Up until 1782 this ancient Saxon church was in the centre of the village of Albury, but when the estate came into the possession of the Honourable William Clement Finch R.N. he enclosed the village green, which formerly came up to the south and west walls of the churchyard, annexed the north-east corner of the churchyard itself to form part of the grounds of the mansion, and embarked on such harassment of the Albury villagers that most of them moved away to Weston Street, a hamlet about half a mile to the west (now called Albury). This harassing process was continued by a later owner, Charles Wall, so that by the time Henry Drummond came to acquire the estate from Wall in 1819 there was little left of the original village save its parish church and the village inn.

The church was by now in a very poor state of repair, and the churchyard too small for the growing population. In 1839 Henry Drummond applied to the Bishop of Winchester for permission to build for the parish a new church at Weston Street (Albury) and to close the old church. That permission having been obtained, no time was lost; before the end of that same year work had begun on the building of the present Albury parish church, and Pugin (then a young man of 27), had been brought in to design the Mortuary Chapel for the Drummond family within the old church.

When Henry Drummond died in 1860 the estate passed to his daughter Louisa. It was her marriage in 1845 to Lord Lovaine, afterwards 6th Duke of Northumberland, that led to the Albury Estate passing later into the hands of the Percy family in 1890, on the death of Lady Lovaine as Duchess of Northumberland.