In 1881, the house was still owned by the head of the Willoughby family, Digby Willoughby, 9th Baron Middleton, but by then it was "too near the smoke and busy activity of a large manufacturing town... now only removed from the borough by a narrow slip of country", so that the previous head of the family, Henry Willoughby, 8th Baron Middleton, had begun to let the house to tenants and in 1881 it was vacant.