On the death of Thomas Pitt, 1st Baron Camelford, in 1793, the Hall was inherited by his son, also Thomas, and then in 1804 by his daughter Anne, who had married William Grenville, 1st Baron Grenville.