The 4th Duke 's younger son Lord George Cavendish and a Devonshire in - law each used the house for at least two separate spells .
Devonshire in - law is the 3rd Duke of Portland .
The 4th Duke 's younger son Lord George Cavendish and a Devonshire in - law , the 3rd Duke of Portland , each used the house for at least two separate spells .