Walpole was Townshend's neighbour in Norfolk as well as his brother-in-law, but political and personal rivalry between them was exacerbated by Walpole's building of Houghton Hall As Lord Hervey observed, "Lord Townshend looked upon his own seat at Raynham as the metropolis of Norfolk, and considered every stone that augmented the splendor of Houghton, as a diminution of the grandeur of Raynham."