The Manor of Stobo had been the family seat of the Graham-Montgomery Baronets from 1767, and the castle itself was constructed between 1803 and 1811 for his grandfather Sir James Montgomery, 2nd Baronet, then remodelled for his father who also had the grounds redesigned in 1872.