He married Harriette Osborne, daughter of the Irish baronet Sir Daniel Toler Osborne and Harriette Trench, daughter of the Earl of Clancarty in Dublin in 1839; they had two sons (Norman survived) and three daughters, Louise (1841–1878), Rachel (1845–1882) and Alice.