Herschel married his cousin Margaret Brodie Stewart (1810–1884) on 3 March 1829 in Edinburgh, and was father of the following children:
# Caroline Emilia Mary Herschel (31 March 1830 – 29 January 1909), who married the soldier and politician Alexander Hamilton-Gordon
# Isabella Herschel (5 June 1831 – 1893)
# Sir William James Herschel, 2nd Bt. (9 January 1833 – 1917),
# Margaret Louisa Herschel (1834–1861), an accomplished artist
# Alexander Stewart Herschel (1836–1907), FRS, FRAS
# Col. John Herschel FRS, FRAS, (1837–1921) surveyor
# Maria Sophia Herschel (1839–1929)
# Amelia Herschel (1841–1926) married Sir Thomas Francis Wade, diplomat and sinologist
# Julia Herschel (1842–1933) married on 4 June 1878 to Captain (later Admiral) John Fiot Lee Pearse Maclear
# Matilda Rose Herschel (1844–1914), a gifted artist, married William Waterfield (Indian Civil Service)
# Francisca Herschel (1846–1932)
# Constance Anne Herschel (1855–20 June 1939), mathematician and scientist who became lecturer in natural sciences at Girton College, Cambridge