He married twice; firstly on 3 April 1745 to Marie Charlotte de La Tour d'Auvergne (20 December 1729 – 6 September 1763), daughter of Emmanuel Théodose de La Tour d'Auvergne and his last wife Louise Henriette Françoise of Lorraine. The couple had one child who married into the Noailles family. Marie Charlotte died of smallpox aged 33 in 1763. The next year, on 14 March, he married again; this time to Marie Charlotte Sylvie de Rohan, a cousin of the Prince of Soubise. The couple had no children. Marie Charlotte Sylvie outliving her husband till 1807.