Gilman has been married three times, to the former Jane Prizant (1927–2000), a lawyer and daughter of a well-known actor of the Yiddish theater (), Rita Kelhofer, and Georgia Tingus. He had five children.