He married the sculptor Lilian Swann Saarinen. They divorced in 1954. He had two children from his first marriage, Eric and Susan. Saarinen then married Aline Bernstein Louchheim (March 25, 1914 – July 13, 1972), an art critic at "The New York Times". They had a son, Eames, named after his collaborator Charles Eames.