Sullivan was engaged to swimmer Sybil Bauer, but she died of cancer in 1927 at the age of 23. He married Sylvia Weinstein on April 28, 1930. She died on March 16, 1973. They had a daughter, Betty Sullivan. Betty married the "Ed Sullivan Show"'s producer, Bob Precht.