Ferrer was married to Rosemary Clooney, aunt of George Clooney, from 1954 through 1961 and again from 1964 through 1967. They had a son, actor Miguel Ferrer. In 1985 he received the National Medal of Arts from President Ronald Reagan, becoming the first actor to receive that honor.