His younger brother John was the first Roman Catholic bishop in the United States (as Bishop of Baltimore, 1790) and founder of Georgetown University; his cousin Charles Carroll of Carrollton signed the Declaration of Independence.