A native of New York City and a member of the prominent and wealthy Lorillard and Claiborne families, Pell was educated at Connecticut's Pomfret School and attended Harvard University, Columbia University, and New York University.