Charlotte Lee was born Charlotte Fitzroy, on 5 September 1664, the fourth child and second daughter of Barbara Palmer, Countess of Castlemaine, the only child of the Royalist commander William Villiers, 2nd Viscount Grandison.