Born Lady Anne Palmer in Westminster, she was the first child of Barbara Villiers, the only child of William Villiers, 2nd Viscount Grandison, and the wife of Roger Palmer, 1st Earl of Castlemaine, who was also one of the mistresses of King Charles II.