Born Charles Bragge, Bathurst was the son of Charles Bragge, of Cleve Hill in Gloucestershire, and his wife Anne Bathurst, the granddaughter of Sir Benjamin Bathurst, younger brother of Allen Bathurst, 1st Earl Bathurst.