Cary was the son of Sir Edward Cary, of Berkhamstead and Aldenham, Hertfordshire, Master and Treasurer of His Majesty's Jewels, and his wife Catherine Knevet or Katherine Knyvett, daughter of Sir Henry Knevet or Knyvett, Master of the Jewel Office to Queen Elizabeth and King James, and wife Anne Pickering, and widow of Henry Paget, 2nd Baron Paget.