Richard and James Rigby were sons of Edward Rigby of Mistley Hall, a London draper and landowner based in Covent Garden, and Anne Hyde, a close cousin of Queen Anne (Hyde), Queen Anne, Queen Mary, and the Earl of Clarendon.