He was succeeded by his son, Arthur, the fourth Baron, whose son and rightful heir James Annesley was overlooked for the succession by his uncle, Richard Anglesey, the fifth Baron Altham and later sixth Earl of Anglesey.