The castle eventually passed out of ownership of the Dukes of Ormonde in the early eighteenth century before it was rebuilt in its entirety in 1816-9 by the Agar family, Viscounts Clifden of Gowran (1781-1974).