The property was originally held by the Comyns, and it is thought that there was a royal castle here in the 13th century - the Exchequer Rolls mention repair of the royal castle in anticipation of Haakon IV of Norway’s invasion of 1263.