Built and remodelled over several centuries, it was started around 1480 for George Ogilvie, and served as a seat of the Ogilvies, who had succeeded the Abbots of Kinloss as the feuars of the district after the Scottish reformation.