It was built in the years 1844–51 as a country house for Sir James Matheson who had bought the whole island a few years previously with his fortune from the Chinese Opium trade.
Det blev bygget i 1847-1857 som et sommerslot til Sir James Matheson, som havde købt hele øen et par år tidligere for sin formue, der var tjent ved opiumshandel i Kina.