Fragmentary ruins of the old Stornoway Castle had survived in the bay until that time, and can even be seen in Victorian photographs, but Matheson destroyed them in 1882, in order to expand the harbour; a few remains of Stornoway Castle still remain, hidden beneath pier number 1, close to the shore, slightly west of centre.