Together with his wife, Margaret Gordon the granddaughter of former Governor Walter Patterson, Bannerman returned to the colony of her birth, when he took up the appointment in 1851 as governor of Prince Edward Island.