In 1825, Bannerman married Margaret Gordon, who was later identified as "Carlyle's first love" by her biographer, who tells of the young schoolmaster Thomas Carlyle, in Kirkcaldy, Scotland, "who was attracted by her intelligence and wit."