Lettsom was born into a Quaker family at Fulham in March 1805.  His paternal grandfather John Coakley Lettsom was a famous physician, philanthropist and abolitionist who held that sea-bathing was good for public health.  His maternal grandfather − with whom he lived in his youth − was Sir William Garrow the celebrated criminal defender, afterwards a judge, who introduced the phrase "presumed innocent until proven guilty" into the common law and whose life inspired the television drama series Garrow's Law.   Lettsom was educated at Westminster School and Cambridge University.