Nutt was in love with Lavinia Warren, another little person like him, at the American Museum. Lavinia was several years older than Nutt. She thought of him only as a "nice little boy". This in some ways, made Nutt depressed later on in life. She married General Tom Thumb in a spectacular wedding masterminded by Barnum in 1863. Nutt went to the wedding as Thumb's best man. He stayed away from women for a long time after the wedding. In 1879, he married Lilian Elston of Redwood City, California.