I le 1813, na faaipoipo ai Ruffin iā Susan Hutchings Travis o Williamsburg. Na siitia le ulugalii i se faatoaga na maua e Ruffin mai lona tamamatua, i Coggin's Point, i le vaitafe o James i Prince George County, lea na maitauina o le amataga o Benedict Arnold ina ua uma ona ia fesuiaʻi le faamaoni ma osofaia le vaitafe o James (e aofia ai Richmond) i le 1781. E toʻasefulu tasi le la fanau a o leʻi maliu Susan Ruffin i le 1846, e ui o le toʻatele o le a feoti a o leʻi uma le American Civil War ma ua pule i le ola lo latou tama: Edmund Ruffin Jr. (1814-1875), Agnes Ruffin Beckwith (1816<unk>1865), Julian Calx Ruffin (1821<unk>1864, na maliu i le Taua o Drewry's Bluff
), Elizabeth Ruffin Sayre (1821<unk>1860), Rebecca Ruffin Bland (1823<unk>1855), Jane Ruffin Dupuy (1829<unk>1855), Charles Lorraine Ruffin (1832<unk>1870; VMI faauu na sao mai le taua), Ella Ruffin (1832<unk>1855), Ann Ruffin (1844<unk>1863) ma George Champion Ruffin (1845-1913).