His family home, Retreat, was also located in Charles County, near one of the largest slave-trading ports of the era, Port Tobacco; it was built in the last quarter of the 18th century and is listed on the National Register of Historic Places.