His older brother, Ephraim Kirby Smith (1807–1847), who graduated from West Point in 1826 and was a captain in the regular army, served with him in the 5th U.S. Infantry in the campaigns with both Taylor and Scott.