He had six children with his first wife Eliza Davison, the daughter of a Scotch-Irish farmer, the most prominent of which were oil tycoons John D. Rockefeller and William Rockefeller Jr., the co-founders of Standard Oil.