In 1849, along with his eldest brother William Montgomery Vermilye and George Carpenter, he founded the firm of Carpenter & Vermilye, which became one of the most prominent banking houses in New York City and was known for selling war bonds during the U.S. Civil War.