From September 1870 to September 1872, Sumner was Rector of the Church of the Redeemer in Morristown, N.J. On April 17, 1871, Sumner married Jeannie Whittemore Elliott, daughter of Henry H. Elliott of New York City.