In 1866, like his father and brother he opposed the Congressional Reconstruction policy, and on that issue left the Republican Party.
I 1866 var han i lighed med sin far og bror modstandere af Kongressens genforeningspolitik, og på den baggrund forlod han Republikanerne.