His conversion to the Republican Party and his cooperation with his old friend, President Ulysses S. Grant, as well as critical comments he wrote in his memoirs about General Lee's wartime performance, made him anathema to many of his former Confederate colleagues.
Hans skifte til det Republikanske Parti og hans samarbejde med sin gamle ven præsident Ulysses S. Grant samt de kritiske bemærkninger, som han skrev i sine erindringer om general Lees indsats i krigen, betød, at han blev bandlyst af mange af sine tidligere kolleger i sydstatshæren.