 He actually had returned the $20,000 several months later. And when that came out, there was immediate bipartisan condemnation of it. This was such a controversy that it ultimately led to Justice Fortas resigning from the Supreme Court. Here's what the law says. Any justice shall disqualify himself in any proceeding in which his impartiality might reasonably be questioned.