That's why so many Labour MPs feared a miscalculation when their party's leader brought his old comrade back into the fold.
But his staggering comments today about Hitler and anti-Semitism crossed a line - they were enough for Jeremy Corbyn to suspend him.
But the problem for the leader doesn't end with that act.
No one believes that Jeremy Corbyn himself tolerates discrimination against Jews.
But on repeated occasions Labour has been slow and clumsy in closing down cases of anti-Semitism among its members when they emerge.
Any moments of delay or doubts about the leadership's determination, open the window a tiny crack to the kind of intolerance that the vast majority of the Labour Party, and indeed the public, find appalling.
Public denials that there is even a issue could make it even worse. Perhaps in politics as in normal life, the first step towards fixing a problem is acknowledging that it exists.
And with only a week before Jeremy Corbyn's first big test at the polls, In elections in London, Scotland, Wales, and al round England, it's the kind of mess, and political distraction Labour could do without.

A southbound Volkswagen Crafter Van collided with a Peugeot Partner Van on the A702, just north of Candymill Road, near Biggar, at 17:35 on Thursday.
A 58-year-old male passenger in the Peugeot is in a serious but stable condition in hospital.
The 32-year-old driver was uninjured as was the 26-year-old male driver of the Volkswagen van.
Police said both vehicles suffered extensive damage in the collision and the road was closed for several hours while officers investigated the scene.

The three children were placed in foster care after the couple's arrest by counter-terrorism officers in 2015.
Mr Justice Cobb said he was suspicious the parents were going to areas controlled by so-called Islamic State.
But he said their "attitude" had "genuinely changed" and he concluded that the children should return home.
The judge, who is based in the Family Division of the High Court, said he was also satisfied that the couple will meet their children's needs.
He said evidence showed that being in the family home will be in their best interests.
Details of the case have emerged in rulings published by Mr Justice Cobb following private family court hearings in Leeds.
The judge did not identify the couple. He said they were from London but lived in the north-east of England.
Council social workers had asked him to make decisions about where the children should live.
The couple were arrested while in a car with their children at the Eurotunnel terminal in Folkestone, Kent, during the summer of 2015.
They had not been accused of any terrorism-related offences as a result of their arrest, but had faced fraud charges.
The couple had told police that they were travelling to Germany to visit relatives, although the man admitted that he had been "drawn into radical extreme thinking".
Mr Justice Cobb said evidence caused him "very considerable concern" and he said he was "suspicious" that the couple had been heading for areas of the Middle East controlled by IS.
But he said suspicion was not enough.
Mr Justice Cobb said he had been told how the couple had co-operated with social services after the children were taken into foster care.
The father said he accepted that things he had said before were "not appropriate" and "not a true expression of Islam", and the mother had accepted that views she expressed "were wrong".
"I am sufficiently satisfied... that the attitudes of these parents have genuinely changed," said Mr Justice Cobb.
"I am satisfied from the information placed before me that these parents [will meet] the needs of these... children in all material respects."
He added: "All relevant considerations point to the conclusion that it is indeed in the best interests of the children that they should [be] at home in the care of their parents."
For years it has been more surprising when Ken Livingstone hasn't raised hackles than when he has.
[NEXT_CONCEPT]
Police have appealed for witnesses to a crash between two vans which closed a Lanarkshire road for several hours.
[NEXT_CONCEPT]
A Muslim couple arrested over fears that they were heading to Syria for "extremist activities" have won a court fight for the care of their children.