He was survived by five children from two marriages, and his twin brother, Byron. A book about Spencer's life called "Gross Misconduct: The Life of Spinner Spencer", was written by Martin O'Malley. The book was later adapted into a made-for-television movie in Canada called Gross Misconduct by Atom Egoyan.