He was born on May 7, 1936 in Collinsville, Mississippi. His brother was born 2 years later. He first started singing with the gospel group, mostly on sessions but he also recorded singles for Miracle label. Ruffin was then drafted for national service. After he left the Army in 1964, Ruffin returned to Motown and was offered the opportunity to join the Temptations to replace Elbridge Bryant. They instead hired Ruffin's brother David after hearing him. Jimmy decided to continue his solo career and recorded for the Soul label, but he didn't have much success.