Scruggs was born on January 6, 1924 in Flint Hill, North Carolina. He was raised in Shelby, North Carolina. Scruggs was married to Louise Certain from 1948 until her death in 2006. They had two sons. Scruggs died on March 28, 2012 in Nashville, Tennessee from natural causes, aged 88.