Miller was born in 1915 in Melbourne, Victoria, the son of Septimus and Helen Miller and he attended Xavier College as a young man where he became friends with Sam Calder whom he would remain friends with for the rest of his life.