Lom was born in Prague, Austria-Hungary (now Czech Republic) on September 11, 1917 to Karl ze Schluderpacheru and to Olga (née Gottlieb). He was raised in London, England. Lom was married three times. He was married to Dina Shea from 1948 until they divorced in 1971. Lom had three children. Lom was friends with actor Peter Sellers.