He was married to Ekaterina Voroshilova (born Golda Gorbman). She came from a Jewish Ukrainian family from Mardarovka. She changed her name when she converted to Orthodox Christianity in order to be allowed to marry Voroshilov. During Stalin's rule, the couple lived in the Kremlin at the Horse Guards.