Fischer was born in Michael Reese Hospital in Chicago, Illinois on March 9, 1943. His birth certificate said that his father was Hans-Gerhardt Fischer, a German. His mother, Regina Wender Fischer, was a Polish-Jewish American citizen. Born in Switzerland, she grew up in St. Louis, Missouri, and later became a teacher, a nurse, and a doctor. The two married in 1933 in Moscow, USSR, where Regina was studying medicine at the First Moscow Medical Institute. They divorced in 1945 when Bobby was two years old, so he grew up with his mother and older sister. In 1948, the family moved to Mobile, Arizona, where Regina taught in an elementary school. The next year, they moved to Brooklyn, New York.