Israel Isaac Rabi was born on July 29, 1898, into a Polish-Jewish Orthodox family in Rymanów, Galicia, in what was then part of Austria-Hungary but is now Poland. Soon after he was born, his father, David Rabi, emigrated to the United States. The younger Rabi and his mother, Sheindel, joined David there a few months later, and the family moved into a two-room apartment on the Lower East Side of Manhattan. At home the family spoke Yiddish. When Rabi was enrolled in school, Sheindel said his name was Izzy, and a school official, thinking it was short for Isidor, put that down as his name. Henceforth, that became his official name. Later, in response to anti-Semitism, he started writing his name as Isidor Isaac Rabi, and was known professionally as I.I. Rabi. To most of his friends and family, including his sister Gertrude, who was born in 1903, he was known simply by his last name. In 1907, the family moved to Brownsville, Brooklyn, where they ran a grocery store.
As a boy, Rabi was interested in science. He read science books borrowed from the public library and built his own radio set. His first scientific paper, on the design of a radio condenser, was published in Modern Electrics when he was in elementary school. After reading about Copernican heliocentrism, he became an atheist. "It's all very simple", he told his parents, adding, "Who needs God?" As a compromise with his parents, for his Bar Mitzvah, which was held at home, he gave a speech in Yiddish about how an electric light works. He attended the Manual Training High School in Brooklyn, from which he graduated in 1916. Later that year, he entered Cornell University as an electrical engineering student, but soon switched to chemistry. After the American entry into World War I in 1917, he joined the Student Army Training Corps at Cornell. For his senior thesis, he investigated the oxidation states of manganese. He was awarded his Bachelor of Science degree in June 1919, but since at the time Jews were largely excluded from employment in the chemical industry and academia, he did not receive any job offers. He worked briefly at the Lederle Laboratories, and then as a bookkeeper.