Kapp was born in New York City where his father Friedrich Kapp, a political activist and later Reichstag delegate for the National Liberal Party, had settled after the failed European revolutions of 1848.
Kapp blev født i New York City, hvortil faderen Friedrich Kapp, politisk aktivist og senere rigsdagsmedlem for det Nationalliberale Parti, var flygtet efter sin deltagelse i revolutionerne i 1848.