Downey akaberekwa muManhattan, New York, ari muduku pavana vaviri. Baba vake, Robert Downey Sr., mutambi uye mugadziri wemufirimu, apo amai vake, Elsie Ann (née Ford), vaive mutambi akaonekwa mumafirimu aDowney Sr. Baba vaDowney ndevehafu yechiLithuanian Jewish, chikamu chimwe chechina chechiHungarian Jewish, uye chikamu chimwe chechina cheIrish chizvarwa, apo amai vaDowney vaive neScotish, chiGerman, neSwitzerland madzitateguru. Zita remhuri rekutanga raRobert raive Elias rakashandurwa nababa vake kunyoresa muuto. Downey nehanzvadzi yake mukuru Allyson vakakurira muGreenwich Village.
Elsie Ann