Kammala jumla mai zuwa: An haifi Grant a Birnin New York, New York, ɗan Gabriel Grant, likita da likitan tiyata na Amurka, da Caroline Manice. Mahaifiyar Madison Grant ta kasance zuriyar Jessé de Forest, Huguenot na Walloon wanda a cikin 1623 ya ɗauki rukunin farko na masu mulkin mallaka don su zauna a New Netherland, yankin Jamhuriyar Holland a gabar gabashin Amurka. A gefen mahaifinsa, magajin Madison Grant na farko na Amurka shi ne Richard Treat, shugaban Ikilisiyar Pitminster a Ingila, wanda a 1630 ya kasance daya daga cikin 'yan Puritan na farko na New England. Kakannin Grant ta hanyar layin Treat sun hada da Robert Treat (gwamnan mulkin mallaka na New Jersey), Robert Treat Paine (wanda ya sanya hannu kan Yarjejeniyar Independence), Charles Grant (kakan Madison Grant, wanda ya yi aiki a matsayin jami'i a cikin War of 1812), da kuma Gabriel Grant (mahaifin Madison), wani shahararren likita da kwamishinan kiwon lafiya na Newark, New Jersey. Dokta Gabriel Grant an ba shi lambar yabo ta girmamawa yayin da yake aiki a matsayin likitan tiyata tare da 2nd New Jersey Volunteers a cikin yakin basasa. A lokacin da
Yaƙin Fair Oaks, ya motsa wadanda suka ji rauni zuwa aminci yayin da yake ƙarƙashin wuta mai ƙarfi. Grant ya kasance mazaunin New York City ne na tsawon rayuwarsa.