Brouthers was born in Sylvan Lake, New York, to Michael and Annie Brooder, Catholic immigrants from Ireland; upon arriving in New York, Michael Brooder was recruited to Dutchess County to work in open pit iron mines in the town of Beekman.