Peabody served from 1884 to 1930 as a trustee of Hampton University, one of Virginia's historically black universities, where he established in the university library the Peabody Collection of rare materials on African-American history, one of the largest collections in the United States.