After transferring from the University of New Hampshire, O'Leary resumed his college football career in 1956 at Colorado College in Colorado Springs, Colorado, playing quarterback for head coach Roy B. Robertson.