 Every time he captured a ship, Fly forced the crew to serve him. It's called impressment. For example, he forced one guy named William Atkinson to be his navigator. Eventually, all those unwilling and probably unhappy sailors outnumbered Fly's original crew.