Early in 1866, the people of Brewster accused Alger of sexually molesting two boys. These boys were 13 and 15 years old. Three men of the church discovered that this was true. Alger said that he had been "unwise", and quit his job with the Brewster church. He left town quickly, and went to his parents' home in South Natick, Massachusetts.