Historians are unsure why, but Hall's life changed. By early 1862, his marriage was in trouble. Biddy left Ben and moved in with a neighbour, Jim Taylor, taking young Henry with her. There were many criminals living and working in the area where Hall lived. He became friends with Frank Gardiner. Gardiner was already wanted for robbery and had shot two policemen before escaping. On 14 April 1862 Gardiner and Hall robbed three bullock wagon drivers. One week later the drivers saw him at the Forbes horse races. Police Inspector Sir Frederick Pottinger, who was also at the races, told the police to arrest Ben for using guns in a robbery with Gardiner. The jury at the courthouse at Orange did not think there was enough evidence to show that Hall had been one of the robbers. After he was released, the police kept a close watch on Ben Hall, to see where he went and what he was doing.