Eventually enough of Custer 's men were amassed to break down the fence .
Men caused the Virginians to retreat .
Eventually enough of Custer 's men were amassed to break down the fence , and they caused the Virginians to retreat .