 To knock the spots off something means to beat easily or completely outdo, and it began in America in the mid 1800s. Carnivals were commonplace all over the country at the time, and the most popular sideshow was the shooting gallery. All comers would test their marksmanship skills, and the most used target was a playing card, the face of which had spots or marks on it to indicate the suit or value of the card.