 And the fly then remarks, all you've done is added insult to injury, which is where the expression comes from to mean making an already bad situation even worse. The phrase began in 25 BC, didn't pass into English until about the mid 1700s. The saying against the grain comes from tradesmen. It means against the natural flow or opposed to one's normal inclination.