 The way that it's trained is it's supposed to maximize the probability score that it should assign to the correct word. If it predicted the wrong word, it means the wrong word got the higher probability. But all that there is is to raise the probability of the correct word that should have happened instead.