 During pre-training, the learning mechanism is basically to maximize the probability score that will be assigned to the correct sequence of words, meaning the exact sequence of words that happen to be on the internet. Now, there's really no reason why that's the only correct sequence of words, by the way, because for any given prefix text, there can be another different word that could