Are BERT models pretrained with Whole Word Masking?
image1748×1196 295 KBAre BERT models in Transformers pretrained with Whole Word Masking?
2020-11-30T14:03:17Z
It depends on the checkpoint you are using, we provide both version. For instancebert-base-uncasedis the first BERT model pretrained without WWM, butbert-large-uncased-whole-word-maskingis pretrained with WWM. Check all the checkpoints availablehere