Why do different tokenizers use different vocab files?
I’ve trained aByteLevelBPETokenizer, which output two files:vocab.jsonandmerges.txt. I want to use this tokenizer with an XLNet model.When I tried to load this into anXLNetTokenizer, I ran into an issue. TheXLNetTokenizerexpects the vocab file to be a SentencePiece model:VOCAB_FILES_NAMES = {"vocab_file": "spiece.model"}I found this confusing, since it’s possible to train aByteLevelBPETokenizerthen load it into aRobertaTokenizerFast:tokenizer = RobertaTokenizerFast.from_pretrained("./EsperBERTo", max_len=512)I looked at the source for the RobertaTokenizer, and theexpected vocab files match the output of theByteLevelBPETokenizer:VOCAB_FILES_NAMES = {
    "vocab_file": "vocab.json",
    "merges_file": "merges.txt",
}Why do these tokenizers expect different vocab file formats? Are they intended to be used in different ways?