What's the best way to load a saved Tokenizer json into a transformers PreTrainedTokenizerFast (or other transformers tokenizer)?
This is kind of a cross-library question, so maybe it belongs in the Tokenizers forum instead. But hopefully this is the right place.I’ve made a custom Roberta-style BPE tokenizer for my project using the tokenizers library, with some useful preprocessors and other helpful goodies. I’m able to load the saved json into a tokenizers Tokenizer and it works as expected. But I’d like to load it as a transformers PreTrainedTokenizerFast instead, and I’m not sure if there’s a good way to do that.I can passtokenizer_file="my_tokenizer.json"while creating aPreTrainedTokenizerFast, but it doesn’t seem to read the json for the padding token information, and several methods raise aNotImplementedError, so I assume that class is not meant to be used directly.Making aRobertaTokenizerFastrequires vocab and merges files. So I created those as well, and I can pass them (redundantly), along withtokenizer_file="my_tokenizer.json". But the resulting tokenizer isn’t respecting the json’s add_prefix_space configuration, so I also have to pass inadd_prefix_space=Trueto get my desired behavior.This is making me wonder if I’m doing something wrong here. Is there a way to load a saved Tokenizers json file directly into some kind of transformers tokenizer?
2021-02-25T20:35:01Z
For now, you do have to specify all the information in the init (even if it’s also in the json). We’ll work on making that more seamless in the future.
2021-02-25T21:40:06Z
Got it. Thanks!So to confirm, the best way right now is loading a specific fast tokenizer (e.g.RobertaTokenizerFast) and notPreTrainedTokenizerFastorAutoTokenizer.from_pretrained()(which I don’t believe has a tokenizer_file parameter)?
2021-02-25T23:50:34Z
You will only be able to load withAutoTokenizerafter doing asave_pretrainedonce you have loaded your tokenizer. ThenRobertaTokenizerFastis better because it already has all the default special tokens, whereas you would need to give them all if you usePreTrainedTokenizerFast.