We now have a tokenizer trained on the files we defined. We can either continue using it in that runtime, or save it to
a JSON file for future re-use.
Loading directly from the tokenizer object
Let's see how to leverage this tokenizer object in the 🤗 Transformers library. The
[PreTrainedTokenizerFast] class allows for easy instantiation, by accepting the instantiated
tokenizer object as an argument:
thon