How to save a fast tokenizer using the transformer library and then load it using Tokenizers?
I want to avoid importing the transformer library during inference with my model, for that reason I want to export the fast tokenizer and later import it using the Tokenizers library.On Transformers side, this is as easy as tokenizer.save_pretrained(“tok”), however when loading it from Tokenizers, I am not sure what to do.from tokenizers import TokenizerTokenizer.from_file(“tok/tokenizer.json”)Seems to work, but it is ignoring the two other files in the directory: tokenizer_config.json and special_tokens_map.json, for that reason I believe it won’t give me the same tokens.Is there a way to import a tokenizer using the whole directory files ? Or better, can we import a pretrained fast tokenizer from the hub ?Thanks
2021-08-30T06:55:13Z
you can load tokenizer from directory withfrom_pretrainedmethod:tokenizer = Tokenizer.from_pretrained("your_tok_directory")
2021-08-31T17:17:35Z
Thanks for your reply, but I am trying to do is load it using the Tokenizers library rather than transformers
2021-09-01T03:12:41Z
maroxtn:tokenizersit seems that theTokenizerlibrary does not directly support loading from directory,an alternative way is to write a wrapper method that looks fortokenize.jsonin the directory first, then load it via  thefrom_filemethod of Tokenizer  :from os.path import isfile, join
from tokenizers import Tokenizer

def load_tokenizer_from_dir(Tokenizer, your_dir):
    # check the existance of tokenizer file
    if isfile(join(your_dir, "tokenizer.json")):
        return Tokenizer.from_file("tok/tokenizer.json")
    else:
        raise ValueError("tokenier.json not exist in dir {}".format(your_dir))thetransformerslibrary implement  thefrom_pretrainedmethod in similar way
2021-09-01T08:07:38Z
Thank you for the response again!And both tokenizers, on the transformer library and on the tokenizers library, would give exactly the same output ? (if yes, what’s the point then of tokenizer_config and special_tokens_map)
2021-09-01T12:27:25Z
The tokenizer_config contains information that are specific to the Transformers library (like which class to use to load this tokenizer when usingAutoTokenizer). As for the other files, they are generated for compatibility with the slow tokenizers. Everything you need to load a tokenizer from the Tokenizers library is in thetokenizer.json.
2021-09-01T13:17:13Z
Crystal clear. Thank you all for the help and assistance
2022-12-14T23:57:01Z
Can you save a tokenizer from transformers into the tokenizer.json format? For example, GPT2Tokenizer.save_pretrained returns vocab.json, merges.txt, etc. Can I make it save a tokenizer.json so it can be loaded from the tokenizers library instead without transformers?