Moving tokenizers between machines
Because of some restrictions I have, I need to use the Huggingface libraries on a server without network access. This is extremely cumbersomeWhat I am trying to do is to download each model, dataset and tokenizer locally, save them and move them to the server. But I am having an issue when moving tokenizers because I still get references to local paths. For instance, I tried doing this:from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
tokenizer.save_pretrained('/Users/andrea/data/models/bert-base-uncased')I thought that the content of/Users/andrea/data/models/bert-base-uncasedwould be enough to reload the tokenizer elsewhere, but unfortunately this is not the case. In particular,/Users/andrea/data/models/bert-base-uncased/tokenizer_config.jsoncontains{
  "tokenizer_file": "/Users/andrea/.cache/huggingface/transformers/534479488c54aeaf9c3406f647aa2ec13648c06771ffe269edabebd4c412da1d.7f2721073f19841be16f41b0a70b600ca6b880c8f3df6f3535cbc704371bdfa4"
}which is a path on my local machine.Now, I could rewrite it, move that file as well, and so on, but still this is unconvenient and I am not sure I will be missing something else. Is there areliableway to move a tokenizer on a different machine in such a way that it will be usable there without internet access?