Using a BertWordPieceTokenizer trained from scratch from transformers
Hey everyone,I’d like to load aBertWordPieceTokenizerI trained from scratch using the interface built intransformers, either withBertTokenizerorBertTokenizerFast. It looks like those two tokenizers intransformersexpect different ways of loading in the saved data fromBertWordPieceTokenizer, and I am wondering what is the best way to go about things.ExampleI am training on a couple test files, saving the tokenizer, and the reloading it intokenizers.BertTokenizer(there is a bit of ceremony here creating the test data, but this is everything you need to reproduce the behavior I am seeing):from pathlib import Path

from tokenizers import BertWordPieceTokenizer
from transformers import BertTokenizer


def test_text():
    text = [
        "This is a test, just a test",
        "nothing more, nothing less"
    ]

    return text


def create_test_files():
    test_path = Path("tmp")
    test_path.mkdir()

    test_data = test_text()

    for idx, text in enumerate(test_data):
        file = test_path.joinpath(f"file{idx}.txt")
        with open(file, "w") as f:
            f.write(text)

    return test_path


def cleanup_test(path):
    path = Path(path)

    for child in path.iterdir():
        if child.is_file():
            child.unlink()
        else:
            rm_tree(child)

    path.rmdir()


def create_tokenizer_savepath():
    savepath = Path("./bert")
    savepath.mkdir()
    return str(savepath)


def main():
    # Saving two text files to train the tokenizer
    test_path = create_test_files()

    files = test_path.glob("**/*.txt")
    files = [str(f) for f in files]

    tokenizer = BertWordPieceTokenizer(
        clean_text=True,
        strip_accents=True,
        lowercase=True,
    )

    tokenizer.train(
        files,
        vocab_size=15,
        min_frequency=1,
        show_progress=True,
        special_tokens=["[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"],
        limit_alphabet=1000,
        wordpieces_prefix="##",
    )

    savepath = create_tokenizer_savepath()
    tokenizer.save_model(savepath, "pubmed_bert")

    tokenizer = BertTokenizer.from_pretrained(
        f"{savepath}/pubmed_bert-vocab.txt",
        max_len=512
    )

    print(tokenizer)

    cleanup_test(test_path)
    cleanup_test(savepath)


if __name__ == "__main__":
    main()Loading the Trained TokenizerSpecifying the path to thepubmed_bert-vocab.txtis deprecated:Calling BertTokenizer.from_pretrained() with the path to a single file or url is deprecated
PreTrainedTokenizer(name_or_path='bert/pubmed_bert-vocab.txt', vocab_size=30, model_max_len=512, is_fast=False, padding_side='right', special_tokens={'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'})But, if I just specify the path to the directory containingpubmed_bert-vocab.txt:Traceback (most recent call last):
  File "minimal_tokenizer.py", line 86, in <module>
    main()
  File "minimal_tokenizer.py", line 76, in main
    max_len=512
  File "/home/ygx/opt/local/anaconda3/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1777, in from_pretrained
    raise EnvironmentError(msg)
OSError: Can't load tokenizer for 'bert'. Make sure that:

- 'bert' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'bert' is the correct path to a directory containing relevant tokenizer filesThe directory I am saving to only containspubmed_bert-vocab.txt. If specifying the full path to that vocab is deprecated, what is the best way to load that tokenizer?Using BertTokenizerFastIf I swap out BertTokenizer for BertTokenizerFast, and pass in the path to the directory where I have saved my tokenizer trained from scratch, I get the same error:Traceback (most recent call last):
  File "minimal_tokenizer.py", line 86, in <module>
    main()
  File "minimal_tokenizer.py", line 76, in main
    max_len=512
  File "/home/ygx/opt/local/anaconda3/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1777, in from_pretrained
    raise EnvironmentError(msg)
OSError: Can't load tokenizer for 'bert'. Make sure that:

- 'bert' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'bert' is the correct path to a directory containing relevant tokenizer filesAnd if I specify the path to file saved by my tokenizer (pubmed_bert-vocab.txt), I get a ValueError (vs the deprecation warning I was getting using BertTokenizer):Traceback (most recent call last):
  File "minimal_tokenizer.py", line 86, in <module>
    main()
  File "minimal_tokenizer.py", line 76, in main
    max_len=512
  File "/home/ygx/opt/local/anaconda3/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1696, in from_pretrained
    "Use a model identifier or the path to a directory instead.".format(cls.__name__)
ValueError: Calling BertTokenizerFast.from_pretrained() with the path to a single file or url is not supported.Use a model identifier or the path to a directory instead.Current ApproachI am currently usingBertTokenizer, specifying the full path the thepubmed_bert-vocab.txtand am ignoring the deprecation warning, but ideally I would like useBertTokenizerFastbut don’t know how to load in my saved tokenizer. What is the best way to go forward on this?
2021-03-15T08:21:09Z
pinging@anthony
2021-03-26T14:30:59Z
Should I open an issue on the tokenizers repository for this?