Loading pretrained SentencePiece tokenizer from Fairseq
Hello. I have a pretrained RoBERTa model on fairseq, which containsdict.txt,model.pt,sentencepiece.bpe.model.I have found a way to convert a fairseq checkpoint to huggingface format inhttps://github.com/huggingface/transformers/blob/master/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.pyHowerver, I couldn’t find a similar method to convert the tokenizer in fairseqsentencepiece.bpe.modelto huggingface’s format.Is there any existing solution? Or do I have to convert it by myself?Thanks.
2020-10-07T09:01:02Z
@proxyhtwere you able to convert sentencepiece model to huggingface tokenizer. As I am facing similar issues as well.
2020-10-07T16:16:57Z
A colleague of mine has figured out a way to work around this issue. Although both Huggingface and Fairseq use spm from google, the tokenizer in Fairseq map the id from spm to the token id in thedict.txtfile, while Huggingface’s does notWe will have to write a custom Tokenizer in Huggingface to simulate the behavior as in Fairseq
2020-10-07T20:12:34Z
Thanks. I will look into creating the custom tokenizer then using sentence piece model and tokenize data before passing it to training.
2020-10-09T18:52:18Z
@proxyhtseems like new version released today added support for loading sentenncepiece model. Details in this PRhttps://github.com/huggingface/tokenizers/pull/292Haven’t tested myself yet. Plan to test in a week or so. Let me know how it goes if you test before.
2020-10-21T12:55:38Z
For reference in tokenizer version 0.9.2 there is a function to load spm model. You need to run these two commands before to install the dependenciespip install protobufwgethttps://raw.githubusercontent.com/google/sentencepiece/master/python/sentencepiece_model_pb2.pythen you can instantiate huggingface tokenizer using pretrain sentencepiece model astok = tokenizers.SentencePieceUnigramTokenizer.from_spm("spm.model")