Tokenizer from tokenizers library cannot be used in transformers.Trainer
Hi,I am trying to train my own model withTrainerwith a pre-trainedSentencePieceBPETokenizerfromtokenizerslibrary. However, it is missing several attributes as well as methods (e.g.,pad), which makes it incompatible withtransformers.Trainer. Is there an easy way to convert it toPretrainedTokenizerfromtransformers?Thanks!
2021-07-28T17:59:47Z
If you want SentencePiece Tokenizer, you should use thesentencepiece library, then pass in the trained model as a parameter into the desired tokenizer model like T5, Bart etc. By doing this the vocab will be yours and the desired tokenizer will handle the padding, I am not sure about whether it will handle the special tokens though.
2021-07-30T11:21:27Z
sguggerhelped with the solution, we simply needtransformers.PretrainedTokenizerFast(tokenizer_object=my_tokenizer).