BertTokenizerFast for stsb-xlm-r-multilingual model
Hi community,Would there be a fast tokenizer for thestsb-xlm-r-multilingualmodel?Thanks !
2021-04-08T07:04:46Z
Hi community and@lewtun,Could anyone have an idea on how to get a fast tokenizer forstsb-xlm-r-multilingualmodel?I am blocked with low latency response due to tokenizer computation. Is there a fast tokenizer model asBertTokenizerFastor is there a way to run tokenizer on GPU ?
2021-04-08T07:38:50Z
hey@Matthieu, as far as i know the “fast” refers to the rust implementations of the tokenizers:tokenizers/tokenizers at master · huggingface/tokenizers · GitHubthere are bindings for python, so perhaps you can adapt the suggestionhereto your use case? e.g. download thetokenizer.jsonfile for stsb and load the fast version as follows:from transformers import RobertaTokenizerFast
tokenizer = RobertaTokenizerFast(tokenizer_file="tokenizer.json")(i’m not super familiar with thestsb-xlm-r-multilingualmodel but am assuming it’s using the same tokenization strategy as XLM-R)
2021-04-08T21:19:47Z
Hi@lewtunthanks. I finally found that there is axlmrobertatokenizerfastimplementation.