Word ids of BioGPT model
As there is no Fast tokenizer availabale for BioGPT, I am not able to get the word_ids. Does anyone have any idea of how to get the word_ids?
2023-05-07T17:33:06Z
Hi,BioGPT doesn’t have a fast tokenizer implementation yet:Unable to convert BioGpt slow tokenizer to fast: token out of vocabulary · Issue #21838 · huggingface/transformers · GitHub. To contribute this, a new BioGPTConverter class would have to be defined here:transformers/convert_slow_tokenizer.py at main · huggingface/transformers · GitHub.