WordLevel error: Missing [UNK] token from the vocabulary
Hi, I am trying to train a basic Word Level tokenizer based on a filedata.txtcontaining5174 5155 4749 4814 4832 4761 4523 4999 4860 4699 5024 4788 [UNK]When I run my codefrom tokenizers import Tokenizer
from tokenizers.models import WordLevel

tokenizer = Tokenizer(WordLevel(unk_token='[UNK]'))
tokenizer.train(files=['data.txt'])
tokenizer.encode('5155')I get the errorException: WordLevel error: Missing [UNK] token from the vocabularyWhy is it still missing despite having[UNK]indata.txtand also settingunk_token='[UNK]'?Any help is very appreciated!
2021-06-17T20:48:16Z
Hi Athena, I’m having the same issue… did you find the root of the problem?
2021-08-26T20:02:29Z
I am experiencing this too
2022-09-21T10:50:30Z
Having the same issue…
2022-10-27T09:09:12Z
from tokenizers import Tokenizer
from tokenizers.models import WordLevel
from tokenizers.trainers import WordLevelTrainer

tokenizer = Tokenizer(WordLevel(unk_token='[UNK]'))

########## Specify [UNK] here ############
trainer = WordLevelTrainer(
    special_tokens=['[UNK]']
)
##########################################

files = ['./datasets/AAABBBCCC.txt']
tokenizer.train(files, trainer) # <--- specify trainer
tokenizer.encode('41').ids