Reused tokenizer returns unk
HelloI’m training a tokenizer from an old one (Bert based)the new tokenizer returns [UNK] for words already exist in vocabulary and run correctly with the old tokenizerfrom transformers import AutoTokenizer

old_tokenizer = AutoTokenizer.from_pretrained("asafaya/bert-base-arabic")

tokens = old_tokenizer.tokenize('مع')
tokensreturns [‘مع’]while  with the new tokenizernew_tokenizer= old_tokenizer.train_new_from_iterator(training_corpus, 10)
tokens = new_tokenizer.tokenize('مع')
tokensreturns [‘[UNK]’]Can any one help me please !
2024-03-14T21:41:51Z
I am also facing similar issue. Were you able to sort it our ?