Adding new tokens to a BERT tokenizer - Getting ValueError
I have a Python list namedunique_listthat contains new words that will be added to my tokenizer usingtokenizer.add_tokens. However, when I run my code I’m getting the following error:File "/home/kaan/anaconda3/envs/env_backup/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 937, in add_tokens
    if not new_tokens:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()When I tested with a test array that contains 10 random words, it worked fine but the larger unique_list is causing a problem.What am I doing wrong here?
2021-12-03T22:16:00Z
Not sure if you already solved this issue, but I stumbled upon it today.Looking closely at the error message, looks like theadd_tokens()method expects thenew_tokenspassed to it as a python list rather than an numpy array. Convertingnew_tokensfrom numpy array to a list and then passing it resolved the issue.added_tokens = tokenizer.add_tokens(new_tokens.tolist())
2022-01-16T19:06:31Z
excuse me if I need to solve the problem of not finding the word after tokenizing using BERT … can I use your solution