Change Gemma tokenizer unused token
I want to add a new token to the gemma tokenizer let’s call it<newtoken>. I would like to take advantage of the fact that the vocabulary of this tokenizer has unused tokens (<unused0>,<unused1>, …).The first thing I tried was toadd_token("<newtoken>")but this only append the token to the end of the vocabulary instead of replacing the unused ones.Is there an expected way to achieve the replacement of the unused tokens? Something like:add_token("<newtoken>", replace_unused=True)