Add_tokens + finetune
I use the huggingface Trainer class to finetune a mT5 model. Since my data (AMR graphs) contains mayn tokens like:ARG0,:op2or:name, which are generally split into wordpieces I added those 219 tokens to the tokenizer withfrom  transformers import MT5ForConditionalGeneration, MT5Tokenizer
tokenizer = MT5Tokenizer.from_pretrained("google/mt5-base")
model     = MT5ForConditionalGeneration.from_pretrained("google/mt5-base")
toks = [ ... my ist of tokens ...]
tokenizer.add_tokens(toks, special_tokens=False)
model.resize_token_embeddings(len(tokenizer))and than starting the training.Once the model is finetuned I run my test, and once again I add the same tokens in the same order to the tokenizer. But the result is catastrophic, it drops from 80% F1 to 50%, so evidently something is going wrong. I compared the tokenisation with and without added tokens and this looks OK. But I have not the slightest idea where to check. Can you give me a hint about the error I’m committing ?I thought (possibly erroneously) that the added tokens will get random vectors which will be updated during the finetuning. If this is not the case, is there a way to do this? If not what is the point of adding new tokens?Could anybody elaborate on this ?Environment infotransformersversion: 4.11.3Platform: Linux-5.13.0-30-generic-x86_64-with-glibc2.17Python version: 3.8.12PyTorch version (GPU): 1.9.1Jax version: not installedJaxLib version: not installedUsing GPU in script?: yesUsing distributed or parallel set-up in script?: no