ByT5: problem with tokenizer.decode()
[ EDIT ]: there is a bug in the 4.11.0. Back to 4.9.2 solves the issue related here (but can create others? like this oneByT5 tokenizer gives indices of chars instead of bytes?)Hi.I’ve created aColab notebookto show a problem when usinggoogle/byt5-smallfrom the model hub of Hugging Face andmodel.generate().Observations:More especifically, the problem comes from the methodtokenizer.convert_tokens_to_string()in source code fortransformers.models.byt5.tokenization_byt5.The same problem happens withgoogle/byt5-base.If someone could run my notebook and tell me what I did wrong or what could be a solution, I would appreciate it because this problem, besides preventing using ByT5 in inference, prevents its finetuning since when evaluating the model at the end of an epoch, the methodtokenizer.convert_tokens_to_string()is called by the script … which suddenly fails). Thanks.cc@patrickvonplaten,@valhalla,@sshleiferScreen shots from the notebookimage1502×562 107 KBimage1720×950 72 KBimage1920×1030 82.4 KB
2021-09-28T17:51:18Z
I just discussed on Twitter withNick Doironthat published the notebookByT5-Finetuning-Datasets.ipynb.He wrote the following tweet on the issue showed in mynotebook:It must be due to recent changes - transformers==4.9.2 works.  decode got brokenThe issue is because instead of 256 byte tokens, there are 3 special bytes (total token ids 0-258) and it isn’t accounting for thoseMy notebook has been updated with the following code and now, it works!pip install transformers==4.9.2I guess that the HF team will solve this issue in the next transformers version.
2021-09-28T18:33:46Z
Issue open in githubByT5: problem with tokenizer.decode() #13779
2021-10-15T11:53:14Z
@patrickvonplatenclosed this issue (seeexplanation) with the return oferrors="ignore"indecode("utf-8", errors="ignore")(seecommit).