How to map generated characters to tokens?
Is there an equivalent ofBatchEncoding.char_to_tokenfor the outputs ofmodel.generate?e.g. suppose I had the following output by callingtokenizer.decode(model.generate(...)["sequences"]):"the cat sat on the mat"where the output oftokenizer.convert_ids_to_tokens(model.generate(...)["sequences"])may look like:["_", "the", "_ca", "t", "_sat", "_on", "_", "the", "_mat"]How would I get the token indices ([2, 3, 4]) for the part of the output corresponding to"cat sat"(e.g. if I wanted to inspect the attention)?If there were an equivalent of theBatchEncoding.char_to_tokenfunction then I could get the character indices of"cat sat"from"".join(tokenizer.convert_ids_to_tokens(model.generate(...)["sequences"]))and then callchar_to_token, but as far as I am aware something like this doesn’t exist? Is there an alternative method?Thanks!