How to decode with custom pad tokens
Hi there,I am implementing a seq2seq model. I am padding the target sequence usingDataCollatorForSeq2Seq(
    tokenizer=t5_tokenizer,
    padding=self.padding,
    label_pad_token_id=-100,
    return_tensors="pt",
)However, when I decode the target sequence (as I want to compute BLUE score, for example) I getOverflowError: out of range integral type conversion attemptedbecause of the -100.Is there a direct way totokenizer.batch_decodepassing a custom padding token?(the alternative would be to manually substitute -100 with 0 before decoding - as donehere- but I am looking for something more straightforward, if it exists).Thanks a lot in advance!
2023-04-13T19:03:34Z
Hey@pietrolescidid you find any solution for this? I’m stuck with the same problem.
2023-11-02T12:14:36Z
I am getting same issue, is there any solution?
2023-12-22T08:47:51Z
Got same problem