How does T5 create the correct decoder_input_ids?
huggingface.coT5 — transformers 3.1.0 documentationI have been reading the documentation for the T5 model and in the training section, for both unsupervised denoising and supervised training, the comment states the model is able to create the correctdecoder_input_ids. However, I’m not sure how the model is able to do. The documentation tells me that if thedecoder_input_idsis None (which it is by default) it takes the values ofinput_ids. Does this mean the start token is the first token of the input sequence?The description also mentions a start-sequence token. But I can’t seem to find where this is generated or what the token is.
2020-09-20T12:14:22Z
TheT5PreTrainedModel._shift_rightmethodheretakeslabelsand preparesdecoder_input_idsBasically, all it does is,takeslabels-> add pad token at the beginning -> remove theeos->decoder_input_ids
2020-09-21T13:30:40Z
Thanks for pointing this out for me!