How is the loss comuted for sequence to sequence models?
If I understand it correctly, sequence-to-sequence models use cross-entropy as the loss function. Here, each output token is compared with the true token.Now, with for example text summarization, does this mean that during training the model is forced to generate a summary of the exact same length as the true summary associated with the input? If not, how is the loss computed in the case where the length of the generated summary differs from the length of the true summary?
2022-06-14T21:56:34Z
Simzalabim:does this mean that during training the model is forced to generate a summary of the exact same length as the true summary associated with the input?More or less, yes, it’s forced to generate an output that’s the same length as the labels. However, the thing is that for training, your model doesn’t actually “generate”; it runs just a singleforwardcall. You feed the labels into the model all at once and the model will output a big set of logits for each token and then compute the loss for each token. For example, you can see thisherein the T5 code. Note that it’s not calling thegeneratemethod or running a loop or anything.
2022-06-16T16:05:32Z
Clear reply. Thank you!
2022-06-17T20:36:49Z
Btw, if you want more background info on how it works, you can also look up “teacher forcing” or “maximum likelihood estimation with teacher forcing”.
2023-06-06T14:21:51Z
@dblakelyTo add to your comment: I found this lecture very useful for understanding how CE for seq2seq models are calculated:Stanford CS224N NLP with Deep Learning | Winter 2021 | Lecture 6 - Simple and LSTM RNNs - YouTube