T5forConditionalGeneration + classification
I would like to do sequence classification over the encoder in parallel with conditional generation using an auxiliary loss. However, I am confused about which hidden state I should take for the classification.Supposing that the hidden state of the last layer has the following dimensions: [batch size, seq length, hidden size] should I take the last one [:, -1, :] ?
2020-12-11T12:44:58Z
It depends on the model. BERT uses the first one (where the CLS token is), some models use a pooling of all hidden states, other the one for the last logits (which is not necessarily-1since you could have padding). I’d look at what is done inT5ForSequenceClassificationand copy the code.
2020-12-11T15:20:14Z
You are absolutely right. That’s what I tried to do at first. However,  the T5 model has noT5ForSequenceClassificationclass or something similar. I think that the most suitable is to use the last logits taking into account padding. Do you have in mind any function that can be helpful?
2020-12-13T14:48:27Z
I would also like to ask if there is any way to tie the weights of the encoder and the decoder.