T5 sequence classification
I would like to do sequence classification over the encoder of the T5 model.Which hidden state of the last layer should I use for the classification? The hidden state of the last timestep or should I take a mean over all timesteps?Thank you in advance.
2022-05-08T06:53:27Z
manzar:advanceare there any suggestions for this? I am not sure ifhidden[:, 0, :]makes sense (since no[CLS]token in T5) but I found that usinghidden[:,0,:]is yielding better results thantorch.mean(hidden_states, dim=1). Any suggestions on whats the best way to do this in T5Encoder?