Why is there no pooler representation for XLNet or a consistent use of sequence_summary()?
I’m trying to create sentence embeddings using different Transformer models. I’ve created my own class where I pass in a Transformer model, and I want to call the model to get a sentence embedding.BothBertModelandRobertaModelreturn apooler output(the sentence embedding).pooler_output(torch.FloatTensorof shape(batch_size, hidden_size)) – Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.Why doesXLNetModelnot produce a similarpooler_output?When I look at the source code forXLNetForSequenceClassification, I see thatthere actually exists code for getting a sentence embeddingusing a function calledsequence_summary().def forward():
        transformer_outputs = self.transformer( ... )
        output = transformer_outputs[0]

        output = self.sequence_summary(output)Why is thissequence_summary()function not used consistently in the other Transformers models, such asBertForSequenceClassificationandRobertaForSequenceClassification?
2020-11-30T02:29:57Z
To be rigorous, I compared (1) BertModel’s pooler output to (2) the output of SequenceSummary for the same sentence. Aren’t these two approaches supposed to produce the same sentence embedding? That’s what I’m led to believe from the comments ofSequenceSummary:classtransformers.modeling_utils.SequenceSummaryCompute a single vector summary of a sequence hidden states.summary_type(str) – The method to use to make this summary. Accepted values are:"last"– Take the last token hidden state (like XLNet)"first"– Take the first token hidden state (like Bert)"mean"– Take the mean of all tokens hidden states"cls_index"– Supply a Tensor of classification token position (GPT/GPT-2)"attn"– Not implemented now, use multi-head attentionReturnsThe summary of the sequence hidden states.I’ve created aGoogle Colab notebookthat computes both (1) pooler output and (2) SequenceSummary output for the same sentence. The last line in the notebook shows that the two approaches to sentence embeddings are not the same.Is this a bug, or was my asssumption wrong? If it’s a bug (in code or documentation), I’ll open a GitHub issue.Any help would be appreciated.
2020-11-30T02:37:33Z
@thomwolfBecause he answered this tangentially-related Githubissue.
2020-11-30T09:44:38Z
The answer for thewhyis in your own question.facehugger2020:The Linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.BERT implements a pooler output because it has been pretrained with an additional objective (next sentence prediction) that uses such output. RoBERTa does not use that objective anymore but it subclasses BERT in code so it also has that functionality. XLNet however is trained on PLM and is different altogether.The BERT pooler is slightly different from what SequenceSummary does. You can compare the code if you want to. The pooler gets the output of CLS and puts it through a linear layer and TanH activation.github.comhuggingface/transformers/blob/d5b3e56de5376aa85ef46e7f0325139d9e299a41/src/transformers/modeling_bert.py#L508-L520class BertPooler(nn.Module):def __init__(self, config):super().__init__()self.dense = nn.Linear(config.hidden_size, config.hidden_size)self.activation = nn.Tanh()def forward(self, hidden_states):# We "pool" the model by simply taking the hidden state corresponding# to the first token.first_token_tensor = hidden_states[:, 0]pooled_output = self.dense(first_token_tensor)pooled_output = self.activation(pooled_output)return pooled_outputThe SequenceSummary has more options: you can choose the activation function, add dropout, and pick a summary type. This can explain the differences that you encounter.github.comhuggingface/transformers/blob/f9a2a9e32bf4f6dc128decd0c124fa1f5507532e/src/transformers/modeling_utils.py#L1535-L1538output = self.first_dropout(output)output = self.summary(output)output = self.activation(output)output = self.last_dropout(output)
2020-11-30T14:07:36Z
Note that theSequenceSummaryobject will disappear soon-ish: it goes against our philosophy of having each model fully defined in one self-contained file (except for obvious subclasses) and adds config arguments to some models that are never actually used. We will use an approach like in the BERT or ELECTRA modeling files, where all the layers for the head are defined in the modeling.py file.
2020-11-30T19:22:24Z
BERT implements a pooler output because it has been pretrained with an additional objective (next sentence prediction) that uses such output. RoBERTa does not use that objective anymore but it subclasses BERT in code so it also has that functionality. XLNet however is trained on PLM and is different altogether.Thanks, but I don’t think that fully explains why XLNet doesn’t have apooler_output. You are right that BERT is trained on next-sentence-prediction, but that only affects the classifier top layer (the linear layer that takes thepooler_outputand produces logits). I’m interested in only thepooler_output, the summarization of all the token embeddings. To be consistent with BERT and Roberta, XLNet should produce apooler_outputof its own. I will use that sentence embedding to train my own classifier top layer using a downstream task.
2020-11-30T19:23:35Z
Note that theSequenceSummaryobject will disappear soon-ish: it goes against our philosophy of having each model fully defined in one self-contained file (except for obvious subclasses) and adds config arguments to some models that are never actually used. We will use an approach like in the BERT or ELECTRA modeling files, where all the layers for the head are defined in the modeling.py file.Thank you. WhenSequenceSummaryis removed, will all models then produce their ownpooler_output?
2020-11-30T22:21:52Z
No, this was just a BERT thing that it had in its original code and is a part of its pretraining objective. There is no reason to add it to other models that did not use that in their pretraining objective.
2020-11-30T23:10:10Z
Users (like me) still need a way to generate sentence embeddings from a token sequence, right? I’m referring to the actual construction of the sentence embedding from the token embeddings, such as:First token embeddingMean of the token embeddingsMax of the token embeddingsEtc.Isn’t that what theSequenceSummaryand/orpooled_outputprovides?
2020-12-01T07:57:49Z
I’m all for usability and have made many suggestions that focus on the user side of things, but decisions need to be made as to what this library should provide and what should be considered user-specific scenarios and how intricate it is for users to implement such functionality themselves. You saying that this libraryshould provide xis your subjective opinion but does not go well with the philosophy of the library. The library cannot implement all small code snippets that individual users want, especially not if those are easy to do yourself.The library does provide some useful, general help such as the Trainer class, but a sequence summary is more subjective: it can be seen as a final classification module on top of a model, but then the decision needs to be made how to implement it. Dropout? Activation? Mlp? How many layers? Additional skip connections? Perhaps even just ensemble first? Etc. There are so many options and the ‘right’ option will depend on the model and your use-case. So rather than implementing something that is just half-and-half, we can leave that implementation up to the user. As you can see from the code from sequence summary, it isn’t hard to get the position of the CLS token or mean of tokens which, as a basic sentence embedding, works well.You have to remember that some models are basically extensions of others (Roberta and Bert) but that others are completely different (Bert and gpt2). It doesn’t make sense to force them into the same position especially if it is easy for users to do this themselves.If you really need this functionality to get sentence embeddings, and you do not have the skill to implement it yourself, you can use thesentence-transformerslibrary which is intended exactly for this purpose.
2020-12-01T23:07:24Z
The library cannot implement all small code snippets that individual users want, especially not if those are easy to do yourself.I’m not asking about some random code snippet. I’m asking about functionality to construct a sentence embedding form the token embeddings, which is core requirement for NLU applications. I’m further not asking anyone to implement anything new. I’m asking why would they remove theSequenceSummaryclass, which already constructs multiple types of sentence embeddings.a sequence summary is more subjective: it can be seen as a final classification module on top of a model, but then the decision needs to be made how to implement it. Dropout? Activation? Mlp? How many layers? Additional skip connections? Perhaps even just ensemble first? Etc.I’m not asking about the classification. I’m asking about how to construct the sentence embedding from the token embeddings, which is universal for all these neural language models because their core competency is to produce token embeddings.The reason I’m asking about all this is that I don’t want to reinvent the wheel. There seems to already exist code to get sentence embeddings from BERT, XLNet, and others, but the code paths are not consistent.
2020-12-02T14:22:25Z
@sguggerAlready explained why it is being removed.“A core requirement for NLU applications” is a stretch. SOTA applications typically finetune full models on their downstream task. Sentence embeddings as such (by averaging or otherwise pooling without any tuning) areaapproach but hardly the best one out there. Averaging token embeddings is as simple astorch.meanif that is what you want, and getting a CLS token’s output (if it even exists for that LM) isn’t hard either (cf. SequenceSummary).I think this topic can conclude here. I would advise you to either implement this yourself, and perhaps start from SequenceSummary and build from that if you are capable, and otherwise use the sentence-transformers library or similar which provides finetuned models to get better sentence representations.
2020-12-03T02:27:57Z
SOTA applications typically finetune full models on their downstream task.We’re in agreement here.Averaging token embeddings is as simple astorch.meanif that is what you wantIt’s not so simple when taking into account attention masks. That’s why I was hoping the HF library could provide pooling functionality appropriate for every model.otherwise use the sentence-transformers library or similar which provides finetuned models to get better sentence representations.Again, I’m not interested in pretrained parameters for a classifier. I just want to generate sentence embeddings so I can fine tune the parameters on my own downstream task. My observations are that:BERT has apooled_output.XLNet does not have apooled_outputbut instead uses SequenceSummarizer.sgugger says that SequenceSummarizer will be removed in the future, and there is no plan to have XLNet provide its ownpooled_output.Folks like me doing NLU need to produce a sentence embedding so we can fine-tune a downstream classifier.
2020-12-03T08:45:05Z
Have you looked at sentence-transformers? That’s what they do, outputting a single embedding for a single sentence. You can then use that with your own downstream classifier. They do not provide many different models though, so that can be an issue for you. But again, this shouldn’t be so hard to implement yourself.I am not sure what you mean with “taking into account attention masks”, why do you need those?
2020-12-04T06:51:15Z
Have you looked at sentence-transformers?Yes, I’m doing research in the same area.I am not sure what you mean with “taking into account attention masks”, why do you need those?BERT will insert[PAD]tokens as needed. The attention masks mark where those tokens are located in the input sequencer. I don’t want to include the embedding for the those tokens when I compute the max or the mean.
2020-12-04T10:40:44Z
You don’t need the attention mask for that explicitly, as you can create it yourself. You just need the input token IDs. Then you can get the index where input_tokens_ids == tokenizer.pad_token_id, and then you can select those indices from the output. If you cannot figure out to do this I can post a snippet here.
2020-12-07T04:30:35Z
Thanks. I’m trying to plug in other encoders like XLNet and Roberta, so I don’t think I can necessarily rely on creating a mask withinput_tokens_ids == tokenizer.pad_token_id. I’ll figure it out.