What is the classification head doing exactly?
Hello,When using a transformer model for text classification, one usually loads a model and then usesAutoModelForSequenceClassificationto train the classifier over the N classes in the data.My question: which model is actually used for classification? Is it alogisticmodel (with uses as input theCLSrepresentation)?In the case of several classes (say bad, neutral, good) the usual methodology in machine learning is to train severalone-vs-allclassifiers and then predict the label with most votes. Is this what is happening under the hood withhuggingface?Thanks!
2021-09-20T14:16:05Z
@nielsrI would be curious to have your take on this, if you have a few moments. Your comments have been incredibly useful so far. Thanks!
2021-09-20T14:21:11Z
The exact implementation of XXXForSequenceClassification differs from model to model. But you can have a look at e.g. the BERT implementation:github.comhuggingface/transformers/blob/04976a32dc555667afa994e8f918cbee88d84a4f/src/transformers/models/bert/modeling_bert.py#L1481)@add_start_docstrings("""Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooledoutput) e.g. for GLUE tasks.""",BERT_START_DOCSTRING,)class BertForSequenceClassification(BertPreTrainedModel):def __init__(self, config):super().__init__(config)self.num_labels = config.num_labelsself.config = configself.bert = BertModel(config)classifier_dropout = (config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob)self.dropout = nn.Dropout(classifier_dropout)In this case it is simply a dropout and then a linear layer on top of the pooled outputs.
2021-09-20T14:27:58Z
oh, thanks@BramVanroy! I am not a hardcore user ofpytorchbut I am looking at the code. You say that this is a linear layer on the pooled output. But can the model generate multiple classes then?
2021-09-20T14:50:13Z
What do you mean? Why wouldn’t it? A linear layer is simply a projection of X dimensions to Y dimensions, e.g. 512 to 512, or, 768 to 1, or any other that you can think of. As you can see here:github.comhuggingface/transformers/blob/04976a32dc555667afa994e8f918cbee88d84a4f/src/transformers/models/bert/modeling_bert.py#L1492def __init__(self, config):super().__init__(config)self.num_labels = config.num_labelsself.config = configself.bert = BertModel(config)classifier_dropout = (config.classifier_dropout if config.classifier_dropout is not None else config.hidden_dropout_prob)self.dropout = nn.Dropout(classifier_dropout)self.classifier = nn.Linear(config.hidden_size, config.num_labels)self.init_weights()@add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))@add_code_sample_docstrings(tokenizer_class=_TOKENIZER_FOR_DOC,checkpoint=_CHECKPOINT_FOR_DOC,output_type=SequenceClassifierOutput,config_class=_CONFIG_FOR_DOC,)The linear layer will output the number of classes that you request. If you have five classes, it will output five values. How you deal with those values (single or multi label classification) then depends on your loss function. See:github.comhuggingface/transformers/blob/04976a32dc555667afa994e8f918cbee88d84a4f/src/transformers/models/bert/modeling_bert.py#L1551-L1562if self.config.problem_type == "regression":loss_fct = MSELoss()if self.num_labels == 1:loss = loss_fct(logits.squeeze(), labels.squeeze())else:loss = loss_fct(logits, labels)elif self.config.problem_type == "single_label_classification":loss_fct = CrossEntropyLoss()loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))elif self.config.problem_type == "multi_label_classification":loss_fct = BCEWithLogitsLoss()loss = loss_fct(logits, labels)
2021-09-20T15:46:05Z
olaffson:My question: which model is actually used for classification? Is it alogisticmodel (with uses as input theCLSrepresentation)?One typically just places a linear layer (nn.Linear) on top of the final hidden state of the [CLS] token. In other words, a linear classifier is sufficient. It converts the final hidden state vector into a vector that represents the classes.olaffson:In the case of several classes (say bad, neutral, good) the usual methodology in machine learning is to train severalone-vs-allclassifiers and then predict the label with most votes. Is this what is happening under the hood withhuggingface?For HuggingFace models, one typically only learns a single classifier on top of the base model, for binary, multi-class and multi-label classification.
2021-09-20T20:09:34Z
Very interesting@nielsr@BramVanroy. Thanks. Its great to know whathuggingfacehas a consistent linear classification head in all cases.If my understanding is correct, then the linear classification head actually learns a big matrix whose dimensions arerows*columns=vocabulary size*classes.Is that right? Thanks!
2021-09-21T07:28:34Z
No, that is not right. As I said before, its input dimensions are usually the output dimensions of the base model (e.g. 768) and the output dimensions are the number of classes, e.g. 3 for “neutral, positive, negative”. The classification weights are, relatively speaking, quite small in many downstream tasks.During language modeling, the LM head has the same input dimensions, but the output dimensions are the same size as the vocabulary: it provides you with a probability for each token how well it fits in a given position. This does lead to a large classifier.
2021-09-21T12:09:58Z
Hi@BramVanroyI think we actually agree. What I am saying, in mathematical terms, is that the linear layers performs the linear operationA*Bwhere A is anumber of obsx768matrix and B is a768xnumber of classesmatrix.Thus the result of the matrix multiplication isnumber of obsxnumber of classeswhose row i gives the score  for each class for observation i (before actually transforming to probabilities via a softmax).Does that make sense?
2021-09-21T12:59:14Z
So the linear layer does a linear transformation on the given data by multiplying it with a learned transformation matrix (and optionally adding bias). This (transposed) transformation matrix is of the shapeoutput_features * input_features, which in the case of the classification layer isn_classes * model_hidden_size. I do not understand why you bring the vocabulary size into play, as it is not important at this stage.
2021-09-21T13:59:24Z
Thanks@BramVanroyfor clarifying your point. You are right I mentioned the vocabulary size early on which is incorrect. What I mean is the following (and again I think we agree).The output of the language model is a vector representation of the[CLS]token for each sentence. This representation is a row vector with 768 dimensions. So if you have N documents in your data you can stack all your row vectors into a big matrix A with dimensions N *768Then what I think the linear layer is doing is to learn essentially another matrix B, whose dimensions are 768 xk number of classesbecause when you do the matrix multiplication of A times B you get as output a matrix with N rows (your N input sentences) and k columns (the score for each possible class). I believe this is also what you are saying (except that your matrices are transposed). Does that make sense?Thanks! It is always interesting to discuss the underlying mechanisms. Helps understand the models better.
2021-09-21T15:38:06Z
Yes, this is almost correct. For completeness’ sake: the linear layer does not output “scores for the classes”. It produces logits. These can be transformed into probabilities with a soft max as was mentioned earlier.It always helps to look directly at some code, I think. At least that’s how I get to better understand these things.
2021-09-21T19:21:55Z
perfect. I am sure this conversation will be helpful to others, too. Thanks!
2022-09-26T18:18:47Z
hugging face gives you a pre-trained model which has an architecture, and the weights are downloaded and added to the model when you initialize the model. If you are using the API AutoModel(checkpoint), when you run it you will get the final layer as the out. here you will need to make a custom head. in your case a classification head, usually, is done using linear layers which can get you an output of dimension (batch_size,n) n- for say (pos, neu, neg,… all the stuff in between), set the final linear layer output to 3 if you want just 3 logits, or huggingface has another API AutoModelFoseuenceClassification(checkpoint,num_labels=3)  num_labels argument can be set to 3  for classification head to output logits of dim (batch_size,3).
2023-03-21T17:50:32Z
I want to use the BertForSequenceClassification for binary classification. If I set the num_labels to 2, it throws an error "ValueError: Target size (torch.Size([16])) must be the same as input size (torch.Size([16, 1]))". If i set it to one then it works can anyone tell me why this happens?
2023-04-14T08:35:28Z
use num_labels=1