Finetuning from multiclass to mutlilabel
Hello,I got a specific classification task where I finetuned a pretrained BERT Model on a specific task concerning customer reviews (classify a text as “customer service text”, “user experience” etc.):num_labels_cla = 8
model_name_cla = "bert-base-german-dbmdz-uncased"
batch_size_cla = 32

model = AutoModelForSequenceClassification.from_pretrained(model_name_cla, num_labels=num_labels_cla)
tokenizer = AutoTokenizer.from_pretrained(model_name_cla)As you can see I got 8 distinct classes. My finetuned classification model scores pretty well with unseen data with a f1 score of 80.1. However, it is possible that one text belongs to 2 different classes. My question now is how I have to change my code achieve that? I already transformed my target variable with MultiLabelBinarizer such that my target variable looks like this:[0, 0, 0, 1, 0, 0, 0, 0],
       [1, 0, 0, 0, 0, 0, 0, 0],
       [1, 0, 0, 1, 0, 0, 0, 0],I am using the HuggingFace Trainer instance for finetuning.Cheers
2021-09-01T13:18:51Z
You can set theproblem_typeof anxxxForSequenceClassificationmodel tomulti_label_classificationwhen instantiating it, like so:from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("bert-base-german-dbmz-uncased", problem_type="multi_label_classification", num_labels=num_labels_cla)This ensures that theBCEWithLogitsLossis used instead of theCrossEntropyLoss, which is necessary for multi-label classification. You can then fine-tune just like you would do with multi-class classification.
2021-09-01T13:46:04Z
Thank you for your reply! Is it correct then that I encode my target like this:marlon89:[0, 0, 0, 1, 0, 0, 0, 0],
       [1, 0, 0, 0, 0, 0, 0, 0],
       [1, 0, 0, 1, 0, 0, 0, 0],
2021-09-01T13:53:16Z
Yesas you can see in thecode, the loss is computed as follows:elif self.config.problem_type == "multi_label_classification":
      loss_fct = BCEWithLogitsLoss()
      loss = loss_fct(logits, labels)The logits will be of shape(batch_size, num_labels). Thedocsof PyTorch’sBCEWithLogitsLossindicates that thelabelsshould have the same shape. So that’s indeed correct.
2021-09-01T13:54:00Z
Thank you so much for your support!