Fine-Tune for MultiClass or MultiLabel-MultiClass
Hi,I want to build a:MultiClass Label(eg: Sentiment with VeryPositiv, Positiv, No_Opinion, Mixed_Opinion, Negativ, VeryNegativ)and a MultiLabel-MultiClass model to detect 10 topics in phrases(eg: Science, Business, Religion …etc)and I am not sure where to find the best model for these types of tasks?I understand this refers to theSequence ClassificationTask. So, I could search for a model tagged with that task on your model repository site - but not all models are tagged like that and the transformers API seems to provide much more task applications beyond the original training.I found with the code below that I can have a model that supports originally 5 labels but load it into aConvBertForSequenceClassificationmodel to support, for example 25 labels. Would this (plussoftmaxorsigmoidand fine-tuning) be the correct way to pick up an existing model and implement 1. or 2. or is there a different more effective way to choose a model and fine tune it?Thanks dirkfrom transformers import pipeline
nlp = pipeline("sentiment-analysis", 'bert-base-multilingual-uncased-sentiment')
result = nlp("I hate you")[0]
print(f"label: {result['label']}, with score: {round(result['score'], 4)}")
result = nlp("I love you")[0]
print(f"label: {result['label']}, with score: {round(result['score'], 4)}")

#label: 1 star, with score: 0.6346
#label: 5 stars, with score: 0.8547from transformers import ConvBertForSequenceClassification, ConvBertTokenizer

convBertModel = ConvBertForSequenceClassification.from_pretrained('bert-base-multilingual-uncased-sentiment', num_labels=25)
convBerttokenizer = ConvBertTokenizer.from_pretrained('bert-base-multilingual-uncased-sentiment')

print ( f"                              num_labels: {model.num_labels}")
print ( f"                              classifier: {model.classifier}")

# num_labels: 25
# classifier: ConvBertClassificationHead(
#   (dense): Linear(in_features=768, out_features=768, bias=True)
#   (dropout): Dropout(p=0.1, inplace=False)
#   (out_proj): Linear(in_features=768, out_features=25, bias=True)
# )
2021-02-27T15:15:07Z
Hi@dikster99,The way I usually search for models on theHubis by selecting the task in the sidebar, followed by applying a filter on the target dataset (or querying with the search bar if I know the exact name). In both your cases, you’re interested in the Text Classification tags, which is a specific example of sequence classification:Screen Shot 2021-02-27 at 4.00.33 pm942×1346 132 KBHowever, this assumes that someone has already fine-tuned a model that satisfies your needs. If not, there are two main options:If you have your own labelled dataset, fine-tune a pretrained language model likedistilbert-base-uncased(a faster variant of BERT). You can find a nice example for text classificationhereand seeherefor the multi-label case. In general this is similar to your second example withConvBertForSequenceClassificationand you were correct to specifynum_labelsin thefrom_pretrainedfunctionIf you have no labelled dataset, then you could try out one of the Zero-Shot Classification models (e.g.here). Seethisblog post for an explanation on how zero-shot works.Hope that helps!PS. thePipelineclasses are typically used for generating predictions from a fine-tuned model, so your example withbert-base-multilingual-uncased-sentimentwouldn’t work because that model was trained on 5 labels and does not know about the 25 labels you are interested in.
2021-02-27T19:03:34Z
Hi Lewtun,thanks for answering so quickly and precise. I have labelled data. So, I am looking at the first option. Its clear to me that I cannot use the pipeline statement since the model does not satisfy my needs out of the box. I was just using it to a have reference to something concrete. I guess I am just confused by the number of different models supported and some let you specifynum_labelsand some will accept exactly one number (eg 5)……but I understand now that I can specify a differentnum_labelsvalue for some models and this will result in:removing the head used to pretrain the model on a masked language modeling objective andreplacing it with a new head` as described inyour linked tutorial(Fine Tuning section).So, to get me started with the MultiClass task where I have 1 label but 6 values (VeryPositiv, Positiv, No_Opinion, Mixed_Opinion, Negativ, VeryNegativ) - I could actually use, for example:a model trained for a True/False sentiment,specifynum_labels=1and train it with my labels containing 6 valuesand that should work if my labels are sensible.Did I get that right?I guess, I’ll just get started with the MultiClass and see how I fareThanks a lot for helping me, DirkPS.: I am working in a cloud environment without Internet. So, I had to resolve the question of how to download a model and use it locally, which I had a hard time finding in documentation. So, Iconstructed 2 notebooksthat show case:How a model can be downloadedHow a local model can be usedI actually needed about 2 days to figure this out just because I was missing thegit lfspart (and it generates an error message shown in the notebook) and the error message from theTransformerslibrary did not see that the partially downloaded files where actually wrong…and I probably made other silly mistakes too … So, I am thinking that others might have the same problem and it might be good to include something as specific as the 2 notebooks in your documentation to be more specific and clear on the what and how can be done with the TransformersWhat do you think?
2021-02-28T20:18:53Z
Hi@dikster99,So, to get me started with the MultiClass task where I have 1 label but 6 values (VeryPositiv, Positiv, No_Opinion, Mixed_Opinion, Negativ, VeryNegativ) - I could actually use, for example:a model trained for a True/False sentiment,specifynum_labels=1and train it with my labels containing 6 valuesand that should work if my labels are sensible.Did I get that right?I think in this case you actually have 6 labels, one for each independent category. For example, such a dataset might take the following form:TextLabelI reallyTransformers!VeryPositivThe pandemic is a dragNegativI went for a walk todayNo_Opinion……so you have one label column, but each row can take one of six values.To get started, my suggestion would be to load up one of the pretrained language models (e.g.distilbert-base-uncased), specifynum_labels=6, and then tokenize the dataset / build theTraineras done in Sylvain’s tutorial, e.g. something likemodel_ckpt = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_ckpt)
model = AutoModelForSequenceClassification.from_pretrained(model_ckpt, num_labels=6).to("cuda")

def tokenize_text(batch):
    # Replace "text" with whatever column name has your text inputs
    return tokenizer(batch["text"], truncation=True)

dataset_enc = dataset.map(tokenize_text, batched=True)
...Regarding, your experience with downloading the models locally, did you know about the “Use in transformers” button from themodel hub?Screen Shot 2021-02-28 at 9.15.59 pm1550×1112 129 KBIf not, perhaps it should be made clearer in the documentation but for that I suggest opening an issue or PR on GitHub so the maintainers can provide you feedback
2021-03-02T20:45:32Z
Hi Lewis,I got to be honest - the GLUE example is a bit out of my league right now as I just started to do some work on TensorFlow……but the good news is that I’ve found aSentiment Classificationsample that seems to have a much lower boundary in order to get it work and make sense of it.So, I took this and developed aMultiClass classificationfrom it (essentially setting thenum_labelsas discussed and using a public demo dataset as you rightfully indicated).I see that the above solutions work with a yield mechanism so I am wondering if this implementation will scale good in terms of memory usage for larger files with more labels? What do you think?Otherwise, I am thinking that I can develop a MultiLabel-MultiClass Classification from theMultiClass classificationby providing a Pandas label column with a list of values (eg ‘[0,1,0,1]’) and setting thenum_labelsto the length of the array in the label column. And replacing the Softmax with a Sigmoid function to yield the correct result. Would you say this is about right or am I missing something again?
2021-03-03T17:32:27Z
@lewtunmhh, it looks like my assumtions about the MultiLabel MultiClass implementation with the Transformers library is missing a detailI tried tochange the code as describedin the last post and now I am receiving an exception from 9 frames deep stacktrace:InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  ValueError: `generator` yielded an element of shape (6,) where an element of shape () was expected.
Traceback (most recent call last):

...

(1) Invalid argument:  ValueError: `generator` yielded an element of shape (6,) where an element of shape () was expected.I get that the shape is not as expected but is that:because I am missing a configuration parameter orbecause I am using the labels in a wrong way? and if it is the 2nd, how am I supposed to tell the model that multiple labels can be applicable for a single text?Does anyone have a hint towards what I am doing wrong here?
2021-03-03T21:41:07Z
Hi@lewtun,I have looked atthis MultiLabel samplethat you previously recommended and found it throws the exception in 1. (see below) - I understand from searching the net that there was a breaking change in Transformers Version 3.0 and was able to resolve this problem by settingtruncation=Truein thetokenizersection of theCustomDatasetclass.But then the code is still throwing the exception in 2. and this is where I am lost without a fix - would you have any idea on how to get this code to work in a current version of transformers version 4.3.3?Truncation was not explicitly activated butmax_lengthis provided a specific value, please usetruncation=Trueto explicitly truncate examples to max length. Defaulting to ‘longest_first’ truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy totruncation.Exception: TypeError: dropout(): argument ‘input’ (position 1) must be Tensor, not str
2021-03-04T12:56:23Z
Hi@dikster99, I had a closer look at the multi-label example I linked to and see that it’s more complicated than it needs to be because:transformersnow has aTrainerclass that dramatically simplifies the training / evaluation loops.thedatasetslibrary is a much better way to prepare the data and works great with theTrainerTo implement multi-label classification, the main thing you need to do is override theforwardmethod ofBertForSequenceClassificationto compute the loss with a sigmoid instead of softmax applied to the logits. In PyTorch it looks something likeclass BertForMultilabelSequenceClassification(BertForSequenceClassification):
    def __init__(self, config):
      super().__init__(config)

    def forward(self,
        input_ids=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        output_attentions=None,
        output_hidden_states=None,
        return_dict=None):
        return_dict = return_dict if return_dict is not None else self.config.use_return_dict

        outputs = self.bert(input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict)

        pooled_output = outputs[1]
        pooled_output = self.dropout(pooled_output)
        logits = self.classifier(pooled_output)

        loss = None
        if labels is not None:
            loss_fct = torch.nn.BCEWithLogitsLoss()
            loss = loss_fct(logits.view(-1, self.num_labels), 
                            labels.float().view(-1, self.num_labels))

        if not return_dict:
            output = (logits,) + outputs[2:]
            return ((loss,) + output) if loss is not None else output

        return SequenceClassifierOutput(loss=loss,
            logits=logits,
            hidden_states=outputs.hidden_states,
            attentions=outputs.attentions)where the only thing that I’ve really changed are these two linesloss_fct = torch.nn.BCEWithLogitsLoss()
loss = loss_fct(logits.view(-1, self.num_labels), labels.float().view(-1, self.num_labels))You can probably adapt the TensorFlow code in a similar fashion (I haven’t used TF in years so can’t be much help there).There are some other things needed (e.g. the metrics), so I put together a hacky notebook here that you can use as a template to get started:https://colab.research.google.com/drive/1X7l8pM6t4VLqxQVJ23ssIxmrsc4Kpc5q?usp=sharing
2021-03-05T09:24:44Z
Hi again@dikster99, thanks to a tip from Sylvain Gugger, I realised that there’s a much simpler way to implement multi-label classification: just override thecompute_lossfunction of theTrainer!Here’s an example in PyTorch:class MultilabelTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.pop("labels")
        outputs = model(**inputs)
        logits = outputs.logits
        loss_fct = torch.nn.BCEWithLogitsLoss()
        loss = loss_fct(logits.view(-1, self.model.config.num_labels), 
                        labels.float().view(-1, self.model.config.num_labels))
        return (loss, outputs) if return_outputs else lossand I’ve updated my Colab notebook to reflect the change. Hope that helps!PS you will need to installtransformersfrom themasterbranch for this to work, i.e.pip install git+https://github.com/huggingface/transformers.git
2021-03-05T13:00:04Z
Hi@lewtun,thanx for you cool inputunfortunately, I am not able to use Datasets because its not installed in my target environment - I usually usePandasorSparkthere - is it possible to substitueteDatasetswithPandasorSparkbased on direct access totrains.csvandtest.csvin your notebook?I also do not have an Internet Connection in this target environment (for security reasons) which is why I cannot use a direct download link from GitHub or anywhere elseWe  have transformers v4.2.1 installed and so I am bound to this version as well.Would you be able to adjust your sample notebook to these requirements:use Pandas or Spark for data retrievalAccess data from file system rather than retrieval through Internet connection as I want to replace this later with my own data in our private environmentUse transformers version v4.2.1Sorry to be such a pain about it……but I hope its easy to meet these requirements as I imagine many others working in similarly locked down environments …Thanx DirkPS.: I have found a notebook in the transformers community (after unsucessfully testing 3 others) which seems to meet my requirements (but is much more complex as your approach) but appears to work with transformers 4.2.1 <=> 4.3.3:github.comDhavalTaunk08/NLP_scripts/blob/master/Transformers_multilabel_distilbert.ipynb{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Transformers_multilabel_distilbert.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "mount_file_id": "15ZXLCtJmBmByqWLkJsyuRuidGunSQt3c",
      "authorship_tag": "ABX9TyMYFn2H5CH0K8Hce4nsXrwt",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [This file has been truncated.show original
2021-03-06T08:33:12Z
Hi@dikster99, sure here’s an example that meets your constraints (I too know the joys of working behind a firewall):https://colab.research.google.com/drive/18vy67le2DC-iMJK-AiB0vVKtMRAxmBnB?usp=sharingYou’ll just have to pointdata_dirto the location of your files on disk and adapt the code a bit to load both the train and test files (I just worked with the training set for simplicity). You can find more details about creating custom datasets here:Fine-tuning with custom datasets — transformers 4.3.0 documentationIf you can’t access the HuggingFace Hub through your firewall, you’ll need to clone the model repo (as mentioned in a previous post) and upload the model files manually to your env.Good luck!
2021-03-09T09:07:55Z
Hi Lewis,your solution works great and its really impressiv how easy it is to generate a good prediction accuracyI found anextremly good intro to transformersand thinks are adding up for me now - so, I can create a TF version based on your code.So, thanx a bunch for getting me startedDirk
2021-03-10T11:48:20Z
mhhh, I guess, I’ve been to positiv too earlyThe problem I am having now is that I can train a model and it realy seems to generate positive accuracy/loss results but the prediction always generates the same classThe CoLab notebook is here using the 20 newsgroups dataset as a sample.Am I using the wrong way to train or is the prediction not correct?github.comDirkster99/PyNotes/blob/master/Transformers/LocalModelUsage_Finetuning/62_Transformer_4_Language_Classification_MultiClass.ipynb{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "62 Transformer 4 Language Classification MultiClass.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOyEXhZWk3x+tCwEb02x5Az",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "c60ae33f833a4093a9323b17a9a6f379": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",This file has been truncated.show originalI’ve tried this with different models and different datasets but the result always seems to be the same - the model just always predicts 1 class no matter what I try to enter as sample text
2021-03-11T19:49:02Z
When you say “same class”, do you mean that you’re always getting something likesci.spacefor different inputs or do you mean you are getting one class (multiclass) when you expect multiple (multilabel)?One thing I wonder is whether you need to pass thelabel2Indexandindex2labelmappings to your model when you initialise it, i.e.bert = TFAutoModel.from_pretrained(tranformersPreTrainedModelName, label2id=label2Index, id2label=index2label)That way you ensure the model’s logits line up with the mappings you’re using to generate the predictions at the end of the notebook.
2021-03-12T00:08:44Z
@lewtunWhen I say ‘same class’ I mean that it always predictssci.spaceno matter what text I enter - even on a sports or religions texteven though this model should generate a MultiClass classification result…Your suggestion with thebert = TFAutoModel.from_pretrained(tranformersPreTrainedModelName, label2id=label2Index, id2label=index2label)seems to change the situation in the way that the model predicts not just one class but about 3 on the 4 sample sentences. But it is still almost always wrong on the actual target even though I trained the model with aseq_len=50over32 Epochswith these results:loss: 0.0456 - accuracy: 0.9882 - val_loss: 0.1228 - val_accuracy: 0.9680I am confused because I don’t understand why I have to supply the parameters since I am using the model without its in-build heads? I have used the exact same model configuration with a classic TensorFlow/Keras GloVe model and reached a worse accuracy but had better actual predictions - so, there still seems to be something strange here because the measured accuracy just does not add up with the actual prediction result
2021-03-12T19:00:04Z
@lewtunI think I got it to work now - thanks to your tip about the label - index mapping. I don’t really understand why thats needed but it seems to do the trick now as expected. What do you think?github.comDirkster99/PyNotes/blob/master/Transformers/LocalModelUsage_Finetuning/66_Transformer_4_Language_Classification_MultiClass.ipynb{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "66 Transformer 4 Language Classification MultiClass.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyPIhUxBDq7TgOhyP/tK4UPR",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU",
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "e050430edc3344be8224e0e8af7c5569": {
          "model_module": "@jupyter-widgets/controls",This file has been truncated.show original
2021-03-12T20:10:59Z
nice to hear you got it working@dikster99! i had a quick look at your code and it looks goodthe reason the label → id and id → label mappings are needed is because the default behaviour is to create a dummy mapping based on the number of labels:transformers/configuration_utils.py at fa35cda91e9a1929f9cebeb973801709ba31dd4b · huggingface/transformers · GitHubso in your case, the accuracy, loss etc were all fine because the model was using an internally-consistent definition for the labels, it’s just that the meaning of those labels did not match the ones in the data.
2021-03-19T13:53:57Z
This awful or least it seems to be less than stableSo, I tried to apply the code that we already had working for English text just to find out that it does not work for German - because there are not many German public datasets around Icreated my own.I then published 3 notebooks on GitHub to show the dilemma:66_Multi_Label_German_text_classification_in_TensorFlow_Keras.ipynbShows a Multilabel approach via a classic Keras/TF implementation to show that the German dataset can yield useful results (the results are not great because I did not clean the data extracted originally somewhere else but this shows the dataset can be used for a classification task)66_Transformer_4_Language_Classification_MultiLabel_DistilBert.ipynbShows a Multilabel approach via Transformers DistillBert Modeling using the English Toxitity dataset collection as we previously discussed.This model is working as expecting and does indeed return useful results.66_Transformer_4_Language_Classification_MultiLabel_DistilBert_German.ipynbThis notebook uses the exact same transformers code approach as we’ve used in 2) with the difference that we try to consume German data here. The accuracy shown during the training phase looks useful but testing at the end of the notebook shows that the actual predictions are less than useful (no single prediction is correct and the model always predicts very similar output no matter what the input is)I wonder if this is a known bug or whether this should be an issue for the HuggingFace repository?
2021-06-15T10:15:46Z
Hi@lewtun,lewtun:class MultilabelTrainer(Trainer):does that meanThe multi-label problem can be solved directly in Trainer without subclassing BertForSequenceClassification asclass BertForMultilabelSequenceClassification(BertForSequenceClassification):Thanks,
2021-06-16T12:22:09Z
hey@Loganathan, it’s actually now possible to do multi-label classification for some models without needing to create your ownTrainersubclassfor example, with BERT you can specify theproblem_typeparameter in the model config as follows:from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", problem_type="multi_label_classification")this ensures that the loss function used in the forward pass isBCEWithLogitsLoss()which is suitable for multi-label tasks. you should then be able to use this model in a standardTrainerif your model is not in one of the types listed underproblem_typein the configherethen you can either subclass theTraineras i discussed earlier in the thread (using a standardModelNameForSequenceClassificationclass) or subclass the model and override the loss calculation the forward passhth!
No date available
No reply text available