Multi-label token classification
Hi!I am trying to solve a token classification problem in a multi-label setup. So far I haven’t found the best path to do it.UnlikeAutoModelForSequenceClassification,AutoModelForTokenClassificationlacks theproblem_typeparameter, where you can specify that you’re working with multi-label.Should I deepen into logits and create a custom loss or is there a more straightforward solution?Thanks!
2022-04-11T21:19:52Z
I would also love to be able to do this. I’m trying to do some NER that has multiple layers of annotation such that a given token could have more than one label…
2022-04-11T22:05:47Z
Actually, I already started implementing a custom loss within thecompute_lossfunction of theTrainer……but the thing surely deserves a feature request on GitHub! Will try to do that
2022-04-12T04:15:09Z
Oh nice. Do you by any chance have code you could share?
2022-04-12T08:01:11Z
I don’t think it should be much harder than swapping outCrossEntropyLossforBCEWithLogitsLossand one-hot encoding your labels and making sure that they are floats.
2022-04-12T20:28:44Z
Yeah, that’s roughly what I am doing. Will share the code once it’s done!
2022-04-19T13:26:02Z
@BunnyNoBugsAny update on this?I have been working on this myself but am running into some challenges…
2022-04-19T13:38:33Z
This is an untested attempt, but I think it should work. Read more about BCEWithLogitsLosshere.class MultiLabelTrainer(Trainer):
    def __init__(self, *args, class_weights: Optional[FloatTensor] = None, **kwargs):
        super().__init__(*args, **kwargs)
        if class_weights is not None:
            class_weights = class_weights.to(self.args.device)
            logging.info(f"Using multi-label classification with class weights", class_weights)
        self.loss_fct = BCEWithLogitsLoss(weight=class_weights)

    def compute_loss(self, model, inputs, return_outputs=False):
        """
        How the loss is computed by Trainer. By default, all models return the loss in the first element.
        Subclass and override for custom behavior.
        """
        labels = inputs.pop("labels")
        outputs = model(**inputs)
        try:
            loss = self.loss_fct(outputs.logits.view(-1, model.num_labels), labels.view(-1))
        except AttributeError:  # DataParallel
            loss = self.loss_fct(outputs.logits.view(-1, model.module.num_labels), labels.view(-1))

        return (loss, outputs) if return_outputs else lossMake sure your data is correctly formatted, e.g., 0 or 1 encoded for each label for each sample.
2022-04-19T13:58:49Z
Yep, I did the same but withoutviewand getting an error with the dims. Will get back to this task on Thursday
2022-04-19T15:52:48Z
Thanks so much@BunnyNoBugsand@BramVanroy! I’m trying this now and getting some other issues but I think they are specific to my situation. I may report back later with an update if I think it will be useful to others. Thanks again!
2022-04-20T19:46:20Z
Here’s a question: doesn’t the custom loss function need to ignore the predictions for special tokens like CLS, PAD, SEP, and (if one is only applying labels to the first subword in a token) the non-first subwords?
2022-04-21T09:10:00Z
Thanks this was super helpful! I followed your tips and got dimension errors, but fixed it by modifying the view parameters, and I also had to cast the labels to floats:loss = self.loss_fct(outputs.logits.view(-1, model.num_labels), labels.view(-1, model.num_labels).float())
2022-04-21T13:59:26Z
I need to look at my results more closely but I think I’ve got this working. My model only gets to about 60% F1 but on the validation set but I think I have ideas about how to improve it.class MultiLabelNERTrainer(Trainer):
    def __init__(self, *args, class_weights: Optional[FloatTensor] = None, **kwargs):
        super().__init__(*args, **kwargs)
        if class_weights is not None:
            class_weights = class_weights.to(self.args.device)
            logging.info(f"Using multi-label classification with class weights", class_weights)
        self.loss_fct = BCEWithLogitsLoss(weight=class_weights)

    def compute_loss(self, model, inputs, return_outputs=False):
        """
        How the loss is computed by Trainer. By default, all models return the loss in the first element.
        Subclass and override for custom behavior.
        """
        labels  = inputs.pop("labels")
        outputs = model(**inputs)
        
        # this accesses predictions for tokens that aren't CLS, PAD, or the 2nd+ subword in a word
        # and simultaneously flattens the logits or labels
        flat_outputs = outputs.logits[labels!=-100] 
        flat_labels  = labels[ labels!=-100]
        
        try:
            loss = self.loss_fct(flat_outputs, flat_labels)
        except AttributeError:  # DataParallel
            loss = self.loss_fct(flat_outputs, flat_labels)

        return (loss, outputs) if return_outputs else loss
2022-04-21T20:46:22Z
It is said, for example,here, that -100 value is automatically ignored by PyTorch loss functions.
2022-04-22T08:41:43Z
That’s only true for CrossEntropyLoss AFAIK. You can compare the signatures.CrossEntropyLosshas an “ignore_index” option,BCEWithLogitsLossdoes not. So I think that@drussellmrichie's adaptation is indeed needed.
2022-04-22T14:17:13Z
I haven’t finished my loss function yet, but here’s what I found out:inputs.pop('labels')in@BramVanroy’s example is very important: otherwise the labels are passed to the model and the standardCrossEntropyLossinside it is computed instead of the custom one.We should make use of theattention_maskin the custom loss. However, it may be the same as@drussellmrichie’s suggestion about the special tokens, maybe someone will correct me.
2022-04-22T14:45:07Z
We should make use of theattention_maskin the custom loss. However, it may be the same as@drussellmrichie’s suggestion about the special tokens, maybe someone will correct me.I’m fairly confident that it’s not necessary given the chunk I added withlabels!=-100.EDIT: Just to elaborate: the proof is in the pudding, and indeed I was able to train some decent multi-label NER models with this approach.
2022-04-22T18:36:35Z
What do you mean with padding?
2022-04-22T19:09:42Z
Are you referring to my phrase “the proof is in the pudding”? If so, I apologize – I am using an old proverb.I am just saying that the proof thatlabels!=100is doing something similar to attention mask lies in the fact that the model performs decently and generates reasonable predictions for the tokens we care about (not CLS, PAD, etc). Also, I noticed that the loss went way up after I introducedlabels!=100, I think because now the easy predictions of the CLS, PAD, and 2nd+ subword tokens can no longer drive the loss down – that is consistent with@BramVanroy's comment thatBCEWithLogitsLossdoes not ignore predictions for special tokens, and needed something like mylabels!=100trick to ignore those.Hope that clarifies.
2022-05-17T17:31:33Z
Hi@drussellmrichie(and@BunnyNoBugsand@BramVanroy!), thanks for the sample code above. A couple questions if you don’t mind:Is the custom trainer all that’s needed to adapt the model to multi-label classification?Also, is this being implemented on top ofAutoModelForTokenClassification?Would you be willing to share an example notebook? I’d love to see the full implementation, including the “right” way to extract the predictions at inference time.If anyone would be willing to share an example notebook, I’m currently trying to get this working and would very much appreciate a sample implementation!Thanks!
No date available
No reply text available