New model output types
As was requested in#5226, model outputs are now more informative than just plain tuples (without breaking changes); PyTorch models now return a subclass ofModelOutputthat is appropriate. Here is an example on a base model:from transformers import BertTokenizer, BertForSequenceClassification
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs = model(**inputs, labels=labels)Thenoutputswill be anSequenceClassifierOutputobject, which has the returned elements as attributes. The old syntaxloss, logits = outputs[:2]will still work, but you can also doloss = outputs.loss
logits = outputs.logitsor alsoloss = outputs["loss"]
logits = outputs["logits"]Under the hood,outputsis a dataclass with optional fields that may be set toNoneif they are not returned by the model (likeattentionsin our example). If you index by integer or by slice, theNonefields are skipped (for backward-compatibility). If you try to access an attribute that’s set toNoneby its key (for instance hereoutputs["attentions"]), it will return an error.You can convert thoseoutputsto a regular tuple/dict withoutputs.to_tuple()oroutputs.to_dict().You can revert to the old behavior of having tuple by settingreturn_tuple=Truein the config you pass to your model, or when you instantiate your model, or when you call your model on some inputs. If you’re usingtorchscript(and the config you passed to your model hasconfig.torchscript = True) this will automatically be the case (because jit only handles tuples as outputs).Hope you like this new feature!
2020-07-10T18:21:13Z
So many quality of life improvements recently.Thanks for all your work and effort.
2020-07-17T10:24:53Z
I hadn’t seen this before. This is great! Might help get some confusion out of the way.
2020-07-18T16:00:40Z
FYI, this new feature breaks when usingnn.DataParallel(multi gpu) as currentlytorch.nn.parallel.scatter_gather.gather can’t gather outputs that are dataclasses.There are workarounds worked on in the core, but until then if you run into this issue in your custom code you may need to add:+            if isinstance(model, torch.nn.DataParallel):
+                inputs["return_tuple"] = Truebefore your call:outputs = model(**inputs)For more details see:https://github.com/huggingface/transformers/issues/5693p.s. voting on thepytorch issuemay help resolve this globally faster.
2020-07-27T08:26:55Z
By voting you mean thumbing-up, right? Done. This would indeed be helpful because I assume that some of the examples may not work with this new functionality out of the box.(As an aside: nice to see thattransformershas grown so much that it is taken into account by the big developer league.)
2020-07-27T15:38:55Z
BramVanroy:By voting you mean thumbing-up, right?Yes and thank you,@BramVanroy.This would indeed be helpful because I assume that some of the examples may not work with this new functionality out of the box.Until pytorch sorts this out a potential workaround is being worked on:a transparent solution for DataParallel.gather not supporting ModelOutput (dataclass) by stas00 · Pull Request #5941 · huggingface/transformers · GitHub(As an aside: nice to see thattransformershas grown so much that it is taken into account by the big developer league.)Indeed!
2020-07-28T22:01:46Z
So after a few weeks of experimenting with this functionality, we have discovered this whole thing is a little bit more complicated than we initially thought.As mentioned above in this thread, there is a problem withtorch.nn.DataParallel. It also causes problem when exporting to ONNX or others. Those can be fixed, but the biggest problem is that TensorFlow does not like those new model output types at all, so we are now left with two different APIs for PyTorch and TensorFlow, which is not what we want for the library.On the other hand, we have a version that works with TensorFlow and doesn’t breaktorch.nn.DataParallelin#5981. Its problem is that it breaks backward-compatibility since this model output behaves more like a dict than tuple, and unpacking it (e.g. doingloss, logits = output) iters through the keys and not the values. This breaks pretty much all tests and examples since we use that unpacking a lot.After lots of internal discussion, and since we can’t find a way to have the best of both world (e.g., fix the current model output without breaking change to work with TensorFlow andtorch.nn.DataParallel) we have decided to revert a bit those changes. I’ll start implementing the following tomorrow:The currentModelOutputwill become the more-dict-version in#5981and it will be used for all TensorFlow models as well as PyTorch.Since this is a breaking change, we’ll switch the flip on the other side: instead of getting the new model outputs and having the possibility of getting back a tuple withreturn_tuple=True, we will go back to return tuple by default and have areturn_dictargument you can pass to get the new model outputs.Update all our examples to usereturn_dict=Trueand discourage the use of unpacking, so that everyone has time to get use to the future breaking change to come. Test and fix potential side effects.In a few months (and probably during a major release), switch the flip and makereturn_dict=Truethe default for all models
2021-03-11T13:08:11Z
@sguggerWhat is the guideline for using ModelOutput classes/objects and the created subclasses in our own LM classes? ModelOutput seems to be HF internal class. TIA