[Announcement] GenerationOutputs: Scores, Attentions and Hidden States now available as outputs to generate
This PR:https://github.com/huggingface/transformers/pull/9150addedGenerationOutputssimilar toModelOutpusfor Pytorch Transformers.=> Check out the corresponding tweet with an example:https://twitter.com/SimonBrandeis/status/1346858472000937984.The community has been asking for quite some time for this feature, seehttps://github.com/huggingface/transformers/issues/7654https://github.com/huggingface/transformers/issues/8656https://github.com/huggingface/transformers/issues/3891When settingreturn_dict_in_generatethe PyTorch.generate()method now returnsGenerationOutputsthat are very similar in style toModelOutputs. The usual generationoutput_idscan then be accessed underoutputs["sequences"].For all “non-beam search” generations (greedy_searchandsample), one has now access toallattentionsof every layer at every generation step (be careful memory might blow up here) ifoutput_attentios=Trueallhidden_statesof every layer at every generation step ifoutput_hidden_states=True.scores. Now the scores correspond to theprocessedlogits -> which means the models lm head output after applying all processing functions (liketop_portop_korrepetition_penalty) at every generation step in addition ifoutput_scores=True.For all “beam_search” methods:allattentionsand allhidden_statesof every layer at every generation step ifoutput_attentionsandoutput_hidden_statesare set toTruescoresnow correspond to all processed lm head logits + the current beam_scores for each output token. This score (next_token_scores + beam_scores) is the most important score at every generation step so we decided to output this score.sequence_scoresIn addition to the three outputs above for beam searchoutput_scores=Truealso returns the final “beam score” for each returnedsequence(seehttps://twitter.com/SimonBrandeis/status/1346858472000937984)This should make it easier to analyze the generation of transformer models and should also allow the user to build “confidence” graphs from thescoresandsequence_scores.For more in-detail information please check out the docs:https://huggingface.co/transformers/master/internal/generation_utils.html#generate-outputsWe would be very happy for some feedback from the community if theGenerationOutputsmeets the expectations.
2021-01-13T12:40:21Z
This might be interesting as well:Generation Probabilities: How to compute probabilities of output scores for GPT2🤗TransformersNow that it is possible to return the logits generated at each step, one might wonder how to compute the probabilities for each generated sequence accordingly. 
The following code snippet showcases how to do so for generation with do_sample=True for GPT2: 
import torch
from transformers import AutoModelForCausalLM
from transformers import AutoTokenizer


gpt2 = AutoModelForCausalLM.from_pretrained("gpt2", return_dict_in_generate=True)
tokenizer = AutoTokenizer.from_pretrained("gpt2")

input_ids …