[Announcement] Generation: Get probabilities for generated output
Hey everyoneWe have just merged aPRthat exposes a new function related to.generate(),compute_transition_scores. With this function, you can quickly solve any problem that requires the probabilities of generated tokens, for any generation strategy. It is also nicely documented – seehere.How can this function help you? Let me give you two simple examples!Example 1 -- print the probabilities for the output generated by Greedy Searchfrom transformers import GPT2Tokenizer, AutoModelForCausalLM
import numpy as np

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer.pad_token_id = tokenizer.eos_token_id
inputs = tokenizer(["Today is"], return_tensors="pt")

outputs = model.generate(**inputs, max_new_tokens=5, return_dict_in_generate=True, output_scores=True)
transition_scores = model.compute_transition_scores(
    outputs.sequences, outputs.scores, normalize_logits=True
)

input_length = inputs.input_ids.shape[1]
generated_tokens = outputs.sequences[:, input_length:]
for tok, score in zip(generated_tokens[0], transition_scores[0]):
    # | token | token string | logits | probability
    print(f"| {tok:5d} | {tokenizer.decode(tok):8s} | {score.numpy():.4f} | {np.exp(score.numpy()):.2%}")
# Expected output:
#|   262 |  the     | -1.4136 | 24.33%
#|  1110 |  day     | -2.6089 | 7.36%
#|   618 |  when    | -2.0096 | 13.40%
#|   356 |  we      | -1.8593 | 15.58%
#|   460 |  can     | -2.5083 | 8.14%Example 2 -- recompute the sequence scores from Beam Searchfrom transformers import GPT2Tokenizer, AutoModelForCausalLM
import numpy as np

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer.pad_token_id = tokenizer.eos_token_id
inputs = tokenizer(["Today is"], return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=5,
    num_beams=4,
    num_return_sequences=4,
    return_dict_in_generate=True,
    output_scores=True,
)
transition_scores = model.compute_transition_scores(
    outputs.sequences, outputs.scores, outputs.beam_indices, normalize_logits=False
)
# If you sum the generated tokens' scores and apply the length penalty, you'll get the sequence scores.
# Tip: set `normalize_logits=True` to recompute the scores from the normalized logits.
output_length = np.sum(transition_scores.numpy() < 0, axis=1)
length_penalty = model.generation_config.length_penalty
reconstructed_scores = transition_scores.sum(axis=1) / (output_length**length_penalty)
print(np.allclose(outputs.sequences_scores, reconstructed_scores))
# Expected output:
#TrueThere is also an interactive demoherethat makes use of these functionalities to color-code generated text according to the probabilitiesLet me know if you have comments, questions, and/or suggestions!(P.S.: This new post is also meant as a replacement for anolder one, which contains stale examples. Let’s keep all further discussion around generated token probabilities here!)
2023-01-20T21:53:44Z
@joaogantecoming from the log probabilities thread here as per your recommendation. This is a great contribution! While at it - do you plan to add utility functions to calculate log probabilities for each token if no scores are given but just token ids? More generally - given a sentence, calculate log probabilities for each token in that sentence.
2023-01-21T10:47:57Z
Hi@vblagojeThat was not on my plans, but you’re not the first person asking for them… so I’ll include it there! In summary, I think.generate()can be modified to also return the scores of the input, for decoder-only models like the GPT family or OPT.
2023-01-21T11:32:09Z
Hey@joaogante,Thank you for the prompt response. However, I meant more like a general utility method to calculate log probs for the given text regardless ofgenerate. Of course, one can do this “by hand” by applying softmax on logits, but then again, we have to check the docs, do we need to shift inputs, need to invoke gather to get logits on individual tokens, whether calculations are different for this architecture from that architecture, etc etc. Why not have some utility method that, for a given model and text, returns tokens and their logprobs?
2023-01-23T16:43:51Z
Getting anAttributeError: 'GPT2LMHeadModel' object has no attribute 'compute_transition_scores'error while trying to run. Installed transformers library usingpip install git+https://github.com/huggingface/transformers
2023-01-24T15:11:12Z
Hey@rsahaperhaps you need to add the--upgradeflag (i.e.pip install --upgrade git+https://github.com/huggingface/transformers.git)If the problem persists, please open an issue in transformers
2023-01-24T15:13:40Z
@vblagojeI see. That is easy to build, added that to my to do list
2023-01-24T15:58:09Z
Awesome,@joaoganteI have my own implementation that works well for the model we currently use, but it would undoubtedly be better to use an official API one day. I think others will find it helpful as well especially now that providers like OpenAI and Cohere offer token logprobs as part of the standard API. All the best
2023-01-26T16:16:47Z
Worked with the upgrade flag!
2023-01-27T07:02:20Z
@joaoganteThanks for your contribution. I’ve been looking for this and it works well with T5 with the latest tag (v 4.26.0). Can you double check if my understanding is correct?transition_scores_{i} = log(P(y_i|y_1, …, y_{i-1}, x)),sequence_scores (from.generate()) = (1/t)* sum_{i=1}^{t} log(P(y_i|y_1, …, y_{i-1}, x))Therefore, as stated in the documentation, the length normalized sum oftransition_scoresequals thesequence_scores? (but the documentation states that it’s only guaranteed whennormalize_logits=False. Does this mean the original implementation ofsequence_scoresdoes not normalize logits across the vocab dimension?)Also, my understanding ofscoresfrom the.generate()is still shaky. Could anyone share their understanding on howscoresfrom the.generate()method is different fromtransition_scoresfrom.compute_transition_scores()?
2023-01-27T12:49:36Z
Hi@vblagoje. I’ve been looking for such implementation. Would you be willing to share yours?
2023-01-27T16:49:33Z
Nice! This is not exposed in the hosted inference API, right?
2023-01-28T11:35:12Z
@jas-holet me answer in two parts:We favor inference endpoints (Inference Endpoints - Hugging Face) to inference API, as the former are much more flexible.Inference endpoints are fully flexible, so it is surely doable. The question is how complex it is to do itI have no idea, so I will build a demo and share it here!
2023-01-28T11:51:46Z
@SeohyeongI agree that formalizing the definitions for those three things (scores,transition_scores, andsequence_scores) can be tricky. Especially because we, intransformers, have to ensure that what we build is backward compatible, which means that our naming of variables can become slightly inaccurate.scoresare the UNNORMALIZED log probabilities (they will only be normalized if you passrenormalize_logits=Trueto.generate()). Sadly, we are stuck with unnormalized log probabilities to ensure backwards compatibilityIt is a tuple containing one entry for each generated token. Each tuple member is a tensor containing the log probabilities from the model, for all words in the vocabulary. These log probabilities are gathered AFTER manipulating them with our logits processors (e.g. after setting the probability of certain words to 0, after applying top k, …);sequence_scorescorrespond to the sum of the log probabilities inscores. Iflength_penaltyis not 0,sequence_scoreswill be divided bysequence_length**length_penalty;transition_scorescontainsscoresfor the tokens that were selected at generation time. You can setnormalize_logits=Trueto ensure they are normalized at a token level (i.e. to ensure the sum of probabilities for all vocabulary at a given generation step is 1).An important detail: becausescoresis unnormalized by default, it also means thatsequence_scoresis the sum of unnormalizedscores, which can be undesirable. A workaround is to createtransition_scoresfromcompute_transition_scoreswithnormalize_logits=True, from which you can recompute the sequence scores using normalized scores.Hopefully we can correct this unintuitive behavior when we release the next major version (transformersv5.0)
2023-01-31T08:22:54Z
Hey@Seohyeonghere it isdef to_tokens_and_logprobs(model, tokenizer, input_texts):
  encoded_input_texts = tokenizer(input_texts, padding=True, return_tensors="pt")
  outputs = model(encoded_input_texts.input_ids, labels=encoded_input_texts.input_ids)
  probs = torch.log(outputs.logits.softmax(dim=-1)/100).detach()

  # collect the probability of the generated token
  # we need to add a dummy dim in the end to make gather work
  gen_probs = torch.gather(probs, 2, input_ids[:, :, None]).squeeze(-1)

  batch = []
  for input_sentence, input_probs in zip(input_ids, gen_probs):
    text_sequence = []
    for token, p in zip(input_sentence, input_probs):  
      if token not in tokenizer.all_special_ids:
        text_sequence.append((tokenizer.decode(token),p.item()))
    batch.append(text_sequence)    
  return batchThen you would call this method with:to_tokens_and_logprobs(model, tokenizer, ["Give me something", "Good morning", "Hello, how are you?"])And get the tokens and probs pairs:[[('Give', -13.964306831359863),
  ('me', -5.264768123626709),
  ('something', -6.42432165145874)],
 [('Good', -20.744667053222656), ('morning', -4.624837398529053)],
 [('Hello', -8.46367073059082),
  (',', -5.478875637054443),
  ('how', -6.255255222320557),
  ('are', -4.785802841186523),
  ('you', -4.666200160980225),
  ('?', -4.6729416847229)]]It would be great if@joaogantecould review this code snippet
2023-02-02T03:03:33Z
Thanks for the explanation! I understand why certain things are implemented in such ways. I personally had a difficult time trying to find definitions of scores, sequence_scores, and transition_scores and how they’re calculated. It’d be great if this chuck of your explanation is added to the official documentation.Also, I do run into RuntimeError by tensor size mismatch by runningcompute_transition_scores()with a batch size > 1. Does this method only support input with batch size 1 for now?
2023-02-02T11:07:33Z
Hey@vblagojeI believe you forgot one tiny detail, other than that looks like a solid implementation! In a nutshell, these models return the probabilities for the next token, which means thatlogits[batch_idx, seq_idx, vocab_idx]actually contains the logits corresponding toinput_ids[batch_idx, seq_idx + 1]. This implies that the pairing between the probabilities and the tokens has to be shifted by oneHere is a modified script:from pprint import pprint
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM


def to_tokens_and_logprobs(model, tokenizer, input_texts):
    input_ids = tokenizer(input_texts, padding=True, return_tensors="pt").input_ids
    outputs = model(input_ids)
    probs = torch.log_softmax(outputs.logits, dim=-1).detach()

    # collect the probability of the generated token -- probability at index 0 corresponds to the token at index 1
    probs = probs[:, :-1, :]
    input_ids = input_ids[:, 1:]
    gen_probs = torch.gather(probs, 2, input_ids[:, :, None]).squeeze(-1)

    batch = []
    for input_sentence, input_probs in zip(input_ids, gen_probs):
        text_sequence = []
        for token, p in zip(input_sentence, input_probs):
            if token not in tokenizer.all_special_ids:
                text_sequence.append((tokenizer.decode(token), p.item()))
        batch.append(text_sequence)
    return batch


tokenizer = AutoTokenizer.from_pretrained("gpt2", padding_side="left")
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained("gpt2")
model.config.pad_token_id = model.config.eos_token_id

input_texts = ["One plus one is two", "Good morning", "Hello, how are you?"]

batch = to_tokens_and_logprobs(model, tokenizer, input_texts)
pprint(batch)which yields[[('One', -5.882715702056885),
  (' plus', -9.785109519958496),
  (' one', -0.7229145169258118),
  (' is', -2.494063377380371),
  (' two', -6.137458324432373)],
 [('Good', -7.5790300369262695), (' morning', -1.826707124710083)],
 [(',', -2.343151807785034),
  (' how', -4.339702606201172),
  (' are', -2.6824729442596436),
  (' you', -0.4109247326850891),
  ('?', -1.8950778245925903)]]Notice how high the logits for certain obvious tokens are, likemorning,you, or?! Checking these tokens is always a good sanity checkAlso, look at the last sentence: there are no logits for the first token. If you want the logits for that token, you need to add extra padding on the left, so that the first text token is not the actual first token fed to the model.
2023-02-02T11:49:18Z
@Seohyeongyeah, we have a very big backlog and improving the docstrings is part of it. Our team is actually quite small (I’m the only one working ongenerateat the moment), so PRs with improvements are very welcomeCan you open an issue on GitHub with what you are experiencing? (compute_transition_scores()with a batch size > 1)
2023-02-02T15:28:27Z
Excellent, you are right, thank you@joaogante
2023-02-04T06:45:05Z
Hi, thanks for the PR and explanations! I am still a bit confusedI want to get the probability of each output under (for example) a beam search. In other words,P_\theta(output|input)where\thetais language model parameters,outputis the output of the LM, andinputis the input. Since it is nothing but a conditional probability, I guess this is clearly defined. Then, how should I get this value?My current guess:# modified from "example 1" and "example 2"
outputs = model.generate(**inputs, max_new_tokens=5, return_dict_in_generate=True, output_scores=True)
transition_scores = model.compute_transition_scores(
    outputs.sequences, outputs.scores, normalize_logits=True # NOTE normalize SHOULD be true?
)
output_length = inputs.input_ids.shape[1] + np.sum(transition_scores.numpy() < 0, axis=1)
length_penalty = model.generation_config.length_penalty
probabilities = torch.exp(transition_scores.sum(axis=1) / (output_length**length_penalty))
# what I want is the `probabilities` (correct?)Looks it is wrong, because I can get probabilities thatsum up to larger than 1
No date available
No reply text available