Guide: The best way to calculate the perplexity of fixed-length models
Hey all. Just thought you might be interested in a page I just added to the research docs on theperplexity of fixed-length models.Perplexity (PPL) is defined as the exponential average of a sequence’s negative log likelihoods. For at-length sequenceX, this is defined,\text{PPL}(X) = \exp \left\{ -\frac{1}{t} \sum_i^t \log p_\theta (x_i|x_{<i})  \right\}But with fixed-length models (like most transformers), we can’t always condition on the entire preceding subsequence when predicting each token.The initial instinct for many in dealing with this problem is to break the whole sequence into segments equal to the model’s max input size and calculate the likelihoods of each segment independently. This not the best approach, however, since it gives the model very little context to use for prediction at the beginning of each segment. I’ll illustrate this with the following gif where we imagine a model with a max input size of 6 adding up the log-likelihoods for the sentence, “Hugging Face is a startup based in New York City and Paris”ppl_chunked1200×160 352 KBWhen the model starts the second segment, it has to try to predict the word “in” without any context, even though we have 5 words before it that the model could be using (since we said the max input size is 6).A better approach is to instead employ asliding windowstrategy, where you continually move the context across the sequence, allowing the model to take advantage of the available context.ppl_sliding1200×160 373 KBThis is slower to compute, but will typically yield better scores and is actually much closer to the way the sequence probabilities are formally decomposed (e.g. see the the equation above).In theguide, we show how to do this in a strided way with GPT-2. When using the first, naive approach, GPT-2 gets a PPL of19.64on WikiText-2. In contrast, when we use a strided sliding window, this score improves dramatically down to16.53.
2020-10-20T20:37:42Z
Hi, I have a question about the perplexity calculation from theguide.Why do we divide byiin the example, seeppl = torch.exp(torch.stack(lls).sum() / i)?If you have a codebase or paper that exemplifies this behaviour could you please share it?Thanks!
2020-10-20T22:01:25Z
Hmm yes, you should actually divide byencodings.input_ids.size(1)sinceidoesn’t account for the length of the last stride.I also just spotted another bug. When the length of the last segment is less thanstride, thelog_likelihoodcalculation is slightly off. The difference in scores won’t be significant, but I’ve update the guide on master. This should be right:max_length = model.config.n_positions
stride = 512

lls = []
for i in tqdm(range(0, encodings.input_ids.size(1), stride)):
    begin_loc = max(i + stride - max_length, 0)
    end_loc = min(i + stride, encodings.input_ids.size(1))
    trg_len = end_loc - i    # may be different from stride on last loop
    input_ids = encodings.input_ids[:,begin_loc:end_loc].to(device)
    target_ids = input_ids.clone()
    target_ids[:,:-trg_len] = -100

    with torch.no_grad():
        outputs = model(input_ids, labels=target_ids)
        log_likelihood = outputs[0] * trg_len

    lls.append(log_likelihood)

ppl = torch.exp(torch.stack(lls).sum() / end_loc)Does that answer your question?
2020-10-21T14:02:18Z
yep thanks Joe!I was thinking something similar but wanted to check in case I was missing something
2021-03-01T22:57:39Z
Hi@joeddav- the input_ids and target_ids are the same. Shouldn’t target_ids be shifted by one?
2021-03-01T23:09:39Z
Nevermind - just found out that labels are shifted inside the model and the loss for last one gets ignored.huggingface.coOpenAI GPT2We’re on a journey to advance and democratize artificial intelligence through open source and open science.labels(torch.LongTensorof shape(batch_size, sequence_length), optional) – Labels for language modeling. Note that the labelsare shiftedinside the model, i.e. you can setlabels = input_idsIndices are selected in[-100, 0, ..., config.vocab_size]All labels set to-100are ignored (masked), the loss is only computed for labels in[0, ..., config.vocab_size]
2021-07-15T19:44:29Z
@joeddavI read and read the page several times. Thank you!What would be the simplest way of accessing a perplexity score for a sentence and its parts? I’m building an application in NodeJS and hoping to access a perplexity score via an API - paid is fine for now. I think I could set up the Python model somewhere and expose it via an API but this hopefully will come later after some MVP testing.Thank you again!
2021-10-16T20:01:19Z
I am wondering whether this is still correct. So what you do is, for all input sequences:neg_log_likelihood = outputs[0] * trg_lenYet the first output of causal LMs isCrossEntropyLoss, not NLLL. So from that you can just get the mean CE loss from all sequences and get the exponential.EDIT: that is also how it is implemented in the Trainer and run_clm.py script. First gather all losses for  all batches in the whole validation set and take the mean.github.comhuggingface/transformers/blob/11c69b80452fae4b13c6d8bc22bdc19f3a752199/src/transformers/trainer.py#L2353-L2354if all_losses is not None:metrics[f"{metric_key_prefix}_loss"] = all_losses.mean().item()Then take the exponential.github.comhuggingface/transformers/blob/11c69b80452fae4b13c6d8bc22bdc19f3a752199/examples/pytorch/language-modeling/run_clm.py#L495# Evaluationif training_args.do_eval:logger.info("*** Evaluate ***")metrics = trainer.evaluate()max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))try:perplexity = math.exp(metrics["eval_loss"])except OverflowError:perplexity = float("inf")metrics["perplexity"] = perplexitytrainer.log_metrics("eval", metrics)trainer.save_metrics("eval", metrics)kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "text-generation"}if data_args.dataset_name is not None:kwargs["dataset_tags"] = data_args.dataset_name
2021-11-26T15:26:25Z
I’d agree with@BramVanroy, any thoughts@joeddavon the above post?BramVanroy:Yet the first output of causal LMs isCrossEntropyLoss, not NLLL. So from that you can just get the mean CE loss from all sequences and get the exponential.I don’t understand the multiplication bytrg_lenin this example. Also on my dataset it explodes the perplexity by orders of magnitude above a uniform upper bound oflog(|Vocab Size|)
2021-12-16T03:36:34Z
I think it is correct forPerplexity of fixed-length modelssince batch size is 1.B.T.W. most libraries like simpletransformers implement perplexity calculation by taking exp(sum_of_loss_in_all_batches / num_of_batch) likesimpletransformers/language_modeling_model.py at 254aaaa218635ef68f80ad1917403e7b7e24d710 · ThilinaRajapakse/simpletransformers · GitHub