What knowledge distillation approaches have been proposed to enable small language models to generate rationales for their predictions?
Knowledge Distillation There exist some works
that explore the idea of distilling rationales knowl-
edge from a large LM to a small LM as the student.
Chan et al. proposed to learn a student model that
only predicts answers from a teacher model that is
augmented with rationales. Eisenstein et al. pro-
posed to train the student to extract the sentence
containing the answer, which is not applicable to
reasoning tasks that require background knowledge.
Shridhar et al. proposed to train the student to ask
and answer sub-questions necessary for decompos-
ing the main question, which is tailored to solve
math word problems (Cobbe et al., 2021) with an
equation generator for guiding the student while
we do not have such a constraint. Li et al. proposed
to train the student on the joint task of generating
the answers and the rationales, which only act as a
regularization and do not affect the student’s pre-
diction during inference. More importantly, both
Shridhar et al. and Li et al. do not consider the
faithfulness of the rationales, which is critical for
examining the behavior of the student.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Explainable question answering systems should produce not only accurate answers but also rationales that justify their reasoning and allow humans to check their work. But what sorts of rationales are useful and how can we train systems to produce them? We propose a new style of rationale for open-book question answering, called \emph{markup-and-mask}, which combines aspects of extractive and free-text explanations. In the markup phase, the passage is augmented with free-text markup that enables each sentence to stand on its own outside the discourse context. In the masking phase, a sub-span of the marked-up passage is selected. To train a system to produce markup-and-mask rationales without annotations, we leverage in-context learning. Specifically, we generate silver annotated data by sending a series of prompts to a frozen pretrained language model, which acts as a teacher. We then fine-tune a smaller student model by training on the subset of rationales that led to correct answers. The student is"honest"in the sense that it is a pipeline: the rationale acts as a bottleneck between the passage and the answer, while the"untrusted"teacher operates under no such constraints. Thus, we offer a new way to build trustworthy pipeline systems from a combination of end-task annotations and frozen pretrained language models.
[{"first":"Jacob","last":"Eisenstein","middle":[]},{"first":"Daniel","last":"Andor","middle":[]},{"first":"Bernd","last":"Bohnet","middle":[]},{"first":"Michael","last":"Collins","middle":[]},{"first":"David","last":"Mimno","middle":[]}]
KNIFE: Knowledge Distillation with Free-Text Rationales
Free-text rationales (FTRs) follow how humans communicate by explaining reasoning processes via natural language. A number of recent works have studied how to improve language model (LM) generalization by using FTRs to teach LMs the correct reasoning processes behind correct task outputs. These prior works aim to learn from FTRs by appending them to the LM input or target output, but this may introduce an input distribution shift or conﬂict with the task objective, respectively. We propose KNIFE, which distills FTR knowledge from an FTR-augmented teacher LM (takes both task input and FTR) to a student LM (takes only task input), which is used for inference. Crucially, the teacher LM’s forward computation has a bottleneck stage in which all of its FTR states are masked out, which pushes knowledge from the FTR states into the task input/output states. Then, FTR knowledge is distilled to the student LM by training its task input/output states to align with the teacher LM’s. On two question answering datasets, we show that KNIFE signiﬁcantly outperforms existing FTR learning methods, in both fully-supervised and low-resource settings.
[{"first":"Aaron","last":"Chan","middle":[]},{"first":"Zhiyuan","last":"Zeng","middle":[]},{"first":"Wyatt","last":"Lake","middle":[]},{"first":"Brihi","last":"Joshi","middle":[]},{"first":"Hanjie","last":"Chen","middle":[]},{"first":"Xiang","last":"Ren","middle":[]}]
Contrastive Decoding: Open-ended Text Generation as Optimization
Given a language model (LM), maximum probability is a poor decoding objective for open-ended generation, because it produces short and repetitive text. On the other hand, sampling can often produce incoherent text that drifts from the original topics. We propose contrastive decoding (CD), a reliable decoding approach that optimizes a contrastive objective subject to a plausibility constraint. The contrastive objective returns the difference between the likelihood under a large LM (called the expert, e.g. OPT-13B) and a small LM (called the amateur, e.g. OPT-125M), and the constraint ensures that the outputs are plausible. CD is inspired by the fact that the failures of larger LMs (e.g., repetition, inco- herence) are even more prevalent in smaller LMs, and that this difference signals which texts should be preferred. CD requires zero additional training, and produces higher quality text than decoding from the larger LM alone. It also works across model scales (OPT-13B and GPT2-1.5B) and significantly outperforms four strong decoding algorithms (e.g., nucleus, top-k) in automatic and human evaluations across wikipedia, news and story domains.
[{"first":"Xiang Lisa","last":"Li","middle":[]},{"first":"Ari","last":"Holtzman","middle":[]},{"first":"Daniel","last":"Fried","middle":[]},{"first":"Percy","last":"Liang","middle":[]},{"first":"Jason","last":"Eisner","middle":[]},{"first":"Tatsunori","last":"Hashimoto","middle":[]},{"first":"Luke","last":"Zettlemoyer","middle":[]},{"first":"Mike","last":"Lewis","middle":[]}]
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
Step-by-step reasoning approaches like chain-of-thought ( CoT ) have proved to be a very effective technique to induce reasoning capabilities in large language models. However, the success of the CoT approach depends primarily on model size, and often billion parameter-scale models are needed to get CoT to work. In this paper, we propose a knowledge distillation approach, that leverages the step-by-step CoT reasoning capabilities of larger models and distils these reasoning abilities into smaller models. Our approach D ECOMPOSI - TIONAL D ISTILLATION learns a semantic decomposition of the original problem into a sequence of subproblems and uses it to train two models: a) a problem decomposer that learns to decompose the complex reasoning problem into a sequence of simpler sub-problems and b) a problem solver that uses the intermediate subproblems to solve the overall problem. On a multi-step math word problem dataset (GSM8K), we boost the performance of GPT-2 variants up to 35% when distilled with our approach compared to CoT . We show that using our approach, it is possible to train a GPT-2-large model (775M) that can outperform a 10X larger GPT-3 (6B) model trained using CoT reasoning. Finally, we also demonstrate that our approach of problem decomposition can also be used as an alternative to CoT prompting, which boosts the GPT-3 performance by 40% compared to CoT prompts.
[{"first":"Kumar","last":"Shridhar","middle":[]},{"first":"Alessandro","last":"Stolfo","middle":[]},{"first":"Mrinmaya","last":"Sachan","middle":[]}]
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
To be trustworthy and useful, a question answering system should be able to explain its reasoning and offer evidence. In open-book question answering, such explanations often take the form of rationale masks, which are subsets of tokens from the original passage (Lei et al., 2016). However, a challenge for mask-based rationales is that subspans of the original passage are not meant to be read alone: coherent texts contain anaphora, ellipsis, and other cohesion-building elements that limit   (Dasigi et al., 2021) with the generated rationale shown in dark text. The markup, shown in square brackets, makes it possible to find a more concise rationale than could be extracted from the original passage.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
the interpretability of individual subspans when extracted from the discourse (Halliday and Hasan, 1976). An example is shown in Figure 1, in which the key sentence mentions the answer only through the nominal the grieving goddess. A sufficient rationale for this answer would have to include an additional sentence introducing the entity Astarte and binding it to the nominal in the sentence that describes the key event.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Despite their limitations, extractive rationales have an important advantage over free-text explanations: they are directly linked to the original passage, making it easy for human readers to assess the reliability of the evidence for themselves. In this paper, we present a new style of explanation, called markup-and-mask, which preserves the attributability of extractive rationales while overcoming the problems created by extracting propositions from the discourse in which they were written. The key idea is that discourse context is made explicit in free-text markup and then rationales are extracted from the marked-up passages.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Rather than annotating markup-and-mask rationales manually, we present a new training method Figure 2: Schematic of the prompt chain used to produce silver data to fine-tune the honest student. At the decontextualization stage, one prompt is applied per sentence in the passage in sequence; the remaining stages use exactly one prompt each. that leverages the in-context learning capability of large pretrained language models ( Figure 2). First, we prompt a frozen language model to produce markup that sequentially decontextualizes each sentence in each passage in the training set. Next, we prompt the same language model to produce answers and chain-of-thought rationales from the decontextualized passage. Finally, we check that the rationale supports the answer by prompting the language model again, this time replacing the full passage with the rationale. When the answer approximately matches the ground truth, we add the rationale and markup to a silver training set. These silver annotations are used to train an "honest student" that is constrained to follow a pipeline: first generate question-neutral markup, then select a question-based rationale, and finally produce an answer using the rationale and not the passage.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Evaluation shows a number of favorable properties to this approach: (1) unlike other maskingbased methods, accuracy on SQuAD is nearly as good as that of an end-to-end system; (2) on QuoRef, markup significantly increases accuracy;
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
(3) answers that can be validated by a rationale are much more likely to be correct (+20 F 1 ); (4) rationales usually entail the answers; (5) despite having access to only five human-annotated examples of decontextualizing markup, the student model produces markup that is more accurate than a system that was fine-tuned on 11,290 gold-labeled training examples. The student models outperform their teacher on all three of our key metrics -overall accuracy, entailment rate of rationales, and accuracy of decontextualizing markup -highlighting the positive impact of distillation from pretrained language models.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
To summarize the contributions of this paper:
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
• We propose markup-and-mask rationales for open-book question answering, which preserve a direct link to the original evidence text but use markup to incorporate non-local information.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
• We show that it is possible to train models to produce markup-and-mask rationales without explicit supervision, by leveraging the capabilities of a pretrained language model.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
• We present a general strategy for using pretrained language models to help supervise interpretable pipeline systems in which annotations are available for only the end task.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
• We empirically validate the proposed approach, showing that the resulting rationales:
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
(1) support accurate question answering; (2) help quantify predictive uncertainty;
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
(3) are more likely to entail the predicted answers than "chain-of-thought" rationales produced alongside the answer; and (4) accurately match human-written decontextualizations.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Our goal is to fine-tune a student model to produce markup-and-mask rationales. Lacking labeled examples, we obtain silver annotations by applying three distinct prompting patterns to the pretrained language model PaLM (Chowdhery et al., 2022) (540-billion parameter version), which we refer to as the teacher model. Each prompt combines passages and questions from open-book question answering datasets, along with the outputs of previous prompts, in an approach that has been called prompt chaining . There are three steps to the silver annotation process: (1) decontextualization;
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
(2) chain-of-thought question answering;
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
(3) rationale validation. The prompt chain is shown in Figure 2.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Decontextualization. The goal of the decontextualization step is to add free-text markup  An example is shown in Figure 4. We use a handcrafted prompt with five examples, shown in appendix A. We proceed incrementally through the document, decontextualizing each sentence using the previous k decontextualized sentences as context. This enables information to propagate through the document. The capabilities and limitations of this approach are highlighted in Figure 3, which shows some typical outputs. The markup resolves pronominal references she and her and the nominal references this painting and this phenomenon. Perhaps most impressively, the elliptical expression despite this is decontextualized with the markup [the fact that nudes were extremely rare. . . ]. However, by the end of the document, we have lost track of the first name of the artist, so that the artist is decontextualized as only [Velázquez], rather than with the full name. Future work may address this issue by exploring more sophisticated strategies than simple autoregressive decontextualization.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Chain-of-thought question answering. In chain-of-thought prompting, the language model is asked to first generate a rationale before producing an answer . For open-book question answering, we take the rationale to be a sentence that is extracted from the passage and which contains the answer, as shown in Figure 5.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
We construct question-specific few-shot prompts by concatenating several exemplars in which a question, passage, rationale, and answer are shown, before providing the question and passage for the instance to be predicted. The exemplars are drawn from the training set, selecting questions with the highest BM25 similarity to the target question (Robertson et al., 2009). Exemplars are added until we reach a limit of 1024 sentencepiece tokens in the prompt (Kudo and Richardson, 2018); for the QuoRef dataset, this amounts to two or three exemplars in most cases.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
To generate the rationales in the exemplars, we enumerate all sentences in the passage that contains an exact match to the answer and select the one with the highest BM25 similarity to the exemplar's question. Each sentence is considered in both its original surface form and with decontextualizing markup. If no sentence contains an exact match to the answer, then the question is not included as an exemplar. However, prompts are constructed for all training set examples, even when no rationale can be extracted using this heuristic.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Rationale validation. Finally, to validate the rationales that were generated in the chain-of-thought stage, we perform a final validation stage in which the teacher model must answer questions based only on the generated rationales. As in the previous stage, we include each training set example and construct in-prompt exemplars by BM25 similarity to other questions in the training set. Because this stage does not include full passages, we can  fit many more exemplars while remaining under the budget of 1024 tokens, on the order of 20 per prompt. The resulting "faithful answers" are then used to filter the fine-tuning data that is exposed to the student model.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
The prompt chain described in Section 2 produces markup-and-mask rationales and uses them to answer questions. However, there are two main reasons to distill this teacher model into a smaller "honest student." The first reason is efficiency: the prompt chain requires several calls to the large language model; because it is more specialized, the student model can potentially be smaller. The second reason is accuracy: in the teacher model, the training set is used only for in-context learning, with only a few examples per prompt; fine-tuning can make use of more gold answers, in combination with silver rationales.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Use each passage to answer the question, and cite the most relevant sentence as an explanation.  To fine-tune the student model, we use as training data the gold answers and the rationales produced by the teacher model. Because our goal is to train an honest student, we implement the student model as a pipeline: it must first produce the decontextualizing markup without seeing the question, then generate a rationale from the passage (conditioned on the question and the marked-up passage), and finally produce an answer (conditioned on the question and the generated rationale). Critically, the student has no access to the full passage when generating the answer. Each step of the pipeline is implemented as a text-to-text model using the t5x library (Roberts et al., 2022), and the steps are trained in a single multi-task model. The specific tasks for the student model are:
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Decontextualizing markup. As in the teacher model, decontextualization is performed autoregressively, with one training example per sentence. The target output is the markup produced by the teacher model.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Span selection. The input to the span selection task is a concatenation of the question and the decontextualized passage, and the target output is the rationale generated by the teacher in the chain-of-thought QA step. At training time the decontextualized passages are from the teacher; at prediction time they are from the decontextualizing markup step in the student pipeline.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Rationale-based reading comprehension. At training time, the input is a concatenation of the question and the teacher model's rationale; the target output is the gold answer. At prediction time, the input includes the rationale produced by the span selection step in the student pipeline.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
End-to-end reading comprehension. For comparison, we also train an end-to-end reading comprehension task, in which the input is a concatenation of the question and the full passage. The target output is the gold answer and no rationale is produced.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
The decontextualization task aligns closely to the decontextualization prompt, but the student model is trained by fine-tuning while the teacher model relies only on in-context learning. Unlike the chainof-thought prompt described in Section 2, the span selection task does not produce an answer; the rationale-based reading comprehension task is conceptually similar to the rationale validation prompt, but again, the student model uses fine-tuning rather than in-context learning. To build a cleaner silver training set, we train only on the rationales that led to approximately correct answers at both the chainof-thought stage (using the entire passage) and the validation stage (using the rationale alone). Specifically, we score the generated answers at both stages, and exclude examples for which either answer has an F 1 < 0.5.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
We evaluate on two datasets: QuoRef (Dasigi et al., 2021) and the version of SQuAD (Rajpurkar et al., 2016) from the MRQA shared task (Fisch et al., 2019). For each dataset, we run PaLM on the training data to produce silver annotations of the markup-and-mask rationales, as described above. The decontextualization step is autoregressive, in the sense that the decontextualization for sentence t is part of the prompt for decontextualizing sentence t + 1. This makes it difficult to use the more efficient bulk inference procedure that we apply in the other parts of the prompt chain. For this reason, we use only a fraction of the SQuAD training data (12000 questions). We then use PaLM's out- Teachers (540B) PaLM in-context 73.7 / 86.2 57.9 / 66.7 PaLM in-context (+markup) 71.9 / 84.9 50.6 / 60.0 Table 1: Overall exact match / F 1 on open-book question answering. The end-to-end system predicts the answer directly from the passage; the markup+mask system predicts the answer from a rationale that includes both masking and markup; the mask-only system uses a rationale based only on masking the original unmarked text; PaLM in-context refers to the teacher model, which uses in-context learning only.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
put as annotations to fine-tune multitask sequenceto-sequence models built on pretrained mT5 backbones (Xue et al., 2021). The results that follow are based on the mT5-XXL backbone. Comparisons across model scales are shown in Figure 6. Table 1 shows the overall performance of the student model, an end-to-end equivalent, and a masking-only ablation. On the SQuAD dataset, performance is similar across all model variants, showing that it is possible to derive causal rationales for SQuAD answers with only a minimal impact on accuracy. In contrast, prior work has found that previous unsupervised techniques for constructing rationales (Paranjape et al., 2020;Guerreiro and Martins, 2021) decreased performance by 10-20 F 1 on SQuAD (Chen et al., 2022). The pipeline method suffers a significant reduction in accuracy on QuoRef, which, as discussed below, is particularly resistant to rationale-based approaches. However, this is mitigated by the use of decontextualizing markup, reducing the gap between the endto-end predictor and the mask-based rationales by almost half.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Selective prediction. The availability of a stepby-step explanation can serve as a coarse form of calibration: examples for which explanations are available may be more likely to be accurately predicted. To test this idea, we perform an evaluation of selective prediction, in which answers are produced only when they are likely to be correct (Rodriguez et al., 2019;Kamath et al., 2020  end-to-end model and the rationale-based pipeline agree and disagree. 1 As shown in Table 2, rationalizable answers are significantly more accurate. The F 1 for rationalizable answers is more than 20 points higher than for non-rationalizable answers on both datasets, and the gap in exact match is even larger. Furthermore, most answers are rationalizable in this way. The markup-and-mask rationales play an important role in selective prediction on the QuoRef dataset, where they increase the fraction of rationalizable answers from 58% to 74%, while enlarging the F 1 gap from 13.0 to 22.1. However, on the QuoRef dataset, a better coverage-accuracy tradeoff can be obtained by thresholding on the predictive probability of the end-to-end model; on SQuAD, the tradeoff is almost identical.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
To test how often rationales are consistent with the answers, we apply natural language inference (NLI). Specifically, we ask a strong NLI system whether the rationale entails the linearization, "The answer to "[question]" is "[predicted-answer]"". This style of evaluation has been applied to other tasks involving factual consistency, such as summarization and fact verification (Honovich et al., 2022). We use a very similar NLI system, trained 1 The link between explanations and calibration was explored by Ye and Durrett (2022), who work in a chain-ofthought prompting framework. They show that when the chain-of-thought explanation is consistent with the passage, the answer is more likely to be correct. by fine-tuning t5-XXL on multiple NLI datasets (MNLI, SNLI, FEVER, PAWS, SciTail, and Vita-minC). As shown in Figure 7, the rationales produced by the pipeline student models are significantly more consistent than the chain-of-thought rationales produced by the teacher model, justifying the "honest student" moniker. On the QuoRef dataset, 64% of the rationales produced by the student model (with markup) entail that model's predicted answers, versus 47% for the teacher model with markup, and 36% without. On the SQuAD dataset, the student model achieves 81% consistency, versus 76% for the teacher model (75.5% without markup). 2 The markup also improves the consistency of the student model by 26% on QuoRef and 1% on SQuAD. It is particularly notable that markup improves the entailment rate despite the fact that the NLI system is trained on data that does not contain any markup.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
A rationale is deemed extractive when it appears as a contiguous substring in the marked-up passage, caseinsensitive and not including punctuation characters or whitespace. Extractiveness is desirable because it means that the rationales are directly grounded in the passage, similar to the notion of "verified quotes" proposed by Menick et al.  (2022). In QuoRef, the student model rationales were extractive for 92.3% of passages; in SQuAD, 90.6%. These rationales yielded 7.9x compression in QuoRef and 4.5x compression in SQuAD. Table 5 shows statistics of the markup and rationales.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Stress test. Can we view the markup-and-mask rationales as a faithful explanation of the reasoning process that produced the answer in the honest student (Jacovi and Goldberg, 2020)? While the honest student does not have access to the full passage, it may rely on knowledge obtained during pretraining. As a robustness check, we created an entityperturbed version of the SQuAD dataset, similar to Longpre et al. (2021) and Yan et al. (2022), in which entity names were automatically substituted in the passages and gold answers. Substitutions were performed by running a named entity recognizer and replacing names that appear in the answer and passage with names of other entities of the same broad class, e.g., Winston Churchill → Patti Smith, AT&T → the New York Knicks. As shown in table 3, all models are approximately 3-4 F 1 points worse than on the original evaluation set, with comparable exact match. Note that in some cases these perturbations affect the grammaticality of the passage, making the task more difficult for reasons that do not relate to the fidelity of the explanations. Overall these results suggest that the predictors mainly relied on the passage and not on knowledge obtained during pretraining.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
To measure the accuracy of the decontextualizing markup, we apply the prompt-based teacher and the fine-tuned student models to a manually decontex-tualized dataset, in which references are replaced inline rather than annotated with markup (Choi et al., 2021). Results are shown in Table 4. Both the student and teacher models exceed the reported results for a T5-base model that was fine-tuned on 11,290 in-domain examples of the decontextualization task. This shows that it is possible to learn to perform the task reasonably well from just five labeled examples, and that distillation improves performance further. Our models produce a different style of decontextualization from the test data, so it is possible that these results could be further improved.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Well-formedness. We treat markup as a freetext generation task, with no constrained decoding. As a result, the markup may not be well-formed: the removal of markup may not yield a passage that is character-wise identical to the original passage (case-insensitive). However, both the student and teacher models usually produce well-formed markup. For more than 96% of sentences in the QuoRef eval set, the decontextualization phase of the student model leaves the original text unaffected, as intended, and in 73% of passages, all markup was well formed. In SQuAD, the markup was well formed in 96% of sentences and in 85% of full passages. The difference at the passage level is due to mainly the greater length of the QuoRef passages (see Table 5). The teacher model markup was slightly less well-formed: on both the SQuAD and QuoRef datasets, approximately 94% of the teacher model's sentence decontextualizations were well formed. This indicates that the language model can learn the format of the markup task from the five incontext examples. Most of the errors were minor, such as omission of sentence-final punctuation and the erroneous movement of text from the original into markup, e.g. As a schoolboy Saint-Saëns was outstanding → As a schoolboy [Charles-Camille Saint-Saëns] was outstanding. More serious errors, such as incorrectly-formatted markup and deletion of significant original content, occurred very rarely.    Amount of markup. On the QuoRef dataset, the decontextualization model added 2.0 markup spans per sentence, with an average length of 5.3 Sen-tencePiece tokens per span (31.6 per document). This almost exactly matches the behavior of the teacher model, which added 2.1 spans, with 5.8 SentencePiece tokens per span (median=4). On the SQuAD dataset, there were fewer opportunities for decontextualization: the teacher model added 0.9 markup spans per sentence, with 6.1 tokens per span. The student model also added 0.9 spans per sentence (4.8 per document), with 5.6 tokens per span (median=4).
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
On both datasets, the biggest source of erroneous answers for the pipeline model was the selection of rationales that do not contain the gold answer. In QuoRef, many questions are multihop, requiring information found in multiple spans in the passage.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
In some cases this information can be localized by the markup -as in the motivating example shown in Figure 1. There were several reasons that markup failed to add the information necessary to provide a localized rationale:
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
• Sometimes, the necessary markup could have been supplied but was erroneously omitted: for example, to the question who is Fran's son?, the pipeline model provides the rationale The spirit reminds Scrooge [Ebenezer Scrooge] that Fran, dead for some years, is the mother of his [Ebenezer Scrooge's] nephew, which would have been sufficient if additional markup had been provided after the word nephew.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
• On the QuoRef dataset, a large class of errors relates to markup that was supplied for names. Many of the questions involve nicknames and pseudonyms, and the markup sometimes included the wrong name, which then propagated to the reading comprehension module. In other cases, part of the name was lost, such as the disappearance of the given name of Diego Velázquez in the markup in Figure 3.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
• Implicit entity references are not disambiguated by markup: for example, the sentence In 1905 Ravel, by now thirty, competed for the last time, causing a furore introduces a piano competition, which would have to be disambiguted for the sentence to serve as a rationale for the question What is the name of the competition Ravel entered for the last time in 1905, inadvertently causing a furore?
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
• Some questions reference multiple facts in the passage, such that it is difficult to imagine any markup making it possible to localize a rationale into a single sentence. For example, for the question in what country did Rakoto Frah's troupe win the gold medal?, the selected rationale is Among the 80 competitors hailing from a variety of countries, Rakoto Frah's [the artist] troupe won the gold medal, which is the only sentence mentioning the event from the question. To provide the answer, the markup would have had to supply location information for the event won the gold medal. If this was done as a general practice, significantly more markup would have been required. In general, when the rationale did not contain sufficient information to answer the question, the pipeline model "hallucinated" the requested details. However, as shown in Section 4.2, this was not typical: on both datasets, the rationales usually entail the predicted answer.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
On the positive side, the markup not only solves a number of challenging coreference problems (as shown in Figures 1 and 3), it also goes well beyond the traditional coreference resolution task. For example, the markup handles groups of entities: This enables the honest student to correctly answer the question What are the names of the three characters who go to a bar in order to celebrate being single?
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
We show how to train an honest student to produce markup-and-mask rationales for open-book question answering. The approach has three key properties: (1) the rationales are more expressive than traditional masks because they include freetext markup to enable each sentence to stand on its own; (2) the rationales are faithful because the student model must first produce the rationale and then discard all other information from the passage when answering the question; (3) the rationalegeneration system is unsupervised, training on silver data created by prompting a large language model. These properties suggest a general methodology for a new generation of pipeline systems, which could offer the benefits of interpretability and controllability while limiting annotation cost and achieving the expressivity of natural language. In future work we will explore the capability of the teacher model to support even more expressive reasoning patterns, through richer prompt chains.
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
Limitations. A number of limitations are highlighted by the error analysis in Section 4.4. More generally, we have assumed that answers can be rationalized by a contiguous span of the passage, after applying query-independent markup. This explains the lower performance of the pipelined methods on QuoRef, which contains questions that are hard to answer from any single sentence, even with query-independent markup. Another limitation is that markup is provided in a single forward pass, making it impossible to handle cataphoric references -for example, when an individual's name is revealed only at the end of a passage.
KNIFE: Knowledge Distillation with Free-Text Rationales
Pretrained language models (LMs) are typically finetuned on downstream tasks using only the task labels as supervision (Raffel et al., 2020;Devlin et al., 2018). Task labels provide feedback on whether the LM's outputs are correct, but not on whether the LM's reasoning process is correct. Consequently, an LM supervised only with task labels may be prone to overfitting the downstream training data. In contrast to conventional supervised learning, explanation-based learning (EBL) aims to improve LM generalization by also * Equal contribution. Figure 1: Learning from Free-Text Rationales. Existing methods for learning from free-text rationales (FTRs) can be categorized as FTR Input Context (input distribution shift between training and inference) or FTR Multitask Learning (task prediction and FTR generation losses may conflict). To address these issues, we propose KNIFE, which uses FTR Knowledge Distillation (KD) from an FTR-augmented teacher LM to improve a student LM's generalization ability. explaining the correct reasoning process behind a given correct output (Hase and Bansal, 2021;Narang et al., 2020;Joshi et al., 2022).
KNIFE: Knowledge Distillation with Free-Text Rationales
Most EBL works have focused on learning from extractive rationales, which explain how to solve a task by highlighting important features (e.g., tokens) in the task input (Sundararajan et al., 2017;Li et al., 2016;Chan et al., 2022b). Since extractive rationales assign an importance score to each input feature, there is a one-to-one correspondence between the features and scores. In light of this, many regularization methods have been proposed for aligning the LM's extractive rationales (i.e., how did the LM solve the task?) with humanannotated extractive rationales (i.e., how did the human solve the task?) (Joshi et al., 2022;Ross et al., 2017;Rieger et al., 2020;Liu and Avci, 2019) Meanwhile, there is growing interest in learning from free-text rationales (FTRs). Following how humans communicate, FTRs explain reasoning processes via natural language (Narang et al., 2020;Rajani et al., 2019;Camburu et al., 2018;Wei et al., 2022). Also, FTRs can reference things beyond the task input, while supporting high flexibility in content, style, and length (Rajani et al., 2019;Chan et al., 2022a). Hence, FTRs can potentially provide much richer knowledge than extractive rationales. However, this also means that, unlike extractive rationales, FTRs do not have a feature-score correspondence to leverage. Without this structure, it is unclear how to effectively incorporate FTRs into LM training.
KNIFE: Knowledge Distillation with Free-Text Rationales
During task finetuning, existing works have attempted to learn from FTRs by appending FTRs to the LM's input (Sun et al., 2022;Hase et al., 2020) or jointly training the LM to generate FTRs (Narang et al., 2020;Rajani et al., 2019;Brahman et al., 2021;Li et al., 2022;Hase et al., 2020) (Fig. 1). Still, these paradigms have key limitations. Appending FTRs to the LM's input as additional context is problematic because it either assumes access to gold (or large-LM-generated) FTRs during inference, or introduces an input distribution shift between training and inference when FTRs are unavailable during inference. Although multitask learning enables the LM to explain its behavior via the FTR during inference, this paradigm can also be problematic since the task prediction and FTR generation objectives may conflict, especially if the generated FTR does not support the LM's predicted label. Thus, both paradigms may even perform worse than vanilla (i.e., non-FTR) finetuning baselines Narang et al., 2020).
KNIFE: Knowledge Distillation with Free-Text Rationales
In light of this, how can we more effectively inject FTR knowledge into LMs, in order to improve the LM's generalization ability? We propose KNowledge DIstillation with Free-Text Rationale BottlEneck ( KNIFE). KNIFE distills FTR knowledge from an FTR-augmented teacher LM (takes both task input and FTR) to a student LM (takes only task input), which is used for inference. Crucially, the teacher LM's forward computation includes a bottleneck stage, where all of its FTR states are masked out and inaccessible beyond this stage. Beyond this bottleneck, the teacher LM is forced to use only its task input states to compute its task output states. As a result, to leverage FTR context for solving the task, knowledge must be transferred from the FTR states to the un-masked task input/output states. Since it requires FTRs as input, we cannot directly use the teacher LM for inference. Therefore, KNIFE distills FTR knowledge to the student LM by training its task input/output states to align with the teacher LM's.
KNIFE: Knowledge Distillation with Free-Text Rationales
Unlike prior FTR-based EBL methods, KNIFE does not suffer from input distribution shift (student LM takes only task input during both training and inference) or conflicting learning objectives (student LM is only trained with KNIFE's distillation losses). Plus, KNIFE has lower computational costs during inference, since the student LM does not require additional FTR inputs or need to generate additional FTR tokens. On two popular question answering (QA) datasets (OBQA, Strat-egyQA), we demonstrate that KNIFE achieves significant performance gains over existing FTRbased EBL methods ( §4.5). Also, KNIFE can outperform baselines in low-resource settings (i.e., few training instances), further verifying KNIFE's ability to improve LM generalization ( §4.6). Additionally, we validate our KNIFE design choices via extensive ablation studies ( §4.7).
KNIFE: Knowledge Distillation with Free-Text Rationales
Problem Definition Let F denote an LM for text classification. Given a task input x and a set of class labels Y = {y i }, F's goal is to predict a confidence score ρ(x, y i ) for each (x, y i ) pair, so that the predicted labelŷ = arg max y i ∈Y ρ(x, y i ) matches the gold (i.e., correct) label y * ∈ Y . In this work, we consider both multi-choice (Y can vary across task instances) and closed-set (Y is the same for all task instances) text classification.
KNIFE: Knowledge Distillation with Free-Text Rationales
Free-Text Rationales Given some (x, y i ) pair, a free-text rationale (FTR) r is a natural language text sequence that explains the reasoning process for predicting label y i (Camburu et al., 2018;Rajani et al., 2019) Most explainability works have focused on extractive rationales (Sundararajan et al., 2017;Li et al., 2016;Chan et al., 2022b), which highlight important features (e.g., tokens) in x, but there is growing interest in FTRs (Narang et al., 2020;Kumar and Talukdar, 2020;Wei et al., 2022). First, compared to extractive rationales, FTRs may be more intuitive and understandable to humans, since they follow how humans communicate in natural language (Camburu et al., 2018;. Second, unlike extractive rationales, FTRs can reference things beyond the task input (Rajani et al., 2019;. Third, in contrast to extractive rationales, FTRs support high flexibility in content, style, and length (Chan et al., 2022a). Hence, FTRs can potentially provide much richer knowledge than extractive rationales. As a result, a number of recent works have explored generating FTRs to explain LM behavior (Narang et al., 2020;Rajani et al., 2019;Camburu et al., 2018) and utilizing FTRs to improve LM decisionmaking (Sun et al., 2022;Li et al., 2022).
KNIFE: Knowledge Distillation with Free-Text Rationales
Learning from Free-Text Rationales Since extractive rationales assign an importance score to each input feature, there is a one-to-one correspondence between the features and scores (Sundararajan et al., 2017;Li et al., 2016;Chan et al., 2022b;Joshi et al., 2022). However, unlike extractive rationales, FTRs do not have such a feature-score correspondence to leverage. Without this kind of structure, it becomes unclear how to effectively incorporate FTRs into LM training. During task finetuning, existing works have attempted to learn from FTRs by appending FTRs to the LM's input (Sun et al., 2022;Hase et al., 2020) or jointly training the LM to generate FTRs (Narang et al., 2020;Rajani et al., 2019;Brahman et al., 2021;Li et al., 2022;Hase et al., 2020).
KNIFE: Knowledge Distillation with Free-Text Rationales
Still, these paradigms have key limitations. First, appending an FTR to the LM's input gives the LM additional context for solving the task (Sun et al., 2022;. However, it is unrealistic to assume access to sufficiently "correct" (i.e., explains correct reasoning process) FTRs during inference, since producing correct FTRs would already be tantamount to solving the task. Appending FTRs to the LM's input is problematic because it either assumes access to correct FTRs during inference, or introduces an input distribution shift between training and inference when correct FTRs are unavailable during inference. Second, multitask learning with task prediction and FTR generation objectives can also be problematic since these objectives may conflict, especially if the generated FTR does not support the LM's predicted label. Plus, if the LM is trained to generate the predicted label and FTR as a single output sequence (Narang et al., 2020;, then it is not uncommon for the LM to generate a predicted label that does not match any valid label candidates or even omit the predicted label entirely. Thus, both paradigms may even perform worse than vanilla (i.e., non-FTR) finetuning baselines Narang et al., 2020).
KNIFE: Knowledge Distillation with Free-Text Rationales
During task finetuning, existing works have attempted to learn from FTRs by appending FTRs to the LM's input (Sun et al., 2022;Hase et al., 2020) or jointly training the LM to generate FTRs (Narang et al., 2020;Rajani et al., 2019;Brahman et al., 2021;Li et al., 2022;Hase et al., 2020). However, these works suffer from input distribution shift or conflicting learning objectives.
KNIFE: Knowledge Distillation with Free-Text Rationales
To address these issues, we propose KNIFE, a knowledge distillation (KD) approach for injecting FTR knowledge into LMs (Fig. 2). First, KNIFE trains an FTR-augmented teacher LM (takes both task input and FTR) to solve the task, while having its FTR states masked beyond a bottleneck stage. As a result, to leverage FTR context for solving the task, knowledge must be transferred from the FTR states to the unmasked task input/output states. Second, KNIFE trains a student LM (takes only task input) so that its task input and/or output states are aligned with the teacher LM's, thus distilling FTR knowledge from the teacher to the student.
KNIFE: Knowledge Distillation with Free-Text Rationales
KNIFE consists of a teacher LM and a student LM. Below, we discuss the design of each LM type.
KNIFE: Knowledge Distillation with Free-Text Rationales
Basic LM Design Following prior works, we use text-to-text (i.e., encoder-decoder) Transformer (Vaswani et al., 2017) architectures for both teacher and student LMs (Raffel et al., 2020;Narang et al., 2020). Building upon the notation in §2, let the task input be denoted as n x -token sequence
KNIFE: Knowledge Distillation with Free-Text Rationales
Given an LM F, we can decouple F as encoder F enc and decoder F dec . By default, given x as input, F enc outputs hidden states
KNIFE: Knowledge Distillation with Free-Text Rationales
Typically, decoding is done by taking only a special start token as F dec 's input, then using greedy search or beam search to sequentially generate the remaining tokens until a special end token is generated (Raffel et al., 2020;Vaswani et al., 2017). However, when using this decoding strategy for classification, it is possible to generate an output sequence that does not match any of the labels Figure 2: KNIFE Learning Framework. KNIFE distills FTR knowledge from an FTR-augmented teacher LM (takes both task input and FTR) to a student LM (takes only task input), which is used for inference. Crucially, the teacher LM's forward computation includes a bottleneck stage, where all of its FTR states are masked out and inaccessible beyond this stage. Beyond this bottleneck, the teacher LM is forced to use only its task input states to compute its task output states. As a result, to leverage FTR context for solving the task, knowledge must be transferred from the FTR states to the unmasked task input/output states. Therefore, KNIFE distills FTR knowledge to the student LM by training its task input/output states to align with the teacher LM's. in Y . To avoid this issue, we instead produce a dedicated decoder output for each label sequence y i , by separately teacher-forcing each y i to be F dec 's input . This gives us a conditional probability P (y
KNIFE: Knowledge Distillation with Free-Text Rationales
i in y i . Then, following  and Shwartz et al. (2020), we compute ρ i = ρ(x, y i ), the confidence score for y i , by aggregating these token probabilities as:
KNIFE: Knowledge Distillation with Free-Text Rationales
Finally, we use the softmax function to normalize ρ i as label confidence probability P (
KNIFE: Knowledge Distillation with Free-Text Rationales
Given gold label y * , the goal of the downstream classification task is to train F such that P (y * | x) is maximized.
KNIFE: Knowledge Distillation with Free-Text Rationales
Teacher LM Design KNIFE first trains an FTRaugmented teacher LM to solve the task, using both the task input x and an FTR r as input. We assume that the FTR expresses a sufficiently correct reasoning process corresponding to the gold label y * . By default, r is a gold FTR, which is humanannotated to support y * .
KNIFE: Knowledge Distillation with Free-Text Rationales
Let T denote the teacher LM, which has encoder T enc and decoder T dec . Let the FTR be denoted as n r -token sequence r = [r (1) , r (2) , ..., r (nr) ].
KNIFE: Knowledge Distillation with Free-Text Rationales
Tr , e Unlike the basic LM design described earlier, T contains a special bottleneck stage in the crossattention between T enc and T dec . During crossattention, the FTR states are masked out so that T dec only has access to the task input states during decoding. Thus, for each label sequence y i , T dec produces hidden states d Ty
KNIFE: Knowledge Distillation with Free-Text Rationales
Ty i ], which we call the task output states. Note that the same FTR r (which supports gold label y * ) is used for decoding each label y i . Intuitively, we can interpret T dec 's decoding process as predicting the compatibility between candidate label y i and the correct reasoning process expressed by r. We assume that, among these labels, only y * is compatible with r.
KNIFE: Knowledge Distillation with Free-Text Rationales
Student LM Design After training the teacher LM T , KNIFE trains a student LM based on KD objectives, given only the task input x. Let S denote the student LM, with encoder S enc and decoder S dec . Since S does not have access to any FTRs, S enc only outputs task input states
KNIFE: Knowledge Distillation with Free-Text Rationales
Sx ]. Hence, unlike T , S does not have an FTR bottleneck. Analogously to T , for each label sequence y i , S dec produces task output states
KNIFE: Knowledge Distillation with Free-Text Rationales
Without access to gold FTRs, S can be realistically used during inference.
KNIFE: Knowledge Distillation with Free-Text Rationales
Recall that the teacher LM T is essentially trained to predict the compatibility of each (y i , r) pair. T 's FTR bottleneck design causes the FTR knowledge to be transferred to other parts of T . First, since T dec can only use the task input states as context, T enc must be trained to distill sufficient knowledge from r into the task input states. Second, since T dec is the component that actually performs the reasoning process to compute the final compatibility scores, T dec 's task output states must also contain useful knowledge. Thus, in KNIFE, KD is done by training the student LM S so that its task input states and/or task output states are aligned with the teacher LM's. Besides the KD losses, we can also train S with the same task loss used to train T . Now, we formally define these learning objectives. Given predicted label distribution P and target label distribution Q, we first define the task loss L task as cross-entropy loss:
KNIFE: Knowledge Distillation with Free-Text Rationales
Next, let dist denote an arbitrary distance function, e.g., mean squared error (MSE). Let L KD-in denote KNIFE's task input states based KD loss, which pushes the student LM's task input states (e 
KNIFE: Knowledge Distillation with Free-Text Rationales
Similarly, let L KD-out denote KNIFE's task output states based KD loss, which pushes the student LM's task output states (e (j)
KNIFE: Knowledge Distillation with Free-Text Rationales
Sy i ) to be closer to the teacher LM's (e (j) Ty i ):
KNIFE: Knowledge Distillation with Free-Text Rationales
Finally, let L denote the total loss, defined as:
KNIFE: Knowledge Distillation with Free-Text Rationales
with loss weights λ task , λ KD-in , and λ KD-out , such that λ task + λ KD-in + λ KD-out > 0. Note that the teacher LM is trained only with L task (i.e., λ task > 0 and λ KD-in = λ KD-out = 0), whereas the student LM may or may not use L task (i.e., λ task , λ KD-in , λ KD-out ≥ 0).
KNIFE: Knowledge Distillation with Free-Text Rationales
This section presents various experiments to demonstrate KNIFE's effectiveness for learning from FTRs. First, in fully-supervised settings, we show that KNIFE achieves significant performance gains over existing FTR-based EBL methods ( §4.5). Second, we show that KNIFE can also outperform baselines in low-resource settings (i.e., few training instances), further verifying KNIFE's ability to improve LM generalization ( §4.6). Third, we validate our KNIFE design choices via extensive ablation studies ( §4.7).
KNIFE: Knowledge Distillation with Free-Text Rationales
We consider both multi-choice (Y can vary across task instances) and closed-set (Y is the same for all task instances) text classification tasks. Since FTRs are commonly annotated for QA datasets (Wiegreffe and Marasovic, 2021), we focus on popular two QA datasets: OBQA and StrategyQA.
KNIFE: Knowledge Distillation with Free-Text Rationales
OBQA (Mihaylov et al., 2018) is a multi-choice (i.e., four-choice) QA dataset that simulates science exams. In OBQA, each question requires both scientific knowledge and commonsense reasoning to solve. StrategyQA (Geva et al., 2021) is a closedset (i.e., binary yes/no) QA dataset that tests opendomain knowledge. In StrategyQA, each question must be solved by inferring the required reasoning steps implicit in the question.
KNIFE: Knowledge Distillation with Free-Text Rationales
For all results, we report the mean and standard deviation (std) accuracy over three random seeds, using the following format: mean ± std.
KNIFE: Knowledge Distillation with Free-Text Rationales
Each table contains a Mode column, which indicates the LM input/output setting used by the given method during training. We consider four different modes: I→O, IR→O, I→OR, and I→RO. Here, I denotes task input (i.e., x), O denotes task output (i.e., y * ), and R denotes rationale (i.e., r). For I→O, the LM is trained to output y * given x as input. For IR→O, the LM is trained to output y * given [x, r] as input. For I→OR, the LM is trained to output [y * , r] given x as input. For I→RO, the LM is trained to output [r, y * ] given x as input.
KNIFE: Knowledge Distillation with Free-Text Rationales
For each On the other hand, Vanilla Finetuning + I→OR and Vanilla Finetuning + I→RO are trained using the typical greedy search decoding strategy described in §3.1 (Raffel et al., 2020;Vaswani et al., 2017;Narang et al., 2020). This is because, unlike task labels, there do not exist multiple FTR choices (besides gold FTR y * ) to separately teacher-force into the decoder. When using this decoding strategy for classification, it is possible to generate an output sequence that does not match any of the labels in Y . Thus, it is not un- common for such methods to perform much worse than random chance. Additionally, we consider the Vanilla Finetuning (KNIFE Teacher Init.) baseline, where an I→O LM is initialized with the KNIFE teacher LM's finetuned parameters. This baseline is designed to verify that the KNIFE student LM's improvement actually comes the KD training process and not just approximating the teacher LM's parameters. Note that the LM is randomly initialized for all other Vanilla Finetuning methods.
KNIFE: Knowledge Distillation with Free-Text Rationales
We consider three KNIFE variants, each of which uses a different combination of the L KD-in (i.e., KD with task input states) and L KD-out (i.e., KD with task output states) losses ( §3.2). KNIFE Student Finetuning (In) denotes the student LM's performance when trained only with L KD-in . KNIFE Student Finetuning (Out) denotes the student LM's performance when trained only with L KD-out . KNIFE Student Finetuning (In+Out) denotes the student LM's performance when trained with both L KD-in and L KD-out . By default, all KNIFE variants are trained without task loss L task . As an upper bound, we also report the performance of the teacher LM, which has access to FTRs during both training and inference. The teacher LM is denoted in the tables as KNIFE Teacher Finetuning.
KNIFE: Knowledge Distillation with Free-Text Rationales
Following many prior FTR works (Narang et al., 2020;Sun et al., 2022;, we use T5-Base and T5-Large (Raffel et al., 2020) as the LM architecture. By default, KNIFE uses the same architecture for both the teacher and student LMs. In the tables, T5-Base means both the teacher and student LMs use T5-Base, while T5-Large means both the teacher and student LMs use T5-Large. For both cases, the student LM's parameters are initialized as the teacher LM's finetuned parameters.
KNIFE: Knowledge Distillation with Free-Text Rationales
We also consider a T5-Large → T5-Base setting, in which the teacher LM uses T5-Large while the student LM uses T5-Base. In this case, the two LMs have different embedding dimensionalities. Thus, for the task input states and task output states, we train a linear projection layer transforming the student LM's states to have the same dimensionality as the teacher LM's. Also, the student LM uses the teacher LM's language modeling (i.e., decoder output) head.   Fifth, as expected, we find that T5-Large based methods generally outperform their T5-Base counterparts. However, although KNIFE Student Finetuning outperforms Vanilla Finetuning for T5-Large → T5-Base, we find that T5-Large → T5-Base does not outperform T5-Base for KNIFE Student Finetuning. This may be due to the linear projection layer causing information loss when transferring knowledge between the two different architectures. Table 2 displays the results of our low-resource learning (i.e., few training instances) experiments. Here, we consider the OBQA dataset, finetuning on only 10% of the training instances. Also, we focus on using T5-Base as a representative architecture. As expected, all methods in the low-resource setting perform worse than in the fully-supervised setting. However, our conclusions here still largely mirror those in the main results.
KNIFE: Knowledge Distillation with Free-Text Rationales
First, although Vanilla Finetuning + I→O and Vanilla Finetuning (KNIFE Teacher Init.) + I→O still perform competitively among all baselines, Vanilla Finetuning + IR→O is the best performing baseline in the low-resource setting. This is somewhat surprising due to the input distribution shift issue, which negatively impacted Vanilla Finetun-  ing + IR→O's performance in the fully-supervised setting. Since the LM sees fewer training instances overall, it also sees fewer training instances with FTR inputs. As a result, the LM is not as strongly finetuned to expect FTR inputs, which may lessen the severity of the input distribution shift. Second, like in the main results, we find that KNIFE Student Finetuning significantly outperforms all baselines. All three KNIFE variants perform similarly, with KNIFE Student Finetuning (Out) performing best by a relatively slim margin. This provides additional evidence of KNIFE's ability to improve LM generalization.
KNIFE: Knowledge Distillation with Free-Text Rationales
Third, we again observe that the KNIFE Teacher Finetuning upper bound outperforms all other methods. However, in the low-resource setting, the performance gap between KNIFE Teacher Finetuning and other methods is noticeably larger. Since all methods in the low-resource setting have limited labeled training data, having FTRs as context during both training and inference now offers a much greater advantage.
KNIFE: Knowledge Distillation with Free-Text Rationales
To validate the effectiveness of our KNIFE design choices, we consider three different ablation studies. These ablation studies analyze the impact of FTR type, FTR bottleneck, and task loss on KNIFE LM performance.
KNIFE: Knowledge Distillation with Free-Text Rationales
By default, for KNIFE, we use gold FTRs to train the teacher LM. Since gold FTRs are humanannotated to support the gold label, we can assume that gold FTRs contain meaningful knowledge for reasoning about the given downstream task. In this ablation study, our goal is to verify that KNIFE's performance gains truly stem from transferring use-   Table 3 displays the results of this ablation study. In this experiment, we consider various datasets (OBQA, StrategyQA) and architectures (T5-Base, T5-Large, T5-Large → T5-Base). For KNIFE configurations, we consider Student Finetuning (In), Student Finetuning (Out), Student Finetuning (In+Out), and Teacher Finetuning. As expected, we observe that Gold consistently outperforms Replace and Shuffle, often by significant margins. This suggests that FTR type/quality matters a lot and that gold FTRs' knowledge is indeed the source of KNIFE's performance improvements over the Vanilla Finetuning baselines. However, on some settings like OBQA + T5-Large → T5-Base, Gold is not quite as dominant. In particular, for Student Finetuning (Out) on OBQA + T5-Large → T5-Base, both Replace and Shuffle slightly outperform Gold. Like we saw in the main results, this may again be due to the linear projection layer causing information loss when transferring knowledge between the two different architectures, so that FTR quality matter less.
KNIFE: Knowledge Distillation with Free-Text Rationales
Previously, we asserted that the teacher LM's FTR bottleneck is critical for distilling knowledge from the teacher LM's FTR states to its task in-put/output states, through which FTR knowledge is ultimately distilled to the student LM. In this ablation study, we justify this assertion by also considering KNIFE teacher LMs that do not have the FTR bottleneck. Specifically, given a teacher LM with no FTR bottleneck, we report the performance of both the teacher LM and its corresponding student LM. Table 4 shows the results of this ablation study. In this experiment, we consider the OBQA and StrategyQA datasets, the T5-Base and T5-Large architectures, and Student Finetuning (In) as a representative KNIFE configuration. For Teacher Finetuning, there seems to be little correlation between task performance and the use of the FTR bottleneck. This shows that the FTR bottleneck is effective at distilling knowledge from the teacher LM's FTR states to its task input/output states, so that the teacher LM can still leverage FTR context for solving the task even when the FTR states are masked out. For Student Finetuning (In), we observe that having the FTR bottleneck always improves task performance, especially for StrategyQA. This supports our hypothesis that, since the teacher LM's task input/output states are well-infused with FTR knowledge, it is also effective to distill knowledge from the teacher LM's task input/output states to the student LM's.
KNIFE: Knowledge Distillation with Free-Text Rationales
By default, KNIFE trains the student LM with only the KD losses, omitting the task loss. In this ablation study, we justify our choice of learning objectives by evaluating the student LM when it is trained on both the KD losses and the task loss.  Consequently, when trained on both the KD losses and task loss, the student LM may get confused and learn a more suboptimal "in-between" reasoning process that leads to even worse generalization.
Contrastive Decoding: Open-ended Text Generation as Optimization
Open-ended text generation aims to craft fluent and coherent textual continuations of given prompts, laying foundations for various downstream applications such as writing assistance and story generation (Brown et al., 2020). The canonical approaches often sample from large pre-trained language models (Holtzman et al., 2020;Fan et al., 2018;Radford et al., 2019), but the generated text is prone to incoherence and topic drift as unlucky sampling choices compound over long sequences (Eikema and Aziz, 2020;Maynez et al., 2020). On the other hand, searching for the most likely se-1 Code is available at https://github.com/ XiangLi1999/ContrastiveDecoding.git Figure 1: Contrastive decoding exploits the contrasts between expert and amateur LM of different sizes by choosing tokens that maximize their log-likelihood difference. CD produces high-quality text that amplifies the good expert behavior and diminishes the undesired amateur behavior. quences often results in short, repetitive and tedious text (Holtzman et al., 2020), indicating that maximizing probability is a wrong decoding objective.
Contrastive Decoding: Open-ended Text Generation as Optimization
We propose a new search-based approach, contrastive decoding (CD), that can generate fluent and lexically diverse text without compromising coherence. As shown in Figure 1, contrastive decoding takes an off-the-shelf large language model such as OPT-13B (that we call the expert) and an off-the-shelf smaller language model such as OPT-125M (that we call the amateur). CD searches for text that maximizes the difference between expert log-probabilities and amateur log-probabilities, subject to plausibility constraints which restrict the search space to tokens with sufficiently high probability under the expert LM.
Contrastive Decoding: Open-ended Text Generation as Optimization
Contrastive Decoding works because many failure modes of language models (short, repetitive, irrelevant or uninteresting strings) are more common under smaller LMs than under larger LMs. Such outputs are further deemphasized by taking the difference between model log-probabilities. Conversely, stronger models tend to put more probability mass on desirable outputs, such as those with factual knowledge that has not been learnt by the weaker model, and these strings are emphasized by contrastive decoding.
Contrastive Decoding: Open-ended Text Generation as Optimization
Taking Figure 1 as an example, the expert model places significant probability mass on previous tokens such as "Hawaii" and "Honolulu", leading to a highly repetitive continuation from greedy search; and nonsensical tokens such as "Washington" may be sampled, leading to an incoherent continuation. A correct continuation "1961" is strongly preferred by contrastive decoding, despite only having a probability of 0.1, and the continuation includes more correct facts. This example suggests that contrastive decoding generates outputs that emphasize the best of the expert LM and remove its amateur tendencies. Moreover, we provide a pragmatic interpretation of contrastive decoding in §4.
Contrastive Decoding: Open-ended Text Generation as Optimization
Compared to recent training-based methods that improve generation quality such as unlikelihood training (Welleck et al., 2020) and contrastive learning (Su et al., 2022;An et al., 2022), contrastive decoding requires zero additional training. We find that by simply contrasting two frozen language models of different sizes, we are able to decode higher quality text than from the larger LM alone. Furthermore, we find that better performance is achieved when the scale difference between expert and amateur is larger ( §7.1). As a result, the optimal amateur model is also cheap to run and incurs very little inference time overhead.
Contrastive Decoding: Open-ended Text Generation as Optimization
We evaluate our contrastive decoding approach for open-ended text generation in three domains: Wikipedia, stories, and news, and we evaluate using different teacher-student combinations, including (GPT2-XL v.s. GPT2-small, OPT-13B v.s. OPT-125M). Compared to four decoding baselines (nucleus sampling, top-k, typical decoding and SimCTG) our contrastive decoding method significantly improves the coherence of generated text, and improves or maintains the same fluency levels, according to both human evaluation and automatic metrics.
Contrastive Decoding: Open-ended Text Generation as Optimization
We consider decoding approaches for open-ended language generation, where the language models receive an input prompt and aim to generate a fluent and coherent continuation. Specifically, we consider a relatively short prompt of length n, denoted as x pre = x 1 · · · x n , where x i is a token in the vocabulary V. The decoder must generate continuations of length m, denoted as x cont = x n+1 , · · · , x n+m .
Contrastive Decoding: Open-ended Text Generation as Optimization
We generate text from a pre-trained autoregressive language model p LM . At decoding time, we iteratively decode one token at a time by conditioning on the preceding context:
Contrastive Decoding: Open-ended Text Generation as Optimization
where p LM (x i | x <i ) is the next token distribution. We use different subscripts to denote different LMs: p AMA is the amateur LM (e.g., GPT-2 small), and p EXP is the expert LM (e.g., GPT-2 XL).
Contrastive Decoding: Open-ended Text Generation as Optimization
One canonical decoding approach is to sample from a truncated next token distribution at each time step. For example, nucleus sampling (Holtzman et al., 2020) draws from the top p percentile of the next token distribution; top-k sampling (Fan et al., 2018) draws from the top k candidates in the next token distribution. Another common approach is to search for the most likely text sequence via greedy decoding or beam search (Wu et al., 2016); but this leads to repetition and tedious outputs.
Contrastive Decoding: Open-ended Text Generation as Optimization
We propose contrastive decoding as a search-based decoding method that optimizes a novel contrastive objective subject to our plausibility constraint. We first provide intuition and define the constrastive objective ( §3.1). Second, we discuss the potential weakness of this objective alone, and introduce the plausibility constraint to correct for the weakness ( §3.2). Then we define the full contrastive decoding method as our contrastive objective subject to the plausibility constraint ( §3.3). Finally, we elaborate on the design spaces by discussing the choices of amateurs ( §3.4).
Contrastive Decoding: Open-ended Text Generation as Optimization
Smaller LMs demonstrate stronger tendencies to produce undesirable patterns (e.g., repetition, topic drift, and self contradiction) than larger LMs. For example, when both expert (larger LM) and amateur (smaller LM) assign highest probability to a repetitive token, the expert LM is often less confident about this decision and assigns non-trivial probability mass to other good, non-repetitive continuations. Contrastive decoding is inspired by these observations. The goal is to factor out undesired behaviors highlighted by the smaller amateur LMs, and generate text from the remaining good behaviors of larger expert LMs.
Contrastive Decoding: Open-ended Text Generation as Optimization
To operationalize this intuition, we propose the contrastive objective L CD (x cont , x pre ):
Contrastive Decoding: Open-ended Text Generation as Optimization
The CD objective rewards text patterns favored by the large expert LMs and penalizes patterns favored by the small amateur LMs. However, amateur LMs are not always mistaken: small language models still capture many simple aspects of English grammar and common sense (e.g., subject verb agreement). Thus, penalizing all behaviors from amateur LMs indiscriminately would penalize these simple aspects that are correct (False negative), and conversely reward implausible tokens (False positive). To tackle this issue, we introduce the plausibility constraint, which complements our CD objective and avoids these failure modes.
Contrastive Decoding: Open-ended Text Generation as Optimization
To tackle the aforementioned issue, we propose an adaptive plausibility constraint (V head ) that exploits the confidence level of the expert LM to restrict the effect of the contrastive objective when the expert LM is highly confident:
Contrastive Decoding: Open-ended Text Generation as Optimization
Here, α is a hyperparameter in [0, 1] that truncates the next token distribution of p EXP . Larger α entails more aggressive truncation, keeping only high probability tokens, whereas smaller α allows tokens of lower probabilities to be generated. We set α = 0.1 throughout the paper. This adaptive plausibility constraint corrects for both false positive and false negative failures of the contrastive objective:
Contrastive Decoding: Open-ended Text Generation as Optimization
False positives. An implausible token may be rewarded with a high score under our unconstrained contrastive objective. For example, the token "Net-Message" is highly implausible under the context of Figure 1, with 3 × 10 −9 of p EXP and 8 × 10 −14 of p AMA ; however, it attains the highest contrast of log p EXP − log p AMA = 10.6, which is much higher than plausible tokens "1961" and "Hawaii". To handle the false positive problem, V head filters out low probability tokens and only keeps high probability tokens in the candidate pool.
Contrastive Decoding: Open-ended Text Generation as Optimization
False negatives. When confronting an easy decision, the correct token that achieves high probability under both amateur LM and expert LM may receive a low score under the contrastive objective. For example, due to tokenization, the word "unicorn" consists of two subwords: "unic" and "#orn", and the probability of "#orn" given the prefix "unic" is close to 0.99 under both LMs, but the contrast log p EXP −log p AMA is only 6×10 −4 , which is much lower than bad continuations.
Contrastive Decoding: Open-ended Text Generation as Optimization
Here, V head uses the expert LM's confidence (as defined by the α ratio with the max probability token in the given timestep) to avoid these false negative cases. The expert LM assigns high confidence to easy decisions, but not to tokens that reflect the undesired behaviors of the amateur, since probability mass is taken up by other candidate tokens the expert is able to consider. Our constraint keeps as few as one token in the candidate pool when the expert is highly confident about this token, which removes the impact of the contrastive objective, because the single token would always be highest ranked regardless of the CD objective.
Contrastive Decoding: Open-ended Text Generation as Optimization
Combining the contrastive objective and the adaptive plausibility constraint, we obtain the full contrastive decoding formulation:
Contrastive Decoding: Open-ended Text Generation as Optimization
The above objective is defined at the sequence level, which is intractable to optimize. Thus, we factor the objective to token level scores:
Contrastive Decoding: Open-ended Text Generation as Optimization
We apply beam search to optimize CD-score, by first filtering tokens based on plausibility constraints V head (x <i ), eliminating tokens that fail to achieve sufficiently high probabilities under the expert LM. Then we score the remaining tokens based on the amount of contrast they demonstrate, according to log p EXP (x i | x <i ) − log p AMA (x i | x <i ). As a result, we end up selecting plausible tokens under the expert LM that least resemble the amateur LM.
Contrastive Decoding: Open-ended Text Generation as Optimization
The choice of amateur LM is an important decision for contrastive decoding. As discussed in §3.1, we should choose amateur LMs that exhibit the behaviors we would like to downweight from the expert LM. Here, we consider three aspects:
Contrastive Decoding: Open-ended Text Generation as Optimization
Scale. Smaller LMs have lower modeling capacity and are more prone to errors. Therefore, we choose the amateur LM to be the smallest model in the same family of the expert LM. For example, for OPT-13B expert, we choose OPT-125M as the amateur; for GPT-2 XL expert, we choose GPT-2 small as the amateur. We verify this design choice in §7.1. On the extreme end, employing n-gram models yields an amateur LM of extremely low capacity. But this choice hurts generation quality, because n-gram LMs incur too many errors to identify similar failure modes of the expert LM.
Contrastive Decoding: Open-ended Text Generation as Optimization
Temperature. We can manipulate the amateur LM behavior by tuning its temperature τ . For example, applying a high temperature (τ > 1) to the amateur LM results in flatter distributions; applying a low temperature (τ close to 0) highlights the mode of the amateur distribution, which is more prone to errors (e.g. repetition). Therefore, we manipulate the temperature of the amateur LM to adjust the amateur behavior that will be penalized in contrastive decoding. In §7.2, we study the impact of τ to generation quality and set τ to 0.5 or 1.0 for our main experiments.
Contrastive Decoding: Open-ended Text Generation as Optimization
Context window. We can also weaken capacity by restricting the context window of the amateur LM (Li et al., 2016). For instance, we can only allow the amateur LM to condition on the last token of x pre , but we allow the expert LM to condition on the entire x pre . In other words, we decode from log pEXP(x cont |x 1:n )
Contrastive Decoding: Open-ended Text Generation as Optimization
pAMA(x cont |xn) . By conditioning the amateur LM only on partial prompts, the coherence of the amateur LM is weakened, and contrastive decoding produces more coherent text by highlighting the coherence nature of the expert LM. In §7.5, we study the impact of this design choice.
Contrastive Decoding: Open-ended Text Generation as Optimization
Having formally described contrastive decoding, we now provide a pragmatic interpretation, justifying its validity through pragmatic communication goals .
Contrastive Decoding: Open-ended Text Generation as Optimization
A line of work in pragmatics (Grice, 1975) characterizes communication as a cooperative process between speakers and listeners. Several of these formalisms (Horn, 1984;Levinson, 2000) describe a tradeoff between speakers and listeners, where a speaker should generally produce language that is high quality (e.g. truthful, fluent, and relevant) while also being informative to a listener.
Contrastive Decoding: Open-ended Text Generation as Optimization
Our contrastive objective can be motivated by this tradeoff, with our expert and amateur LMs modeling a knowledgable speaker and a lessinformed listener: (1) Upweighting tokens by p EXP and using our expert-based plausibility constraints generates tokens that have high probability under the expert LM, encouraging generated text to be fluent and relevant (e.g. upweighting '1961' in Figure 1). (2) Downweighting tokens by p AMA suppresses language that is predictable by (i.e. less informative to) the amateur LM (e.g. downweighting 'Honolulu' and 'Washington'), and by proxy encourages the language to be informative to a listener in context. By combining these two criteria, our contrastive decoding method produces high quality text that satisfies the communicative goal of transferring relevant but not predictable information.
Contrastive Decoding: Open-ended Text Generation as Optimization
Maximum probability. Setting the amateur LM to a uniform distribution reduces CD to maximize log-probabilities under the expert LM.
Contrastive Decoding: Open-ended Text Generation as Optimization
N-gram blocking. If we set the amateur LM as an n-gram model whose n-gram counts are updated to fit the generated prefix, this yields a decoding algorithm with soft n-gram blocking. If we also set the amateur temperature to be very small, then it approaches the canonical heuristic of forbidding repeated n-grams (Paulus et al., 2018).
Contrastive Decoding: Open-ended Text Generation as Optimization
Diverse decoding. If we use the same LM as both amateur and expert and restrict the context window of the amateur LM ( §3.4), our method is equivalant to the MMI decoding objective (Li et al., 2016) sometimes used in dialog systems, which explicitly maximizes the pointwise mutual information between the x pre and x cont . We use the first 32 words in the passage as the prompt, and decode for 256 tokens for the continuations. We evaluate generated text with both automatic and human evaluation.
Contrastive Decoding: Open-ended Text Generation as Optimization
Diversity. This metrics aggregate n-gram repetition rates: DIV = 4 n=2 |unique n-grams (x cont )| total n-grams (x cont )| . A low diversity score suggests the model suffers from repetition, and a high diversity score means the model generated text is lexically diverse.
Contrastive Decoding: Open-ended Text Generation as Optimization
MAUVE. MAUVE (Pillutla et al., 2021) score (the higher the better) measures the distribution similarity between the set of generated text and the set of gold reference.
Contrastive Decoding: Open-ended Text Generation as Optimization
Coherence. We follow Su et al. (2022) and approximate coherence by cosine similarity between the sentence embeddings of prompt x pre and generated continuation Human Eval. In order to evaluate the quality of the generated text, we consider two critical aspects: fluency and coherence. A fluent piece of text is written in grammatical English and has a natural flow (e.g. excluding unnatural repetition or web formatting). A coherent piece of text should stay on topic with the prompt and avoid unnatural topic drift. We ask Amazon Mechanical Turkers to read two continuations (A and B) of the same prompt, and choose the more fluent/coherent continuation or decide they are similar.
Contrastive Decoding: Open-ended Text Generation as Optimization
We compare contrastive decoding with three sampling methods, each with the recommended hyperparameters: nucleus sampling (p = 0.95), top-k sampling (k = 50), typical decoding (Meister et al., 2022) (τ = 0.95); and two search-based methods: greedy (max prob) decoding that uses log p EXP as the objective, and contrastive search (CS) (Su et al., 2022;Su and Collier, 2022). Among them, nucleus sampling is the standard approach for open-ended text generation whose performance has been verified in various domains (Holtzman et al., 2020;DeLucia et al., 2020), and typical decoding is a recently proposed approach that excels in lexical diversity (Meister et al., 2022). We therefore conduct human evaluation by comparing CD against these two methods.
Contrastive Decoding: Open-ended Text Generation as Optimization
In order to demonstrate that our approach generalizes across various LM families and sizes, we consider GPT-2 XL (1.5B), OPT (6.7B) and OPT (13B) as expert LMs and employ the smallest LM in their respective family as the amateurs: GPT-2 small (100M) and OPT (125M).
Contrastive Decoding: Open-ended Text Generation as Optimization
Recall that contrastive decoding introduces two hyperparameters: α is the parameter to adjust the plausibility threshold, and τ is the temperature of the amateur LM. We always set α = 0.1 for the main results in the paper -we find that this setting is quite robust and generalizes across various domains. For OPT experiments, we set the amateur temperature to 1.0 and for GPT-2 experiments, we set the amateur temperature to 0.5. We use a beam size of 5. We also study the impact of these hyperparameters in the ablation study §7.2, and we find that our method is robust to various hyperparameter values.
Contrastive Decoding: Open-ended Text Generation as Optimization
6 Main Results
Contrastive Decoding: Open-ended Text Generation as Optimization
As shown in Table 1, contrastive decoding outperforms all other decoding baselines in MAUVE score and coherence score (COH) across three different domains (news, Wikipedia, stories) and two model sizes (1.5B, 13B). Contrastive decoding achieves comparable or slightly worse diversity compared to nucleus and typical sampling, but it achieves substantially better diversity than other search based methods.
Contrastive Decoding: Open-ended Text Generation as Optimization
Typical decoding and nucleus sampling produce lexically diverse text by choosing low probability tokens, at the expense of topic drift. For instance, in the story domain we observe the largest diversity gap between contrastive decoding and nucleus sampling (0.83 v.s. 0.94) in the 1.5B model, but we find that the gap shrinks (0.89 v.s. 0.93) as the model size increases to 13 billion, suggesting that our decoding method would continue to improve as expert models continue to scale.
Contrastive Decoding: Open-ended Text Generation as Optimization
CD outperforms all the baselines in coherence scores by a large margin, followed by greedy decoding. Greedy decoding achieves good coherence despite being highly repetitive, because always repeating the same sentence is a degenerate way to circumvent topic drift. We believe our gain in coherence comes from three aspects: (1) CD searches to optimize our objective, avoiding the topic drift that can happen by chance in sampling-based generation techniques.
Contrastive Decoding: Open-ended Text Generation as Optimization
(2) Our contrastive objective implicitly rewards coherence, because large LMs are typically more coherent than smaller LMs. (3) Finally, we restrict the context length of the amateur LM ( §3.4), further encouraging CD to reward text that is connected with the prompt (Li et al., 2016).
Contrastive Decoding: Open-ended Text Generation as Optimization
We conduct human evaluation to compare our contrastive decoding approach against nucleus sampling (the canonical method that scores high under MAUVE) and typical decoding (the winning method for diversity metrics). 3 As shown in Table 2, contrastive decoding generates significantly more coherent text compared to nucleus and typical decoding across three domains and two models: on average across settings, evaluators preferred CD 2.6x more than nucleus sampling and 6.4x more than typical decoding when evaluating coherence. As for fluency, CD is preferred 1.4x more than nucleus sampling and 3.5x more than typical decoding.
Contrastive Decoding: Open-ended Text Generation as Optimization
We include a truncated qualitative example in Table 3. The nucleus sampling output shows a topic drift from a video game to music, and part of the generated text includes the format of an email; moreover, there is a style shift from third person narrative style to first person conversational style. These features match the noisy pre-training distribution of internet data, but are not desirable in the context of this prompt. Contrastive decoding output stays on topic with the prompt and elaborates on various aspects of the game, making it more coherent in both content and style. We include more qualitative examples in the appendix.  Figure 2: Generation quality when applying contrastive decoding to expert and amateur LMs of different scales ( §7.1). We explore the expert-amateur combination within GPT-2 family (OPT family results in the appendix). We find the larger scale gap between the expert and the amateur LMs, the more text quality improves.
Contrastive Decoding: Open-ended Text Generation as Optimization
7 Ablation Studies
Contrastive Decoding: Open-ended Text Generation as Optimization
Recall in §3.4, we provide intuition that choosing smaller LMs as the amateur should improve contrastive decoding results. We empirically verify this in Figure 2. The diagonal entries use the same model as expert and amateur, yielding highly repetitive text (low diversity score), because we cannot exploit any contrast between two identical LMs. The upper triangular entries use an expert LM that is smaller than the amateur LM, and this counter-intuitive setup leads to inferior text quality. The lower triangular entries use an expert LM that is larger than the amateur LM, resulting in higher quality text, as measured by both diversity and MAUVE. In particular, the optimal design is to select the largest LM as the expert and the smallest one as the amateur (lower left corner).
Contrastive Decoding: Open-ended Text Generation as Optimization
Does this trend generalize to extremely low capacity LMs like n-gram models? We find that employing a trigram LM as the amateur produces low quality text with a MAUVE score of only 0.73. Our findings indicate that contrastive decoding benefits most with an amateur LM that can emphasize the failure modes of the expert LM, and the mistakes of a low-capacity n-gram model do not highlight failure modes of an expert LM.
Contrastive Decoding: Open-ended Text Generation as Optimization
Recall in §3.3, we introduced the amateur LM temperature τ as a hyperparameter. We study how sensitive our method is to τ as shown in Figure 3.
Contrastive Decoding: Open-ended Text Generation as Optimization
Large τ brings the amateur distribution closer to the uniform distribution, which makes contrastive    Table 2: Human evaluation results for wikipedia, wikinews, story datasets. We boldfaced the max column and * the numbers that are significantly better than the two other columns combined. Contrastive decoding generates significantly more coherent text compared to nucleus and typical decoding across three domains and two models. CD also generates better or comparably fluent text compared to two baselines.
Contrastive Decoding: Open-ended Text Generation as Optimization
Prompt A version of Sonic the Hedgehog was developed by Ancient and released in 1991 for Sega's 8 -bit consoles, the Master System and Game CD (1.5B) A version of Sonic the Hedgehog was developed by Ancient and released in 1991 for Sega's 8 -bit consoles, the Master System and Game Gear. It featured a new level editor called the Sonic Editor and was released as an expansion pack to Sonic CD in 1992, which was itself a port from the Genesis version. The editor allowed the creation and playtesting of custom levels for Sonic CD, and the expansion was also bundled with Sonic CD. It is also available for the PC as Sonic Editor. The game is considered a spiritual successor to the Genesis game Sonic 3 & Knuckles and the Genesis port Sonic 3D Blast, as it features the same gameplay elements, including the Spin Dash, and also shares many levels and music with those games..    decoding generate repetitive text, as repetition is no longer penalized. Small τ makes the amateur LM more spiky and emphasizes undesired amateur behaviors, leading to better outputs from contrastive decoding. As shown in Figure 3, we find that setting τ in [0.5, 1.5] attains good and robust performance in coherence and fluency.
Contrastive Decoding: Open-ended Text Generation as Optimization
Recall that contrastive decoding is a search-based approach that maximizes the contrastive objective subject to plausibility constraints. We explore a sampling alternative based on the same objective. Specifically, we normalize the CD-score(x i ; x <i ) (defined in §3.3) via softmax into a probability distribution from which we sample the next token.  As shown in Table 4 and Table 5, we find that sampling from this objective produces lower quality text than searching under the objective. According to automatic and human evaluations, CD (sample)'s fluency and coherence rating consistently falls behind CD (search), but sampling still yields reasonably good outputs.
Contrastive Decoding: Open-ended Text Generation as Optimization
In §3.2, we describe why including the feasibility constraints is critical. Here, we conduct an ablation study verifying this claim by removing the plausibility constraints V head . We find that the generation outputs suffers from severe fluency issues, as easily shown by its MAUVE score of 0.01 in the CD(-V head ) row of Table 4.
Contrastive Decoding: Open-ended Text Generation as Optimization
We further experiment with ablating the prompt context on the amateur LM ( §3.4), by letting the expert LM and amateur LM both condition on the entire x pre . Table 5 shows that the ablation slightly hurts coherence and fluency. 
Contrastive Decoding: Open-ended Text Generation as Optimization
We propose contrastive decoding, a search-based decoding approach that contrasts LMs of different scales. We evaluate our approach on open-ended text generation, and find that it improves over the prevalent methods like nucleus sampling in both fluency and coherence. As future work, the idea of contrasting an expert (larger LM) and an amateur (smaller LM) can be expanded to myriad setups, for instance, contrasting an early checkpoint of an LM and a later checkpoint of the LM. We hope that this paper can encourage more exploration of how to use contrasting language models.
Contrastive Decoding: Open-ended Text Generation as Optimization
In this paper, we focus on open-ended text generation and demonstrate the effectiveness of contrastive decoding. We would like contrastive decoding to also work well for task-oriented generation settings such as summarization and machine translation. However, the idea of contrasting models across different scales (larger expert LM and smaller amateur LM) is not directly applicable, because the modes of both amateur LM and expert LM are of high quality. Empirically, having a smaller summaization model (BART-small finetuned on summarization data) as the amateur LM yields lower ROUGE score than employing a uniform distribution as the amateur LM, which is equivalent to beam search based on log-probabilities. As future work, we aim to study the necessary properties of amateur LM to empower task-oriented generation (e.g. summarization, table-to-text). 
Contrastive Decoding: Open-ended Text Generation as Optimization
In order to emprically justify our contrastive objective, we report the likelihood scores and contrastive scores for repetitive text, reference and sampling outputs. As shown in Table 6, we find that reference text scores highest under our contrastive loss objective, whereas the likelihood maximization objective ranks the undesired repetitive text the highest. Averaging across the wikitext data, repetitive text receives a likelihood score of -0.79 per token, reference text receives -3.20, and sampling output receives -2.93. Contrastive objective on the other hand, assigns 0.21 to repetitive text, 0.62 to reference text, and 0.59 to sampling text. This trend is consistent with observation in the Table 6, and contrastive scores correctly assigns highest ranking to reference text.
Contrastive Decoding: Open-ended Text Generation as Optimization
The pre-trained LMs are flawed in both coherence and repetition, and they make similar mistakes regardless of the sizes: for maxprob decoding, the 4-gram repeat rate is 71% for GPT-2 XL, and 40% for GPT-3 Davinci (both are unacceptably high). For sampling, the coherence score is 0.56 for GPT-2 XL and 0.57 for GPT-3 Davinci (both are lower than GPT-2 XL's CD results of 0.69).
Contrastive Decoding: Open-ended Text Generation as Optimization
Recall from §3.3, our objective log pEXP(x cont |xpre) pAMA(x cont |xpre) can intuitively be interpreted as factoring out amateur tendencies from the expert LM. Formally, the argmax x cont of our contrastive objective also maximizes the pointwise mutual information PMI(x cont , I = 1), where I is an indicator variable that determines the source of generated text: I = 1 for text generated by the expert and I = 0 for text generated by the amateur.
Contrastive Decoding: Open-ended Text Generation as Optimization
This leads to a formal interpretation of our objective: it favors text that has high PMI with the indicator variable I = 1, i.e., the most distinguishable text as having originated from the expert LM, rather than the amateur LM.
Contrastive Decoding: Open-ended Text Generation as Optimization
Training Methods. Prior works often aim to improve text generation quality by further training a given LM. A common approach is to fine-tune the LMs on domain specific data, which improves the relevance of generated text, but fails to fundamentally address fluency or coherence problems (DeLucia et al., 2020). To tackle these model specific issues, many works craft novel training objectives. Despite the effectiveness of these approaches, they require training model parameters on these crafted objectives, which can be prohibitively expensive for ever-larger models. In contrast, our method uses frozen LMs and requires no training. We simply take off-the-shelf pre-trained language models of different sizes, and exploit their differences to improve text generation quality.
Contrastive Decoding: Open-ended Text Generation as Optimization
Contrast in Text Generation. The idea of contrast for text generation has been explored in diverse settings. In pun generation, He et al. (2019) contrasts the same LM with global versus local context to select tokens that are plausible globally but surprising locally. In dialog generation, Li et al. (2016) contrasts the same dialog model with and without preceding chat history in order to generate relevant responses. Su et al. (2022) fine-tuned language models on a contrastive training objective to separate token representations, which in turn improves generation diversity. The closest work to ours is DExpert (Liu et al., 2021), which studies controllable text generation by contrasting an trained expert model (on nontoxic data) and a trained anti-expert model (on toxic data) to produce text that is non-toxic. In this work, we focus on open-ended text generation and show that it is possible to get domain-and task-agnostic anti-experts simply by using a smaller LM. Contrastive decoding uses the observation that smaller LMs are more susceptible to the undesirable behaviors, and contrasts off-the-shelf LMs of different scales to produce high quality text, without any training.  Table 6: We report the likelihood scores and contrastive scores for repetitive text, reference and sampling outputs. We find that reference text scores highest under our contrastive loss objective, whereas the likelihood maximization objective ranks the undesired repetitive text the highest.
Contrastive Decoding: Open-ended Text Generation as Optimization
Contrastive decoding aims to produce fluent and coherent continuation of a given prompt. However, as the generation quality improves, one can imagine more powerful disinformation (e.g., automatic generation of fake news) that are hard to distinguish from human written text. Towards this end, it might be worth augmenting current decoding techniques to also watermark the generated outputs without affecting its quality.
Contrastive Decoding: Open-ended Text Generation as Optimization
We use NVIDIA RTX A5000 and A100 GPU to run the decoding experiments. All the decoding is done by one GPU. For OPT-13b, we use fp16 to reduce the required amount of GPU memories. CD generates one continuation of length 256 tokens (with batchsize of 1) in 8 seconds on NVIDIA RTX A5000.
Contrastive Decoding: Open-ended Text Generation as Optimization
We report the instruction given to the Amazon mechanical turkers in Figure 4, and we explain the annotation results will be used towards distinguishing text generation qualities. We conduct a pre-qualification round of 60 people to ensure the participants understand the task and are capable of judging fluency and coherence, resulting in around 20 people qualified.
Contrastive Decoding: Open-ended Text Generation as Optimization
We assign 20 minutes to each HITs, which consists of three comparison tasks. Each HITs takes 14 minutes on average to complete. We pay $4.5 for each HITs, which adds up to an hourly payment of $18, which is adequate given the participants' demographic. Our human evaluation project received approval from the ethics review.
Contrastive Decoding: Open-ended Text Generation as Optimization
In the main paper, we focus in the settings where the experts and the amateurs come from the same model family (e.g., GPT-2 small v.s. GPT-2 XL; OPT-125M v.s. OPT-13B), because the tokenizer is the same within each model family. However, contrastive decoding still works when the expert and amateur models come from different model families. In particular, we use GPT-J as the expert and GPT-2 small as the amateur (the two models are pre-trained on different datasets by different companies, but share the same tokenizer). We find that CD yields MAUVE=0.93, DIV=0.91, which is better than GPT-2 XL's CD results.
Contrastive Decoding: Open-ended Text Generation as Optimization
In Table 1, we report diversity, MAUVE, and COH. In the tables (Table 7 for wikitext, Table 8 for wikinews, Table 9 for story), we also include REP-N metrics for n = 2, 3, 4 and perplexity (PPL) under GTP-2 medium, along with MAUVE, COH and DIV.
Contrastive Decoding: Open-ended Text Generation as Optimization
As shown in Figure 5, we report additional results for the ablation study of amateur temperature. We find that τ ∈ [0.5, 1.0] robustly result in high generation quality. In Figure 6, we provide additional results on the amateur-expert size combinations for the OPT family and GPT-2 family. We find that within the same LM family, the larger scale gap between the expert LM versus the amateur LM, the more text quality improves.
Contrastive Decoding: Open-ended Text Generation as Optimization
Recall in §7.3, we compare sampling CD objective and searching CD objective. Here, we include extra results in Table 10. We find that CD (search) outperform CD (sample) consistently across three domains and three model sizes.
Contrastive Decoding: Open-ended Text Generation as Optimization
We include 6 randomly sampled qualitative examples in Table 12 -17.
Contrastive Decoding: Open-ended Text Generation as Optimization
As we mentioned in §3.4, an ideal amateur LM should summarize the failure mode of the expert LM, and we have been using a off-the-shelf amateur LM in the main text (e.g., GPT-2 small, OPT-125m). Here, we experiment with learning an amateur model that mimics the degenerate behavior of the expert LM. Precisely, we first randomly sample some prompt of different length from wikipedia dataset, and generate training data by beam searching the expert LM conditioned on the prompts. This training data is representative of the degeneration       2). The left plot is based on automatic metrics, and it shows how MAUVE and diversity score change as we vary the τ values, labeled next to each dot. The right table is based on human evaluation, and we report coherence and fluency preference in the same format as Table 2. We find that τ ∈ [0.5, 1.0] robustly result in high generation quality. For main results we use τ = 0.5 for GPT-2 and τ = 1.0 for OPT.   Figure 6: Generation quality when applying contrastive decoding to expert and amateur LMs of different scales ( §7.1). The left two plots explore the expert-amateur combination within GPT-2 family. The right two plots explore size combination in the OPT family. We find that within the same LM family, the larger scale gap between the expert LM versus the amateur LM, the more text quality improves. wikinews wikitext story name div mauve coh div mauve coh div mauve coh  Table 10: Full ablation studies results comparing CD (sample) and CD search §7.3. We find that CD (search) outperform CD (sample) consistently across three domains and three model sizes.
Contrastive Decoding: Open-ended Text Generation as Optimization
The 40 -metre -long ( 131 ft 3 in ) hangar was 20 metres ( 65 ft 7 in ) wide forward and 11 metres CD (1.5b) The 40 -metre -long ( 131 ft 3 in ) hangar was 20 metres ( 65 ft 7 in ) wide forward and 11 metres wide ( 33 ft 6in) aft. The hangar had two doors on the port and starboard sides and a single door in front. The port door opened to a large, open hangar deck which extended forward to about 20 metres ( 65ft 7 in ) in width, then curved around and closed on a small, enclosed platform which had two small, circular, windows in it, each about 10 feet by 6 feet ( 3m x 1.3 m ). The port and forward door opened onto the forward part ( port side) of a hangar which extended to a platform on which was suspended the main wing assembly, a single vertical tailplane and two vertical fin assemblie s, each with two horizontal planes and two horizontal stabilizer planes, all suspended by cables from a central vertical stabilizing strut. The platform had two small windows on it and a small door in it, which cou ld also open. The platform was suspended from a large vertical strut which ran the entire width and which could also swing out to form a small platform for a single plane, or could be swung into position for the en tire assembly to rest upon. The hangar deck extended to the rear, where it curved around to close on the small platform which had the single, small, round, window. This small platform Nucleus (1.5b) The 40 -metre -long ( 131 ft 3 in ) hangar was 20 metres ( 65 ft 7 in ) wide forward and 11 metres ( 32 ft 3 in ) wide aft of the posvailar aircraft Long house tractor OHP (on ramp) for refuelling of aircraft Multi-armed bus for each aircraft Type 60 1012 with full size service ship kitted out to have a toilet on the U-shaped port side Alternating twin Mk-III main guns on the port side could each be fitted with fuzes to fight various targets Interior of two cabin cabins aboard YK-SATF Modern sleeping quarters aboard YK-SATF with internal bunk room, practically designed for two officers and 128 people (51 passengers, 193 space on D-handlers) Room to move and all-round living facilities on U-shaped bar stand on the port side (covering port side) Elevator to the Laundry Arcade on the port side and overhead hoist to the garage with parking (drivers) and service bus Medium weight serial 717-5005 complete with enclosed treadle trailer and winch, bought in BOM-BARDIER'S FORGOT LANDING Modern passenger transporter and ( Grumman HUU-21 ) with new advance Z CD (13b)
Contrastive Decoding: Open-ended Text Generation as Optimization
The 40 -metre -long ( 131 ft 3 in ) hangar was 20 metres ( 65 ft 7 in ) wide forward and 11 metres wide aft ( 34 ft 7 in ). The hangar had two doors on the port and starboard sides and a single door in the centre, which could be closed to form a sealed room. It could accommodate a Boeing B-17F Flying Fortress bomber. The aircraft were parked nose to tail, with the nose of one aircraft abeam of the centreline of the other aircraft, with their wings touching. The aircraft were parked in this fashion to prevent them colliding in flight, which was a serious hazard in the days of wooden aircraft. The hangar doors could be opened in flight, but were normally left closed. The hangar had an internal height of 8 metres ( 26 ft ) and an internal volume of 5,500 cubic feet ( 152 m 3 ). Nucleus (13b)
Contrastive Decoding: Open-ended Text Generation as Optimization
The 40 -metre -long ( 131 ft 3 in ) hangar was 20 metres ( 65 ft 7 in ) wide forward and 11 metres ( 36 ft 10 in ). At the rear of each side of the hangar, there was a covered gangway, for walking up and down. This went from the winch room on the starboard side and descended over the steel deck until reaching the 2 x 10 sheet metal Whitworth Type 328's head. The starboard side of the hangar swung 90 degrees, towards the bridge, and ended with a 10 metre ( 33 ft ) long steel skid flat at the rear which neatly incorporated the lockers for the G-Turn seats. Total hangar deck length was 183 metres ( 603 ft ). Above the single 40m x 10m hangar, the hanger deck's Foxtrot and RMS sections wereEach section had a length of 5.0 metres ( 16 ft 3 in ) a height of 2.6 metres ( 8 foot ) and a width of 2.7 metres ( 7 ft 2 in ). The engine room was A 40m x 10m hangar built in 1937 had a longer engine hatch to both forward ends of the deep narrow profile of the FOxtrot section, so that when doors were closed supplemented the long afterbonnet doors.The crawlway from FOxtrot to RMS halved the length of the engine room.The Table 11: Qualitative Example 1.
Contrastive Decoding: Open-ended Text Generation as Optimization
In some images, Hu employed a blind embossing technique ( known as " embossed designs " ( gonghua, 拱花 CD (1.5b)
Contrastive Decoding: Open-ended Text Generation as Optimization
In some images, Hu employed a blind embossing technique ( known as " embossed designs " ( gonghua, 拱花) ), in others he employed an ink brush ( gonghuai, 拱花), in yet other images he employed both techniques , or used neither technique. The technique used to emboss Hu's characters was not known to me. The images below were all created by Hu in the late Qing Dynasty, around 1900-1900 C.E.. The first three are all from a book titled " The Book on the Great Learning ", and were all created around the year 1900-1905. The last image, which was made sometime around the early 1920's is from an unknown work. The book in which these images are contained was published sometime around 1920-1920. The images are all in the Hu style, which was a style of calligraphic Chinese script used from around the mid 19 th Century until around 1900. Hu was the most famous and prolific of calligraphy painters of this style. The style is characterized by a strong and distinct character set and a strong, distinct line-width, and was used to write many famous Chinese poems and poems by famous Chinese poets such as Li Bai ( 杨瑜), and Zhuangzi ( Zhūngzì, 老子 Nucleus (1.5b) In some images, Hu employed a blind embossing technique ( known as " embossed designs " ( gonghua, 拱花); In which images are embossed by straight red lines on the front and back of the paper, and the paper is then pasted to the frame. There are another method in which the paper is pinned down or carried over the back of the paper to the base of the image. Chinese calligraphy artists often used this " embossed poster, ", a style developed by my great-grandfather Tang  In some images, Hu employed a blind embossing technique ( known as " embossed designs " ( gonghua, 拱花), where a design was embossed onto a surface and then covered by another material, usually paper or si lk, which would reveal the design when the covering material was peeled away. In other images, Hu employed an ink wash technique ( guancha, 档拱花), which involved painting designs with ink onto silk, which was then folded over and pressed to reveal the ink designs when the folded silk was opened. In both techniques, Hu's designs often incorporated Chinese characters, flowers, birds, or other images, often in a floral arr angement or other geometric pattern. The designs were often repeated in different colors or in different arrangements to create an overall pattern or composition. The embossed and ink wash paintings were then often mounted together on a silk cloth or paper to create a scroll painting or scroll painting panel ( jiaochaoren, 街上篆), or were framed to make a wall scroll ( xuanhua, 序印), which were often given as gifts o r sold to raise funds for Hu's travels or other expenses. Hu often signed the scrolls he produced, but his signatures are often difficult or impossible to read, as they have been rubbed away over Nucleus (13b) In some images, Hu employed a blind embossing technique ( known as " embossed designs " ( gonghua, 拱花, 覆盖恢, 2008, Scale-packaging Checkpoint (Luntai)), which enables the smoothing out of its distinct design features. The result becomes a multi-level one of its staggering packaging. Meanwhile, in order to turn the expected carrier type into a visual one, he designed a unique structural design in order to have multi hiding spaces for its series of Brown Helpful Products( Thanks-Sweet Brown) wit h innovative distribution of compressed air between them. The manufacturer was originally originally known as this address: Of course some users are: enter you read how to play or watch this game, please feel free to ask, please, tell some friends did you find the site in google search. Thank You. Por favor, leas cualquier otro tema. Gracias, recio los comentarios. You can leave a comment. Thank You. Leave a comment. CONTACTS less surprisingly, cutting produced a very good contrast between its high quality and its price. To refer to three of his early works: Bao Bao, Nanji Yaem, Ganzi Schneider. A Lexicon of Military Land Art. Globorum Logos: avant-garde  Sonic CD in 1992, which was itself a port from the Genesis version. The editor allowed the creation and playtesting of custom levels for Sonic CD, and the expansion was also bundled with Sonic CD. It is also available for the PC as Sonic Editor. The game is considered a spiritual successor to the Genesis game Sonic 3 & Knuckles and the Genesis port Sonic 3D Blast, as it features the same gameplay elements, including the Spin Dash, and also shares many levels and music with those games, though it was developed by a completely new team and has no connection with either of them, and was not intended as an official sequel to either game. It also shares some level design with Sonic Adventure and Sonic Adventure DX, but not Sonic CD or Sonic CD Special Delivery, as those were not developed by Ancient at all, but by a separate developer called Team Sonic, and are considered separate titles in the franchise, with their own storylines, gameplay and characters, though the two share some level designs and some characters, A version of Sonic the Hedgehog was developed by Ancient and released in 1991 for Sega's 8 -bit consoles, the Master System and Game Gear. While the game struggled to sell and was poorly received, it represented an experiment in Sonic game capability by requiring that the game's code be 1,000 times faster as a response to Sega's attempts to catch up to Nintendo. This early version shares only a few very minor features with the official Sonic the Hedgehog game released three and a half years later for the Super Nintendo Entertainment System by Sonic Team, notably features from Sonic Spinball (working as a gravity suit for the player and a non-invincible roll ability)are retained. Though it is possible that the data from this very early build was also used in a test build of Sonic's follow-up entries to Sonic Gems: Team Sonic Racing 2, a point that has since been supported by Victor Coelho, one of the creators of the ROM, in the Sonic Mega Collection Plus special discs. Unfortunately, and uniquely compared to its very similarly named paleo relative of 1991 in Mobile Suit Gundam: MSI,both 2D and its computer version are far less distinctly different than the 2D Game Gear version even while being different to and games. There were also Sonic propositions for a Game Boy Advance port due to which Sega developed a prototype for it using Game Studio, an in-house game- CD (1.5b) Writing for The A.V. Club, television reviewer Hayden Childs commented that " Exit Through the Kwik -E -Mart " is "the best episode of the show yet." I agree with Childs. This episode is a masterpiece of the kind of television that The Simpsons is so often lauded as having. The story, written by showrunner Mike Scully and starring the always excellent John Di Mambro as the voiceover narrator of "Exit Through The Kwik-E -Mart," has the feel and feel-ness of the classic "Bart to the Future," but with the wit of a modern Simpsons. The story, in its first two acts, follows Bart's attempts to find the perfect gift for a girl he's crushing on, while his parents are away at the hospital for the birth of Bart's sister Lisa (the episode was written before Lisa was conceived, so Bart is not aware that Lisa's father, Homer Simpson, will one day marry Marge Simpson and become Homer Simpson, and thus the father of Lisa and the future Homer). The third and fourth acts are devoted entirely to the birth and early childhood years of Lisa, as Bart is left behind at home to raise her while Homer is off working for the nuclear plant, leaving Bart with the task of looking after her while he goes out and gets drunk and/or high and/or whatever else happens when you have Nucleus (1.5b) Writing for The A.V. Club, television reviewer Hayden Childs commented that " Exit Through the Kwik -E -Mart " is one of " the fucking worst shows ever made on television," which makes sense since it made no sense at all. Why get all obsessive about specific comic book characters when you can "relate" with people from any fictional realm? Always good to know. The one-shot from Grant Morrison and Darick Robertson sets up the Missing Link story of the issue, a side narrative that follows a resistance mission into a parallel universe and leads to a spooky-as-hell finale. Next week: The Walking Dead So why would Morrison want to direct a feature film, or at least in such a genre? Well, he's already turned his attention to television and presumably want to turn his hand to the tiny screen again. That's where this comes in-the TV series he's being given the job to write and direct is The Walking Dead, the $1.2 billion-selling zombie apocalypse franchise on AMC that's probably the most viewed zombie phenomenon on the planet (have you seen the true facts? Those ratings really aren't close) The show is about a kid from the first wave of zombies, a few years after the zombie apocalypse has come and gone. He doesn't know anything about zombies. He just knows that you can't leave the woods around a campground CD (13b)
Contrastive Decoding: Open-ended Text Generation as Optimization
Writing for The A.V. Club, television reviewer Hayden Childs commented that " Exit Through the Kwik -E -Mart " is "an entertaining, if not particularly insightful documentary about graffiti artist Kaws, whose work has become increasingly commercial and mainstream." He added that "Kawasaki is a likable enough subject, and the film does manage to capture some interesting footage of Kaws at work in his studio and around New York. But the documentary never quite manages to explain what exactly it's trying to say about art or commerce or the nature of fame." He concluded by writing that "Exit Through the Kwik -E -Mart is a decent enough documentary about an intriguing figure. But it's also an example of how, even with a talented subject and some decent access, the best documentaries still can't quite escape their limitations as entertainment.  Prompt DreamWorks contracted Amazon.com to create the official website, marking the first time that Amazon had created a special section devoted to a feature film. The website CD (1.5b) DreamWorks contracted Amazon.com to create the official website, marking the first time that Amazon had created a special section devoted to a feature film. The website features the official trailers, cast and credits of all five DreamWorks features. The website is also the first time Amazon has provided the official website of any feature film to the public, as they had previously only released the film's trailers and posters to their employees and select press. The official site is available in both a standard version, which is the one you are currently reading, or an "Enhanced" version that has additional content and is optimized to load quickly and load the most information possible on a mobile screen. The enhanced site also includes a "DreamWorks Home Page," a section that provides a more personalized experience and offers a "DreamWorks Home Screen." The site will be available to all Amazon Prime customers, but is currently available to Prime customers in the US, the United Kingdom and Germany, with the site's availability expanding to the remaining markets later. The official website for The Croods is currently unavailable in Australia. The site is expected to return to Australia later in 2012. The site is currently unavailable in Brazil, China and the Netherlands, while it will be added in these countries in the coming days, but not all features are yet available. The site is also unavailable for users in India. The website for the animated feature Trolls was also launched today. The site features Nucleus (1.5b) DreamWorks contracted Amazon.com to create the official website, marking the first time that Amazon had created a special section devoted to a feature film. The website is also available on AOL Instant Messenger f or voice-enabled users. The company announced it was doing the project in July but didn't say what kind of publicity it had gathered. Participating film-makers included students at Amazon's "digital artist lab," which has worked with companies such as Dell, Redwood City, Calif.-based TV maker Vizio and Daniel Crown, a Dominican immigrant who made his name in the 1980s as the lead singer of the Demolition Dog. DreamWorks already had a large, dedicated online fans site for the film, which stars the popular actors Jason Bateman, Owen Wilson, Christian Bale, and Michelle Pfeiffer, as well as more recent comic-book favorites Emily Blunt and Paul Rudd. The home page had a map of Manhattan with other New Yorkers who would be familiar to the film-viewing public. From various corners of the Internet The site is not quite as polished as an official website for an old movie. The 1920s-era design isn't very elegant and Microsoft's Internet Explorer plugin doesn't load smoothly. For example, a rudimentary search function is available, but not very sophisticated. Google is also a huge presence, as users can consult news about the project on Google News CD (13b) DreamWorks contracted Amazon.com to create the official website, marking the first time that Amazon had created a special section devoted to a feature film. The website featured exclusive interviews and footage, a blog by Dreamworks' Jeffrey Katzenberg and Meg LeFauve and a section where visitors could create a personalized character. The film's website received a nomination for Best Website at the 2008 Streamys. Release Marketing DreamWorks began its campaign for Shrek Forever After on May 7 with a teaser website, featuring the tag line "The adventure continues", and a countdown clock to May 17, when the teaser trailer would premiere on ABC's Jimmy Kimmel Live!. The trailer debuted during the show and was also made available online. The official trailer premiered on the same date. The film's first poster was revealed in July, featuring the four maincharacters in silhouette. A second poster, featuring the four characters in full color and the tag line "The adventure never ends" debuted at San Diego Comic-Con. A series of character posters was also revealed. The first of which featured Fiona, Donkey and Puss in Boots in silhouette and was revealed on August 1. A second character poster featuring Shrek, Princess Fiona and Donkey debuted in August, followed by posters featuring Shreks son, Farquad and the villainous Lord Farquaad in September, and the four characters together in October. The first television spot Nucleus (13b) DreamWorks contracted Amazon.com to create the official website, marking the first time that Amazon had created a special section devoted to a feature film. The website previously served as store and promotional hub for all DreamWorks films. According to ThinkFilm, the website is sort of like a cross between Backstage and IMDbPro. Users can even take a short quiz and get a free preview of the film. Members of the website are also privy to additional content and promotions. Other sites frequented by the celebs during their promotional tour were Artsy.net, wellcome.org, Mashable and amfam.com. http://www.ingorepub.com/ Insight-it's who you know by Emily Kreps (If Magazine) Wal-Mart is watching you. Could that even be an exaggeration? Wal-Mart, the world's largest retailer, is moving quickly to fill the information void, in providing a signal its general awareness of what goes on from brick and mortar location to location. The chain has equipped what's essentially online GPS monitoring data of nearly every metric found in traditional retail, and is adding capabilities on the fly as it knows what's measurable-but still not. While Wal-Mart isn't tapping into cook's sales in the chilled salad department, the ability to 12306 in the expert LM, and tends to be highly repetitive. We then prefix-tune (Li and Liang, 2021) a GPT-2 model on this training data to obtain the final amateur LM. Here, we use prefix-tuning as the lightweight adaptation method which only requires learning and storing a soft prompt of length 10. At decoding time, we just use the prefix-tuned model as the amateur, and apply contrastive decoding in §3.3. We denote this variant of CD as beamprefix and report automatic evaluation results in Table 7, Table 8, and Table 9. We also include human evaluation results, which compares the beamprefix variant of CD with nucleus sampling results. As shown in Table 18, we find that CD (beamprefix) also attain significantly better performance than nucleus sampling.  Table 18: Human evaluation results for wikipedia, wikinews, story datasets. We describe the details of CD (beamprefix) in Appendix M.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
Large language models (LLMs) have demonstrated strong performance on a wide range of reasoning tasks (Brown et al., 2020;Hoffmann et al., 2022;Chowdhery et al., 2022, inter alia).
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
One particularly interesting strategy is chain-ofthought (CoT) prompting, which has been shown to elicit strong reasoning abilities in LLMs by asking the model to include intermediate reasoning * Equal contribution; Figure 1: As opposed to the chain of thought reasoning (CoT) that enables LLM to generate a series of natural language-based intermediate steps to solve a reasoning task, the DECOMPOSITIONAL DISTILLATION approach asks a question at each intermediate step corresponding to the various CoT reasoning steps. Here, Q refers to the input problem, and A denoted the LLMs reasoning processes after prompting. steps while solving a problem (Wei et al., 2022b;. However, the CoT prompting has only been made possible due to the scaling of these models to hundreds of billions of parameters, making it one of the emergent properties of LLMs (Wei et al., 2022a).
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
For this reason, significant compute resources or expensive API calls are required for CoT reasoning. Therefore, we deem it desirable to transfer such reasoning capabilities into smaller models. One way to attempt this would be to induce CoTlike reasoning abilities in smaller models by knowledge distillation, i.e., training small models with the CoT outputs from the larger model. However, i . The generated subquestions-solutions are used to train two student models: a) the Problem Decomposer that learns to mimic the subproblem generator of the LLM and b) the Problem Solver, which learns to solve each subquestion. At the bottom, the inference process is depicted for an unseen problem and no LLM is present. The Problem Decomposer breaks the unseen problem into simpler subquestions and the Problem Solver solves each one of them eventually leading to the final answer a (ni) i . small-size language models are known to be poor reasoners (Stolfo et al., 2022), and the CoT reasoning abilities cannot be transferred directly to the smaller models. In our work, we justify that training smaller models 1 directly with CoT outputs to implicitly learn to generate a sequence of reasoning steps to answer a given problem is a suboptimal strategy (details in Section 5).
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
In this paper, we introduce DECOMPOSITIONAL DISTILLATION, a general framework that takes advantage of the reasoning abilities of LLMs and transfers these capabilities into smaller models. Our methodology is motivated by the Knowledge distillation framework (Hinton et al., 2015), where a larger teacher model transfers knowledge to a smaller student model.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
In contrast to standard knowledge distillation, DECOMPOSITIONAL DISTILLATION distils the teacher model's reasoning abilities using its generated solutions as a proxy. Our approach consists of two steps (illustrated by the two upper boxes in Figure 2): first, we prompt an LLM with examples of problems decomposed in intermediate subquestion-solution pairs. This way, the model, using the same question-answer structure, generates the intermediate steps to obtain the solution for a new problem. (We report an example of our question-answer decomposition and a comparison to a traditional CoT prompt in Figure 1.) Second, we transfer the decomposed intermediate reasoning into a student structured as a pair of smaller models: a problem decomposer and a problem solver. The former acts as a decomposition unit that learns to break down a complex reasoning problem into smaller subproblems, while the latter solves the intermediate subproblems produced by the decomposition unit leading to a solution of the overall problem.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
To demonstrate the effectiveness of our approach, we consider a multistep math word problems dataset, GSM8K (Cobbe et al., 2021) due to its popularity and presence of step-wise reasoning solution steps. Our distilled models outperform our baseline models of the same size that were trained to reproduce the standard chain of thought reasoning without decomposed sub-questions by over 35%. While the results provide a proof of concept of our proposed methodology, it is essential to note that the approach is not specifically tailored to the given domain and could, in the future, be applied to other reasoning tasks.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
In our work, we use a math word problem (MWP) dataset. Solving MWPs has been an interesting area of research for the last couple of years Hosseini et al., 2014;Roy et al., 2015;Seo et al., 2015;Sachan and Xing, 2017;Amini et al., 2019;Xie and Sun, 2019;Zhang et al., 2020). However, the majority of the modern approaches are shifting towards the use of LLMs, often relying on approaches involving prompting or incontext learning (Cobbe et al., 2021;Kojima et al., 2022a;Wei et al., 2022b;Chowdhery et al., 2022;Lewkowycz et al., 2022;Srivastava et al., 2022). The most popular approach is the chain of thought prompting (Wei et al., 2022b), which prompts the language model to generate a series of natural language-based intermediate steps that improve the reasoning capabilities in LLMs.  took another step forward and sampled multiple reasoning paths and selected the most relevant output using majority voting. Huang et al. (2022) used the most voted outputs to further fine-tune the model for better performance. Kojima et al. (2022b) further improved the reasoning of LLM in a zero-shot manner by appending "Let's think step by step" to the prompt. Our work, however, not only trains the model rather than just prompting but also explicitly guides the reasoning path using sub-questions at each step which is missing in all the above reasoning chains. One work that is close to our work is by Zhou et al. (2022) which decomposes the questions into multiple sub-questions and asks the language model to solve each subquestion sequentially. However, this work is also restricted to prompting and only works with LLMs with billions of parameters and cannot be directly transferred to smaller models.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
We use a student network that imitates the teacher and this goes in the direction of knowledge distillation (Ba and Caruana, 2014;Hinton et al., 2015). Snell et al. (2022) demonstrated the usefulness of providing instruction that can assist the models in achieving better reasoning abilities. Similar to our hypothesis, Eisenstein et al. (2022) argued that questionanswering systems should not only focus on the final answer but also rationales that justify their reasoning to help reason better. We go beyond this in our work and work on not just the questionanswering system but also on what questions need to be asked at each step that can assist in learning that reasoning step better. Finally, similar to our hypothesis of injecting reasoning capabilities in smaller models, Li et al. (2022) used CoT-like reasoning from LLMs to train smaller models on a joint task of generating the solution and explaining the solution generated. We, on the other hand, use the LLM to generate subquestions and solution pairs and use them together to inject reasoning capabilities into smaller models.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
Motivation from Learning Sciences Explicit guidance through sub-questioning at each step helps in better reasoning and has a strong basis in learning science literature (Wood et al., 1976;Fantuzzo et al., 1989;Wood, 1994). We see our proposed learning strategy DECOMPOSITIONAL DISTILLATION as a form of collaboration script. A collaboration script is a set of instructions which aims to guide and support two or more learners to interact and behave during collaborative learning in a way that all learning partners benefit from collaboration. In our case, an LLM (acting as a teacher) decomposes the complex problem into simpler subquestion-solutions, and two smaller models act as learners and learn to mimic the reasoning steps and the guiding questions while benefiting from each other's learning.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
We describe our DECOMPOSITIONAL DISTILLA-TION approach in the setting of math word problems (MWPs). We consider a dataset D in which each problem P i is paired with a solution S i that contains a numerical answer a i . The task of solving the problem using a model f consists of predicting a solutionâ = f (P ) such thatâ = a. In this section, we describe the two steps that constitute our distillation method and the inference procedure.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
For a small subset of D, we break down each prob-
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
We refer to the ordered list of subquestion-solution pairs for an example as (q 
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
By combining this intermediate annotation, we construct a few-shot prompt (example provided in the Appendix Section A).
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
For each remaining problem P ∈ D that was not manually decomposed, we then prompt a large language model M to generate the intermediate reasoning steps for the remaining problems in D in the form of subquestion-solution pairs (process illustrated by the top left box in Figure 2).
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
We make sure that the chain of intermediate question-solution pairs is meaningful by checking whether the last solution matches with the groundtruth numerical answer, i.e., whether a
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
If that is not the case, we discard the problem and re-sample a new chain by prompting the model again (for a maximum of 3 times). This way, we obtain an augmented dataset D * in which a portion of the problems is paired with a sequence of subquestion-solution pairs leading to the correct result.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
We break the student models into two separate models: a problem decomposer that is trained to produce meaningful subquestions that can assist in reaching the correct solution, and a problem solver, that solves the problem in a step-by-step manner given the subquestions produced by the problem decomposer.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
Problem Decomposer Using the problems in D * that contain the augmented chain of intermediate questions and solutions, we train a problem decomposer model M P D that learns to produce the sequence of subquestions {q (1) , q (2) , . . . } needed to the solve a given problem. We use a pre-trained transformer-based model (Vaswani et al., 2017), and train it on the given sequence of subquestions. Given a subquestion q (j) of problem P , consisting of a sequence of m j tokens {x
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
}, we use a typical autoregressive language modelling loss, L:
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
where IP PD (x|c) is the probability assigned by M PD to token x given context c, and x :(y) indicates the sequence {x (1) , . . . , x (y) }. The learning objective for problem P is then modelled as:
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
However, training the problem decomposer directly on the sequence of subproblems leads to worse performance due to no control over the generated sequence of subquestions (more details in the Ablation Section 6).
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
We propose a guiding mechanism that constrains the question-generation process to follow a strict set of constraints.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
Following Shridhar et al. (2022), we consider a second sequence-to-sequence model that is trained to generate the equations eq (j) , later used as a guiding mechanism for the sequence and the number of questions to be generated. The equations are a taskspecific example (for the MWP case) of a guiding mechanism for M P D , and they can be possibly replaced with any set of constraints for other tasks.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
The equation generation follows the same objective defined in Equation 1 with questions replaced with the equations from the solutions at each step. The generated equations 2 are later used to guide the M P D , where the generated equations are appended to the encoder to be conditioned on, and the updated learning objective is defined as:
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
where ⊕ depicts the concatenation operation and eq = [eq (1) , eq (2) , . . . ] is the list of the equations leading to the intermediate numerical results a (j) 's.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
The problem solver component consists of a pre-trained language model M PS , trained independently of the problem decomposer, using the teacher-generated chain of subquestionsolution pairs. Similarly to the PD model, the learning objective of the problem solver is computed at a token level for each intermediate solution:
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
where l j and the y j 's represent, respectively, the length and the tokens of the intermediate solution s (j) . Then, the objective is combined for all the substeps of problem P:
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
L PS (P ) = n i j=1 L(P, s (j) ).
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
During training, the previous intermediate solutions generated by the problem decomposer are replaced with the teacher-generated solutions using teacher forcing (Cho et al., 2014). However, the solutions generated by the problem solver are used at inference time.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
When predicting an unseen problem P , the problem decomposer is queried to produce the sequence of subquestions {q (1) , . . . ,q (n) } meant to guide the reasoning of the problem solver model. After these questions are generated, they are shown to the problem solver iteratively, decoding the intermediate solutionŝ (j) at step j token by token according to the model's probability distribution over its vocabulary:
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
where y (k) j is the k-th token being decoded in greedy fashion.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
After the last solutionŝ (n) has been generated, the numerical predictionâ (n) is parsed from the text using simple heuristics.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
We study how DECOMPOSITIONAL DISTILLA-TION can assist smaller models to learn better reasoning on a multi-hop math word problems dataset, GSM8K 3 (Cobbe et al., 2021). GSM8K consists of 8.5K grade school math word problems with each problem requiring 2 to 8 reasoning steps to solve. The solutions primarily involve a sequence of elementary calculations using basic arithmetic operations (+ − * /). The dataset is partitioned into 7.5K training problems and 1K test problems. Since the GSM8K dataset has problem solutions in a step-by-step format leading to the final answer, we used the ground truth as a proof of concept for our experiments. However, for other datasets with just the final answer, the step-by-step decomposition will be provided by the teacher model.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
We used GPT-2 variants (Radford et al., 2019) as the main student question-answering model in our work. GPT-3 (Brown et al., 2020) served as the teacher model for decomposing the complex problem into a series of simpler substeps. We used T5-large (Raffel et al., 2020) as the backbone of our question generator module with the main problem as input and sub-questions generated as output.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
All models were trained using the Huggingface library (Wolf et al., 2020) on an NVIDIA Tesla A100 GPU with 40GB of GPU memory. Each experiment ran for the same number of iterations to ensure fairness with a periodic evaluation over the validation set. Teacher forcing was used during training to replace the generated answers with true answers from the training dataset.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
For the evaluation of the question-answering performance on the GSM8K dataset, we compute the accuracy based on the final answer provided by the problem solver. For the question generation part, we report automatic evaluation using SacreBLEU (Post, 2018) which is based on exact word overlap, BERT F1 score (Zhang et al., 2019) which is based on DeBERTa (He et al., 2020) as the similarity model. We also report #Q, the number of questions generated compared to the number of reasoning steps needed to solve a problem.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
Does DECOMPOSITIONAL DISTILLATION help in achieving better reasoning performance? The GSM8K dataset consists of a Socratic version where subquestion-solution pairs are provided for each MWP. We assumed this as a substitution for a teacher model as this is the best result that a teacher model can achieve (we interpret this as an upper bound). We also experimented with generating these subquestions using a GPT-3 model with prompts, obtaining similar results (BERT F 1 score of 95%). Since the end goal is to induce the reasoning capabilities into smaller models, we use CoT reasoning as our baseline. We used these sub-questions to train variants of the GPT-2 model (small, medium, and large) and compared our results with the CoT baseline. Table 1 demonstrates the effectiveness of our method with all models achieving higher accuracy when trained with our proposed methodology DECOMPOSITIONAL DIS-TILLATION (up to 38%). The approach favours the larger model as larger models have a better capacity to encode the reasoning capabilities in their parameters. We also demonstrate that training with DECOMPOSITIONAL DISTILLATION can help a much smaller model (GPT-2 large with 774M parameters) reason equally well to an almost 10X larger model (GPT-3 with 6B parameters).
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
Methodology Accuracy GPT-2 Small  Our approach can also be extended to promptingbased models, where DECOMPOSITIONAL DISTIL-LATION can improve the performance of GPT-3 with 1-shot prompting by over 40% (Table 2). Here, LLM like GPT-3 is used to decompose the main problem into simpler problems and then GPT-3 is used to solve it in a 1-shot setting with decomposed sub-questions and solutions as prompts.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
Methodology Accuracy GPT-3 (1-shot)
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
CoT 27.5 (175B) Sub-ques 47.1 (↑ 41%) Can we model the sub-questioning capabilities into a smaller model? Using a large language model like GPT-3 can be an obvious choice for decomposing a complex problem into step-by-step reasoning steps. However, relying on a large model at test time is undesirable (e.g., for resource constraints). Therefore, we study whether a smaller model can learn to decompose a problem into subquestions required to guide the reasoning process. This can again be thought of as a knowledge distillation between a teacher and student model with the major difference being this time the student learns to ask a series of suitable questions needed to solve the problem.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
We use a T5 model that learns to ask a series of questions needed to answer the problem in a chain-of-thought style. We calculate the BLEU and BERT F 1 scores alongside the match in the number of questions generated (#Q) between the generated sub-questions and the ground truth (for other datasets where we don't have access to subquestions, this will be generated by a prompt-based model like GPT-3) on the GSM8K dataset. Directly fine-tuning a T5 model without any guidance towards what kind of questions needs to be generated was not effective in our case as the model was generating sometimes more, sometimes fewer questions that were confusing the question answering model . This led to a sharp performance drop (more discussion in the ablation studies in section 6). We used a guidance-based strategy where we used the equations from the ground truth to assist the models in generating the right number of questions and the performance increased on all three metrics -BLEU, BERT F 1 , and the number of questions. The results are reported in Table 3. Note that here we additionally train another model for generating equations at test time to assist the problem decomposer to condition the sub-questions from it.  Finally, we compare the reasoning capabilities of the models when a smaller problem decomposer is used instead of ground truth (or even prompt-based models like GPT-3) in Table 4. Even with a much smaller question generator model (200X smaller than GPT-3), DECOMPOSITIONAL DISTILLATION achieved better reasoning performance than the CoT approach for all different GPT-2 models.  
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
Impact of non-guided question generator on question answering performance. When a problem decomposer is not guided (with equations, in our case) to know how many questions to generate, the model asks a lot of redundant questions that sometimes confuse the model, leading to a decrease in accuracy. Figure 3 shows a comparison between the non-guided problem decomposer effects on the reasoning accuracy compared to the equation-guided model. It is worth noting that using subquestions that are not ordered or well structured can harm the reasoning capabilities of the model, just like students in the real world.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
Eliminating the need for a problem decomposer model. We studied the impact of the problem decomposer by removing it completely from the training and testing process. We trained a student to answer sub-questions based on the intermediate steps directly meaning depending on what step question is asked, the problem solver learns to answer that step. If sub-question q (1) is asked, the model learns to answer s (1) , and when q (2) is asked, the model learns to build the steps {s (1) , s (2) } and not just s (2) . This means that during the test time, a multistep question can range from 2 to 8 steps and the model builds the needed steps to reach the final answer a. This phenomenon can be thought of as curriculum learning (Bengio et al., 2009) where simple to more complex concepts are taught to the model and the learning happens gradually. However, in our case, the model did not learn to map solution steps iteratively; leading to a much worse performance during test time. The results demonstrating the need for the problem decomposer are summarised in Table 5.  No sub-ques sub-ques QG Small 2.70 5.98 Medium 7.20 11.57 Large 8.18 17.89 Table 5: Accuracy comparison (in %) of using no student questioning module (No sub-ques) vs using one (Sub-ques QG) on the GSM8K dataset.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
Chain of thought-style step-by-step reasoning methods has proven to be very effective for reasoning in LLMs. In this work, we propose ways to further improve it by explicitly asking questions at each step, thus making it possible to train smaller models with this approach. We demonstrate the effectiveness of our proposed methodology on a popular multi-step math word problem dataset. However, we have used the provided ground truth solution in our experiments, which is very specific to the GSM8K dataset. In many real-world scenarios, such intermediate solutions are not available, and in such cases, we rely on LLMs to gen-erate those steps leading to additional noise in the training data. Recent works like STaR (Zelikman et al., 2022) can prove very useful for such cases, where if the generated rationale leading to the final answer is incorrect, the rationale is regenerated given the correct answer (ground truth). This idea can also be applied to the student models where the problem decomposer and the problem solver are trained jointly to improve each other's performance.
Distilling Multi-Step Reasoning Capabilites of Large Language Models into Smaller Models via Semantic Decompositions
Finally, we only use one solution from the LLM to distil information in the student model and following  multiple subquestionsolution pairs can be sampled and using majority voting, all the correct pairs can be used to distil knowledge in the student models. We leave this for future work.