How are explanations used to improve model performance for NLP tasks?
2.3 Usage of Explanations for SOTA models
Existing works have been exploring circumstances
in which explanations could improve model per-
formance; for example, Hase and Bansal (2021)
argues that explanations are most suitable for use
as model input for predicting, and Kumar and
Talukdar (2020) proposed a system to generate
label-specific explanations for the NLI task specif-
ically. Some recent works have tried to generate
better explanations with a self-rationalization set-
ting (Wiegreffe et al., 2020; Marasovi ́c et al., 2021),
where a model is asked to generate the prediction
label and explanation simultaneously. We conduct
a preliminary experiment to find the best model
setting to leverage explanations in Section 4.1.
There exists many recent works (Paranjape et al.,
2021; Liu et al., 2021; Chen et al., 2022) that ex-
plore the usage of prompts to complete explana-
tions, generate additional information for the orig-
inal task, or examine whether generated explana-
tions can provide robustness to adversarial attacks.
Ye and Durrett (2022) showed that simply plugging
explanations into a prompt does not always boost
the in-context learning performance, and model-
generated explanations can be unreliable for few-
shot learning. Another related line of research fo-
cuses on extracting or generating explanations with
a unified framework (Chan et al., 2022b) or with a teachable reasoning system that generates chains of reasoning (Dalvi et al., 2022).
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Many methods now exist for conditioning models on task instructions and user-provided explanations for individual data points. These methods show great promise for improving task performance of language models beyond what can be achieved by learning from individual (x,y) pairs. In this paper, we (1) provide a formal framework for characterizing approaches to learning from explanation data, and (2) we propose a synthetic task for studying how models learn from explanation data. In the first direction, we give graphical models for the available modeling approaches, in which explanation data can be used as model inputs, as targets, or as a prior. In the second direction, we introduce a carefully designed synthetic task with several properties making it useful for studying a model’s ability to learn from explanation data. Each data point in this binary classification task is accompanied by a string that is essentially an answer to the why question: “why does data point x have label y?” We aim to encourage research into this area by identifying key considerations for the modeling problem and providing an empirical testbed for theories of how models can best learn from explanation data.
[{"first":"Peter","last":"Hase","middle":[]},{"first":"Mohit","last":"Bansal","middle":[]}]
Can Rationalization Improve Robustness?
A growing line of work has investigated the development of neural NLP models that can produce rationales–subsets of input that can explain their model predictions. In this paper, we ask whether such rationale models can provide robustness to adversarial attacks in addition to their interpretable nature. Since these models need to first generate rationales (“rationalizer”) before making predictions (“predictor”), they have the potential to ignore noise or adversarially added text by simply masking it out of the generated rationale. To this end, we systematically generate various types of ‘AddText’ attacks for both token and sentence-level rationalization tasks and perform an extensive empirical evaluation of state-of-the-art rationale models across five different tasks. Our experiments reveal that the rationale models promise to improve robustness over AddText attacks while they struggle in certain scenarios–when the rationalizer is sensitive to position bias or lexical choices of attack text. Further, leveraging human rationale as supervision does not always translate to better performance. Our study is a first step towards exploring the interplay between interpretability and robustness in the rationalize-then-predict framework.
[{"first":"Howard","last":"Chen","middle":[]},{"first":"Jacqueline","last":"He","middle":[]},{"first":"Karthik","last":"Narasimhan","middle":[]},{"first":"Danqi","last":"Chen","middle":[]}]
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Does prompting a large language model (LLM) like GPT-3 with explanations improve in-context learning? We study this question on two NLP tasks that involve reasoning over text, namely question answering and natural language inference. We test the performance of four LLMs on three textual reasoning datasets using prompts that include explanations in multiple different styles. For these tasks, we find that including explanations in the prompts for OPT, GPT-3 (davinci), and InstructGPT (text-davinci-001) only yields small to moderate accuracy improvements over standard few-show learning. However, text-davinci-002 is able to benefit more substantially. We further show that explanations generated by the LLMs may not entail the models' predictions nor be factually grounded in the input, even on simple tasks with extractive explanations. However, these flawed explanations can still be useful as a way to verify LLMs' predictions post-hoc. Through analysis in our three settings, we show that explanations judged by humans to be good--logically consistent with the input and the prediction--more likely cooccur with accurate predictions. Following these observations, we train calibrators using automatically extracted scores that assess the reliability of explanations, allowing us to improve performance post-hoc across all of our datasets.
[{"first":"Xi","last":"Ye","middle":[]},{"first":"Greg","last":"Durrett","middle":[]}]
Measuring Association Between Labels and Free-Text Rationales
In interpretable NLP, we require faithful rationales that reflect the model’s decision-making process for an explained instance. While prior work focuses on extractive rationales (a subset of the input words), we investigate their less-studied counterpart: free-text natural language rationales. We demonstrate that *pipelines*, models for faithful rationalization on information-extraction style tasks, do not work as well on “reasoning” tasks requiring free-text rationales. We turn to models that *jointly* predict and rationalize, a class of widely used high-performance models for free-text rationalization. We investigate the extent to which the labels and rationales predicted by these models are associated, a necessary property of faithful explanation. Via two tests, *robustness equivalence* and *feature importance agreement*, we find that state-of-the-art T5-based joint models exhibit desirable properties for explaining commonsense question-answering and natural language inference, indicating their potential for producing faithful free-text rationales.
[{"first":"Sarah","last":"Wiegreffe","middle":[]},{"first":"Ana","last":"Marasović","middle":[]},{"first":"Noah A.","last":"Smith","middle":[]}]
Towards Teachable Reasoning Systems
Our goal is a teachable reasoning system for question-answering (QA), where a user can interact with faithful answer explanations, and correct errors so that the system improves over time. Our approach is three-fold: First, generated chains of reasoning show how answers are implied by the system’s own internal beliefs . Second, users can interact with the explanations to identify erroneous model beliefs and provide corrections. Third, we augment the model with a dynamic memory of such corrections. Retrievals from memory are used as additional context for QA, to help avoid previous mistakes in similar new situations - a novel type of memory-based continuous learning. To our knowledge, this is the ﬁrst system to generate chains that are both faithful (the answer follows from the reasoning) and truthful (the chain reﬂects the system’s own beliefs, as ascertained by self-querying). In evaluation, users judge that a majority (65%+) of generated chains clearly show how an answer follows from a set of facts - substantially better than a high-performance baseline. We also ﬁnd that using simulated feedback, our system (called EntailmentWriter) continually improves with time, requiring feedback on only 25% of training examples to reach within 1% of the upper-bound (feedback on all examples). We observe a similar trend with real users. This suggests new opportunities for using language models in an interactive setting where users can inspect, debug, correct, and improve a system’s performance over time.
[{"first":"Bhavana","last":"Dalvi","middle":[]},{"first":"Oyvind","last":"Tafjord","middle":[]},{"first":"Peter","last":"Clark","middle":[]}]
Generated Knowledge Prompting for Commonsense Reasoning
It remains an open question whether incorporating external knowledge benefits commonsense reasoning while maintaining the flexibility of pretrained sequence models. To investigate this question, we develop generated knowledge prompting, which consists of generating knowledge from a language model, then providing the knowledge as additional input when answering a question. Our method does not require task-specific supervision for knowledge integration, or access to a structured knowledge base, yet it improves performance of large-scale, state-of-the-art models on four commonsense reasoning tasks, achieving state-of-the-art results on numerical commonsense (NumerSense), general commonsense (CommonsenseQA 2.0), and scientific commonsense (QASC) benchmarks. Generated knowledge prompting highlights large-scale language models as flexible sources of external knowledge for improving commonsense reasoning. Our code is available at https://github.com/liujch1998/GKP
[{"first":"Jiacheng","last":"Liu","middle":[]},{"first":"Alisa","last":"Liu","middle":[]},{"first":"Ximing","last":"Lu","middle":[]},{"first":"Sean","last":"Welleck","middle":[]},{"first":"Peter","last":"West","middle":[]},{"first":"Ronan","last":"Bras","middle":["Le"]},{"first":"Yejin","last":"Choi","middle":[]},{"first":"Hannaneh","last":"Hajishirzi","middle":[]}]
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Many commonsense reasoning NLP tasks involve choosing between one or more possible answers to a question or prompt based on knowledge that is often implicit. Large pretrained language models (PLMs) can achieve near-human performance on such tasks, while providing little human-interpretable evidence of the underlying reasoning they use. In this work, we show how to use these same models to generate such evidence: inspired by the contrastive nature of human explanations, we use PLMs to complete explanation prompts which contrast alternatives according to the key attribute(s) required to justify the correct answer (for example, peanuts are usually salty while raisins are sweet). Conditioning model decisions on these explanations improves performance on two commonsense reasoning benchmarks, as compared to previous non-contrastive alternatives. These explanations are also judged by humans to be more relevant for solving the task, and facilitate a novel method to evaluate explanation faithfulfness.
[{"first":"Bhargavi","last":"Paranjape","middle":[]},{"first":"Julian","last":"Michael","middle":[]},{"first":"Marjan","last":"Ghazvininejad","middle":[]},{"first":"Luke","last":"Zettlemoyer","middle":[]},{"first":"Hannaneh","last":"Hajishirzi","middle":[]}]
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
An extractive rationale explains a language model’s (LM’s) prediction on a given task instance by highlighting the text inputs that most influenced the prediction. Ideally, rationale extraction should be faithful (reflective of LM’s actual behavior) and plausible (convincing to humans), without compromising the LM’s (i.e., task model’s) task performance. Although attribution algorithms and select-predict pipelines are commonly used in rationale extraction, they both rely on certain heuristics that hinder them from satisfying all three desiderata. In light of this, we propose UNIREX, a flexible learning framework which generalizes rationale extractor optimization as follows: (1) specify architecture for a learned rationale extractor; (2) select explainability objectives (i.e., faithfulness and plausibility criteria); and (3) jointly the train task model and rationale extractor on the task using selected objectives. UNIREX enables replacing prior works’ heuristic design choices with a generic learned rationale extractor in (1) and optimizing it for all three desiderata in (2)-(3). To facilitate comparison between methods w.r.t. multiple desiderata, we introduce the Normalized Relative Gain (NRG) metric. Across five English text classification datasets, our best UNIREX configuration outperforms the strongest baselines by an average of 32.9% NRG. Plus, we find that UNIREX-trained rationale extractors’ faithfulness can even generalize to unseen datasets and tasks.
[{"first":"Aaron","last":"Chan","middle":[]},{"first":"Maziar","last":"Sanjabi","middle":[]},{"first":"Lambert","last":"Mathias","middle":[]},{"first":"Liang","last":"Tan","middle":[]},{"first":"Shaoliang","last":"Nie","middle":[]},{"first":"Xiaochang","last":"Peng","middle":[]},{"first":"Xiang","last":"Ren","middle":[]},{"first":"Hamed","last":"Firooz","middle":[]}]
NILE : Natural Language Inference with Faithful Natural Language Explanations
The recent growth in the popularity and success of deep learning models on NLP classification tasks has accompanied the need for generating some form of natural language explanation of the predicted labels. Such generated natural language (NL) explanations are expected to be faithful, i.e., they should correlate well with the model’s internal decision making. In this work, we focus on the task of natural language inference (NLI) and address the following question: can we build NLI systems which produce labels with high accuracy, while also generating faithful explanations of its decisions? We propose Natural-language Inference over Label-specific Explanations (NILE), a novel NLI method which utilizes auto-generated label-specific NL explanations to produce labels along with its faithful explanation. We demonstrate NILE’s effectiveness over previously reported methods through automated and human evaluation of the produced labels and explanations. Our evaluation of NILE also supports the claim that accurate systems capable of providing testable explanations of their decisions can be designed. We discuss the faithfulness of NILE’s explanations in terms of sensitivity of the decisions to the corresponding explanations. We argue that explicit evaluation of faithfulness, in addition to label and explanation accuracy, is an important step in evaluating model’s explanations. Further, we demonstrate that task-specific probes are necessary to establish such sensitivity.
[{"first":"Sawan","last":"Kumar","middle":[]},{"first":"Partha","last":"Talukdar","middle":[]}]
Few-Shot Self-Rationalization with Natural Language Prompts
Self-rationalization models that predict task labels and generate free-text elaborations for their predictions could enable more intuitive interaction with NLP systems. These models are, however, currently trained with a large amount of human-written free-text explanations for each task which hinders their broader usage. We propose to study a more realistic setting of self-rationalization using few training examples. We present FEB -- a standardized collection of four existing English-language datasets and associated metrics. We identify the right prompting approach by extensively exploring natural language prompts on FEB. Then, by using this prompt and scaling the model size, we demonstrate that making progress on few-shot self-rationalization is possible. We show there is still ample room for improvement in this task: the average plausibility of generated explanations assessed by human annotators is at most 51% (with GPT-3), while plausibility of human explanations is 76%. We hope that FEB and our proposed approach will spur the community to take on the few-shot self-rationalization challenge.
[{"first":"Ana","last":"Marasovi'c","middle":[]},{"first":"Iz","last":"Beltagy","middle":[]},{"first":"Doug","last":"Downey","middle":[]},{"first":"Matthew","last":"Peters","middle":["E."]}]
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
A long line of past work has sought to use freetext explanations, rationales, and other similar data to improve machine learning models. Proposed methods use explanations to constrain or regularize the learned model (Zaidan et al., 2007;Small et al., 2011;Ba et al., 2015;Zhang et al., 2016;Srivastava et al., 2017;Liang et al., 2020), to automatically label data for data augmentation (Hancock et al., 2018;Wang et al., 2019a;Awasthi et al., 2020), as additional supervision (Narang et al., 2020; Hase Here, x is an input that one might expect a model to produce the correct output for after fitting to (x, y) pairs. For some models, x may be sufficient, while others may benefit from additional information provided by e. Pruthi et al., 2021) or intermediate structured variables (Camburu et al., 2018;Rajani et al., 2019;Wiegreffe et al., 2020), and simply as model inputs (Rupprecht et al., 2018;Co-Reyes et al., 2019;Zhou et al., 2020).
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
However, there are many tasks in NLP where improvements in performance prove elusive even when using thousands of explanations as additional data (Narang et al., 2020;Hase et al., 2020). A few observations could explain this situation: (1) the modeling space has not been fully explored for these tasks, but improvements are possible; (2) pretrained language models already store the knowledge that the explanations would have provided, so they do not need them; (3) the language models do not need any information that is not already learnable from the task's input-output pairs. We do not yet know which explanation is best, and therefore it would be helpful to more deeply understand the motivations behind existing modeling approaches.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
In this paper, we (1) present a formal framework for characterizing approaches to learning from explanation data, and (2) we propose a synthetic task for studying how models learn from natural language data. Specifically, we first present graphical models for various approaches where explanation data is used either as model inputs, targets, or priors, and we characterize existing methods according to these graphical models. Then, based on past results, we suggest which models might be most appropriate for explanation data. Next, we present a synthetic task which shares important properties with NLP tasks involving explanation data. Constructing this task helps us carefully specify the manner in which we expect explanations to be useful to models. We provide simple experimental verification that the task is solvable by existing Transformer models when using explanations as additional data but very difficult to solve without them. Our aim is to outline promising approaches in the area and contribute a concrete test bed to assist others in developing new models for learning from natural language explanations.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
In what follows, we discuss our framework for modeling with explanations and relevant work (Sec. 2.1), as well as promising approaches for learning from explanations (Sec. 2.2).
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
What Is an Explanation? We use the term "explanation" to refer to the data one might collect if asking a person to answer the question, "Why does data point x have label y?" This is a formulation of the explanation as an answer to a why-question of the kind discussed in Miller (2019). Rather than try to give a formal definition of the kind of data generated from this question, we proceed with some illustrative examples, shown in Fig. 1.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
In this section, we lay out our theory of how explanations may be used in modeling a task, in a standard supervised learning setup for obtaining a MAP estimate of model parameters:
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
where Y is a set of labels for inputs X. We refer to the role of Y in this probabilistic model as the target, X as an input, and p(θ) as a prior. Below we describe existing approaches to adding explanations into this framework. An overview of the corresponding graphical models is shown in Fig. 2  2: Graphical models for several approaches to using explanations as targets, as inputs, and as priors. Typically past works do not condition on human-given explanations at test time, unless they are designed to not leak the data point label.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
as Multi-Task in Fig. 2). For instance, Pruthi et al. (2021) consider using attention weight explanations (from a model) as targets in a multitask framework, and they observe accuracy improvements in what is essentially model distillation. Meanwhile, natural language explanations appear as targets in a multi-task framework, using datasets with explanations for each data point (Camburu et al., 2018;Narang et al., 2020;Hase et al., 2020;Wiegreffe et al., 2020). None of these works find improvements in task performance from incorporating explanations. It is perhaps even concerning that a model could learn to generate coherent "explanations" without the learning of this ability influencing the models that are found for the task.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Using Explanations as Inputs. Additional inputs may be valuable for solving some tasks. One family of approaches uses explanations as model inputs for each data point (Per Data Point Input in Fig. 2 ExpBERT (Murty et al., 2020), a model conditions on vector representations of an input x and a single "global" set of explanations in order to make each prediction (Global Set in Fig. 2). This approach may not scale well to large numbers of explanations, however. Zhou et al.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
(2020) treat explanations as latent variables, and at inference time they retrieve explanations from the training data (Retrieval in Fig. 2). A number of works condition on explanations generated at test time using generative models learned with human explanations as supervision, which are represented as Structured Variable and Per-Label Structured Variable in Fig. 2 (Camburu et al., 2018;Rajani et al., 2019;Kumar and Talukdar, 2020;Hase et al., 2020;Wiegreffe et al., 2020;Zhao and Vydiswaran, 2021). While such structured variables could be useful in principle, these methods have not produced sustained improvements in model accuracy.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Lastly, large language models have recently opened the door for using explanations in few-shot in-context learning (Brown et al., 2020). We represent this approach as Few-shot In-context Learning in Fig. 2. We do not draw the dependencies between distinct data points in the context that would be implied by the attention graph of Transformers, but instead represent the dependence of each data point on the unknown task τ , which models evidently do inference over at test time. Initial work in this direction suggests that models of a sufficiently large size (280B parameters) can learn from explanations provided in a few-shot in-context learning setting (Lampinen et al., 2022).
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Using Explanations as Priors. We group together approaches to defining a distribution over model parameters, including those conditioning on data, p(θ|data). This is a prior over model weights not in the sense that the distribution is independent of data (which it is not), but rather that the posterior parameters are conditioned on the prior. Explanations have been used to constrain the learned model (Srivastava et al., 2017(Srivastava et al., , 2018 or to place priors over how features are weighted or extracted (Zaidan et al., 2007;Small et al., 2011;Zhang et al., 2016;Ross et al., 2017;Bao et al., 2018;Selvaraju et al., 2019;Liang et al., 2020;Stammer et al., 2020;Pruthi et al., 2021;Stacey et al., 2022). Other works map directly from text to model parameters (Ba et al., 2015;Andreas et al., 2018). These meth-ods are all effectively described by Regularizer or Hypernetwork in Fig. 2. Lastly, a few approaches learn to use explanations for automatically labeling data for data augmentation purposes (Hancock et al., 2018;Wang et al., 2019b;Awasthi et al., 2020), which is effectively fitting to data from a prior distribution given by the labeling mechanism (Data Augmentation in Fig. 2).
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Based on our review of existing approaches, we make a few key observations that we believe will assist in the design of future techniques: The upshot of these results is that the most promising approaches for learning from explanation data are likely those treating explanations as inputs (in a manner that does not require new explanations at test time). However, we recommend that other graphical models not be ruled out completely, in case there are promising methods in those families that have yet to be explored.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Following recent work using synthetic data to investigate sequence modeling questions (Liu et al., 2021;Lovering et al., 2021), we design a synthetic dataset so that we can carefully control several important data properties. In Fig. 3, we show an example data point and description of how it gets its label. The premise of our task is to classify sequences by counting different integers in them. Core Idea Behind Data. We wish to design a task where, for a data point (x, y), an explanation Analogous Components to Real Data Index An easily computable feature connecting the input to its explanation
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
A feature indicating what information from the explanation is relevant for the input's label An explanation that says why the input received its label, when understood properly
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Description: The sequence has label because there are more s than s. The index maps to , and indicator says to count rather than . If there were more s than s, the label would be . There is a one-to-one map between index values and tuples.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Count whether there are more of integer a than integer b e communicates information about why input x receives label y. The premise of the task is that a binary label for a sequence of integers x is determined by whether there are more of an integer a in the sequence than there are of an integer b. We refer to integers (a, b) that need to be counted as the label reason. This label reason forms the basis of the explanation for each data point, and it is always exactly specified by the first two integers in x, which we term the index and indicator. For every data point x, there is an explanation e = (index, m, n, r, d) where the label reason is given by either (m, n) or (r, d). Whether the label reason is the (m, n) integer pair or the (r, d) pair is dictated by the indicator. As represented in Fig. 3, (a, b) = (m, n) if the indicator is 1 and (a, b) = (r, d) if the indicator is 2. We call the data e an explanation because it is a direct encoding of a natural language explanation for the data (x, y).
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
For the data point in Fig. 3, this natural language explanation is "input x receives label 1 because it contains more 80's than 40's, and we do not need to count 17's or 27's for this sequence." Proposed Dataset. We describe the proposed dataset using some default data parameters for preliminary experiments, but any specific numbers appearing below are easily adjusted. See Supplement D for the full generative process. 1. Train set: 5000 sequences of 20 integers (including index and indicator), each accompanied by an explanation. There are 500 unique values of index in the dataset drawn from unif (1, 10000), so there are 10 points for each index, whose values of m, n, r, and d are drawn from unif (1, 100) while requiring that m =n =r =d. The corresponding 10 values of indicator are split between 1 and 2. Half of the points have label y=1, i.e. either #m>#n or #r>#d, depending on which feature is causal.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
In each x i , after m, n, r, and d have been randomly placed into the sequence, unfilled slots are filled with samples from unif (1, 100). 2. Dev set: 10,000 points, none appearing in Train, with the same 500 index values, and twice the number of points per index as Train. 3. Test set: 50,000 points of similar construction to the Dev set, but with five times the points per index as Train. Analogous Properties to Human-Curated Data. We claim that aspects of our synthetic task are analogous to properties that natural language data might take on, which we represent in Fig. 3. First, e is an explanation in the sense that, when understood properly, it is a plausible answer to the question: "why does point x have label y?" The explanation describes the feature that causes the label, i.e. the integers that should be counted. We suggest that the index in a sequence is analogous to the topic of some text or the things it refers to: it is an easily computable feature that connects the input to the appropriate explanation. Meanwhile, the indicator is a feature that tells how information from an explanation is relevant to deciding the label. Similarly, an explanation might only be understood in the context of the input it explains.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
We include experiments below that (1) show explanation data is helpful for solving our task and (2) demonstrate why the task is hard without explanation data. We make use of a retrieval-based model similar to Zhou et al. (2020), which learns to retrieve explanations from the training dataset to help with prediction at test time (details in Appendix B and C). This model is composed of a RoBERTabase classifier (Liu et al., 2019) and a SentenceR-oBERTa model used for retrieval (Reimers and Gurevych, 2019). The baseline in our experiments is the RoBERTa classifer on its own.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Design. Using our default dataset containing one explanation per training point, we measure model accuracy with retrieval in a 3 × 2 design. There are three conditions for the retrieval model: (1) fixed, where the Sentence-RoBERTa retriever is fixed and only the classifier is trained, (2) learned, where both classifier and retriever are trained end- of tasks increases (equivalent to the number of points per task decreasing), reaching accuracies as low as 62.2% at num-tasks= 500. Meanwhile, we observe that providing the index does slightly ease the task inference, but the models can by no means memorize the map from index to the task information. Regarding model capacity, we find that using RoBERTa-large increases model accuracy when the number of num-tasks is relatively low (less than 250), but after this point RoBERTa-base performs better (see Fig. 13 in Appendix B). Lastly, we see that increasing the training set size can greatly improve model performance even with num-tasks= 500, reaching 87.11% with 50,000 training points (trend shown in Fig. 14 in Appendix B). However, we will see in the next section that, in terms of improving model accuracy, even this 10x increase in training size is less effective than using retrieved explanations with 5000 training points. [ roberta-large results in appendix. better at low-task regime, worse in high-task regime -PH ] 6.2. RQ2: Can retrieval of past explanations enable a model to solve our task?
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Design. Using the full-info explanations and data with num-tasks= 500, we measure model accuracy with retrieval in a 3 ⇥ 2 design. There are three conditions for the retrieval model: (1) fixed, where the Sentence-RoBERTa retriever is fixed and only the classifier is trained, (2) learned, where both classifier and retriever are trained end-to-end, and (3) optimal where the optimal retrieval model is used and the classifier is trained. Note that we know the optimal retrieval model assigns the highest probabilities to explanations with index e matching the query point's index x , so by using a retriever p(e i |x i ) = exp ( [index e = index x ]) and a context size lower than n task , we can ensure the retrieved explanations are always relevant. There are two conditions for the conditioning mechanism used: (1) TEXTCAT with C=k=6, Results. Shown in Fig. 6, the results show that retrieval with Sentence-BERT improves model accuracy by around 29 percentage points over a no-retrieval baseline. Each conditioning mechanism sees roughly the same improvement. Additionally, we can learn a retrieval model that does nearly as well as the optimal retrieval model, improving over the fixed condition by another 7 points. [ should we add some more reasons + conclusions/takeaways of these numerical results? [ added a couple takeaway sentences -PH ] -MB ]
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Thus, retrieval of explanations allows the model to perform much better than a no-retrieval baseline. We see a large improvement in performance from retrieval even when the baseline could learn to infer the task information directly from the index value in each input. In fact, explanation retrieval outperforms a no-retrieval baseline with as many as 50,000 training data points (a 10x increase), which obtains 87.11% accuracy.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
6.3. RQ3: Can models aggregate information across explanations for better prediction?
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Design. We run the same experiment design as for RQ2, using evidential and recomposable explanations (see Sec. 3.3). With evidential explanations, we shift each integer in the explanation (excluding the index) independently by zero-mean, discrete noise ✏ ⇠ unif( 2, 2). We use the 2-piece condition for recomposable explanations, meaning two explanations combine to give the full task information.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
As in RQ1, we show results here for values of C=k=6 for TEXTCAT and C=k=4 for H-MEAN.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Results. We display the results in Fig. 7. First, we observe that for evidential explanations, learned retrieval is close to-end, and (3) optimal where the optimal retrieval model is used and the classifier is trained. We know the optimal retrieval model retrieves explanations with an index matching the query point's index.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
The two conditioning mechanisms, H-MEAN and TEXTCAT, differ in how they combine information across multiple retrieved explanations to produce a final prediction (see Appendix B.1).
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Results. The results in Fig. 4 show that explanation retrieval can reach accuracies above 98%, improving accuracy by around 37 points over a no-explanation baseline. We also find that the learned retrieval model does as well as the optimal retrieval model, improving over the fixed condition by about 7 points. Thus, access to explanations allows the model to perform much better than a no-explanation baseline. In fact, the explanation retrieval model outperforms a no-explanation baseline with as many as 50,000 training data points (a 10x increase), which obtains 87.11% accuracy.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Design. We measure test accuracy as a function of how many unique explanations (and therefore label reasons) there are in the data. While keeping the train set size fixed at 5000 points, we vary how many points share the same explanation (index, m, n, r, d). By default there are 10 points per index, and with 5000 points this means that there are 500 unique explanations in the data. We use many as 2500 points per index, meaning using two unique explanations. The experiment conditions also vary in how task information is available in the input: (1) for With Explanation, each 20-integer sequence x i has its explanation appended to it; (2) for No Explanation, only x i is given, which requires the model to learn the map index → (m, n, r, d); (3) for No Index, the index is omitted from the input, so the model must infer the label reason from the sequence's contents alone.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
The results are shown in Fig. 5. We see that, when the number of unique explanations (and therefore possible label reasons) is small, the No Explanation model can achieve an accuracy as high as if it had been directly given the label reason, i.e. as high as the With Explanation condition. Yet, No Explanation model accuracy falls off quickly with the number of unique explanations, reaching accuracies as low as 62.2% with 500 explanations. Evidently, with this many unique explanations, it is too difficult to learn the map between the index and the latent label reason. Without the index in the input (No Index condition), it is even harder to infer the label reason. While accuracy does rise significantly with the size of the training data (see Fig. 4), even using 10x as much train data does not close the gap with the explanation retrieval model.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
We present a synthetic dataset with key similarities to natural language explanation data, and we show that our explanations are highly useful for model learning. However, we emphasize that if a model already "knew" the information in some explanations, it might not need them. This may plausibly occur with sufficiently large pretrained models that store a great deal of factual knowledge (Petroni et al., 2019). Similarly, the necessary information might be learnable from (X, Y ) data alone. Future work on modeling approaches we outline in this paper (Fig. 2) will benefit from testing their methods on controlled synthetic tasks as a test of their ability to learn from explanation data. Then, further analysis will be helpful for understanding how explanations contain novel information that is not learned elsewhere in pretraining or finetuning. 
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
There are several positive broader impacts from designing methods for learning from human explanations. Foremost among them is the promise of better aligning learned models with human priors on what kinds of behaviors are good, which could be especially helpful when these priors are hard to robustly encode in supervised learning objectives or unlikely to be learned from the available data. Explanations can also greatly improve model sample efficiency, which is broadly beneficial for difficult, time-consuming, or human-in-the-loop tasks where acquiring a large amount of data is expensive and slow. There are still some possible risks to this methodology, mainly involving overconfidence in what explanations can provide. For instance, just because explanations improve a model's performance does not mean the model will behave exactly as a human would. We risk anthropomorphizing machine learning models when we suppose their learned interpretations of explanations matches our own. Wang, and Felix Hill. 2022 Figure 6: A depiction of our retrieval-based method TEXTCAT. A total of Ck explanations are retrieved and allocated into k latent variables, each a set of explanations E, which are marginalized over to produce a final prediction.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
We represent the overall approach in Fig. 6 for one method of computing p θ (y|x, E) (described fully in Sec. B.1), where explanations are concatenated with the query sequence. Flowing from left to right, Fig. 6 shows how explanations are retrieved from the training data conditioned on a query sequence x, then allocated into k classifier inputs with C explanations each. The k classifier predictions are aggregated by marginalizing over the latent variable, Z = E.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Modeling Assumptions. In using retrieval, we make a few assumptions. First, since the number of forward passes per data point scales with k, we require a relatively small value of k, i.e. k ≤ 10, for reasonable computational efficiency in SGDbased training. Hence, we must assume that this summation is sufficiently similar to the full summation over latent variables. This assumption is more likely to hold when (1) a small number of documents account for most of the probability mass in p η (e|x), and (2) a pretrained model p η (e|x) yields a decent initial rank-ordering, such that some of the best documents are in the top-k. The exact value of k we use depends on the experiment. A second, more basic assumption is that explanations will be useful in predicting other data points' labels. Such an assumption is needed since we never condition on a data point's own explanation. Lastly, during retrieval we assume that explanations are independent given x, i.e. p(E|x) = e∈E p(e|x). This could be a poor assumption when, for instance, explanations each contribute one of a number of needed facts, in which case it would be helpful to retrieve additional explanations conditioned on what has already been retrieved.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
In this section we describe the methods used to compute p θ (y|x, E) and p η (e|x) (see Sec. B for the overall model description). For the classifier p θ (y|x, E), we use two methods, TEXTCAT and H-MEAN, which are described below. Then we describe the retrieval model, which is based on Sentence-BERT (Reimers and Gurevych, 2019).
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
TEXTCAT. Represented in Figure 6, this method takes a straightforward approach to conditioning on a set of explanations: concatenating C explanations and the input x to form a longer sequence of text. Each of the original sequences is separated by a special token, e.g.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
[SEP] for BERT. In our experiments, we pass this longer sequence into a RoBERTa-base model. After pooling the output token representations, we pass the resulting vector to a 1-layer MLP for classification. We use mean pooling for our synthetic task and NLI; for relation extraction tasks, we concatenate the representations corresponding to the initial tokens in the subject and object words, since this is an especially effective pooling technique (Baldini Soares et al., 2019).
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
This approach allows the model to reason over all of the explanations and the input together. While the method may be limited by the fact that some models can face difficulties in processing long pieces of text (Beltagy et al., 2020), this issue is partly mitigated by marginalizing over k sets of explanations. As a result of the marginalization, the final prediction can be conditioned on a far higher number (Ck) of individual explanations than could fit in the context alone. RoBERTa (x ) which is then passed to the MLP for classification. H-MEAN does not face the same sequence length limitations as TEXTCAT, but by separately processing of each explanations H-MEAN may fail to integrate information across explanations. This method also becomes expensive when we marginalize over E (which is what allows retrieval to be learned), as it requires Ck forward passes for a single prediction.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
We use a similar approach to retrieval as in Lewis et al. (2020), namely using vector representations of sequences from a pretrained transformer to compute p η (e|x) ∝ exp (f η (e) T f η (x)), which is followed by computing top-Ck(p η (·|x). We use an approximate but sub-linear time search method (FAISS) to find the top-Ck points (Johnson et al., 2017). In our experiments we find that it is necessary to use Sentence-BERT (Reimers and Gurevych, 2019) as our pretrained f η , rather than simply a pretrained RoBERTa model. Sentence-BERT is a network trained to produce semantic representations of sentences that can be compared under cosine similarity. In our experiments, we use the Sentence-RoBERTa-base model trained on a combination of several NLI and semantic textual similarity tasks, with mean pooling of token representations. We normalize the representations we obtain from this model, so that our inner product is equivalent to a cosine similarity.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Note that during training, we never condition on a data point's own explanation when predicting its label. This is an important constraint for matching the train and test-time distributions. At test time, we assume we have access only to past (training) explanations, since they can be expensive to collect and conditioning on explanations at test time can lead to label leakage, meaning what is essentially the benefit of human labeling could be mistaken as improvements in model performance.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
C Training Details C.1 Runtimes.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Regarding training times, we run most experiments on a single NVIDIA RTX 2080 GPU, with run-times as follows: 4.0 hours for 40 epochs of the noretrieval RoBERTa-base using the synthetic dataset; 5.7 hours for 40 epochs of RoBERTa-large in the same setting; 8.6 hours for 20 epochs of learned retrieval with RoBERTa-base models on synthetic data.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
For optimization, we use AdamW with a learning rate of 1e−5 and gradient norm clipping at norm 1. For the LR, we use a linear warmup and decay schedule peaking at 10% of the training steps for experiments with synthetic data and at 1% for experiments with existing datasets (given the larger training set sizes). The batch size is set to 10 across all experiments.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
We decide how often to rebuild the representations of training explanations while learning the retrieval model by tuning across frequency values in the range {10%, 20%, 33%, 50%, 100%} (i.e. to rebuild at this percentage of every epoch), as well as never rebuilding. In our synthetic setting, the only noticeable drop in performance comes from never rebuilding. As long as representations are re-encoded at least as often as every epoch, we notice no difference in final test accuracy, though in early experiments we observed that rebuilding more often improved training stability. To err on the safe side of training stability, we re-encode the representations every 20% of each epoch in all experiments except e-SNLI with full data, where we re-encode every 30% of each epoch.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
Additionally, we use the stop-gradient function when computing the gradient of p η (e|x) as follows:
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
∇ η exp (sg[f η (e)] T f η (x)), meaning that we do not differentiate through the explanation embeddings, but only through the query data point embeddings. In early experiments, we found that this decision contributed to training stability, while improving computational efficiency, and we confirm that we observe no differences in model accuracy as a result.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
We compute confidence intervals for our synthetic data tasks to represent seed variance around some mean seed performance. We represent seed variance in figures rather than sample variance because the sample variance is fairly low with 50,000 test points and could be driven arbitrarily low with more generated test points. For instance, the 95% confidence interval for a model accuracy of 90% would be ±0.26. To calculate seed variance, we run 10 random seeds for our baseline condition (no-retrieval) with the default synthetic task setup.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
The required parameters to the data generation include: (1) a training sample size sample-size and (2) num-tasks, the number of unique integer pairs to be counted, or, equivalently, the number of points per index, n task . In all experiments, we use a maximum integer value of 100 to appear in the sequences, and a maximum index value of 10,000. We give the general generative process below. Note that the dev and test sets are constructed with the extra constraint that sequences must not appear in the training data. Further note that this is the generic version of generative process, and in some experiments the process is altered. For example, in RQ3, indicator is always 1 and the construction of the map from index values to (m, n) tuples occurs in a special way described in the experimental design for RQ3.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
from the uniform distribution over integers {1,...,10000} without replacement.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
2. Sample {(m, n, r, d) t } num-tasks τ =1 from the uniform distribution over integers, unif ([1, 100] 4 ), without replacement and requiring that m = n = r = d.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
3. Define the set {(index, m, n, r, d) index )} for index and (m, n, r, d) drawn from their respective sets, without replacement, in an arbitrary order.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
4. Compute the number of points per index, n task = sample-size // num-tasks.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
τ =1
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
: (a) Sample a vector of length n task , balanced between 1s and 2s, that gives the values of {indicator p } P p=1 for the P points with that index. should correlate (1 implies they are equal, and 0 unequal). This balance changes when the strong-weak correlation is intended to change.
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
(c) Sample a vector of length n task , balanced between 0s and 1s, representing whether (m, n) or (r, d) should be the more numerous integers in the sequence (so that there is no bias, even randomly, between features by size). (d) For i ∈ 1 : n task :
When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Data
i. Place the index in the first element of an empty array, and the indicator in the second. ii. Based on the i th elements of the three vectors described above, allocate samples of the integers in (m, n, r, d) index into the remaining 18 slots. iii. If there are any remaining slots after these integers are randomly allocated, fill them with i.i.d. samples from unif (1, 100).
Can Rationalization Improve Robustness?
Rationale models aim to introduce a degree of interpretability into neural networks by implicitly baking in explanations for their decisions (Lei et al., 2016;Bastings et al., 2019;. These models are carried out in a two-stage 'rationalizethen-predict' framework, where the model first selects a subset of the input as a rationale and then makes its final prediction for the task solely using the rationale. A human can then inspect the selected rationale to verify the model's reasoning Figure 1: Top: input text is processed by a rationale model (rationalizer and predictor) and a full-context model (making predictions based on the whole input) separately in a beer review sentiment classification dataset. Both models make correct predictions. Bottom: when an attack sentence "The tea looks horrible." is inserted, the full-context model fails. The rationalizer successfully excludes the negative-sentiment word "horrible" from the selected rationales (yellow highlights). The predictor is hence not distracted by the attack sentence. over the most relevant parts of the input for the prediction at hand.
Can Rationalization Improve Robustness?
While previous work has mostly focused on the plausibility of extracted rationales and whether they represent faithful explanations (DeYoung et al., 2020), we ask the question of how rationale models behave under adversarial attacks (i.e., do they still provide plausible rationales?) and whether they can help improve robustness (i.e., do they provide better task performance?). Our motivation is that the two-stage decision-making could help models ignore noisy or adversarially added text within the input. For example, Figure 1 shows a state-of-the-art rationale model (Paranjape et al., 2020) smoothly handles input with adversarially added text by selectively masking it out during the rationalization step. Factorizing the rationale prediction from the task itself effectively 'shields' the predictor from having to deal with adversarial inputs.
Can Rationalization Improve Robustness?
To answer these questions, we first generate adversarial tests for a variety of popular NLP tasks ( §4). We focus specifically on model-independent, 'AddText' attacks (Jia and Liang, 2017), which augment input instances with noisy or adversarial text at test time, and study how the attacks affect rationale models both in their prediction of rationales and final answers. For diversity, we consider inserting the attack sentence at different positions of context, as well as three types of attacks: random sequences of words, arbitrary sentences from Wikipedia, and adversarially-crafted sentences.
Can Rationalization Improve Robustness?
We then perform an extensive empirical evaluation of multiple state-of-the-art rationale models (Paranjape et al., 2020;Guerreiro and Martins, 2021), across five different tasks that span review classification, fact verification, and question answering ( §5). In addition to the attack's impact on task performance, we also assess rationale prediction by defining metrics on gold rationale coverage and attack capture rate. We then investigate the effect of incorporating human rationales as supervision, the importance of attack positions, and the lexical choices of attack text. Finally, we investigate an idea of improving the rationalizer by adding augmented pseudo-rationales during training ( §7).
Can Rationalization Improve Robustness?
Our key findings are the following: 1. Rationale models show promise in providing robustness. Under our strongest type of attack, rationale models in many cases achieve less than 10% drop in task performance while fullcontext models suffer more (11%-27%). 2. However, robustness of rationale models can vary considerably with the choice of lexical inputs for the attack and is quite sensitive to the attack position. 3. Training models with explicit rationale supervision does not guarantee better robustness to attacks. In fact, their accuracy drops under attack are higher by 4-10 points compared to rationale models without supervision. 4. Performance under attacks is significantly improved if the rationalizer can effectively mask out the attack text. Hence, our simple augmented-rationale training strategy can effectively improve robustness (up to 4.9%). Overall, our results indicate that while there is promise in leveraging rationale models to improve robustness, current models may not be sufficiently equipped to do so. Furthermore, adversarial tests may provide an alternative form to evaluate rationale models in addition to prevalent plausability metrics that measure agreement with human rationales. We hope our findings can inform the development of better methods for rationale predictions and instigate more research into the interplay between interpretability and robustness.
Can Rationalization Improve Robustness?
Extractive rationale models 2 output predictions through a two-stage process: the first stage ("rationalizer") selects a subset of the input as a rationale, while the second stage ("predictor") produces the prediction using only the rationale as input. Rationales can be any subset of the input, and we characterize them roughly into either token-level or sentence-level rationales, which we will both investigate in this work. The task of predicting rationales is often framed as a binary classification problem over each atomic unit depending on the type of rationales. The rationalizer and the predictor are often trained jointly using task supervision, with gradients back-propagated through both stages. We can also provide explicit rationale supervision, if human annotations are available.
Can Rationalization Improve Robustness?
Formally, let us assume a supervised classification dataset D = {(x, y)} , where each input x = x 1 , x 2 , ..., x T is a concatenation of T sentences and each sentence x t = (x t,1 , x t,2 , ...x t,nt ) contains n t tokens, and y refers to the task label. A rationale model consists of two main components: 1) a rationalizer module z = R(x; θ), which generates a discrete mask z ∈ {0, 1} L such that z x selects a subset from the input (L = T for sentence-level rationalization or L = T i=1 n i for token-level rationales), and 2) a predictor moduleŷ = C(x, z; φ) that makes a predictionŷ using the generated rationale z. The entire model M (x) = C(R(x)) is trained end-to-end using the standard cross-entropy loss. We describe detailed training objectives in §5.
Can Rationalization Improve Robustness?
Rationale models are traditionally evaluated along two dimensions: a) their downstream task performance, and b) the quality of generated rationales.
Can Rationalization Improve Robustness?
To evaluate rationale quality, prior work has used metrics like token-level F1 or Intersection Over Union (IOU) scores between the predicted rationale and a human rationale (DeYoung et al., 2020):
Can Rationalization Improve Robustness?
where z * is the human-annotated gold rationales. A good rationale model should not sacrifice task performance while generating rationales that concur with human rationales. However, metrics like F1 score may not be the most appropriate way to capture this as it only captures plausibility instead of faithfulness (Jacovi and Goldberg, 2020).
Can Rationalization Improve Robustness?
Our goal is to construct attacks that can test the capability of extractive rationale models to ignore spurious parts of the input. Broadly, we used two guiding criteria for selecting the type of attacks: 1) they should be additive since an extractive rationale model can only "ignore" the irrelevant context. For other attacks such as counterfactually edited data (CAD) (Kaushik et al., 2020), even if the rationalizer could identify the edited context, the predictor is not necessarily strong enough to reason about the counterfactual text; 2) they should be modelindependent since our goal is to compare the performance across different types of rationale and baseline models. Choosing strong gradient-based attacks (Ebrahimi et al., 2018; would probably break all models, but that is beyond the scope of our hypothesis. An attack is suitable as long as it reduces performance of standard classification models by a non-trivial amount (our attacks reduce performance by 10%-36% absolute).
Can Rationalization Improve Robustness?
Keeping these requirements in mind, we focus on label-preserving text addition attacks Jia and Liang (2017), which can test whether rationale models are invariant to the addition of extraneous information and remain consistent with their predictions. Attacks are only added at test time and are not available during model training.
Can Rationalization Improve Robustness?
Attack construction Formally, an AddText attack A(x) modifies the input x by adding an attack sentence x adv , without changing the ground truth label y. In other words, we create new perturbed test instances (A(x), y) for the model to be evaluated on. While some prior work has considered the addition of a few tokens to the input , we add complete sentences to each input, similar to the attacks in Jia and Liang (2017). This prevents unnatural modifications to the existing sentences in the original input x and also allows us to test both token-level and sentence-level rationale models ( §5.1). We experiment with adding the attack sentence x adv either at the beginning or the end of the input x. 3
Can Rationalization Improve Robustness?
Types of attacks We explore three different types of attacks: (1) AddText-Rand: we simply add a random sequence of tokens uniformly sampled from the task vocabulary. This is a weak attack that is easy for humans to spot and ignore since it does not guarantee grammaticality or fluency.
Can Rationalization Improve Robustness?
(2) AddText-Wiki: we add an arbitrarily sampled sentence from English Wikipedia into the task input (e.g., "Sonic the Hedgehog, designed for . . . "). This attack is more grammatical than AddText-Rand, but still adds text that is likely irrelevant in the context of the input x.
Can Rationalization Improve Robustness?
(3) AddText-Adv: we add an adversarially constructed sentence that has significant lexical overlap with tokens in the input x while ensuring the output label is unchanged. This type of attack is inspired by prior attacks such as AddOneSent (Jia and Liang, 2017) and is the strongest attack we consider since it is more grammatical, fluent, and contextually relevant to the task. The construction of this attack is also specific to each task we consider, hence we provide examples listed in Table 1 and more details in §5.3.
Can Rationalization Improve Robustness?
We measure the robustness of rationale models under our attacks along two dimensions: task performance, and generated rationales. The change in task performance is simply computed as the differ-3 In §6.4, we also consider inserting the attack sentence at a random position for studying the effect of attack positions. ence between the average scores of the model on the original vs perturbed test sets:
Can Rationalization Improve Robustness?
where f denotes a scoring function (F1 scores in extractive question answering and I(y =ŷ) in text classification). To measure the effect of the attacks on rationale generation, we use two metrics:
Can Rationalization Improve Robustness?
Gold rationale F1 (GR) This is defined as the F1 score between the predicted rationale and a humanannotated rationale, either computed at the token or sentence level. The token-level GR score is equivalent to F1 scores reported in previous work (Lei et al., 2016;DeYoung et al., 2020). A good rationalizer should generate plausible rationales and be not affected by the addition of attack text.
Can Rationalization Improve Robustness?
Attack capture rate (AR) We define AR as the recall of the inserted attack text in the rationale generated by the model:
Can Rationalization Improve Robustness?
where x adv is the attack sentence added to each instance (i.e., A(x) is the result of inserting x adv into x), z A(x) is the predicted rationale. The metric above applies on both token or sentence level (|x adv | = 1 for sentence-level rationalization and number of tokens in the attack sentence for token-level rationalization). This metric allows us to measure how often a rationale model can ignore the added attack text-a maximally robust rationale model should have an AR of 0.  Table 1: AddText-Adv attack applied to three datasets. The query (blue) is transformed into an attack (red). The query together with the context forms the input. The attack is inserted to the context. We only show insertion at the end, but the attack can be inserted at any position between sentences. A model needs to associate the query and the evidence (ground truth rationale) in the context and not be distracted by the inserted attack to make the correct prediction. Note that the Beer dataset doesn't have a query and the attack sentence is dependent on the label ( §5.3).
Can Rationalization Improve Robustness?
Variational information bottleneck (VIB) This model (Paranjape et al., 2020) imposes a discrete bottleneck objective (Alemi et al., 2017) to select a mask z ∈ {0, 1} L from the input x.
Can Rationalization Improve Robustness?
The rationalizer samples z using Gumbel-Softmax and the predictor uses only z x for the final prediction. During inference, we select the top-k scored rationales, where k is determined by the sparsity π.
Can Rationalization Improve Robustness?
Sparse structured text rationalization (SPEC-TRA) This model (Guerreiro and Martins, 2021) extracts a deterministic structured mask z by solving a constrained inference problem by applying factors to the global scoring function while optimizing the end task performance. The entire computation is deterministic and allows for backpropagation through the LP-SparseMAP solver (Niculae and Martins, 2020). We use the BUDGET factor to control the sparsity π.
Can Rationalization Improve Robustness?
Full-context model (FC) As a baseline, we also consider a full-context model, which makes predictions directly conditioned on the entire input. The model is a standard BERT model which adds taskspecific classifiers on top of the encoder (Devlin et al., 2019). The model is trained with a crossentropy loss using task supervision.
Can Rationalization Improve Robustness?
VIB with human rationales (VIB-sup) When human-annotated rationales z * are available, they can be used to guide the prediction of the sampled masks z by adding a cross entropy loss between them (more details in §A.1). VIB-sup leverages this supervision signal to guide rationale prediction.  Full-context model with human rationales (FCsup) We also extend the FC model to leverage human-annotated rationales supervision during training by adding a linear layer on top of the sentence/token representations. Essentially, it is multi-task learning of rationale prediction and the original task, shared with the same BERT encoder. The supervision is added by calculating the cross entropy loss between the human-annotated rationales and the predicted rationales ( §A.1).
Can Rationalization Improve Robustness?
We evaluate the models on five datasets that cover both sentence-level (FEVER, MultiRC, SQuAD) and token-level (Beer, Hotel) rationalization (examples in Table 1). We summarize the dataset characteristics in Table 2.
Can Rationalization Improve Robustness?
FEVER FEVER is a sentence-level binary classification fact verification dataset from the ERASER benchmark (DeYoung et al., 2020). The input contains a claim specifying a fact to verify and a passage of multiple sentences supporting or refuting the claim. For the AddText-Adv attacks, we add modified query text to the claims by replacing nouns and adjectives in the sentence with antonyms from WordNet (Fellbaum, 1998).
Can Rationalization Improve Robustness?
MultiRC MultiRC (Khashabi et al., 2018) is a sentence-level multi-choice question answering task (reformulated as 'yes/no' questions). For the AddText-Adv attacks, we transform the question and the answer separately using the same procedure we used for FEVER.
Can Rationalization Improve Robustness?
SQuAD SQuAD (Rajpurkar et al., 2016) is a popular question answering dataset. We use the Ad-dOneSent attacks proposed in Adversarial SQuAD (Jia and Liang, 2017), except that they always insert the sentence at the end of the paragraph and we consider inserting at the beginning, the end, and a random position. Since SQuAD does not contain human rationales, we use the sentence that contains the correct answer span as the ground truth rationale sentence. We report F1 score for SQuAD.
Can Rationalization Improve Robustness?
Beer BeerAdvocate is a multi-aspect sentiment analysis dataset (McAuley et al., 2012), modeled as a token-level rationalization task. We use the appearance aspect in out experiments. We convert the scores into the binary labels following Chang et al. (2020). This task does not have a query as in the previous tasks, we insert a sentence with the template "{SUBJECT} is {ADJ}" into a negative review where the adjective is positive (e.g., "The tea looks fabulous.") and vice versa. We consider one object "car" and eight adjectives (e.g., "clean/filthy", "new/old").
Can Rationalization Improve Robustness?
Hotel TripAdvisor Hotel Review is also a multiaspect sentiment analysis dataset (Wang et al., 2010). We use the cleanliness aspect in our experiments. We generate AddText-Adv attacks in the same way as we did for the Beer dataset. We consider three objects ranging from more relevant words such as "tea" to less related word "carpet" and six adjectives (e.g., "pretty/disgusting", "good/bad", "beautiful/ugly").
Can Rationalization Improve Robustness?
For all attacked test sets, we report the average scores with attack sentence inserted at the beginning and the end of the inputs. Our findings shed light on the relationship between GR, AR, and drop in performance, which eventually lead to a promising direction to improve performance of rationale models under attacks ( §7). Figure 2 summarizes the average scores on all datasets for each model under the three attacks we consider. We first observe that all models (including the full-context models FC and FC-sup) are mildly affected by AddText-Rand and AddText-Wiki, with score drops of around 1-2%. However, the AddText-Adv attack leads to more significant drops in performance for all models, as high as 46% for SPECTRA on the Hotel dataset. We break out the AddText-Adv results in a more fine-grained manner in Table 3 
Can Rationalization Improve Robustness?
In Table 4, we report the Gold Rationale F1 (GR) and Attack Capture Rate (AR) for all models. When attacks are added, GR consistently decreases for all tasks. However, AR ranges widely across datasets. VIB and SPECTRA have lower AR and higher GR compared to FC-sup across all tasks, which is correlated with their superior robustness to AddText-Adv attacks. Next, we investigate the poor performance of VIB and SPECTRA on the Hotel dataset by analyzing the choice of words in the attack. Using the template "My car is {ADJ}.", we measure the percentage of times the rationalizer module selects the adjective as part of its rationale. When the adjectives are "dirty" and "clean", the VIB model selects them a massive 98.5% of the time. For "old" and "new", VIB still selects them 50% of the time. On the other hand, the VIB model trained on Beer reviews with attack template "The tea is {ADJ}." only selects the adjectives 20.5% of the time (when the adjectives are "horrible" and "fabulous"). This shows that the bad performance of the rationale models on the Hotel dataset is due to their inability to ignore task-related adjectives in the attack text, hinting that the lexical choices made in constructing the attack can largely impact robustness.   We examine where the rationale model gains robustness by inspecting the generated rationales. Table 5 shows the accuracy breakdown under attack for VIB and VIB-sup models. Intuitively, both models perform best when the gold rationale is selected and the attack is avoided, peaking at 91.1 for VIB and 92.4 for VIB-sup. Models perform much worse when the gold rationale is omitted and the attack is included (73.6 for VIB and 74.1 for VIB-sup), highlighting the importance of choosing good and skipping the bad as rationales.
Can Rationalization Improve Robustness?
Perhaps surprisingly, adding explicit rationale supervision does not help improve robustness (Ta-ble 3). Across FEVER, MultiRC and SQuAD, VIBsup consistently has a higher ∆ between its scores on the original and perturbed instances. We observe that models trained with human rationales generally have higher GR, but they also capture a much higher AR across the board (Table 4). On MultiRC, for instance, the VIB-sup model outperforms VIB in task performance because of its higher GR (36.1 versus 15.8). However, when under attack, VIBsup's high 58.7 AR, hindering the performance compared to VIB, which has a smaller 35.8 AR. This highlights a potential shortcoming of prior work in only considering metrics like IOU (similar in spirit to GR) to assess rationale models. The finding also points to the risk of straightforwardly  incorporating supervised rationale as it could result in the existing model overfitting to them.
Can Rationalization Improve Robustness?
We further analyze the effect of attack text on rationale models by varying the attack position. Figure 3 displays the performance of VIB, VIB-sup and FC on FEVER and SQuAD when the attack sentence is inserted into the first, last or any random position in between. We observe performance drops on both datasets when inserting the attack sentence at the beginning of the context text as opposed to the end. For example, when the attack sentence is inserted at the beginning, the VIB model drops from 77.1 F1 to 40.9 F1, but it only drops from 77.1 F1 to 72.1 F1 for a last position attack on SQuAD. This hints that rationale models may implicitly be picking up positional biases from the dataset, similar to their full-context counterparts (Ko et al., 2020). We provide fine-grained plots for AR versus attack positions in §A.4.   
Can Rationalization Improve Robustness?
From our previous analysis on the trade-off between GR and AR ( §6.2), it is clear that avoiding attack sentences in rationales is a viable way to make such models more robust. Note that this is not obvious by construction since the addition of attacks affects other parameters such as position of the original text and discourse structure, which may throw off the 'predictor' component of the model. As a more explicit way of encouraging 'rationalizers' to ignore spurious text, we propose a simple method called augmented rationale training (ART). Specifically, we sample two sentences at random from the Wikitext-103 dataset (Merity et al., 2017) and insert them into the input passage at random position, setting their pseudo rationale labels z pseudo = 1 and the labels for all other sentences as z = 0. We limit the addition to only inserting two sentences to avoid exceeding the rationalizer maximal token limit. We then add an auxiliary negative binary cross entropy loss to train the model to not predict the pseudo rationale. This encourages the model to ignore spurious text that is unrelated to the task. Note that this procedure is both model-agnostic and does not require prior knowledge of the type of AddText attack. Table 6 shows that ART improves robustness across the board for all models (FC-sup, VIB and VIB-sup) in both FEVER and MultiRC, dropping ∆ scores by as much as 5.9% (VIB-sup on FEVER). We further analyzed these results to break down performance in terms of attack and gold sentence capture rate. Table 7 shows that ART greatly improves the percentage of sentences under the "Gold Attack " category (31.8% → 65.4% for VIB and 11.3% → 63.5% for VIB-sup). This corroborates our expectations for ART and shows its effec-tiveness at keeping GR high while lowering AR.
Can Rationalization Improve Robustness?
Interestingly, the random Wikipedia sentences we added in ART are not topically or contextually related to the original instance text at all, yet they seem to help the trained model ignore adversarially constructed text that is tailored for specific test instances. This points to the promise of ART in future work, where perhaps more complex generation schemes or use of attack information could provide even better robustness.  
Can Rationalization Improve Robustness?
In this work, we investigated whether neural rationale models are robust to adversarial attacks. We constructed a variety of AddText attacks across five different tasks and evaluated several state-of-theart rationale models. Our findings raise two key messages for future research in both interpretability and robustness of NLP models: Interpretability We identify an opportunity to use adversarial attacks as a means to evaluate rationale models (especially extractive ones). In contrast to existing metrics like IOU used in prior work (DeYoung et al., 2020;Paranjape et al., 2020), robustness more accurately tests how crucial the predicted rationale is to the model's decision making. Further, our analysis reveals that even stateof-the-art rationale models may not be consistent in focusing on the most relevant parts of the input, despite performing well on tasks they are trained on. This points to the need for better model architectures and training algorithms to better align rationale models with human judgements.
Can Rationalization Improve Robustness?
Robustness For adversarial attack research, we show that extractive rationale models are promising for improving robustness, while being sensitive to factors like the attack position or word choices in the attack text. Research that proposes new attacks can use rationale models as baselines to assess their effectiveness. Finally, the effectiveness of ART points to the potential for data augmentation in improving robustness of NLP systems, even against other types of attacks beyond AddText.
Can Rationalization Improve Robustness?
We hope our results can inspire more research at the intersection of interpretability and robustness.
Can Rationalization Improve Robustness?
Training Both the rationalizer and the predictor in the rationale models are initialized with pretrained BERT (Devlin et al., 2019). We predetermine rationale sparsity before fine-tuning based on the average rationale length in the development set following previous work (Paranjape et al., 2020;Guerreiro and Martins, 2021). We set π = 0.4 for FEVER, π = 0.2 for MultiRC, π = 0.7 for SQuAD, π = 0.1 for Beer, and π = 0.15 for Hotel. The hyperparameter k (for top-k ratioanle extraction) is selected based on the percentage π of the human annotated rationales in the development set (following Paranjape et al. (2020)). During evaluation, for each passage k = π × #sentences. We select the model parameters based on the highest fine-tuned task performance on the development set. The models with rationale supervision will select the same amount of text as their no-supervision counterparts. The epoch/learning rate/batch size for the different datasets are described in Table A 
Can Rationalization Improve Robustness?
We provide qualitative examples of the rationale model predictions for each dataset in Table 8. Figure 4 shows a more fine-grained trend reflecting the sensitivity of AR against inserted attack position. As the attack position move from the beginning of the passage towards the end, AR decreases across all models. With ART training (R6 in §6), the AR also becomes less sensitive to positions. We also experimented with various adjectives related to appearance as the attack and observe the same trend. For example, when inserting "The carpet looks really ugly/beautiful." to the Beer dataset, VIB performance drops 93.8 → 83.1 while FC drops 93.8 → 61.6. The Silver Surfer is a fictional superhero appearing in American comic books published by Marvel Comics. The character also appears in a number of movies , television , and video game adaptations. The character was created by Jack Kirby , and first appeared in the comic book Fantastic Four # 48 , published in 1966. The Silver Surfer is a humanoid with metallic skin who can travel space with the aid of his surfboard-like craft. Originally a young astronomer named Norrin Radd on the planet Zenn-La , he saved his homeworld from the planet devourer , Galactus , by serving as his herald. Imbued in return with a tiny portion of Galactus 's Power Cosmic , Radd acquired vast power , a new body and a surfboard-like craft on which he could travel faster than light. Now known as the Silver Surfer , Radd roamed the cosmos searching for planets for Galactus to consume. When his travels took him to Earth , he met the Fantastic Four , a team of powerful superheroes who helped him rediscover his humanity and nobility of spirit. Betraying Galactus , the Surfer saved Earth but was exiled there as punishment. The Carey Hayes appears only in scottish comic books.
Can Rationalization Improve Robustness?
MultiRC What did Jenny and her friends enjoy when they walked to the sand ? || The sun
Can Rationalization Improve Robustness?
Jenny was a 13 year old girl with blond hair and blue eyes . She had gotten out of her last day of school and was free for the summer. Two of her friends were going to the nearby beach to do some swimming and enjoy the sun. Jenny went with them and when they got there the beach was very full and there were people everywhere. They changed into their bathing suits and went to the water. The water was very cold. They chose not swim and walked to the sand. Then they laid down on some towels and enjoyed the sun. After several hours Jenny and her friends fell asleep. Jesse and her foe enjoy the moon when they walked to the sand. They woke up and the sun was beginning to set. When Jenny sat up she found that it was painful to touch her skin. When she looked down she saw that she had a very bad sunburn. Her friends were also very badly sunburned so they went home. Jenny 's mother gave her a cream to put on the sunburn.
Can Rationalization Improve Robustness?
SQuAD When did oil finally returned to its bretton woods levels ?
Can Rationalization Improve Robustness?
This contributed to the "oil shock". After 1971, opec was slow to readjust prices to reflect this depreciation. From 1947 to 1967, the dollar price of oil had risen by less than two percent per year. Until the oil shock, the price had also remained fairly stable versus other currencies and commodities. Opec ministers had not developed institutional mechanisms to update prices in sync with changing market conditions, so their real incomes lagged. The substantial price increases of 1973 -1974 largely returned their prices and corresponding incomes to bretton woods levels in terms of commodities such as gold. Oil finally returned to its colossus mickelson levels in 1898. 1973-1974 / 1973-1974 Beer The tea looks horrible. Poured from a 12oz bottle into a delirium tremens glass. This is so hard to find in columbus for some reason, but I was able to get it in toledo... murky yellow appearance with a very thin white head. The aroma is bready and a little sour. The flavor is really complex, with at least the following tastes: wheat, spicy hops, bread, bananas, and a toasty after -taste. It was really outstanding. I'd recommend this to anyone, go out and try it. I think it's the best so far from this brewery.
Can Rationalization Improve Robustness?
Hotel My car is very filthy. The hotel was in a brilliant location and very near a metro station. Yes the room was small but it was clean and very well equipped. the bathroom was a really good size and lets face it how long do you spend in your hotel room anyway? The breakfast was fantastic and the staff were really friendly and helpful. We will definately stay here when we return to barcelona. It's worth going up to the roof of the hotel for the view over the city.
Can Rationalization Improve Robustness?
Negative / Positive 
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Recent scaling of pre-training has empowered large language models (LLMs) to learn NLP tasks from just a few training examples "in context," without updating the model's parameters (Brown et al., 2020). However, this learning process is still poorly understood: models are biased by the order of in-context examples (Zhao et al., 2021) and may not leverage the instructions or even the labels of the examples in the ways one expects (Min et al., 2022;Webson and Pavlick, 2022). Existing tools for interpreting model predictions have high computational cost (Ribeiro et al., 2016) or require access to gradients (Simonyan et al., 2014;Sundararajan et al., 2017), making them unsuitable for investigating in-context learning or explaining the predictions of prompted models.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
One appealing way to gain more insight into predictions obtained through in-context learning is to let the language model "explain itself" (Nye et al., 2021;Wei et al., 2022;Chowdhery et al., 2022;Marasović et al., 2022;Lampinen et al., 2022). In addition to input-label training pairs in context, one can prompt the language model with an explanation for each pair and trigger the model to generate an explanation for its prediction (Figure 1). Prompting with explanations introduces much richer information compared to using labels alone, which might guide the inference process and allow the model to learn more information from the examples.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
In this work, we investigate the nature of the explanations that LLMs generate and whether they can improve few-shot in-context learning for textual reasoning tasks, specifically QA and NLI. Recent prior work that finds success with this approach largely targets symbolic reasoning tasks with a very different structure, such as math word problem solving (Nye et al., 2021;Wei et al., 2022). We experiment on three different datasets spanning QA and NLI with four LLMs: OPT, , , and text-davinci-002. The results suggest that explanations only substantially improve accuracy for text-davinci-002, but give a smaller improvement or even hurt the performance with the other LLMs.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Surprisingly, we find that the explanations generated by LLMs can be unreliable, even for a very simple synthetic dataset. We evaluate the explanations along two axes: factuality, whether the explanation is correctly grounded in the input, and consistency, whether the explanation entails the final prediction. LLMs tend to generate consistent explanations that account for the predictions, but the explanations may not be factual, as as shown in Figure 1. Furthermore, our analysis suggests an unreliable explanation more likely indicates a wrong prediction compared to a reliable explanation.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Despite LLMs' failures here, we can still benefit from model-generated explanations by using them for calibration. If we are able to automatically assess the reliability of an explanation, we can allow an LLM to return a null answer when its explanation is unreliable, since the prediction in this case is less likely to be correct. Unfortunately, there is no automated way to perfectly assess the reliability, but we can extract features that approximately reflect it. We use these features to calibrate InstructGPT's 2 predictions, and successfully improve the in-context learning performance across all the datasets.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
In summary, our main findings are: (1) Simply plugging explanations into the prompt does not always substantially boost the in-context learning performance for textual reasoning. (2) LLMs generate explanations consistent with their predictions, but these explanations might not be factually grounded in the inputs. (3) The factuality of an explanation can serve as an indicator for the correctness of the corresponding prediction. (4) Using features that can approximate the factuality of explanations, we successfully use explanations to improve the in-context learning performance across all tasks.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
In this paper, we specifically focus on tasks involving reasoning over natural language. These are tasks where explanations have been traditionally studied Rajani et al., 2019), but which are more complex than tasks like sentiment analysis which are well explained by extractive rationales (Zaidan et al., 2007;DeYoung et al., 2020). We experiment on two tasks, 2 Throughout our paper, we primarily test on InstructGPT for two reasons. First, it was the most capable model available at the time we were conducting the majority of our experiments. Second, it still has significant room to improve on the datasets we explore in this work. This setting is a representative testbed for the situation where an LLM-based system does not yet give satisfactory performance on a target task, causing the system designer to turn to explanations in prompts to improve things. 
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Premise: A toddler in a green jersey is being followed by a wheelchair bound woman in a red sweater past a wooden bench. Hypothesis: A toddler is walking near his wheelchair bound grandmother.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Label: Neither Explanation: the woman may not be his grandmother. Figure 2: A SYNTH example and an E-SNLI example. See Figure 3 for ADVHOTPOT examples.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
reading comprehension question answering (QA) and natural language inference (NLI), on three English-language datasets. For each dataset, we create a test set with 250 examples.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Synthetic Multi-hop QA (SYNTH) In order to have a controlled setting where we can easily understand whether explanations are factual and consistent with the answer, we create a synthetic multi-hop QA dataset. Shown in Figure 2, each example in this dataset asks a bridge question (using the terminology of Yang et al. (2018)) over a context consisting of supporting facts paired with controlled distractors. This dataset is carefully designed to avoid spurious correlations, giving us full understanding over the correct reasoning process and the explanation for every example, which naturally consists of the two supporting sentences. See Appendix B for full details of this dataset. 3
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Adversarial HotpotQA (ADVHOTPOT) We also test on the English-language Adversarial Hot-potQA dataset (Yang et al., 2018;Jiang and Bansal, 2019). We use the adversarially augmented version since InstructGPT achieves high performance on the distractor setting of the original dataset. We make a challenging set of examples by balancing sets of questions on which InstructGPT makes correct and incorrect predictions. The context of each question includes two ground truth supporting paragraphs and two adversarial paragraphs. Full details of preprocessing the ADVHOTPOT dataset can be found in Appendix C.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
For ADVHOTPOT, we manually annotated explanations for the training examples. Figure 1 shows an example of such an explanation, highlighted in orange. We could use the supporting sentences as the explanations, but we found they are usually too verbose and not sufficient, e.g., with anaphors that resolve outside of the supporting sentences. Therefore, we manually annotate a set of explanations which clearly describe the reasoning path for each question.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
E-SNLI E-SNLI  is an English-language classification dataset commonly used to study explanations, released under the MIT license. Shown in Figure 2, each example consists of a premise and a hypothesis, and the task is to classify the hypothesis as entailed by, contradicted by, or neutral with respect to the premise. As a notable contrast to the other datasets, the explanations here are more abstract natural language written by human annotators, as opposed to mostly constructed from extracted snippets of context.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We study the effectiveness of plugging in explanations by comparing the in-context learning performance of prompting with or without explanations. Prompting without explanations resembles the standard few-shot in-context learning approach (Few-Shot). To incorporate explanations into the prompt, we consider the following two most commonly used paradigms:
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Explain-then-Predict (E-P) prepends an explanation before the label (Figure 1). The language model is expected to generate an explanation first followed by the prediction. The prompting style of past work involving computational traces can be categorized into this paradigm, including Nye et al. (2021) and Wei et al. (2022). This approach is also called a pipeline model in other literature on training models using explanations (Jacovi and Goldberg, 2021;Wiegreffe et al., 2021). Predict-then-Explain (P-E) generates the explanation after the prediction. Unlike E-P, the predicted explanation does not influence the predicted label, since we use greedy inference and the explanation comes afterwards. However, the explanations in the prompt still impact the predictions.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
For few-shot learning, we use roughly the maximum allowed shots in the prompt that can fit the length limit of OPT (Zhang et al., 2022) and GPT-3 (Brown et al., 2020), which is 16 for SYNTH, 6 for ADVHOTPOT, and 32 for E-SNLI, respectively. 4 We experiment with four LLMs, including OPT (175B), GPT-3 (davinci), InstructGPT (text-davinci-001), and text-davinci-002. OPT and GPT-3 are trained using the standard causal language modeling objective, whereas InstructGPT and text-davinci-002 are trained with special instruction data and human annotations. We generate outputs with greedy decoding (temperature set to be 0). Our prompt formats follow those in Brown et al. (2020). The explanations are inserted before/after the prediction with conjunction words like because. Please refer to Appendix A for full prompts. Because the results of in-context learning vary with the examples presented in the input prompt, for each dataset, we randomly sample multiple groups of training shots, and report the mean and standard deviation of the results (subscript). We use 5 groups for InstructGPT, the primary LM we are using throughout our paper, and 3 groups for the rest.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
As shown in Table 1, OPT, GPT-3, and InstructGPT show small to moderate improvements from using explanations for textual reasoning tasks. On the two QA tasks, SYNTH and ADVHOTPOT, E-P improves the performance of InstructGPT, the best among these three LMs, from 54.8 to 58.5 and 56.8 to 59.4, respectively. 5 On E-SNLI, P-E outperforms FEW-SHOT by 2.6, whereas E-P substantially lags FEW-SHOT. Comparing E-P against P-E on SYNTH and E-SNLI, E-P typically degrades performance (except on SYNTH for InstructGPT) and P-E is inconsistent across the different models, whereas E-P consistently leads to performance improvements on ADVHOTPOT. There is no single winner between the two paradigms of using explanations; choosing the most effective way is task-specific. Overall, vanilla LLMs (OPT and GPT-3) see limited benefit from producing explanations, and even the Instruct-series InstructGPT does not see substantial improvements.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
The only exception is text-davinci-002. text-davinci-002 greatly benefits from explanations in the prompt across all three tasks, and E-P is consistently more effective than P-E. However, it is unclear what contributes to this difference. As far as we are aware, the differences between text-davinci-002  and InstructGPT are not described in any publication or blog post. 6 Comparing GPT-3 and Instruct-GPT, we see the move to Instruct series models is not sufficient to explain the difference. Given the lack of transparency with this model, we hesitate to make scientific claims about the results it yields.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Our results do not suggest immediate strong improvements from incorporating explanations across all LLMs, even for our synthetic dataset, contradicting recent prior work. This can be attributed to the difference between the tasks we study. The tasks that receive significant benefits from using explanations in Nye et al. (2021) and Wei et al. (2022) are all program-like (e.g., integer addition and program execution), whereas the tasks in this work emphasize textual reasoning grounded in provided inputs. In fact, in Wei et al. (2022) and Chowdhery et al. (2022), explanations only show mild benefit on open-domain QA tasks like StrategyQA (Geva et al., 2021) that are closer to our setting.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Prompting LLMs with explanations and having models generate them may not guarantee higher performance on our tasks. But what about the quality of the model-generated explanations themselves?
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We assess the reliability of the explanations for the three datasets, measured in terms of two aspects.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Factuality refers to whether a generated explanation is faithfully grounded in the corresponding input context (context for QA and premise/hypothesis pair for NLI). A factual explanation should not contain hallucinations that contradict the context. See Figure 3 for a nonfactual explanation.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Consistency measures if the explanation entails the prediction. Our concept of consistency resembles plausibility as described in Jacovi and Goldberg (2021), in that we assess whether the prediction follows from the explanation as perceived by a human. See Figure 3 for an inconsistent explanation.  Figure 4: Explanations are more likely to be nonfactual than to be inconsistent, and a nonfactual explanation usually indicates an incorrect prediction.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
For SYNTH, we use rules to automatically judge whether an explanation is factual and consistent on all four LLMs. For ADVHOTPOT and E-SNLI, the authors manually inspected the explanations generated by InstructGPT and annotated them for these two characteristics (more details in Appendix D). Note for each setting, the results are based on the explanations and predictions obtained with a single set of training shots. We only show the results of P-E on E-SNLI, as E-P is substantially worse here.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We summarize the results in Table 2. We only report consistency on E-SNLI, as the explanations for E-SNLI often require some external commonsense knowledge which cannot be easily grounded in the inputs or judged as true or false (examples in Appendix F). The results suggest a disconnect between the model predictions and the "reasoning" in explanations. On InstructGPT, though using explanations improves its performance across three tasks, the generated explanations are unreliable (upper section), even for the straightforward synthetic setting. Comparing the factuality of explanations for SYNTH generated by GPT-3, InstructGPT, and text-davinci-002, we see that instruction tuning improves the factuality, but even the most powerful text-davinci-002 still fails to generate explanations that are perfectly grounded in the input context. Overall, LLMs tend to generate consistent explanations (>80% for all three datasets with the right prompt structure), but the explanations are less likely to be factual, which is concerning as they can deceive a user of the system into believing the model's answer.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
LLMs may hallucinate problematic explanations, but this could actually be advantageous if it gives us a way of spotting when the model's "reasoning" has failed. We investigate the connection between the reliability of an explanation and the accuracy of a prediction and ask whether a reliable explanation indicates an accurate prediction. (This resembles the linguistic calibration of Mielke et al. (2022), but using a different signal for calibration.)
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
As shown in Table 2 (right), accuracy and factuality/consistency are typically correlated, especially factuality. By knowing whether an explanation is factual, we can guess the model's accuracy a high fraction of the time (Accuracy = Factuality). A nonfactual explanation very likely means an incorrect prediction on the SYNTH dataset across all four LLMs. On ADVHOTPOT, factuality and Instruct-GPT's prediction correspond 80.0% of the time, substantially surpassing the prediction accuracy itself. We show fractions of correct and incorrect predictions when the explanations are factual/nonfactual and consistent/inconsistent in Figure 4 for two of our settings. Factual explanations are much more likely paired with correct predictions compared to nonfactual explanations. Consistency is also connected to accuracy but is an inferior indicator compared to factuality in general ( Table 2).
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We first show how post-hoc calibration functions in the controlled SYNTH setting, where we can simply check the factuality of an explanation. Since the generated explanation always follows the format "B is [profession] and A [verb] B." (example in Figure 2), we can split the explanation into two sentences. The explanation is factual if and only if each of the two sentences exactly matches one of the sentences in the context.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We use the assessment to improve the performance of P-E for SYNTH, where a nonfactual explanation typically indicates an incorrect prediction. This gives us a way to reject presumably incorrect answers. Specifically, we iterate through the top 5 candidate answers (restricted by the API) given by InstructGPT and reject any answer-explanation pair if the explanation is nonfactual until we find a factual one. This procedure dramatically improves the accuracy from 52.4% to 74.8%. Note that this SYNTH dataset is a challenging task given its lack of reasoning shortcuts: for reference, neither ROBERTA  nor DEBERTA (He et al., 2021) finetuned with 16 examples can achieve an accuracy surpassing 50%. With the help of the explanations and the checking procedure, we can use InstructGPT to achieve strong results using few-shot learning.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Framework We now introduce the framework that can leverage the factuality assessment of an explanation to calibrate a prediction. Let p be the vector of predicted probabilities associated with each class label in NLI (or the probability score of predicted answer in QA). Let v be a scalar value extracted from the explanation to describe the factuality. Then, we can adjust the probabilities accordingly using a linear model:
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Our calibration framework is extended from classical calibration methods (Platt, 1999;Guo et al., 2017;Zhao et al., 2021), which apply an affine transformation on the probabilities alone:p = softmax(W p + b). In contrast, we use an additional factor v in calibration to incorporate the factuality assessment of the explanation.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
There are a small number of parameters (W and b) that need to be trained in such a calibration framework. We will rely on a few more examples in addition to the shots we use in the prompt to train the calibrator. Specifically, we use the prompt examples to generate the predictions and explanations for these extra examples, and extract predicted probabilities, factors, and target probabilities triples to construct training data points used to train the calibrator. Note this procedure requires no explanation annotations for the extra examples.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Approximating Factuality We approximate the factuality using lexical overlap between the explanations and the inputs, which we found to work fairly well for our tasks.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
ADVHOTPOT: We use an explanation consisting of two sentences (examples in Figure 3) as an illustration. Let E = (E (1) , E (2) ) be the generated explanation, where E (1) and E (2) are the two sentences, and the E (i) = (e 1 , e 2 , · · · ) contain tokens e 1 , e 2 , · · · . Similarly, let P = (P (1) , P (2) , P (3) , P (4) ) be the context paragraphs, and P (i) = (p 1 , p 2 , · · · ) be the tokens. The factuality estimation of one explanation sentence E (i) is defined as: V(E (i) ) = max P ∈P
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Intuitively, the factuality score for a sentence E is defined as the maximum number of overlapping tokens over all paragraphs P , normalized by the number of tokens in E. We then define the factuality score for the whole explanation as V(E) = min E∈E V(E), as it requires all sentences to be factual in order to make the entire explanation factual. 8 E-SNLI: The explanations of E-SNLI do not really involve a concept of factuality. Nevertheless, we use an analogous score following the same principle by viewing the premise as the context. Let E = (e 1 , e 2 , · · · ) be the explanation and P = (p 1 , p 2 , · · · ) be the premise. We simply score the explanation by V(E) = |E|∩|P | |E| . The more an explanation overlaps with the premise, the more factual we judge it to be. Setup For E-SNLI, we use calibration methods to postprocess the final probabilities. Unlike classical temperature scaling (Platt, 1999), note that the methods we use here can actually change the prediction; we will therefore evaluate on accuracy of the calibrated model.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We study the effectiveness of our explanationbased calibrator under different training data sizes varying from 32 to 128. Recall that we only require explanation annotations for 32 data points, and only need the labels for the rest to train the calibrator. For E-SNLI, we calibrate P-E, which is shown to be more effective than E-P in this setting (Section 2.4).
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Baselines We provide the performance of finetuned ROBERTA  model as a reference, finding this to work better than De-BERTa (He et al., 2021). To isolate the effectiveness of using explanations for calibration, we introduce three additional baselines using nonexplanation-based calibrators. We apply the probability-based calibrator as described in Section 4.2 on the results obtained on few-shot learning (FEW-SHOT+PROBCAL) and predict-then-explain pipeline (P-E+PROBCAL). We note that the parameters of these calibrators are trained using the additional data points, as opposed to being heuristically determined as in Zhao et al. (2021). Furthermore, we experiment with a recently proposed supervised calibrator from , which uses the CLS representations from an additional language model as features in the calibrator. The probabilities are tuned usingp = softmax(W [p; h] + b), where h is the CLS representation. Since we do not have access to the embeddings obtained by GPT-3, we use ROBERTA to extract the vectors instead. We use such a calibrator on top of our best-performing base model, P-E, resulting P-E+ ZHANG ET AL. (2021).
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Limited by the maximum prompt length, in-context learning is not able to take as input the additional data used for training the calibrator. For a fair comparison, we can allow the in-context model to use this data by varying the prompts across test examples, dynamically choosing the prompt examples to maximize performance. Choosing closer data points for prompting is a common and effective way of scaling up the training data size for in-context learning (Shin et al., 2021;. Following , we test the performance of choosing nearest neighbors for the prompt based on CLS embedding produced by a ROBERTA model , referred as FEW-SHOT(NN). It is worth clarifying that the FEW-SHOT and FEW-SHOT+PROBCAL approaches use the same set of 32 training shots in the prompt for every test example, whereas the shot sets vary from example to example in FEW-SHOT(NN).
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We show the results in Table 3. We use 5 different groups of training examples and report the mean and standard deviation across the groups. For FEW-SHOT(NN), we only report the results obtained using 128 examples; results using a smaller number of examples will be worse than this.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Under 128 training examples, applying a trained calibrator on top of prompting with explanation (i.e., P-E+EXPLCAL) achieves the best accuracy of 68.5%, which is 12% higher than the performance of the vanilla uncalibrated few-shot in-context learning (FEW-SHOT). P-E+EXPLCAL also outperforms FEW-SHOT+PROBCAL and P-E+PROBCAL by 5% and 3%, respectively. Using explanations is more effective than using probabilities alone. In addition, P-E+EXPLCAL also outperforms P-E+ZHANG ET AL. (2021), whose performance is on par with P-E+PROBCAL. This suggests the additional CLS information is not very helpful in this setting.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
As the data size increases from 32 to 128, the performance of the explanation-based calibrator keeps improving notably, whereas the performance of probability-based calibrators nearly saturates at a data size of 96. The performance of FEW-SHOT(NN) with 128 training instances only improves the performance by 3.3%, compared to FEW-SHOT with 32 training instances. Choosing nearest  neighbors as the shots, while being effective when having access to a large amount of data, is not helpful in the extreme data-scarce regime. Calibrating using explanations is an effective way of using a few extra data points that cannot fit in the prompt, which is a pitfall of standard in-context learning.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Finally, ROBERTA finetuned using 128 shots only achieves an accuracy of 54.9%, lagging the performance of GPT-3 based models. The limited training data size is insufficient for finetuning smaller language models like ROBERTA, but is sufficient for P-E+EXPLCAL to be effective.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Setup For the ADVHOTPOT dataset, our calibration takes the form of tuning the confidence scores of the predicted answers to better align them with the correctness of predictions. These confidence scores can be used in a "selective QA" setting (Kamath et al., 2020), where the model can abstain on a certain fraction of questions where it assigns low confidence to its answers. We use the area under coverage-accuracy curve (AUC) to evaluate how well a model is calibrated as in past literature (Kamath et al., 2020;Garg and Moschitti, 2021;Ye and Durrett, 2022). The curve plots the average accuracy with varying fractions (coverage) of questions being answered (examples in Figure 5). For any given coverage, a better calibrated model should be able to identify questions that it performs best on, hence resulting a higher AUC.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We experiment with training data set sizes of 6, 32, and 64. We report the results averaged from 5 trials using different training sets. For ADVHOTPOT, we calibrate E-P, which is shown to be more effective than P-E in this setting (Section 2.4). Our approach is also effective for calibrating P-E; please refer to Appendix E for details.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We show the AUC scores in Table 4. By leveraging explanations, E-P+EXPLCAL successfully achieves an AUC of 68.8, surpassing both FEW-SHOT by 7 points and E-P by 4 points. We note this is a substantial improvement, given that the upperbound of AUC is constrained by the accuracy of the answers and cannot reach 100. Figure 5 shows the coverage-accuracy curves of various methods averaged across the 5 training runs. E-P+EXPLCAL always achieves a higher accuracy than its uncalibrated counterpart, E-P, under a certain coverage, and the gap is especially large in the most confident intervals (coverage < 50%). E-P+ZHANG ET AL. (2021) is able to calibrate the predictions on this dataset, but still lags our explanation-based calibrator, E-P+EXPLCAL.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
In addition, the explanation-based calibrator can be effective with as few as 32 examples. This is because there are only two parameters (the probability of predicted answer and the explanationbased factor) in the calibrator, which can be easily learned in this few-shot setting. Comparing E-P+EXPLCAL against FEW-SHOT(NN), using nearest neighbors in the prompt is also able to improve the performance compared to using a fixed set of shots (FEW-SHOT), yet our lightweight calibrator can better utilize such a small amount of data, and learn to distinguish more accurate predictions based on the explanations.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Caveats and Risks of Explanations from Large Language Models Our analysis suggests that LLMs' internal "reasoning" does not always align with explanations that it generates, as shown by our consistency results. More concerning, the explanations might not be factually grounded in the provided prompt. This shortcoming should caution against any deployment of this technology in practice: because the explanations are grammatical English and look very convincing, they may deceive users into believing the system's responses even when those responses are incorrect. Section 6 of Bender et al. (2021) discusses these risks in additional detail. The fact that language models can hallucinate explanations is also found in other work (Zhou and Tan, 2021). This result is unsurprising in some sense: without sufficient supervision or grounding, language models do not learn meaning as distinct from form (Bender and Koller, 2020), so we should not expect their explanations to be strongly grounded.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We have shown that even explanations which don't lead to accuracy gains can still be useful for calibration. However, the lexical overlap feature we use here is a weak signal of explanation correctness (see the example in Figure 1). Strong enough entailment models should theoretically be able to perform this task and work across a range of tasks without fine-tuning. This explanation assessment model can even be a language model itself trained for this particular propose to approach the verification tasks for a given domain by in-context learning.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Conclusion We have explored the capabilities of LLMs in using explanations in in-context learning for textual reasoning. Through our experiments with four LLMs and on two QA datasets and an NLI dataset, we find that simply including explanations in the prompt does not always improve the performance of in-context learning. Our manual analysis demonstrates that LLMs tend to generate nonfactual explanations when making wrong predictions, which can be a useful leverage to assess the correctness of the predictions. Lastly, we showcase how to use explanations to build lightweight calibrators, which successfully improve InstructGPT's in-context learning performance across all three datasets.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We show examples of the prompts used for SYNTH, ADVHOTPOT, and E-SNLI in Figure 6, Figure 7, and Figure 8, respectively. Our prompts follow the original formats in Brown et al. (2020). For approaches that use explanations (E-P and P-E), we insert explanations before/after with necessary conjunction words.  
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We create a controlled synthetic multi-hop QA dataset. Each context consists of four reasoning chains, where each chain contains two sentences following a template: "A [verb] B. B is [profession].". We fill in A and B in the reasoning chain templates using randomly selected names from a pool of 50 names. To fill in the [verb] and [profession] in the four reasoning chain templates, we first select two verbs from a pool of 30 verbs and two professions from a pool of 30 professions. Next, we fill in the four chains using the combination of these two verbs and professions, which give a set of completely symmetric chains. Finally, we sample one reasoning chain from all of the four to derive a asking: "Who [verb] [profession]?" (example in Figure 2).
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Such a design ensures there are no reasoning shortcuts (Chen and Durrett, 2019), making it a difficult dataset even despite the regular structure of the task. A ROBERTA model needs roughly 500 data points to tackle this problem and achieve near 100% accuracy on the test set.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We observed a Cohen's Kappa of 0.85 between the factuality annotations of 100 examples (obtained using the E-P paradigm) on which the annotations of the authors overlapped.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
E Calibrating P-E on ADVHOTPOT We also experiment with calibrating predictions obtained using P-E on ADVHOTPOT, following the same setup as described in Section 4.4.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
As in Table 5, using explanations is also effective for calibrating the predictions obtained using P-E. P-E+EXPLCAL successfully improves the AUC of P-E from 58.4 to 62.6, outperforming FEW-SHOT and FEW-SHOT (NN).
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
SYNTH: E-P Nonfactual Jason helps Christopher. Kevin helps Stephanie. James punches Jonathan. Daniel is a typist. Jonathan is a tailor. Christopher is a typist. Richard punches Daniel. Stephanie is a tailor. Q: Who punches a typist? A: Because Jonathan is a typist and James punches Jonathan, the answer is James.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Matthew blames Tiffany. Lisa is a chef. Christopher helps Kelly. Angela helps Jessica. Rachel blames Lisa. Jessica is a farmer. Kelly is a chef. Tiffany is a farmer. Q: Who helps a farmer? A: Because Kelly is a farmer and Christopher helps Kelly, the answer is Christopher.  Figure 9: Examples of failure cases on SYNTH. GPT-3 almost always generates nonfactual explanations when giving an incorrect answer, using either P-E or E-P.  
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Our questions follow the template Who V1 P1?. Our generated explanations follow the template N1 is P2 and N2 V2 N3. Our answers are always a name, e.g., N4. Because large language models almost always produce well-formed explanations, we can match the output against these patterns and extract variables V 1 , P 1 , etc. using simple regular expressions.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We say that an explanation is consistent if and only if the following conditions are satisfied: (1) N2 = N4 and N1 = N3. (2) P2 = P1 and V2 = V1. These rules ensure the explanation matches the intent of the question and entails the answer at the same time.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We say an explanation is factual if and only if both N1 is P2 and N2 V2 N3 appear exactly in the context. We also experimented with using an alternative style of explanations for SYNTH, where we reversed the order of the two sentences in the explanations shown in We show the performance of using reversed explanations in Table 6 and the reliability in Table 7. In general, this alternative style of explanations yields inferior performance compared to the original style (Table 1). Using explanations leads to no improvements on GPT-3, and InstructGPT. P-E is consistently better than E-P across GPT-3, InstructGPT, and text-davinci-002.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
Furthermore, using such a reversed style, language models almost always generate consistent explanations when being prompted in either E-P or P-E paradigm. The factuality almost always indicates the correctness of predictions.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We believe these two prompts cover the most natural explanation styles for this problem. While small format changes or modifications to the general QA prompt format are also possible, we observed these to have minor impacts on the results (as we see in Appendix I).
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We test whether including a trigger for multi-step reasoning can help LLMs better learn from explanations in the prompt for multi-step reasoning. Following Kojima et al. (2022), we prepend "Let's think step by step." to the exemplar explanations used in the E-P paradigm. For this experiment, we only test on SYNTH and ADVHOTPOT, which involve multi-step reasoning. We do not experiment with text-davinci-002, which has already achieved substantial performance improvement from using explanations, and we omit OPT because its performance is too low.  58.0 6.2
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
As shown in Table 8, adding triggers in the prompts does not lead to statistically significantly improvements in E-P for GPT-3 and InstructGPT. In fact, it typically causes a performance degradation.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
The cost of our experiments, described as follows, is estimated based on using the GPT-3 API with the largest models available (davinci, text-davinci-001, and text-davinci-002) as of August 2022 ($0.06 per 1,000 tokens). The setting in Table 1 uses 250 examples for each result, with roughly 1400 tokens per example using the FEW-SHOT paradigm and 2000 tokens per example using the E-P or E-P paradigm. The cost of evaluating FEW-SHOT, P-E, and E-P for 5 trials on a single dataset is roughly $105, $150, and $150, respectively. The total price for reproducing results on three datasets as in Table 1 using a single language model is roughly $1200.
The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
We subsample 250-example sets to reduce cost rather than running on full datasets. Based on the significance tests in this paper and the reported confidence intervals, this size dataset is sufficient to distinguish between the performance of different approaches.
Measuring Association Between Labels and Free-Text Rationales
Interpretable NLP aims to better understand predictive models' internals for purposes such as debugging, validating safety before deployment, or revealing unintended biases and behavior (Molnar, 2019). These objectives require faithful rationalesexplanations of the model's behavior that are accurate representations of its decision process (Melis and Jaakkola, 2018).
Measuring Association Between Labels and Free-Text Rationales
One way towards faithfulness is to introduce architectural modifications or constraints that produce rationales with desirable properties (Andreas et al., 2016;Schwartz et al., 2018;Jiang et al., 2019, inter alia). For example, pipeline models ( Figure 2) were designed for information extraction (IE) tasks Figure 1: A categorization of interpretable NLP on an illustrative faithfulness spectrum. Two predominant forms of explanation exist that align with two predominant classes of NLP tasks. Unlike models for IE tasks, the desirable properties of interpretable models for reasoning tasks have not been explored. We investigate architectures and tests for explaining reasoning tasks.
Measuring Association Between Labels and Free-Text Rationales
for which a rationale can be extracted as a subset of the input and is sufficient to make a prediction on its own, without the rest of the input (Lei et al., 2016). Such models approach faithfulness by construction (Jain et al., 2020).
Measuring Association Between Labels and Free-Text Rationales
There is a growing interest in tasks that require world and commonsense "knowledge" and "reasoning", such as commonsense question-answering (CommonsenseQA; Talmor et al., 2019) and natural language inference (SNLI; Bowman et al., 2015). Here, extractive rationales necessarily fall short-rationales must instead take the form of free-text natural language to fill in the reasoning or knowledge gap (Camburu et al., 2018;Rajani et al., 2019). 2 In Table 1, for example, the highlighted extractive rationale of the first problem instance lacks at least one reasoning step to adequately justify the answer; the natural language rationale (which is not extractive) fills in the gap.
Measuring Association Between Labels and Free-Text Rationales
Question: While eating a hamburger with friends , what are people trying to do? Answer choices: have fun, tasty, or indigestion Natural language rationale: Usually a hamburger with friends indicates a good time.
Measuring Association Between Labels and Free-Text Rationales
Natural Language Inference (E-SNLI) Premise: A child in a yellow plastic safety swing is laughing as a dark-haired woman stands behind her. Hypothesis: A young mother is playing with her daughter in a swing. Label choices: neutral, entailment, or contradiction Natural language rationale: Child does not imply daughter and woman does not imply mother. Table 1: Examples from the CoS-E v1.0 and E-SNLI datasets ( §2). Extractive rationales annotated by humans are highlighted, while human-written free-text rationales are presented underneath the answer/label choices. These examples illustrate that the extractive rationales fail to adequately explain the correct (underlined) label.
Measuring Association Between Labels and Free-Text Rationales
We study two distinct model classes: selfrationalizing models, which are fully differentiable and jointly predict the task output with the rationale; and pipelines, which rationalize first and then predict task output with a separate model. We first show that, for CommonsenseQA and SNLI, a selfrationalizing model provides rationales that better indicate the correct label than a pipeline ( §3.1). Next, we show that sufficiency is not universally applicable: a natural language rationale on its own does not generally provide enough information to arrive at the correct answer ( §3.2). These findings suggest that a faithful-by-construction pipeline is not an ideal approach for reasoning tasks, leading us to ask: is there is a way to achieve faithful freetext rationalization with self-rationalizing models?
Measuring Association Between Labels and Free-Text Rationales
We note that there is currently no way to assess the relationship between a prediction and a freetext rationale within the same fully differentiable model. Jacovi and Goldberg (2020) argue for the development of evaluations that measure the extent and likelihood that a rationale is faithful in practice (illustrated in Figure 1). To do so, we propose two measurements to initiate testing the extent to which predicted labels and explanations are associated within the model that produces them.
Measuring Association Between Labels and Free-Text Rationales
The first experiment, robustness equivalence ( §4.1), analyzes whether a predicted label and generated rationale are similarly robust to noise. The second, feature importance agreement ( §4.2), analyzes whether the gradient-attributions of the input with respect to the predicted label are similar to those with respect to the predicted rationale. We show that a self-rationalizing finetuned variant of T5 (Raffel et al., 2020;Narang et al., 2020) demonstrates good robustness equivalence and feature importance agreement on the datasets investigated. This result motivates future work on more measurements for testing label-rationale association.
Measuring Association Between Labels and Free-Text Rationales
Before we turn to our analyses we introduce datasets and models used for our experiments.
Measuring Association Between Labels and Free-Text Rationales
We explore two large-scale datasets for textual reasoning tasks that contain human-written natural language rationales: E-SNLI (Camburu et al., 2018), an extension of SNLI (Bowman et al., 2015);and CoS-E (Rajani et al., 2019), an extension of CommonsenseQA (Talmor et al., 2019) (both in English). For the former, the task is to infer whether a given hypothesis sentence entails, contradicts, or is neutral towards a premise sentence. For the latter, the task is to select the correct answer from 3 (v1.0) or 5 (v1.11) answer choices for a question. We use both versions of CoS-E in our experiments (see Appendix A.2). Table 1 contains examples and Table 7 (Appendix A.2) data statistics. 3 T5 Models All of the models in this work are based on T5, though our methods can in principle be applied to any architecture. The base version of T5 is a 220M-parameter transformer encoderdecoder (Vaswani et al., 2017). To carry out supervised finetuning, T5 is trained by maximizing the conditional likelihood of the correct text output (from annotated data), given the text input.
Measuring Association Between Labels and Free-Text Rationales
We finetune five T5-Base models for each dataset, supervising with ground-truth labels and rationales (further details in Appendix A.3-A.4):
Measuring Association Between Labels and Free-Text Rationales
• I→R, which maps task inputs to rationales, without ever being exposed to task outputs. • R→O, which maps rationales to task outputs.
Measuring Association Between Labels and Free-Text Rationales
The only input elements this model is exposed to are answer choices (for CoS-E). • I→OR, which maps inputs to outputs and rationales.  Table 2: An overview of text-only datasets and rationale types (E for extractive, NL for natural language rationales) used in prior work on pipeline architectures. We focus on the two tasks we believe require a more complex notion of "reasoning" to solve: CommonsenseQA (CQA) and NLI. Unlike the other tasks in the table, prior work for rationalizing these two tasks lacks consensus on (1) the type of rationales best-suited, and (2) the form of the model for these tasks. We argue for natural language rationales, and demonstrate that pipeline models are poorlysuited for CQA and SNLI given this choice. Dataset citations: Appendix A.1. • IR→O, which maps pairs of inputs and rationales to outputs. • I→O, which maps inputs to outputs. We provide input-output formatting in Table 8 (Appendix A.3). Using these building blocks, we can instantiate two important approaches.
Measuring Association Between Labels and Free-Text Rationales
Pipeline Model (I→R;R→O) This architecture composes I→R with R→O, each of which is trained entirely separately, for a total of 440M parameters. It is illustrated in Figure 2 and is faithfulby-construction (with caveats; see Jacovi and Goldberg, 2021). The vast majority of prior work using pipelines has focused only on extractive rationales (see Table 2).
Measuring Association Between Labels and Free-Text Rationales
Self-Rationalizing Model (I→OR) A joint, self-rationalizing model (Melis and Jaakkola, 2018), illustrated in Figure 3, predicts both a label and rationale. This is the most common approach to free-text rationalization (Hendricks et al., 2016;Kim et al., 2018;Hancock et al., 2018;Camburu et al., 2018;Ehsan et al., 2018;Liu et al., 2019a;Wu and Mooney, 2019;Narang et al., 2020;Do et al., 2020;Tang et al., 2020), but little is understood about model internals. I→OR models are desirable for their ease-of-use, task-effectiveness, parameter efficiency, and their ability to generate fluent and plausible rationales. We expect models of this kind to play an important role in continuing research on explainable AI for these reasons.
Measuring Association Between Labels and Free-Text Rationales
We use the I→OR variant of T5 (Narang et al., 2020). Because only one instance of T5 is used to instantiate it, the total number of parameters is half that of the pipeline. We replicate two prior findings (Tables 9-10 in Appendix B): the T5 pipeline does not perform as well as the self-rationalizing model (despite having double the parameters), and T5-Base outperforms pretrained models used in prior work.
Measuring Association Between Labels and Free-Text Rationales
Evaluation We do not report BLEU scores (Papineni et al., 2002), because BLEU and related metrics do not measure plausibility (Camburu et al., 2018;Kayser et al., 2021;Clinciu et al., 2021) or faithfulness (Jacovi and Goldberg, 2020). In addition to low correlation with human scores, there can be many valid rationales for a given instance (Miller, 2019); metrics that compare generated rationales to a single ground-truth do not address this and are thus a poor measure of quality.  Human simulatability (Doshi-Velez and Kim, 2017) has a rich history in machine learning interpretability as a reliable measure of rationale quality from the lens of utility to an end-user (Kim et al., 2016;Chandrasekaran et al., 2018;Hase and Bansal, 2020;Yeung et al., 2020;Poursabzi-Sangdeh et al., 2021;Rajagopal et al., 2021, i.a.). Rather than computing word-level overlap with a ground-truth explanation, simulatability measures the additional predictive ability a rationale provides over the input, computed as the difference between task performance when a rationale is given as input vs. when it is not (IR→O minus I→O). Historically, humans have served as the predictors, but recent work has shown that the computation of simulatability can be automated using trained models. Hase et al. (2020) demonstrate that automated metrics for simulatability have moderate to high correlation with human scores in both an expert and a crowdsourced setting. 4 We thus use simulatability score as a measure of rationale quality.
Measuring Association Between Labels and Free-Text Rationales
We first analyze "faithful-by-construction" pipeline models (I→R;R→O) for free-text rationalization with respect to two properties: quality of generated rationales ( §3.1) and appropriateness of the sufficiency assumption ( §3.2).
Measuring Association Between Labels and Free-Text Rationales
Rationales should be a function of the input and the predicted label. To demonstrate why this is the case, consider training an I→R model on a dataset with multiple annotation layers, e.g., OntoNotes, that contains word sense, predicate structure, and coref-4 Given the large scale of our analysis (>250K instances evaluated), an automated metric provides coverage, reproducibility and consistency not achievable with human annotation. An author of this paper annotated 60 instances from both E-SNLI and CoS-E v1.11, and found 82.5% agreement between their simulatability score and the automated metric. 19.09 -12.77 -6.06 Table 4: Simulatability scores (higher is better) of ground-truth rationales (R * ) and rationales generated from two model architectures: I→OR and I→R. These results demonstrate that rationales generated as a function of the input and the predicted label (I→OR) are higher quality than those generated as a function of the input alone (I→R) across datasets ( §3.1).
Measuring Association Between Labels and Free-Text Rationales
erence (Pradhan et al., 2007). Without additional task-specific input, this model would produce the same rationale, regardless of the task being rationalized. Prior work has also critiqued I→R;R→O models because it is counter-intuitive to generate a rationale before deciding the label to explain (Kumar and Talukdar, 2020;Jacovi and Goldberg, 2021). Therefore, the I→R model will first need to implicitly predict a label. But can I→R infer the label well, when it is trained without label signal? To address this question, we study whether I→OR rationales are better at predicting the gold labels than I→R rationales. We train a R→O model on ground-truth rationales (R * ), and evaluate on the following inputs:
Measuring Association Between Labels and Free-Text Rationales
• test set ground-truth R * rationales, • test set rationales generated by I→OR, and • test set rationales generated by I→R.
Measuring Association Between Labels and Free-Text Rationales
In Table 2, we show that I→OR rationales recover 8-9% more ground-truth (R * ) performance than R→O rationales on both versions of CoS-E, and 1% on E-SNLI. A smaller improvement for E-SNLI could be explained by the fact that E-SNLI has substantially more training examples for each label than CoS-E, which helps a pipeline model learn features predictive of each label. 5 We additionally demonstrate I→OR rationales are higher quality than R→O's, as measured by the simulatability metric ( Table 4). The fact that the pipeline's strong performance does not generalize to a complex prediction task such as CoS-E empirically demonstrates that training on label signal O is important to generate good-quality rationales and avoid cascading errors. Figure 3: An example of a joint architecture (I→OR; §2) for CoS-E v1.0 with a human-written rationale. Trained on both task signal and human rationales, these models are effective at generating fluent rationales while retaining good task performance (Table 9-10 in Appendix B).  Table 5: A comparison of the IR→O and R→O models ( §2) evaluated with ground-truth natural language rationales (R * ). In some cases accuracy improves substantially with the addition of the input, indicating that rationales are not always sufficient and pipelines are not always effective.
Measuring Association Between Labels and Free-Text Rationales
"Faithful-by-construction" pipelines rely on the sufficiency assumption: the selected rationale must be sufficient to make the prediction without the remaining input. This assumption is suitable for IE tasks for which a subset of the input tokens is predictive of the label. Indeed, humans can serve as R→O models on certain IE tasks and make accurate predictions, validating that rationales are sufficient for these tasks (Jain et al., 2020).
Measuring Association Between Labels and Free-Text Rationales
To illustrate why sufficiency might not be justified for reasoning tasks, consider the example in Figure 2. The task of the R→O model is to select between the answer choices "have fun", "tasty", and "indigestion" given the rationale "Usually a hamburger with friends indicates a good time". The rationale is designed to complement the input question, but the R→O model does not see the question, changing the fundamental nature of the task it is solving. We thus wonder: does task obfuscation hurt pipelines' ability to perform the task?
Measuring Association Between Labels and Free-Text Rationales
We report the accuracy difference between a R→O model and a model that receives both the input and rationale (IR→O), both trained on R * . We evaluate on test set R * . 6 In Table 5, the IR→O models on CoS-E have a 5-12% increase in accuracy over R→O, indicating that the rationales are not sufficient. The difference is much smaller for E-SNLI (1%), likely due to the fact that E-SNLI was collected by instructing annotators to provide 6 Evaluating on R * instead of generated rationales serves as an upper-bound on pipeline performance, removing the confounding factor that I→R rationales can be poor ( §3.1). self-contained rationales. However, using datasetcollection to explicitly collect sufficient rationales does not address the unnaturalness of such a task formulation (Wiegreffe and Marasović, 2021). 7 Table 5 indicates that (especially) in the case of CoS-E, sufficiency is not a valid assumption, and the use of I→R;R→O models is sub-optimal in these cases.
Measuring Association Between Labels and Free-Text Rationales
So far, we have highlighted shortcomings of pipelines for reasoning tasks:
Measuring Association Between Labels and Free-Text Rationales
• cascading errors caused by low-quality rationales that are not indicative of labels ( §3.1), • missing information due to rationales not being sufficient ( §3.2), • double the number of parameters and more manual labor needed to reach comparable performance to an end-to-end (I→O) model; still often performing worse ( §2). We next turn our focus to self-rationalizing (I→OR) models currently in widespread use, which in contrast to pipelines are high-performing, easy to implement via a multi-task loss, and more parameter-efficient ( §2).
Measuring Association Between Labels and Free-Text Rationales
Despite their popularity and widespread use, the extent to which self-rationalizing models exhibit faithful rationalization has not been studied. To illustrate this point, we reference Narang et al.  models cannot be treated as faithful explanations without further investigation. At minimum, rationales must be implicitly or explicitly tied to the model's prediction. We present two metrics to analyze the association between the mechanisms that produce labels and rationales in a multi-task, I→OR model: robustness equivalence ( §4.1) and feature importance agreement ( §4.2). These experiments serve as a necessary sanity check for the reliability of I→OR models' explanations.
Measuring Association Between Labels and Free-Text Rationales
We aim to analyze whether predicted labels and rationales are similarly or dissimilarly robust to noise applied to the input. The former indicates predicted labels and rationales are strongly associated, while the latter indicates the opposite. Given some amount of noise, there are four possible cases for a model's output: {l stable , l unstable } × {r stable , r unstable }, where l is a label and r is a rationale.
Measuring Association Between Labels and Free-Text Rationales
The case where l and r are both stable or both unstable indicates that both tasks are similarly affected by noise. The case where l is unstable but r is stable (or vice versa) is a failure case-if only one output is stable, we conclude the two generation mechanisms cannot be strongly associated within the model.
Measuring Association Between Labels and Free-Text Rationales
Method Following related work Lakshmi Narayan et al., 2019;Liu et al., 2019b), we add zero-mean Gaussian noise N (0, σ 2 ) to each input embedding in the I→OR encoder at inference time. We measure changes in label prediction as the number of predicted test set labels that flip, i.e., change from their original prediction to something else, alongside changes in accuracy of the I→OR model. We measure changes in rationale quality using simulatability ( §2). We illustrate details of the simulatability calculation in Figure 6a. We report metrics on rationales generated by I→OR under different levels of noise, controlled by σ 2 .
Measuring Association Between Labels and Free-Text Rationales
An example of noisy outputs for the running CoS-E v1.0 example is presented in Table 6.
Measuring Association Between Labels and Free-Text Rationales
Results We present results on the effect of noise on labels in Figure 4 (E-SNLI and CoS-E v1.11 in Figure 8 of Appendix B). As expected, the accuracy of the I→OR model (red line) and the percent of labels in the I→OR model which have not flipped (black line) are almost identical for all three datasets. We present results on the effect of noise on rationales in Figure 5 for CoS-E (E-SNLI in Figure 9 of Appendix B).
Measuring Association Between Labels and Free-Text Rationales
By examining the regions of largest slope, we gain insights into model behavior. On the rationale quality measure, both versions of CoS-E's rationales reach a minimum contribution to task accuracy at σ 2 = 20 ( Figure 5). We similarly observe the largest drop in task accuracy ( Figure 4) for CoS-E v1.0 between σ 2 = 15 to σ 2 = 20. 8 Thus, at lower noise levels (0-15), the model exhibits both stable labels and rationales, and at higher levels (20+), both unstable, indicating robustness equivalence. Similar conclusions can be reached for E-SNLI and CoS-E v1.11; we conclude that the I→OR model demonstrates high label-rationale association for all 3 datasets as measured by robustness equivalence.
Measuring Association Between Labels and Free-Text Rationales
We expect that the simulatability in Figure 5 does not monotonically decrease because as rationales continue to worsen in quality (see the example in Table 6), the IR→O model may ignore them completely and more closely emulate the σ 2 Predicted Output 0 have fun explanation: having fun is the only thing that people are trying to do. 5 have fun explanation: having fun is the only thing that people are trying to do. 10 have fun explanation: eating a hamburger with friends is fun. 15 have fun explanation: having fun is the only thing people are trying to do while eating a hamburger with friends. 20 <extra_id_0> a hamburger with friends<extra_id_1> are people trying to do? explanation: a hamburger is a hamburger. . . 25 "" is the only thing that is """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" 30 a indigestion: a indigestion: a indigestion: a indigestion: a indigestion: a indigestion: a indigestion: a indigestion: a indigestion:. . . 35 a """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" Table 6: Output of the I→OR model for the running CoS-E v1.0 example under differing noise levels. While the rationale changes from variance 0-15, it is still valid for the given (correct) predicted label. At a variance of 20 and beyond, the model fails to generate both the correct label and a valid rationale. The model's predictions for this instance exhibit robustness equivalence.   Table 6, at σ 2 = 30, the rationale provides signal for an incorrect answer choice ("indigestion") that does not exist at σ 2 = 35. Therefore, we consider rationales produced with σ larger than the value at which simulatability reaches the minimum as unstable (see further examples corroborating this in Appendix B, Tables 11-13).
Measuring Association Between Labels and Free-Text Rationales
If label prediction and rationale generation are associated, input tokens important for label prediction should be important for rationale generation and vice versa. We refer to this property as feature importance agreement. To measure to what extent I→OR models exhibit this property, we use gradient-based attribution (Baehrens et al., 2010;Simonyan et al., 2014) to identify tokens important for label prediction, and the Remove and Retrain (ROAR) occlusion method (Hooker et al., 2019) to analyze their impact on rationale generation (or vice versa).
Measuring Association Between Labels and Free-Text Rationales
Gradient Attribution For a predicted class p, gradient attribution is a function of the gradient of the predicted class' logit l p with respect to an input token embedding x (i) ∈ R d :
Measuring Association Between Labels and Free-Text Rationales
where the function f reduces the gradient to a scalar. Choices for f include L 1 or L 2 norm (Atanasova et al., 2020), or an element-wise sum (Wallace et al., 2019). Intuitively, the gradient measures how much an infinitesimally small change in the input changes the predicted class' logit, using a first-order Taylor series approximation of the logit function. Such methods have been extended to sequence-output models such as neural machine translation ( p } m k=1 with respect to the input:
Measuring Association Between Labels and Free-Text Rationales
The attribution of a sequence of n input token embeddings, X ∈ R n×d , is a vector a(X) = [a(x (1) ), . . . , a(x (n) )] ∈ R n , where a(x (i) ) is shorthand for the value defined in Equation 2. By decomposing the term in Equation 2 into two parts, we obtain two attribution vectors over the input tokens; one for the predicted label logits L, and one for the predicted rationale logits R in the decoded output:
Measuring Association Between Labels and Free-Text Rationales
a(X) = a(X)L + a(X)R.
Measuring Association Between Labels and Free-Text Rationales
Reliability of Gradient Attribution Before we measure feature importance agreement, it is critical to evaluate whether the gradient-attribution scores truly capture token importance, since these methods can be unreliable for certain datasets or architectures (Kindermans et al., 2019). To validate that our attributions are reliable, we perform the ROAR test (Hooker et al., 2019). Using attribution scores, we obtain the top-k% attributed tokens for every instance and occlude them following T5's pretraining procedure and mask tokens. We retrain a model on the occluded training set and evaluate on the occluded test set. We repeat this procedure for k ∈ {10%, 20%, 30%}, and compare the drop in performance as k increases to a baseline in which k% random tokens are dropped.
Measuring Association Between Labels and Free-Text Rationales
To the extent that the occluded model fails to match the random model's performance, we can attribute such degradation to the removal of tokens that the original model finds informative. A large drop in performance indicates that gradient attributions successfully identify important tokens in the input.
Measuring Association Between Labels and Free-Text Rationales
We first use this method to select an optimal gradient-attribution method and f function (Figure 11 in Appendix B). We find the L 1 norm of the embedding vector as f to outperform the element-wise sum (which may suffer from dampened magnitudes). Unlike prior work in computer vision (Hooker et al., 2019), we find raw gradients to perform comparably to the input*gradient variant (Shrikumar et al., 2017). Thus, we compute attributions in subsequent experiments following Equation 1 with f equal to the L 1 norm.
Measuring Association Between Labels and Free-Text Rationales
We validate that attributions from the label logits, a(X) L , degrade label accuracy when compared to random occlusion (orange vs. blue line in Figure 7, left). The two simulatability lines (Figure 7, right) for CoS-E v1.0 have an inflection point. We illustrate how simulatability is calculated in Figure 6b. Similar to §4.1, we expect this is due to rationales so noisy that IR→O ignores them and behaves like I→O. If an input attribution degrades rationale quality (as measured by simulatability) more than a random attribution, then the line corresponding to that attribution (for values of k for which neither that attribution nor the random attribution have reached the inflection point) has to be below the "random" line. For values of k for which both attributions have passed the inflection point, the "random" line should be below the attribution line, assuming that after this point, a noisier ratio-nale is more similar is IR→O to I→O and hence simulatability is closer to 0. Both criteria hold for attributions from the rationale logits, a(X) R (green vs. blue line in Figure 7, right) for CoS-E v1.0 and other datasets (see Figure 12 in Appendix B). This reliability check confirms that gradient-attribution works well in our setting.
Measuring Association Between Labels and Free-Text Rationales
To measure feature importance agreement-whether tokens important for label prediction are important for rationale generation (and vice versa)-we repeat the same experiment, but measure performance with respect to the other output's metric. For attributions computed from label logits, a(X) L , we measure the effect of their occlusion on rationale quality using simulatability score. For attributions with respect to rationale logits, a(X) R , we measure the effect of their occlusion on label accuracy. If at least one of these values is notably different from random, we can conclude that the I→OR model displays feature-importance similarity in a given direction.
Measuring Association Between Labels and Free-Text Rationales
Results for CoS-E v1.0 are once again in Figure 7 (and for other datasets in Figure 12 in Appendix B). In Figure 7 (left), we find that removing top-k% tokens by a(X) R magnitude degrades label performance compared to the baseline (green vs. blue line). Intuitively, this drop is less than token attributions by a(X) L magnitude (orange line). In Figure 7 (right), we observe that removing top-k% tokens by a(X) L consistently degrades rationale performance more than random according to the two criteria for comparing simulatability lines (orange vs. blue line). This also holds for E-SNLI and CoS-E v1.11 ( Figure 12). We conclude that the I→OR model demonstrates label-rationale association as measured by feature importance agreement for the datasets studied.
Measuring Association Between Labels and Free-Text Rationales
After demonstrating the weaknesses that pipeline models exhibit for free-text rationalization tasks, we propose two measurements of label-rationale association in self-rationalizing models. We find that on three free-text rationalization datasets for CommonsenseQA and SNLI, models based on T5 exhibit high robustness equivalence and feature importance agreement, demonstrating that they pass a necessary sanity check for generating faithful free-text rationales.
Measuring Association Between Labels and Free-Text Rationales
Future work can expand analysis to more properties. We believe this research direction to be important moving forward due to the advantages of large multi-task explanation models, and as a complement to development of interpretable architectures that can be fickle and task-specific. Although our measurements address only necessary and not sufficient properties, by viewing faithful interpretability as a spectrum, we make a step to quantitatively situate common models on it. 
Measuring Association Between Labels and Free-Text Rationales
A.1 Overview of Prior Work on Pipelines
Measuring Association Between Labels and Free-Text Rationales
In Table 2, we overview the datasets and types of rationales used in prior work on pipelines. The sources of datasets are: CommonsenseQA (Talmor et al., 2019), SNLI (Bowman et al., 2015), SST (Socher et al., 2013), AgNews (Del Corso et al., 2005), Evidence Inference (Lehman et al., 2019), Movie Reviews (Zaidan and Eisner, 2008), MultiRC (Khashabi et al., 2018), LGD (Linzen et al., 2016), 20 News (Lang, 1995, Amazon Reviews (McAuley and Leskovec, 2013), Beer Reviews (McAuley et al., 2012), BoolQ (Clark et al., 2019), FEVER (Thorne et al., 2018).
Measuring Association Between Labels and Free-Text Rationales
We summarize dataset statistics in Table 7. The two versions (v1.0, v1.11) of CoS-E correspond to the first and second versions of the Common-senseQA dataset. CoS-E v1.11 has some noise in its annotations (Narang et al., 2020). 9 This is our primary motivation for reporting on v1.0 as well, which we observe does not have these issues.
Measuring Association Between Labels and Free-Text Rationales
The T5 model (Raffel et al., 2020) is pretrained on a multi-task mixture of unsupervised and supervised tasks, including machine translation, question answering, abstractive summarization, and text classification. Its inputs and outputs to every task are text sequences; we provide the input-output formatting for training and decoding of our T5 models in Table 8. T5 can provide any word in the vocabulary as an answer.
Measuring Association Between Labels and Free-Text Rationales
We use Huggingface Datasets 10 to access all datasets, and Huggingface Transformers (Wolf et al., 2020) to access pretrained T5 weights and tokenizer. To optimize, we use Adam with = 1e-8, β 1 = 0.9, and β 2 = 0.99. We use gradient clipping to a maximum norm of 1.0 and a dropout rate of 0.1. We train each model on a NVIDIA RTX 8000 GPU (48GB memory) for maximum 200 epochs with a batch size of 64 and a learning rate linearly decaying from 5e-5. Training ends if the validation set loss has not decreased for 10 epochs. Early stopping occurs within 15 epochs for most models. Most CoS-E models train in less than 1 hour and most E-SNLI models in around 30. At inference time, we greedy-decode until an EOS token is generated (or for 200 tokens). Approximating the 64-batch model with a batch-size of 16 and 4 gradient accumulation steps on 8GB memory cloud GPUs, we sweep starting learning rates of 1e-2, 1e-3, 1e-4, 5e-5, and 1e-5. The two largest learning rates never result in good performance. Among the smallest three rates, performance across all model variants (I→R, I→OR, R→O, I→O, and IR→O) on E-SNLI and CoS-E v1.0 never varies by more than 1.58% accuracy or 0.34 BLEU.
Measuring Association Between Labels and Free-Text Rationales
Worst-case model performance under large noise values in the Robustness Equivalence experiments (Figures 4 and 8) reaches 0 rather than random accuracy due to structure of the models' output. The I→OR model is trained to produce a delimiter to distinguish the label from the rationale in a long string of output tokens. When it fails to produce the delimiter under high noise, we cannot delineate the label from the rationale in the output where multiple answer choices are often mentioned, so we mark the label as incorrect.
Measuring Association Between Labels and Free-Text Rationales
We provide additional results that supplement the main body of the paper:
Measuring Association Between Labels and Free-Text Rationales
• Table 9 presents results comparing the selfrationalizing T5 model to baselines. • Table 10 presents results comparing the selfrationalizing T5 model to its pipeline variant (from §2).      house explanation: a house is the only place that would have air conditioning. 10 house explanation: a house is the only place that would have air conditioning. 15 <extra_id_0> house explanation: a house is the only place that will have air conditioning. 20 <extra_id_0> movie theatre explanation: movie theatre is the only option that is not a movie. 911 911 911. . . 25 <extra_id_0> explain<extra_id_1> explain<extra_id_2> explain<extra_id_3> explain<extra_id_4> explain<extra_id_5> movie theatre<extra_id_6>. . . 30 house of house of house of house of house of house of house of house of house of house of house of house of house . . . 35 house of house of office office office office office office office office office office office office office office office office office office. . . Table 11: Noised output of the I→OR model for the CoS-E v1.0 example "A man wants air conditioning while we watches the game on Saturday, where will it likely be installed?" The correct answer is "house". σ 2 Predicted Output 0 stress explanation: a computer is used to communicate with a granddaughter. 5 stress explanation: a computer is used to communicate with a granddaughter. 10 stress explanation: a computer is used to talk to people. 15 stress explanation: a computer is used to talk to people. 20 <extra_id_0> is using a computer to<extra_id_1> to talk to<extra_id_2> is using a computer to talk to a person is using a computer to talk to a person. . . 25 <extra_id_0> answer: answer: answer: answer: answer: answer: answer: answer: answer: answer: answer: answer: answer:. . . 30 <extra_id_0> answer: answer: answer: answer: answer: answer: answer: answer: answer: answer: answer: answer: answer:. . . 35 office of the office of the office of the office of the office of the office of the office of the office of the office of the office of the office . . . Table 12: Noised output of the I→OR model for the CoS-E v1.0 example "If a person is using a computer to talk to their granddaughter, what might the computer cause for them?" The correct answer is "happiness". σ 2 Predicted Output 0 transfer of information explanation: transfer of information is the only option that would be appropriate when communicating with a boss. 5 transfer of information explanation: transfer of information is the only option that would be appropriate when communicating with a boss. 10 transfer of information explanation: transfer of information is the only option that would be appropriate when communicating with a boss. 15 transfer of information explanation: transfer of information is the only option that would be appropriate when communicating with my boss. 20 transfer of information: transfer of information is the only thing that is transfer of information. transfer of information is the only thing that is. . . 25 transfer of information: transfer of information is information. transfer of information is information. transfer of information is information.. . . 30 i believe that is the answer of the question. argument is the answer of the question. argument is the answer of the question. argument is the answer of the question.. . . 35 i can't handle the argument argument argument is the answer of the argument argument is the answer of the argument. . . Table 13: Noised output of the I→OR model for the CoS-E v1.0 example "When communicating with my boss, what should I do?". The correct answer is "transfer of information". Accuracy of the I→OR model (red) and % stable labels in the I→OR model (black) show that most changes take place in the 10-20 σ 2 range for CoS-E and 15-30 σ 2 for E-SNLI. See §4.1. Figure 9: Results of the rationale portion of the robustness equivalence test for E-SNLI.  Figures 2-3. The decoded label is "have fun" and generated rationale "having fun is the only thing that people are trying to do". Important input terms vary across the two loss terms. For example, the predicted label term assigns high importance to the predicted answer choice, "have fun", while the explanation attends more uniformly over the input with peaks on relevant entities and verbs such as "trying". In this example, the explanation-and label-attribution vectors are each L 1 -normalized in order to compare the relative importance of tokens (irrespective of gradient magnitudes). Figure 11: Effect of various gradient attribution methods on the ROAR test at k = 10 − 30% occlusion for the CoS-E v1.0 validation set. We compute attributions with respect to the label logit and measure label accuracy of the resulting model after masking and re-training (see §4.2 for details). The largest drop in performance comes from the L 1 norm embedding-combination method, and raw gradients are not significantly different from input*gradient. On average, input*gradient and raw gradients share 17% of tokens in the top 10%, 24% of tokens in the top 20%, and 31% of tokens in the top 30%.
Measuring Association Between Labels and Free-Text Rationales
(a) Impact of occlusion by source of attribution on label accuracy.
Measuring Association Between Labels and Free-Text Rationales
(b) Impact of occlusion by source of attribution on rationale quality. Simulatability is calculated as IR→O minus I→O. Figure 12: ROAR Feature Importance Agreement results on E-SNLI (left) and CoS-E v1.11 (right). Figure 12a shows label accuracy of the I→OR model. Figure 12b shows quality of generated rationales from the I→OR model.
Towards Teachable Reasoning Systems
Our goal is a teachable question-answering (QA) system -one that a user can interact with to see reasoning-based explanations for its answers, debug errors, and correct them so that the system gradually improves over time (sometimes referred to an explanatory interactive machine learning (XIL) (Teso and Kersting, 2019)). While the benefits Figure 1: Given a question, EntailmentWriter searches for an answer hypothesis that is supported by an entailment proof. First it over-generates candidate proofs, then it removes those that the model itself does not "believe" (i.e., confirms via self-querying that it considers all the generated proof elements to be true). Finally it selects the best verified proof. Multistep proofs are generated by iteratively backward chaining on the premises (Section 3.1.3). of such a system are evident (Lakkaraju et al., 2022), the challenges are evident also: despite recent progress in explainability (Wiegreffe and Marasović, 2021), it is often hard to understand how a model arrived at an answer, and even harder to correct it if it made a mistake. In contrast, people are typically able to provide a chain of reasoning for their decisions, and may change their mind if a flaw in their knowledge or reasoning is exposed. Our goal is to similarly have machines provide reasoned answers to questions, showing how the answer follows from its internal knowledge (and possibly externally available information), and where it is capable of changing its answer if errors in that knowledge are identified.
Towards Teachable Reasoning Systems
Our approach has three components. First, the Given a new question, facts retrieved from memory are used as additional context by EntailmentWriter, influencing its answers and proofs. (B) If the user disagrees with an answer, they localize the error in the explanation and offer corrective feedback, which is added to memory. (C) These new facts can then be retrieved if the query is re-asked, helping the system avoid repeating mistakes. Note that these also help improve answers on new, similar questions that are asked later, helping the system improve over time.
Towards Teachable Reasoning Systems
system constructs answers supported by a faithful chain of reasoning, showing how the answer follows from the system's own internal knowledge via a tree of multi-premise entailments (Figures 1,3,and 4). Chains are constructed by backward chaining from candidate answers, using a language model (LM) trained for the basic inference operation. For each step, EntailmentWriter over-generates candidate entailments, then filters out those that do not conform to its own internal knowledge ("beliefs") by self-querying, asking itself whether (a) the generated premises (leaves of the proof) are true, and (b) each entailment step is valid. As a result, the system has materialized some of its latent knowledge from which the selected answer follows. The resulting proof is thus both faithful (the answer follows from the proof) and truthful (the proof reflects the system's beliefs 1 ). Second, if an answer is wrong, a user can inspect the reasoning chain to diagnose the failure. Given a chain supporting an incorrect answer, either (at least) one of the leaf facts that the model believes must be false, and/or an entailment inference itself must be invalid. For example in Figure 2, the system incorrectly concludes that "a magnet can pick up a penny" from the over-general (false) fact that "metals are magnetic". The user can thus correct the mistake by asserting that "not all metals are magnetic", in particular copper.
Towards Teachable Reasoning Systems
Finally, to store and apply the user's feedback, we augment the model with a dynamic memory. Given a new question, EntailmentWriter retrieves user-supplied facts from this memory. These are then used as context while generating an entailment-supported answer to the question, e.g., step (C) in Figure 2, helping override prior, erroneous model beliefs and thus biasing Entail-mentWriter to avoid similar mistakes in future -a novel type of memory-based continual learning in which the model itself remains fixed (frozen), and retraining is not required.
Towards Teachable Reasoning Systems
We evaluate EntailmentWriter in several ways. First, human evaluation suggests 70% of Entail-mentWriter's chains clearly show how an answer followed from their premises, substantially higher than for explanations produced by a highperformance QA system, Macaw . Second, using simulated user feedback on two reasoning-based datasets, OBQA (Mihaylov et al., 2018) and QuaRTz (Tafjord et al., 2019), we find that EntailmentWriter is able to continuously improve with time, without retraining, requiring only a quarter of the feedback annotations available in the original dataset to reach within 1% of the upper-bound (using all gold annotations). Finally, in a small-scale experiment with real users, we find that after they interact with Entailment-Writer on training questions, the system's performance on hidden test questions similarly improves without retraining. Our contributions are thus:
Towards Teachable Reasoning Systems
1. 
Towards Teachable Reasoning Systems
, namely deriving conclusions from known facts via step-wise application of well-defined inference operations. One approach is to retrain a blackbox model end-to-end (Clark et al., 2020a), but has been limited to small rulebases. An alternative approach, which we follow here, is to have an outside loop around a model, where the model generate individual inference steps (i.e., rules), and a controller chains them together. SCSearch (Bostrom et al., 2021) and ProofWriter (iterative version)  do this in a forward-chaining fashion, while (like us) Braid (Kalyanpur et al., 2020) does this in a backward-chaining fashion. In all these systems, the required facts were expected to be provided explicitly to the model. In contrast, EntailmentWriter's reasoning also uses its own internal, latent knowledge as well as externally provided facts. LeapOfThought (Talmor et al., 2020) demonstrated that reasoning with a combination of implicit and explicit knowledge was possible, for simple 1-step inferences. We expand this for multi-step inference, and (unlike LeapOfThought) have the system explicitly articulate the implicit knowledge it uses. Recent work has shown that generating a freeform explanation ("chains of thought") before an answer can improve performance on a variety of tasks (Wei et al., 2022;Cobbe et al., 2021;Nye et al., 2021). In these works, however, the explanations are unstructured, and there are no claims of faithfulness that the answer follows from the generation, nor that the explanations themselves represent model beliefs. Materializing a Model's Internal Knowledge: Pretrained LMs contain a vast amount of knowledge, and can be thought of as a kind of knowledge base to tap into (Petroni et al., 2019). Recent work has shown that this latent knowledge can be materialized as explicit English sentences or a knowledge graph using generative techniques, e.g., COMeT (Bosselut et al., 2019), ParaCOMET (Gabriel et al., 2021). Our work with EntailmentWriter similarly materializes its latent knowledge, but here in a goaldirected way, namely by producing a faithful chain of reasoning from facts it validates ("believes") as true to an answer. This articulation can be seen as a kind of self-talk, where a self-generated context can improve QA (Shwartz et al., 2020). However, here our generations are not used as context for opaque problem-solving, but are assembled into a well-defined chain of reasoning.
Towards Teachable Reasoning Systems
Guiding Frozen Language Models and Memory: Our use of context to modify a (run-time) frozen model's behavior is similar to retrievalbased QA (Ni et al., 2019;Clark et al., 2020b), where retrieved context can improve QA performance. In our case, however, retrieval is from a dynamic memory of user-supplied facts, rather than a static corpus. To date this approach has proved more effective than directly editing the model itself (Mitchell et al., 2021). It also has some similarities to prompt engineering (Brown et al., 2020;Rubin et al., 2021), except that our context contains supporting facts rather than few-shot QA examples, and we specifically train our model to respond to context before freezing it (Section 4.1.3).
Towards Teachable Reasoning Systems
EntailmentWriter's memory-based feedback is most similar to that of BeliefBank (Kassner et al., 2021), in which retrieved memories were similarly used as context to improve future QA. In Belief-Bank, however, memories were previous system answers, while in EntailmentWriter memories are provided by a user. EntailmentWriter's memory is also related to work by Tandon et al., where user feedback memories were used but in different ways, namely to repair erroneous model outputs via postprocessing (Tandon et al., 2022a), or to clarify user intent in prompts for GPT3 (Tandon et al., 2022b). In contrast, EntailmentWriter's feedback contains corrections and elaborations to the model's internal beliefs themselves.
Towards Teachable Reasoning Systems
Feedback and Interaction: While interaction has been successfully used recently to learn users' preferences (interactive recommender systems, e.g., (Kang et al., 2019;Li et al., 2021)), knowledge graphs (Hixon et al., 2015), and procedural tasks (Li et al., 2020), our work focuses on learning corrective feedback in the context of reasoning. Early AI attempts at having users debug rule-based representations had limited success, e.g., Teiresias Figure 3: An entailment tree is a set of multi-premise, 1-step entailments (red boxes) showing how the hypothesis (root node, green) is entailed from the leaf nodes (white). If all the leaf nodes are true, and all the 1-step entailment relations are valid, then we say the tree is a valid chain of reasoning for the hypothesis. (Davis, 1977), Convince (Kim and Pearl, 1987). Our work can be viewed as the first modern formulation of this goal, using NL representations of the knowledge stored latently in a model.
Towards Teachable Reasoning Systems
We now describe EntailmentWriter's three key functions: 1. Entailment Proof Generation: in which EntailmentWriter searches for an entailmentbased chain of reasoning for different candidate answers, and select the supported answer (or best supported if more than one). 2. Interaction: where the user can inspect, locate, and correct errors in those chains 3. Dynamic Memory: in which Entailment-Writer maintains a dynamic "belief bank" (Kassner et al., 2021) of user-corrected beliefs to help answer future questions. Note that EntailmentWriter is a frozen, generalpurpose, zero-shot system, that is not fine-tuned on new datasets. Rather, to create EntailmentWriter, we train a general model to make 1-step inferences, then freeze that model and embed it in a larger system with a control algorithm and memory. It is then applied to new tasks and datasets zero-shot, with no further fine-tuning. Rather, all updates are handled through the memory.
Towards Teachable Reasoning Systems
Like several previous works (Section 2), Entail-mentWriter treats reasoning as Natural Language Inference (NLI). In NLI, the basic unit of knowledge is (represented as) a sentence rather than a structure, and a proof 2 is a tree of multi-step, multi-premise entailments, e.g., Figures 3 and 4.
Towards Teachable Reasoning Systems
Within this framework, EntailmentWriter's novel contribution is to construct entailment trees grounded in the system's latent "beliefs" about the world, rather than (just) external knowledge sources. Following (Kassner et al., 2021), we say a system believes a fact F if it answers "yes" to the question "Is F true?" -in other words, we establish (a simple notion of) belief through self-querying. Note we use the term "belief" rather than "knowledge", as the system's beliefs may be wrong. Given a hypothesis H, EntailmentWriter searches for a chain of reasoning showing how H follows from facts it believes. Similarly, given multiple competing hypotheses (e.g., from a multiple-choice question) EntailmentWriter searches for the one supported by a chain of reasoning (or the one supported by the best chain, if more than one). Entail-mentWriter's answers are thus both: faithful: The answer follows from the proof (rather than being generated post-hoc), and truthful: The proof reflects the system's own beliefs. These are prequisites for interaction (Section 3.3).
Towards Teachable Reasoning Systems
Given a question, EntailmentWriter first generates candidate answers and converts these into declarative hypotheses (e.g., "Is the sky (A) blue (B) yellow" → { H 1 = "The sky is blue.", H 2 = "The sky is yellow."). 3 An N -way multiple choice question yields N hypotheses. A true/false question yields 2 hypotheses. For open-ended questions, Entail-mentWriter first collects N candidate answers generated by an external source (Macaw  using nuclear sampling (Holtzman et al., 2019)) then forms N hypotheses from them.
Towards Teachable Reasoning Systems
A chain of reasoning is a multi-step entailment tree containing a hierarchy of 1-step, multi-premise entailments from two or more premises P = {p 1 , ..., p n } to a conclusion (either a higher-level premise or the root hypothesis H). For each 1-step entailment, we say the root node is true if all its children are also true and the entailment relation between the children and root is valid. 4 We say the Q: A magnet will stick to (A) a belt buckle (B) a wooden table (C) a plastic cup (D) a paper plate A: (A) A magnet will stick to a belt buckle because A belt buckle is sometimes magnetic. because Metal is sometimes magnetic. A belt buckle is made of metal. A magnet will stick to magnetic metals. Q: You can make a telescope with a (A) straw (B) Glass (C) Candle (D) mailing tube A: (D) You can make a telescope with a mailing tube. because A telescope is made of a tube for observing / seeing. A mailing tube is a kind of tube.
Towards Teachable Reasoning Systems
Q: Quartz may produce rainbows when light is shined (A) around the crystal's area (B) through any of its sides (C) in the room its in (D) in to a mirror at it A: (B) Quartz may produce rainbows when light is shined through any of its sides. because A rainbow is produced when light shines through a prism. because The rainbow is made of all different colors in visible light. A prism can split light into different colors. A quartz is a kind of prism. 
Towards Teachable Reasoning Systems
Input
Towards Teachable Reasoning Systems
Steel is a metal. I:"
Towards Teachable Reasoning Systems
→ 0.998 Table 1: Examples of the three input/output angles used by EntailmentWriter. The first generates a candidate entailment rule P H given H. The second and third score whether each premise, and the entailment itself, is valid, using tokens V/I in the input to indicate that S d /S e is the desired output.
Towards Teachable Reasoning Systems
entire chain of reasoning is valid if all the leaves of the tree are true, and all the individual 1-step entailment relations are also valid. If the reasoning is valid, then the hypothesis H necessarily follows. Conversely, if the hypothesis H is false, then at least one fact in the tree must be false and/or one of the 1-step entailment relations is invalid.
Towards Teachable Reasoning Systems
The core of EntailmentWriter is generating and validating a single entailment step that entails a hypothesis. We define the following "angles" (types) of data: H: A hypothesis (English statement) to prove. P: A set of premises {p 1 ,...,p i } (sentences) that together may entail the hypothesis H. Together, P and H form a one-deep entailment step, denoted by P H. Q: A question posed to EntailmentWriter. A: A candidate answer for consideration. C: A context (set of sentences) relevant to the problem We train a model (details in Section 4) with the (Section 4). three input/output behaviors below (optional inputs shown in parentheses): (QAC)H → P : Given a hypothesis H, generate a set of premises P that may entail H (QAC)H → S d : Score whether a hypothesis H (or premise p i ) is true (S d > 0.5) or not, according to the model (i.e. perform yes/no QA). We call this the "direct" score (range 0-1). (QAC)P H → S e : Score whether a candidate entailment (P H) is valid (S e > 0.5) or not, i.e., P validly entails H (range 0-1). Examples of these three angles are in Table 1. The procedure to find a 1-step inference is called 1STEP(H) in Algorithm 1 (line 19). It uses an overgenerate-and-filter approach, also found useful elsewhere (Cobbe et al., 2021;Li et al., 2022). Given a hypothesis H, we use the angle H → P to over-generate a set of k alternative backwardchaining steps P H using nucleus sampling (line 21). We then check that the model believes all the premises p i ∈ P using the H(= p i ) → S d angle, and that it believes the inference step P H itself is valid (independent of whether the p i are true or not) using the P H → S e angle (line 24). The overall score, denoting how well the 1-step proof Algorithm 1 EntailmentWriter's backchaining algorithm for searching for the best proof tree(H) with score s(H) for a hypothesis H.
Towards Teachable Reasoning Systems
ii. find reasoned score sr(H), confidence cr(H), & subtrees tree(pi): 6:
Towards Teachable Reasoning Systems
call 1STEP(H) →P to find best premises P that entail H 7:
Towards Teachable Reasoning Systems
find se(P H) via model P H → Se // score the entailment 8:
Towards Teachable Reasoning Systems
if se(P H) > c d (H) or H=H † 0 // if reasoning conf cr(H) might beat direct conf c d (H) (lines 13,11), backchain 9:
Towards Teachable Reasoning Systems
then forall pi ∈ P do 10:
Towards Teachable Reasoning Systems
call PROVE(pi) to find s(pi), tree(pi) // get scores and subtrees for each pi 11: reasoned score sr(H) = average({s(pi)}).se(P H) // overall score for reasoned answer 12:
Towards Teachable Reasoning Systems
else sr(H) = 0 13: reasoning confidence cr(H) = sr(H) // reason confidence = score 14:
Towards Teachable Reasoning Systems
iii. return direct or reasoned answer (pick most confident):  The highest-scoring proof P H is returned.
Towards Teachable Reasoning Systems
Backward Chaining Procedure PROVE(H ) in Algorithm 1 generalizes this to multi-step entailments by recursively generating support for each premise p i in P (line 10). The stopping condition is when the model's direct confidence 6 in p i is greater than the proof-derived confidence in p i , i.e., c d (p i ) > c r (p i ). When this condition is met, the subtree P p i for p i is discarded and p i becomes a leaf node of the tree (line 17). We also impose a maximum depth d on the tree.
Towards Teachable Reasoning Systems
Because premises are self-generated rather than externally provided, this stopping condition has a different semantics to earlier work, e.g., (Kalyanpur et al., 2020;Bostrom et al., 2021): Rather than 5 We also tried product() as a combination function, but empirically we found average() worked the best, likely because it is more robust to noisy scores than product(). 6 As the direct score s d (pi) in pi ranges from 0 (definitely false) to 1 (definitely true), we define the corresponding confidence to be the score's distance from 0.5, c d (pi) = |2 × (s d (pi) − 1)|. In contrast, the proof score ranges from 0 (no proof) to 1 (perfect proof), hence the proof confidence is simply the proof score, cr(pi) = sr(pi). Note that unlike s d (pi), sr(pi) = 0 means no proof, not false. stopping when externally known facts are reached, EntailmentWriter stops when strongly believed facts are reached, and more backward chaining can no longer improve a hypothesis' confidence. Finally the system selects the answer corresponding to the hypothesis with the top-scoring proof s(H).
Towards Teachable Reasoning Systems
The second key component of EntailmentWriter is the interactive interface, allowing users to inspect, debug, and correct system mistakes. Corrected model beliefs are stored in a memory, described in more detail in Section 3.3, and used as context for future questions. Facts in memory override model confidences when verifying facts (H → S d ). Our goal is that such corrections will not only fix a specific error but also help on new, unseen questions in future. We evaluate this later in Section 5.
Towards Teachable Reasoning Systems
The interface is currently experimental and so we do not make strong claims for it. For interaction purposes, the interactive system uses just 1-deep proofs (d=1) of the candidate hypotheses, with the option to interactively drill down on premises that the user wants more detail on. A snapshot of the full EntailmentWriter interface is shown in Ap-pendix C. and explained shortly. The overall flow of interaction is as follows:
Towards Teachable Reasoning Systems
1. The user poses a question to the system 2. EntailmentWriter shows its selected answer and its proof 3. If the user disagrees with the answer, they have several options to indicate why:
Towards Teachable Reasoning Systems
• can explicitly provide the first premise of the proof. That premise is noted in memory, and if the question is re-asked, the proof generation is forced to start with that given premise (forced generation).
Towards Teachable Reasoning Systems
In addition, EntailmentWriter shows its attempted proofs for other answer options for the user to inspect (under "Trace output", Appendix C). This is particularly useful for the user to see why the proof for the correct answer failed, and correct it so it succeeds. It is also useful if no successful proof is found. 4. After making a change, the system re-attempts the question, and shows its updated answer and reasoning. When the user is satisfied, they indicate so (clicking "Yes" to "Do you agree?"). The premises and answer are then noted in memory, for consistency if those premises and answers are queried about in future. Thus the user can indicate why a bad answer is wrong, or offer corrections to help show why the correct answer is right. The updated beliefs are stored in memory, and override the model's original beliefs for future questions (Section 3.3).
Towards Teachable Reasoning Systems
As an example below, the user (playing the role of teacher) has asked EntailmentWriter a probing question "Can a magnet attract a penny?" in the upper box. (The correct answer is "no", as pennies are made of the non-magnetic metal copper). The system has responded with an (incorrect) answer and proof (blue):
Towards Teachable Reasoning Systems
The error in this case is the system's belief that "A penny is made of magnetic metal." (pennies are in fact made of non-magnetic copper). To indicate the error, the user clicks on the "No" button in response to "Do you agree?", and then here chooses to correct the error by entering the missing, required knowledge that "A penny is made of copper.":
Towards Teachable Reasoning Systems
On reasking the question, EntailmentWriter uses this user-supplied fact as the first part of the proof (via forced generation), but still gets the answer wrong due to a different misunderstanding, namely that "A magnet can attract copper."
Towards Teachable Reasoning Systems
Here the user chooses to correct this mistake by clicking on [but it's not true!] next to "A magnet can attract copper", causing the opposite fact ("A magnet cannot attract copper") to be added to memory. Finally the system gets the answer right:
Towards Teachable Reasoning Systems
Confirming the answer causes EntailmentWriter to note both the facts, conclusion, and proof in memory, so that it will produce the same answer if the question is reasked. In addition, the stored facts can be retrieved to help answer future questions, as we now describe.
Towards Teachable Reasoning Systems
The third key component of EntailmentWriter is a dynamic memory, containing a list of assertions (English sentences), collected through interaction. The memory serves as a "belief bank" of additions and overrides to the model's latent beliefs, and to our knowledge is the first to show that userprovided and model-internal beliefs can be integrated together for systematic reasoning.
Towards Teachable Reasoning Systems
Note that during interaction, EntailmentWriter's neural component (for 1-step inference) is frozen: all updates are made by modifying the memory, rather than the model parameters themselves ("model editing" (Mitchell et al., 2021) 
Towards Teachable Reasoning Systems
Up to r (= 5) sentences are retrieved from memory using the question as the search query, using a standard BM25 search algorithm. To ensure Entail-mentWriter pays attention to facts in the context, incorporating them in its generated proofs if it can, we use two mechanisms: As Context: Retrieved facts are provided to EntailmentWriter using the optional context field (angle) C in its input (Section 3.1.3) to influence its behavior. In our case, as these retrievals are high-quality, user-approved facts rather than noisy retrievals from a heterogenous corpus, we train EntailmentWriter to pay for Ai ∈ A do 5:
Towards Teachable Reasoning Systems
// Generate a hypothesis Hi for each choice Ai 6:
Towards Teachable Reasoning Systems
Hi = Hypothesisi = QA2D(Q,Ai) 7:
Towards Teachable Reasoning Systems
for Cj ∈ C ∪ {none} do // for each sent Cj 8:
Towards Teachable Reasoning Systems
// 1. Gen proof of Hi with first premise = Cj: 9:
Towards Teachable Reasoning Systems
call 1STEP(Hi) † in Algorithm 1 to find support-10:
Towards Teachable Reasoning Systems
ing premises Pi and score s r-1deep (Hi) 11:
Towards Teachable Reasoning Systems
// 2. Add the proof to the list of proofs so far: 12: E = E ∪ <(Pi Hi), s r-1deep (Hi), Ai> 13: <E best , score best , A best > = Max(E) 14:
Towards Teachable Reasoning Systems
return answer=A best , explanation=E best † When the model generates premises Pi in 1STEP(Hi), Q, Ai, and C are provided as additional model inputs, and the output is constrained to start with Cj (forced generation). more attention to them than if they had come (say) from the Web (Section 4). This encourages (but does not force) EntailmentWriter to use these facts in a generated proof. Forced Generation: Given r retrieved sentences, interactive EntailmentWriter generates r + 1 proofs, where each of the first r proofs uses one of the retrieved sentences as the first sentence in the proof. This is done using forced generation during decoding time so that the proof starts with the rth retrieved sentence as its first premise. The last r + 1 proof is generated as normal (no forced generation). This forcing can be seen as a way of encouraging diversity in the generations, as each of the r proofs necessarily starts with a different premise. Note that many of these proofs may later be rejected if verification fails. The highest-scoring proof is then selected. The overall control algorithm with memory is shown in Algorithm 2. Given a question Q with answer options A, EntailmentWriter first retrieves potentially relevant sentences C from memory (line 3). Then, for each answer option, it creates a hypothesis (line 6), then generates proofs using each sentence in C in turn as the first premise, as well as free-form generation (line 9). Finally, the answer option associated with the highest scored proof is returned (line 14).
Towards Teachable Reasoning Systems
As a complete system, EntailmentWriter consists of a model with the three functionalities in Table 1, a controller which iterates over it to build proofs (Algorithms 1 and 2), and a dynamic memory (Sec- tion 3.3). Before use, the model is trained for its three functionalities, described below. Then, during interaction with users, the model is frozen: any subsequent improvements in EntailmentWriter's behavior come solely from users' contributions to the memory, rather than changes to the model's parameters themselves.
Towards Teachable Reasoning Systems
To train EntailmentWriter's model before deployment, we leaverage (the training partition of) the EntailmentBank dataset (Dalvi et al., 2021). Entail-mentBank contains 1840 multiple-choice science questions (1313 in the training partition) along with their correct answer, a hypothesis expressing the question and answer in declarative form, and a multistep entailment tree expressing how the hypothesis follows from a set of facts drawn from a corpus of science facts (the WorldTree corpus (Xie et al., 2020)). Using the notation introduced earlier, each EntailmentBank example is of the form:
Towards Teachable Reasoning Systems
< Q, A, H 0 , { (P i H i ) * } > where (P i H i )* denotes a set of entailment steps be assembled into a tree (with root H i = H 0 ), describing how the corpus facts (the leaves p i ) entail the hypothesis. An example is shown in Figure 5.
Towards Teachable Reasoning Systems
EntailmentBank only contains positive examples of entailments. To obtain negative examples, we first ran an earlier, positive-only-trained version of EntailmentWriter to generate proofs of (hypotheses for) all the answer options in EntailmentBank's questions (all 4-way multiple choice), resulting in one proof for the true hypothesis, and three (necessarily bad) proofs for the three false hypotheses.
Towards Teachable Reasoning Systems
(This was done using just the H→P angle, without verification). Note that EntailmentWriter will happily generate some kind of proof even for false hypotheses, e.g., A rabbit has six legs because: 1. A rabbit is an animal 2. Animals have six legs These invalid proofs will be incorrect either because a generated fact is false, and/or because the inference itself is not a valid entailment. To distinguish these, we use crowdworkers to assess whether the generated facts were true, and if the entailment itself was valid. The 1313 questions result in 3939 proofs for false hypotheses. Dropping the few with more than two premises (to simplify the crowdsourcing interface), crowdworkers annotated 3674 proofs, using labels T/F/? for each premise and T/F/? for the entailment itself. Each proof was annotated by 3 crowdworkers, then an additional 3 workers provided additional annotations for cases with no initial majority verdict. Dropping premises/entailments without a final majority verdict, we end up with 7013 additional examples of labeled premises (3900 T, 3113 F) for the H → S d angle, and 3391 additional examples of labeled entailments (1827 T, 1564 F) for the P H → S e angle. The crowdworker interface is in Appendix B.
Towards Teachable Reasoning Systems
We also augment the training data with duplicate examples but with additional, optional input fields: QA: The input QA question/answer-option pair, as well as the hypothesis H C: A context consisting of up to 5 relevant sentences At inference (test) time, C will contain retrievals from the user-provided memory. To simulate this during training, we use a mixture of sentences drawn from (a) the gold (target) entailment (i.e., the gold premises), and (b) sentences retrieved from a corpus of similar-styled knowledge (namely all the leaf sentences used in the EntailmentBank entailment trees), mixed in different ratios so the model is exposed to a mixture of high and medium relevance sentences. Further details of training are given in Appendix A.
Towards Teachable Reasoning Systems
We train a T5-11B multi-angle model, Entail-mentWriter, following the multi-task setup similar to  for both en-tailment generation and verification using this training data. Specifically, we train it for three separate input/output "angles" described earlier: H → P (proof generation), H → S d (fact scoring/verification), and P H → S e (entailment scoring/verification), including with and without the optional input fields QAC. 7 We used the default hyperparameters (including the Adafactor optimizer) in the T5 library, 8 fine-tune the model for 20K steps with batch size of 8, selecting the checkpoint with highest validation score (usually the final step).
Towards Teachable Reasoning Systems
At inference time, EntailmentWriter's primary function is to generate candidate entailments for each answer choice for a given question, which are then scored by the same model using its verification angles. To generate multiple explanations for a single input (answer hypothesis, IR retrieved context), in addition to the greedy decoding we use nuclear sampling (for the experiments in this paper we set "temperature= 2.0", and "top_p= 0.95").
Towards Teachable Reasoning Systems
We have two key research questions:
Towards Teachable Reasoning Systems
1. Does entailment-based reasoning lead to better answers, both in terms of explaining a model's inference and answer accuracy? 2. Can our dynamic memory mechanism help improve EntailmentWriter's performance on new, unseen test questions without requiring model retraining, allowing users to "teach" the system? Our focus here is on the feedback memory mechanism itself, rather than the user interface design and interaction protocol, for which we do not make any claims given its preliminary form. For the first question, we use two measures: (a) We collect human judgements of whether the hypothesis clearly follows from facts in the proof (b) we compare EntailmentWriter's reasoning-based answer accuracy with its "direct" question-answering capability (both zero shot).
Towards Teachable Reasoning Systems
For the second question, we primarily evaluate EntailmentWriter using simulated user feedback on two datasets (training partitions) then test the taught system on the test partitions. Finally, in the subsequent section, we report results on a smallscale experiment with real users, in which they 7 The verification scores are obtained by training the model to output the text "true" or "false", the score being the model's confidence in the "true" output. 8 https://github.com/google-research/text-to-text-transfertransformer repair EntailmentWriter's performance on a handful of OBQA training questions, then test on hidden test questions. In all cases, our primary metric is question-answering accuracy.
Towards Teachable Reasoning Systems
We evaluate EntailmentWriter using two existing multiple-choice datasets, namely OBQA (Mihaylov et al., 2018) and QuaRTz (Tafjord et al., 2019). These datasets contain multiple-choice questions that (typically) require multihop reasoning, along with a (crowdworker created) gold 1step entailment proof for every correct answer option. In addition, among the premises in those gold proofs, one has been tagged as the "core" (most important) fact of the proof (e.g., "Metals conduct electricity"), with several questions sharing the same core fact. For our purposes, these gold proofs are important as we can use facts in them to simulate user feedback.
Towards Teachable Reasoning Systems
For meaningful feedback experiments, there should be at least topical overlap between train (teaching) and test (evaluation) partitions. In OBQA, this topical overlap occurs naturally because the train/test partitions were created randomly, meaning that questions based on the same core fact are distributed between train and test. 9 QuaRTz, however, was originally partitioned to remove topical (core fact) overlap between train and test. As a result, we use just the training partition of QuaRTz, and repartition it randomly into NewTrain/NewDev/NewTest, leading to a natural topical overlap between the new partitions.
Towards Teachable Reasoning Systems
The sizes of the partitions we use are OBQA train/test = 4957/500 examples, and QuaRTz New-Train/NewDev = 1348/557 examples.
Towards Teachable Reasoning Systems
To first evaluate the quality of EntailmentWriter's proofs, before considering interaction, six annotators compared 1-deep EntailmentWriter proofs with explanations generated by Macaw, a stateof-the-art QA system . Examples of the explanations are shown in Appendix D. Annotators scored each explanation along four dimensions, and finally compared the two:
Towards Teachable Reasoning Systems
1. Does the conclusion clearly follow from the premises? Figure 6: Users' assessments of EntailmentWriter's proofs (red), compared to Macaw's explanations (blue), showing percent of times annotators answered "Yes". EntailmentWriter's conclusions were judged to "clearly follow from the premises" in over 65% of the proofs (first bar), substantially more than Macaw's explanations (30%).
Towards Teachable Reasoning Systems
2. Are all the premises correct? 3. Are all of the premises relevant? 4. Does the explanation (premises) introduce something new and useful, i.e., does more than just restate the conclusion in different words? 5. Which explanation do you prefer? Answer options for questions 1-4 were yes/somewhat/no, and for question 5 were first/similar/second. The ordering of explanations were scrambled so the annotators did not know which explanation was which, and in fact they were unaware that the different explanations had been generated by different systems. We collected annotations for 78 pairs of explanations for correct answers to 78 OBQA dev questions. The results ( Figure 6) suggest several findings: 1. In absolute terms, EntailmentWriter's conclusions were judged to clearly follow from the premises in the large majority (over 65%) of the explanations, and substantially more than Macaw's explanations (30%). This potentially contributes to system trustworthiness, where understanding how evidence supports a conclusion is critical. 2. ≈90% of EntailmentWriter's self-verified premises were judged correct by users. Of the remainder, virtually all were labeled "unsure" (only 1 EntailmentWriter fact was labeled not correct), indicating that the there are few false beliefs in proofs for correct answers. 10 Rather, vague facts (e.g., "Claws are used for cracking open shells") make up the remainder.
Towards Teachable Reasoning Systems
EntailmentWriter's explanations were clearly preferred (60% to 20%, last bar) over Macaw's. In particular, EntailmentWriter's arrangement of facts into a tree discourages irrelevant information (bar #3). Finally we note EntailmentWriter's proofs are faithful (showing how an answer was derived) and truthful (reflecting the system's own beliefs), while Macaw's explanations are post-hoc generated text. These all suggest the value of entailment trees as trustable explanations of a system's behavior.
Towards Teachable Reasoning Systems
We now consider EntailmentWriter's ability to learn through interaction, first using a simulated user (teacher). In this scenario, we consider the teacher working through the training questions, and behaving as follows:
Towards Teachable Reasoning Systems
1. If EntailmentWriter answers the question correctly then no action is taken. This makes the simplifying assumption that the generated chain of reasoning is also correct.
Towards Teachable Reasoning Systems
correctly, then the user will provide feedback to help correct the system. In the simulated scenario, we take the core fact in the gold entailment proof as that user feedback: As the system was wrong, we here assume that either the model did not know this core fact, or failed to attend to it when trying to generate a chain of reasoning for the correct answer. The (simulated) user thus aims to correct this by providing that fact. This new fact is then added to the system's memory, where it may be recalled and used for future questions to avoid a similar mistake in future. Although only an approximation, it allows us to assess whether this failure-driven provision of knowledge also helps on future, unseen questions. Note that throughout this simulation, the model itself is frozen; the overall system behavior is modified purely by the growing contents of the dynamic memory. Once simulated teaching is completed, we then test the system on a hidden test set (no further interaction), measuring QA accuracy.
Towards Teachable Reasoning Systems
We compare the following configurations: Figure 7: EntailmentWriter's performance on the hidden test sets improves with user feedback, improving over direct QA and coming close (within ≈ 1%) of the upper bound of using feedback on all answers.
Towards Teachable Reasoning Systems
Here we measure the model's ability to directly answer the test questions (zero shot) using the H → S d angle.
Towards Teachable Reasoning Systems
One can loosely think of this as the "fast thinking" answer.
Towards Teachable Reasoning Systems
Here we measure EntailmentWriter's basic ability to answer the test questions (again zero shot) by generating, scoring, and comparing entailment proofs for each answer option. One can loosely think of this as the "slow thinking" answer.
Towards Teachable Reasoning Systems
3. EntailmentWriter (after teaching): This is the end of the simulated teaching scenario, where the user provides feedback (the appropriate core fact) for all training questions that Entailment-Writer answered incorrectly. To simulate this, En-tailentWriter's memory is set to the core facts used in all failing training questions. It is then tested on the test set.
Towards Teachable Reasoning Systems
As an upper bound, we imagine the user providing feedback on all training questions, regardless of whether Entailment-Writer answered them correctly or not. To simulate this, EntailmentWriter's memory is set to all the core facts used in all training questions. In this upper-bound scenario, the simulated user is doing approximately the same work as it took to create the training dataset proofs in the first place.
Towards Teachable Reasoning Systems
The results are shown in Figure 7. Our main findings are as follows: Figure 8: EntailmentWriter's performance on OBQA test improves as it sees a larger fraction of training data and stores feedback for wrong answers in its memory.
Towards Teachable Reasoning Systems
EntailmentWriter's Basic Accuracy is Close to that of Direct Answering: Comparing Entail-mentWriter (before teaching) with direct QA, we see EntailmentWriter's proof-based answer accuracy is close, but not quite as good as, the accuracy for direct QA (72.6% vs. 75.2% OBQA 11 , 73.6% vs. 74.1% QuaRTz). It is encouraging that the scores are loosely comparable, as it suggests users are critiquing proofs of reasonable quality. A primary cause of failure is errors by the two verifiers, in particular the entailment verifier P H → S e sometimes mis-recognizes a bad entailment as valid.
Towards Teachable Reasoning Systems
Feedback helps on new questions. Most significantly, feedback on the training questions has helped improve performance on the test questions, without requiring model retraining (OBQA: 72.6% to 77.0%; QuaRTz: 73.6% to 75.9%), indicating the viability of the paradigm we are exploring. The with-memory scores also exceed the direct QA scores on both datasets.
Towards Teachable Reasoning Systems
Feedback reaches within 1% of the upper bound while only requiring feedback on ≈30% of the training questions (namely those that the model answered incorrectly). This suggests that targeted feedback is sufficient to obtain near-optimal performance, avoiding the high cost of exhaustively annotating the proofs for all the training questions, as was done in the original datasets.
Towards Teachable Reasoning Systems
How does the interactive EntailmentWriter's performance improve with time? To track this, we re-used the OBQA dataset and measured Entail-mentWriter's performance on the test set as it sees a Category Question
Towards Teachable Reasoning Systems
Model generated answer, best proof Comments Uses Gold core fact Q: Owls are likely to hunt at (A) 3pm (B) 2am (C) 6pm (D) 7am
Towards Teachable Reasoning Systems
Owls are likely to hunt at 2am because: Nocturnal predators hunt during the night. Owls are a kind of nocturnal predator. 2:00 AM is after midnight.
Towards Teachable Reasoning Systems
Model generated reasonable chain of reasoning justifying the correct answer using the gold core fact.
Towards Teachable Reasoning Systems
Wax can be used similarly to (A) wood (B) rubber (C) water (D) metal Wax ...similarly to rubber because: Rubber is a thermal insulator. Wax is a thermal insulator.
Towards Teachable Reasoning Systems
Gold core fact: "wax is a electrical insulator" was not in top-k retrieval. But the model used another relevant fact to construct a good proof.
Towards Teachable Reasoning Systems
Uses remotely relevant fact Q: They looked where the log decayed to garden as it would leave the earth (A) richer (B) dryer (C) sandy (D) harder ...leave the earth richer because:
Towards Teachable Reasoning Systems
The mantle is a layer of the Earth.
Towards Teachable Reasoning Systems
Decaying leaves a layer of soil on the Earth. Soil is a kind of resource. If something decays then that something will leave the earth richer.
Towards Teachable Reasoning Systems
Model includes a remotely relevant fact in the proof. As the answer is still entailed, the verifier approves the proof.    Table 2).
Towards Teachable Reasoning Systems
larger fraction of training data, storing the feedback for wrong answers it has seen so far in its memory. The results were averaged over 3 random orderings of OBQA training data, and are shown in Figure 8. As can be seen, the performance gradually improves as more feedback is collected on failing training questions. Note that a larger memory does not guarantee better performance, e.g. when training data increases from 20% to 30% in Figure 8, because Entailmentwriter may retrieve distracting facts from memory, resulting in spurious proofs supporting wrong answers.
Towards Teachable Reasoning Systems
When EntailmentWriter changed its (test set) answer from a wrong answer option (no feedback) to the correct answer option (with feedback), was that change for a good reason? ("right for the right reasons" (Ross et al., 2017)). Our interest here is whether EntailmentWriter did indeed recall and use relevant domain knowledge appropriately. To explore this, we analyzed a random sample of 50 of the 74/500 test cases where such positive flips occurred. Of these, we found approximately 3/4 resulted from good reasoning, while approximately 1/4 were not. Comparing the generated and gold test set proofs, we found four classes of reasoning, illustrated in Figure 9 and Table 2 and described below: 28% (14/50) : the gold core fact was included in the best scoring proof, resulting in the correct answer. 28% (14/50) : a relevant core fact (though not exactly gold core fact) was used in the best scoring proof resulting in the correct answer. 20% (10/50) : a remotely related fact was retrieved and used by the model as the first premise in the proof due to forced generation (Algorithm 1 line 9), followed by a proof of the right answer.
Towards Teachable Reasoning Systems
(The retrieved fact is partially redundant). 24% (12/50) : a spurious fact was retrieved due to word overlap with question, then the model produced an incoherent proof connecting it to the correct answer hypothesis and scored this proof highest among all candidates. Although this error was advantageous in these cases, there are analogous failure cases where a spurious fact changes a previ- Gold science fact: "A mammal is warm-blooded." is part of memory but not in top-k retrieval Bad Reasoning Q: What has more gravity force than Earth but less than the sun? (A) Jupiter (B) the moon (C) a space station (D) a comet The moon has ... because:
Towards Teachable Reasoning Systems
The Moon orbits the Earth. The Earth has more mass than the Moon. The Moon has more gravity force than the Earth. The Sun has more mass than the Moon.
Towards Teachable Reasoning Systems
Gold science fact: "as mass of a celestial body increases, the force of gravity on that planet will increase" is at rank 4 in the retrieval. Model incorrectly starts with less relevant fact and completes a proof for the wrong answer option.
Towards Teachable Reasoning Systems
Bad Scoring Q: Which of these is required for a plant to enjoy the product of a rain storm?
Towards Teachable Reasoning Systems
Dirt is required...because: Clouds produce rain. A plant requires dirt to grow.
Towards Teachable Reasoning Systems
Model generated valid proof for correct answer "xylem" using Gold science fact (from top-k retrievals): "Xylem carries water from the roots of a plant to the leaves of a plant." but premise verifier scored it as incorrect premise (score=0.045). Hence the proof for wrong answer was scored higher than that for the correct answer.   Table 3).
Towards Teachable Reasoning Systems
ously correct answer to incorrect. A possible cause of this is that the entailment verifier (P H → S e ) has not seen many training examples where where the two premises in a proof are unrelated, hence wrongly assigns high scores to such proofs resulting in bad ranking of generated proofs. We plan to alleviate this by synthetically adding such training examples during future verifier training.
Towards Teachable Reasoning Systems
In cases where retrieved feedback did not help on new questions, there are several candidate reasons: Knowledge: the relevant knowledge was simply not in memory Retrieval: the knowledge was there but not retrieved
Towards Teachable Reasoning Systems
Reasoning: the knowledge was there and retrieved, but EntailmentWriter chose to ignore it Scoring: the knowledge was retrieved and used, but the correct answer was still not chosen as a proof for a different answer option scored higher To measure the relative frequency of these failures, we examine 50 randomly sampled failure cases, described below. Figure 10 Table 3 gives an example for each of them: 24% (12/50) missing knowledge: The gold science fact for the test question was not present in the corpus. Instead, the model tried to make use of the facts retrieved from the corpus to construct proofs but ended up selecting a wrong answer option. 54% (27/50) bad retrieval: The gold science fact for the test question was present in the corpus but IR module failed to retrieve it among top-k. 12% (6/50) bad reasoning: The proof generated for gold answer option was not good, even when the retrieval was good. In 5/6 cases, the model created a bad proof, even though it had correctly start with the correct core fact. In the remaining case, the gold core fact was retrieved, but the model picked an unrelated retrieved fact to create a proof Figure 11: EntailmentWriter's performance (% correct) substantially improves on a hidden test set (from 38% to 55%), a subset of OBQA, after users correct/expand its knowledge for the training questions. (Results are averaged over 8 users). from instead. 10% (5/50) bad scoring: While a good proof for the right answer was generated, it was not scored highest either due to some of its (true) premises or entailment being disbelieved by the model, or a false premise or bad entailment for a wrong answer being scored highly. Again, further training of the verifiers would help alleviate this problem.
Towards Teachable Reasoning Systems
Although our experiments have primarily been with a simulated user, we also piloted a small-scale experiment with real users, to test whether users could in practice improve the system's performance. For this, we took a small subset of 31 OBQA questions based on five core facts that EntailmentWriter struggled with (getting 20/31 of the questions wrong), and then split them into a training set containing 1 failing question for each core fact (total 5 questions), and the remaining 26 questions as a test set. Our interests were (a) whether users could successfully interact with the system to identify and correct EntailmentWriter's misunderstandings on the 5 training questions, so it could answer them correctly, and then (b) whether the result of the "teaching" carried over to improved performance on the test set.
Towards Teachable Reasoning Systems
The results were averaged over eight users (from within our organization), and are in Figure 11 showing EntailmentWriter's scores before and after user interaction. Users were able to correct/expand EntailmentWriter's knowledge to remove almost all its errors on the training set (raising Entail-mentWriter's training score to 97%). In addition, the taught system's score on the hidden test set increased by 17% (38% to 55%), indicating the knowledge provided by the users generalized to the test set.
Towards Teachable Reasoning Systems
Of the 208 test answers (21 questions x 8 users), 41 answers changed from incorrect to correct, and 7 changed from correct to incorrect. Of the 41 that changed to correct (based on an analysis of a subset):
Towards Teachable Reasoning Systems
• ≈50% a relevant fact in memory was recalled and used in a good proof • ≈20% a slightly relevant fact was recalled and used in the good proof • ≈10% the recalled facts altered the model behavior so it generated a good proof with a (generated) relevant fact • ≈20% the newly generated proof was bad but (fortuitously) scored highest For example, for the question:
Towards Teachable Reasoning Systems
[correct] the model originally selected a wrong answer (eggs), and could not generate a proof for the correct answer. With memory, its retrieval included the user-supplied fact "Animals can use magnetic patterns to navigate.", providing crucial knowledge that the model apparently did not know, and allowing a proof for the right answer to be found.
Towards Teachable Reasoning Systems
Similarly for the 7 cases that changed from correct to incorrect: about half the time (4/7) the system did recall a relevant fact, but either ignored it (2/7) or generated a bad proof (2/7). In the remaining 3/7 cases, there was no relevant fact retrieved, but the retrievals served to confuse the generator. For example, for the question:
Towards Teachable Reasoning Systems
Gills are used to breath water by what? (A) salmon [correct] (B) fishing boats (C) mammals (D) penguins the system originally selected the right answer (salmon), with an (incorrect) proof for penguins close behind. With memory, it retrieved the usersupplied fact "Animals can use magnetic patterns to navigate.", irrelevant to the question, but enough when added to the context to slightly change the verification scores, resulting in the (bad) proof for penguin being scored highest.
Towards Teachable Reasoning Systems
Our goal is a teachable reasoning system, where users can interact with it to see its beliefs and reasoning, and correct it when it is wrong. To this end, we have presented the first system that can generate a chain of reasoning showing how an answer systematically follows from its own internal beliefs, thus revealing relevant beliefs to a user. Its answers are thus faithful, systematic, and truthful, allowing users to gain some understanding of how the model's beliefs entail the conclusions it draws.
Towards Teachable Reasoning Systems
We have also shown that by embedding the model in a larger system with a dynamic, persistent memory, users can correct and override model beliefs, resulting in an overall system that can improve over time without retraining. To our knowledge, this is the first system to show that userprovided and model-internal beliefs can be integrated together for systematic reasoning. This is significant as it is a step towards systems that can not only interact with users, but continually learn from them.
Towards Teachable Reasoning Systems
Although we have created and evaluated an integrated system, numerous issues still remain. For reasoning, the accuracy of the fact and entailment verifiers could be further improved, and methods to avoid uninteresting (near-tautologous) proofs are needed. For interaction, we have treated "teaching" primarily as question-centric debugging, but clearly there are other styles to explore, e.g., the user volunteering general knowledge up-front, probing what the system already knows, and following a curriculum. Finally while the memory usefully biases EntailmentWriter for new tasks, the effects of placing new knowledge in an input context are not fully predictable, despite careful training. These are all areas for future exploration.
Towards Teachable Reasoning Systems
Despite these, the research agenda is an exciting one, pointing towards future systems that can learn directly from users in a conversational way, rather than solely training on large datasets. It also suggests a way of overcoming the opaqueness of neural systems, by viewing models as components in a larger system with a persistent memory and that can systematically reason. We look forward to future developments in these directions. 5. Each EntailmentBank proof also comes with a set of associated relevant facts (sentences), only some of which are used in the entailment proofs. We use these to create a "relevant context" containing these sentences, sorted into two buckets: high-relevance: sentences actually used in the proof of the correct answer medium-relevance: the remaining sentences This context is the same across all answer options at this point.
Towards Teachable Reasoning Systems
6. For every hypothesis and premise appearing in this dataset, we run a simple BM25 IR search against a larger science text corpus (about 1.5 million sentences from a science-filtered Wikipedia) to obtain noisier sentences to use as a low-relevance context. 7. For the final training dataset, we create 4 different contexts, using 4 sampling strategies. First we create a "full" context of sentences sorted into three buckets, with added noise: high: high-relevance facts + 10% chance of a random sentence from medium/low medium: medium-relevance facts + 20% chance of random sentences from high/low low: 5 low-relevance facts + 20% chance of random sentence from high/medium Then, we use four sampling strategies to create actual contexts for the Entailment-Writer training data. Each strategy is defined by the per-sentence chance of a sentence coming from one of the high/medium/low buckets. For example, a context sampled with 0.2/0.4/0.6 means that 20% of its sentences came from high, 40% from medium, an 60% from low. If no sentence is selectable (e.g., a bucket is exhausted), we use one from low. The final context is syntactically expressed as "[HIGH] <high sen-tences> [MEDIUM] <medium sentences> [LOW] <low sentences>". The 4 sampling strategies are: 1/1/1 (full context, all available sentences) 0.5/0.5/0.5 (half of sentences from each category) 0.2/0.4/0.6 (more emphasis on lower categories) 0/0/1 (only low sentences)
Towards Teachable Reasoning Systems
For the training set we store these as a list of contexts, to be sampled at random when generating the training instances. We do the same for hypotheses, so these are also sampled at random. 8. The model is trained across the following angles, each sampled equally: {H→P, H→V, HP→I, QAH→P, QAH→V, QAHP→I, HC→P, HC→V, HPC→I, QAHC→P, QAHC→V, QAHPC→I} This is just the core angles H→P, H→V, HP→I, with optional QA and/or C added. The individual premise verification (leaves) uses the H→V angle (where H is now a premise), for those angles we don't have associated proofs P so we limit to the angles: H→V, QAH→V, HC→V, QAHC→V
Towards Teachable Reasoning Systems
The full dataset is available on request.
Generated Knowledge Prompting for Commonsense Reasoning
It remains an open research question whether external knowledge is needed for commonsense reasoning. On one hand, a substantial body of prior work has reported that integrating external knowledge can help improve task performance (Mitra et al., 2019;Bian et al., 2021, inter alia), especially if the knowledge is high quality (e.g. hand-crafted by experts). On the other hand, recent leaderboards are often dominated by large-scale pretrained models that are fine-tuned on a target benchmark (Khashabi et al., 2020;Lourie et al., 2021), suggesting that the benefits of external knowledge may wash away as the underlying models increase in size and are pretrained on ever larger amounts of raw text.
Generated Knowledge Prompting for Commonsense Reasoning
Even if external knowledge is found to be effective on a particular task, flexibility remains a fundamental hurdle to integrating external knowl- Figure 1: Generated knowledge prompting involves (i) using few-shot demonstrations to generate questionrelated knowledge statements from a language model; (ii) using a second language model to make predictions with each knowledge statement, then selecting the highest-confidence prediction. edge, as many benchmarks currently lack appropriate knowledge bases with sufficient coverage. Furthermore, prior methods often require task-specific, custom supervision for knowledge integration (Mitra et al., 2019;Chang et al., 2020), introducing a burden for rapidly adapting new pretrained models to a wide variety of tasks.
Generated Knowledge Prompting for Commonsense Reasoning
In this paper, we investigate whether external knowledge can be helpful for commonsense reasoning, even on top of the largest state-of-the-art pretrained models (e.g. T5-11b (Raffel et al., 2019) and its variants), with a focus on four recent commonsense benchmarks. To facilitate easier adaptation with any zero-shot or finetuned models, we propose an approach that does not require access to a structured knowledge base or joint finetuning for knowledge integration.
Generated Knowledge Prompting for Commonsense Reasoning
The key insight behind our method, Generated Knowledge Prompting (sketched in Figure 1), is that we can generate useful knowledge from a language model, then provide the knowledge as an input prompt that is concatenated with a question. To Part of golf is trying to get a higher point total than others. yes 1.00 | 0.00 The player with the lowest score wins. no 1.00 QASC Sponges eat primarily cartilage 0.95 | 0.00 Sponges eat bacteria and other tiny organisms.
Generated Knowledge Prompting for Commonsense Reasoning
krill and plankton 0.99 Table 1: Examples where prompting with generated knowledge rectifies model prediction. Each section shows the correct answer in green, the incorrect answer in red, and the prediction scores from the inference model that only sees the question (top) and the same model that sees the question prompted with the given knowledge (bottom). support a variety of settings without finetuning, the quality and flexibility of knowledge is crucial. We propose a simple, yet effective, method that elicits knowledge statements (i.e. knowledge expressed as natural language statements) from generic language models in a few-shot setting. Compared to prior work that elicits knowledge via clarification questions (Shwartz et al., 2020) or contrastive explanations (Paranjape et al., 2021), our approach can generate knowledge flexibly, beyond the scope of pre-defined templates (Table 1). Experiments show that our method improves both zero-shot and finetuned models on numerical commonsense (NumerSense (Lin et al., 2020)), general commonsense (CommonsenseQA (Talmor et al., 2019), CommonsenseQA 2.0 (Talmor et al., 2021)), and scientific commonsense (QASC ) benchmarks, setting a new state-ofthe-art on three of these datasets. It outperforms the template-based knowledge generation method self-talk (Shwartz et al., 2020), while performing comparably to retrieval-based systems.
Generated Knowledge Prompting for Commonsense Reasoning
We find three factors contribute to the performance of generated knowledge prompting: (i) the quality of knowledge, (ii) the quantity of knowledge where the performance improves with more knowledge statements, and (iii) the strategy for integrating knowledge during inference. Our qualitative analysis suggests that the generated knowledge statements cover a variety of types, and can transform commonsense question answering to explicit reasoning procedures, e.g. deduction, that are supported by off-the-shelf and finetuned language models.
Generated Knowledge Prompting for Commonsense Reasoning
A multiple-choice commonsense reasoning task involves predicting an answer a ∈ A q given a ques-tion q ∈ Q, where the set of choices A q is finite and can vary by question, and both questions and answers are variable-length text sequences. Our method answers commonsense questions in two steps.
Generated Knowledge Prompting for Commonsense Reasoning
The first step is knowledge generation, where we use a language model p G (k|q) to generate knowledge statements conditioned on the question:
Generated Knowledge Prompting for Commonsense Reasoning
where each knowledge statement k m is a variablelength text sequence. Intuitively, each statement contains information that is helpful for answering the question (e.g. Table 1).
Generated Knowledge Prompting for Commonsense Reasoning
The second step is knowledge integration, where generated knowledge is integrated into the decision process of a language model used for inference, a = arg max a∈Aq p I (a|q, K q ).
Generated Knowledge Prompting for Commonsense Reasoning
In contrast, the vanilla setting of using the inference model without knowledge is represented bŷ a = arg max a∈Aq p I (a|q).
Generated Knowledge Prompting for Commonsense Reasoning
Next, we describe the knowledge generation and integration steps in detail.
Generated Knowledge Prompting for Commonsense Reasoning
We generate question-related knowledge statements by prompting a language model. The prompt consists of an instruction, a few demonstrations that are fixed for each task, and a new-question placeholder. The demonstrations are human-written, and each consists of a question in the style of the task and a knowledge statement that is helpful for answering this question. For a given task, we write five demonstrations using the format in Table 2.
Generated Knowledge Prompting for Commonsense Reasoning
We write questions (or select them from the training set, when available) that are representative of Table 2: Prompts for knowledge generation for two of our tasks, NumerSense and QASC. The prompt consists of an instruction, five demonstrations of question-knowledge pairs, and a new question placeholder. For full prompts on all the tasks we evaluate on, see Appendix A.2. challenges posed by the task (e.g. numerical commonsense, scientific commonsense). We pair each question with a knowledge statement that turns the commonsense problem posed by the question into an explicit reasoning procedure, without directly answering the question. For example, the knowledge statement Birds have two wings. Penguin is a kind of bird. is helpful for the question Penguins have <mask> wings, because it turns the problem into deductive reasoning. Meanwhile, Penguins have two wings. would be a poor knowledge statement to demonstrate according to our guideline.
Generated Knowledge Prompting for Commonsense Reasoning
When generating knowledge for a new question q, we plug the question into the placeholder, and repeatedly sample generated continuations of this prompt to obtain a set of knowledge statements K q = {k 1 , k 2 , . . . , k M }. For full prompts on all the tasks we evaluate on, see Appendix A.2.
Generated Knowledge Prompting for Commonsense Reasoning
In the knowledge integration step, we use a language model -called the inference model -to make predictions with each generated knowledge statement, then select the highest-confidence prediction. Specifically, we use each knowledge statement to prompt the model, forming M knowledgeaugmented questions:
Generated Knowledge Prompting for Commonsense Reasoning
where [·||·] denotes text concatenation.
Generated Knowledge Prompting for Commonsense Reasoning
We compute an aggregated score for each answer choice a using the augmented question that best supports it under the inference model:
Generated Knowledge Prompting for Commonsense Reasoning
Intuitively, this favors knowledge statements that strongly support one of the choices.
Generated Knowledge Prompting for Commonsense Reasoning
The predicted answer is then,
Generated Knowledge Prompting for Commonsense Reasoning
which is the choice that gets most support from one of the knowledge statements. This prediction uses a single knowledge statement, which we refer to as the selected knowledge:
Generated Knowledge Prompting for Commonsense Reasoning
The inference model may be any existing language model taken off-the-shelf (i.e. zero-shot) or finetuned on the task. We do not do any further finetuning with knowledge prompting.
Generated Knowledge Prompting for Commonsense Reasoning
Here, we describe the implementation details of our method and how they are adapted to each task.
Generated Knowledge Prompting for Commonsense Reasoning
For knowledge generation, we use GPT-3 (Brown et al., 2020) as the underlying language model, where our few-shot prompting method is most effective. We generate M = 20 knowledge statements for each question with nucleus sampling p = 0.5 (Holtzman et al., 2019), and discard repetitions and empty strings. Generation is terminated when it exceeds 64 tokens or hits the \n token. 1 For inference, we use off-the-shelf T5 (Raffel et al., 2019) and GPT-3, as well as finetuned models that are state-of-the-art on each dataset, including UnifiedQA (UQA) (Khashabi et al., 2020) and Unicorn (Lourie et al., 2021). See details in the task setup below.
Generated Knowledge Prompting for Commonsense Reasoning
We evaluate our method on four commonsense reasoning datasets which cover a variety of challenges and problem formats.
Generated Knowledge Prompting for Commonsense Reasoning
NumerSense (Lin et al., 2020) consists of numerical statements about common objects and concepts where for each sentence we need to recover a masked number word. The choices are integers ranging from zero to ten, plus the word no, so the task can be framed as a multiple-choice problem. Since NumerSense is a diagnostic dataset, we only use zero-shot inference models, which is the current SOTA. We follow Zhang (2021) who uses the state-of-the-art zero-shot T5 with text-infilling setup and select the choice with highest likelihood on its token(s). We also implement zero-shot GPT-3 inference, where we plug in each choice to the question and compute the choice probability as the generative probability of the entire sentence, normalized over all the choices. CommonsenseQA (CSQA) (Talmor et al., 2019) is a 5-way multiple-choice QA dataset about common world scenarios. We do inference with the zero-shot and finetuned T5 models. For zero-shot T5, we format the question as text-infilling, and predict the choice with highest sequence-to-sequence language modeling probability. For finetuned T5 (including UnifiedQA which is SOTA), we use the same setup as Khashabi et al. (2020). CommonsenseQA 2.0 (CSQA2) (Talmor et al., 2021) is a binary classification dataset where we need to judge whether commonsense statements are true or false. We only do inference with the finetuned model, due to poor calibration of zero-shot models on this dataset. We use finetuned Unicorn (Lourie et al., 2021), which is the current SOTA, following the setup in Talmor et al. (2021). QASC  is an 8-way multiplechoice QA dataset about grade school science. This dataset also includes two pieces of background knowledge per question, whose composition fully answers the question. We do inference with zeroshot T5 and finetuned T5 (including UnifiedQA which is SOTA), using the same setups as CSQA.
Generated Knowledge Prompting for Commonsense Reasoning
Since all the inference models we use (T5, Uni-fiedQA, Unicorn) are generative language models, the support to a choice by the inference model is
Generated Knowledge Prompting for Commonsense Reasoning
and a i is the i-th token of choice a.
Generated Knowledge Prompting for Commonsense Reasoning
We study the impact of our knowledge generation method (shorthanded as K) by comparing with the following baselines:
Generated Knowledge Prompting for Commonsense Reasoning
No knowledge (∅) We refer to inference without any knowledge statements as the vanilla baseline. Random sentences (R) Sampling random sentences from the language model without conditioning on the question. We use the same implementation setup as our knowledge generation method (i.e. also using GPT-3, with the same hyperparameters).
Generated Knowledge Prompting for Commonsense Reasoning
Context sentences (C) Sampling sentences from the context of the question. This is implemented by sampling text continuations of the question from the language model. We use the same implementation setup as our knowledge generation method.
Generated Knowledge Prompting for Commonsense Reasoning
Template-generated knowledge (T ) Self-talk (Shwartz et al., 2020) uses manually-designed templates to elicit knowledge statements from language models. For fair comparison, we use GPT-3 as the knowledge generator in self-talk, and bound the number of generations to M = 20 per question.
Generated Knowledge Prompting for Commonsense Reasoning
Templates and other hyperparameters are kept the same as their original paper.
Generated Knowledge Prompting for Commonsense Reasoning
Retrieval-based knowledge (IR) Instead of being generated, knowledge can be retrieved from appropriate sources. We consider the following retrieval-based methods. For NumerSense, knowledge is retrieved from sentences in Wikipedia and GenericsKB. For CSQA2, we use snippets returned by Google when querying the question. For QASC, we use the associated fact sentences that are used to create each question. Answers (A) Instead of generating knowledge, GPT-3 can be prompted to generate direct answers to questions. In the prompts, we use the same input questions as those in knowledge generation, while replacing the knowledge statement with the ground truth answer. We consider two baselines:
Generated Knowledge Prompting for Commonsense Reasoning
(1) Generate one answer per question and use this to measure the performance of the few-shot GPT-3 inference model;
Generated Knowledge Prompting for Commonsense Reasoning
(2) Generate M = 20 answers per question, and use these answers to prompt the SOTA inference models.
Generated Knowledge Prompting for Commonsense Reasoning
As we will show, our generated knowledge prompting method sets new state-of-the-art results on most  We bold the best and underline the second best numbers. Previous SOTA and retrieval-based methods are also based on the inference model in their corresponding column: * T5-11b 1.1 +digits (Submission by ISI Waltham); ** T5-11b + IR (Yan, 2021); # UQA-11b-ft (Khashabi et al., 2020) (SOTA of single-model methods without referencing ConceptNet); † Unicorn-ft (Talmor et al., 2021); † † Unicorn-ft + Google snippets (Talmor et al., 2021); ‡ UQA-11b-ft (Khashabi et al., 2020).
Generated Knowledge Prompting for Commonsense Reasoning
datasets we evaluate on, and works well under both zero-shot and finetuned settings. In particular, our knowledge generation outperforms naive baselines as well as template-based knowledge generation, and is on-par with retrieval-based systems. Table 3 shows the results on zero-shot and finetuned models following our task setups. New state-of-the-art. We apply our method on top of the same inference model used in the previous state-of-the-art. On NumerSense, we achieve a 6% (66.18 → 72.47) improvement over the previous best method based on the zero-shot T5 model.  Table 3 indicate that our method consistently improves upon the vanilla baseline set by finetuned inference models (though by smaller margins than in the zero-shot settings). Table 3 reports the performance with different knowledge generation baselines. Generally, random sentences barely help and even hurt the inference model, whereas context sentences of the question provide some gain. In contrast, knowledge generated by our method consistently leads to substantial performance improvements, which implies that our knowledge is of high quality.
Generated Knowledge Prompting for Commonsense Reasoning
Knowledge is an essential factor. The few-shot GPT-3 model is poorly calibrated to directly answer commonsense questions, underperforming our best models by 14% to 20% across all tasks. Even when we use answers generated by few-shot GPT-3 to prompt the SOTA inference models, this still significantly falls behind our method on almost all the tasks and models we consider (with one exception -CSQA with T5 inference). Through the medium of knowledge, our method can effectively leverage useful information possessed by GPT-3 to help improve even the SOTA models on various commonsense reasoning tasks.
Generated Knowledge Prompting for Commonsense Reasoning
Our knowledge outperform template generated knowledge. We compare our knowledge generation method with the template-based self-talk on the CSQA dev set. (CSQA is the only task we experiment with that has self-talk templates available.) Our method leads to a larger improvement over the T5-11b baseline than self-talk (by 1.89%), showing that it is better at eliciting helpful knowl-  edge from models.
Generated Knowledge Prompting for Commonsense Reasoning
Our knowledge is comparable with retrievalbased knowledge. On NumerSense, the retrieved knowledge only improves inference performance by 0.18% on test-core and 1.02% on test-all, while our method further outperforms it by 8.83% and 7.37%, respectively. This shows that knowledge retrieved from a loosely-related knowledge base can be far less useful than our generated knowledge. On CSQA2, although we are not able to beat the web-retrieved knowledge, our method still bridges the performance gap without referring to Google search. For QASC, the "retrieved" knowledge is actually gold knowledge from a knowledge base that was used to construct the dataset. As a result, our generated knowledge falls significantly short of the retrieved knowledge. In summary, our generated knowledge is roughly comparable with retrieved knowledge in terms of downstream performance, and is most valuable when there is no appropriate in-domain knowledge base to retrieve from.
Generated Knowledge Prompting for Commonsense Reasoning
Better performance with more knowledge.
Generated Knowledge Prompting for Commonsense Reasoning
We analyze the impact of the number of generated knowledge statements, M , and show the results in Figure 2. Generally, the performance increases with the quantity of knowledge statements. It saturates at M = 20 and begins to decline when more knowledge statements are introduced, which may be because more noisy knowledge is generated.  The knowledge integration method. In addition to the knowledge integration method described in §2.2, we experiment with two alternatives: Mixture-of-Experts (MoE) and Product-of-Experts (PoE) (Hinton, 2002). These make the following modifications to Equation 1, respectively:
Generated Knowledge Prompting for Commonsense Reasoning
The results in Table 4 indicate that our knowledge integration method -i.e. adaptively choosing the best knowledge to rely on -is best among the three. Lightweight inference models and amplification. We found that the size of inference model affects the magnitude of improvement. Figure 3 shows the NumerSense performance gain on top of different sizes of inference model. As we use smaller inference models, the performance gain increases drastically. In particular, with our method the smallest T5 model is as powerful as the T5-3b baseline, and T5-large outperforms the GPT-3 baseline. This indicates that model-generated knowledge can enable high performing, yet lightweight, inference models. Furthermore, the improvement does not diminish as the inference model becomes as big as the knowledge generation model, as the inference by GPT-3 can benefit by 9.0% from the knowledge elicited from itself. This indicates that our method can somewhat amplify the useful knowledge already possessed by the model, leading to better predictions.
Generated Knowledge Prompting for Commonsense Reasoning
The size of knowledge generation model. Figure 4 shows the NumerSense performance gain when using different sizes of GPT-3 as the knowledge generation model. On top of the T5-11b inference model, The 6.7B knowledge model gives a 5.0% improvement, narrower than the 10.5% improvement given by the 175B knowledge model. The 1.3B and 0.4B knowledge models do not give a significant improvement. Therefore, we do not necessarily need the largest version of GPT-3 as the knowledge source, though we do need the model to be relatively large in order to generate useful and reliable knowledge.
Generated Knowledge Prompting for Commonsense Reasoning
We conduct a human evaluation on NumerSense and QASC to study the quality of generated knowledge and the interpretability of its impact on task performance. Evaluation. We report the quality of knowledge statements along four axes: (1) Grammaticality: whether it is grammatical; (2) Relevance: whether it is relevant to the topic or concepts mentioned on the question; (3) Factuality: whether it is (mostly) factually correct; and (4) Helpfulness: whether it helps answering the question in an either direct or indirect way, and may fall into one of the three categories: helpful (i.e. supports the correct answer), harmful (i.e. negates the correct answer or supports an incorrect answer), or neutral (neither helpful nor harmful). These metrics are adapted from Shwartz et al. (2020) and are defined in Appendix A.3.
Generated Knowledge Prompting for Commonsense Reasoning
From each dataset, we sample up to 50 selected knowledge ( §2.2) that change the correctness of T5-11b's prediction (i.e. rectifies model prediction from wrong to right, or misleads model prediction from right to wrong). The knowledge are labeled by two NLP experts and a moderate level of agreement was reached (Fleiss Kappa κ = 0.57 (Landis and Koch, 1977)). To ensure objectivity, it is not revealed to the annotators whether the knowledge rectifies or misleads the model prediction.
Generated Knowledge Prompting for Commonsense Reasoning
Results. Figure 5 summarizes the results. The vast majority of selected knowledge are grammatical and relevant to the question, and 83% of them are factually correct. 72% are seen as being helpful for answering the question according the human evaluators, whereas 13% are harmful. Out of the knowledge statements that rectify the model predictions, 93% are labeled as helpful by the human evaluators; in contrast, when the knowledge statement misleads the model, only 21% are labeled as helpful, and 39% harmful. Of the knowledge deemed helpful by human and rectifies model prediction, 95% are factual, while of those deemed harmful by human and misleads model prediction, 86% are non-factual, suggesting that improving knowledge factuality is a promising path towards more helpful knowledge. We also analyzed the nonselected knowledge and found that these statements have slightly lower factuality and helpfulness than the selected knowledge.  Table 5: More examples where prompting with generated knowledge reduces the reasoning type and rectifies the prediction. The first row of each section is the original question and the inference results associated with it; the second row is a model-generated knowledge statement that prompts the inference model. We show correct answers in green, incorrect answers in red, and their corresponding scores assigned by the inference model.
Generated Knowledge Prompting for Commonsense Reasoning
rect answer, while with knowledge prompting, the correct answer is assigned a much higher score. Prompting with generated knowledge can transform commonsense reasoning into explicit reasoning procedures such as paraphrasing, induction, deduction, analogy, abductive reasoning, logical elimination, negation, and numerical reasoning.
Generated Knowledge Prompting for Commonsense Reasoning
We introduce generated knowledge prompting, a simple method to elicit and integrate knowledge from language models so as to improve performance on commonsense reasoning tasks. In particular, we generate knowledge statements by prompting a language model with task-specific, human-written, few-shot demonstrations of questionknowledge pairs. We show that knowledge can be integrated by simply plugging it in at inference time, with no need to finetune the model for knowledge integration. Our method shows effectiveness across multiple datasets, sets the new state-of-theart on three commonsense reasoning tasks, and works under a variety of settings. The method's success highlights language models as sources of flexible, high-quality knowledge for commonsense reasoning. Table 6 summarizes the comparison between our generated knowledge prompting method and prior methods that add generated text to an inference model for commonsense reasoning tasks. Our method is unique because it uses few-shot demonstrations to prompt for knowledge generation, and can apply to finetuned inference models without joint finetuning with knowledge. Table 7 through 10 shows the full prompts for knowledge generation that we use for each evaluated task: NumerSense, CSQA, CSQA2, and QASC. Table 11 and 12 shows the detailed guidelines we use for human evaluation of generated knowledge.
Generated Knowledge Prompting for Commonsense Reasoning
Limitations. Our method is tested on a representative selection of commonsense reasoning tasks and datasets. Applying this method to other tasks may require people with moderate expertise to craft a task-specific prompt to feed into the method.
Generated Knowledge Prompting for Commonsense Reasoning
Risks. It is possible that our proposed method may lower the performance of commonsense reasoning systems, if not implemented properly or using badly-designed prompts. Such risk can be mitigated by following the prompt design guidelines in this paper ( §2.1).
Generated Knowledge Prompting for Commonsense Reasoning
We do not train any new model in this paper. Inference is conducted on Quadro RTX 8000 GPUs and costs about 200 GPU hours in total. Knowledge generation is done with the OpenAI GPT-3 API, with an approximate cost of $500.
Generated Knowledge Prompting for Commonsense Reasoning
Our method is implemented with PyTorch and the Huggingface Transformers library.
Generated Knowledge Prompting for Commonsense Reasoning
Knowledge Generator Inference Model CAGE (Rajani et al., 2019) task-finetuned joint-finetuned Latcinnik and Berant (2020) task-finetuned joint-finetuned DynaGen  task-finetuned joint-finetuned Self-talk (Shwartz et al., 2020) template-prompted 0-shot Contrastive expl. (Paranjape et al., 2021) template-prompted 0-shot & joint-finetuned Generated knowledge prompting (ours) demonstrations-prompted 0-shot & task-finetuned Table 6: Comparison of methods that add generated text to an inference model. Knowledge Generator: taskfinetuned -a model finetuned to generate task-specific knowledge; template-prompted -an off-the-shelf LM from which knowledge statements are elicited via templates; demonstration-prompted -an off-the-shelf LM from which knowledge statements are elicited via few-shot demonstrations ( §2.1). Inference Model: 0-shot -an off-the-shelf LM that is set up to make predictions; task-finetuned -a model finetuned with task training data (and without seeing extra knowledge); joint-finetuned -a model finetuned with task training data and generated knowledge.  Input: Glasses always fog up. Knowledge: Condensation occurs on eyeglass lenses when water vapor from your sweat, breath, and ambient humidity lands on a cold surface, cools, and then changes into tiny drops of liquid, forming a film that you see as fog. Your lenses will be relatively cool compared to your breath, especially when the outside air is cold.
Generated Knowledge Prompting for Commonsense Reasoning
Input: A fish is capable of thinking. Knowledge: Fish are more intelligent than they appear. In many areas, such as memory, their cognitive powers match or exceed those of 'higher' vertebrates including non-human primates. Fish's long-term memories help them keep track of complex social relationships.
Generated Knowledge Prompting for Commonsense Reasoning
Input: A common effect of smoking lots of cigarettes in one's lifetime is a higher than normal chance of getting lung cancer. Knowledge: Those who consistently averaged less than one cigarette per day over their lifetime had nine times the risk of dying from lung cancer than never smokers. Among people who smoked between one and 10 cigarettes per day, the risk of dying from lung cancer was nearly 12 times higher than that of never smokers.
Generated Knowledge Prompting for Commonsense Reasoning
Input: A rock is the same size as a pebble. Knowledge: A pebble is a clast of rock with a particle size of 4 to 64 millimetres based on the Udden-Wentworth scale of sedimentology. Pebbles are generally considered larger than granules (2 to 4 millimetres diameter) and smaller than cobbles (64 to 256 millimetres diameter).
Generated Knowledge Prompting for Commonsense Reasoning
Knowledge: Table 9: Prompt for knowledge generation on CSQA2. Demonstration examples are selected from the CSQA2 training set; we use the annotated Google featured snippet as the knowledge.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Pretrained Language Models (PLMs) (Raffel et al., 2020;Lewis et al., 2020;Radford et al., 2019;Brown et al., 2020) have been shown to encode substantial amounts of knowledge in their parameters (Petroni et al., 2019;Talmor et al., 2020;Roberts et al., 2020) and have achieved impressive performance on commonsense reasoning (CSR) tasks without the use of external knowledge (Trinh and Le, 2018;Yang et al., 2020). However, these models provide little human-interpretable evidence of the intermediate commonsense knowledge or reasoning they use, and have been observed to overly rely on superficial dataset artifacts (Poliak et al., 2018;Geva et al., 2019). To overcome this limitation, recent work has shown that PLMs can i) I picked up a bag of peanuts and raisins for a snack. I wanted a sweeter snack out so I ate the for now. Contrastive Expl. -Peanuts are salty while raisins tend to be sweet.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
ii) The geese prefer to nest in the fields rather than the forests because in the predators are more hidden. Contrastive Expl. -Forests are denser than fields Table 1: Examples of Winograd Schema Instances where the correct and incorrect answer choices are highlighted in blue and red respectively. Choices are contrasted along attributes like taste (for i) and density of vegetation (for ii) by humans to explain why they prefer some answer choice. explain themselves by generating free-form natural language explanations of their reasoning patterns (Rajani et al., 2019a;Camburu et al., 2018;Narang et al., 2020). However, the space of possible free-form explanations is incredibly large, inherently ambiguous, and difficult to annotate or evaluate (Wiegreffe et al., 2020;Latcinnik and Berant, 2020). Furthermore, quantifying the model's dependence on free-form explanations is also challenging (Camburu et al., 2020). We address these challenges by proposing an unsupervised method that uses contrastive prompts, which require the model to explicitly contrast different possible answers in its explanation (Table 1).
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Our approach is based on a key observation: Many commonsense reasoning tasks require the comparison or contrast of plausible alternatives along a distinguishing attribute. For instance, in Table 1, the differentiating attributes for the two answer choices maybe taste (for i) and vegetation density (for ii). People commonly use contrastive explanations to explain their reasoning (Miller, 2018). Rather than asking "Why P?", they ask "Why P rather than Q?", where Q may be implicit from the context. For example, instead of justifying why raisins are the appropriate choice, people tend to ex-plain why they are more likely than peanuts. Miller (2018) also argues that such contrastive explanations are computationally efficient, as they only require focusing on the limited set of reasons that might make one answer more likely than the other instead of exhaustively enumerating all possible reasons for an answer. For instance, the raisin's taste (not its size, temperature, etc.) in Table 1 is adequate to explain why it is the best answer.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Our goal is to enable PLMs that explain their predictions to similarly benefit from such constraints. We develop a small set of contrastive generation prompts that can be in-filled by a PLM such as T5 (Raffel et al., 2020) or BART (Lewis et al., 2020) (see Table 3). These templates are designed to cover a multitude of language patterns used by humans to compare and contrast entities. Another PLM then conditions on both the original input and the generated contrastive explanation, to predict the final answer. This approach is inspired by Shwartz et al. (2020), who also use textual prompts to query the PLM with clarification questions. However, their prompts are generic while we prompt for instance-specific information.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Our approach shows quantitative improvements in task performance over two existing methods for model explainability (Shwartz et al., 2020;Latcinnik and Berant, 2020), for two commonsense reasoning tasks: the Winograd Schema Challenge (Levesque et al., 2012) and multiple-choice question answering about physical commonsense (Bisk et al., 2020). Our gains in the zero-shot setting are especially notable, outperforming the best reported results on publicly available PLMs and improving over Shwartz et al. (2020) by up to 11%. We also show, through human evaluations, that contrastive explanations are deemed more useful for solving the original task compared to generic clarification questions. Finally, contrastive explanations can be semantically perturbed to quantify the model's dependence on them by flipping the contrast in the explanation to support the foil, facilitating quantification of model faithfulness. 1
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Human-Authored Contrastive Explanation Winograd Schema 1. The party was more interesting and uplifing than the • Parties are for celebrating while funerals are for mourning funeral because the was rigid.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
• People wear colorful clothes at parties and black at funerals 2. The geese prefer to nest in the fields rather than the • Forests are dense while fields are sparse forests because in the predators are more hidden.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
• Forests have more predators than fields.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
PIQA 1. How do you get strong hamstrings?
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
• Hamstrings are located in the legs while biceps are located in (a) work out your upper body (b) work out your legs the upper body 2. How do you flood a room?
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
• Filling it with objects can clutter a room while filling it (a) fill it with objects (b) fill it with water with water floods the room. Table 2: Examples of commonsense tasks that can be explained using contrastive language and some contrastive explanations authored by in-house annotators. The Fact and Foil are marked in the input.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
We present the theory of contrastive explanations adopted in this work (Section 3.1) and the intuition behind using them for commonsense reasoning tasks (Section 3.2).
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
A contrastive explanation is generally defined as an answer to a counterfactual question of the form "Why P rather than Q?" for two potential hypotheses P and Q that can follow from some event E. It explains why some fact P occurred instead of some foil Q, where Q can be implicit (Hesslow, 1988;Lipton, 1990;Miller, 2019). A good contrastive explanation points to differences between the fact and foil with regard to certain attributes, not just conveying that the fact has a certain attribute. Table  1 shows examples of contrastive explanations that differentiate between peanuts and raisins (on the basis of taste) or forests and fields (on the basis of vegetation densities) to explain the more probable answers to Winograd Schema instances. Previous studies (Miller, 2019) in philosophy, psychology, and cognitive science show that humans use such contrastive explanations when explaining their decisions to each other. Importantly, Miller (2018) also argues that contrastive explanations are computationally efficient -exhaustively describing all causes for the occurrence of an event P is harder than only enlisting causes for why another event Q did not occur instead of P .
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Many recently proposed commonsense reasoning tasks are framed in a multiple-choice format that facilitates contrastive explanation (see Table 2). In this study, we focus on the following two tasks.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
The Winograd Schema Challenge (Levesque et al., 2012, WSC) is a pronoun coreference resolution task designed as a hard benchmark for evaluating everyday knowledge and commonsense reasoning (Zhang et al., 2020). For instance, in the sentence "The city councilmen refused the demonstrators a permit because they feared violence," the pronoun they must be disambiguated between fact (the city councilmen) and foil (the demonstrators). Both fact and foil are explicit in such sentences.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
The Physical Interaction Question Answering (Bisk et al., 2020, PIQA) challenge is designed to test knowledge of physical commonsense. PIQA requires choosing between which one of two solutions is a better way of achieving a goal posed as a question (see Table 2). PIQA questions relate to physical properties of entities, their affordances, and how they can be manipulated. The fact and foil are explicit in the two solutions, which typically differ from one another by a short noun phrase.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
To validate our intuition that contrastive reasoning is instrumental in these tasks, we performed a pilot study with 10 annotators over 100 commonsense questions from Winogrande and PIQA. We instructed them to answer the questions and explain their reasoning, but gave no specific instructions about what the explanations should look like. Examples are shown in Table 2. In 76% of Winogrande and 64% of PIQA examples, annotators explicitly contrasted the fact and foil. The frequent use of certain phrase structures, like P are while Q are , strongly informed our method for generating them (Section 4).
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
We assume the input to a commonsense reasoning problem consists of a textual context c which contains a placeholder , and two marked answer Prompt Pattern
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Personal Characteristics =⇒ P likes/likes to while Q likes/likes to Megan said it would be liberating to go out without makeup like P likes/likes to while Q does not like/like to Elena does since never wore makeup P prefers/prefers to while Q prefers Explanation: Elena likes to be natural while Q prefers while P does not prefer/prefer to Megan likes to wear lipstick Q thinks while P thinks/does not think Object Characteristics P is taller/shorter/smaller/larger/slower/faster than Q How to tie pieces of paper together? =⇒ P is/are while/but/however Q is/are (a) Thread ruler through the holes Q has/have while/but/however P has/have (b) Thread ribbon through the holes P has/have more/less than Q Explanation: Ruler is hard while a ribbon is P is/are than Q flexible Spatial/Temporal Contrast =⇒ P is inside/outside/above/below Q Emily looked up and saw Patricia racing by overhead. was on the is closer to P and farther away from Q ramp. P is to the right/left of Q Explanation: Emily is below Patricia Q takes longer to than P Use cases and causes P is used for Q To prepare the puff pastry for your pie, line a baking sheet with P is used to do Q parchment. Then =⇒ P is used for/to/in while Q is used for/to/in (a) Unroll the pastry, lay it over baking twine. Q is used while P is used (b) Unroll the pastry, lay it over fishing line. Q because while P because Explanation: Baking twine is used in Q can cause while P results in baking while fishing line is used in fishing choices a 1 and a 2 corresponding to the fact and foil ( Table 2, left column). Let c x denote substitution of x for the placeholder in c. The task is to predict whether c a 1 or c a 2 is more likely to be true, i.e., whether a 1 or a 2 best completes the context. Our approach has two stages: First, an Explainer PLM P expln generates contrastive explanations (Section 4.2) by infilling preset contrastive templates (Sec. 4.1) on the basis of c, a 1 , and a 2 . Then, a Task Model P LM selects the correct answer conditioned on both the context and the generated explanations (Sec. 4.3).
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
We develop a list of contrastive templates on the basis of an annotation study. For 250 instances from Winogrande and PIQA, we asked three annotators to explain why one answer is more likely than the other. We manually examined these explanations and abstracted them into templates containing at least two placeholders: two for the fact and foil being contrasted, and possibly more corresponding to the properties they are being contrasted on. For instance, peanuts are salty while raisins are sweet becomes Q are while P are . We retained templates used by annotators at least 10 times. Table 3 shows several examples. A template is converted into an explanation by replacing placeholders for the fact and foil with answers a 1 and a 2 and the remaining placeholders with the appropriate contrastive information.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
We evaluate the quality and coverage of our templates with another round of human evaluation. For 100 WSC and PIQA examples, we ask three annotators to either write contrastive explanations using one or more of the templates, or indicate that none of the them were appropriate. Annotators used the templates in over 82% of cases, indicating high coverage for the tasks we study.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Let t denote a contrastive template. We write t a 1 ,a 2 to denote the customization of t to an input by filling its marked placeholders for fact and foil with the answer choices. For instance, in Figure 1, the template P are while Q are is customized to Fields are while forests are . 2 A full explanation may be produced by filling the remaining gaps in t a 1 ,a 2 by leveraging an infilling language model, the explainer P expln .
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
We first construct a neutral context c a 0 by filling c's placeholder with a task-specific neutral answer  Figure 1: (1) A commonsense reasoning instance (c, a 1 , a 2 ) is converted into a custom prompt c a0 ⊕ t a1,a2 as input for the explainer PLM (2) The combination of input and explanation (c ai ⊕ e j ) is used by task model to score a i ∀i∀j. For a 1 and a 2 , scores are aggregated over templates.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
that does not indicate if a 1 or a 2 is correct. For Winogrande Schemas, c a 0 is constructed using the ambiguous pronoun in c (them in Figure 1). For PIQA, c a 0 is constructed as "c ⊕ a 1 or a 2 ", where ⊕ is string concatenation, e.g., upper body or legs in Figure 1 (More dataset-specific details are in Section 5.2). We then prepend c a 0 to the customized template t a 1 ,a 2 and use it as input to the infilling language model to fill in the remaining gaps in the template. We use the maximum likelihood candidate phrases from top-K decoding to transform the template into a full explanation e.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
We use a list of templates t 1 , . . . , t n to generate a list of candidate explanations e 1 , . . . , e n for each input, which are all fed into the task model. We also use some task-specific heuristics to reduce the number of prompts for each example, detailed in Appendix A.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Given the context and answer choices (c, a 1 , a 2 ) and a list of explanations e 1 , . . . , e n , the second stage of our pipeline is a binary classifier between a 1 and a 2 which marginalizes over the explanations. We first assign a score to each answer a ∈ {a 1 , a 2 } and explanation e ∈ {e 1 , ..., e n }:
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
where c a denotes the substitution of a into c, P LM is string probability under the task language model, and k is the string length of c a ⊕ e. We use φ as input to a logistic regression classifier which marginalizes over explanations:
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
where Z is a normalizer over a 1 and a 2 . At initialization, φ uses a pretrained language model, and we fine-tune it to minimize the cross-entropy loss of P(a * | c, a 1 , a 2 ), where a * is the correct answer. We do not fine-tune the explainer PLM since the top-K beam decoding is a discrete operation that is hard to backpropagate through. In the zero-shot setting (where the task PLM is not fine-tuned) and during inference, the answer is predicted by aggregating scores assigned to an answer by all n explanations: argmax a i j φ(c, a i , e j ).
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Context-Only We experiment with a baseline that does not condition on explanations at all. Here,
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
and gold answer is argmax a i φ(a i , c)
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Unconstrained Generation Latcinnik and Berant (2020) generate explanations from a PLM by beam-decoding a free-form sequence termed a hypothesis which is then used by a classifier to solve the task. The model is trained end-to-end and loss terms are added to encourage the hypothesis to sound natural. Explanation generation is otherwise unconstrained. For fair comparison with our approach, we do not fine-tune the explainer PLM (more details are in Appendix C). Shwartz et al. (2020) propose an unsupervised model that uses a PLM as the answer scorer and a (possibly different) PLM as a knowledge source, similar to our framework. They formulate the process of obtaining relevant knowledge as self-talk with the following steps: 1) completing clarification question prefixes such as "what is the definition of ..." conditioned on input context, 2) generating their corresponding answers (clarifications), and 3) conditioning on the clarification questions and answers to make predictions. The key difference between their approach and ours is in the choice of prompts for the PLM, and the kinds of knowledge the prompts seek. While Shwartz et al.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
(2020) draw inspiration from inquiry-based discovery learning (Bruner, 1961), we target contrastive reasoning.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
We Each instance provides two answer choices, which we use directly as a 1 and a 2 . For the neutral answer c a 0 , we use the sentence with the original ambiguous pronoun. Since Winogrande has a blank space for the answer, we replace it with the most likely pronoun under a masked language model (BERT), following Shwartz et al. (2020). c a 1 , c a 2 are obtained by replacing the blank space or pronoun with the answer choice.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Physical Interaction Question Answering (PIQA) (Bisk et al., 2020) PIQA provides two answer choices which mostly vary from each other on a substring (e.g., "work out your [upper body]/[legs]"). We use these differing substrings as a 1 =legs and a 2 =upper body. For the neutral answer a 0 , we combine the answers into "a 1 or a 2 " (upper body or legs). In the cases where a 1 or a 2 is longer than 2 words, we include an or between the full answers. More details and examples are presented in Appendix A. We use question-answer pairs for c a 1 and c a 2 .
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
In this section, we present an extensive evaluation of our approach, demonstrating performance gains which are independently verified by human judges.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
We report task accuracy as a proxy for explanation quality. Table 4 compares the task performance of our model with the baselines defined in Section 5.1. We observe that generating and conditioning on additional information from PLMs improves performance over just using the original input (Row 1 vs. 2-6). Using templates to prompt the PLM for specific knowledge is better than unconstrained generation of text (Row 2 vs. 3-6). Contrastive explanations outperform previous work that use clarification questions in self-talk (Shwartz et al., 2020). The T5-Large explainer already surpasses the results of self-talk despite being smaller than GPT2-XL, demonstrating the impact of using contrastive explanations over clarification questions.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
We also observe that larger explainer PLMs (going from T5-Large to T5-11B) yield higher performance. Our zero-shot results with T5-11B are the highest reported on Winogrande, PIQA and WSC for an open-sourced model. 3 Finally, our approach gets smaller improvements when finetuning the task model. This suggests that some of the reasoning is still learned implicitly by the task model. Figure 2 shows task performance with various training data sizes of Winogrande, indicating a larger gap between the Context-Only baseline and our approach when training data is scarce.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Setup Following the human evaluation setup of Shwartz et al. (2020), we sample up to 50 highest-    Crowd workers are presented with a commonsense instance, the correct answer, and an explanation, and are asked to judge for: 1) Grammaticality, whether the explanation is grammatical; 2) Relevance, whether it's relevant to the topic of the text; 3) Factual Correctness, whether it's factually correct or likely true; and 4) Helpfulness, whether it adds helpful evidence for the correct answer. These metrics and definitions follow from Shwartz et al.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
(2020) with more details in Appendix B. The annotators are also shown the same explanation with fact and foil flipped (details in Section 6.3) and are asked to judge if the other answer is more likely than before if they assume the flipped explanation to be hypothetically true.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Results Table 5 shows the results of human evaluation of contrastive and self-talk explanations. The contrastive explanations are overwhelmingly preferred over self-talk explanations for relevance, factual correctness and helpfulness. They may be con-sidered less grammatical because of in-filling noise (such as incomplete phrases). Table 6 presents some qualitative examples of instances where contrastive explanations improve over all baselines.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
We also analyze how much the task model relies on contrastive explanations for its decisions.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Flipping Explanations Our choice of contrastive language templates facilitates a novel way to evaluate explanation usefulness in prediction. The contrast in the explanation can be reversed by flipping the position of the fact and the foil in the explanation. If the choice between fact and foil actually depends on the contrastive explanation, then the flipped explanation should provide a hypothetical situation where the foil is more likely than the fact. For instance, "peanuts are salty while raisins are sweet," when switched to "raisins are sweet while peanuts are salty," may provide evidence that peanuts is a more likely label for the example in Table 1 (i). This may cause a model that uses the explanation to flip its prediction and lead to a drop in accuracy. The magnitude of drop can quantify the extent to which the model relies on the contrast    provided in the explanation. In fact, humans also deem the flipped explanation to imply the opposite label in a majority of cases (Table 5), indicating that our contrastive explanations frequently capture contrastive properties that the labels truly rely on. Table 7 shows the flipped evaluation results. We observe declines in accuracy of up to 8%, indicating that the model does use some contrastive knowledge to reason about the task. Finetuned models show a smaller decline in accuracy compared to the zero-shot setting. In this case, the task model may be directly fitting the data in lieu of relying on the knowledge conveyed by the explanation.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Abstracting Fact and Foil Given input context c (consisting of the fact and foil a 1 , a 2 ) and an explanation e, the explainer PLM P expl infills its explanation e on c while the task model P LM conditions on both c and e. We can test the quality of the generated explanations and the task model's reliance on them by forcing the task model to rely on e when information in input c is restricted. One potential way to do so is to scrub the identities of the fact and foil, a 1 and a 2 , from c.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
We replace the fact and foil with placeholder tokens to create an abstract context c . For instance, the example in Table 6 (ii) becomes "The <mask1> and <mask2> helped me navigate ... down.", where the model must now choose between <mask1> and <mask2>. 4 Running the task model on c lower-bounds the performance possible without knowing answer identities. We can now test the relevant answer-based knowledge contrastive contained in the explanations by allowing the explanation model to see the original answers in c, but then abstracting them out when passing the input context and explanations to the task model. More formally, the task model conditions its decision on c and e . For Table 6 (ii) c and e are "The <mask1> and <mask2> helped me navigate ... down." and "The <mask1> is right-side-up while the <mask2> is upside down." Since only the explainer PLM is shown answer identities, the task model's decision is conditionally independent of the answer identities given the explanation.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Experiments on Winogrande and PIQA in the Banerjee and Baral (2020a) 38.8 Table 9: Zero-shot test performance on Common-senseQA for baselines as well as contrastive models which ensemble fact/foil pairs by voting (V) and maximum margin (MM). The best reported unsupervised performance (Banerjee and Baral, 2020b) uses Con-ceptNet, which was used to construct the dataset.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
fine-tuned setting (Table 8) show that performance improves significantly when the task model conditions on both c and e compared to a fully abstracted contrastive baseline that only conditions on c (from 63.2 to 70.4 for Winogrande), covering almost half of the gap between the fully abstracted setting and the non-abstracted original model (79.1). This indicates that our contrastive explanations encode a significant amount of information required for commonsense tasks. Even if the full model does not always use the explanations, these evaluations show that our contrastive explanations contain rich task-relevant knowledge, and suggest that future work might focus on how to better make use of this signal.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
The set of contrastive prompts used in our framework are curated from an in-house analysis of training instances from Winogrande and PIQA datasets.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
To determine the generalizability of these prompts for other commonsense reasoning tasks, we also experiment with the CommonsenseQA dataset (Talmor et al., 2019), which consists of multiple-choice questions created over ConceptNet -"Where on a river can you hold a cup upright to catch water on a sunny day? a) waterfall, b) bridge, c) valley, d) pebble, e) mountain". Since there are more than two answer choices to contrast, we convert each instance into 10 pairwise (binary) classification instances. Contrastive explanations are generated for each pairwise decision in the zero-shot setting, similar to Winograd and PIQA datasets. To choose the final answer, we consider two inference procedures: (a) Vote: The answer that receives the maximum number of votes across all binary clas-sification instances is selected, and (b) Maximum Margin: The choice with the maximum difference (margin) between answer likelihoods for any binary classification instance is selected. In Table  9, we observe that self-talk significantly hurts performance for this dataset. On the other hand, contrastive explanations are found to be useful and approach the zero-shot performance of the stateof-the-art, which uses ConceptNet (Banerjee and Baral, 2020b). These results demonstrate that the set of contrastive prompts are generalizable to other commonsense reasoning datasets, and that while our contrastive prompts are limited to contrasting two answer choices at a time, the framework can be easily extended to tasks with multiple foils.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
We show it is possible to prompt pretrained language models (PLMs) to generate contrastive explanations of their reasoning patterns, inspired by explanations that humans naturally provide for their reasoning. Conditioning model decisions on these explanations improves performance on two commonsense reasoning benchmarks, and humans judge the explanations to be highly relevant and helpful in comparison to prior work. We also showed how contrastive explanations can facilitate in-depth evaluations of faithfulness by flipping or abstracting the fact and foil, finding that our explanations encode a significant amount of information relevant to the classification decision, and in many cases models rely on the contrast in the expected way. While we have shown that our method is flexible enough to apply to multiple-choice commonsense tasks with many foils, leveraging contrastive reasoning in a wider variety of open-ended tasks remains an exciting challenge for future work. A Generating Contrastive Templates Table 12 shows the complete list of contrastive patterns used in our work, categorized under different types of attributes/properties. For templates with no place holders for the explainer to fill out, we only replace placeholders for answers (fact and foil). Table 10 lists a 0 , a 1 , a 2 , c a 0 , c a 1 , c a 2 for different examples in Winogrande and PIQA to explain dataset specific transformations made by our approach. Detection of P , Q: For WSC, the fact and foil are typically 1-word nouns. However, they may by qualified in the context and these qualifiers are important for contrasting. For instance, in the WSC example "She remembered how annoying it is to dust her wood chair so she bought a plastic 
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
The annotation task was carried out in Amazon Mechanical Turk, following (Shwartz et al., 2020).
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
To ensure the quality of annotations, workers were required to be located in the US, UK, or Canada, and have a 99% approval rate for at least 5000 prior tasks. Annotators were paid .30$ per HIT to ensure participants get approximately $15/hr if they are doing the task. Annotation were aggregated from 3 workers using majority vote. The annotations yielded moderate levels of agreement, with Fleiss Kappa κ = 0.43 (Landis and Koch, 1977).
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Explainer PLM For T5 we use special symbols <extra id 0> and <extra id 1> in place of the blanks ( ) in our templates. We observe that T5 is able to replace these tokens with multi-word phrases. For BART, we substitute blanks with a sequence with four [MASK] tokens to encourage generating multiple words. BART can choose to delete a [MASK] token during generation. Top-K decoding was done with a beam size of 200 and a maximum output sequence length of 20 for T5 models and 100 for BART. This is because both T5 is pre-trained to in-fill by only generating missing phrases while BART is pre-trained to decode the entire input with missing phrases filled in. We used early stopping for BART.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Task PLM Task PLM was finetuned for 20 epochs, using BertAdam optimizer with a learning rate of 2e − 5, batch size of 8, and dropout of 0.1, following (Latcinnik and Berant, 2020).
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Self-Talk (Shwartz et al., 2020) generate multiple clarification questions conditioned on the context, by 1) concatenating one of several question prefixes to the input prompt or question; and 2) generating 5 questions for each prefix using Nucleus sampling with p = 0.2, i.e., sampling from the top 20% tokens (Holtzman et al., 2019) limiting the question length to up to 6 tokens excluding the prefix. For each well-formed question, they generate multiple answers using a similar method. They Winogrande Ian volunteered to eat Dennis's menudo after already having a bowl because despised eating a1 : Ian a2 : Dennis a0 : he ca 0 : Ian volunteered to eat Dennis's menudo after already having a bowl because he despised eating ca 1 : Ian volunteered to eat Dennis's menudo after already having a bowl because Ian despised eating ca 2 : Ian volunteered to eat Dennis's menudo after already having a bowl because Dennis despised eating PIQA (difference between answers is 1-2 words)
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
To prepare carrots before cooking with them, you can a1 : Run them in the sink under boiling water a2 : Run them in the sink under cold water a0 : boiling water or cold water ca 0 : To prepare carrots before cooking with them, you can run them in the sink under boiling water or cold water ca 1 : To prepare carrots before cooking with them, you can run them in the sink under boiling water ca 2 : To prepare carrots before cooking with them, you can run them in the sink under cold water PIQA (difference between answers is larger)
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
To prevent gunk buildup in cup holders of a car, a1 : place coffee filters inside of the cup holders. a2 : pour a thin layer of oil into the cup holders. a0 : place coffee filters inside of the cup holders or pour a thin layer of oil into the cup holders. ca 0 : To prevent gunk buildup in cup holders of a car, place coffee filters inside of the cup holders or pour a thin layer of oil into the cup holders ca 1 : To prevent gunk buildup in cup holders of a car, place coffee filters inside of the cup holders ca 2 : To prevent gunk buildup in cup holders of a car, pour a thin layer of oil into the cup holders Table 10: Examples of Winogrande and PIQA, with fact, foil , neutral answer and respective substituted contexts used in our approach for prompting the explainer PLM or computing answer likelihood.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Original Input: The geese prefer to nest in the fields rather than the forests because in the predators are more hidden.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
(i) Context-Only Input to task model: The geese prefer to nest in the <mask1> rather than the <mask2> because in the predators are more hidden.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
(ii) Fully Abstracted Input to explainer: The geese prefer to nest in the <mask1> rather than the <mask2> because in the predators are more hidden. Generated Explanation: <mask1> is smaller than <mask2> Input to task model: The geese prefer to nest in the <mask1> rather than the <mask2> because in the predators are more hidden. <mask1> is smaller than <mask2> (iii) Abstraction after Explanation Input to explainer: The geese prefer to nest in the fields rather than the forests because in the predators are more hidden. Generated Explanation: Forests have more predators than fields Input to task model: The geese prefer to nest in the <mask1> rather than the <mask2> because in the predators are more hidden. <mask2> have more predators than <mask1> Table 11: Input to Explainer and Task model for Abstractive Evaluation limit the answer length to 10 generated tokens, and use Nucleus sampling with p = 0.5. Shwartz et al.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
(2020) only condition task prediction on a single clarification question and answer pair that increases the model's belief of a certain answer choice. Thus, the score of each answer choice is selected as the score of the text containing the clarification that most supports it, i.e., whose combination with it yields maximum language model likelihood.
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Unconstrained Generation For unconstrained explanation baseline, maximum output sequence length was set to 20 and beam size for beam decoding was set to 200. Again we use early stopping. WSC and PIQA OPT1 is/are smaller than OPT2 OPT1 is/are larger than OPT2 OPT1 is/are slower than OPT2 OPT1 is/are faster than OPT2 OPT1 is than OPT2 OPT1 are than OPT2 OPT1 is while OPT2 is OPT1 is but OPT2 is OPT1 is however OPT2 is OPT1 are while OPT2 are OPT1 are but OPT2 are OPT1 are however OPT2 are OPT1 has while/but/however OPT2 has/does not have OPT1 have while/but/however OPT2 have/do not have OPT1 is made of/to however OPT2 is made of/to OPT1 is made of/to while OPT2 is made of/to Spatial:
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
WSC and PIQA OPT1 is above OPT2 OPT1 is below OPT2 OPT1 is to the right of OPT2 OPT1 is to the left of OPT2 OPT1 is inside OPT2 OPT1 is outside OPT2 is closer to OPT1 and father away from OPT2 OPT1 is closer to while OPT2 is father away from Usecase:
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
WSC(No PERSON entity) and PIQA OPT1 can while OPT2 can/cannot OPT1 is/can be used for OPT2 OPT1 is/can be used to do OPT2 OPT1 is/can be used for but OPT2 cannot OPT1 is/can be used for while OPT2 is used for OPT1 is/can be s used for but OPT2 is used for OPT1 is/can be used to while OPT2 is used to OPT1 is/can be used to but OPT2 is used to Causes:
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
WSC (No PERSON entity) and PIQA OPT1 has because while OPT2 is because OPT1 can cause while OPT2 causes/results in Since it can OPT1 but not OPT2 Since it can OPT1 but because it is not it can't OPT2
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
WSC (No PERSON entity) and PIQA can be OPT1 but cannot be OPT2 OPT1 means to while OPT2 means to OPT1 is defined as while OPT2 is defined as OPT1 OPT2 OPT1 but not OPT2 OPT1 exists while an OPT2 doesn't Table 12: Complete list of contrastive patterns used in this work.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Large neural language models (LMs) have yielded state-of-the-art performance on various natural language processing (NLP) tasks (Devlin et al., 2018;Liu et al., 2019). However, LMs' complex reasoning processes are notoriously opaque (Rudin, 2019), posing concerns about the societal implications of using LMs for high-stakes decision-making * Work done while AC was a research intern at Meta AI.  (Bender et al., 2021). Thus, explaining LMs' behavior is crucial for promoting trust, ethics, and safety in NLP systems (Doshi-Velez and Kim, 2017;Lipton, 2018). Given a LM's (i.e., task model's) predicted label on a text classification instance, an extractive rationale is a type of explanation that highlights the tokens that most influenced the model to predict that label (Luo et al., 2021). Ideally, rationale extraction should be faithful (Ismail et al., 2021;Jain et al., 2020) and plausible (DeYoung et al., 2019), without hurting the LM's task performance (DeYoung et al., 2019) (Fig. 1).
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Configuring the rationale extractor and its training can greatly impact these desiderata, yet prior works have commonly adopted two suboptimal heuristics. First, many works rely in some way on attribution algorithms (AAs), which extract rationales via handcrafted functions (Sundararajan et al., 2017;Ismail et al., 2021;Situ et al., 2021). AAs cannot be directly trained and tend to be computeintensive (Bastings and Filippova, 2020). Also, AAs can be a bottleneck for plausibility, as producing human-like rationales is a complex objec-tive requiring high capacity rationale extractors (Narang et al., 2020;DeYoung et al., 2019). Second, many works use a specialized select-predict pipeline (SPP), where a predictor module is trained to solve the task using only tokens chosen by a selector module (Jain et al., 2020;Yu et al., 2021;Paranjape et al., 2020). Instead of faithfulness optimization, SPPs heuristically aim for "faithfulness by construction" by treating the selected tokens as a rationale for the predictor's output (which depends only on those tokens). Still, SPPs typically have worse task performance than vanilla LMs since SPPs hide the full input from the predictor.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
To tackle this challenge, we propose the UNIfied Learning Framework for Rationale EXtraction (UNIREX), which generalizes rationale extractor optimization as follows: (1) specify architecture for a learned rationale extractor; (2) select explainability objectives (i.e., faithfulness and plausibility criteria); and (3) jointly train the task model and rationale extractor on the task using selected objectives (Sec. 3). UNIREX enables replacing prior works' heuristic design choices in (1) with a generic learned rationale extractor and optimizing it for all three desiderata in (2)-(3).
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
UNIREX provides significant flexibility in performing (1)-(3). For (1), any model architecture is applicable, but we study Transformer LM based rationale extractors in this work (Zaheer et al., 2020;DeYoung et al., 2019). We focus on two architectures: (A) Dual LM, where task model and rationale extractor are separate and (B) Shared LM, where task model and rationale extractor share parameters. For (2), any faithfulness and plausibility criteria can be used. Following DeYoung et al. (2019), we focus on comprehensiveness and sufficiency as faithfulness criteria, while using similarity to gold rationales as plausibility criteria. For (3), trade-offs between the three desiderata can be easily managed during rationale extractor optimization by setting arbitrary loss weights for the faithfulness and plausibility objectives. Plus, though computing the faithfulness criteria involves discrete (nondifferentiable) token selection, using Shared LM can approximate end-to-end training and enable both task model and rationale extractor to be optimized w.r.t. all three desiderata (Sec. 3.3).
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
To evaluate all three desiderata in aggregate, we introduce the Normalized Relative Gain (NRG) metric. Across five English text classification datasets -SST, Movies, CoS-E, MultiRC, and e-SNLI (Carton et al., 2020;DeYoung et al., 2019)our best UNIREX configuration outperforms the strongest baselines by an average of 32.9% NRG (Sec. 4.2), showing that UNIREX can optimize rationale extractors for all three desiderata. In addition, we verify our UNIREX design choices via extensive ablation studies (Sec. 4.3). Furthermore, UNIREX-trained extractors have high generalization power, yielding high plausiblity with minimal gold rationale supervision (Sec. 4.4) and high faithfulness on unseen datasets and tasks (Sec. 4.5).
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Finally, our user study shows that humans judge UNIREX rationales as more plausible than rationales extracted using other methods (Sec. 4.6).
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Rationale Extraction Let F task = f task (f enc (·)) be a task model for M -class text classification (Sec. A.1), where f enc is the text encoder and f task is the task output head. Typically, F task has a BERT-style architecture (Devlin et al., 2018), in which f enc is a Transformer (Vaswani et al., 2017) while f task is a linear layer with softmax classifier. Let x i = [x t i ] n t=1 be the n-token input sequence (e.g., a sentence) for task instance i, and F task (x i ) ∈ R M be the logit vector for the output of the task model. Letŷ i = arg max j F task (x i ) j be the class predicted by F task . Given F task , x i , andŷ i , the goal of rationale extraction is to output vector s i = [s t i ] n t=1 ∈ R n , such that each s t i ∈ R is an importance score indicating how much token x t i influenced F task to predict classŷ i . Let F ext be a rationale extractor, such that s i = F ext (F task , x i ,ŷ i ). F ext can be a learned or heuristic function. In practice, the final rationale is often obtained by binarizing s i as r i ∈ {0, 1} n , via the top-k% strategy: DeYoung et al., 2019;Jain et al., 2020;Pruthi et al., 2020;Chan et al., 2021). For top-k%, let r (k) i be the "important" (i.e., ones) tokens in r i , when using 0 ≤ k ≤ 100.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Faithfulness means how well a rationale reflects F task 's true reasoning process for predictingŷ i (Jacovi and Goldberg, 2020). Hence, faithfulness metrics measure how much the r (k) i tokens impact pŷ i (x i ), which denotes F task 's confidence probability forŷ i when using x i as input (DeYoung et al., 2019;Shrikumar et al., 2017;Hooker et al., 2018;Pruthi et al., 2020) 
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
High faithfulness is signaled by high comp and low suff.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Plausibility means how convincing a rationale is to humans (Jacovi and Goldberg, 2020). This can be measured by automatically computing the similarity between F ext 's rationales (either s i or r i ) and human-annotated gold rationales (DeYoung et al., 2019), or by asking human annotators to rate whether F ext 's rationales make sense for predictingŷ i (Strout et al., 2019;Doshi-Velez and Kim, 2017). Typically, a gold rationale is a binary vector r * i ∈ {0, 1} n , where ones/zeros indicate important/unimportant tokens (Lei et al., 2016).
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Task Performance, w.r.t. rationale extraction, concerns how much F task 's task performance (on test set) drops when F task is trained with explainability objectives (i.e., faithfulness, plausibility) for F ext . As long as F task is trained with non-task losses, F task 's task performance can be affected.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Given task model F task , UNIREX generalizes rationale extractor optimization as follows: (1) choose architecture for a learned rationale extractor F ext ; (2) select explainability objectives (i.e., faithfulness loss L faith and plausibility loss L plaus ); and (3) jointly train F task and F ext using L task (task loss), L faith , and L plaus . UNIREX training consists of two backpropagation paths (Fig. 2). The first path is used to update F task w.r.t. L task and L faith . Whereas L task is computed w.r.t. the task target y i , L faith is computed only using the task input x i and the top-k% important tokens r (k) i (obtained via F ext ), based on some combination of comp and suff (Sec. 2). The second path is used to update F ext w.r.t. L plaus , which encourages importance scores s i to approximate gold rationale r * i . Thus, UNIREX frames rationale extraction as the following optimization problem:
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
where α f and α p are loss weights. If F task and F ext share parameters, then the shared parameters will be optimized w.r.t. all losses. During inference, for task input x i , we first use F task to predict y i , then use F ext to output a rationale r i for F task 's predictionŷ i . Below, we discuss options for the rationale extractor and explainability objectives.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
In UNIREX, F ext is a learned function by default. Learned F ext can be any model that transforms x t i into s t i . Given their success in NLP explainability (DeYoung et al., 2019), we focus on pre-trained Transformer LMs and highlight two architectures: Dual LM (DLM) and Shared LM (SLM) (Fig. 3). For DLM, F task and F ext are two separate Transformer LMs. DLM provides more dedicated capacity for F ext , which can help F ext output plausible rationales. For SLM, F task and F ext are two Transformer LMs sharing encoder f enc , while F ext has its own output head f ext . SLM leverages multitask learning between F task and F ext , which can improve faithfulness since F ext gets more information about F task 's reasoning process. Unlike heuristic F ext (Sec. A.2), learned F ext can be optimized for faithfulness/plausibility, but cannot be used out of the box without training. Learned F ext is preferred if: (A) optimizing for both faithfulness and plausibility, and (B) gold rationales are available for plausibility optimization (Sec. A.3).
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
After selecting F ext , we specify the explainability objectives, which can be any combination of faithfulness and plausibility criteria. In prior approaches (e.g., AA, SPPs), the rationale extractor is not optimized for both faithfulness and plausibility, but UNIREX makes this possible. For any choice of learned F ext , UNIREX lets us easily "plug and play" different criteria and loss weights, based on our needs and domain knowledge, to find those that best balance the rationale extraction desiderata.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Faithfulness Evaluating rationale faithfulness is still an open problem with many existing metrics, and UNIREX is not tailored for any specific metric. Still, given the prevalence of comp/suff (Sec. 2), we focus on comp/suff based objectives.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Recall that comp measures the importance of tokens in r (k) i as how pŷ i (x i ), F task 's predicted probability for classŷ i , changes when those tokens are removed from x i . Intuitively, we want pŷ i (x i ) to be higher than pŷ i (x i \r (k) i ), so higher comp is better. Since comp is defined for a single class' probability rather than the label distribution, we can define the comp loss L comp via cross-entropy loss L CE , as in 53 Figure   the following difference criterion for L comp :
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
For training stability, we compute comp loss for target class y i here instead of F task 's predicted classŷ i , sinceŷ i is a moving target during training. Using L comp-diff , it is possible for
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
i ), y i )) to become much larger than L CE (F task (x i ), y i ), leading to arbitrarily negative losses. To avoid this, we can add margin m c to the loss function, giving the margin criterion:
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Recall that suff measures the importance of tokens in r (k) i as how pŷ i (x i ), F task 's predicted probability for classŷ i , changes when they are the only tokens kept in x i . Based on suff's definition, we want pŷ i (r (k) i ) to be higher than pŷ i (x i ), so lower suff is better. For suff loss L suff , we define the difference and margin criteria analogously with margin m s but the opposite sign (since lower suff is better):
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
In our experiments, we find that the marginbased comp/suff criteria are effective (Sec. 4.3), though others (e.g., KL Div, MAE) can be used too (Sec. A.4.1). Note that r (k) i is computed via top-k% thresholding (Sec. 2), so we also need to specify a set K of threshold values. We separately compute the comp/suff losses for each k ∈ K, then obtain the final comp/suff losses by averaging over all k values via area-over-precision-curve (AOPC) (DeYoung et al., 2019). To reflect this, we denote the comp and suff losses as L comp,K and L suff,K , respectively. Let α f L faith = α c L comp,K + α s L suff,K , where α c and α s are loss weights.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Plausibility Plausibility is defined as how convincing a rationale is to humans (Jacovi and Goldberg, 2020), i.e., whether humans would agree the rationale supports the model's prediction. While optimizing for plausibility should ideally involve human-in-the-loop feedback, this is prohibitive. Instead, many works consider gold rationales as a cheaper form of plausibility annotation (DeYoung et al., 2019;Narang et al., 2020;Jain et al., 2020). Thus, if gold rationale supervision is available, then we can optimize for plausibility. With gold rationale r * i for input x i , plausibility optimization entails training F ext to predict binary importance label r * ,t i for each token x t i . This is essentially token classification, so one natural choice for L plaus is the token-level binary cross-entropy (BCE) criterion:
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Besides BCE loss, we can also consider other criteria like sequence-level KL divergence and L1 loss. See Sec. A.4.2 for discussion of these and other plausibility criteria.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
After setting F ext , L faith , and L plaus , we can move on to training F task and F ext . Since top-k% rationale binarization (Sec. 3.2) is not differentiable, by default, we cannot backpropagate L faith through all of F ext 's parameters. Thus, F task is trained via L task and L faith , while F ext is only trained via L plaus . This means F ext 's rationales r i are indirectly optimized for faithfulness by regularizing F task such that its behavior aligns with r i . The exception is if we are using the SLM variant, where encoder f enc is shared by F task and F ext . In this case, f enc is optimized w.r.t. all losses, f task is optimized w.r.t. L task and L faith , and f ext is optimized w.r.t. L plaus . SLM is a simple way to approximate end-to-end training of F task and F ext . In contrast, past SPPs have used more complex methods like reinforcement learning (Lei et al., 2016) and the reparameterization trick (Bastings et al., 2019), whose training instability can hurt task performance (Jain et al., 2020). Now, we summarize the full learning objective. Given that cross-entropy loss L task = L CE (F task (x i ), y i ) is used to train F task to predict y i , the full learning objective is:
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
During inference, we use F task to predict y i , then use F ext to output r i for F task 's predicted labelŷ i .
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
We present empirical results demonstrating UNIREX's effectiveness in managing trade-offs between faithfulness, plausibility, and task performance during rationale extractor optimization.  To aggregately evaluate multiple desiderata, we introduce the Normalized Relative Gain (NRG) metric, which is based on the ARG metric from Ye et al. (2021). NRG normalizes raw metrics (e.g., F1, sufficiency) to scores between 0 and 1 (higher is better). Given a set of raw metric scores Z = {z 1 , z 2 , ...} (each from a different method), NRG(z i ) captures z i 's value relative to min(Z) and max(Z). If higher values are better for the given metric (e.g., F1), then we have:
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
. If lower values are better (e.g., sufficiency), then we have:
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
. After computing NRG for multiple raw metrics, we can aggregate them w.r.t. desiderata via averaging. Let FNRG, PNRG, and TNRG be the NRG values for faithfulness, plausibility, and task performance, respectively. Finally, we compute the composite NRG as: CNRG = FNRG+PNRG+TNRG 3
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
. Results Reporting For all results, we report average over three seeds and the five k values. We   jan et al., 2017). We also experiment with IG for L2E (Situ et al., 2021), which distills knowledge from an AA to an LM. The second category is SPPs: FRESH (Jain et al., 2020) and A2R (Yu et al., 2021). For FRESH, we use a strong variant where IG rationales are directly given to the predictor, rather than output by a trained selector. A2R aims to improve SPP task performance by regularizing the predictor with an attention-based predictor that uses the full input. In addition, we introduce FRESH+P and A2R+P, which augment FRESH and A2R, respectively, with plausibility optimization. The third category is AA-based regularization: SGT (Ismail et al., 2021), which uses a sufficiency-based criterion to optimize for faithfulness. We also consider SGT+P, which augments SGT with plausibility optimization.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Fig. 4-6 display the main results. In Fig. 4/5, we compare the CNRG for all methods and datasets, without/with gold rationales. In both plots, we see that UNIREX variants achieve the best CNRG across all datasets, indicating that they are effective in balancing the three desiderata. In particular, UNIREX (DLM-FP) and UNIREX (SLM- Figure 6: NRG Comparison by Desiderata. We show FNRG, PNRG, and TNRG for all methods, averaged over all datasets. FP) have very high CNRG scores, both yielding more than 30% improvement over the strongest baselines. Fig. 6 compares methods w.r.t. desiderata NRG (i.e., FNRG, PNRG, TNRG). Here, the left/right plots show methods without/with gold rationales. Again, we see that UNIREX variants achieve a good NRG balance of faithfulness, plausibility, and task performance. Meanwhile, many baselines (e.g., AA (IG), A2R, SGT+P) do well on some desiderata but very poorly on others.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
We present five ablation studies to validate the effectiveness of our UNIREX design choices. The ablation results are displayed in Table 1. In this table, each of the five sections shows results for a different ablation. Thus, all numbers within the same section and column are comparable.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Extractor Type In the Ext Type (F) section, we compare four heuristic rationale extractors, using AA-F. Rand uses random importance scores, Gold directly uses the gold rationales, Inv uses the inverse of the gold rationales, and IG uses IG. All heuristics yield similar task performance, but IG dominates on all faithfulness metrics. This makes sense because IG is computed using F task 's inputs/parameters/outputs, while the others do not have this information. For plausibility, Gold is the best, Inv is the worst, and Rand and IG are about the same, as none of the heuristics are optimized for plausibility. In the Ext Type (FP) section, we compare four learned rationale extractors. By default, attribution algorithms' dimension scores are pooled into token scores via sum pooling. AA-FP (Sum) uses IG with sum pooling, while AA-FP (MLP) replaces the sum pooler with a MLP-based pooler to increase capacity for plausibility optimization. Task performance for all four methods is similar, AA-FP (Sum) dominates on faithfulness, and DLM-FP and SLM-FP dominate on plausibility. AA-FP (MLP) does not perform as well on faithfulness but slightly improves on plausibility compared to AA-FP (Sum).
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Comp/Suff Losses The Comp/Suff Loss section compares different combinations of Comp and Suff losses, using SLM-FP. Note that SLM-FP (Comp+Suff) is equivalent to SLM-FP shown in other tables/sections. As expected, SLM-FP (Comp) does best on Comp, but SLM-FP (Comp+Suff) actually does best on Suff. Meanwhile, SLM-FP, (Suff) does second-best on Suff but is much worse on Comp. This shows that Comp and Suff are complementary for optimization.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Suff Criterion The Suff Criterion section compares different Suff criteria, using SLM-FP. SLM-FP (KLDiv) uses the KL divergence criterion, SLM-FP (MAE) uses the MAE criterion, and SLM-FP (Margin) uses the margin criterion. SLM-FP (Margin) is equivalent to SLM-FP in other ta- bles/sections. All criteria yield similar performance and plausibility, while Margin is slightly better on faithfulness.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
The SLM Ext Head section compares different extractor heads, using SLM-FP. Linear is the default choice and uses a linear layer. MLP-2048-2 uses a MLP with two 2048-dim hidden layers. MLP-4096-3 uses a MLP with three 4096-dim hidden layers. All three output head types yield similar performance, but decreasing head capacity yields better faithfulness, while increasing head capacity heads yields better plausibility. This trades off faithfulness and plausibility, although larger heads will be more compute-intensive.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
UNIREX supports arbitrary amounts of gold rationale supervision and allows us to account for data efficiency. In Fig. 7, we compare plausibility (in AUPRC) for γ = [0.5, 1, 5, 10, 20, 100] (i.e., % of train instances with gold rationales). We compare AA (IG) and four UNIREX variants (AA-F, AA-FP, DLM-FP, SLM-FP). AA (IG) and AA-F do not use gold rationales and thus have the same AUPRC for all γ. Standard deviation is shown by the error bands. UNIREX (DLM-FP) and UNIREX (SLM-FP) dominate across all γ values, with AUPRC slowly decreasing as γ decreases. Even at γ = 0.5, they can still achieve high AUPRC. This suggests that UNIREX's gold rationale batching procedure (Sec. A.3) is effective for learning from minimal gold rationale supervision and demonstrates how UNIREX enables us to manage this trade-off. See Sec. A.6 for similar results on CoS-E.  
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
In We want to show that, even if F task yields poor task performance on unseen datasets, F ext 's rationales can still be faithful. As expected, all methods achieve much lower task performance in the third setting than in the first two settings. However, faithfulness does not appear to be strongly correlated with task performance, as unseen tasks' comp/suff scores are similar to seen tasks'. Across all datasets, DLM-FP has the best faithfulness and is the only method whose comp is always higher than suff. AA-F is not as consistently strong as DLM-FP, but almost always beats AA (IG) on comp and suff. Meanwhile, AA (IG) has the worst comp and suff overall. Ultimately, these results suggest that UNIREX-trained models' faithfulness (i.e., alignment between F task 's and F ext 's outputs) is a dataset/task agnostic property (i.e., can generalize across datasets/tasks), further establishing UNIREX's utility in low-resource settings.  
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Gold rationale based plausibility evaluation is noisy because gold rationales are for the target label, not a model's predicted label. Thus, we conduct two five-annotator user studies (Table 3) to get a better plausibility measurement. Given 50 random test instances from SST, we get the rationales for SGT+P, A2R+P, UNIREX (AA-FP), and UNIREX (DLM-FP), plus the gold rationales. For each instance, we threshold all rationales to have the same number of positive tokens as the gold rationale. The first user study is forward simulation (Hase and Bansal, 2020; Jain et al., 2020). Here, the annotator is given an input and a rationale for some model's prediction, then asked what (binary) sentiment label the model most likely predicted. For forward simulation, we also consider a No Rationale baseline, where no tokens are highlighted. For No Rationale and Gold, the target label is the correct choice. Annotators are also asked to rate their confidence (4point Likert scale) in their answer to this question. The second user study involves giving a subjective rating of how plausible the rationale is (Hase and Bansal, 2020). Here, the annotator is given the input, rationale, and model's predicted label, then asked to rate (5-point Likert scale) how aligned the rationale is with the prediction. In both forward simulation and subjective rating, we find that DLM-FP performs best among all non-oracle methods and even beats Gold on accuracy, further supporting that DLM-FP rationales are plausible. As expected, the fact that Gold does not achieve near-100% accuracy shows the discrepancy between evaluating plausibility based on the target label (i.e., gold rationale similarity) and F task 's predicted label (forward simulation). Meanwhile, SGT+P and AA-FP, which had lower AUPRC/TF1 in our automatic evaluation, also do worse in accuracy/alignment. Also, users found SGT+P and AA-FP rationales harder to understand, as shown by their lower confidence scores. Meanwhile, A2R+P had high AUPRC/TF1, but gets very low accuracy/alignment because A2R+P's predicted label often not the target label, leading to misalignment with its gold-like rationale. A2R+P is a great example of how automatic plausibility evaluation can be misleading. For the accuracy, confidence, and alignment questions, we achieved Fleiss' Kappa (Fleiss, 1971) inter-annotator agreement scores of 0.2456 (fair), 0.1282 (slight), and, 0.1561 (slight), respectively. This lack of agreement shows the difficulty of measuring plausibility. Connection to UNIREX Unlike prior works, UNIREX enables both the task model and rationale extractor to be jointly optimized for faithfulness, plausibility, and task performance. As a result, UNIREX-trained rationale extractors achieve a better balance of faithfulness and plausibility, without compromising the task model's performance. Also, by using a learned rationale extractor, which generally only requires one model forward pass, UNIREX does not have the computational expenses that limit many AAs. 
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Here, we formalize the text classification problem in more detail.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
are the labels, and N is the number of instances (x i , y * i ) in D. We also assume D can be partitioned into train set D train , dev set D dev , and test set D test . Let F task = f task (f enc (·)) be a task LM, where f enc is the text encoder, and f task is the task output head. Typically, F task has a BERT-style architecture (Devlin et al., 2018), in which f enc is a Transformer (Vaswani et al., 2017) while f task is a linear layer. Below, we define the sequence classification (SST, Movies, MultiRC, e-SNLI) and multi-choice QA (CoS-E) tasks, which are different types of text classification.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
In sequence classification, x i is a token sequence (e.g., a single sentence, a pair of sentences), while y * i is the target class for x i . Here, we assume a fixed label space Y = {1, ..., M } of size M , where y * i ∈ Y for all i. Thus, f task outputs a vector of size M , such that F task (x i ) = f task (f enc (x i )) =ŷ i ∈ R M is the logit vector used to classify x i . Givenŷ i = [ŷ i,j ] M j=1 , let y i = arg max jŷi,j be the class predicted by F task . The goal of sequence classification is to learn F task such that y * i = y i , for all (x i , y * i ) (Minaee et al., 2021).
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Multi-Choice QA Instead of a fixed label space, multi-choice QA has a different (but fixed-size) set of answer choices per instance. For instance i, let q i be the question (e.g., "A friend is greeting me, what would they say?") and
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
be the corresponding answer choices (e.g., {"say hello", "greet", "associate", "socialize", "smile"}), where M is now the number of answer choices. Define x i,j = q i ⊕ a i,j , where ⊕ denotes concatenation. In multi-choice QA, we have
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
is the correct answer for x i . Thus, f task outputs a scalar, such that F task (x i,j ) = f task (f enc (x i,j )) =ŷ i,j ∈ R is the logit for x i,j . Givenŷ i = [ŷ i,j ] M j=1 , let j ′ = arg max jŷi,j , where y i = a i,j ′ is the answer predicted by F task . The goal of multi-choice QA is to learn F task such that y * i = y i , for all (x i , y * i ) (Talmor et al., 2018).
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
A heuristic F task is an AA, which can be any handcrafted function that calculates an importance score s t i for each input token x t i (Bastings and Filippova, 2020). AAs are typically gradient-based (Sundararajan et al., 2017;Denil et al., 2014;Lundberg and Lee, 2017;Li et al., 2015) or perturbationbased (Li et al., 2016;Poerner et al., 2018;Kádár et al., 2017) methods. Gradient-based methods compute s t i via the gradient of F task 's outputŷ i w.r.t. x t i , via one or more F task backward passes. Perturbation-based methods measure s t i asŷ i 's change when perturbing (e.g., removing) x t i , via multiple F task forward passes.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
AAs can be used out of the box without training and are designed to satisfy certain faithfulnessrelated axiomatic properties (Sundararajan et al., 2017;Lundberg and Lee, 2017). However, AAs' lack of learnable parameters means they cannot be optimized for faithfulness/plausibility. Thus, if F task is trained for explainability using AA-based rationales, then only F task is optimized. Also, faithful AAs tend to be compute-intensive, requiring many F task backward/forward passes per instance (Sundararajan et al., 2017;Lundberg and Lee, 2017;Li et al., 2016).
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
If a learned rationale extractor is chosen, UNIREX enables users to specify how much gold rationale supervision to use. Ideally, each train instance would be annotated with a gold rationale. In this case, we could directly minimize the plausibility loss for each train instance. However, since gold rationales can be expensive to annotate, UNIREX provides a special batching procedure for training with limited gold rationale supervision.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Given N train = |D train | train instances, let 0 < γ < 100 be the percentage of train instances with gold rationales, N gold = ⌈ γ 100 N train ⌉ ≥ 1 be the number of train instances with gold rationales, b be the desired train batch size, and β > 1 be a scaling factor. Define D gold ⊆ D train as the set of train instances with gold rationales, where |D gold | = N gold . Note that, if all train instances have gold rationales, then D gold = D train and γ = 100.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Each batch is constructed as follows: (1) randomly sample b gold = max(1, b β ) instances from D gold without replacement, then (2) randomly sample b − b gold instances from D train \D gold without replacement. This results in a batch with b total train instances, b gold with gold rationales and the rest without. Since N gold is generally small, we only sample from D gold without replacement for a given batch, but not a given epoch. Thus, instances from D gold may appear more than once in the same epoch. However, we do sample from D train \D gold without replacement for each batch and epoch, so every instance in D train \D gold appears exactly once per epoch.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
After constructing the batch, we compute the plausibility loss for the batch as follows:
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
where L plaus is the plausibility loss for train instance (x i , y * i ). This function zeroes out the plausibility loss for instances without gold rationales, so that plausibility is only being optimized with respect to instances with gold rationales. However, in Sec. ??, we show that it is possible to achieve high plausibility via rationale extractors trained on minimal gold rationale supervision.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Sufficiency In addition, to the criteria presented in Sec. 3.2, we consider two other sufficiency loss functions. The first is the KL divergence criterion used in (Ismail et al., 2021), which considers the entire label distribution and is defined as L suff-KL = KL(F task (r (k) i )) || F task (x i )). The second is the mean absolute error (MAE) criterion, which is defined as L suff-MAE = |L CE (F task (r
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Unlike the difference criterion L suff-diff and margin criterion L suff-margin (Sec. 3.2), the MAE criterion assumes that using r (k) i as input should not yield better task performance than using x i as input. In our experiments, we find that L suff-margin is effective, though others (e.g., KL divergence, MAE) can be used too.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Similar to faithfulness, UNIREX places no restrictions on the choice of plausibility objective. As described in Sec. 3.2, given gold rationale r * i for input x i , plausibility optimization entails training F ext to predict binary importance label r * ,t i for each token x t i . This is essentially binary token classification, so one natural choice for L plaus is the token-level binary cross-entropy (BCE) criterion: L plaus-BCE = − t r * ,t i log(F ext (x t i )) (Sec. 3.2). Another option is the sequence-level KL divergence criterion, which is defined as: L plaus-KL = KL(F ext (x i ) || r * i ). Additionally, we can directly penalize F ext (x i ) in the logit space via a linear loss, defined as:
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
where Φ(u) = −2u + 1 maps positive and negative tokens to −1 and +1, respectively. The linear loss directly pushes the logits corresponding to positive/negative tokens to be higher/lower and increase the margin between them. To prevent linear loss values from becoming arbitrarily negative, we can also lower bound the loss with a margin m p , yielding: L plaus-linear-margin = max(−m p , L plaus-linear ) + m p .
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
A.5 Implementation Details LM Architecture While many prior works use BERT (Devlin et al., 2018) Transformer LMs, BERT is limited to having sequences with up to 512 tokens, which is problematic since many datasets (e.g., Movies) contain much longer sequences. Meanwhile, BigBird (Zaheer et al., 2020) is a state-of-the-art Transformer LM designed to handle long input sequences with up to 4096 tokens. Thus, we use BigBird-Base, which is initialized with RoBERTa-Base (Liu et al., 2019), in all of our experiments (i.e., both baselines and UNIREX). We obtain the pre-trained BigBird-Base model from the Hugging Face Transformers library (Wolf et al., 2019). Note that UNIREX is agnostic to the choice of LM architecture, so RNNs, CNNs, and other Transformer LMs are also supported by UNIREX. However, we leave exploration of other LM architectures for future work.
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
Training Building upon Sec. ??, we discuss additional training details here. We find that α c = 0.5 and α s = 0.5 are usually best. For the batching factor β (Sec. A.3), we use 2. For model selection, we choose the model with the best dev performance averaged over three seeds. We can also perform model selection based on dev explainability metrics, but we leave this extended tuning for future work. All experiments are implemented us-  creases. Interestingly, UNIREX (AA-FP) yields a noticeable dip in AUPRC for lower γ values. Since AA-FP has limited capacity (via the task model) for plausibility optimization, it is possible that this fluctuation is due to random noise. We leave further analysis of this for future work. 
UNIREX: A Unified Learning Framework for Language Model Rationale Extraction
In this subsection, we present additional results from our experiments. Besides the aggregated results shown in Sec. 4 of the main text, Tables 4-10 contain more detailed results, using both raw and NRG metrics. Specifically, Tables 4-8 show all raw/NRG results for each dataset, Table 9 shows the ablation results for all raw metrics, and Table 10 includes the zero-shot explainability transfer results for UNIREX (SLM-FP). Generally, the computation of NRG should involve globally aggregating the raw metrics for all available methods, as done in the main results. However, for a number of more focused experiments (Tables 9-10), only a subset of the available methods are considered. Thus, to make the faithfulness results in Tables 9-10 easier to digest, we introduce a metric called Comp-Suff Difference (CSD), which locally aggregates comp and suff as: CSD = comp − suff. Therefore, since higher/lower comp/suff signal higher faithfulness, then higher CSD signals higher faithfulness.        
NILE : Natural Language Inference with Faithful Natural Language Explanations
Deep learning methods have been employed to improve performance on several benchmark classification tasks in NLP (Wang et al., 2018(Wang et al., , 2019. Typically, these models aim at improving label accuracy, while it is often desirable to also produce explanations for these decisions (Lipton, 2016;Chakraborty et al., 2017). In this work, we focus on producing natural language explanations for Natural Language Inference (NLI), without sacrificing much on label accuracy.
NILE : Natural Language Inference with Faithful Natural Language Explanations
There has been growing interest in producing natural language explanations for deep learning systems (Huk Park et al., 2018;Kim et al., 2018;Ling et al., 2017), including NLI (Camburu et al., 2018). In general, the explanations from these methods can typically be categorized as post-hoc explanations (Lipton, 2016). Camburu et al. (2018) propose an NLI system which first produces an explanation and then processes the explanation to produce the final label. We argue that these explanations also resemble post-hoc explanations (Section 4.2). Further, existing methods don't provide a natural way to test the faithfulness of the generated explanations, i.e., how well do the provided explanations correlate with the model's decision making.
NILE : Natural Language Inference with Faithful Natural Language Explanations
We therefore propose Natural-language Inference over Label-specific Explanations (NILE) 1 , which we train and evaluate on English language examples. Through NILE, we aim to answer the following question:
NILE : Natural Language Inference with Faithful Natural Language Explanations
Can we build NLI systems which produce faithful natural language explanations of predicted labels, while maintaining high accuracy?
NILE : Natural Language Inference with Faithful Natural Language Explanations
Briefly, in NILE, we first generate natural language explanations for each possible decision, and subsequently process these explanations to produce the final decision. We argue that such a system provides a natural way of explaining its decisions. The key advantage is the testability of these explanations, in themselves, as well as in terms of the sensitivity of the system's prediction Figure 1: Overview of NILE: A Premise and Hypothesis pair is input to label-specific Candidate Explanation Generators G which generate natural language explanations supporting the corresponding label. The generated explanations are then fed to the Explanation Processor S, which generates label scores using the evidence present in these explanations (see Figure 3 for the architectures used in this work). In addition to the explanations, NILE also utilizes the premise and hypothesis pair (See Section 4.4.2 for a discussion on the challenges in building such a system). Please see Section 4 for details.
NILE : Natural Language Inference with Faithful Natural Language Explanations
to these explanations.
NILE : Natural Language Inference with Faithful Natural Language Explanations
We choose NLI due to its importance as an NLP task, and the availability of e-SNLI, a large dataset annotated both with entailment relation labels and natural language human explanations of those labels (Camburu et al., 2018;Bowman et al., 2015).
NILE : Natural Language Inference with Faithful Natural Language Explanations
In summary, we make the following contributions in this work.
NILE : Natural Language Inference with Faithful Natural Language Explanations
1. We propose NILE, an NLI system which generates and processes label-specific explanations to infer the task label, naturally providing explanations for its decisions. 2. We demonstrate the effectiveness of NILE compared to existing systems, in terms of label and explanation accuracy. 3. Through NILE, we provide a framework for generating falsifiable explanations. We propose ways to evaluate and improve the faithfulness of the system's predictions to the generated explanations. We claim that task-specific probes of sensitivity are crucial for such evaluation. We have released the source code of NILE to aid reproducibility of the results.
NILE : Natural Language Inference with Faithful Natural Language Explanations
In this section, we discuss the datasets (Section 3.1) and pre-trained models (Section 3.2) used to build NILE.
NILE : Natural Language Inference with Faithful Natural Language Explanations
SNLI: The Stanford NLI dataset (Bowman et al., 2015) contains samples of premise and hypothesis pairs with human annotations, using Amazon Mechanical Turk. The premises were obtained from pre-existing crowdsourced corpus of image captions. The hypotheses were obtained by presenting workers with a premise and asking for a hypothesis for each label (entailment, neutral and contradiction), resulting in a balanced set of ∼570K pairs. e-SNLI: Camburu et al. (2018) extend the SNLI dataset with natural language explanations of the ground truth labels. The explanations were crowdsourced using Amazon Mechanical Turk. Annotators were first asked to highlight words in the premise and hypothesis pairs which could explain the labels. Next, they were asked to write a natural language explanation using the highlighted words.
NILE : Natural Language Inference with Faithful Natural Language Explanations
Similar to Camburu et al. (2018), for all our experiments, we filter out non-informative examples where the explanations contain the entire text of the premise or hypothesis. In particular, we drop any training example where the uncased premise or hypothesis text appears entirely in the uncased explanation. This leads to a training data size of ∼532K examples.
NILE : Natural Language Inference with Faithful Natural Language Explanations
Transformer architectures (Vaswani et al., 2017) pre-trained on large corpora with self-supervision have shown significant improvements on various NLP benchmarks (Devlin et al., 2019;Radford et al., 2019;Yang et al., 2019;Lan et al., 2019). Improvements have been demonstrated for text classification as well as text generation tasks Raffel et al., 2019). In this work, we leverage the implementation of transformer architectures and pre-trained models provided by Wolf et al. (2019).
NILE : Natural Language Inference with Faithful Natural Language Explanations
GPT-2: We use the GPT-2 architecture (Radford et al., 2019), which is trained using a causal language modeling loss (CLM), and includes a leftto-right decoder suitable for text generation. In particular, we use the gpt2-medium model. This model has 24 layers, 16 attention heads and a hidden size of 1024 (∼345M parameters). For text generation, the model can be finetuned using CLM on desired text sequences.
NILE : Natural Language Inference with Faithful Natural Language Explanations
RoBERTa: For classification modules, we leverage RoBERTa , which is trained using a masked language modeling loss (MLM). In particular, we use the roberta-base model. This model has 12 layers, 12 attention heads and a hidden size of 768 (∼125M parameters). For downstream classifications tasks, a classification layer is added over the hidden-state of the first token in the last layer.
NILE : Natural Language Inference with Faithful Natural Language Explanations
4 Natural-language Inference over Label-specific Explanations (NILE)
NILE : Natural Language Inference with Faithful Natural Language Explanations
The overall architecture employed in NILE is shown in Figure 1. We introduce the notation used in this paper in Section 4.1. We then discuss the motivation for the major design choices in Section 4.2. NILE performs the following steps to produce labels and explanations:
NILE : Natural Language Inference with Faithful Natural Language Explanations
1. Candidate Explanation Generators: Labelspecific Candidate Explanation Generators first generate explanations supporting the respective labels (Section 4.3).
NILE : Natural Language Inference with Faithful Natural Language Explanations
Processor takes the explanations and also the premise and hypothesis pairs as input to produce the task label (Section 4.4). We also build NILE-PH, where the Explanation Processor has access only to the generated explanations (Section 4.4.1).
NILE : Natural Language Inference with Faithful Natural Language Explanations
We note that NILE-PH more naturally fits the desiderata described in Section 1, while we design and evaluate NILE for the more general case where the Explanation Processor also accesses the premise and hypothesis pair.
NILE : Natural Language Inference with Faithful Natural Language Explanations
In Section 4.5, we describe comparable baseline architectures.
NILE : Natural Language Inference with Faithful Natural Language Explanations
We denote each data point by (p, h), where p is the premise and h the hypothesis sentence. G denotes a model trained to generate natural language explanations. Specifically, G x denotes a model which generates natural language explanations t x of type x, where x ∈ {entail, contradict, neutral}. We denote the human-provided gold explanation for the correct predictions as t g . S denotes a module which predicts label scores. The true label for an example is denoted by y, while a model prediction is denoted by y , and label scores by l x . Posthoc generation: Given an input instance, first the label is predicted and then an explanation generated conditioned on the label and the input text. B. ExplainThen-Predict (Camburu et al., 2018): Given the input instance, first the desired explanation is generated, and then the label is predicted using only the generated explanation. We argue that neither architecture provides a natural way to test the sensitivity of the model's predictions to the generated explanation. Please see Section 4.2 for details.
NILE : Natural Language Inference with Faithful Natural Language Explanations
In this section, we describe the motivation for adopting a two-step pipelined approach.
NILE : Natural Language Inference with Faithful Natural Language Explanations
Label-specific explanations: Consider two alternative existing architectures in Figure 2. In Figure 2A, a model S pre is trained directly on the example sentences (p & h) to produce a label (y ), which together with the example sentences are used to produce an explanation t g using G post . It can be argued that while the target explanations may regularize the system, there is no reason for t g to be aligned with the reason why the model chose a particular label. Figure 2B corresponds to a model which has also been trained on e-SNLI (Camburu et al., 2018).
NILE : Natural Language Inference with Faithful Natural Language Explanations
G pre is first trained to produce natural language explanations t g using human-provided explanations (t g ) as targets, using only the example sentences as inputs. A model S post then chooses the label corresponding to the generated explanation t g . While at first, it appears that this system may provide faithful explanations of its decisions, i.e., the generated explanations are the reason for the label prediction, we argue that it may not be so.
NILE : Natural Language Inference with Faithful Natural Language Explanations
In Figure 2B, G pre is required to generate the explanation of the correct label for an example. It must first infer that label and then produce the corresponding explanation. Further analysis of the free-form human-provided explanations has revealed clear differences in the form of explanations, through alignment to label-specific templates (Camburu et al., 2018;Oana-Maria et al., 2019a). The Explanation Processor S post then only needs to infer the form of t g . G pre then resembles post-hoc generation methods, with the label (as the form of t g ) and explanation t g being produced jointly. The claim is supported by inconsistencies found in the generated explanations (Oana-Maria et al., 2019a).
NILE : Natural Language Inference with Faithful Natural Language Explanations
Neither architecture allows a natural way to test the sensitivity of the model's predictions to its explanations. In NILE, we first allow explanations for each label, and then require the Explanation Processor to select the correct explanation. This allows us to naturally test whether the model's predictions are indeed due to the selected explanation. This can be done, for example, by perturbing the input to the Explanation Processor.
NILE : Natural Language Inference with Faithful Natural Language Explanations
A pipelined approach: We use a pipelined approach in NILE ( Figure 1). The Candidate Explanation Generators are first trained using humanprovided explanations. The Explanation Processor takes as input the generated label-specific explanations. This prevents the system from producing degenerate explanations to aid task performance. It also allows perturbing the generated explanations to probe the system in a more natural way compared to an unintelligible intermediate state of a learnt model. We believe that systems can be designed to work in this setting without compromising task performance.
NILE : Natural Language Inference with Faithful Natural Language Explanations
We train label-specific explanation generators, G x , x ∈ {entail, contradict, neutral}, using humanprovided explanations of examples with the corresponding label. For example, to train G entail , we are special tokens added to the vocabulary. During fine-tuning, the language modeling loss function is used only over the explanation tokens. Next, we create prompts of the form "Premise: p Hypothesis: h [EXP]" and require each trained language model to independently complete the sequence. In this way we obtain label specific expla-
NILE : Natural Language Inference with Faithful Natural Language Explanations
The Explanation Processor in NILE takes as input the generated label-specific explanations, as well as the premise and hypothesis pair to generate label scores l x , x ∈ {entail, contradict, neutral}. During training, these scores are passed through a softmax layer and a cross-entropy loss is used to generate the training signal. During testing, the label with the maximum score is selected.
NILE : Natural Language Inference with Faithful Natural Language Explanations
We leverage a pre-trained roberta-base model for all our experiments, and fine-tune it as specified in the following subsections. In each case, any intermediate scores are generated through transformations of the first token ([CLS]) embedding from the last layer. We define:
NILE : Natural Language Inference with Faithful Natural Language Explanations
where inp is a pair of sequences in NILE, a single sequence in NILE-PH, and W are the learnable parameters for the model.
NILE : Natural Language Inference with Faithful Natural Language Explanations
For simplicity, and to elucidate the desired behavior, we first describe how explanations are processed in NILE-PH (Section 4.4.1). We then dis-cuss the construction of NILE, a potential issue, and a fix for the same (Section 4.4.2).
NILE : Natural Language Inference with Faithful Natural Language Explanations
In this section, we describe how explanations are processed in NILE-PH, which is generalized in NILE (Section 4.4.2). We experiment with three architectures, described below (also see Figure 3).
NILE : Natural Language Inference with Faithful Natural Language Explanations
A. Independent: In the Independent model, explanations are fed to F Ind , which generates a score for each explanations independently:
NILE : Natural Language Inference with Faithful Natural Language Explanations
where x ∈ {entail, contradict, neutral}. We expect this score to represent the truthfulness of the input explanation.
NILE : Natural Language Inference with Faithful Natural Language Explanations
B. Aggregate: The Independent model would need all three explanations to be available to reliably produce label scores. We believe a system should be able to handle one or more missing or ambiguous explanations. For example, the entailment explanation: "t entail : A dog is a cat" would provide evidence for contradiction. To capture this notion, we require the Explanation Processor to produce two intermediate scores V 1 and V 2 , where we expect V 1 to collect evidence supporting an input claim and V 2 to collect evidence against an input claim:
NILE : Natural Language Inference with Faithful Natural Language Explanations
The intermediate score are then aggregated into the final label scores:
NILE : Natural Language Inference with Faithful Natural Language Explanations
where Cmb is the LogSumExp function. The reason for this choice of aggregation is that while evidence against entailment might point to contradiction and vice versa, evidence against neutral doesn't necessarily provide any information about entailment or contradiction relations.
NILE : Natural Language Inference with Faithful Natural Language Explanations
C. Append: Finally, to allow the model to reason arbitrarily between the three generated explanations, we created a single sequence, concat ecn : "entailment: t entail contradiction: t contradict neutral: t neutral ", and generate the scores as follows:
NILE : Natural Language Inference with Faithful Natural Language Explanations
where x ∈ {entail, contradict, neutral}.
NILE : Natural Language Inference with Faithful Natural Language Explanations
In NILE, to process premise p and hypothesis h, we first concatenate p and h into concat ph : "Premise: p Hypothesis: h". The label scores are then obtained as in Section 4.4.1, by modifying Equation 1, 2 and 4 as follows: replace F z (x) by F z (concat ph , x), where z ∈ {Ind, Agg, Apn}. We note that appending the example sentences to the generated explanations (as in Append) would result in having no control over whether the explanations are used for the final prediction. The case for Independent and Aggregate is not immediately clear. We now discuss a potential issue with these architectures when processing premise and hypothesis text, and suggest a fix for the same.
NILE : Natural Language Inference with Faithful Natural Language Explanations
The issue: We expect NILE to answer the question: Is (concat ph , t x ), where x ∈ {entail, contradict, neutral}, a valid instance-explanation pair? The Independent and Aggregate architectures for NILE have been designed such that the model can't ignore the label-specific explanations. For example, the Independent model will produce identical scores for each output label, if it chooses to completely ignore the input explanations. However, the model is still free to learn a different kind of bias which is an outcome of the fact that natural language explanations convey ideas through both content and form. If the form for explanations of different labels is discriminative, an unconstrained learning algorithm could learn to infer first the type of explanation and use it to infer the task. For example, given the input (concat ph , t x ), where x ∈ {entail, contradict, neutral}, if a model could learn whether t x is an entailment explanation, it then only has to output whether concat ph corresponds to an entailment relation. Essentially, high label accuracy can be achieved by inferring first what task to do using only the form of t x .
NILE : Natural Language Inference with Faithful Natural Language Explanations
The fix: To prevent NILE from exploiting the form of an explanation as described above, we create additional training examples, where we require NILE to score valid instance-explanation pairs higher. In particular, we sample negative explanations for an instance, of the same form as the correct label. For example, an instance labeled as entailment would have an additional training signal: Score (concat ph , t entail ) higher than (concat ph , t entail ) and (concat ph , t entail ), where t entail and t entail are randomly sampled entailment form explanations. We note that the fix leaves room for other kinds of biases to be learnt. However, the key advantage with NILE is that it is easy to design probes to test for such biases and subsequently fix them (see Section 5.3).
NILE : Natural Language Inference with Faithful Natural Language Explanations
We now describe baselines which use the same underlying blocks as NILE, for generating explanations and classification.
NILE : Natural Language Inference with Faithful Natural Language Explanations
NILE:post-hoc: To understand the drop in performance which could be associated with constraining models as we have done, we train a model with full access to input examples (See Figure 2A).
NILE : Natural Language Inference with Faithful Natural Language Explanations
Further, we provide a strong baseline for posthoc generators using this model, where using the model's predictions, we simply pick the corresponding label-specific generated explanation.
NILE : Natural Language Inference with Faithful Natural Language Explanations
We note that the model's predictions have no sensitivity to the generated explanations in NILE: posthoc.
NILE : Natural Language Inference with Faithful Natural Language Explanations
ExplainThenPredictAttention (ETPA): Following (Camburu et al., 2018), (see Figure 2B), we train a pipelined system, where we first learn to generate the gold explanation t g , followed by a classification of t g to predict the label:
NILE : Natural Language Inference with Faithful Natural Language Explanations
where x ∈ {entail, contradict, neutral}.  (Camburu et al., 2018) as well as the results with our reproduction of ETPA. ETPA (reproduced) is directly comparable with NILE (Section 4.5). NILE-PH competes with or outperforms ETPA baselines on label accuracy, while NILE-NS and NILE provide significant gains in label accuracy. NILE and NILE-NS are competitive with the best reported results in terms of label accuracies. We report the number of correct explanations, averaged across annotators (B) as well as when all annotators agree on correctness (C). All NILE variants are able to provide more correct explanations than the ETPA baseline. We also report the percentage of correct explanations in the subset of correct label predictions (B/A, C/A). On this metric, NILE variants are comparable with the ETPA baseline. However, the real value of NILE lies in being able to probe the faithfulness of its decisions (Section 5.3). Further, NILE explanations generalize significantly better on out-of-domain examples (See Table 2). Please see Section 5.1 for details.
NILE : Natural Language Inference with Faithful Natural Language Explanations
In this section, we aim to answer the following questions: We provide training details in Appendix A, and examples of generated label-specific explanations in Appendix B.
NILE : Natural Language Inference with Faithful Natural Language Explanations
We report the label accuracies of the baselines and proposed architectures on the SNLI Dev and Test set in Table 1. We also report explanation accuracies, obtained through human evaluation of the generated explanations in the first 100 test examples. Binary scores on correctness were sought from five annotators (non-experts in NLP) on the generated explanations. For both label and explanation accuracies, we report using a model selected using the SNLI Dev set label accuracy across 5 runs with 5 different seeds of random initialization. Please see the Appendix for more details on the the 5 runs. First, through NILE:post-hoc, we provide a strong baseline for obtaining high label and explanation accuracy. Our aim in this work is to learn explanations that serve as the reason for the model's  We report the number of correct explanations, averaged across annotators (B) as well as when all annotators agree on correctness (C). All NILE variants provide more correct explanations than the ETPA baseline (B, C). Further, the percentage of correct explanations in the subset of correct label predictions (B/A, C/A) is significantly better for all NILE variants. The results demonstrate that NILE provides a more generalizable framework for producing natural language explanations. Please see Section 5.2 for details.
NILE : Natural Language Inference with Faithful Natural Language Explanations
predictions. Nevertheless, we are able to compete or outperform this baseline, in terms of explanation accuracy, while incurring a only a small drop in label accuracy. All variants of NILE, including NILE-PH and NILE-NS (which is not trained using negative samples of explanations as described in Section 4.4.2), produce more correct explanations than the ETPA baseline. NILE-PH:Append, NILE and NILE-NS provide gains over label accuracies compared to the ETPA baseline. Additionally, NILE and its variants provide natural ways to probe the sensitivity of the system's predictions to the explanations, as demonstrated in the subsequent sections. Finally, the explanations generated by all NILE variants generalize significantly better on out-of-distribution examples when compared to the ETPA baseline (See Section 5.2).
NILE : Natural Language Inference with Faithful Natural Language Explanations
To test the generalization capability of NILE, we do training and model selection on the SNLI dataset (Section 5.1), and evaluate on the out-of-domain MNLI (Williams et al., 2018) development sets. Transfer without fine-tuning to out-of-domain NLI has been a challenging task with transfer learning for generating explanations in MNLI being particularly challenging (Camburu et al., 2018). We report label accuracies on the Dev (matched) and Dev-mm (mismatched) sets, and explanation evaluation on the first 100 Dev samples in Table 2. Explanation evaluation was done by three annotators (who also annotated the SNLI explanations). While the label accuracies follow a similar pattern as the in-domain SNLI Test set, all variants of NILE provide gains in the quality of generated explanations. All variants of NILE produce more correct explanations (B, C) as well as a higher percentage of correct generated explanations among correct predictions (B/A, C/A). This demonstrates that NILE, through intermediate label-specific natural language explanations, provides a more general way for building systems which can produce natural language explanations for their decisions.
NILE : Natural Language Inference with Faithful Natural Language Explanations
NILE and its variants allow a natural way to probe the sensitivity of their predictions to the generated explanations, which is by perturbing the explanations themselves. In this way, NILE resembles   Table 4: Probing the sensitivity of the system's predictions by shuffling instance-explanation pairs. Each instance is attached to a randomly selected explanation of the same form as the original pair. The results demonstrate a much weaker link between NILE-NS's predictions and associated explanations. On the other hand, NILE behaves more expectedly. Note that the baselines don't allow a similar mechanism to test their faithfulness, and such testability is a key advantage of NILE. Please see Section 5.3 for details. explanation systems which provide input text fragments as reasons for their decisions. DeYoung et al. (2019) propose metrics to evaluate the faithfulness of such explanations. Following their work, we first attempt to measure the explanations generated by the methods proposed in this paper for comprehensiveness (what happens when we remove the explanation from the input) and sufficiency (what happens if we keep only the explanations). In Table 3, we show these measures for NILE and NILE-NS. The results seem to indicate that explanations for both NILE and NILE-NS are comprehensive, while having higher sufficiency in the case of NILE-NS. We first note that the comprehensiveness of these systems is ensured by design, and the input is indistinguishable without an explanation. Second, we argue that sufficiency may indicate correlations which don't necessarily exist in the system otherwise. We study the sensitivity of the explanations through a probe motivated by an understanding of the task and the training examples (see Section 4.4.2). We perturb the instanceexplanation inputs such that for each test instance, the explanation is replaced by a randomly selected explanation of the same label. The results (Table 4) indicate that NILE-NS is more robust to random perturbations of input explanations, and presumably uses the form of the explanation to infer the task (see Section 4.4.2 for a discussion). It is true that NILE behaves expectedly as we have specifically designed NILE to prevent the associated bias, and that this could potentially lead the system to learn other such biases. However, a key advantage of the proposed architecture is the ability to identify and fix for such biases. We leave it as an interesting and challenging future work to find and fix more such biases.
NILE : Natural Language Inference with Faithful Natural Language Explanations
In this paper we propose NILE, a system for Natural Language Inference (NLI) capable of generating labels along with natural language explanations for the predicted labels. Through extensive experiments, we demonstrate the effectiveness of this approach, in terms of both label and explanation accuracy. NILE supports the hypothesis that accurate systems can produce testable natural language explanations of their decisions. In the paper, we also argue the importance of explicit evaluation of faithfulness of the generated explanations, i.e., how correlated are the explanations to the model's decision making. We evaluate faithfulness of NILE's explanations using sensitivity analysis. Finally, we demonstrate that task-specific probes are necessary to measure such sensitivity.  For fine-tuning gpt2-medium language models for explanation generation as well as roberta-base models, we leverage code and pre-trained models from the "transformers" library available at https://github.com/huggingface. In each case we train on the train split for three epochs. Apart from batch size, sequence length and seed for random initialization, we keep the other hyperparameters fixed throughout the experiments. We don't do any fine-tuning on seeds of random initialization.
NILE : Natural Language Inference with Faithful Natural Language Explanations
For roberta-base models, we report results through model selection on models trained using 5 seeds of random initialization -42, 219, 291, 67 and 741. Model selection is done using label accuracies on SNLI Dev set. In Table 5, we report the mean and standard deviation for the label accuracies across 5 runs.
NILE : Natural Language Inference with Faithful Natural Language Explanations
We ran our experiments on GeForce GTX 1080 Ti GPUs. We adjust the batch size to be the largest multiple of 16 to fit on the GPU memory (∼12GB). We now list all the hyper-parameters used.
NILE : Natural Language Inference with Faithful Natural Language Explanations
GPT2: The hyper-parameters used for finetuning gpt2-medium include a maximum sequence length of 128, batch size of 2, learning rate of 5e-5, Adam epsilon of 1e-8, max gradient norm of 1.0 and a seed of 42. For generating text, we used greedy decoding.  Table 6: Hyper-parameters (batch size and maximum sequence length) used for fine-tuning roberta-base RoBERTa: The fixed parameters for fine-tuning robera-base included a learning rate of 2e-5, Adam epsilon of 1e-8 and max gradient norm 1.0. Other parameters are captured in Table 6.
NILE : Natural Language Inference with Faithful Natural Language Explanations
In this section, we list the label-specific explanations generated for five Dev set examples, along with the premise, hypothesis and gold labels, for both SNLI (Section B.1) and MNLI (Section B.2) datasets.
NILE : Natural Language Inference with Faithful Natural Language Explanations
holding to go packages. Hypothesis: The sisters are hugging goodbye while holding to go packages after just eating lunch.
NILE : Natural Language Inference with Faithful Natural Language Explanations
entailment: Sisters are women. contradiction: The women cannot be embracing if they are hugging goodbye. neutral: Two women embracing does not imply they are sisters hugging goodbye.
NILE : Natural Language Inference with Faithful Natural Language Explanations
Two young children are two kids, and blue jerseys implies numbered jerseys. contradiction: The kids are either standing on wooden steps or wash their hands. neutral: The kids could be washing their hands in a sink for any reason, not just because they wash their hands. entailment: A propeller is a type of ball. contradiction: A child cannot touch the propeller of a plane while playing with a ball. neutral: Just because the child reaches up to touch the propeller of a plane does not mean the child is playing with a ball.
Few-Shot Self-Rationalization with Natural Language Prompts
Models constrained to be more understandable to people are easier to troubleshoot and more useful in practice (Rudin et al., 2021). For instance, constraining a model that answers the question "Which linguist invented the lightbulb?" with "none" to also provide the reason-"Thomas Edison is the inventor of the lightbulb and he was not a linguist"makes the model easier to control and interact with (Kim et al., 2021). Models that jointly predict task labels and generate free-text explanations for their predictions (as in the previous example) are known as self-rationalization models . Their explanations are arguably more faithful and stable than post-hoc explanations since * Equal contributions. they are intrinsic to the model (Melis and Jaakkola, 2018). The free-text format is essential for explaining tasks requiring reasoning about unstated knowledge such as commonsense (Marasović et al., 2020), and it makes explanations more intuitive to people compared to highlights of individual words (Camburu et al., 2018). Despite these benefits, selfrationalization models are not widely used, in part because their training currently requires an abundance of human-authored explanations for each task (Narang et al., 2020). A possible solution is few-shot learning, which has shown promising results in recent years. To help the research community begin tackling self-rationalization with only a few examples, we present (i) FEB-a standardized collection of four existing English-language datasets and associated metrics, and (ii) the first approach for the task established through an extensive evaluation of natural language prompts. 1 One approach to few-shot learning is promptbased finetuning with natural language prompts. Such prompts are produced by formatting finetuning instances using a format similar to that used in pretraining, based on the idea that finetuning examples that look similar to pretraining ones will be more informative in the fewshot setting. A few prompts are then used for finetuning. In this paper, we explore whether prompt-based finetuning can be extended to induce few-shot self-rationalization behavior in addition to few-shot prediction. To measure our progress, we first introduce FEB as a benchmark dataset consisting of human authored free-text explanations across four distinct end tasks including natural language inference and commonsense tasks ( §2). Since finding appropriate prompts is often challenging (Gao et al., 2021), we then extensively explore natural language prompts for few-shot self-rationalization. In our experiments, we fine-tune the T5 and UNIFIEDQA pretrained encoder-decoder transformers (Raffel et al., 2020;Khashabi et al., 2020), and show that versatile question-answering prompts (defined in §3.1) outperform prompts based on span infilling by 8.7 accuracy points, as well as prompts designed by following the most similar T5's supervised pretraining task by 3.2.
Few-Shot Self-Rationalization with Natural Language Prompts
We then study the impact of model size on fewshot self-rationalization to investigate whether the quality of generated explanations scales with the size as good as the accuracy of predicting task labels. To this end, we also evaluate GPT-3's (Brown et al., 2020) self-rationalization behavior. Our experiments show that explanation plausibility scored by human annotators (which range from 0-100) and end-task accuracy improve with increasing model size. Specifically, the difference in plausibility scores between the BASE and 3B model ranges from [6.2, 24.8] (on average 14.8). The average plausibility across datasets is 43.4 (UNIFIEDQA-3B) and 50.6 (GPT-3). While encouraging, our results show that there is still a large gap between model and human performance (25.7 for GPT-3), and we hope this work will help enable the research community to take on the few-shot selfrationalization challenge.
Few-Shot Self-Rationalization with Natural Language Prompts
Our code for producing data splits, prompt construction, model training/evaluation, and human evaluation templates are publicly available. 2
Few-Shot Self-Rationalization with Natural Language Prompts
There has been an explosion of interest in generating free-text explanations and in few-shot learning in the last 1-2 years. However, appropriate datasets and metrics for few-shot self-rationalization have not yet been established. We thus introduce the FEB benchmark-a suite of existing Englishlanguage datasets with human-authored free-text explanations and associated metrics for few-shot self-rationalization. We expect that FEB will simplify future model comparison and lower barriers to entry for those interested in working on this task.
Few-Shot Self-Rationalization with Natural Language Prompts
Datasets in FEB To identify available datasets suitable for few-shot self-rationalization, we start with a recent overview of datasets with free-text explanations  and filter them according to the following criteria: (i) the input is textual, (ii) the explanation consists of one sentence or 2-3 simple sentences, (iii) the task has a fixed set of possible labels, (iv) the explanation is  (Camburu et al., 2018), ECQA (Aggarwal et al., 2021), COMVE , SBIC (Sap et al., 2020).
Few-Shot Self-Rationalization with Natural Language Prompts
human-authored, and (v) the dataset has at least 389 instances. We use the second and third criteria to narrow the scope to easier self-rationalization since we expect that few-shot self-rationalization is very challenging. The last requirement is introduced to have 48 training and 350 evaluation examples. This gives us 5 datasets, 4 of which are included in FEB and overviewed in Table 1. These datasets span 4 different tasks: natural language inference, multiple-choice commonsense QA, nonsensical sentence selection, and offensiveness classification. We exclude COS-E (Rajani et al., 2019) as it is too noisy to be useful for modeling and evaluating self-rationalization (Narang et al., 2020), but we do not support using COS-E in the future, especially since ECQA is introduced. 3 ECQA contains not only justifications of the correct answer, but also justifications that refute the incorrect answer choices. We use only the former since they answer "why is [input] assigned [label]?", just as explanations in other datasets that we have included in FEB. The SBIC dataset contains annotations of frames representing the social biases that are implied in language. We format these frames as a self-rationalization task as follows. We allow only two labels: "offensive" and "not offensive". If a post is not offensive, we assign it the explanation: "This post does not imply anything offensive." A post can be offensive because it targets an individual or a demographic group. In the former cases, a post is assigned the explanation: "This post is a personal attack." Otherwise, we define a set of rules to transform SBIC annota-tions of which identity-based group is targeted and what stereotypes of this group are referenced or implied into a single, coherent sentence; e.g., group: women, stereotype: can't drive → "This post is offensive because it implies that women can't drive". This is, to the best of our knowledge, the most comprehensive collection of self-rationalization tasks with textual inputs that could also be used even when working in a high-resource setting.
Few-Shot Self-Rationalization with Natural Language Prompts
Evaluation Evaluating selfrationalization (predicting task labels and generating explanations for the predicted labels) requires end-task evaluation and assessing the explanation plausibility. We use accuracy as our end-task evaluation metric. Explanation plausibility may be described as a subjective satisfaction with how a given explanation justifies a label/answer . Kayser et al. (2021) present the largest currently available study on the correlation of 10 NLG metrics with human judgments of free-text explanation plausibility and report that BERTscore (Zhang et al., 2020) is most correlated (although the correlation is still weak). Thus, we use BERTscore to evaluate the similarity between gold and generated explanations. Following Kayser et al., we assign zero BERTscore to explanations of incorrectly predicted instances. 4 We follow recent recommendations for reliable few-shot evaluation (Bragg et al., 2021 Table 7 in Appendix.
Few-Shot Self-Rationalization with Natural Language Prompts
Human Evaluation For our final models ( §4), we conduct a human evaluation of plausibility of generated explanations following prior work (Kayser et al., 2021;Marasović et al., 2020). For each model evaluation, Kayser et al. (2021) take the first 300 dev examples that are correctly predicted by the model. This means that the dev set subsets used for human evaluation differ across models that are evaluated. However, the overlap between the evaluation sets is maximized by fixing the order of dev instances and taking the first 300.
Few-Shot Self-Rationalization with Natural Language Prompts
Prior work used a single train-dev split, while FEB has 60 train-dev splits. Multiple splits provides the opportunity to account for the variance caused by changing the random seed to produce a reliable estimate of plausibility of explanations produced with only a few examples. Therefore, we take the first 6 correctly predicted examples per train-dev split, i.e., 6*60=360 total instances. Moreover, for classification tasks, we propose to take the first 6/#labels correctly-predicted examples per label to have a balanced evaluation set.
Few-Shot Self-Rationalization with Natural Language Prompts
Following Kayser et al. (2021), we conduct the human evaluation in two steps:
Few-Shot Self-Rationalization with Natural Language Prompts
• Step1: Select the correct label/answer. • Step2: Assess whether two explanations (gold and generated) justify the label/answer above.
Few-Shot Self-Rationalization with Natural Language Prompts
The first step makes sure the annotators understood the task correctly and they are not able to submit their annotations if the answers are wrong. 7 Ground-truth explanations are evaluated to implicitly influence annotators with a gold reference point when they evaluate generated explanations, and to measure the quality of explanation datasets. To evaluate explanations, annotators are asked "Does the explanation justify the answer?" and given the options {"yes","weak yes","weak no","no"}. These options are mapped to plausibility scores of {1, 2 3 , 1 3 ,0}, respectively. For each of the 360 examples, we calculate the mean plausibility score of 3 annotators and report the mean and the standard error of 360 mean scores. We also report the inter-annotator agreement calculated with Fleiss' kappa. Finally, models are evaluated independently to avoid penalizing worse models in the presence of explanations generated by a better model.
Few-Shot Self-Rationalization with Natural Language Prompts
We approach few-shot self-rationalization with prompt-based finetuning using natural language (NL) prompts. The key idea behind NL prompts is that a pretrained language model (LM) is already well-positioned to solve the end-task if we format finetuning end-task examples as similar as possible to the format used in the LM's pretraining. Following that principle, in this section, we describe our prompting approach with T5 (Raffel et al., 2020) and comprehensively evaluate three distinct prompt types with FEB. Our results show that a unified question-answering (QA) prompt combined with a T5 variant that includes additional supervised multitask QA training (UNIFIEDQA; Khashabi et al., 2020) performs the best overall across tasks, when compared to three different alternative prompts as described below.
Few-Shot Self-Rationalization with Natural Language Prompts
Self-rationalization models (Narang et al., 2020; are currently based on T5 for at least two reasons. First, T5 has been pretrained with many supervised tasks including classification and generation tasks, and self-rationalization involves both classification and generation. Second, T5 is one of the largest open-sourced and widely studied pretrained models, and higher LM performance is correlated with larger model size (Kaplan et al., 2020). Thus, all of our experiments are based on T5 (and the UNIFIEDQA variant when evaluating prompts based on a QA format). In this section, all results are obtained with the base version of these models and in §4 we scale model size.
Few-Shot Self-Rationalization with Natural Language Prompts
When a LM is pretrained with masked language modeling (Devlin et al., 2019) only, an appropriate NL prompt is constructed by adding and infilling masked tokens (Jiang et al., 2020). T5, however, has been pretrained with span infilling and a suite of supervised tasks whose instances were formatted in various ways. One of these supervised tasks includes SQUAD 1.1 (Rajpurkar et al., 2016) which allows us to experiment with prompts based on QA templates. As a result, we were able to design several different types of NL prompts for T5 consistent with different aspects of its pretraining:
Few-Shot Self-Rationalization with Natural Language Prompts
1. QA prompts ( SQUAD T5 , QA SIMPLE ). 2. span-filling prompts ( INFILLING ), 3. prompts designed by following the formatting of the most similar T5's pretraining task ( ≈T5 ; see Table 6 in Appendix), We illustrate these prompt types for COMVE in Table 11 in Appendix. The following sections de-scribe these formats in detail and compare their performance using FEB.
Few-Shot Self-Rationalization with Natural Language Prompts
Formatting new instances as QA pairs has been shown to be useful for transfer learning from a QA model (Gardner et al., 2019). We first evaluate options for a versatile QA NL prompt for self-rationalization of tasks in FEB before comparing this approach with the other two prompt types ( INFILLING and ≈T5 ) in §3.3. As alternative QA models, we investigate two models: T5 (which has been pretrained with QA supervision from SQUAD 1.1), and UNIFIEDQA (a T5 variant described in detail below). Since UNIFIEDQA was trained on a multitask mixture of many different QA datasets, these T5 variants allow us to examine the extent to which additional QA supervision can transfer to the few-shot self-rationalization setting.
Few-Shot Self-Rationalization with Natural Language Prompts
Prior work (Bragg et al., 2021) introduced UNIFEW, a model based on UNIFIEDQA, that is finetuned on a few task-specific instances posed as QA. Despite its simplicity, UNIFEW achieves competitive few-shot learning performance with strong baselines for classification tasks. However, Bragg et al.'s prompts do not cover all task types in FEB, and the question structure in their prompts is highly task-specific (see Appendix A.1).
Few-Shot Self-Rationalization with Natural Language Prompts
Alternatively, we propose to design QA prompts with a simple principle in mind: Given a non-QA task, construct an equivalent QA task in the form of short "Is...?" or "What is...?" questions. Here, "Is...?" questions have yes/no answers (sometimes "maybe"), and task labels verbatim are answers to "What is...?" questions (e.g., "offensive" and "not offensive"). Given such question-answer pairs, we develop prompts following the formats proposed in UNIFIEDQA (see Appendix A) and prompt UNI-FIEDQA. We denote these prompts as QA SIMPLE . For T5, we develop prompts following the SQUAD format for the T5's pretraining ( SQUAD T5 ; see Appendix A).
Few-Shot Self-Rationalization with Natural Language Prompts
There is another factor to consider. We need to decide whether to add tags-a single description of each input element. Examples of tags are "premise:" and "hypothesis:" before the first and second sentence in the E-SNLI input. Without these tags the task seems impossible to understand, but UNIFIEDQA has not been trained with any tags.
Few-Shot Self-Rationalization with Natural Language Prompts
The output always takes the form of "[answer/label] because [explanation]". See Table   11 (Appendix) for examples of our various QA prompts.
Few-Shot Self-Rationalization with Natural Language Prompts
We present the results of UNIFIEDQA with QA SIMPLE in Table 2, and due to space limits, T5's results with SQUAD T5 prompts in Table 10 in Appendix.
Few-Shot Self-Rationalization with Natural Language Prompts
We observe that for E-SNLI and COMVE it is crucial to add tags ("premise:"/"hypothesis:"; "choice1:"/"choice2:"). 8 This result is intuitiveit should be difficult to pick one of the two sentences, or classify a relation between them, if sentences are not marked. On the other hand, adding label choices is not beneficial and in some cases can even decrease the performance. When tags are included, we see that across all the tasks the "What is...?" question performs the best. This also holds for T5 and SQUAD T5 prompts (see Table 10). Finally, the prompt with the "What is...?" question and tags in the input outperforms UNIFEW for both tasks UNIFEW can be applied to. This result shows that this prompt is both versatile and effective.
Few-Shot Self-Rationalization with Natural Language Prompts
Finally, we compare the best performing prompts we get with UNIFIEDQA with QA SIMPLE and T5 wtih SQUAD T5 . See prompts " SQUAD T5 × WHAT IS...? + TAGS" and " QA SIMPLE × WHAT IS...? + TAGS" in Table 11. For ECQA and COMVE, we observe notable improvements from using UNIFIEDQA, and minor improvements for SBIC. For E-SNLI, T5 is better, presumably because UNIFIEDQA has lost some useful information from NLI after extensive continued pretraining for QA. These results suggest that UNIFIEDQA is a better model for prompting self-rationalization with QA prompts.
Few-Shot Self-Rationalization with Natural Language Prompts
To recap, the analysis presented in this section suggests that QA prompting for inducing selfrationalization behavior is best done when UNI-FIEDQA is combined with the NL prompt below. For true QA tasks, we use the original UNIFIEDQA formats. 9  
Few-Shot Self-Rationalization with Natural Language Prompts
The simplest way to design an infilling prompt for self-rationalization with T5 is to add the span "<extra_id_0> because <extra_id_1>" to the input. A model should then replace <extra_id_0> with a label/answer and <extra_id_1> with an explanation. Besides being similar to T5's span infilling pretraining task, another benefit of this prompt is that it is very flexible-the span above can be added to any task input. This basic infilling prompt could be easily made more natural by prepending phrases such as: "The answer is" (ECQA), "Less common is" (COMVE), or "This  is" (E-SNLI, SBIC). We hypothesize that these additional phrases could be beneficial because they suggest which subset of the vocabulary is the right word for filling in <extra_id_0>. We test whether it is beneficial to make the infilling prompt more natural-sounding.
Few-Shot Self-Rationalization with Natural Language Prompts
Results T5 results are shown in Table 3. The outcome is mixed-while we observe notable benefits for ECQA/SBIC, for E-SNLI/COMVE there is a minor difference in favor of the basic prompt. A way to explain this is that T5 learned about NLI labels from MNLI during pretraining, so it does not an need additional phrase to nudge it in the right direction. COMVE results are comparable to the random performance, and the model could not learn the task from the infilling prompt, with or without the additional phrases. Thus, we recommend using the more natural version as it is not detrimental to E-SNLI/COMVE performance while it leads to big improvements for ECQA/SBIC.
Few-Shot Self-Rationalization with Natural Language Prompts
We have established appropriate QA and IN-FILLING prompts in §3.1 and §3.2. We now turn to a comparison between all three prompt types: (i) INFILLING (natural), (ii) ≈T5 , and (iii) QA SIMPLE ("What is...?" with tags). The first two are used to prompt T5 and the last type UNI-FIEDQA. To construct ≈T5 prompts, for each task in FEB, we identify the most similar T5's pretraining task (see Table 6, Appendix) and use that task's formatting (see, e.g., ≈T5 × COPA in Table 11).
Few-Shot Self-Rationalization with Natural Language Prompts
Results A comparison of the three prompt types is presented in  cept E-SNLI for which unsurprisingly ≈T5 is the best. Finally, this brings us to the end of our extensive exploration of natural language prompts for a prompt-based finetuning approach to few-shot selfrationalization. We identify the QA SIMPLE prompt as the most effective and we use it to study how few-shot self-rationalization performance scales with the size of the UNIFIEDQA model.
Few-Shot Self-Rationalization with Natural Language Prompts
In §3, we discovered that a QA prompt combined with the base UNIFIEDQA model version is as an effective combination for few-shot selfrationalization through prompt-based finetuning.
Few-Shot Self-Rationalization with Natural Language Prompts
In this section, we provide two additional evaluations to establish the first approach to few-shot self-rationalization. First, we assess how plausible the generated explanations are when evaluated by annotators on Amazon MTurk. Details of how we conduct human evaluation of plausibility are given in §2. One HIT contains 10 instances and we pay $1 per HIT. Next, we investigate how self-rationalization performance changes with the model size since larger pretrained language models typically give better few-shot performance (Brown et al., 2020). We wonder whether the same trend will hold for a complex generation task of self-rationalization where  it is conceivable that an enormous model could overfit on a few examples. To this end, we evaluate three versions of UNIFIEDQA (BASE, LARGE, 3B) and GPT-3 (Brown et al., 2020). We use davinci-instruct-beta which is a beta version of the INSTRUCTGPT model (Ouyang et al., 2022).
Few-Shot Self-Rationalization with Natural Language Prompts
We evaluate GPT-3 using its API and "incontext demonstrations" (Brown et al., 2020). We pack as many training examples (demonstrations) as we can fit in the input, followed by the input of the test example, then run GPT-3 to generate its output. The number of demonstrations we are able to fit ranges from [28,45] which are randomly selected from the 48 used for UNIFIEDQA. Since evaluation using a single prompt costs us $1,050, we do not do prompt search for GPT-3. We use the prompts shown in Fig. 1 in Appendix.
Few-Shot Self-Rationalization with Natural Language Prompts
A detailed description of evaluation metrics is given in §2. The dev set size (of each out of 60 dev sets) for GPT-3 is 18 instead of 350 (because of the API cost). Ground-truth explanations are evaluated together with explanations generated by 4 models. Therefore, for GOLD explanations, we report the average of 4 plausibility scores, std. errors, and κ values calculated with 4 Mturk batches (corresponding to 4 models).
Few-Shot Self-Rationalization with Natural Language Prompts
Results are shown in Table 5. Note that we use T5 with the ≈T5 prompt for E-SNLI, and UNI-FIEDQA with QA SIMPLE ( §3) for other datasets to establish the best possible performance for each dataset. The exact prompts for each task are given in Appendix A.2. We observe that all metrics-accuracy, BERTscore, and plausibilitymonotonically increase with the model size for all datasets. That is, larger models learn to predict task labels and generate explanations from a few examples better. UNIFIEDQA-3B has a higher accuracy/BERTscore than GPT-3 for all datasets except SBIC, but GPT-3 generates explanations that are notably more plausible.
Few-Shot Self-Rationalization with Natural Language Prompts
The following observations suggest that few-shot self-rationalization is a promising research direction. The difference in plausibility scores between the BASE and 3B model versions ranges from [6.2, 24.8] (on average 14.8). In other words, since it is possible to generate more plausible explanations by only increasing the model size, it is conceivable that further progress could be made with more creative approaches. Next, the plausibility score of the best model (GPT-3) ranges from [42.2, 72.7] ([42.2, 52.6] if we consider only SBIC "offensive" (Label 1 ) subset. This shows that a moderate plausibility can already be achieved with current models without any task-specific enhancements.
Few-Shot Self-Rationalization with Natural Language Prompts
Despite that, the gap between our best models and human-authored explanations remains large. The average plausibility score across datasets is 43.4 (UNIFIEDQA-3B), 50.6 (GPT-3), and 76.3 (GOLD). In other words, the difference in plausibility scores between UNIFIEDQA-3B's and human explanations is 33.0, and between GPT-3's and human explanations is 25.7. We expect that the FEB benchmark, our UNIFIEDQA approach, and first results, present a good starting point to tackle this challenge.
Few-Shot Self-Rationalization with Natural Language Prompts
Performance w.r.t. Labels For E-SNLI and SBIC, we can inspect the metrics with respect to labels. In E-SNLI part of the Table 5, Label 1 marks "entailment", Label 2 "neutral", and Label 3 "contradiction". There are notable differences between the plausibility scores for each label. The plausibility score for "entailment" does not scale with the model size and it is much lower than scores for other labels (the best score is 27.3 vs. 66.0/50.9). This issue stems from the difficulty of explaining the entailment label (Camburu et al., 2018). Even people struggle with explaining "entailment" as evident by the lower GOLD score for "entailment" compared to the other two labels. An interesting observation from the other two labels is that UNI-FIEDQA-3B is explains "contradiction" instances best and GPT-3 "neutral" instances.
Few-Shot Self-Rationalization with Natural Language Prompts
In SBIC part of the Table 5, Label 1 marks "offensive" and Label 2 "not offensive" instances. The latter achieve almost perfect plausibility since the models learn to generated "This post does not imply anything offensive". Thus, main plausibility scores for SBIC are those of offensive instances. We can observe that the relative differences between models for offensive instances are much larger than the relative differences when examples of both labels are accounted for (column "All / Score"). If we had only looked into a single plausibility score we would not notice these differences. This result is in line with Carton et al. (2020) who also recommend breaking down the evaluation of explanations w.r.t. labels whenever possible.
Few-Shot Self-Rationalization with Natural Language Prompts
Annotator Agreement Finally, we observe challenges in collecting human judgments of plausibility. For all datasets except ECQA, Fleiss' κ is either moderate (between 0.41-0.6) or substantial (between 0.61-0.8). One exception is GPT-3 on SBIC (Label 1 ; offensive) where κ is only 0.34. We also observe that κ for GPT-3's explanations is lower than κ for UNIFIEDQA's or GOLD explanations, with the exception of COMVE. The most concerning is ECQA where κ is on average 0.35 for UNIFIEDQA's explanations, 0.34 for GOLD explanations, and only 0.12 for GPT-3's. Future work should investigate the reasons behind these differences more carefully.
Few-Shot Self-Rationalization with Natural Language Prompts
We study natural language prompts (Brown et al., 2020;Schick and Schütze, 2021) to establish the first approach to few-shot self-rationalization. Alternatively, few-shot learning researchers are studying prompts in the form of continuous/soft vectors that do not correspond to real tokens (e.g., Qin and Eisner, 2021). Such methods present a promising research direction for few-shot self-rationalization. Namely, we show that larger models generate notably more plausible explanations, and "prefix tuning" (Li and Liang, 2021) has been show to learn two condition generation tasks using only 0.1% of the parameters, while maintaining comparable performance. In practice, such approaches still require a notable amount of GPU memory. Thus, any efforts to reduce required memory such as compression (Ganesh et al., 2021) may be valuable for few-shot self-rationalization.
Few-Shot Self-Rationalization with Natural Language Prompts
We draw attention to the task of few-shot selfrationalization: predicting task labels and generating free-text explanations for the prediction using only a few human-written explanations. We present (i) the FEB benchmark, (ii) the first prompting approach for FEB established through a comprehensive search of natural language prompts, and (iii) results using models with a number of parameters ranging from 220M to 175B. Our human evaluation results show that progress is possible on this task given that just scaling the model size increases both the plausibility of generated explanations and task accuracy by a very large margin. Despite that, few-shot self-rationalization remains very challenging, with the plausibility of explanations generated by the best model being 27.7 points behind that of human-authored explanations. We hope that work presented in this paper spurs the community to work on this challenging problem to enable more intuitive interaction with NLP systems.   Tasks   E-SNLI MNLI (Williams et al., 2018) Classify the entailment relation between two sequences ECQA RECORD (Zhang et al., 2018) Answer a cloze-style query about a passage given entities in it COMVE COPA (Roemmele et al., 2011) Select one of two sequences as the cause/effect of a premise SBIC COLA (Warstadt et al., 2019) Classify a sentence as acceptable or not Table 6: The first column shows tasks that we have included in FEB. Tasks on the right are included in T5's pretraining and they are similar to FEB's tasks. We explore self-rationalization prompts for FEB's tasks based on the tasks on the right, and compare them to prompts designed as span infilling and QA ( §3).