What are the existing approaches for zero-shot commonsense reasoning?
Zero-shot commonsense reasoning. We categorize zero-shot approaches for commonsense reasoning into two groups. The first group uses pretrained language models as a source of world knowledge. Shwartz et al. (2020); Zhou et al. (2021a) query the language models with information seeking questions to identify background knowledge relevant to specific examples, and the answers returned by the models are later used as additional information for producing the final outputs. Dou and Peng(2022) convert multiple-choice QA to cloze-style sentences and have the language models score different answers. Qin et al. (2020) proposed a decoding algorithm that generates free-form explanations by considering the future contexts through backpropagation. Our approach also uses pretrained language models as a source of knowledge, but we perform additional maximum likelihood fine-tuning to fit the abductive task data. The second group leverages external knowledge bases (KBs). Bosselut et al. (2021) leverage COMET (Bosselut et al., 2019), a dynamic knowl-
edge graph, to generate a chain of commonsense inferences based on contexts of QA examples, which can be treated as explanations. Banerjee and Baral(2020); Ma et al. (2021) pretrain language models on artificial question answering (QA) datasets, created from knowledge graphs; a system trained on such datasets can directly perform zero-shot QA. Huang et al. (2021) formulate multiple-choice QA as natural language inference (NLI) and leverage both existing NLI datasets and KBs to identify answer choices in a zero-shot manner
Think Before You Speak: Learning to Generate Implicit Knowledge for Response Generation by Self-Talk
Humans make appropriate responses not only based on previous dialogue utterances but also on implicit background knowledge such as common sense. Although neural response generation models seem to produce human-like responses, they are mostly end-to-end and not generating intermediate grounds between a dialogue history and responses. This work aims to study if and how we can train an RG model that talks with itself to generate implicit knowledge before making responses. We further investigate can such models identify when to generate implicit background knowledge and when it is not necessary. Experimental results show that compared with models that directly generate responses given a dialogue history, self-talk models produce better-quality responses according to human evaluation on grammaticality, coherence, and engagingness. And models that are trained to identify when to self-talk further improves the response quality. Analysis on generated implicit knowledge shows that models mostly use the knowledge appropriately in the responses.
[{"first":"Pei","last":"Zhou","middle":[]},{"first":"Behnam","last":"Hedayatnia","middle":[]},{"first":"Karthik","last":"Gopalakrishnan","middle":[]},{"first":"Seokhwan","last":"Kim","middle":[]},{"first":"Jay","last":"Pujara","middle":[]},{"first":"Xiang","last":"Ren","middle":[]},{"first":"Yang","last":"Liu","middle":[]},{"first":"Dilek","last":"Hakkani-Tur","middle":[]}]
Unsupervised Commonsense Question Answering with Self-Talk
Natural language understanding involves reading between the lines with implicit background knowledge. Current systems either rely on pre-trained language models as the sole implicit source of world knowledge, or resort to external knowledge bases (KBs) to incorporate additional relevant knowledge. We propose an unsupervised framework based on \emph{self-talk} as a novel alternative to multiple-choice commonsense tasks. Inspired by inquiry-based discovery learning (Bruner, 1961), our approach inquires language models with a number of information seeking questions such as"$\textit{what is the definition of ...}$"to discover additional background knowledge. Empirical results demonstrate that the self-talk procedure substantially improves the performance of zero-shot language model baselines on four out of six commonsense benchmarks, and competes with models that obtain knowledge from external KBs. While our approach improves performance on several benchmarks, the self-talk induced knowledge even when leading to correct answers is not always seen as useful by human judges, raising interesting questions about the inner-workings of pre-trained language models for commonsense reasoning.
[{"first":"Vered","last":"Shwartz","middle":[]},{"first":"Peter","last":"West","middle":[]},{"first":"Ronan","last":"Bras","middle":["Le"]},{"first":"Chandra","last":"Bhagavatula","middle":[]},{"first":"Yejin","last":"Choi","middle":[]}]
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
The aim of all Question Answering (QA) systems is to be able to generalize to unseen questions. Most of the current methods rely on learning every possible scenario which is reliant on expensive data annotation. Moreover, such annotations can introduce unintended bias which makes systems focus more on the bias than the actual task. In this work, we propose Knowledge Triplet Learning, a self-supervised task over knowledge graphs. We propose methods of how to use such a model to perform zero-shot QA and our experiments show considerable improvements over large pre-trained generative models.
[{"first":"Pratyay","last":"Banerjee","middle":[]},{"first":"Chitta","last":"Baral","middle":[]}]
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Understanding narratives requires reasoning about implicit world knowledge related to the causes, effects, and states of situations described in text. At the core of this challenge is how to access contextually relevant knowledge on demand and reason over it. In this paper, we present initial studies toward zero-shot com- monsense question answering by formulating the task as inference over dynamically generated commonsense knowledge graphs. In contrast to previous studies for knowledge integration that rely on retrieval of existing knowledge from static knowledge graphs, our study requires commonsense knowl- edge integration where contextually relevant knowledge is often not present in existing knowledge bases. Therefore, we present a novel approach that generates contextually-relevant symbolic knowledge structures on demand using generative neural commonsense knowledge models . Empirical results on two datasets demonstrate the efﬁcacy of our neuro-symbolic approach for dynamically constructing knowledge graphs for reasoning. Our approach achieves signif- icant performance boosts over pretrained language models and vanilla knowledge models, all while providing interpretable reasoning paths for its predictions.
[{"first":"Antoine","last":"Bosselut","middle":[]},{"first":"Ronan","last":"Le Bras","middle":[]},{"first":"Yejin","last":"Choi","middle":[]}]
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Recent developments in pre-trained neural language modeling have led to leaps in accuracy on commonsense question-answering benchmarks. However, there is increasing concern that models overﬁt to speciﬁc tasks, without learning to utilize external knowledge or perform general semantic reasoning. In contrast, zero-shot evaluations have shown promise as a more robust measure of a model’s general reasoning abilities. In this paper, we propose a novel neuro-symbolic framework for zero-shot question answering across commonsense tasks. Guided by a set of hypotheses, the framework studies how to transform various pre-existing knowledge resources into a form that is most effective for pre- training models. We vary the set of language models, training regimes, knowledge sources, and data genera- tion strategies, and measure their impact across tasks. Extending on prior work, we devise and compare four constrained distractor-sampling strategies. We provide empirical results across ﬁve commonsense question- answering tasks with data generated from ﬁve external knowledge resources. We show that, while an individ- ual knowledge graph is better suited for speciﬁc tasks, a global knowledge graph brings consistent gains across different tasks. In addition, both preserving the structure of the task as well as generating fair and informative questions help language models learn more effectively.
[{"first":"Kaixin","last":"Ma","middle":[]},{"first":"Filip","last":"Ilievski","middle":[]},{"first":"Jonathan","last":"Francis","middle":[]},{"first":"Yonatan","last":"Bisk","middle":[]},{"first":"Eric","last":"Nyberg","middle":[]},{"first":"Alessandro","last":"Oltramari","middle":[]}]
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
Recent methods based on pre-trained language models have shown strong supervised performance on commonsense reasoning. However, they rely on expensive data annotation and time-consuming training. Thus, we focus on unsupervised commonsense reasoning. We show the effectiveness of using a common framework, Natural Language Inference (NLI), to solve diverse commonsense reasoning tasks. By leveraging transfer learning from large NLI datasets, and injecting crucial knowledge from commonsense sources such as ATOMIC 2020 and ConceptNet, our method achieved state-of-the-art unsupervised performance on two commonsense reasoning tasks: WinoWhy and CommonsenseQA. Further analysis demonstrated the beneﬁts of multiple categories of knowledge, but problems about quantities and antonyms are still challenging.
[{"first":"Canming","last":"Huang","middle":[]},{"first":"Weinan","last":"He","middle":[]},{"first":"Yongmei","last":"Liu","middle":[]}]
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Abductive and counterfactual reasoning, core abilities of everyday human cognition, require reasoning about what might have happened at time t, while conditioning on multiple contexts from the relative past and future. However, simultaneous incorporation of past and future contexts using generative language models (LMs) can be challenging, as they are trained either to condition only on the past context or to perform narrowly scoped text-infilling. In this paper, we propose DeLorean, a new unsupervised decoding algorithm that can flexibly incorporate both the past and future contexts using only off-the-shelf, left-to-right language models and no supervision. The key intuition of our algorithm is incorporating the future through back-propagation, during which, we only update the internal representation of the output while fixing the model parameters. By alternating between forward and backward propagation, DeLorean can decode the output representation that reflects both the left and right contexts. We demonstrate that our approach is general and applicable to two nonmonotonic reasoning tasks: abductive text generation and counterfactual story revision, where DeLorean outperforms a range of unsupervised and some supervised methods, based on automatic and human evaluation.
[{"first":"Lianhui","last":"Qin","middle":[]},{"first":"Vered","last":"Shwartz","middle":[]},{"first":"Peter","last":"West","middle":[]},{"first":"Chandra","last":"Bhagavatula","middle":[]},{"first":"Jena","last":"Hwang","middle":[]},{"first":"Ronan","last":"Bras","middle":["Le"]},{"first":"Antoine","last":"Bosselut","middle":[]},{"first":"Yejin","last":"Choi","middle":[]}]
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Commonsense question answering (CQA) aims to test if models can answer questions regarding commonsense knowledge that everyone knows. Prior works that incorporate external knowledge bases have shown promising results, but knowledge bases are expensive to construct and are often limited to a fixed set of relations. In this paper, we instead focus on better utilizing the \textit{implicit knowledge} stored in pre-trained language models. While researchers have found that the knowledge embedded in pre-trained language models can be extracted by having them fill in the blanks of carefully designed prompts for relation extraction and text classification, it remains unclear if we can adopt this paradigm in CQA where the inputs and outputs take much more flexible forms. To this end, we investigate four translation methods that can translate natural questions into cloze-style sentences to better solicit commonsense knowledge from language models, including a syntactic-based model, an unsupervised neural model, and two supervised neural models. In addition, to combine the different translation methods, we propose to encourage consistency among model predictions on different translated questions with unlabeled data. We demonstrate the effectiveness of our methods on three CQA datasets in zero-shot settings. We show that our methods are complementary to a knowledge base improved model, and combining them can lead to state-of-the-art zero-shot performance. Analyses also reveal distinct characteristics of the different cloze translation methods and provide insights on why combining them can lead to great improvements.
[{"first":"Zi-Yi","last":"Dou","middle":[]},{"first":"Nanyun","last":"Peng","middle":[]}]
Think Before You Speak: Learning to Generate Implicit Knowledge for Response Generation by Self-Talk
End-to-end response generation (RG) models based on pre-trained transformer-based (Vaswani et al., 2017) language models (LM) have shown to produce human-like responses Lewis et al., 2020;Roller et al., 2020). Humans, however, not only "just utter the right sentence", but also contribute to the common ground consisting of mutual beliefs and common knowledge "whose truth is taken for granted as part of the background of the conversation" (Stalnaker, 1978;Clark and Schaefer, 1989;Clark and Brennan, 1991). For example, consider the utterance "I need to buy some flowers for my wife", a potential appropriate response is "Perhaps you'd be interested in red roses". To produce this response, the participant needs to understand relevant background knowledge such as "rose is a type of flower". Note that this background knowledge is implicit in the dialogue turns, meaning that an end-to-end RG system that takes utterances as input and outputs responses will omit this intermediate process that is crucial for humans in communications.
Think Before You Speak: Learning to Generate Implicit Knowledge for Response Generation by Self-Talk
To fill the gap between current model's RG process and how humans use implicit background knowledge in conversations, we focus on commonsense inference as a type of background knowledge and aim to train RG models that first generate implicit commonsense inferences before producing a response given previous utterances. Inspired by inquiry-based discovery learning (Bruner, 1961) and the self-talk procedure (Shwartz et al., 2020), we encourage the RG model to talk with itself to elicit implicit knowledge before making a response.
Think Before You Speak: Learning to Generate Implicit Knowledge for Response Generation by Self-Talk
Collecting training data is challenging for our purpose since commonsense knowledge is by definition often omitted in conversations. We use Con-ceptNet triples (Speer et al., 2017) as the knowl-edge schema to represent knowledge and align common sense-focused dialogues (Zhou et al., 2021) with knowledge to train RG models. We conduct extensive human evaluation on different variants of our training procedure and find that models that generate implicit background knowledge before responding produce more grammatical, coherent, and engaging responses compared to RG models that directly generate responses. Further analysis shows that models can sometimes even learn to distinguish when it is necessary to self-talk to generate implicit knowledge, i.e., be aware of potential knowledge gaps in dialogues. We hope our findings encourage more future studies on making RG models better emulate human communication process and produce better-quality responses.
Think Before You Speak: Learning to Generate Implicit Knowledge for Response Generation by Self-Talk
Given a dialogue history (multi-turn or single-turn) U , the task is to generate implicit background knowledge I and then produce a response R given both U and I. Extending most neural RG models that treat this as a conditional language modeling problem, we aim to learn the conditional probability distribution P (I, R|U ) by training on human dialogues.
Think Before You Speak: Learning to Generate Implicit Knowledge for Response Generation by Self-Talk
Knowledge Schema We consider Concept-Net (Speer et al., 2017) as our knowledge schema, which is a large-scale crowdsourced commonsense knowledge base consisting of triples such as "buy, RelatedTo, money". We have explored several other sources such as LMs trained to generate knowledge (Hwang et al., 2021;Becker et al., 2021) but observe much noise while aligning knowledge to dialogue turns.
Think Before You Speak: Learning to Generate Implicit Knowledge for Response Generation by Self-Talk
Dialogues We use dialogue datasets from Zhou et al. (2021) as they propose "common sensefocused dialogues" by filtering three existing di-alogue datasets DailyDialog (Li et al., 2017), Em-patheticDialogues (Rashkin et al., 2019), Mu-Tual (Cui et al., 2020) using ConceptNet triples, and also crowdsourced SocialIQA-prompted dialogues (Zhou et al., 2021). We extract dialogues that each has at least one matched triple from Con-ceptNet in one of its consecutive turn pairs, resulting in around 31k dialogues and 159k turns we can use for training instances (excluding the first turn). The total number of turn pairs that have at least one matched triple is around 57k.
Think Before You Speak: Learning to Generate Implicit Knowledge for Response Generation by Self-Talk
Each of our training instance is a sequence of tokens with three components: a dialog history U , implicit knowledge (empty or non-empty) I, and a response R. We enclose the implicit knowledge I with special symbols "<implicit>" and "< \implicit>" and add it between U and R. For example: "<bos> <speaker1> I need to buy some flowers for my wife. <implicit> rose is a type of flower < \implicit> <speaker2> Perhaps you'd be interested in red roses <eos>". We train the models to generate everything after <implicit> till <eos>, i.e. generate I and R conditioned on U . We use GPT2-medium (Radford et al.) and train the model for 3 epochs with batch size 4 and learning rate 6.25e-5.
Think Before You Speak: Learning to Generate Implicit Knowledge for Response Generation by Self-Talk
We also explore different ways to better train a model that can both self-talk and respond in dialogues. 1). Two-Step training Instead of jointly modeling P (I, R|U ) in a one-shot fashion, we consider split the instance to first train the model to generate I only based on U and then train it to generate R based on both I and U , i.e. P (I|U ) and then P (R|U, I).
Think Before You Speak: Learning to Generate Implicit Knowledge for Response Generation by Self-Talk
2). Learning to distinguish when knowledge is needed. In addition to using only the 57k instances where there is always non-empty implicit knowledge, we also evaluate using all dialogue instances, some of which have empty implicit knowledge. This allows us to evaluate if models can determine when to self-talk and when it is not needed (e.g., for simple chitchat utterances). 3). Imbalanced VS balanced training set Since we have more instances with empty implicit knowledge than nonempty knowledge (102k vs. 57k), we consider an imbalanced training set where we include all 159k instances and a balanced version where we randomly sample the same number of empty knowledge instances as non-empty ones (114k total).   
Think Before You Speak: Learning to Generate Implicit Knowledge for Response Generation by Self-Talk
For evaluation, we randomly sample 300 instances from unseen test dialogues for human evaluation.
Think Before You Speak: Learning to Generate Implicit Knowledge for Response Generation by Self-Talk
We provide a dialogue history and two model responses and ask three workers on Amazon Mechanical Turk (AMT) platform to select which one is better or not sure. We consider three criteria following most previous RG evaluation: which response is more grammatical, coherent, and engaging. We generally find trained models follow the template of first generating implicity knowledge and then respond. Some example outputs to the utterance "I need to buy some flowers for my wife" are "(buy is related to expensive) Can you afford the expensive flowers?" and "(buy is related to money) How much money do you want to spend?".
Think Before You Speak: Learning to Generate Implicit Knowledge for Response Generation by Self-Talk
Self-Talk models produce higher quality responses than vanilla RG models Table 1 shows that when comparing with a vanilla RG trained on dialogues, the two-step self-talk models improve on all three aspects, especially on coherence and engagingness. Table 2 shows that although imbalanced self-talk model contains 45k more training instances than the balanced version, models trained on balanced data perform better on all three criteria, posing interesting phenomenon for later studies.
Think Before You Speak: Learning to Generate Implicit Knowledge for Response Generation by Self-Talk
Due to page limit, we will summarize results for other model comparisons. We find two-step training outperforms one-step and learning to distinguish outperforms always outputs some implicit knowledge. We also find interesting new knowledge generated from models (not included in Con-ceptNet we used), e.g. "bus is located at airport".
Unsupervised Commonsense Question Answering with Self-Talk
Human level natural language understanding involves reading between the lines and relying on implicit background knowledge. Consider the scene in Figure 1: Alice let Bob stand in front of her at the concert. Using physical and social commonsense -(i) Bob and Alice want to see the stage, and (ii) If Bob is taller, they would block Alice's view -one can infer that Alice is taller than Bob. Such examples are ubiquitous across natural language understanding (NLU) tasks such as reading comprehension (Hirschman et al., 1999) and recognizing textual entailment (Dagan et al., 2013), and Alice let Bob stand in front of her at the concert. Most current NLU models rely on pre-trained language models (LMs; e.g. Radford et al., 2019;Devlin et al., 2019;. The standard practice is to use task-specific data to fine-tune a pre-trained LM in a supervised manner. Alternatively, LM score is used to rank answer choices in a zero-shot setup (Wang et al., 2019;Sakaguchi et al., 2020). In both setups, pre-trained LMs yield improved performance upon prior methods, greatly due to the world knowledge that such LMs capture, having been trained on massive texts (Petroni et al., 2019;Davison et al., 2019).
Unsupervised Commonsense Question Answering with Self-Talk
Despite the performance boost, LMs as knowledge providers suffer from various shortcomings: (i) insufficient coverage: due to reporting bias, many trivial facts might not be captured by LMs (purple set in Figure 1), because they are rarely written about (Gordon and Van Durme, 2013). (ii) insufficient precision: the distributional training objective increases the probability of non-facts (light green set in Figure 1) that are semantically similar to true facts, as in negation ("birds cannot fly"; Kassner and Schütze, 2019). LMs excel in predicting the semantic category of a missing word, but might predict the wrong instance in that category (e.g., depending on the phrasing, BERT sometimes Dataset Context + Question Choices
Unsupervised Commonsense Question Answering with Self-Talk
The man broke his toe. 1) He got a hole in his sock. What was the cause of this?
Unsupervised Commonsense Question Answering with Self-Talk
2) He dropped a hammer on his foot.
Unsupervised Commonsense Question Answering with Self-Talk
Where on a river can you hold a cup 1) waterfall 2) bridge 3) valley SenseQA upright to catch water on a sunny day? 4) pebble 5) mountain MC-TACO [...] dream of becoming a judge. How many 1) 63 years 2) 7 weeks years did it take for Mark to become a judge? 3) 7 years 4) 7 seconds 5) 7 hours
Unsupervised Commonsense Question Answering with Self-Talk
In the school play, Robin played a hero in the 1) sorry for the villain struggle to the death with the angry villain. 2) hopeful that Robin will succeed How would others feel as a result?
Unsupervised Commonsense Question Answering with Self-Talk
3) like Robin should lose the fight PIQA To separate egg whites from the yolk using a 1) [...] Release, which creates suction and lifts the yolk. water bottle, you should 2) [...] Keep pushing, which creates suction and lifts the yolk.
Unsupervised Commonsense Question Answering with Self-Talk
Katrina had the financial means to afford 1) Katrina a new car while Monica did not, 2) Monica since had a high paying job. Table 1: An example from each dataset used in this paper. The correct choice in each example is given in bold text.
Unsupervised Commonsense Question Answering with Self-Talk
predicts red as the color of a dove). Finally, (iii) it is unclear that LMs are capable of performing multiple reasoning steps involving implicit knowledge.
Unsupervised Commonsense Question Answering with Self-Talk
To increase the coverage of high-precision world knowledge and facilitate multi-hop reasoning by making intermediate reasoning steps explicit, prior work incorporated KBs (e.g. ConceptNet;Speer and Havasi, 2012) and knowledge-informed models into LM-based models (Xia et al., 2019;Bosselut and Choi, 2019;Chen et al., 2019).
Unsupervised Commonsense Question Answering with Self-Talk
In this paper, we study pre-trained LMs as an alternative to external KBs in providing knowledge to commonsense question answering tasks. We propose an unsupervised model that uses an LM as the answer scorer, and a (possibly different) LM as a knowledge source. We formulate the process of obtaining relevant knowledge as a self-talk, inquirybased discovery learning (Bruner, 1961), with the following steps: 1) seeking out knowledge by generating natural-language "clarification questions" conditioned on a given context; 2) generating their corresponding answers ("clarifications"); and 3) incorporating the clarifications as additional context.
Unsupervised Commonsense Question Answering with Self-Talk
Our model does not rely on external knowledge or additional supervision. Yet, we show that on 4 out of 6 tasks it substantially improves upon a zero-shot baseline that relies on LM score alone and performs on par, and sometimes better than, models that use external knowledge sources.
Unsupervised Commonsense Question Answering with Self-Talk
Integrating external knowledge warrants discerning relevant and helpful facts for solving a particular instance. LMs further require identifying that a clarification is factually-correct. We show that even among the clarifications that helped the pre-diction, humans perceived many as unhelpful or even incorrect, demonstrating that LM-based models often solve problems correctly for seemingly incorrect reasons. Our results call for future research on robust and correct knowledge integration to LM-based question answering systems.
Unsupervised Commonsense Question Answering with Self-Talk
We focused on the multiple-choice question answering tasks exemplified in Table 1 and detailed below. Each instance consists of an optional context, an optional question, and several answer choices. The development sets sizes vary from 100 (COPA) to 1,954 (Social IQa).
Unsupervised Commonsense Question Answering with Self-Talk
COPA: Choice of Plausible Alternatives (Gordon et al., 2012): Asking about either a plausible cause or a plausible result, among two alternatives, of a certain event expressed in a simple sentence.
Unsupervised Commonsense Question Answering with Self-Talk
Answering (Talmor et al., 2019). General questions about concepts from ConceptNet. To increase the challenge, the distractors are related to the target concept either by a relationship in ConceptNet or as suggested by crowdsourcing workers.
Unsupervised Commonsense Question Answering with Self-Talk
Multiple Choice Temporal commonsense (Zhou et al., 2019). Questions about temporal aspects of events such as ordering (Table 1), duration, stationarity, frequency, and typical time. The distractors were selected in an adversarial way using BERT. 1 Figure 2: Model illustration for WinoGrande. Each answer choice (Brett, Ian) is assigned to the concatenation of the context and a clarification. The score for each choice is the best LM score across clarifications (2 in this case).
Unsupervised Commonsense Question Answering with Self-Talk
Social IQa: Social Interaction Question Answering (Sap et al., 2019b). Questions regarding social interactions, based on the ATOMIC dataset (Sap et al., 2019a). Contexts describe social interactions and questions refer to one of a few aspects (e.g. the subject's motivation, following actions, etc.). The answers were crowdsourced. ing (Bisk et al., 2020). Questions regarding physical commonsense knowledge. Contexts are goals derived from an instruction website, typically involving less prototypical uses of everyday objects (e.g., using a bottle to separate eggs). The answers were crowdsourced, and an adversarial filtering algorithm was used to remove annotation artifacts. 2
Unsupervised Commonsense Question Answering with Self-Talk
WinoGrande (Sakaguchi et al., 2020). A largescale version of WSC that exhibits less bias thanks to adversarial filtering and use of placeholders instead of pronouns. As opposed to WSC that was curated by experts, WinoGrande was crowdsourced with a carefully designed approach that produces diverse examples which are trivial for humans.
Unsupervised Commonsense Question Answering with Self-Talk
A given instance consists of an optional context c, an optional question q, and answer choices: a k i=1 . We first describe the baseline model, which makes the prediction based on the instance alone (Section 3.1). We then describe a knowledgeinformed model that relies on external resources (Section 3.2). Finally, we discuss the proposed inquiry-based model, which uses a pre-trained LMs to produce clarifications (Section 3.3).
Unsupervised Commonsense Question Answering with Self-Talk
We use a pre-trained language model LM s to score the plausibility of different text fragments. We ex-periment with the various LMs provided by the transformers package (Wolf et al., 2019): GPT (Radford et al., 2018), GPT2 (Radford et al., 2019, all sizes), a distilled GPT2 (Sanh et al., 2019), and XLNet (Yang et al., 2019, both sizes).
Unsupervised Commonsense Question Answering with Self-Talk
We assign each of the answer choices a i into the combination of the context and the question, and obtain opt i = combine(c, q, a i ). The combine function is computed differently for each task. For example, in COPA, where the question might be either about the cause or the effect of the context, we create the following texts for cause: " [context]. As a result, [choice]" and for effect: "[context]. The cause for it was that [choice]".
Unsupervised Commonsense Question Answering with Self-Talk
We denote the score of each answer choice as score(a i ) = CE(opt i ), where CE is cross-entropy loss defined as:
Unsupervised Commonsense Question Answering with Self-Talk
We predict the a i with the lowest score as the correct answer, which is the most likely option according to LM s : y = argmin i score(a i ).
Unsupervised Commonsense Question Answering with Self-Talk
In the setup illustrated in Figure 2, each instance consists of an additional clarification list: CL = {cl 1 , ..., cl m }. Those are text fragments containing potentially relevant knowledge for solving the instance. For instance, the clarification "The purpose of the internship is to help people find jobs" might help answering the question "which of Brett and Ian found a job less quickly after graduation?". We don't expect all the clarifications to be relevant and helpful for answering the main question. Instead, the model relies on the single clarification that increases its belief of a certain answer choice. Thus, the score of each answer choice is selected as the score of the text containing the clarification that most supports it, i.e., whose combination with it yields the minimal loss: score(a i ) = min cl∈CL CE(opt i + cl). Again we predict y = argmin i score(a i ). We extract clarifications from the following sources, exemplified in Figure 3.
Unsupervised Commonsense Question Answering with Self-Talk
ConceptNet. Similarly to previous work, we extract relation paths between words from the context and the question, and words from the answer choices. Since we incorporate the knowledge into the model as text, we convert each ConceptNet relation to a natural language template as in Davison et al. (2019). We limit the path length to 2 edges in order to maintain high precision.
Unsupervised Commonsense Question Answering with Self-Talk
Corpus. For pairs of words from the context and question and from the answer choices, we extract their joint occurrences (with minimum frequency of 100) in Google N-grams (Brants and Franz). This yields text fragments of up to 5 words rather than well-formed sentences, with the potential of describing the relationship between the two words (Shwartz and Dagan, 2018).
Unsupervised Commonsense Question Answering with Self-Talk
COMET. COMET (Bosselut et al., 2019) is a knowledge base construction model trained on the ATOMIC resource (Sap et al., 2019a) which consists of everyday situations along with multiple commonsense dimensions such as their causes, effects, pre-and post-conditions, etc. We generate all the dimensions unless we can generate specific relations that are more likely to help. Specifically, in Social IQa, we heuristically try to understand which type of relation in COMET the question asks for. In COPA, we use the pre-condition relations for cause questions (xIntent, xNeed) and the postcondition relations for effect questions (xEffect, xReact, xWant, oEffect, oReact, oWant). When possible, we replace personX with the syntactic subject of the context or the question.
Unsupervised Commonsense Question Answering with Self-Talk
Our proposed model makes the prediction identically to Figure 2, but extracts the clarifications from pre-trained LMs. We treat the knowledge extrac-
Unsupervised Commonsense Question Answering with Self-Talk
The purpose of the internship is to help people find jobs.
Unsupervised Commonsense Question Answering with Self-Talk
Because Brett found an internship while in college but Ian was unable to, _ found a job less quickly after graduation.
Unsupervised Commonsense Question Answering with Self-Talk
What is the purpose of the internship?
Unsupervised Commonsense Question Answering with Self-Talk
Because Brett found an internship while in college but Ian was unable to, _ found a job less quickly after graduation.  Figure 4: Generating a clarification with LM: 1) Generate a question, conditioned on the context (pink) and question prefix (yellow). 2) Generate an answer, conditioned on the context, generated question and a corresponding answer prefix. The clarification is a concatenation of the answer prefix and generated text (green).
Unsupervised Commonsense Question Answering with Self-Talk
tion from LMs as a process of self-asking clarification questions about the context and "discovering" their answers. Figure 4 exemplifies this process for WinoGrande with a generator language model LM g . For the sake of simplicity, the illustration depicts the process of generating a single pair of clarification question and answer. We start by generating multiple clarification questions conditioned on the context, by 1) concatenating one of several question prefixes, which we curated for each task (e.g. "What is the purpose of", see the appendix); and 2) generating 5 questions for each prefix using Nucleus sampling with p = 0.2, i.e., sampling from the top 20% tokens (Holtzman et al., 2019). 3 We limit the question length to up to 6 tokens excluding the prefix.
Unsupervised Commonsense Question Answering with Self-Talk
For each well-formed question that we obtained at the previous step, e.g. "What is the purpose of the Austin did this because they wanted to keep him alive.
Unsupervised Commonsense Question Answering with Self-Talk
Austin fought for Quinn's life, but they eventually died on the operating table.
Unsupervised Commonsense Question Answering with Self-Talk
LM they wanted to keep him alive.
Unsupervised Commonsense Question Answering with Self-Talk
___ did this because p₂ p₂ Figure 5: Generating a clarification for Social IQa conditioned on the context, the given question (pink), and a heuristically matched answer prefix (yellow).
Unsupervised Commonsense Question Answering with Self-Talk
internship?", we generate multiple answers using a similar method. Each question prefix corresponds to an answer prefix. We use the concatenation of the context, generated clarification question, and answer prefix as the prompt for generating an answer (clarification). We limit the answer length to 10 generated tokens, and use Nucleus sampling with p = 0.5. We generate 10 answers for each clarification question and keep all the well-formed clarifications. Note that the clarification questions themselves are only means to generate the clarifications, and they are not used by our model.
Unsupervised Commonsense Question Answering with Self-Talk
In some datasets, an instance consists of both a context and a question. In this case, we can use the instance question as a "clarification" question and generate additional clarification questions similar to it. Figure 5 exemplifies this shortcut for Social IQa: instead of generating a clarification question, the given question "Why did Austin do this?" is used, and together with a heuristically matched answer prefix, the model can generate a potentially direct solution: "Austin did this because they wanted to keep him alive".
Unsupervised Commonsense Question Answering with Self-Talk
Since we did not train the clarification generator to ask sensical, relevant, and helpful questions, nor did we train the answer generator to generate coherent and factually correct answers, we can assume that some of the generated clarifications do not provide useful information to the model. Table 2 displays the performance of the best model in each category according to the development accuracy. We report the performance of the following models: majority baseline, LM baseline (Baseline), LM-based model with external knowledge  (Ext. Knowledge), Self-talk, supervised models from prior work when applicable (Pre. Sup), 4 and human performance. Our zero-shot models are highlighted in purple. As expected, the overall performance is worse for the zero-shot models compared to the state-ofthe-art supervised models, but they perform substantially better than the majority baselines on most tasks, with the exception of WinoGrande where they only slightly outperform it. Among the LMbased models, self-talk performs on par or within a few points from the external knowledge model.
Unsupervised Commonsense Question Answering with Self-Talk
Best LM. Table 3 shows the ranking of the LMs according to their development accuracy averaged across the different knowledge sources. In general there is a preference to GPT-2, and in particular to the larger models, except for COPA in which the distilled version works best. A possible explanation might be that the language model distillation reduces the likelihood of rare words (Tang and Lin,    2018), which works well for the simple sentences in COPA. The XLNet models perform poorly, perhaps due to their smaller training corpus (16GB vs 40GB in GPT-2, both using web text).
Unsupervised Commonsense Question Answering with Self-Talk
Best Knowledge Source. Among the knowledge informed models, COMET achieves the best performance across tasks. This likely happens first because COMET can dynamically generate predictions for any context, while the other two knowledge sources are static and lack coverage. Second, as expected, COMET improves the predictions for Social IQa, which was built based on the ATOMIC resource on which COMET is trained. Table 4 sorts the knowledge sources based on the average development accuracy across LMs. PIQA and MC-TACO, tasks that require different types of knowledge from social commonsense, perform well with ConceptNet and Google Ngrams. With respect to self-talk models, there is a rather small difference in performance between the different LMs used as knowledge sources, with slight preference to GPT-2 in most datasets.
Unsupervised Commonsense Question Answering with Self-Talk
We also experimented with combining the clarifications from all the knowledge sources, which didn't prove beneficial except for MC-TACO (where it added +7.9 points to the dev accuracy, bringing it to 66.7). We assume that some resources added noise, making the whole smaller than the sum of its parts.
Unsupervised Commonsense Question Answering with Self-Talk
While the performance on the end task serves as an extrinsic evaluation for the quality of the generated clarifications, we are also interested in evaluating it intrinsically. From preliminary experiments we know that there is a high ratio of noisy clarifications. Thus, we analyze the clarifications that help predict the correct answer, i.e. clarifications with the best LM score in their instance and whose existence change the answer from an incorrect prediction by the baseline to a correct prediction by the model.
Unsupervised Commonsense Question Answering with Self-Talk
We sampled up to 50 such clarifications for each combination of task and knowledge source, using the best performing LM. 5 We showed crowdsourcing workers an instance along with a clarification question and its answer, and asked them: 1) whether the question is grammatical, not entirely grammatical but understandable, or completely not understandable; and if the answer was anything but "completely not understandable", 2) whether the question is relevant, i.e. on topic with the instance. We asked the same questions about the answer, in addition to: 3) whether the answer is factually correct or likely true; and 4) whether the answer adds helpful information to solve the instance.
Unsupervised Commonsense Question Answering with Self-Talk
The annotation task was carried out in Amazon Mechanical Turk. To ensure the quality of annotations, we required that the workers be located in the US, UK, or Canada, and have a 99% approval rate for at least 5,000 prior tasks. We aggregated annotation from 3 workers using majority vote. The annotations yielded moderate levels of agreement, with Fleiss Kappa κ = 0.43 (Landis and Koch, 1977). Among the different categories of annotations we measured pairwise accuracy, which ranged from 60.41% (the answer is factually correct) to 92.26% (the question is completely not understandable). For the sake of brevity, we focus on the analysis of the answers to the clarification questions. Figure 6 shows the human evaluation results for each combination of task and knowledge source. The top part of the figure shows that across tasks and resources, most clarifications are grammatical or at least understandable, with the exception of XLNet. The bottom part shows the percentage of clarifications considered relevant, correct, and helpful. 6 Most clarifications were considered relevant to the context, around half of them were considered factually correct, and some 20-40% were considered helpful. Considering that these are all clarifications that indeed helped the model, this is an interesting though not completely unexpected finding: the model utilizes knowledge that humans wouldn't consider as helpful, and likely also vice versa.
Unsupervised Commonsense Question Answering with Self-Talk
Breaking down by knowledge source, we observe that when the datasets were created using a knowledge source (ConceptNet for Com-monSenseQA, and Social IQa uses ATOMIC, on which COMET is trained), clarifications from that resource are considered correct. We also note that somewhat surprisingly, relatively few ConceptNet clarifications were considered correct, despite limiting the relation paths up to 2 edges. The external knowledge is typically incorporated into the neural model by learning a vector representation of the symbolic knowledge (e.g. subgraphs from ConceptNet), and attending to it via attention mechanism when representing the inputs (Bauer et al., 2018;Paul and Frank, 2019;Lin et al., 2019). Alternative approaches include using the knowledge to score answer candidates and prune implausible ones (Lin et al., 2017;Tandon et al., 2018), and training in a multi-task setup via auxiliary tasks pertaining to knowledge (Xia et al., 2019).
Unsupervised Commonsense Question Answering with Self-Talk
Pre-trained LMs such as GPT2 (Radford et al., 2019) and BERT (Devlin et al., 2019) capture various types of world knowledge. Petroni et al. (2019) showed that such LMs can be used in a KB completion task over ConceptNet and Wikidata (Vrandečić and Krötzsch, 2014) by converting KB relations into natural language templates and querying the LM for the missing part in the triplet (concept 1 , relation, concept 2 ). For instance, querying BERT for suitable substitutes to the mask in "Dante was born in [MASK]" assigns the highest probability to Rome. Davison et al. (2019) similarly showed that BERT assigns higher scores to natural language fragments of true rather than fictitious ConceptNet triplets, and semi-automated the template creation by using GPT2 to score hand-crafted templates.
Unsupervised Commonsense Question Answering with Self-Talk
While both works have shown somewhat promising results, other work showed that knowledge extracted from LMs is expectantly not always accurate. Specifically, Kassner and Schütze (2019) showed that negated facts are also considered likely by the LM, while Logan et al. (2019) pointed out that LMs may over-generalize and produce incorrect facts such as "Barack Obama's wife is Hillary".
Unsupervised Commonsense Question Answering with Self-Talk
There are numerous research directions investigating automatic question generation (Vanderwende, 2008). Motivations vary from data augmentation to QA tasks (Du et al., 2017;Dhingra et al., 2018;Du and Cardie, 2018;Sachan and Xing, 2018) through conversational machine reading (Saeidi et al., 2018;Pan et al., 2019), simplifying questions to make them more easily answerable (Buck et al., 2018;Talmor and Berant, 2018;Perez et al., 2020), to using questions as means for other purposes such as sentence representation and summarization (Guo et al., 2018;Potash and Suleman, 2019).
Unsupervised Commonsense Question Answering with Self-Talk
In particular, our work is pertinent to previous work in producing clarifications questions and explanations. Rao and Daumé III (2019) worked on question from forums (e.g. Stack Exchange). They proposed a model that generates clarification questions and corresponding answers for a given question, using the question's comments (clarification questions and answers) as supervision. Questionanswer pairs were scored based on how much relevant information they add to the context. Shen et al. (2019) developed an active learning framework for image captioning that learns to detect uncertainty about generated words and ask natural language questions to reduce its uncertainty. A visual question answering (VQA) model provides an answer which is then used to change the caption. The framework is trained with reinforcement learning, but the gold standard captions are used during a warmup steps and the VQA model is supervised.
Unsupervised Commonsense Question Answering with Self-Talk
Klein and Nabi (2019) proposed a joint question generation and question answering framework. They fine-tuned GPT2 on a question answering dataset to generate a question and an answer span for a given passage, and trained BERT to answer the generated question given the passage. Finally, Rajani et al. (2019) proposed a model for Com-monSenseQA that generates explanations for its predictions. They collected human explanations and used them to fine-tune LMs to automatically generate explanations. These explanations were then added as additional inputs. The shortcoming of this approach is that it requires collecting specific human explanations for each new dataset.
Unsupervised Commonsense Question Answering with Self-Talk
We presented an unsupervised framework for multiple choice commonsense tasks that generates and integrates background knowledge from pre-trained LMs. On most tasks, it performs substantially better than the baseline and similarly to a model that had access to external knowledge resources.
Unsupervised Commonsense Question Answering with Self-Talk
By design, our model makes a single additional reasoning step explicit. A preliminary experiment in which we incorporated clarification pairs to facilitate two hops got mixed results. An interesting future direction is to generate each clarification in response to the previous ones, in a dialogue setup (Saeidi et al., 2018). Another challenge is the "needle in a haystack" problem of the clarifications, and one way to address it is to develop a model that is capable of "introspection", specifically knowing what it doesn't know. A more structured knowledge generation might also make the combination of various knowledge sources more successful.
Unsupervised Commonsense Question Answering with Self-Talk
Filling in knowledge gaps and making implicit intermediate reasoning steps explicit is imperative going forward. We hope that our framework will facilitate future research in this area.
Unsupervised Commonsense Question Answering with Self-Talk
Our code and data is available at github.com/vered1986/self talk.  The question and answer prefixes used for each task. " " in the answer prefix is replaced with the generated question (excluding the question mark), e.g. "What is the definition of a cat?" yields the following answer prefix: "The definition of a cat is". The question and answer templates in Social IQa correspond to COMET dimensions. "[NAME]" is replaced with the syntactic subject of the sentence (see for example the first row in Table 6). Before that, what did the player need?
Unsupervised Commonsense Question Answering with Self-Talk
Before, the player needed to go to the ball field.
Unsupervised Commonsense Question Answering with Self-Talk
The boy skipped dinner. What was the cause for it?
Unsupervised Commonsense Question Answering with Self-Talk
He ate a big lunch. What is the relationship between 'dinner' and 'lunch'?
Unsupervised Commonsense Question Answering with Self-Talk
Lunch is the opposite of dinner.
Unsupervised Commonsense Question Answering with Self-Talk
Working on the elaborate task was taxing, it require extreme what?
Unsupervised Commonsense Question Answering with Self-Talk
concentration What is the relationship between 'working' and 'concentration'? In order for working to happen, concentration needs to happen.
Unsupervised Commonsense Question Answering with Self-Talk
What do professors primarily do?
Unsupervised Commonsense Question Answering with Self-Talk
What is the main function of a professor's teaching career,? The main function of a professor's teaching career, is to teach students how they can improve their knowledge.
Unsupervised Commonsense Question Answering with Self-Talk
GPT2-XL Growing up on a farm near St. Paul, L. Mark Bailey didn't dream of becoming a judge. How many years did it take for Mark to become a judge?
Unsupervised Commonsense Question Answering with Self-Talk
What happened first?
Unsupervised Commonsense Question Answering with Self-Talk
The first thing that happened was he went to law school and became an attorney.
Unsupervised Commonsense Question Answering with Self-Talk
After several years of private practice from 1982-90, he became the judge of Decatur County Court for a year. When did he get a law degree?
Unsupervised Commonsense Question Answering with Self-Talk
What happened first?
Unsupervised Commonsense Question Answering with Self-Talk
The first thing that happened was that his wife, linda, had passed away.
Unsupervised Commonsense Question Answering with Self-Talk
Bailey saved money because they wanted to buy something good. How would you describe Bailey?
Unsupervised Commonsense Question Answering with Self-Talk
Bailey is seen as good with money.
Unsupervised Commonsense Question Answering with Self-Talk
How would you describe Bailey?
Unsupervised Commonsense Question Answering with Self-Talk
Bailey is seen as hardworking.
Unsupervised Commonsense Question Answering with Self-Talk
Jan had been on vacation in Hawaii for a week. Last night, she finally returned home. What will Jan want to do next?
Unsupervised Commonsense Question Answering with Self-Talk
As a result, Jan wanted to return to work. What is the relationship between 'vacation' and 'work'?
Unsupervised Commonsense Question Answering with Self-Talk
Vacation is the opposite of work.
Unsupervised Commonsense Question Answering with Self-Talk
Robin proposed another solution after the first one was not successful. What does Robin need to do before this?
Unsupervised Commonsense Question Answering with Self-Talk
Before, Robin needed to come up with other solutions.
Unsupervised Commonsense Question Answering with Self-Talk
What does Robin need to do before this?
Unsupervised Commonsense Question Answering with Self-Talk
Before doing that, Robin first had to do some research on how he could make it.
Unsupervised Commonsense Question Answering with Self-Talk
Skylar always arrives 15 minutes early for class, takes extensive notes, and studies hard before tests. How would you describe Skylar?
Unsupervised Commonsense Question Answering with Self-Talk
Skylar is seen as a dedicated student.
Unsupervised Commonsense Question Answering with Self-Talk
How would you describe Skylar?
Unsupervised Commonsense Question Answering with Self-Talk
Skylar is a very good student who does well in the classroom.
Unsupervised Commonsense Question Answering with Self-Talk
GPT2-M Carson was at a friends house and they decided to play video games. What does Carson need to do before this?
Unsupervised Commonsense Question Answering with Self-Talk
Before, Carson needed to know about their friends house.
Unsupervised Commonsense Question Answering with Self-Talk
What does Carson need to do before this?
Unsupervised Commonsense Question Answering with Self-Talk
Before doing that, Carson first had to get his mother's permission before playing the game.
Unsupervised Commonsense Question Answering with Self-Talk
GPT2-L Taylor gave a raise and a promotion to Kendall. How would you describe Taylor?
Unsupervised Commonsense Question Answering with Self-Talk
Taylor is seen as as someone who appreciates what Kendall does.
Unsupervised Commonsense Question Answering with Self-Talk
How would you describe Taylor?
Unsupervised Commonsense Question Answering with Self-Talk
Taylor is a good guy, but he's also an idiot.
Unsupervised Commonsense Question Answering with Self-Talk
To cut circles out of dough without a cookie cutter.
Unsupervised Commonsense Question Answering with Self-Talk
Use a glass by turning it upside down and using the rim.
Unsupervised Commonsense Question Answering with Self-Talk
What does you want as a result?
Unsupervised Commonsense Question Answering with Self-Talk
As a result, you wants to get the dough out of the oven.
Unsupervised Commonsense Question Answering with Self-Talk
How can I slice up a banana?
Unsupervised Commonsense Question Answering with Self-Talk
Use a knife, on a cutting board, and slice the banana into quarter sized slices.
Unsupervised Commonsense Question Answering with Self-Talk
What is the relationship between 'slice' and 'knife'?
Unsupervised Commonsense Question Answering with Self-Talk
Slice is a specific way of doing cut. Knife is used for cut.
Unsupervised Commonsense Question Answering with Self-Talk
GPT2-XL What pan to use for frying eggs?
Unsupervised Commonsense Question Answering with Self-Talk
Nonstick pans are perfect for frying eggs.
Unsupervised Commonsense Question Answering with Self-Talk
What is the definition of a 'pan'?
Unsupervised Commonsense Question Answering with Self-Talk
The definition of a 'pan' is : A metal or ceramic vessel used in cooking.
Unsupervised Commonsense Question Answering with Self-Talk
XLNet-base Sharpen a pencil.
Unsupervised Commonsense Question Answering with Self-Talk
Use a knife. What are the properties of a you sharpen a pencil,? The properties of a you sharpen a pencil, are that it will not break or be dulled.
Unsupervised Commonsense Question Answering with Self-Talk
Joel complained to Ian about the condition of the house. preferred a messy space.
Unsupervised Commonsense Question Answering with Self-Talk
What does Joel want as a result?
Unsupervised Commonsense Question Answering with Self-Talk
As a result, Joel wants to get rid of the clutter.
Unsupervised Commonsense Question Answering with Self-Talk
The hotel had a terrible gym but a nice restaurant; the was something management attended to.
Unsupervised Commonsense Question Answering with Self-Talk
restaurant What is the definition of 'restaurant'?
Unsupervised Commonsense Question Answering with Self-Talk
'restaurant' is defined as an establishment that serves food, drinks and services.
Unsupervised Commonsense Question Answering with Self-Talk
GPT2-M Mark wanted to lose some weight by starting a diet and workout routine. He focused mainly on the because he was lazy.
Unsupervised Commonsense Question Answering with Self-Talk
diet What is the definition of 'workout routine'?
Unsupervised Commonsense Question Answering with Self-Talk
'workout routine' is defined as any kind of exercise that involves doing physical activity.
Unsupervised Commonsense Question Answering with Self-Talk
The parakeet flew to Hunter but flew past Samuel because had some bird seed in their hand.
Unsupervised Commonsense Question Answering with Self-Talk
What does it mean to be a 'bird'?
Unsupervised Commonsense Question Answering with Self-Talk
Be a 'bird' means the same thing as it does for us humans.
Unsupervised Commonsense Question Answering with Self-Talk
GPT Matt wanted to change either the plain bathroom or the colorful bedroom, but the was already colorful.
Unsupervised Commonsense Question Answering with Self-Talk
bedroom What is the definition of 'bedroom'?
Unsupervised Commonsense Question Answering with Self-Talk
'bedroom' is defined as a room with a bed and nightstand in it.
Unsupervised Commonsense Question Answering with Self-Talk
XLNet-base My employer offers a bonus of either a phone or a television, but unfortunately the is just way too large to be useful.
Unsupervised Commonsense Question Answering with Self-Talk
phone What is the definition of 'phone'?
Unsupervised Commonsense Question Answering with Self-Talk
'phone' is defined as device that connects a person directly into the world. Table 6: Example instances from each dataset and the clarifications generated for them in various resources. We only include clarifications that helped the model predict the correct answer.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
The ability to understand natural language and answer questions is one of the core focus in the field of natural language processing. To measure and study the different aspects of question answering, several datasets are developed, such as SQuaD (Rajpurkar et al., 2018), HotpotQA (Yang et al., 2018), and Natural Questions (Kwiatkowski et al., 2019) which require systems to perform extractive question answering. On the other hand datasets such as SocialIQA (Sap et al., 2019b), Common-senseQA (Talmor et al., 2018), Swag (Zellers et al., 2018) and Winogrande (Sakaguchi et al., 2019) require systems to choose the correct answer from a given set. These multiple-choice question answering datasets are very challenging, but recent large pre-trained language models such as BERT (Devlin et al., 2018), XLNET (Yang et al., 2019b) and RoBERTa (Liu et al., 2019b) have shown very strong performance on them. Moreover as shown in Winogrande (Sakaguchi et al., 2019), acquiring unbiased labels requires a "carefully designed crowdsourcing procedure", which adds to the cost of data annotation. This is also quantified in other natural language tasks such as Natural Language Inference ( Gururangan et al., 2018) and Argument Reasoning Comprehension (Niven and Kao, 2019), where such annotation artifacts lead to "Clever Hans Effect" in the models (Kaushik and Lipton, 2018;Poliak et al., 2018).
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
One way to resolve this is to design and create datasets in a clever way, such as in Winogrande (Sakaguchi et al., 2019), the other way is to ignore the data annotations and to build systems to perform unsupervised question answering (Teney and Hengel, 2016;Lewis et al., 2019). In this paper, we focus on building unsupervised zero-shot multiple-choice question answering systems.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
The task of unsupervised question answering is very challenging. Recent work (Fabbri et al., 2020;Lewis et al., 2019) try to generate a synthetic dataset using a text corpus such as Wikipedia, to solve extractive QA. Other work (Bosselut and Choi, 2019;Shwartz et al., 2020) uses large pretrained generative language models such as GPT-2 (Radford et al., 2019) to generate knowledge, questions, and answers and compare against the given choices.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
In this work, we utilize the information present in Knowledge Graphs such as ATOMIC (Sap et al., 2019a) and ConceptNET (Liu and Singh, 2004) and define a new task of Knowledge Triplet Learning. Knowledge Triplet Learning is similar to Knowl-edge Representation Learning but not limited to it. Knowledge Representation Learning (Lin et al., 2018) learns the low-dimensional projected and distributed representations of entities and relations defined in a knowledge graph. As shown in Figure 1, we define a triplet (h, r, t), and given any two we try to recover the third. This forces the system to learn the all possible relations between the three inputs. We map the question answering task to Knowledge Triplet Learning, by mapping the context, question and answer to (h, r, t) respectively. We define two different ways to perform self-supervised Knowledge Triplet Learning. This task can be designed as a representation generation task or a language modeling task. We compare both the strategies in this work. We show how to use models trained on this task to perform zero-shot question answering without any additional knowledge or additional supervision. We also show how models pre-trained on this task perform considerably well compared to strong pre-trained language models on few-shot learning. We evaluate our approach on the SocialIQA dataset.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
The contributions of this paper are summarized as follows:
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
• We define the Knowledge Triplet Learning over Knowledge Graph and show how to use it for zero-shot question answering. • We compare two strategies for the above task.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
• We achieve state-of-the-art results for zeroshot and propose a strong baseline for the fewshot question answering task.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
We define the task of Knowledge Triplet Learning (KTL) in this section. We define G = (V, E) as a Knowledge Graph, where V is the set of vertices, E is the set of edges. V consists of entities which can be phrases or named-entities depending on the given input Knowledge Graph. Let S be a set of fact triples, S ⊆ V ×E×V with the format (h, r, t), where h and t belong to set of vertices V and r belongs to set of edges. The h and t indicates the head and tail entities, whereas r indicates the relation between these entities. For example, from the ATOMIC knowledge graph, (PersonX puts PersonX's trust in PersonY, How is PersonX seen as?, faithful) is one such triple. Here the head is PersonX puts PersonX's trust in PersonY, relation is How is PersonX seen as? and the tail is faithful. Do note V does not contain homogenous entities, i.e, both faithful and PersonX puts PersonX's trust in PersonY belong to V .
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
We define the task of KTL as follows: Given input a triple (h, r, t), we learn the following three functions.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
That is, each function learns to generate one component of the triple given the other two. The intuition behind learning these three functions is as follows. Let us take the above example: (PersonX puts PersonX's trust in PersonY, How is PersonX seen as?, faithful). The first function f t (h, r) learns to generate the answer t given the context and the question. The second function f h (r, t) learns to generate one context where the question and the answer may be valid. The final function f r (h, t) is a Jeopardy-style generating the question which connects the context and the answer.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
In Multiple-choice QA, given the context, two answer choices may be true for two different questions. Similarly, given the question, two answer choices may be true for two different contexts. For example, given the context: PersonX puts Per-sonX's trust in PersonY, the answers PersonX is considered trustworthy by others and PersonX is polite are true for two different questions How does this affect others? and How is PersonX seen as?. Learning these three functions enables us to score these relations between the context, question, and answers.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
After learning this function in a self-supervised way, we can use them to perform question answering. Given a triple (h, r, t), we define the following scoring function:
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
where h is the context, r is the question and t is one of the answer options. D is a distance function which measures the distance between the generated output and the ground-truth. The distance function varies depending on the instantiation of the framework, which we will study in the following sections. The final answer is selected as:
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
Since the scores are the distance from the groundtruth we select the answer which has the minimum score.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
In the following sections, we define the different ways we can implement this framework.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
In this implementation, we use Knowledge representation learning to learn equation (1). In contrast to Knowledge representation learning, where systems try to learn a score function f r (h, t), i.e, is the fact triple (h, r, t) true or false; in this work we learn to generate the inputs vector representations, i.e, f r (h, t) ⇒ r. We can view equation 1 as generator functions, which given the two input learns to generate a vector representation of the third. As our triples (h, r, t) can have a many to many relations between each pair, we first project the two inputs from input encoding space to a different space similar to the work of TransD (Ji et al., 2015). We use a Transformer encoder Enc to encode our triples to the encoding space. We learn two projection functions, M i1 and M i2 to project the two inputs, and a third projection function M o to project the entity to be generated. We combine the two projected inputs using a function C. These functions can be implemented using feedforward networks.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
where I i is the input,Ô is the generated output vector and O p is the projected vector. M and C functions are learned using fully connected networks. In our implementation, we use RoBERTa as the Enc transformer, with the output representation of the [cls] token as the phrase representation. We train this model using two types of loss functions, L2Loss where we try to minimize the L2 norm between the generated and the projected ground-truth, and Noise Contrastive Estimation (Gutmann and Hyvärinen, 2010) where along with the ground-truth we have k noise-samples. These noise samples are selected from other (h, r, t) triples such that the target output is not another true fact triple, i.e, (h, r, t noise ) is false. The NCELoss is defined as:
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
where N k are the projected noise samples, sim is the similarity function which can be the L2 norm or Cosine similarity,Ô is the generated output vector and O p is the projected vector.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
The D distance function (2) for such a model is defined by the distance function used in the loss function. For L2Loss, it is the L2 norm, and in the case of NCELoss, we use 1 − sim function.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
In Span Masked Language Modeling, we model the equation 1 as a masked language modeling task. We tokenize and concatenate the triple (h, r, t) with a separator token between them, i. . We feed this tokens to a Transformer encoder Enc, and use a feed forward network to unmask the sequence of tokens. Similarly we mask h to learn f h and t to learn f t .
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
We train the same Transformer encoder to perform all the three functions. We use the crossentropy loss to train the model:
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
where P M LM is the masked language modeling probability of the token t i , given the unmasked tokens h and r and other masked tokens in t. Do note we do not do progressive unmasking, i.e, all the masked tokens are jointly predicted. The D distance function (2) for this model is same as the loss function defined above.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
To study our framework we evaluate it on So-cialIQA. This dataset is about reasoning over social interactions and the implications of social events. Each instance in this dataset contains a context C which is a social situation, a question Q about this situation, and three answer options. There are several question types that are derived from the different ATOMIC inference dimensions, such as Intent, Effect, Attributes, etc. There are 33,410 training samples and 1954 validation samples, with a withheld test set.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
Though this dataset is derived from the ATOMIC knowledge graph, the fact triples present in the graph are considerably different than the context, question and answers present in the dataset as they are crowdsourced. The average length of the event description in ATOMIC is 10, max length is 18. Whereas in SocialIQA the average length is 36 and max is 124. This shows the varied type of questions present in SocialIQA, and possess a much more challenge in unsupervised learning.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
We compare our models to three strong baselines. The first one is a pre-trained language model, GPT-2 (all sizes) which are scored using language modeling cross-entropy loss. We concatenate the context and question and find the cross-entropy loss for each of the answer choices and choose the answer which has the minimum loss. The second baseline is another pre-trained language model, RoBERTa-large. We follow the same Span Masked Language Model (SMLM) scoring using the pretrained RoBERTa model. For the third baseline, we finetune the RoBERTa-large model using the original Masked Language Modeling task over our concatenated fact triples (h, r, t).
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
We train the Knowledge Representation Learning (KRL) model using both L2Loss and NCELoss. For NCELoss we also train it with both L2 norm and Cosine similarity. Both the KRL model and SMLM model uses RoBERTa-large as the Transformer encoder. We train the model with the following hyper-parameters: batch sizes 16,32; learning rate in range: [1e-5,5e-5]; warm-up steps in range [0,0.1]. We use the transformers package (Wolf et al., 2019). From the ATOMIC knowledge graph, we generate 595595 unique triplets. All these triplets are positive facts. We learn using these triplets. For NCE, we choose k equal to 10, i.e, 10 negative samples. We perform 3 hyperparameter trials for each model, and train models with 3 different seeds [0,21,42].  Table 1: Accuracy comparison with our baseline models on the SocialIQA dataset. We compare the models on the Zero-shot task. We compare them on both Train-Val split (35k) and the Validation split (2k), to enable measuring better statistical significance. Table 1 shows our evaluation and baseline comparisons for the zero-shot task. We can observe our KTL trained models perform significantly well compared to the baselines. When comparing the different KRL models, the NCELoss with Cosine similarity performs the best. This might be due to the additional supervision provided by the negative samples as the L2Loss model only tries to minimize the two projections. When comparing different KTL instantiations we can see the SMLM model performs the best overall but has a slightly higher deviation. We are analyzing our model to understand this phenomenon better. Our KRL model does perform equal to the current state-of-the-art model, Self-Talk (Shwartz et al., 2020) which uses two GPT type models. Our models have half the parameters compared to Self-Talk. We also compare our model on the entire Train-Val set of 35,364 questions in the zero-shot setting to better gauge the statistical significance of the different model accuracies.    2 compares the different pre-trained Transformer encoders in the few-shot question answering task. We randomly sample three sets of 2,400 samples as training data from the training set. We train our models on these three sets and measure the validation set accuracy. We can see the Transformer encoder trained on KTL perform significantly better than the baseline models in this setting. This shows that encoders trained on KTL are able to learn with a few samples. We plan to continue analyzing our models and evaluate the KTL framework on different datasets. 6 Related Work
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
Recent work on unsupervised question answering approach the problem in two ways, a domain adaption or transfer learning problem (Chung et al., 2018), or a data augmentation problem Dhingra et al., 2018;Wang et al., 2018;. The work of (Lewis et al., 2019;Fabbri et al., 2020;Puri et al., 2020) use style transfer or template-based question, context and answer triple generation, and learn using these to perform unsupervised extractive question answering. There is also another approach of learning generative models, generating the answer given a question or clarifying explanations and/or questions, such as GPT-2 (Radford et al., 2019) to perform unsupervised question answering (Shwartz et al., 2020;Bosselut and Choi, 2019;Bosselut et al., 2019). In contrast, our work focuses on learning from knowledge graphs and generate vector representations or sequences of tokens not restricted to the answer but including the context and the question using the masked language modeling objective.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
There are several approaches to add external knowledge into models to improve question answering. Broadly they can be classified into two, learning from unstructured knowledge and structured knowledge. In learning from unstructured knowledge, recent large pre-trained language models (Peters et al., 2018;Radford et al., 2019;Devlin et al., 2018;Liu et al., 2019b;Clark et al., 2020;Lan et al., 2019;Joshi et al., 2020;Bosselut et al., 2019) learn general-purpose text encoders from a huge text corpus. On the other hand, learning from structured knowledge includes learning from structured knowledge bases (Yang and Mitchell, 2017;Bauer et al., 2018;Mihaylov and Frank, 2018;Wang and Jiang, 2019; by learning knowledge enriched word embeddings. Using structured knowledge to refine pre-trained contextualized representations learned from unstructured knowledge is another approach (Peters et al., 2019;Yang et al., 2019a;Zhang et al., 2019;Liu et al., 2019a). Another approach of using external knowledge includes retrieval of knowledge sentences from a text corpora (Mitra et al., 2019;Banerjee et al., 2019;Baral et al., 2020;Das et al., 2019;Chen et al., 2017;Banerjee and Baral, 2020;Banerjee, 2019) or knowledge triples from knowledge bases (Min et al., 2019;Wang et al., 2020) that are useful to answer a specific question. In our work, we use knowledge graphs to learn a self-supervised generative task to be able to perform zero-shot multiple-choice QA.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
Over the years there are several methods discovered to perform the task of knowledge representation learning, i.e., embedding entities and relations in knowledge graphs to low-dimensional continuous vector space. We mention few of them here, such as TransE (Bordes et al., 2013) which views relations as a translation vector between head and tail entities, TransH (Wang et al., 2014) which overcomes TransE's inability to model complex relations, and TransD (Ji et al., 2015) which aims to reduce the parameters by proposing two different mapping matrices for head and tail entities. For much detailed reading, we refer to this survey by Lin et al.. KRL has been used in various ways to generate natural answers (Yin et al., 2016;He et al., 2017) and generate factoid questions (Serban et al., 2016). In our work, we modify TransD and adapt it to our KTL framework to perform zero-shot QA.
Self-supervised Knowledge Triplet Learning for Zero-shot Question Answering
In this work, we propose a new framework of Knowledge Triplet Learning over Knowledge Graphs. We show learning all three possible functions, f r ,f h , and f t helps the model to perform zero-shot multiple-choice question answering. We learn from the ATOMIC knowledge graph and evaluate our framework on the SocialIQA dataset. Our framework achieves state-of-the-art in the zero-shot question answering task and sets a strong baseline in the few-shot question answering task. 
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Understanding narratives requires reasoning about all the implicit, but trivially inferable, details of a situation based only on what is explicitly stated in text. A statement as simple as "they went to the club" instantly invokes a bank of commonsense expectations: they had to get dressed, they were going dancing, they likely had drinks, and so forth. These reasoning capabilities are missing in most existing neural language understanding models that learn task-specific representations without acquiring rich background knowledge about the social and physical world.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
In response, recent work has investigated augmenting deep learning models with retrieval mechanisms over large-scale commonsense knowledge graphs (Mihaylov and Frank 2018;Bauer, Wang, and Bansal 2018;Paul and Frank 2019). However, these approaches assume an entity linking step between the written text and knowledge graph. By canonicalizing entities, they discard key context surrounding the input, and  often retrieve semantically irrelevant knowledge (e.g., a club being a blunt weapon is irrelevant to the earlier situation).
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
In this paper, we propose to generate new knowledge that is contextually relevant instead of retrieving existing knowledge as is. Bosselut et al. (2019) recently introduced Commonsense Transformers (COMET), a new framework for training neural representations of knowledge graphs. This new class of neural knowledge model provides a powerful representational tool for connecting commonsense knowledge to downstream task models. Because COMET represents knowledge graphs neurally, it can generate commonsense inferences for any entity that can be encoded by the neural model (i.e., described with language). With no need to canonicalize context entities to link to a static knowledge graph, the knowledge model can be queried directly with complex compositional structures, and even full narrative contexts. This flexibility has led them to be used out-of-the-box in a variety of settings requiring contextual knowledge, such as sarcastic comment generation (Chakrabarty et al. 2020), therapy chatbots (Kearns et al. 2020), and story plot generation (Ammanabrolu et al. 2020).
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
In this work, we use COMET to dynamically construct context-relevant knowledge graphs that can be reasoned over for commonsense question answering. Given a raw context, COMET generates commonsense inferences that provide world knowledge about the situation depicted in the context. These inferences can be used as additional context to score answer candidates or to generate additional inferences. By generating new inferences and connecting them to the raw context and answers, COMET dynamically constructs a symbolic graph of commonsense knowledge. The raw context is the root node, answer choices are leaf nodes and generated commonsense inferences provide intermediate nodes between them, instantiating different reasoning paths between the context and answers. Using COMET generated scores as factors weighting these paths, we propose new inference algorithms to reason over the generated graph and identify the most likely answers to questions about the situation.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
We evaluate our approach in a zero-shot setting on the SocialIQa (Sap et al. 2019b) benchmark, a question answering dataset for evaluating social commonsense, and the Sto-ryCS benchmark (Rashkin et al. 2018), a story understanding dataset. Empirical results show that our neuro-symbolic approach, COMET -DynaGen, outperforms purely neural large-scale pretrained language models (Radford et al. 2018(Radford et al. , 2019 and knowledge models that evaluate QA examples directly without dynamically generating an intermediate symbolic commonsense knowledge graph (i.e., reasoning with COMET with no inference hops).
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Our approach uses a knowledge model, COMET , to dynamically construct a context-relevant commonsense knowledge graph about a presented situation. COMET is trained using transfer learning from large-scale pretrained language models (Radford et al. 2018) to knowledge graphs. When trained on the Atomic knowledge graph (Sap et al. 2019a), it learns to generate social commonsense inferences of situations depicted in text. Importantly, unlike static knowledge graphs (e.g., ConceptNet; Speer, Chin, and Havasi 2017), which require canonicalizing input entities to link to the graph, COMET represents knowledge neurally, allowing it to generate commonsense for arbitrary input forms. In Figure 1, for example, the context "Kai knew things were getting out of control and managed to keep his temper in check" is unlikely to be found in any existing knowledge graph. It describes a very specific situation. However, COMET can parse this full context and generate commonsense knowledge about Kai's reactions and motivations, such as "Kai stays calm" or "Kai wants to avoid trouble," as downstream inferences. We exploit this generalization property of knowledge models to dynamically construct knowledge graphs for presented situations that can be reasoned over to answer commonsense questions about them.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Notation. Formally, we assume a dataset of examples, each with an associated context c describing a situation, a question q asked about that situation, and a set of n possible answers A = {a 0 , ..., a n−1 } to that question. Each answer is composed of multiple tokens Y a = {y 1 , ..., y |a| }.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Generating Commonsense Inferences. We generate commonsense inferences for a situational context c by concatenating the context with relation types from the Atomic knowledge graph and using COMET to produce candidates G. Each candidate g ∈ G is associated with a score φ g that approximates the model's confidence in the inference:
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
where x t are the tokens of g, |g| is the token length of g, r is an arbitrary commonsense relation type for which COMET can generate inferences, and:
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
P (x t |x <t , c, r) = COMET(c, r, x <t ) (2) where the tokens of c and r are concatenated with the tokens x <t to be input to COMET. Any generation g ∈ G conditioned on c can be seen as a 1-hop commonsense inference of c.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Using a Markov assumption, we can generalize this approach by conditioning on generated commonsense inferences to generate G , a set of -hop inferences from c:
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
where φ g is a generation score for any g ∈ G , g −1 is an arbitrary inference from G −1 , the set of inferences of the previous hop, and φ −1 g is the generation score of that seed inference. Using this approach, we can use COMET to construct a graph where commonsense inferences g are nodes.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
For an arbitrary node g , its parent is the node from the previous level G −1 that COMET conditions on to generate g . The children of g are nodes generated when COMET conditions on g to generate new commonsense inferences. We set g 0 = c because the context is the root node of the graph, and φ 0 g = 0 because the original context c is deterministic. Answers as Leaf Nodes. The final step in constructing the knowledge graph is to connect the answer choices a ∈ A to the generated commonsense inferences. We initialize a node in the graph for each answer choice a and connect it as a child node to each commonsense inference in the graph: g ∈ G for ∈ [0, L) where L is the number of levels in the final graph. In Figure 2b, we see that the answer choices A = {relieved, scared, anxious} are connected to the root node and each generated commonsense inference in the L = 2 level graph.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Being designed as a conditional language model, COMET can also be used to score candidate commonsense inferences. We use this property to score answer (a) COMET receives the context c and generates new commonsense inferences to construct a local graph of knowledge about the situation (Section ).
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
generation factor (Eq. 1, 3) layer aggregate (Eq. 7,8) answer factor (Eq. 4, 6)
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
(b) Our inference algorithms reason over the graph by aggregating commonsense paths to answer questions about the situation (Section ). candidates a ∈ A conditioned on the generated commonsense inferences g ∈ G that are connected to them. The scores from COMET are used to initialize factor nodes between each generated commonsense inference (at all levels of the graph) and each answer choice. Using these scores, and scores between commonsense inferences (Eqs. 1, 3), as a set of factors, our generated knowledge graph implicitly encodes a factor graph that can be reasoned over to evaluate each answer candidate.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
COMET is originally trained to maximize the conditional loglikelihood of the tokens of a target entity e 2 from a knowledge graph tuple (e 1 , r, e 2 ). As a result, the knowledge model can measure the log-likelihood of a candidate entity e 2 given a source entity e 1 and relation r. For a given example, we treat each answer candidate a as an e 2 candidate for COMET, map the parent nodes of a (e.g., g nodes) to be equivalent to e 1 , and set the question q as r, allowing COMET to evaluate each answer candidate according to its implicit knowledge representations. For each answer a ∈ A, we define a factor based on each token's conditional log-likelihood as computed by COMET:
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
where y s corresponds to the token in a at time step s, y <s is all the tokens preceding y s in a, and |a| is the total number of tokens making up a. In this way, for any QA example, we define a set of factor nodes φ ga connecting the answer candidates a ∈ A to the commonsense inferences g ∈ G generated by COMET about the situational context c.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Overcoming Answer Priors. Because certain answer candidates have a high probability of occurring for certain questions regardless of the context (e.g., happy is a common answer for questions about emotional reactions), we redefine φ ga (Eq. 4) in terms of the point-wise mutual information between the commonsense path g and answer a:
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
logP (y s |y <s , g, q)
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
− log P (y s |y <s , q)
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
where log P (y s |y <s , q) is the log-likelihood of each token in the answer given only the question and previous answer tokens. We describe our approximation of this distribution in Appendix B.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Each φ g scores a unique reasoning path at a particular depth in the graph. The composition γ g φ g + γ ga φ ga can then be seen as scoring a path to a particular answer. To find the most likely answer, we marginalize over all paths to the answers at layer :
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
where φ g (Eq. 3) and φ ga (Eq. 5) are the path and answer score, respectively, for generation g ∈ G . γ g and γ ga are hyperparameters balancing the contribution of both scores. Because the path and answer scores are log-probabilities, we set f as the LogSumExp, yielding Eq. 6 as a variable elimination over g ∈ G . We also define an extremum estimator over the distribution of generated inferences G :
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
At a high level, φ amax can be interpreted as approximating the likelihood of answer a given a singular reasoning path: {c → g 1 → · · · → g → a}, rather than by computing an aggregation of all paths in the graph to the answer (Eq. 6). Once the answer scores at different levels in the graph are computed, {φ a } L 0 , the final score for each answer can be evaluated by averaging over the graph levels ∈ [0, L):
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
whereâ is the selected best answer by the approach, L is the number of generation hops made by the COMET model (i.e., the number of levels in the graph), φ a is the score that is propagated from each hop of the constructed knowledge graph, and β is hyperparameter scaling the contribution of each hop score. We note that φ 0 a is the result from evaluating the answer candidates directly against the original context c, and that φ a is replaced by φ amax if the extremum estimator (Eq. 7) is used instead of variable elimination (Eq. 6).
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
We evaluate our approach in a zero-shot experimental setting. It is a well-studied phenomenon that neural methods trained on crowdsourced data often learn to shortcut reasoning to arrive at a correct answer (Gururangan et al. 2018;Li and Gauthier 2017). We use a zero-shot setting to simulate the model having to reason about situations it has never encountered before, forcing it to construct reasoning graphs from explicit knowledge it can generate (e.g., knowledge learned by COMET), and precluding it from learning dataset-specific artifacts. As such, we do not use training data to update model parameters. Furthermore, any result presented on the test set does not have hyperparameters tuned on the development set.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
We evaluate our method on two datasets: SocialIQa (Sap et al. 2019b) and StoryCS (Rashkin et al. 2018).
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
SocialIQa. The SocialIQa dataset evaluates a model's ability to understand the social dynamics underlying situations described in short text snippets. Each example in the dataset consists of a context, a question about that context, and three multiple choice answers. An example from the dataset is shown in Figure 2. We outline pre-processing steps for the data in Appendix A.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
StoryCS. The StoryCS dataset consists of short 5-sentence stories with annotated motivations and emotional responses whose labels are drawn from classical theories of psychology (e.g., Plutchik 1980). We map the emotion classification task to a QA task by posing an individual question for each emotion label (disgust, surprise, fear, anger, trust, anticipation, sadness, joy) that must be predicted for each example. We outline this procedure in Appendix B.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Hyperparameters. We use most of the same hyperparameters to train the COMET model on the Atomic knowledge graph as in Bosselut et al. (2019). However, we use GPT2-345M (Radford et al. 2019) as the pretrained language model that seeds COMET and freeze the position embeddings so we can generalize to longer contexts. We note that the So-cialIQa dataset is partially derived from Atomic knowledge base tuples. However, we do not allow Atomic tuples used to seed SocialIQa evaluation examples to be used as training examples for COMET. We provide more details of this splitting in Appendix A. The number of levels in the graph L is set to 2. As we operate in the zero-shot setting, we do not tune hyperparameters. For the SocialIQa dataset, we set γ g = γ ga = 1.0 and β = 1.0 ∀ . For StoryCS, we do the same except that γ g = 0. Unless stated otherwise, we use argmax decoding to generate inferences from COMET, and use variable elimination over the graph to select answers.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
To predict an answer on the SocialIQa dataset, we use Equation 9. Prediction for StoryCS is less straightforward, as the task is originally binary multi-label classification. To make a prediction, we treat φ a (Eq. 8) for each label j independently and select an answer based on whether φ a,j is above a label-specific threshold, κ j . To avoid violating the zero-shot setting (i.e., tuning thresholds on the development set), we select the threshold using the score at the percentile of the positive label distribution (e.g., if the joy emotion is present for 20% of examples, we set the threshold at the score of the 20th percentile of the CDF). Thresholds are reported in Appendix Table 10 for each label.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Baselines. As baselines in the SocialIQa study, we use large-scale pretrained language models: GPT (Radford et al. 2018), GPT2-117M, GPT2-345M, and GPT2-762M (Radford et al. 2019). To adapt these language models optimally to the QA task, question-answer pairs are automatically converted to a templated form, a process we outline in Appendix B. We also report the results of a model, COMET -Direct, that only uses φ 0 a to select answers (i.e., answers are evaluated with respect to the context with no dynamic graph construction). Additionally, we compare against the Self-Talk model of Shwartz et al. (2020), which queries pretrained language models to generate additional details about a presented situ-
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Jesse drove Ash to the airport and dropped them off at the airport with ease. Taylor was close to winning the game. Taylor ran straight for home plate.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Taylor wants to celebrate a) try to get over that they did win b) celebrate the win c) wanted to score Taylor wants to be home a) try to get over that they did win What will Taylor want to do next? b) celebrate the win c) wanted to score Table 1: Example contexts, paths, and answers for the COMET -DynaGen model on SocialIQa. We bold the predicted answer and its most contributing path. We italicize the most likely answer for each path. Incorrect high-scoring answers for a path are marked with and correct answers are marked with . We only present a subset of the generated paths.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Dev Acc. Test Acc.  ation and appends these to the original context. Finally, we report the result of supervised BERT (Devlin et al. 2018) and RoBERTa (Liu et al. 2019) models, and random and human baselines from Sap et al. (2019b).
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Overall Performance. We report the main results of our So-cialIQa study in Table 2. First, our approach achieves an absolute improvement of ∼10.2% over the top performing language model baseline, GPT2-762M, showing the importance of using knowledge models to represent commonsense.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Additionally, our approach of dynamically constructing a knowledge graph on demand (COMET -DynaGen) performs better than using the knowledge model to directly evaluate answers (COMET -Direct) by ∼3.6%, highlighting the value in representing more complex reasoning paths. Finally, the improvement over Self-Talk depicts the benefit of using a structured graphical representation for reasoning compared to one that uses language models to generate additional situational context sentences for conditioning. We note, however, that the state-of-the-art performance of the supervised BERT and RoBERTa models is significantly higher, meaning there is room for improvement in developing comparable zero-shot approaches to QA. However, one point of interest is that the performance of training BERT with only 5000 training examples (rather than the full 30k) is close (54%) to the performance of COMET -DynaGen, indicating that knowledge models and joint neuro-symbolic solutions are already promising in low-data regimes.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Qualitative Analysis. In Table 1, we present top reasoning paths from the graphs generated by COMET -DynaGen. The strength of our approach can be seen in the first example, where the correct answer, drained, is more likely to be a feeling associated with wanting "to go home," a post-condition in the graph generated by COMET -DynaGen. In the original context, this condition is implicit. This benefit to leveraging graph reasoning is also seen in the second example, where Quinn's foolishness is linked to "[getting] hurt." We note that COMET -Direct, RoBERTa-large, and GPT2-345M all  Table 3: Development set accuracy for different graph construction techniques. The average number of nodes and edges in the constructed graphs is presented.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
answer this question incorrectly, reinforcing the importance of explicit reasoning graphs.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
In the final two examples, we present uninteresting or failure cases. In the first, the model predicts that Alex will experience joy after reasoning through the path that he will be "happy," which, while correct, is merely leveraging synonymy. In the final example, we show a case where the model selects an incorrect answer by reasoning through an incorrect path. By recognizing that "Taylor wants to celebrate" as a likely post-condition of the context, the model selects an answer that is incorrect. An interesting secondary failure mode in this example is in the second path through the inference "Taylor wants to be home." While this path selects the correct answer, it would not be considered explanatory by humans. In general, we find these cases to be more common in multi-sentence situations. The compositionality of the context makes it more challenging to generate directed inferences, and the factor nodes become less reliable in the graph. We observe that performance on multi-sentence contexts drops by ∼5%.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Graph Construction Algorithm. As the quality of the reasoning paths is key to our approach, we investigate the effect of the inference generation algorithm. We evaluate the following generation algorithms: argmax decoding, beam search with beam size b = 5, 10 and top-k sampling (Fan, Lewis, and Dauphin 2018;Holtzman et al. 2018) with k = 5, 10. For each decoding method, we generate a graph using every candidate produced by the decoder (e.g., argmax decoding produces 1 candidate, top-10 sampling produces 10).
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Our results in Table 3 show that the performance COMET -DynaGen is not dependent on the decoding strategy used to dynamically generate the commonsense knowledge graph. This result is promising as it shows that the reasoning procedure is robust to variability in the candidate generations (larger graphs will be less precise). However, it also shows that the approach has difficulty using richer dynamicallygenerated commonsense knowledge representations to answer questions correctly. These results point to the need for future work in developing algorithms that can aggregate larger sets of commonsense inference paths.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Baselines. As with SocialIQa, we report the results of a random baseline, pretrained language models adapted to the task, and a model that only uses φ 0 a to select answers (COMET -Direct). As supervised comparison models, we report the   Gaonkar et al. (2020) that are state-of-the-art for the task.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Overall Performance. Our results indicate that our zero-shot algorithm, COMET -DynaGen, significantly outperforms other zero-shot baselines such as language models, including models with twice the number of parameters. Importantly, again, we see consistent improvement from dynamically generating a contextual commonsense knowledge graph, rather than directly evaluating the answer choices with COMET -Direct. Our full approach yields higher precision, recall, and F1, than the COMET -Direct baseline.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Qualitative Analysis. We once again see the benefit of generating a reasoning graph in Table 5. COMET -DynaGen is able to select the two correct answers to "How does Daniel feel?" leveraging the path through the commonsense inference that "His Dad is helpful" to predict that Daniel is trusting, and the path through the commonsense inference "Daniel wants to try something new" to predict that Daniel is excited. However, there is still much room for improvement, as largescale pretrained language models that are fine-tuned using supervised data perform considerably better on the task.
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Few-shot Tuning. To evaluate the quality of our untuned thresholds from Section based on the label distribution threshold of the CDF of the model's scores (CDF-label in Table 6), we also report the results of our approach using different strategies to set thresholds κ. First, we explore the impact of tuning the κ thresholds on varying amounts of the development set data: 4 examples, 10 examples, 20 examples, and 20% of the development data (the same amount used for validation in Rashkin et al. 2018). In each of these settings, we run a study with 5 different randomly selected sets of examples, and report the average performance. We also report the performance of using the 50 th percentile score of the CDF as the threshold (CDF-50). In Table 6 
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-Shot Commonsense Question Answering
Our neuro-symbolic approach uses neural representations of large-scale commonsense knowledge graphs (COMET) to generate contextual knowledge graphs on demand for zeroshot question answering. Our approach dynamically constructs a knowledge graph of commonsense inferences related to a presented context and uses it to evaluate answer options for a posed question. A novel inference algorithm reasons over the constructed graph to select the most likely answer to a question. Our approach shows promising results at answering questions without training on the end task on two datasets, SocialIQa and StoryCS, outperforming zero-shot pretrained language models. Finally, our analysis indicates that dynamically generating a contextualized commonsense knowledge graph for inference performs better than using vanilla knowledge models (COMET -Direct) to directly answer questions.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Common sense is key to efficient communication in everyday situations, as it enables natural language understanding through contextual reasoning. Machine question answering (QA) benchmarks, like SocialIQA (Sap et al. 2019b) and PhysicalIQA (Bisk et al. 2020), are effective behavioral tests of commonsense reasoning in machines, each focusing on different capabilities. Answering a question in SocialIQA might require the knowledge that readers typically prefer heroes over villains in fantasy novels; whereas, in PhysicalIQA, the knowledge that metal stools can break windows, because windows are made of glass and metal is a more enduring material than glass. Although such tasks had been traditionally difficult for machines, recent developments in pre-trained neural language modeling have Copyright c 2021, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. led to leaps in accuracy-closing the gap between human and machine performance to single-digit percentage points. 1 However, due to increasing concern that large-capacity neural systems are modeling individual datasets, rather than learning how to perform logical reasoning or to utilize external knowledge effectively (Mitra et al. 2019), focus is shifting to alternative training and evaluation strategies. In particular, zero-shot evaluation shows promise as an efficient measure of model generalisability across tasks Li et al. 2020). Here, models are trained and validated on task A, and tested on a different task B, without access to B's training data or labels. This leads state-of-the-art models from individual tasks to falter, sometimes by as much as a 50% decrease in performance .
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Repositories of commonsense knowledge, like ConceptNet (Speer, Chin, and Havasi 2017) and ATOMIC (Sap et al. 2019a), can be beneficial for commonsense QA, especially when little or no training data is available. Enriching the training data with ConceptNet and ATOMIC has been shown (Ma et al. 2019;Mitra et al. 2019) to improve accuracy on datasets derived from these graphs: CommonSenseQA (Talmor et al. 2019) and SocialIQA. Knowledge bases (KBs) can be used to generate questionanswer pairs and distractors automatically, in order to test a model's reasoning ability (Petroni et al. 2019;Richardson and Sabharwal 2019) or provide additional supervision (Ye et al. 2019;Yang et al. 2020). While KBs have been shown to help in a zero-shot transfer setting recently (Banerjee and Baral 2020), no comprehensive study exists on the relation between various knowledge, its usage method, and neural models for zero-shot transfer across commonsense tasks. Moreover, while adversarial filtering techniques ) improve the quality of a manually created question set, their impact on automatically generated questions from a variety of KBs has not been investigated yet.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
In this paper, (1) we compile a set of hypotheses and design a novel neuro-symbolic framework that investigates the dependency between knowledge sources, question generation techniques, language model (LM) variants, and tasks. Our framework leverages a wide range of KBs, covering visual, social, and concept-based knowledge, to pre-train LMs for zero-shot evaluation on multiple-choice commonsense QA tasks. (2) Recognizing that the aspect of question generation is especially understudied, we expand on prior work to devise and test four distractor-sampling strategies for effective question generation. We analyze their impact on model performance across tasks, conditioned on model class and (pre-)training regime, and show that generating questions that are simultaneously fair and informative is difficult but beneficial for LM pre-training. (3) We determine which combination of knowledge graphs (KGs), data construction/training, and architectures is most effective and can utilize appropriately rich contexts across five tasks. We observe that diversifying knowledge generally improves performance, under the condition of it being aligned with the task, and that preserving the structure of the task is desired.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
(4) We make our code and resulting datasets available to the community to facilitate future research in this direction. 2
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Strong performance on standard multiple-choice QA benchmarks, like SocialIQA and PhysicalIQA, has been achieved by fine-tuning a task-specific prediction layer, placed atop pre-trained LMs, such as BERT (Devlin et al. 2019), RoBERTa , and GPT (Radford et al. 2019). As shown by Ma et al. (2019) and Mitra et al. (2019), combining neural methods with structured background knowledge from ConceptNet, WordNet (Miller 1995), and ATOMIC works well for commonsense datasets that have been partially derived from these resources, such as SocialIQA and CommonSenseQA. Here, the structured knowledge, formalized as lexicalized task-targeted evidence paths, is injected into an LM, either via an attention mechanism (Bauer, Wang, and Bansal 2018) or through an auxiliary training objective (Xia, Wu, and Yan 2019). Graph and relation networks can also be used to score answer candidates, by informing the graph structure with data from LMs (Lin et al. 2019;Zhong et al. 2019). Finally, complete KGs can be incorporated directly in training by introducing additional modeling objectives, to teach a model about general commonsense regardless of the task at hand (Peters et al. 2019;Levine et al. 2020;Liu et al. 2020;Zhang et al. 2019;Talmor et al. 2020). This line of work resembles our approach of including background knowledge in a general, task-agnostic way; however, it still relies on the task training data and has generally not been tested in a zero-shot regime. Richardson and Sabharwal (2019) use links in WordNet to generate question-answer pairs, then leverage the resulting dataset to evaluate language models. Petroni et al. (2019) prompt the skills of language models by sentences instead of questions, generated from sources like ConceptNet and SQuAD (Rajpurkar et al. 2016). Previous works have generated synthetic QA sets to complement existing training data. Ye et al. (2019) proposed an 'align-mask-select' method to generate questions using ConceptNet and Wikipedia. Kocijan et al. (2019) constructed a large set of pronoun resolution questions using Wikipedia sentences. Yang et al. (2020) generate QA pair and distractors using generative models. Regarding zero-shot evaluation, the Self-Talk model of ) generates clarification prompts based on a template prefix, which are leveraged to elicit knowledge from another LM, which is used jointly with the original context and question to score each answer candidate. Given a task context, one can use COMET (Bosselut and Choi 2019), a generative model trained on commonsense KGs, to generate background knowledge statements, and to compute scores for each answer candidate based on the context, question, and generated knowledge. Banerjee and Baral (2020) pre-train the LM with three representation learning functions which aim to complete a knowledge triple given two of its elements. These functions jointly compute the distance for each answer candidate. The ambition of this paper is to provide a comprehensive framework for such prior efforts on zero-shot QA with KGs. By covering a wider set of KGs, question generation techniques, and tasks, we can systematically investigate the effect of using different KGs, generation methods, and techniques across tasks.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Given a natural language question Q, and n possible answers A 1 , ..., A n , the task is to select the most probable single answer A. We refer to the remaining n − 1 possible answers: D 1 , ..., D n−1 as distractors. In a zero-shot QA evaluation mode, the system has no access to the task training or development data. We assume a setup where the system is pretrained once and then applied across different tasks in a zeroshot manner. Our zero-shot evaluation framework addresses this task by variants of pre-training an LM on an artificial QA set, created from KG data. Next, we describe its covered tasks, sources of knowledge, question generation strategies, LM techniques, and training regimes, in turn.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
We generate questions, answers, and distractor options from five KGs: ATOMIC, ConceptNet, WordNet, VisualGenome (Krishna et al. 2017), and Wikidata (Vrandečić and Krötzsch 2014), found in the Commonsense Knowledge Graph (CSKG) (Ilievski et al. 2020). Notably, ATOMIC differs from the other KGs in two ways: 1) its relations have a different focus than those of the other sources; and 2) its node labels are longer and formalized as templates. Due to these considerations, we prepare two sets of QA sets: one based on ATOMIC and one based on the remaining four knowledge sources. Figure  1 illustrates our question generation pipeline.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Data partitions ATOMIC expresses pre-and post-states for events and their participants with nine relations. Its head nodes are events, whereas the tail nodes are either events or attributes. Its nodes have two particularities: 1) irrelevant parts of the node text are replaced with blanks (' '); and 2) references to fictional agents are indicated with special tokens (e.g., PersonX). We follow the SocialIQA's ATOMIC train/dev/test splits, to ensure that the facts of the dev and test partitions are excluded in training.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Our second partition, CWWV, covers three other KGs in CSKG that express commonsense facts between concepts: ConceptNet, WordNet, and Wikidata. We use them jointly to generate questions, and we enrich them with additional distractors from VisualGenome. Treating these four sources as a single one is enabled by their CSKG mapping to a single set of relations, defined by ConceptNet. We focus on 14 semantic relations that are grounded on strong psycholinguistic and pragmatic evidence (Murphy 2003), like /r/Causes and /r/HasPrerequisite. Since there is no pre-defined train/dev/test split for CSKG, we randomly sample 5% of generated questions as development set, while the other 95% are used for training, to maximize the coverage of the knowledge.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Generating questions and answers If a triple (h, r, t) has an associated sentence, we directly employ it for question generation; otherwise, we generate a sentence in a lexicalization step, using a set of pre-defined templates. Next, we generate the question Q by removing the tail of the sentence, and extract this tail as the correct answer, A. Here, we ensure that there is no token overlap between the head and the correct answer. For ATOMIC, we: 1) compare the keyword tokens instead of all tokens, in order to avoid stopwords; and 2) the agent templates (e.g., 'PersonX') are replaced with randomly sampled gender-neutral names from a pre-defined set. For CWWV, we filter out questions where either the head or the tail are not common concepts or they are named entities. We use corpus frequency as a proxy for commonness, 3 while named entities are filtered by removing all concepts whose labels start with a capital letter.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Generating negative samples (distractors) We seek to generate distractor options that satisfy two criteria: informativeness and fairness. Namely, a good distractor has semantic relatedness with the context (informative), while being relatively easy to discriminate from the correct answer (fair). We create the pool of distractors D for every sample as follows: 1. The distractor candidates are the tails of knowledge triples (h , r , t ) with the same relation r = r, randomly sampled from the KGs. This would ensure that the distractors can fill the same semantic role as the correct answer. 2. The head h of the sampled triples does not have non-stop word overlap with h. 3. The distractor tail t is not part of the correct answer set, i.e., there exist no triples, (h, 3 https://pypi.org/project/wordfreq/ (Accessed 9 Sept. 2020) Question: Robin takes the fifth. As a result, Robin wanted to A1: go to the cinema. A2: withhold information. (*) A3: hear what they think.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Question: losing weight is for A1: being healthier. (*) A2: embedded software. A3: buying things in store. Table 1: Generated questions from ATOMIC (top) and CWWV (bottom). (*) denotes the correct answer.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
r, t ). Considering the example in Figure 1, the triple (gaining weight, CausesDesire, change appearance) will be filtered out by rule (1), (losing weight, UsedFor, feeling better) will be ruled out by both (2) and (3), and (relaxing, Used-For, feeling better) will be ruled out by (3). Here, we replace any references to fictional ATOMIC agents in the distractors with the same names used in the question. We then randomly select two distractors (D 1 , D 2 ) from D. We refer to this distractor pooling strategy as random, and propose three alternative strategies in the next Section.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Example questions with each partition are shown in Table  1. For ATOMIC, this procedure generates 535K QA pairs for training and 60K for development. For CWWV, the training set contains 157K and the dev set has 8K QA pairs.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Existing data generation procedures are likely to introduce annotation artifacts in datasets (Zellers et al. 2019;Sakaguchi et al. 2019). Models may exploit these artifacts to achieve spuriously strong performance during training, at the expense of degradation in robustness. To generate more challenging QA pairs from KGs and to alleviate potential biases in our synthetic sets, we test two other distractor sampling strategies in addition to the random strategy: 1) we select distractors that are as similar as possible to the answer, while being under a certain threshold (adv-answer); and 2) we select distractors that are as similar as possible to the question, while being under a certain threshold (advquestion). Here we define similarity of two nodes to be their proximity in the embedding space, measured by cosine similarity. The intuition is that, by generating more challenging QA pairs for the models, we could achieve better generalization across tasks. We use the RoBERTa sentence embedding model (Reimers and Gurevych 2020) to compute embeddings for all KG nodes. For these two strategies, we set an upper bound on the similarity score to avoid unfair distractors, i.e., paraphrases of the correct answer. Based on manual observations, we set their distractor similarity upper bound to be 0.6 for CWWV and 0.4 for ATOMIC.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Sample filtering Besides these distractor sampling strategies, we test another condition (3), where we select the distractors randomly, but only keep the questions whose distractors are sufficiently challenging at training time (advfilter). The intuition is that QA pairs generated using the aforementioned methods might still be too easy for the mod-els, thus we would like to only keep the most challenging subset to train our models. We employ the AFLite algorithm ) for our purpose. Given a train and dev split of our synthetic QA set, we use 5% of the train set to finetune a RoBERTa model with a classification head (4% training, 1% validation). These 5% are discarded from train after this step. We then compute the fixed embeddings for the remaining 95% of train and the entire dev, denoted as Trn and Dev. Next, we feed Trn and Dev along with their labels to the AFLite algorithm, which iteratively filters out easy examples using an ensemble of linear classifiers. Finally, we retain (101K training, 11K dev) samples for ATOMIC and (29K training, 1.5K dev) samples for CWWV subset. The details of AFLite can be found in the appendix.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
We consider 2 types of language models: auto-regressive language models and masked language models (MLM). Specifically, we use GPT-2 and RoBERTa to select the best answer candidate. Given a context C, a question Q, and a list of answer options (A 1 , A 2 ...), we concatenate C and Q with each answer option to build input sequences (T 1 , T 2 ...). We also use templates to convert a sequence T into a natural language sentence following . For example, we transform the sequence: [C] What will X want to do next? [A i ] into: [C], as a result, X want to [A i ]. The score S for the resulting sequence using an auto-regressive LM is computed as follows:
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
where n is the number of tokens in the sequence and P is the conditional probability provided by the LM. To evaluate MLMs, we mask out one token at a time and compute its loss ). We repeat this process for every token in the sequence. The final MLM score is:
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
The predicted option is the one with the lowest score.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
LM Finetuning In the typical model architecture for finetuning LM for multiple-choice tasks, a linear layer is added on top of the LM encoder to predict the answer. The model inputs are separated by a model-specific delimiter. However, as this architecture introduces randomly initialized parameters, it may not be able to fully utilize the pre-trained weights (Tamborrino et al. 2020). Instead, we re-use the GPT-2 and RoBERTa with LM head for finetuning. By keeping the model intact, we can reuse the same converting templates and scoring functions. To train the model, given the scores computed for each answer candidate S 1 , S 2 , ...S m , we use the marginal ranking (MR) loss defined as:
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Here, η represents the margin and y is the index of the correct answer. For a MLM model, the computation cost for the scoring function scales quadratically with the input length.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
To make the training more efficient, we only mask out nonstop tokens in the head and tail nodes.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Training Regimes In order to disentangle the contribution of the KGs from the structure of the QA pairs, we consider different training methods for augmentation of language models with KGs. Specifically, we compare marginal ranking (MR) training with masked language modeling (MLM) training. For MLM, we directly concatenate the question and the correct answer in our synthetic QA set and then train RoBERTa on the these sentences using the MLM objective.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
We select commonsense tasks based on two criteria. Firstly, we strive to cover a diverse set of tasks, both in terms of their format (question answering, pronoun resolution, natural language inference), as well as their type of knowledge (e.g., social or physical knowledge). Secondly, we prefer larger task datasets that are manually constructed. For this reason, we do not include datasets like COPA (Gordon, Kozareva, and Roemmele 2012), or HellaSwag (Zellers et al. 2019). We opt for the following five task datasets:
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
1. Abductive NLI (aNLI) ) is posed as a natural language inference task. Given the beginning and the ending of a story, the task is to choose the more plausible hypothesis out of two options. The dataset consists of nearly 170k entries. 3. PhysicalIQA (PIQA) (Bisk et al. 2020) is a two-choice question answering dataset which focuses on physical reasoning. Given a question, the system (or human) is asked to pick the more plausible out of two possible continuations.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
4. SocialIQA (SIQA) (Sap et al. 2019b) is a questionanswering dataset which requires reasoning about social interactions. Each entry contains a context, a question, and 3 answer candidates. The context is derived from the ATOMIC knowledge graph, the questions are generated based on nine templates (corresponding to the relations in ATOMIC), and the answers are crowdsourced.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
44 thousand pronoun resolution problems. Each entry consists of a context description with an emphasized pronoun, and two options are offered as its possible references.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
We compare our results with the following baselines. Majority answers each question with the most frequent option in the entire dataset. 'Vanilla' versions of the language   models are used in order to understand the impact of further tuning. Here we directly uses the LMs to score the QA pairs without any finetuning. We also show the results of other unsupervised systems that leverage KGs: Self-talk, COMET-DynaGen, and SMLM. To indicate the upper bound of this work, we include results of a supervised fine-tuned RoBERTa system and of human evaluation.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
For the LM baselines, we directly load the weights from the Transformers library (Wolf et al. 2019) and evaluate on the downstream tasks. The finetuned LMs are trained for a single epoch on our synthetic QA set. For Adv-filter, we train the models for 5 epochs to compensate for less training data. We use our synthetic dev set to select the best model. We describe other hyper-parameters used and computing infrastructure in the appendix.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Based on individual prior findings and understanding of different components of our framework, we put forward a set of hypotheses which will be validated in our experiments:
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
H1: RoBERTa would have better performance than GPT-2. This is in line with prior findings that RoBERTa has the advantage of bi-directional context .
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
H2: Pre-training a language model with artificially created question-answer sets enhances zero-shot performance. This is also supported in previous study about unsupervised QA (Li et al. 2020) H3: The impact of more knowledge depends on the alignment between KGs and the task, partial evidence for which is provided by (Ma et al. 2019;Mitra et al. 2019).
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
H4: Adding diverse knowledge (from different KGs) improves performance. This is the initial motivation behind the creation of CSKG (Ilievski et al. 2020), but has not been investigated in detail. H5: When selecting negative samples for a question, it helps to use an adversarial strategy that ensures the question is not trivial for a language model. H5 is inspired by adversarial filtering, which has not been investigated in detail for automatically-generated questions and across KGs. H6: Preserving the task structure when generating synthetic data leads to better accuracy. This is implicitly assumed in prior data augmentation work (Kocijan et al. 2019). H7: The automatically created questions are notably easier for humans than they are for machines -a general assumption made by commonsense task creators and typically correct for any existing, human-generated benchmark.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
We evaluate various combinations of: knowledge sources, question generation strategies, LMs, training regimes, and tasks. We use accuracy as a metric. All our experiments are performed in a zero-shot setting, i.e., the models do not leverage the official training data of the task. We report results on the dev sets of these tasks, as the official test sets are not publicly available. We note that, since we did not use the tasks' dev sets for hyperparameter tuning or checkpoint selection, the dev sets can be used effectively as test sets. Table 2 shows that GPT-2 and RoBERTa outperform the majority baseline by a large margin on all tasks, indicating that  the LMs have already learned relevant knowledge during pretraining. Despite being a smaller model, RoBERTa outperforms GPT-2 on 4 out of 5 tasks without pretraining, and on all tasks when pretraining over different synthetic QA sets. This shows the advantage of leveraging bi-directional context, and confirms our hypothesis H1. As expected (H2), training RoBERTa on our ATOMIC or CWWV synthetic sets brings notable performance gain on all 5 tasks. We observe that models trained on ATOMIC sets have a large advantage on SIQA compare to models trained on CWWV, while CWWV brings advantage on the CSQA task. This is not surprising as these two tasks are derived from ConceptNet and ATOMIC, respectively. The difference between ATOMIC and CWWV on the remaining three tasks is relatively small. This supports our hypothesis H3: knowledge alignment is crucial for obtaining better performance.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Training on the combined question set (CSKG) is mostly able to retain the best of its both partitions. Training on CSKG leads to best performance on three out of five tasks, showing that a global commonsense resource is able to bring consistent gain across different tasks. This supports our hypothesis H4: adding more diverse knowledge is beneficial for language models. Finally, even with this knowledge, we recognize that there is still a large gap between our model's accuracy and that of the supervised RoBERTa model. Table 3 shows the results with different sampling strategies, thus addressing H5. The best performing adversarial algorithm, Adv-answer, yields comparable accuracy to the random strategy, revealing that distractors sampled with a more sophisticated strategy are not necessarily more informative for the LMs. Adv-question and Adv-filter typically lead to declines in accuracy. Considering Adv-question, this could be due to the similarity of the distractors to the question, which might guide the model to learn to pick the most dissimilar candidate as the correct answer, which is an artifact of our question generation and cannot be expected to work well for downstream tasks. Our manual inspection of the remaining questions prefered by Adv-filter indicates that many questions are unfair, as some distractors are also correct answers, which is a consequence of the incompleteness of the KGs. Adv-filter prioritizes these questions as they are "difficult" for LMs, however, training on them might teach the LM incorrect knowledge and harm downstream accuracy. Table 4 presents results with two different training regimes. In comparison to the baseline without additional training, MLM training on ATOMIC only improves on the SIQA task, and harms on the rest. With CWWV, it brings large gain on CSQA and small improvements on SIQA and WG. At the same time, marginal ranking training on either question set consistently outperforms MLM training by a large margin, suggesting that preserving the task structure is beneficial in addition to the question content and validating H6.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Ideally, the generated question-answer pairs should be challenging for the models but easy for humans to solve (H7). Here, we probe this hypothesis by assessing the difficulty of our synthetic QA sets both by humans and 'vanilla' LMs. We evaluated both models on the dev sets of our synthetic data. For human evaluation, we randomly sample 50 questions from ATOMIC and 50 questions from CWWV. A total of five researchers were asked to first provide the correct answer, then rate the question difficulty. For the latter, the annotator chose between easy, moderate, hard, or non-sensical -as a guideline, nonsensical questions have unfair distractors and cannot be easily understood. Following this procedure, we obtained three judgments for each question.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
The inter-annotator agreement on selecting the correct answer is 0.62 using Fleiss Kappa score, which is substantial agreement. The Kripendorf alpha (Krippendorff 2004) for rating question difficulty is 0.35, which is fair agreement. The results of the baseline LMs and human performance (Table 5) show that the ATOMIC subset presents a harder challenge for both models, as well as for humans. Overall, the results support our hypothesis H7: the synthetic questions are relatively easy for humans to solve and much harder for models. However, the annotation pointed to several directions for improving the synthetic QA sets. A number of questions generated from ATOMIC are ungrammatical, which makes them harder to understand, while some questions from CWWV were rated as unfair. For example, all answer options for the question A person can are valid: (a) cost several thousand dollars (b) expressing boredom (c) check snow level. As discussed earlier, this is due to the incompleteness of our KGs, and the current lack of understanding on how to generate fair, yet informative, distractors.  
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
The overarching pursuit of this paper is to understand whether generating artificial QA sets with KGs improves the zero-shot QA performance of LMs. We observe a consistent leap in accuracy across tasks, LMs, knowledge sources, and question generation strategies. While these accuracies are notably below supervised LM accuracies, they might further improve by architectural improvements of the LMs, knowledge sources with wider coverage and stronger semantics, and well-tuned scoring functions. 4 In addition, despite its complexity and diversity, commonsense knowledge (unlike knowledge on entities and events, which changes rapidly) is largely static and evolves slowly over time, thus making the dataset-specific finetuning unnecessary in theory. A natural question arises: can we build a sufficiently reliable, general commonsense service, by pretraining a LM on a rich set of questions covering a wide spectrum of knowledge types?
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
In general, we observed that using knowledge from a wider set of sources is beneficial. However, on aNLI and CommonSenseQA, the best accuracy was obtained with a subset of all questions. This could be due to the kinds of knowledge covered: 1. aNLI focuses on expectations of agents and events, for which ATOMIC is directly useful, whereas the other KGs might be delusive; 2. CommonSenseQA mostly requires knowledge about properties of objects (e.g., function or appearance) which is the focus of ConceptNet, Wikidata, and VisualGenome, but not of ATOMIC. This indicates a tension between H3 and H4: while more knowledge often helps, it might not be the case when the task and the knowledge are not well-aligned. Our current understanding of the dimensions of commonsense knowledge in knowledge sources and benchmarks is limited, and would benefit from further study.
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Alternatively, this result may be explained by our human evaluation: not all automatically generated questions are fair and a subset has more than one correct answer, as a direct consequence of the inherent incompleteness of KGs. Besides being a disadvantage of automatic question generation,  this finding points to a more substantial challenge: generating fair and informative multiple-choice questions is not yet well-understood. Adversarial strategies yield more plausible candidates than random sampling, making the task less fair; yet, fully relying on random sampling would generate distractors that are trivially discernible from the correct option. Balancing between fairness and informativeness, thus, is essential for multiple-choice question generation. Our empirical evidence suggests that it could be achieved by a mixed approach, where part of distractors is generated randomly, and part by adopting suitable adversarial strategies. 5
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
While zero-shot QA is gaining focus, no study so far has investigated systematically the dependency between the task, the technique, and any additional knowledge used. To address this gap, this paper proposed a framework for zeroshot QA by pretraining LMs on artificial data created based on KGs. We studied the interplay between five knowledge sources, four data generation strategies, two LMs, two training regimes, and five tasks. We put forward seven hypotheses, which guided our experiments. We observed that language models benefit from pretraining on artificially created QA sets. Similar to prior work, we observed that the best performance is obtained by the RoBERTa model. Furthermore, combining knowledge sources leads to best performance, indicating that diverse relevant knowledge is desired for pretraining. However, this is conditioned on a good alignment between knowledge and the task. The training regime had a role too: marginal ranking is superior to vanilla language modeling, as it preserved better the structure of the task. Our analysis and human evaluation indicated that the generated questions were typically easier for humans than for language models, which is appropriate for commonsense questions. Yet, the human evaluation revealed that a notable portion of the questions is nonsensical or difficult, hinting that automatically generating high-quality, informative commonsense questions is non-trivial and should be revised in subsequent work. Future work should also investigate the impact of this approach on knowledge injection systems (Ma et al. 2019) and graph relational networks (Lin et al. 2019). It should also consider: (1) other, less structured knowledge sources, like WikiHow; (2) different distractor sampling strategies, e.g., based on iterative sampling (Niu et al. 2020); and (3) additional LM scoring functions, e.g., based on scoring sequences of tokens (Tamborrino et al. 2020).
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
Recently, the task of commonsense reasoning has attracted much attention, as believed to be a critical and yet challenging component of human-level intelligence (Levesque et al., 2012, Davis, 2017Wang et al., 2019a). To test models' ability to understand natural language and reason with external commonsense knowledge, efforts have been made towards building many challenging WSClike (Winograd Schema Challenge) tasks and QA (question-answer) tasks. Specifically, (Zhang et al., 2020a) crowd-sourced human-provided justifications as reasons for the WSC problems, resulting in a new dataset called WinoWhy. An example of WinoWhy is shown in Table 1, the model is asked to determine whether the given reason for the WSC problem is correct. Meanwhile, constructed based on ConceptNet (Speer et al., 2017), Comon-senseQA (Talmor et al., 2019) is designed as a fivechoice QA dataset that requires model to capture * Corresponding Author A WinoWhy Example WSC Question: Joan made sure to thank Susan for all the help she had received. She refers to Joan because Reason: Joan is doing the thanking so she must have received the help. Label: Positive Convert WinoWhy to NLI Premise: Joan is doing the thanking so she must have received the help. Hypothesis: Joan made sure to thank Susan for all the help Joan had received. Label: entailment Table 1: A WinoWhy example consists of WSC question and reason, while the label is "Positive" or "Negative". We use NLI as a common task and convert WinoWhy to NLI form.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
the relation between the question and the correct answer. In this work, we experiment on these two commonsense datasets.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
Although diverse methods based on pre-trained language models and external knowledge have shown very strong supervised performance on commonsense reasoning, the solution process is usually complex and expensive. Generally, we should gather task-specific training data and then train models to learn the patterns in data. However, as shown in WinoGrande , acquiring unbiased labels requires a carefully designed crowd-sourcing procedure, which greatly adds to the cost of data collection. Moreover, supervising on large training sets is usually timeconsuming. Therefore, instead of applying specific methods to the corresponding task, a reasonable framework is to convert diverse commonsense reasoning tasks to a common task and use a general unsupervised method to solve it. Furthermore, some tasks that lack sufficient annotations can be solved by the framework.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
We attempt to use Natural Language Inference (NLI) as the common task mentioned above. NLI is the task of determining whether a hypothesis is "entailment" or "not entailment" to a given premise. Figure 1: Overview of our NLI framework with injected knowledge from the knowledge base. The NLI-LM and NLI-Classifier denote LM with a classification head fine-tuned on NLI. We convert the original example of the source task, e.g., WinoWhy or CommonsenseQA, to NLI form and combine KB sentences as input.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
NLI task is well-suited to be a common task, as it assembles the skills involved in sentence understanding, from the resolution of syntactic ambiguity to pragmatic reasoning with world knowledge (Wang et al., 2019b). Furthermore, the NLI task has been actively studied, especially since the emergence of large-scale datasets (Bowman et al., 2015;Williams et al., 2018), and we can directly leverage the progress. Moreover, we explore whether injecting external knowledge from knowledge bases to our framework can enhance the model's performance over commonsense reasoning tasks.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
We apply RoBERTa , which has shown powerful performance on NLI tasks, as the backbone network of our NLI framework. We first convert a commonsense reasoning task to an NLI form as the original input. As shown in Table 1, we replace the pronoun "she" of the original WSC sentence with the correct candidate "Joan" and treat the replaced sentence as the hypothesis, while the given reason is the premise. Next, to leverage external knowledge, we use the recently-introduced ATOMIC 2020 (Hwang et al., 2020) and Concept-Net as knowledge bases (KBs). Specifically, we extract KB triples from KB by matching semantic similarity between the embeddings of KB and the source task and then combine the triples and the original input for RoBERTa. Our experimental results on WinoWhy and CommonsenseQA suggest that the NLI framework is suitable for commonsense tasks and external knowledge can provide useful information to help the model make the correct prediction. Furthermore, more improvements can be obtained by combining multiple effective categories of knowledge. In addition, models perform worse when facing problems about quantity knowledge and antonym relation.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
In this section, we describe the details of 1) using the NLI framework to solve commonsense reason-ing tasks and 2) extracting knowledge from KBs, and 3) injecting the external knowledge into the NLI framework. The overview of our framework is shown in Figure 1.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
The key to solving commonsense reasoning tasks such as WinoWhy and CommonsenseQA is to determine the relation between question-answer pairs. Follow this intuition, we use a general task NLI, which aims at identifying whether a hypothesis sentence can be entailed by a premise sentence. We first convert the original example of the source task to the NLI form. In this work, we define the source task as to predict whether an answer is entailed given a question. We can find that the question corresponds to the premise and the answer to the hypothesis. Moreover, for source tasks like WinoWhy, we can also try to convert the question (e.g., WSC question shown in Table 1) to a statement as the hypothesis, and treat the reason as a premise, following the if-then relation. Then, we use pre-trained language models (LM) with a classification head to solve the NLI task. Specifically, given a premise and a hypothesis, we concatenate them as the "NLI sentence": [CLS] Premise [SEP] Hypothesis [SEP]. The LM with the classification head then predicts the entailment relation.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
To mitigate the data scarcity in an unsupervised setting, we consider transferring knowledge from large NLI datasets. Specifically, we fine-tuned the LM and classification head on either MNLI (Williams et al., 2018) or QNLI (Wang et al., 2019b). We use the RobertaForSequenceClassification from the transformers library (Wolf et al., 2020). It is the RoBERTa with a classification head on top. When evaluating our framework on source task, because MNLI has three labels: "entailment", "neutral", and "contradiction", we treat the last two labels as "not entailment". We denote the LM and classification head fine-tuned on NLI datasets as NLI-LM and NLI-classifier.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
In the following, we show how to extract knowledge from KB and inject the matched knowledge into the NLI framework. We first convert the triples in KB to natural language sentences and extract KB triple from KB by calculating cosine similarity between the embeddings of KB sentence and source task example. Finally, we combine the external KB sentence and original example to help NLI-LM and NLI-Classifier perform the correct prediction.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
Convert KB Triple to Natural Language Inspired by ATOMIC (Sap et al., 2019a), which is unique in that the entity in a triple is mostly short sentences, we try to convert KB triple to natural language sentence, then capture helpful knowledge for original example by matching semantic similarity between them. For example, (Per-sonX thanks PersonY afterwards, isAfter, PersonX asked PersonY for help on her homework), a triple in ATOMIC, can be extended to "After PersonX asked PersonY for help on her homework, PersonX thanks PersonY afterwards", and (having_no_food, CausesDesire, go_to_a_store), a triple in Concept-Net, corresponds to "having no food makes someone want go to a store". In this work, we use ATOMIC 2020 (we call it ATOMIC for short in the following) and ConceptNet as knowledge bases. We define templates for every relation in ATOMIC and ConceptNet. Then we convert the triples in ATOMIC and ConceptNet to natural language sentences automatically using the templates. We named the natural language sentence "KB sentence".
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
Extract Knowledge from KB Inspired by (Reimers and Gurevych, 2019), we use NLI-LM to generate the token embeddings of an NLI sentence or a KB sentence. Then we compute the mean of all token embeddings. As shown in Figure 2, an NLI sentence and a KB sentence are input into NLI-LM, and two mean embeddings are output. Then we calculate the cosine similarity between two embeddings as the semantic similarity of two input sentences. When input into NLI-LM, a NLI sentence is composed of the form "[CLS] Premise [SEP] Hypothesis [SEP]" and a KB sentence is wrapped by [CLS] and [SEP] as well. When evaluating our framework on a commonsense dataset, for each example, we extract the KB sentences with TopK semantic similarity.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
To combine a KB sentence and an NLI sentence, we inject the KB sentence into the middle of the NLI sentence to form a combined sentence. Thus, the form of the combined sentence is "
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
For an NLI sentence with TopK matched KB sentences, we can generate K combine sentences. All of them are input into NLI-LM and K CLS-token embeddings are output. Then we compute the mean of all CLS-token embeddings. Finally, the mean embedding is input into the NLI-Classifier and the entailment relation is output.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
We evaluate our framework on two commonsense reasoning datasets, WinoWhy and Common-senseQA. Both require commonsense knowledge beyond textual understanding to perform well. All of our experiments use an unsupervised setting, i.e., our model does not train on the source task.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
WinoWhy (Zhang et al., 2020a) contains 2,865 reasons, which belongs to 273 WSC examples respectively. We evaluate models on the full set. A WinoWhy example consists of a WSC question and reason. We use two strategies to convert an example to an NLI sentence. As the example shown in CommonsenseQA is a multiple-choice QA dataset that specifically measures commonsense reasoning. This dataset is constructed based on ConceptNet. We evaluate models on the development set with 1,221 questions since the answers to the test set are not publicly available. A Com-monsenseQA example consists of a question and 5 choices. We regard the question and a choice as a NLI sentence with the form "[CLS] Q: question [SEP] A: choice [SEP]" (The additional "Q" and "A" follows the recommendation from the FairSeq repo on how to fine-tune RoBERTa on CommonsenseQA 1 ). Then entailment score of every choice is calculated. Finally, the choice with the highest score is selected as the answer to the question. In addition, the form of combine sentence is "
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
ATOMIC (Sap et al., 2019a) is a knowledge base consists of 880K of triples across 9 relations that cover social commonsense knowledge, e.g., (X gets X's car repaired, xIntent, to maintain the car), including aspects of events such as mental states, personal attribute, and social effect. As the later work, (Hwang et al., 2020) extends ATOMIC to ATOMIC 2020 with 1.33M triples. ATOMIC 2020 introduces 23 commonsense relations. Triples are of the form ({Event | Entity}, r, {Entity | Event | Behavior | Persona | Mentalstate}), where head and tail are nouns or short sentences and r represents an if-then relation type or physical property (e.g., xIntent and ObjectUse). We define 23 templates for every relation in ATOMIC 2020 to automatically convert triple to natural language sentences.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
ConceptNet (Speer et al., 2017) is a knowledge base focus mostly on taxonomic and lexical knowledge (e.g., IsA, PartOf) and physical commonsense knowledge (e.g., MadeOf, UsedFor). We extracted 29 relations to form a subset with 485K entityrelation triples. Similar to ATOMIC, we define 29 templates for every relation. In this work, we   Table 3: Performance comparison on the dev set of CommonsenseQA. The accuracies of Self-Talk and SMLM are reported by (Shwartz et al., 2020) and (Banerjee and Baral, 2020). "BERT-Base Sup." denote the base model of BERT training on CommonsenseQA training set and the result is the accuracy of the test set reported by the official leaderboard.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
use ATOMIC 2020 and ConceptNet for injecting external knowledge to NLI framework.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
For WinoWhy, we consider the pre-trained lan-  +WinoGrande. We directly use the results reported by (Zhang et al., 2020a).
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
Same as WinoWhy, we use RoBERTa as baselines for CommonsenseQA. Specifically, we use RobertaForMaskedLM from the transformers library (Wolf et al., 2020). It can be regarded as a RoBERTa Model with a masked language modeling head on top. Given a CommonsenseQA question and one of the five choices, we mask the choice tokens and use the masked LM head to predict them. For example, a CommonsenseQA sentence input to model consists of the form: "[CLS] question [SEP] choice [SEP]". Then the choice will be masked and the masked LM head is used to predict the cross-entropy loss for it. Finally, the choice with the lowest loss will be selected as the answer to the question.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
In addition, we compare our model with Self-Talk (Shwartz et al., 2020) and SMLM (Banerjee and Baral, 2020). These two models both propose an unsupervised framework to multiplechoice commonsense tasks and show considerable improvements over large pre-trained language models. So we report their dev-set accuracies on Com-monsenseQA as baselines. Table 2 and Table 3 show results of applying NLI framework and external knowledge to WinoWhy and CommonsenseQA. Our framework has achieved state-of-the-art (SOTA) unsupervised performance on WinoWhy by a large margin. Specifically, using the same language model RoBERTa, we observed improvements ranging from +8.52% (66.70% by Base+MNLI+CN)  As for results on CommonsenseQA, we first observe that RoBERTa is struggling near the Random Guess baseline. This result illustrates that RoBERTa completely cannot deal with Common-senseQA without training. However, after converting CommonsenseQA to NLI form and injecting KB sentences, RoBERTa behaves a lot better. RoBERTa-Base + MNLI + CN/ATOMIC gets a comparable result compared to Self-Talk, while RoBERTa-Base + QNLI + CN/ATOMIC have already exceeded SMLM, the previous SOTA method. Finally, it is interesting to note that RoBERTa-Large + QNLI + ATOMIC is slightly worse than BERT-Base model training on the Com-monsenseQA training set. Now we focus on results applying the NLI framework without injected knowledge. For WinoWhy, RoBERTa can achieve a considerable boost after being fine-tuned on either QNLI or MNLI. For CommonsenseQA, RoBERTa fine-tuned on QNLI can get +4.62% higher dev-set accuracy than Self-Talk and comparable result to SMLM. The experiment clearly illustrates the effectiveness of the NLI framework and transfer learning from NLI datasets.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
When we inject KB sentences to RoBERTa finetuned on QNLI, improvement can be observed on full-set accuracy for WinoWhy and dev-set accuracy for CommonsenseQA. This indicates that the knowledge from QNLI and that extracted from KB complement each other. On the other hand, external knowledge, either from ATOMIC or ConceptNet, is not much help to RoBERTa fine-tuned on MNLI and even causes a drag. We hypothesize that there Figure 3: Effect of K on WinoWhy and Common-senseQA. We inject KB sentences with topK similarity into NLI-LM. The figure above is the results on WinoWhy, while the below is on CommonsenseQA. The "Base" and "Large" denote RoBERTa-Base/Large + QNLI. is a high overlap or even contradiction between the knowledge of KB and MNLI, which causes the incompatibility between them.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
In summary, We think the reasons leading to the significant improvement are 1) NLI framework is better suited for such tasks; 2) RoBERTa picks up necessary knowledge from the NLI datasets; 3) ATOMIC and ConceptNet provide some useful information to source tasks and help models make the correct prediction.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
Category of Injected KB Sentences In order to study whether the different categories of external knowledge will have a large impact on the model's performance, we divide ATOMIC into five categories: Physical-Entity, Event-Centered, Mental-State, Persona, and Behavior, basically following the definition of (Hwang et al., 2020). Physical-Entity deals with inferential knowledge about common entities and objects. Event-Centered provides intuitions about how common events are related to one another. Mental-State addresses the emotional or cognitive states of the participants in a given event. Persona describes a person's attribute as perceived by others given an event. Behavior address the socially relevant responses to an event. We inject each category separately into RoBERTa-Large + QNLI (the best NLI-LM in our experiment). The results are shown in Table 4. Similar to ATOMIC, we divide ConceptNet into four categories: Physical-Entity, Event-Centered, Social-Interaction, and Taxonomic-Lexical. The meanings of the former two categories are the same as ATOMIC. Social-Interaction focuses on socially triggered states and behaviors. Taxonomic-Lexical focus on taxonomic and lexical. The results are shown in Table 5. It is not surprising that there are some categories of knowledge dragging down the performance. For example, for ATOMIC, injecting the knowledge of Physical-Entity obtains the worst results on either WinoWhy or CommonsenseQA.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
Next, we wonder if we can get higher accuracies after combining the effective categories. So we combine the two most effective categories for each task. On ATOMIC, they are Behavior and Event-Centered. On ConceptNet, they are Taxonomic-Lexical and Event-Centered for WinoWhy, while Physical-Entity and Event-Centered for Common-senseQA. The results show that this strategy makes the performance exceed the "Overall". The slight boost of accuracies illustrates that our assumption is correct. We also find that combining any two categories does not necessarily work through the results of ConceptNet. For example, combining Taxonomic-Lexical and Event-Centered does not get a higher result than "Overall" on Com-monsenseQA, because of the bad performance of Taxonomic-Lexical. It tells us that we need to identify effective knowledge when combining different categories.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
As mentioned before, we extract KB sentences with topK similarity. Now we investigate the impact of hyperparameter K with experimental results shown in Figure 3. The results generally follow the intuition that the more knowledge is injected, the better the performance is until the amount of injected sentence reaches a threshold. Then the accuracy begins to decrease. We think the reason is that knowledge with lower semantic similarity introduces noise to the model and then plays a distracting effect.   
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
To discuss the performance when the model faces different knowledge types, we follow the knowledge types defined in (Zhang et al., 2020a) and divide WinoWhy into five subsets. We evaluate RoBERTa-Base/Large + QNLI on each subset. The results are shown in Table 6. "Property" denotes the knowledge about the property of objects. "Object" represents that about objects. "Eventuality", "Spatial" and "Quantity" corresponding to eventualities, spatial position, and numbers, respectively. Comparing RoBERTa-Large fine-tuned on WinoGrande (the best model reported by Zhang et al., 2020a) and RoBERTa fine-tuned on QNLI, the latter goes beyond the former on all knowledge types. It is no doubt that QNLI contains more commonsense knowledge needed by WinoWhy than WinoGrande. Now let us focus on the comparison between models with and without KB sentence. It is shown that KB sentences matched for WinoWhy examples can provide some performance boost on most knowledge types, suggesting that we successfully inject the effective knowledge from KB to RoBERTa. Further, we can find that whether for RoBERTa + QNLI or RoBERTa + QNLI + KB, the worst performances appear on "Quantity". What's more, the injected knowledge, either from ATOMIC or Con-ceptNet, brings the lowest benefit to "Quantity", and even results in the only drag for RoBERTa-Large + QNLI + CN (-1.00). The reason for this result may be due to the lack of knowledge about numbers in QNLI, ATOMIC, and ConceptNet. We can find that large corpora do often lack quantity knowledge. This gives us the idea that constructing and encoding quantity knowledge into LM in the future.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
Similar to WinoWhy, we follow the experiment described in (Ma et al., 2019) and divide Com-monsenseQA into five subsets. We classify questions based on the ConceptNet relation between the question concept and the correct answer concept. Then we select the relations with more than 40 questions as knowledge types. Observing experimental results shown in Table 7, we can derive the same conclusion as WinoWhy that injecting knowledge following our method can provide useful information to LM and help make the correct decision. However, accuracies on "Antonym" are the lowest compared with other knowledge types. And the boosts are also the lowest after injecting knowledge. "Antonym" denotes that A and B are opposites in some relevant way, such as black and white. We guess it is because the language model has a weak ability to deal with antonym relations.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
In addition, we find that ATOMIC can bring more benefits to RoBERTa compared with ConceptNet. As described in (Hwang et al., 2020), triples in ConceptNet are limited to mostly taxonomic, lexical, and object-centric physical knowledge, making the commonsense portion of ConceptNet relatively small. While ATOMIC has more knowledge related to social commonsense, and relatively, the coverage is more extensive and balanced. Our experimental results are consistent with these descriptions.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
Commonsense Reasoning Recent commonsense reasoning datasets Zhou et al., 2019;Sap et al., 2019b;Bisk et al., 2020;Talmor et al., 2019 ) have motivated research in several domains of commonsense: abductive, temporal, social, and physical. SOTAs for most of them have achieved over 80% accuracy, which is close to human performance (e.g., Brown et al., 2020;Khashabi et al., 2020;Raffel et al., 2020). However, their success is due to larger pre-trained corpora and much more parameters, which is difficult to be followed for most researchers. In addition, other useful methods (Yasunaga et al., 2021;Feng et al., 2020; generally require training on training sets and knowledge graphs. When applying them to different tasks, the same running and tuning process should repeat for several times to find the best fit. Thus, we propose a framework to convert diverse commonsense reasoning tasks to a common task, NLI, and use a general unsupervised method to solve it.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
Natural Language Inference Since GLUE regards NLI as a benchmark task for testing the natural language understanding capability of the model, NLI has been well studied, and language models have achieved performance beyond humans on some NLI datasets. Furthermore, by leveraging transfer learning from large NLI datasets, great performances have been achieved in several tasks, such as story ending prediction (Li et al., 2019), intent detection (Zhang et al., 2020b), semantic textual similarity (Reimers and Gurevych, 2019). Therefore, we attempt to use NLI as the common task to solve commonsense reasoning.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
External Knowledge Most commonsense reasoning tasks require models to synthesize external commonsense knowledge and leverage more sophisticated reasoning mechanisms. The key is to extract effective information from commonsense sources, such as ATOMIC, ConceptNet, and Wikipedia. Methods learn commonsense knowledge either by KGs pre-training Ye et al., 2019) or by reasoning on knowledge graphs (Feng et al., 2020;Lv et al., 2020;Lin et al., 2019). In order to cooperate with our NLI framework, we convert the triples in KB to natural language sentences and extract triples by calculating cosine similarity between the embeddings of KB sentence and source task example.
Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference
In this work, we propose a framework to convert diverse commonsense reasoning tasks to a common task, NLI and use a pre-trained language model, RoBERTa to solve it. By leveraging transfer learning from large NLI datasets, QNLI and MNLI, and injecting crucial knowledge from knowledge bases such as ATOMIC and ConceptNet, our framework achieved SOTA unsupervised performance on two commonsense reasoning tasks: WinoWhy and CommonsenseQA. Experimental results show that knowledge from QNLI and extracted from either ATOMIC or ConceptNet can complement each other to enhance the model's performance on commonsense reasoning. More improvements can be obtained by combining multi categories of effective knowledge. Further experiment shows that ATOMIC can bring more benefits to RoBERTa compared with ConceptNet. However, injected knowledge is not much help to RoBERTa finetuned on MNLI and even causes a drag. In addition, models perform worse when facing problems about quantity knowledge and antonym relation. The code is publicly available: https: //github.com/sysuhcm/NLI-KB.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Everyday causal reasoning requires reasoning about the likely explanations to partially observable past and future (abductive reasoning (Peirce, 1960)) and reasoning about the alternative future based on counterfactual past (counterfactual reasoning). Such nonmonotonic reasoning requires She hit the rope and the tire fell on top of her.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Ray hung a tire on a rope to make his daughter a swing.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Ray ran to his daughter to make sure she was okay.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Zeke thought about being a vampire or a wizard.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Then he decided on a scarier costume.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Zeke dressed up like a skeleton.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Lannister, but he didn't want to look like a Lannister.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
He wanted to look like a Stark.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Zeke dressed up like a Stark.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Zeke was throwing a party.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
All his friends were dressing up for this Halloween party.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
All his friends were dressing up for this Game of Thrones themed party.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
[Counterfactual]
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Hypothesis Counterfactual Reasoning Figure 1: DELOREAN, our proposed method, with generated reasoning results. Top: the goal in abductive reasoning is to generate a hypothesis (Y ) of what happened between the observed past (X) and future (Z) contexts. Bottom: In counterfactual reasoning, given a story context altered by a counterfactual condition, X, and the original ending Z, the goal is to generate a new ending Y which is coherent with X while remaining similar to Z. The story from TIMETRAVEL (Qin et al., 2019a) consists of five sentences. Our approach alternates forward (left-to-right) and backward (rightto-left) passes that iteratively refine the generated texts w.r.t context from each side.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
inferring plausible but potentially defeasible conclusions from incomplete or hypothetical observations (Reiter, 1988). While humans are remarkably good at this type of causal reasoning, developing AI systems capable of nonmonotonic reasoning for a wide range of situations describable in natural language has been a major open research question.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
More concretely, with abductive reasoning, the goal is to find the most plausible explanation for incomplete observations (Peirce, 1960). In the top part of Figure 1, given the first observation that Ray is "making his daughter a swing" and the later observation that he "ran to [her] to make sure she was okay," we can hypothesize that she somehow got hurt by the swing.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
In contrast, counterfactual reasoning concerns the causal changes to future events given a change in the past condition (i.e., "counterfactual condition"; Goodman, 1947). For example, the bottom part of Figure 1 shows the original five sentence story (S 1 , ..., S 5 ) and an alternative counterfactual condition given in S 2 -that instead of being a generic "Halloween party", the new counterfactual condition is that it is going to be a "Game of Thrones themed party"! Given these, the problem we want to solve is to update the future events (S 3 , ..., S 5 ), so that instead of "Zeke dressed up as skeleton", we have "Zeke dressed up like a Stark". 2 Recently, two tasks and corresponding benchmarks have been introduced to tackle languagebased nonmonotonic reasoning: the ART dataset for abductive NLG , and the TIMETRAVEL dataset for counterfactual story rewriting (Qin et al., 2019a). Both tasks are framed as conditional generation, with multiple contexts to condition on. The currently dominant paradigm for conditional text generation tasks is fine-tuning pre-trained language models (LMs), such as GPT2 (Radford et al., 2019a), on large-scale training data for supervision. However, despite the large number of training examples, supervised approaches still perform considerably worse than humans and are subject to developing superficial strategies such as repeating the observations as is or memorizing prevalent surface patters specific in the dataset (Qin et al., 2019a). Furthermore, having to require largescale training data for each domain and task would be utterly inefficient for broad-coverage nonmonotonic reasoning in language.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
In this paper, we investigate an alternative path toward language-based nonmonotonic reasoning using pre-trained language models as is. Intuitively, both the abductive and counterfactual reasoning requires learning coherent patterns in narrative, which should be already available in large-scale pretrained language models. However, the key challenge is that most generative language models are trained to condition only on the left context, or to perform narrowly scoped text-infilling. This paper presents DELOREAN: DEcoding for nonmonotonic LOgical REAsoNing, an unsupervised decoding algorithm that only assumes off-theshelf left-to-right language models with no supervision. The key intuition of our algorithm is incorporating the future through back-propagation, during which, we only update the internal representation of the output while fixing the model parameters. More specifically, DELOREAN alternates between the forward and backward passes, where the forward pass performs left-to-right inference given the left context (roughly maximizing P (Y |X) in Figure 1), while the backward pass instills the right constraint through right-to-left backpropagation with a task-specific loss (roughly maximizing P (Z|XY )). The forward and backward outputs are mixed into a single vector, from which tokens are sampled to generate the desired output. To choose the best output across iterations, we employ an unsupervised ranking step based on BERT's next sentence prediction task to measure coherence (Devlin et al., 2018).
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
On both tasks, DELOREAN outperforms all other unsupervised methods in terms of both automatic metrics and human evaluation, demonstrating that nonmonotonic reasoning through conditional decoding is a promising research direction. Moreover, outputs produced by our model are judged as more coherent than those from the supervised models. In sum, our study shows that backpropagation-based decoding may enable additional future applications of unsupervised generation and reasoning.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Most NLP benchmarks have focused on reasoning about information that is entailed from the premise. For instance, natural language inference (NLI;Bowman et al., 2015) focuses primarily on whether a hypothesis is entailed from a given premise, which means the information stated in the hypothesis is a subset of the information provided in the premise. However, it has been noted that human reasoning is often the other way, where hypotheses often contain new information that was not available in the premise, but plausibly true (but  Figure 2: Illustration of the DELOREAN decoding procedure, using abductive reasoning as an example. At initialization (upper-left box), the language model (LM) initializes the logits
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Past context X Input: Ray hung a tire on a rope to make his daughter a swing. based on the future constraint Z (red box). The backward pass then performs back-propagation and produces the backward logits
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Past context X Input: Ray hung a tire on a rope to make his daughter a swing. 
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
In the subsequent forward pass, for each step n, we compute the forward logits
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Ray hung a tire on a rope to make his daughter a swing. 
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Ray hung a tire on a rope to make his daughter a swing.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Ray ran to his daughter to make sure she was okay.  
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Ray hung a tire on a rope to make his daughter a swing. possibly defeasible with new additional context) (Johnson-Laird, 2006;Mercier and Sperber, 2017). This type of reasoning corresponds to nonmonotonic reasoning (Kraus et al., 1990), as it contradicts the monotonicity property according to which valid arguments cannot be made invalid by adding premises. We study two tasks of that nature: abductive reasoning ( §2.1) and counterfactual reasoning ( §2.2).
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Abductive reasoning aims at finding the most likely explanation to partial observations (Peirce, 1960). It has a central role in the human ability to "read between the lines," and is crucial for language acquisition (Andersen, 1973), understanding sentences in discourse (Hobbs et al., 1993), and many more. Despite the importance, however, relatively little focus has been given to it in NLP research.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Recently,  propose the abductive reasoning task. Given two observations, the goal is to determine the most likely explanation of what happened in-between. The dataset introduced for the task, ART, consists of 20k observations derived from the first and last sentence of stories in the ROCStories dataset (Mostafazadeh et al., 2016a). We focus on the abductive NLG setup introduced in the paper, which is framed as a conditional generation task where a plausible explanation to the observations must be generated using language. The authors reported the performance of several pre-trained LM-based baselines and showed promises and limitations of such approaches.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Counterfactual reasoning aims at inferring alternative past events that could have happened given a certain change in conditions (Goodman, 1947;Starr, 2019). While counterfactual reasoning plays an important role in AI systems (Isard, 1974;Gins-berg, 1986), it requires causal reasoning abilities, which are arguably absent from current associationbased AI (Pearl and Mackenzie, 2018). While there has been work on counterfactual reasoning in NLP, including recognizing counterfactuals in text (Son et al., 2017), and improving the performance of NLP tasks using counterfactual learning (Lawrence et al., 2017;Lawrence and Riezler, 2018), it remains a major research challenge.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Recently, Qin et al. (2019a) introduce the task of counterfactual story generation. Given a 5-sentence original story, and an alternative context in which the second sentence of the story was altered by a counterfactual, the task is to generate a new 3sentence story ending that addresses the alternative beginning while minimally editing the original ending. The associated TIMETRAVEL dataset is based on fictional narratives from ROCStories, for which counterfactual contexts and alternative endings are crowdsourced, yielding 29,849 problem instances. Qin et al. (2019a) report several baseline performances, and find that models based on pre-trained LMs produce output that recognize the counterfactual, but generated endings which deviated considerably from the original storyline. In contrast, in the supervised setup, models optimize the easier of the two goals and generate endings that are overly similar to the original endings.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Humans make inferences based on available information and refine them when new information arrives. Since currently available pre-trained LMs generate text by sequentially predicting the next token from left to right, they are incapable of conditioning on future constraints. Therefore, we propose DELOREAN: an unsupervised backprop-based decoding algorithm, which is summarized in Algorithm 1, illustrated in Figure 2, and detailed below. DELOREAN intermittently refines the predictions to cohere with either the context or the constraints (Section 3.1). The candidate generations are then ranked by coherence (Section 3.2).
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Given context text X, the goal is to generate continuation text Y = (y 1 , . . . , y N ), such that Y satisfies certain constraints according to the reasoning tasks, usually defined based on another context Z (see Figure 1; we discuss the task-specific constraints in the respective task sections).
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Input: Pre-trained language model (LM) Context X Future constraint Z 1: Initialize logitsỸ (0) 2: Initialize Ys, list of candidate generations 3: for t ← 1 to T do 4: // Backward pass 5:
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
for n ← N to 1 do 6:
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Compute backward logitsỹ b n , Eq.(1) 7:
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
end for 8:
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
// Forward pass 9:
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
for n ← 1 to N do 10:
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Compute forward logitsỹ f n , Eq.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
(2) 11:
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Mix forward and backward logits, Eq.(3) 12: end for 13:
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Sample candidate Y from logitsỸ and add to Ys 14: end for 15: Rank Ys by coherence Output: The most coherent generated text Y from Ys
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
The proposed approach interleaves two procedures, namely, forward and backward, that produce and iteratively refine the generation, for a predefined number of iterations T . In particular, the forward pass ensures the generated text is a fluent continuation of the context X, while the backward pass informs the model about the constraint and steers the generation to satisfy it.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
As detailed below, the backward pass uses gradient descent to update the generation Y . However, Y is a discrete text that is not differentiable. Instead, throughout the algorithm, we maintain a soft representation of the sequenceỸ = (ỹ 1 , . . . ,ỹ N ), whereỹ n ∈ R V represents the logits of the n-th token and V is the vocabulary size. After the logits are refined over multiple iterations of the forward and backward passes, we generate discrete text at each step by sampling from y n ∼ softmax(ỹ n /τ ), where τ > 0 is the temperature.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
We start by initializing the logits before the first iteration,Ỹ (0) = (ỹ (0) 1 . . .ỹ (0) N ), by feeding the context X into the LM and greedily decoding N continuation tokens.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Backward The backward pass uses gradient backpropagation to update the generation with respect to the constraint. Specifically, we express the task-specific constraint as a loss function L(X,Ỹ (t−1) , Z) that evaluates how well the generation Y (approximated with the soft representatioñ Y ) obeys the constraint (see the subsequent sections for concrete instantiations of the loss). The goal of this pass is thus to minimize the loss w.r.t the generation. Specifically, at iteration t, for each step n in the generation, we update its logits with:
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
where ∇ỹ n L(X,Ỹ (t−1) , Z) is the gradient of the constraint-informed loss L w.r.t the n-th logits, and λ ∈ R is the step size. In practice, we may repeat the gradient updates multiple times in a single pass.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Forward The forward pass ensures that Y is fluent and coherent with the preceding context X. At iteration t, for a particular step n, we compute the forward logits with the LM:
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
1:n−1 ).
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
We then mix the nth-step forward and backward logits to get the final logits of iteration t:
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
where 0 < γ < 1 is the mixing weight. The resulting logitsỹ (t) n are then fed to the LM to compute the forward logits at the (n + 1)th step (Eq.2). This way, information from the backward pass is integrated into the left-to-right generation process to produce text that is informed by the constraint.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
We pre-define the number of tokens N required by the backward pass, but we allow the forward pass to generate more than N tokens if those are needed to obtain complete sentences. In that case, we set the logits of the extra tokens to the forward logits, without mixing:ỹ (t) n =ỹ (t),f n for n > N . We then prune any trailing tokens in the sampled text to get complete sentences.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
The output of the decoding step is a list of candidate generations for each iteration: Y s = {Y (t) |t = 1, ..., T }. We further use an unsupervised approach to rank and pick the best sample as the final output. Specifically, we take advantage of the BERT model, which was pre-trained with a next-sentence prediction (NSP) objective. Given two sentences A and B, we use NSP to compute the likelihood of B following A as a proxy for coherence:
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
where c(·, ·) denotes the coherence score. This score is used to evaluate the quality of a given candidate continuation Y by measuring (1) its compatibility with the subsequent text of the context X, (2) the internal consistency of Y if it consists of multiple sentences, and (3) the compatibility of Y with its right-side text when it is applicable.  Table 1: Automatic evaluation results on the abductive task, using the test set of ART.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Each instance in the ART dataset consists of two observations O 1 , O 2 and a hypothesis H that explains the two observations. These inputs naturally map to X, Z and Y in our framework. Formally, the abductive generation task aims to maximize P (Y |X, Z) -i.e. models must consider both left and right contexts (X and Z) jointly.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Constraints We maximize Z given XỸ by defining the loss function as the cross-entropy loss of generating Z given XỸ with the LM: 3 L(X,Ỹ , Z) := − N Z n=1 log P LM (z n |X,Ỹ , Z 1:n−1 ), (5) where P LM (a j |a 1:j−1 ) is the likelihood of generating token a j given the preceding text a 1:j−1 . Following the earlier study of the task (Bhagavatula et al., 2019), we also prepend Z to X to "leak" the future information to the LM. That is, we replace X with Z e X in the above equation, where e denotes a special end-of-text token. However, the comparisons with respective baselines below show the prepended Z is minor to the performance.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Ranking We rank candidates by the overall coherence after inserting Y in between X and Z: ranking score(Y ) = c(XY, Z) + c(X, Y Z). (6) Hyperparameters We use GPT2-345M (Radford et al., 2019b) as the pre-trained LM for all models. We use the ART development set to select hyperparameters. We use greedy decoding for our method and top k decoding (Fan et al., 2018) (k = 40, τ = 0.7) for our baselines. Other hyperparameters are outlined in Appendix A.1.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Ray was fine but his car was totaled.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
As he drives the car to the top of the mountain his car is hit by a car.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Peter was excited to go to the Sanders rally in New Hampshire.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
He couldn't wait to vote for him.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
He has a long history of supporting Bernie Sanders and was excited to see him in person. Figure 3: Examples of generated hypotheses on three abductive reasoning cases. Given observations O1 and O2, DELOREAN generates a hypothesis explaining the observations.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Baselines We compare our method against baselines from . The unsupervised baselines use a pre-trained GPT-2 model to generate Y given a prompt text-either the observation X alone (Zero-Shot X ) or Z e X (Zero-Shot ZX ). The supervised method (Sup) follows the same input format as Zero-Shot ZX , but finetunes GPT-2 on the ART training set. Finally, our knowledge-informed baseline (+COMET-Emb) further augments the representation of Sup with knowledge from COMET .
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
To separately study the contribution of our decoding strategy and ranking component, we also report the performance of ranking the baseline outputs. Specifically, we let each baseline generate 20 candidates and rank them by coherence (Eq. 6). 4
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Automatic Evaluation We report the same metrics as : BLEU-4 (Papineni et al., 2002), ROUGE-L (Lin, 2004) and BERTSCORE (Zhang et al., 2019) (with the bertbase-uncased model). The results in Table 1 show that DELOREAN performs best among the unsupervised systems across all metrics. We also note that our ranking step improves both the performance of our model and that of the zero-shot baselines.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Human Evaluation We conduct two sets of human evaluations on 100 test examples using crowdworkers from Amazon Mechanical Turk. In the scoring setting, presented in Table 2, workers were presented a pair of observations (X and Z) and a generated hypothesis Y , and asked to rate the 4 We tried ablating the ranking component from our method in preliminary experiments, and found that ranking is essential to obtaining good performance. By adding ranking to our baselines, we assess the contribution of our decoding strategy.   coherence of the hypothesis with respect to the observation X (X-Y ), the observation Z (Y -Z), and both (X-Y -Z), on a 4-point Likert scale. In the pairwise comparison setting, presented in Table 3, workers were presented the outputs from a pair of systems (DELOREAN and baseline) and asked to choose the better output in terms of the same coherence criteria. Each example was labeled by 3 workers. 5 In both evaluation setups, our method substantially outperform the unsupervised baselines, achieving a relative improvement of 36% − 215% with respect to Y -Z coherence. Our method also  outperform the supervised methods with respect to X-Y coherence (Table 2), and achieve competitive performance in the pairwise comparison (Table 3). Again, the ranking component contributes to increasing performance for the zero-shot baselines. Finally, the large performance gap between the methods and human-written explanations stresses the difficulty of this reasoning task and warrants future research. Figure 3 presents two example outputs produced by DELOREAN. We can see our approach generates reasonable hypotheses by taking into account both the past and future contexts. For instance, in the first example, the future observation (O2) "car was totaled" indicates that Ray had a car accident, which is correctly captured in the generated hypothesis "car is hit by a car".
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Given an original story ending Z of story context X ori , and a counterfactual condition X that changes X ori to invalidate Z (see Fig. 1), the task is to generate a new story ending Y that minimally edits the original ending Z to regain coherence with the counterfactual condition X (Qin et al., 2019a).
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Constraints The constraint we enforce is that Y is close to Z (i.e., minimal edits). We impose this constraint by minimizing their KL divergence:
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
where, with a slight abuse of notation, Z is the one-hot distribution of the tokens in the original ending. That is, we encourage the generated logits to recover the original ending.  Figure 4: Human calibration results for counterfactual generation in terms of weighted harmonic mean of coherence and min-edit, H β = (1+β 2 )·coherence·min edit β 2 ·coherence+min edit , as a function of the scaling factor β. Low β values assign more weight to coherence, and high β values emphasize more on min-edit.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Ranking We rank the candidates based on both their coherence with the context, as well as the internal coherence between the multiple sentences of each candidate (rewritten ending, consists of 3 sentences). More concretely, given a candidate Y , we compute the aggregated coherence score: Hyperparameters We largely follow the same setting as in the abductive reasoning task, but tune hyperparameters on the TIMETRAVEL development set. Deviations from these settings are outlined in Appendix A.2.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Baselines We compare our method with baselines from Qin et al. (2019a). The zero-shot baseline uses the pre-trained GPT-2 model to generate Y as a continuation to the counterfactual condition X. It is the most apt comparison to our method which also doesn't require additional supervision. We also experiment with two baselines that finetune GPT-2 on the original story X ori Z to fit the model to the story domain, either with an LM objective (FT) or a tailored conditional objective that encourages minimal edits of Z (Recon+CF). 6 Finally, we report the performance of a supervised that the price was $1,000. Kay was not happy with the price.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
She knew of a cool place online that did custom fits really cheaply, and ordered from there.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
They browsed shirts from a variety of stores. Tara picked out a floral patterned shirt that she liked best. Tara looked forward to wearing it.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
They sent her a shirt that fit her perfectly. Tara was so excited to wear it. She looked forward to wearing it.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Tara wanted to buy a new shirt for her upcoming school formal. She went to the mall with her mom.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Original Ending
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Shane enjoyed volunteering his time helping others.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
John was not allowed to be friends with Shane anymore. this bothered John greatly but his mom explained the reasons. She explained that Shane was a bad influence on John.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
John was a good student and was always looking for ways to help others. They were both very kind and caring people. Shane was a member of the Boy Scouts of America.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Shane and John were best friends at school. Shane was caught stealing and got suspended from school.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Original Ending Rewritten Ending Figure 5: Examples of generated story endings on three counterfactual reasoning cases. Given a story context, a counterfactual condition, and a original ending, DELOREAN generates a rewritten ending which is coherent with the counterfactual condition and is similar to the original ending.  baseline (Sup), in which GPT-2 is fine-tuned to produce the gold Y from X ori Z and X.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Automatic Evaluation Following Qin et al. (2019a), we report BERTSCORE (Zhang et al., 2019), which was shown to best correlate with human judges' notion of counterfactual coherence, and BLEU-4 and ROUGE-L, which better measure minimum-edits. We find that the discriminative baselines achieve the highest degree of plot fidelity. Meanwhile, DELOREAN achieves the highest BERTSCORE for counterfactual coherence.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Human Evaluation We repeat the human evaluation setup from Section 4.3. Presented with the original story, the counterfactual condition X, and the generated ending Y , workers were asked to judge (1) the coherence of Y with respect to the X; and (2) to what extent the generated ending minimally-edits the original ending. 7 In order to judge both criteria, we report the weighted harmonic mean H β of these scores across a range of weights β (Figure 4). Our results show that DELOREAN is the only model that maintains a consistent balance between coherence (1.66) and minimal edits (1.54). While the ranking-augmented zero-shot model produces the most coherent endings (coherence = 1.8), it deviates from the original ending. As β is increased (i.e., increasing importance of minimal edits), its weighted performance drops considerably, indicating it cannot generate new endings that follow the original plot of the story (min-edit = 1.25). Conversely, Recon+CF generates stories that are faithful to the original endings, but are far less coherent with the counterfactual condition (coherence = 1.23). Through human annotation, we found that Recon+CF copies the original ending word-forword in a 84% of cases.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
The pairwise comparison results in Table 5 parallel these observations. DELOREAN significantly outperforms the discriminative approaches (Recon+CF and Sup+Disc) in coherence, while falling short of the Zero-shot re-ranked baselines.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
In minimal edits, this pattern is flipped with our approach outperforming Zero-shot baselines considerably and losing to the discriminative baselines.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
Qualitative Analysis Figure 5 provides two example results for counterfactual story rewriting by DELOREAN. The approach successfully captures the causal relations between events and properly rewrites the endings with minimal edits. For instance, in the first example, given the counterfactual condition that "Tara ordered a shirt online" (as opposed to the original "went to mall"), the rewritten ending is about "sent shirt" to Tara (as opposed to the original "browsed from stores"). The last sentence of the original ending "She looked forward to wearing it" is correctly preserved as it is coherent with the counterfactual condition.
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning
We presented DELOREAN, an unsupervised LMbased approach to generate text conditioned on past context as well as future constraints, through forward and backward passes considering each condition. We demonstrated its effectiveness for abductive and counterfactual reasoning, on which it performed substantially better than unsupervised baselines. Our method is general and can be easily adapted for other generative reasoning tasks.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Commonsense knowledge consists of widely known facts that humans use to reason and react to everyday situations. Recently, empowering machines with such commonsense reasoning abilities has become an active research topic (Lin et al. 2019;Lv et al. 2020) and various commonsense question answering (CQA) benchmarks have been constructed (Zellers et al. 2018;Sap et al. 2019a;Zellers et al. 2019). Different from other types of QA tasks, CQA usually does not require domain-specific knowledge and sophisticated natural language understanding. Rather, it Copyright © 2022, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. 1 Code/dataset is available at https://github.com/PlusLabNLP/ zero shot cqa.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Swap the first two children of "SQ" Inverse wh-movement Replace wh-word with the mask token Figure 1: An example of natural-to-cloze translation with our syntactic-based method. 'SQ' is defined as the subconstituent of questions excluding wh-word or wh-phrase. relies on inference over implicit commonsense knowledge that is not given in the QA contexts.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
To tackle this problem, researchers have attempted to construct commonsense knowledge bases (Vrandečić and Krötzsch 2014;Speer, Chin, and Havasi 2017;Sap et al. 2019a), which can be integrated into downstream models (Bosselut, Le Bras, and Choi 2019). However, knowledge bases are often limited to a pre-defined set of relations and are expensive to construct. On the other hand, language models (LMs; e.g. Devlin et al. (2019); Liu et al. (2019); Lan et al. (2019)) pre-trained on large textual corpora are easy to extend to more data and allow users to query about an open class of relations. In addition, it has been demonstrated that pre-trained LMs contain a certain amount of world knowledge implicitly (Roberts, Raffel, and Shazeer 2020;Talmor et al. 2020) which can be extracted by having LMs fill in the blanks of carefully designed prompts (Petroni et al. 2019;Zhou et al. 2020;Jiang et al. 2020). However, these previous work only focuses on the settings where there is a fixed set of relations or output classes, thus knowledge can be induced by designing a limited amount of hand-crafted or automatically-generated rules. For example, to obtain a birthplace of one person X, we can just have an LM fill in the blank of 'X was born in '. In contrast, natural questions are much more flexible and it is non-trivial to design general rules to transform different natural questions into cloze forms. How to better solicit implicit knowledge in the pretrained LMs for CQA is an open question and no previous work has explored cloze translation for CQA to our knowledge.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
In this paper, we propose to better exploit the knowledge embedded in LMs for CQA by translating natural commonsense questions into "fill-in-the-blank" cloze sentences (see Figure 1 for an example). We investigate four translation methods, including 1) a syntactic-based model that performs a sequence of syntactic transformations on the source questions; 2) an unsupervised neural sequence-tosequence (seq2seq) model that does not require any naturalcloze question pairs inspired by Lewis, Denoyer, and Riedel (2019); 3) a supervised seq2seq model (Lewis et al. 2020) that is trained on our constructed dataset of natural-cloze question pairs; 4) a sequence tagging model (Omelianchuk et al. 2020) that performs operations such as word insertions and deletions on the source natural questions and transforms them into the target cloze questions. In addition, to combine the strengths of different translation models, we propose a consistency optimization objective which encourages the consistency between model predictions on the different translated cloze questions of the same instance using only unlabeled data.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
We mainly focus on the zero and few-shot settings as commonsense QA should be questions that any human can answer without specific training, so we want to equip models with similar ability. Moreover, these settings are robust measures of the models' general reasoning abilities (Ma et al. 2020). We experiment on three CQA datasets, including CommonsenseQA (Talmor et al. 2019), OpenbookQA (Mihaylov et al. 2018, and SocialIQA (Sap et al. 2019b). Results demonstrate that our cloze translation methods can achieve significant improvements on both zero and fewshot settings, and the consistency optimization objective can lead to further improvements. In addition, we show that our methods are complementary to a state-of-the-art knowledge base improved method and can bring extra gains. Analyses provide insights on distinct characteristics of the different cloze translation methods and why combining them can lead to greater improvements. Finally, we demonstrate that our methods can be beneficial in high-resource settings when the models are trained with both natural questions and our translated cloze-form queries.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
We first present four different cloze translation methods, discuss how we make use of the cloze questions, then illustrate how we combine them using consistency optimization on unlabeled data. 2
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
We investigate four methods for cloze translation:
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Syntactic-based Rewriting. Transforming natural questions to cloze questions can be understood as a series of syntactic transformation rules. While it can be nontrivial to design a perfect set of rules (Heilman and Smith 2010), here we adopt some simple heuristics and our general idea is shown in Figure 1. We use the constituency parser in (Joshi, Peters, and Hopkins 2018) to get the part-of-speech tags and syntactic structure of the input questions. The syntacticbased rewriting model does not require any training data, but it can be inflexible as it is hard to take all kinds of natural questions into consideration.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Specifically, we mainly consider two cases in this paper. First, if there exist nodes with the type 'SQ' in the input sentence, where 'SQ' is defined as the constituent of questions excluding wh-word or wh-phrase, we apply Algorithm 1 on the sentence. To illustrate, Algorithm 1 mainly swaps the first two children of the 'SQ' node, then performs an inverse wh-movement on the inputs, and finally replaces the whword with the mask tooken. Unsupervised Seq2Seq. Lewis, Denoyer, and Riedel (2019) have shown that we can perform unsupervised cloze translation by training neural seq2seq models with denoising auto-encoding and iterative back-translation objectives on unparallel natural and cloze question data. Their unsupervised cloze translation method (Lewis, Denoyer, and Riedel 2019) borrows some ideas from unsupervised neural machine translation methods (Lample et al. 2018a,b). Concretely, first, they create a cloze question corpus by masking noun phrases and named entities in statements sampled from Wikipedia, and a natural question corpus by mining questions containing some common wh-words from Common-Crawl. Then, they train both cloze-to-natural and naturalto-cloze translation methods with denoising auto-encoding and iterative back-translation objectives as in unsupervised machine translation. The denoising auto-encoding objective first masks some of the tokens in the source questions, and the model is trained to reconstruct the original questions. For  the iterative back-translation objective, a target-to-source model is first used to translate a target question into the source side, then a source-to-target model is trained to output the original target question given the translated source sentence, and this process will be repeated in both directions iteratively. Their model architecture uses a 4-layer Transformer (Vaswani et al. 2017) encoder and a 4-layer Transformer decoder.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
While Lewis, Denoyer, and Riedel (2019) mainly focus on cloze-to-natural translation and using it to perform data augmentation for question answering, a by-product of their method is a natural-to-cloze translation model, and here we directly use their pre-trained model. 3 The unsupervised model is much more flexible than the syntactic-based rewriting one, but the translated questions can be deviated from the original inputs due to the lack of supervision signals and the uncontrollable nature of seq2seq models.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Supervised Seq2Seq. To provide models with supervisions, we manually create a dataset of natural-cloze question pairs. Concretely, we manually translate all the natural questions in the original CommonsenseQA training and development data into cloze questions. We create a (8,500/1,221/1,241) split as in our main experiments. It takes a person around 40 hours to construct such a dataset. We sample several examples from the dataset as shown in Table 1. We can see that there exist different kinds of transformation rules and previous methods on designing the prompts for pre-trained language models cannot be applied in commonsense question answering.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
We fine-tune BART-Large (Lewis et al. 2020), a representative seq2seq model, on the dataset and perform naturalto-cloze translation. The inputs and outputs are the natural and cloze questions respectively. The model is trained with maximum likelihood estimation objective and we employ beam search during decoding. We choose BART-Large as our supervised seq2seq model because it is widely used in text generation tasks such as text summarization. BART is based on the Transformer model (Vaswani et al. 2017) and is pre-trained by corrupting documents and then optimizing a reconstruction loss. Its architecture consists of 12 Transformer encoding and decoding layers.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Supervised Sequence Tagging. While seq2seq models have been the de facto choice for many sequence generation tasks, cloze translation mainly involves word movements, deletions, and insertions which do not require a whole rewriting. Therefore, we also formulate cloze translation as a sequence tagging problem. A tagging model identifies which words need to be changed and modifies them with predefined word-level transformations (e.g. keep, delete, append), which may generate more faithful cloze questions than seq2seq models. The sequence tagging task has been widely investigated in the task of grammatical error correction and here we train GECToR (Omelianchuk et al. 2020), a popular model in grammatical error correction, on our constructed dataset.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
The GECToR model (Omelianchuk et al. 2020) employs a Transformer encoder and its parameters are initialized with RoBERTa. They have pre-defined token-level transformations. For example, given a source and target sentence 'A ten years old boy go school' and 'A ten-year-old boy goes to school.', it first pre-processes the pair to convert it to a sequence of transformation rules. Concretely, first, they map each token from the source sentence to subsequence of tokens from the target sentence: Because there is only a single tag for each token, this method is not suitable for all the situations. To solve the problem, they propose to process the pairs iteratively and at each step there is one single tag for each token.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
We can see that after the pre-processing, sequence generation is turned to a sequence classification task. Therefore, we can encode the entire input sentence using its encoder and feed the encoded representations to a classifier. The classifier will decide which transformation rule to apply for each token. We refer readers to their codebase 4 for more details.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Once we have the cloze question x for a natural question q, we can replace the mask token in x with each of the candidate answers in {a i }, and feed each replaced sentence r i = r i1 , · · · , r in to a pre-trained LM. The pre-trained LM can assign a score for r i with
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
and r i with the highest score is treated as our prediction. 5 We can apply softmax on the resulting scores {s i } to get the prediction probabilities.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
The above methods can generate different cloze questions {x j } for q. Instead of picking a single one for LMs, we propose a consistency optimization objective to combine them with unlabeled data. Inspired by work on multi-view learning (Wang, Ruder, and Neubig 2021), we encourage the consistency between predictions on different cloze translations for the same question. For each training instance, we first ensemble the model predictions: formally, given each cloze question x j and its candidate answers {a i }, we obtain the score {s ij } for {a i } as in the previous section, then apply softmax on {s ij } to get the prediction probabilities {p ij }; then, we sum the probabilities for each answer from different translation as p i = j p ij , and take the answer with the highest probability as the ensemble prediction a * .
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
We then treat a * as the pseudo-label to supervise the LM in a self-training manner. Concretely, we fine-tune the LM to maximize the probability of x j with its mask token replaced with a * while minimizing the probability of all the other candidate answers in {a i } with the cross-entropy loss.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Note that the consistency optimization objective does not need any gold labels, making it possible to be integrated into the zero-shot and few-shot settings.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
We experiment on CommonsenseQA (Talmor et al. 2019), OpenbookQA (Mihaylov et al. 2018), and So-cialIQA (Sap et al. 2019b). Because the standard test set labels of CommonsenseQA are unavailable, we create a (8,500/1,221/1,241) split for (train/dev/test) following Lin et al. (2019); Wang et al. (2020). We use the standard splits for the other datasets. We compare with two methods that do not use any knowledge base, including 1) natural questions ('Base'), which are directly concatenated with each answer choice and then fed to pre-trained LMs; 2) self-talk (Shwartz et al. 2020), which gets additional background for commonsense questions by querying LMs. We also report the results of Ma et al. (2020), which is a state-of-the-art zeroshot CQA model using knowledge bases to construct CQA datasets automatically. We use ALBERT-xxlarge-v2 (Lan et al. 2019) as the base LM. We will illustrate the details of our experimental setup in the following paragraphs.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Datasets. For the CommonsenseQA dataset (Talmor et al. 2019), because its test set is not publicly available, the predictions for it can only be evaluated once every two weeks via the official leaderboard. Therefore, following previous work (Lin et al. 2019;Wang et al. 2020), we separate the training data into training and test sets consisting of 8,500 and 1,241 instances respectively. We use the standard development set consisting of 1,221 instances. The Open-bookQA (Mihaylov et al. 2018) dataset consists of 5,957 multiple-choice questions with 4,957 training, 500 development, 500 testing instances. While it provides a small "book" of 1,326 core science facts, we do not include this additional information because our focus is on the implicitly learned knowledge in pre-trained language models. The SocialIQA (Sap et al. 2019b) dataset contains 33,410 training, 1,954 development, 2,224 testing instances, the aim of which is to probe the emotional and social intelligence of models in a variety of everyday situations.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Cloze Translation Methods. For the unsupervised cloze translation method, we use the pre-trained model (sentence cloze boundaries, named entity answers) provided by Lewis, Denoyer, and Riedel (2019). For the seq2seq model, we follow the setting in text summarization on XSUM and finetune the BART-Large model on the training set of our cloze data for 15k steps with a batch size of 16,384 tokens and a learning rate of 3e-5. For the sequence tagging model, we fine-tune the RoBERTa-based GECToR model on our cloze translation data with default parameters. 6 We select the models that achieve the best BLEU scores on the development set for cloze translation.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Consistency Optimization. For the consistency optimization objective, we use the training data of each dataset without using their labels. We encourage the model prediction consistency on the data generated by syntacticbased rewriting, supervised seq2seq, and supervised sequence tagging model. The models are trained with a learning rate of 1e-5 for 2k/1k/2k steps for Common-senseQA/OpenbookQA/SocialIQA respectively.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Zero-shot Settings. In the zero-shot settings, for the baseline ALBERT-xxlarge-v2 model, we directly concatenate the questions and answers together, and feed the concatenated sentences to the model to get the language modeling scores. For the self-talk baseline, we try both GPT2-Large and ALBERT-xxlarge-v2 for querying the external contexts and getting the language modeling scores using the default parameters. For Ma et al. (2020), we use their constructed CWWV data that utilizes three knowledge bases: Concept-Net, WordNet, and Wikidata, then we train both ALBERTxxlarge-v2 and RoBERTa-Large on the CWWV data with their default parameters.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Few-shot Settings. For the few-shot settings, we randomly sample 16/32/64/128 datapoints from the training data and fine-tune ALBERT-xxlarge-v2 on both the natural   Table 3: Accuracy (%) in zero-shot settings. '*' indicates that we run the experiments three times with different random seeds. For self-talk and Ma et al. (2020), we try both ALBERT and the best LMs used in their papers. The best scores are in bold.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
and cloze translated data. Similar to the consistency optimization objective, a question is concatenated with each answer choice and then fed to the model to get its corresponding score. The scores for all the choices are then normalized using softmax to get the prediction probability for the choices, then we use the cross-entropy loss on the prediction probability to train the model. The learning rate is set to 1e-5 and the number of epoch is set to 10 or 20, selected on the development sets. We run the experiments three times with different random seeds for each setting.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
In this section, we will present the main results of our methods in both zero-and few-shot settings.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
We first compare with methods that utilize knowledge base. Table 3 shows that cloze translation can generally improve the zero-shot performance of ALBERT significantly across settings (one exception is unsupervised seq2seq on SocialIQA), demonstrating that the knowledge in LMs can indeed be better extracted with cloze questions. Unsupervised seq2seq is the least effective translation method, potentially due to its lack of supervisions. Our models cannot outperform self-talk on So-cialIQA, possibly because self-talk manually designs taskspecific question transformation rules for SocialIQA, which inserts strong supervision into their models.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
We then combine all the translation methods except unsupervised seq2seq. We find that directly ensembling them as in Section cannot always lead to improvements, but consistency optimization can improve the performance by a large margin across settings. This demonstrates that it is highly nontrivial to combine the strengths of different translation methods and our designed objective is one very effective way of combining them. Note that after using the consistency optimization objective, our method can even achieve comparable performance with Ma et al. (2020) that leverages external knowledge base on two datasets.   
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
We also try to compare our models with a method that uses external knowledge base and achieves state-of-the-art performance in zero-shot settings. While our methods cannot always outperform Ma et al. (2020), which is intuitive considering that our model is given less information, we can combine Ma et al. (2020) with our method. Specifically, because Ma et al. (2020) mainly use knowledge base to construct datasets, we can first fine-tune LMs on their constructed data and then use our cloze translation and consistency optimization methods on the commonsense question answering datasets. As we can see from Table 3, this combination strategy can lead to the best performance across datasets, indicating the complementarity of the two methods.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Few-shot Evaluations. We also experiment in few-shot settings where only 16/32/64/128 instances are available. In this part, we mainly compare the baseline with the syntacticbased translation because 1) it does not need any supervisions; 2) it works well in zero-shot settings as shown in Table 3. As illustrated in Figure 2, our syntactic-based translation method consistently outperforms the baseline which is trained on natural questions and consistency training can also be helpful in these settings. Also, it is interesting to note that zero-shot performance of cloze translation is better than supervised models trained with 100 natural questions.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
We then perform several analyses on our methods.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Translation Errors. We first try to analyze the errors of different cloze translation methods. To this end, we randomly sample 100 questions from the CommonsenseQA dataset and perform human evaluation on the translation errors as in Table 2 and 4. As we can see from the tables, the unsupervised seq2seq method is the least effective one as it can often generate meaningless questions, which can explain why it performs in worst in the main experiment section (Table 3. The syntactic-based method, on the other hand, is inflexible, thus it can generate unnatural sentences or put [MASK] at the wrong place when dealing with complex syntactic structures. In the example in Table 2, we can see that it can generate the sentence "the island country is a popular ferret in [MASK]" which is hard to parse and quite unnatural.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
The supervised methods are more much flexible. However, the supervised seq2seq method can sometimes generate unfaithful outputs. Interestingly, in Table 2, it replaces the person name 'Blue' in the original question with 'James', possibly because 'James' appears more frequently in the training data. Note that even though the output is unfaithful, it does not affect the correct answer choice. For the supervised tagging model, because it mainly deals with word deletions and insertions, sometimes over-deletions or insertions may occur, resulting in unfaithful or unnatural sentences. But it should be noted that the supervised tagging method can generate more faithful outputs than the seq2seq method, confirming our previous hypothesis.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
We can see that different cloze translation methods have rather distinct characteristics, indicating that the translation outputs can be rather diverse, which can be the reason why our consistency optimization objective can greatly improve the model performance across settings as in Table 3.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
High-resource Settings. We also test the model performance on natural and cloze questions in high-resource settings where all the labeled training data are used. In the high-resource settings, we fine-tune ALBERT-xxlarge-v2 on both the natural and cloze translated data with all the training data. The models are trained with a learning rate of 1e-5 for 2k/1k/2k steps for Common-senseQA/OpenbookQA/SocialIQA. For the ensemble, we apply softmax on the prediction scores for each model, and add the prediction probabilities together. We try to ensemble 3 models trained on natural data, 3 models trained on cloze data, and 2 models trained on natural and cloze data respectively. As in Figure 3, our model ('1 Rule') cannot always outperform the baseline ('1 Base'). We hypothesize that this is due to the translation errors as we have analyzed in the previous part. Concretely, the translation errors can alter the meaning of the original questions, and training on these noisy data can lead to degraded performance. However, because of the diversity among different translation methods, we can ensemble each of the models trained on different data ('1 Base + 1 Rule'), which is better than ensembling 3 models trained on the same data with different random seeds ('3 Base' and '3 Rule').
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Applicability to Other LMs. All of the previous experimental results are obtained using the ALBERT model. In this part, we also test if our methods are applicable to other LMs as well. As shown in Figure 4, our methods can improve all the pre-trained LMs. Also, we can see that our model has a tendency to favor bidirectional LMs (e.g. AL-BERT and RoBERTa) than uni-directional models. One possible reason is that when using cloze translation, the LM  probability of all the words will be affected in bidirectional LMs, while only the probability of words appearing after [MASK] are changed for unidirectional LMs. For example, cloze translation improves GPT2 the least possibly because it is unidirectional and only the probability of words succeeding [MASK] are affected. Similarly, consistency training improves T5 marginally because it is a sequence-tosequence model and the target side are the label words. Therefore, only the label word probabilities will be affected which can limit the model performance.  (Bauer, Wang, and Bansal 2018;Lin et al. 2019;Xu et al. 2021;Lv et al. 2020;Wang et al. 2020). For example, Ma et al. (2020) use knowledge bases to automatically construct data for commonsense question answering and train the models on their constructed datasets. Xu et al. (2021) try to fuse knowledge into models by having the models being able to attend to knowledge bases such as ConceptNet. Different from these methods, we focus on better utilizing the knowledge embedded in pre-trained LMs.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
We aim to better utilize the implicitly learned knowledge in pre-trained LMs for commonsense question answering by natural-to-cloze question translation. To this end, we construct a dataset of natural-question pairs and investigate four translation methods. In addition, we demonstrate that different translation methods have distinct characteristics and we propose a consistency optimization objective to combine the strengths of different translations using unlabeled data. We demonstrate the effectiveness of our methods in zero and few-shot settings and show that our methods are complementary to a state-of-the-art knowledge base method.
Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
In the future, we can investigate more cloze translation methods and develop better ways of utilizing the translated questions. Also, so far we only perform experiments on English datasets and our syntactic-based method is specifically designed for English questions, thus it can be interesting to see if our methods can generalize to other languages.