How has clarification question generation been studied in the NLP community?
Question Generation Clarification question gen-
eration has been studied for various domains from
image recognition questions to product description
questions (Rao and Daumé III, 2018; Majumder
et al., 2021; White et al., 2021), defining the good-
ness of clarification questions along the lines of in-
formation theoretic measures such as relevance, in-
formativeness or utility (Rao and Daumé III, 2018;
White et al., 2021; Warstadt and Agha, to appear;
Rao and Daumé III, 2018, 2019). Most of ex-
isting works focus on questions that lead to sin-
gle true answer, whereas we focus on generating
clarification questions based on social situations,
defining the relevance and utility of a question in
terms of defeasibility. Additionally, we offer a
high-quality clarification question dataset for social
and moral situation—comprising of more than 30K
questions—that breaks the mold from the domain-
specificity of previous clarification datasets (Kumar
and Black, 2020; Aliannejadi et al., 2019). 
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
The ability to generate clarification questions i.e., questions that identify useful missing information in a given context, is important in reducing ambiguity. Humans use previous experience with similar contexts to form a global view and compare it to the given context to ascertain what is missing and what is useful in the context. Inspired by this, we propose a model for clarification question generation where we first identify what is missing by taking a difference between the global and the local view and then train a model to identify what is useful and generate a question about it. Our model outperforms several baselines as judged by both automatic metrics and humans.
[{"first":"Bodhisattwa Prasad","last":"Majumder","middle":[]},{"first":"Sudha","last":"Rao","middle":[]},{"first":"Michel","last":"Galley","middle":[]},{"first":"Julian","last":"McAuley","middle":[]}]
Open-domain clarification question generation without question examples
An overarching goal of natural language processing is to enable machines to communicate seamlessly with humans. However, natural language can be ambiguous or unclear. In cases of uncertainty, humans engage in an interactive process known as repair: asking questions and seeking clarification until their uncertainty is resolved. We propose a framework for building a visually grounded question-asking model capable of producing polar (yes-no) clarification questions to resolve misunderstandings in dialogue. Our model uses an expected information gain objective to derive informative questions from an off-the-shelf image captioner without requiring any supervised question-answer data. We demonstrate our model’s ability to pose questions that improve communicative success in a goal-oriented 20 questions game with synthetic and human answerers.
[{"first":"Julia","last":"White","middle":[]},{"first":"Gabriel","last":"Poesia","middle":[]},{"first":"Robert","last":"Hawkins","middle":[]},{"first":"Dorsa","last":"Sadigh","middle":[]},{"first":"Noah","last":"Goodman","middle":[]}]
Answer-based Adversarial Training for Generating Clarification Questions
We present an approach for generating clarification questions with the goal of eliciting new information that would make the given textual context more complete. We propose that modeling hypothetical answers (to clarification questions) as latent variables can guide our approach into generating more useful clarification questions. We develop a Generative Adversarial Network (GAN) where the generator is a sequence-to-sequence model and the discriminator is a utility function that models the value of updating the context with the answer to the clarification question. We evaluate on two datasets, using both automatic metrics and human judgments of usefulness, specificity and relevance, showing that our approach outperforms both a retrieval-based model and ablations that exclude the utility model and the adversarial training.
[{"first":"Sudha","last":"Rao","middle":[]},{"first":"Hal","last":"Daumé III","middle":[]}]
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Inquiry is fundamental to communication, and machines cannot effectively collaborate with humans unless they can ask questions. In this work, we build a neural network model for the task of ranking clarification questions. Our model is inspired by the idea of expected value of perfect information: a good question is one whose expected answer will be useful. We study this problem using data from StackExchange, a plentiful online resource in which people routinely ask clarifying questions to posts so that they can better offer assistance to the original poster. We create a dataset of clarification questions consisting of 77K posts paired with a clarification question (and answer) from three domains of StackExchange: askubuntu, unix and superuser. We evaluate our model on 500 samples of this dataset against expert human judgments and demonstrate significant improvements over controlled baselines.
[{"first":"Sudha","last":"Rao","middle":[]},{"first":"Hal","last":"Daumé III","middle":[]}]
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Context is everything, even in commonsense moral reasoning. Changing contexts can flip the moral judgment of an action; Lying to a friend is wrong in general, but may be morally acceptable if it is intended to protect their life.We present ClarifyDelphi, an interactive system that learns to ask clarification questions (e.g., why did you lie to your friend?) in order to elicit additional salient contexts of a social or moral situation. We posit that questions whose potential answers lead to diverging moral judgments are the most informative. Thus, we propose a reinforcement learning framework with a defeasibility reward that aims to maximize the divergence between moral judgments of hypothetical answers to a question. Human evaluation demonstrates that our system generates more relevant, informative and defeasible questions compared to competitive baselines. Our work is ultimately inspired by studies in cognitive science that have investigated the flexibility in moral cognition (i.e., the diverse contexts in which moral rules can be bent), and we hope that research in this direction can assist both cognitive and computational investigations of moral judgments.
[{"first":"Valentina","last":"Pyatkin","middle":[]},{"first":"Jena D.","last":"Hwang","middle":[]},{"first":"Vivek","last":"Srikumar","middle":[]},{"first":"Ximing","last":"Lu","middle":[]},{"first":"Liwei","last":"Jiang","middle":[]},{"first":"Yejin","last":"Choi","middle":[]},{"first":"Chandra","last":"Bhagavatula","middle":[]}]
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
An important but under-explored aspect of text understanding is the identification of missing information in a given context i.e., information that is essential to accomplish an underlying goal but is currently missing from the text. Identifying such missing information can help to reduce ambiguity in a given context which can aid machine learning models in prediction and generation (De Boni and Manandhar, 2003;Stoyanchev et al., 2014). Daumé III (2018, 2019) recently proposed the task of clarification question generation as a way to identify such missing information in context. They propose a model for this task which while successful at generating fluent and relevant questions, still falls short in terms of usefulness and identifying missing information. With the advent of large-scale pretrained generative models (Radford et al., 2019;Lewis et al., 2019;Raffel et al., 2019), generating fluent and coherent text is within reach. However, generating clarification questions requires going beyond fluency and relevance. Doing so requires understanding what is missing, which if included could be useful to the consumer of the information. †   Humans are naturally good at identifying missing information in a given context. They possibly make use of global knowledge i.e., recollecting previous similar contexts and comparing them to the current one to ascertain what information is missing and if added would be the most useful. Inspired by this, we propose a two-stage framework for the task of clarification question generation. Our model hinges on the concept of a "schema" which we define as the key pieces of information in a text. In the first stage, we find what's missing by taking a difference between the global knowledge's schema and schema of the local context ( §3.1). In the second stage we feed this missing schema to a fine-tuned BART (Lewis et al., 2019) model to generate a question which is further made more useful using PPLM (Dathathri et al., 2019) ( §3.2). 1 We test our proposed model on two scenarios ( §2): community-QA, where the context is a product description from amazon.com (McAuley and Yang, 2016) (see e.g. Table 1); and dialog where the context is a dialog history from the Ubuntu Chat forum (Lowe et al., 2015). We compare our model to several baselines ( §4.2) and evaluate outputs using both automatic metrics and human evaluation to show that our model significantly outperforms baselines in generating useful questions that identify missing information in a given context ( §4.4). BART-decoder (fine-tuned) Figure 1: Test-time behaviour of our proposed model for useful clarification question generation based on missing information in a Community-QA (amazon.com) setup. 1. We obtain a local schema from the available context for a product: description and previously asked questions. 2. We obtain the global schema of the category of the product. 3. We estimate the missing schema that is likely to guide clarification question generation. 4. A BART model fine-tuned on (missing schema, question) pairs to generate a question ("Is it true leather?"). 5. A PPLM model with usefulness classifier as its attribute model further tunes the generated question to make it more useful ("Is there room to also carry a cooling pad?").
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Furthermore, our analysis reveals reasoning behind generated questions as well as robustness of our model to available contextual information. ( §5).
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Rao and Daumé III (2018) define the task of clarification question generation as: given a context, generate a question that identifies missing information in the context. We consider two scenarios:
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Community-QA Community-driven questionanswering has become a common venue for crowdsourcing answers. These forums often have some initial context on which people ask clarification questions. We consider the Amazon questionanswer dataset (McAuley and Yang, 2016) where context is a product description and the task is to generate a clarification question that helps a potential buyer better understand the product.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Goal Oriented Dialog With the advent of high quality speech recognition and text generation systems, we are increasingly using dialog as a mode to interact with devices (Clark et al., 2019). However, these dialog systems still struggle when faced with ambiguity and could greatly benefit from having the ability to ask clarification questions. We explore such a goal-oriented dialog scenario using the Ubuntu Dialog Corpus (Lowe et al., 2015) consisting of dialogs between a person facing a technical issue and another person helping them re-solve the issue. Given a context i.e a dialog history, the task is to generate a clarification question that would aid the resolution of the technical issue.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
3 Approach Figure 1 depicts our approach at a high level. We propose a two-stage approach for the task of clarification question generation. In the first stage, we identify the missing information in a given context. For this, we first group together all similar contexts in our data 2 to form the global schema for each high-level class. Next, we extract the schema of the given context to form the local schema. Finally, we take a difference between the local schema and the global schema (of the class to which the context belongs) to identify the missing schema for the given context. In the second stage, we train a model to generate a question about the most useful information in the missing schema. For this, we fine-tune a BART model (Lewis et al., 2019) on (missing schema, question) pairs and at test time, we use PPLM (Dathathri et al., 2019) with a usefulness classifier as the attribute model to generate a useful question about missing information.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Motivated by (Khashabi et al., 2017) who use essential terms from a question to improve performance of a Question-Answering system, we see the need of identifying important elements in a context to ask a better question. We define schema of sentence s as set consisting of one or more triples of the form (key-phrase, verb, relation) and/or one or more key-phrases.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
(1) Schema Extraction Our goal is to extract a schema from a given context. We consider (keyphrase, action verb, relation) as the basic element of our schema. Such triples have been found to be representative of key information in previous work (Vedula et al., 2019). Given a sentence from the context, we first extract bigram and unigram key-phrases using YAKE (Yet-Another-Keyword-Extractor) (Campos et al., 2020) and retain only those that contain at least a noun. We then obtain the dependency parse tree (Qi et al., 2020b) of the sentence and map the key-phrases to tree nodes. 3 Now, to obtain the required triple, we need to associate a verb and a relation to each key-phrase. This procedure is described in Alg 1. At a high-level, we use the path between the key-phrase and the closest verb in the dependency tree to establish a relation between the key-phrase and the verb. In cases where there is no path, we use only the keyphrase as our schema element. Figure 2 shows an example dependency tree for a sentence. Creating local schema Given a context, we extract a schema for each sentence in the context. The local schema of a context c is a union of schemata of each sentence s in the context.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
3 In the case of bigram phrases, we merge the tree nodes. 
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
We define global schema at the class level where a 'class' is a group of similar contexts. For Amazon, classes consist of groups of similar products and for Ubuntu, classes consist of groups of similar dialogs (see §4.1 for details). The global schema of a class K is a union of local schemata of all contexts c belonging to K.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
A naive union of all local schemata can result in a global schema that has a long tail of low-frequency schema elements. Moreover, it may have redundancy where schema elements with similar meaning are expressed differently (e.g. OS and operating system). We therefore use word embedding based similarity to group together similar key-phrases and retain only the most frequent elements (see appendix). Creating a missing schema Given a context c, we first determine the class K to which the context belongs. We then compute its missing schema by taking the set difference between the global schema of class K and the local schema of the context c:
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
More specifically, we start with the elements in the global schema and remove elements that have a semantic match (see appendix) with any element in the local schema to obtain the missing schema.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Our goal is to generate a useful question about missing information. In §3.1, we explained how we compute the missing schema for a given context; here we describe how we train a model to generate a useful question given the missing schema.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
BART-based generation model Our generation model is based on the BART (Lewis et al., 2019) encoder-decoder model, which is also a state-ofthe-art model in various generation tasks including dialog generation and summarization. We start with the pretrained base BART model consisting of a six layer encoder and six layer decoder. We finetune this model on our data where the inputs are the missing schema and the output is the question. The elements of the missing schema in the input are separated by a special [SEP] token. Since the elements in our input do not have any order, we use the same positional encoding for all input positions. We use a token type embedding layer with three types of tokens: key-phrases, verbs, and relations.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
PPLM-based decoder We observed during our human evaluation 4 that a BART model fine-tuned in this manner, in spite of generating questions that ask about missing information, does not always generate useful questions. We therefore propose to integrate the usefulness criteria into our generation model. We use the Plug-and-Play-Language-Model (PPLM) (Dathathri et al., 2019) during decoding (at test time). The attribute model of the PPLM in our case is a usefulness classifier trained on bags-of-words of questions. In order to train such a classifier, we need usefulness annotations on a set of questions. For the Amazon dataset, we collect usefulness scores (0 or 1) on 5000 questions using human annotation whereas for the Ubuntu dataset we assume positive labels for (true context, question) pairs and negative labels for (random context, question) pairs and use 5000 such pairs to train the usefulness classifier. Details of negative sampling for Ubuntu dataset is in Appendix.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
We aim to answer the following research questions (RQ):
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
1. Is the model that uses missing schema better at identifying missing information compared to models that use the context directly to generate questions? 4 See results of BART+missinfo in Table 5 Train  2. Do large-scale pretrained models help generate better questions? 3. Does the PPLM-based decoder help increase the usefulness of the generated questions?
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Amazon The Amazon review dataset (McAuley et al., 2015) consists of descriptions of products on amazon.com and the Amazon question-answering dataset (McAuley and Yang, 2016) consists of questions (and answers) asked about products. Given a product description and N questions asked about the product, we create N instances of (context, question) pairs where context consists of the description and previously asked questions (if any). We use the "Electronics" category consisting of 23,686 products. We split this into train, validation and test sets ( Table 2). The references for each context are all the questions (average=6) asked about the product. A class is defined as a group of products within a subcategory (e.g. DSLR Camera) as defined in the dataset. We restrict a class to have at most 400 products, and a bigger subcategory is broken into lower-level subcategories (based on the product hierarchy) resulting in 203 classes. While creating global schema, we exclude target questions from validation and test examples. The product descriptions and associated metadata come as inputs during test time. Hence, including them from all splits while creating the global schema does not expose the test and validation targets to the model during training. Ubuntu The Ubuntu dialog corpus (Lowe et al., 2015) consists of utterances of dialog between two users on the Ubuntu chat forum. Given a dialog, we identify utterances that end with a question mark. We then create data instances of (context, question) where the question is the utterance ending with a question mark and the context consists of all utterances before the question. We consider only those contexts that have at least five utterances and at most ten utterances. Table 2 shows the number of data instances in the train, validation and test splits. Unlike the Amazon dataset, each context has only one reference question. A class is defined as a group of dialogs that address similar topics. Since such class information is not present in the dataset, we use k-means to cluster dialogs into subsequent classes. Each dialog was represented using a TF-IDF vector. After tuning the number of clusters based on sum of squared distances of dialogs to their closest cluster center, we obtain 26 classes.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
We follow a similar scheme as with Amazon for not including target questions from validation and test sets while building the global schema.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Retrieval We retrieve the question from the train set whose schema overlaps most with the missing schema of the given context. 
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Similar to Rao and Daumé III (2019), we conduct a human evaluation on Amazon Mechanical Turk to evaluate model generation on the four criteria below. Each generated output is shown with the context and is evaluated by three annotators.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Relevance We ask "Is the question relevant to the context?" and let annotators choose between Yes (1) and No (0). Fluency We ask "Is the question grammatically well-formed i.e. a fluent English sentence?" and let annotators choose between Yes (1) and No (0). Missing Information We ask "Does the question ask for new information currently not included in the context?" and let annotators choose between Yes (1) and No (0). Usefulness We perform a comparative study where we show annotators two model-generated questions (in a random order) along with the context. For Amazon, we ask "Choose which of the two questions is more useful to a potential buyer of the product". For Ubuntu, we ask "Choose which of the two questions is more useful to the other person in the dialog".
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Amazon   produces the most diverse questions (as also observed by Rao and Daumé III (2019)) since it selects among human written questions which tend to be more diverse compared to model generated ones. Among other baselines, transformer interestingly has the lowest diversity whereas GAN-Utility and BART come very close to each other. Model ablations that use missing schema produce more diverse questions further strengthening the importance of training on missing schema. Our model i.e., BART+missinfo+PPLM, in spite of outperforming all baselines (except retrieval), is still far from reference questions in terms of diversity, suggesting room for improvement.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Ubuntu Table 4 shows the results of automatic metrics on Ubuntu. 6 The overall BLEU-4 and ME-TEOR scores are much lower compared to Amazon since Ubuntu has only one reference per context. Under BLEU-4 and METEOR scores, similar to Amazon, we find that the retrieval baseline has the lowest scores. Transformer baseline outperforms the retrieval baseline but lags behind BART, again showing the importance of large-scale pretraining. The difference between the BLEU-4 scores of BART+missinfo and our final proposed model is not significant but their METEOR score difference is significant suggesting that our model produces questions that may be lexically different from references but have more semantic overlap with the reference set. Under Distinct-2 scores, we find the same trend as in Amazon, with the retrieval model being the most diverse and our final model outperforming all other baselines.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Amazon Table 5 shows the human judgment results on model generations for 300 randomly 6 We do not experiment with the GAN-Utility model (since it requires "answers") and the BART+missinfo+WD model (since usefulness labels are not obtained from humans). sampled product descriptions from the Amazon test set. Under relevancy and fluency, all models score reasonably with our proposed model producing the most relevant and fluent questions. Under missing information, the BART model, finetuned on context instead of missing schema, has the lowest score. GAN-Utility outperforms BART but significantly lags behind BART+missinfo and BART+missinfo+PPLM reaffirming our finding from the automatic metric results that our idea of feeding missing schema to a learning model helps.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
We additionally observe that the human-written questions score lower than model-generated questions under 'fluency' and 'missing information' criteria, mirroring similar observations from Daumé III, 2018, 2019). We believe the reason for this is that human-written questions often have typos or are written by non-native speakers (leading to lower fluency). Moreover, humans may miss out on reading full product descriptions causing them to ask about details that are already included in the description (leading to lower missing information scores). Figure 3a shows the results of pairwise comparison on the usefulness criteria. We find that our model wins over GAN-Utility by a significant margin with humans preferring our model-generated questions 77% of the time. Our model also beats BART-baseline 66% of the time further affirming the importance of using missing schema. Finally, our model beats BART+missinfo model 61% of the time suggesting that the PPLM-based decoder that uses usefulness classifier is able to produce much more useful questions (RQ3). The annotator agreement statistics are provided in appendix.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Ubuntu Table 6 shows the results of human judgments on the model generations of 150 randomly sampled dialog contexts from the Ubuntu test set. In terms of relevance, we find that the transformer and BART baselines produce less relevant    Figure 3: Results of a pairwise comparison (on usefulness criteria) between our model and baseline generated question on (a) 300 randomly sampled product descriptions from the Amazon test set, (b) 150 randomly sampled dialogs from the Ubuntu test set as judged by humans.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
questions. With the addition of missing schema (i.e., BART+missinfo), the questions become more relevant and our proposed model obtains the highest relevance score. The reference obtains slightly a lower relevance score which can possibly be explained by the fact that humans sometimes digress from the topic. Under fluency, interestingly, the transformer and BART baselines obtain high scores. With the addition of missing schema, fluency decreases and the score reduce further with the PPLM model. We suspect that the usefulness classifier trained with a negative sampling strategy (as opposed to human labelled data, as in Amazon) contributes to fluency issues. Under missing information, all models perform well which can be explained by the fact that in Ubuntu, the scope of missing information is much larger (since dialog is much more open-ended) than in Amazon. Figure 3b shows the results of pairwise comparison on usefulness criteria. We find that humans choose our model-generated questions 85% of time when compared to either transformer or BART generated questions. When compared to BART+missinfo, our model is selected 71% of the time, further affirming the importance of using the PPLM-based decoder.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Robustness to input information We analyze how a model is robust toward the amount of information present. To measure the amount of informa-tion, we look for context length (description length for Amazon, dialog context length for Ubuntu) and the size of global schema since these two directly control how much knowledge regarding potential missing information is available to the model. We measure the difference in BLEU score between two groups of data samples where context length/size of global schema is either high or low. Figure 5 shows that our model is the least variant toward the information available hence more robust for the Amazon dataset. 7 Owing to our modular approach for estimating missing information, we seek to analyze whether a question is really asking about missing information in an automatic fashion. This also allows us to explain the reasoning behind a particular generation as we are able to trace back to the particular missing information that is used to generate the question. We run a YAKE extractor on the generated questions to obtain key-phrases. We calculate the ratio between the number of key-phrases in the output that belong to the original missing schema and the total number of key-phrases present in the output. Table 8 shows that when we use our framework of estimating missing information coupled with BART, both models achieve very high missing information overlap, thus suggesting that we can obtain the reasoning behind a generated question reliably by tracing the missing information overlap, as shown in Table 9. Can you tell the output after you install them? BART+missinfo+PPLM Can you try rebooting from the start and removing the software after installation?  Question length We also observe in Table 9 that baseline models tend to generate short and generic questions as compared to our model that often chooses longer schema key-phrases (e.g. bigrams) to generate a more specific question. We further looked into annotated (for usefulness) questions from the Amazon dataset and we observed that 70% of questions that were annotated as useful are longer than not-useful questions. The average length of gold useful questions is 10.76 words and 8.21 for not-useful questions. The average length of generated questions for BART, BART+MissInfo and BART+MissInfo+PPLM (ours) are 5.6, 6.2, 12.3 respectively. We also find a similar trend in the Ubuntu dataset as well.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Dynamic expansion of global schema We anticipate that even if we build the global schema from the available offline dataset, it is possible that new entries may appear in a real application. We investigate how our framework responds to the dynamic expansion of global schema. We simulate a scenario where we extend the "Laptop Acces- sories" category in the Amazon dataset, with 100 new products (those that appeared on Amazon.com after the latest entry in the dataset). We obtain key-phrases from their product descriptions and include them in the global schema for the category which amounts to a 21% change in the existing global schema. For 50 random products in the test set from the same category, we found that in 28 out of 50 cases (56%), the model picked a new schema element that is added later. This indicates that our framework is capable of supporting dynamic changes in the global schema and reflecting them in subsequent generations without retraining from scratch.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
We propose a model for generating useful clarification questions based on the idea that missing information in a context can be identified by taking a difference between the global and the local view.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
We show how we can fine-tune a large-scale pretrained model such as BART on such differences to generate questions about missing information. Further, we show how we can tune these generations to make them more useful using PPLM with a usefulness classifier as its attribute model. Thorough analyses reveal that our framework works across domains, shows robustness towards information availability, and responds to the dynamic change in global knowledge. Although we experiment only with Amazon and Ubuntu datasets, our idea is generalizable to scenarios where it is valuable to identify missing information such as conversational recommendation, or eliciting user preferences in a chit-chat, among others.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
We do not foresee any immediate ethical concerns since we assume that our work will be restricted in domain as compared to free-form language generation. We still cautiously advise any developer who wishes to extend our system for their own use-case (beyond e-commerce, goal-oriented conversations) to be careful about curating a global pool of knowledge for data involving sensitive user information. Finally, since we are finetuning a pretrained generative model, we inherit the general risk of generating biased or toxic language, which should be carefully filtered. In general, we expect users to benefit from our system by reducing ambiguity (when information is presented in a terse fashion, e.g. in a conversation) and improving contextual understanding to enable them to take more informed actions (e.g. making a purchase). 0.6. Finally, we order all key-phrase clusters by their frequencies and retain only the top 60% thus removing low-frequency schema elements.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
While creating the missing schema, we do the match based on semantic similarity of key-phrases (even for a tuple we only look at key-phrase similarity) and we consider two key-phrases to be matched if the cosine similarity is above a threshold, that we set as 0.8 since we want to match only highly similar key-phrases.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
GloVe embeddings on Amazon and Ubuntu datasets We train 200 dimensional GLoVE embeddings on the vocabulary of both Amazon and Ubuntu dataset separately. We set a vocabulary frequency threshold at 50, i.e. we only obtain embeddings for words that appears at least 50 times in the whole corpus.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Datasets Downloadable links to each datasets are provided here: Amazon 10 , Ubuntu 11 .
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Collecting human annotations for usefulness scores For the Amazon dataset, Rao and Daumé III (2019) define the usefulness of a question as the degree to which the answer provided by the question would be useful to potential buyers or current users of the product. We use the annotation scheme defined in Rao and Daumé III (2019) to annotate a set of 5000 questions from the amazon dataset. 12 We show annotators product details (title, category, and description) and a question asked about that product and ask them to give it a usefulness score between 0 to 5. 13 Each question was annotated by three annotators. We average the three scores to get a single usefulness score per question. We use the YAKE extractor to extract the schema elements for each question and assign the usefulness score of the question to each of its schema elements.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Since our aim is to assign a usefulness score to each missing element of each product in our dataset, we train a usefulness classifier on the manually annotated schema elements. Although our usefulness score is a real value between 0 and 5, we find that training a regression model gives us poor performance. Hence we convert the real value into a binary value by threshold at 3 (i.e. values below 3 are assigned label 0 and values above 3 are assigned label 1).
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Usefulness classification with negative sampling Collecting usefulness annotation on questions, as we do for the Amazon dataset, can be expensive and may not always be possible in different scenarios. Therefore, for the Ubuntu dataset, we experiment with a classifier where instead of using human annotations are true labels, we use a negative sampling strategy. Specifically, we assume that all (context, question) pairs in the Ubuntu dataset can be labelled 1 and any (context, random question) can be labelled 0. We sample a set of 2500 questions from the Ubuntu dataset and them label 1 and sample an equivalent number of negative samples and assign them label 0.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
BART and PPLM For question generation model, we use BART-base (6 encoder layers, 6 decoder layers, 117M parameters) 14 ). For PPLM usefulness classifier, we use a bag-of-word model, that uses the pretrained subword embedding layers from BART-base model. We average the subword embeddings to obtain a sentence representation and a usefulness score is predicted via a linear layer projection with softmax. We use the the PPLM code from official repository 15 .
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Each BART variant converged in 3 epochs on an average with batch size 4 in a TITAN X (Pascal) GPU that took 12 hours in total. While training, we only observe perplexity on the validation set to employ an early-stopping criteria.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
We train an SVM (support vector machines) classifier on this data. We use word emebddings as our features by training a 200 dimensional GLoVE model trained on individual dataset. We average the word embeddings of all words in a schema element and use it as a feature. We obtain an F1score of 80.6% on a held out test set. 16 We use this classifier to predict a usefulness score for each missing schema element of each instances from a class for each dataset, which was required for the BART+MissInfo+WD model. 14 https://huggingface.co/transformers/ model_doc/bart.html 15 https://github.com/uber-research/PPLM 16 In comparison, humans get an F1-score of 82.7% in Amazon dataset Figure 5: Average BLEU score difference between classes having longer (more than 200 (median) words) and shorter descriptions larger (more than 200 (median) key-phrases) and shorter global schema for Ubuntu dataset. Lower difference indicates more invariance towards information available.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
We additionally report Krippendorff's alpha, a measure of annotator agreement for our human evaluation, on Amazon dataset.They are : for fluency 0.408, for relevancy 0.177, for missinginfo 0.226, and for usefulness 0.0948. For usefulness, we observe, if the systems are more distinct (GAN-Utility vs BART+missinfo+PPLM), then the agreement is higher i.e. 0.163. For missinginfo, again, 3-way gives higher agreement (0.434), and a probable cause would be that more annotations are going into the undecided category.
Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge
Additionally, Figure 5 shows the BLEU difference across different data samples (based on context length and global schema size) that follow a similar trend to Amazon. Table 9 shows generations from all the models, with a case the our best model trades off with missing information to improve the usefulness.
Open-domain clarification question generation without question examples
Human-machine interaction relies on accurate transfer of knowledge from users. However, natural language input can be ambiguous or unclear, giving rise to uncertainty. A fundamental aspect of human communication is collaborative grounding, or seeking and providing incremental evidence of mutual understanding through dialog. Specifically, humans can correct for uncertainty through cooperative repair (Clark, 1996;Purver et al., 2002;Arkel et al., 2020) which involves interactively asking questions and seeking clarification. Making and recovering from mistakes collaboratively through question-asking is a key ingredient in grounding meaning and therefore an important feature in dialog systems (Benotti and Blackburn, 2021). In this work, we focus on the computational challenge of generating clarification questions in visually grounded human-machine interactions.
Open-domain clarification question generation without question examples
One popular approach is to train an end-to-end model to map visual and linguistic inputs directly to questions (Yao et al., 2018;Das et al., 2017). This approach is heavily data-driven, requiring large annotated training sets of questions under different goals and contexts. Another approach has drawn from work on active learning and Optimal Experiment Design (OED) in cognitive science to search for questions that are likely to maximize expected information gain from an imagined answerer (Wang and Lake, 2019;Lee et al., 2018;Misra et al., 2018;Rao and Daumé III, 2018;Rothe et al., 2017;Kovashka and Grauman, 2013). Much of this work has relied on large-scale question-answer datasets (Kumar and Black, 2020;de Vries et al., 2017) for training or retrieval to propose candidate questions or evaluate their expected utility. Others, like (Yu et al., 2020), derive questions from attribute annotations for domain-specific systems.
Open-domain clarification question generation without question examples
In this paper, we address an open-domain setting where one cannot rely on an immediate grounding of the meaning of questions in the target domain (in contrast to end-to-end approaches, which assume Figure 2: A set of candidate questions are produced by our question generator, and then ranked according to their expected utility in the question selector module. After posing the highest-ranked question and receiving an answer, the belief distribution over images is updated in the answer handler module and these updated beliefs are then either used to guess the target image or are fed back to the question selection module for the process to be repeated. examples of questions to train on, or semantic parsing approaches, which assume a logical form for questions). Our key contribution is a lightweight method to ground question semantics in the open image domain without observing question examples. Instead, our framework builds a visually grounded question-asking model from image captioning data, deriving question selection and belief updating without existing semantics. Our model generates candidate polar questions, arguably the most common form of clarification in dialogue (Stivers, 2010), by applying rule-based linguistic transformations to the outputs of a pretrained image captioner. We then use self-supervision to train a response model that predicts the likelihood of different answers. Given these predictions, we estimate the expected information gain of each question and select the question with the highest utility. We demonstrate our method's ability to pose questions that improve communicative success in a questiondriven communication game with synthetic and human answerers.
Open-domain clarification question generation without question examples
We study interactions between questioners and answerers in a visually grounded 20-questions paradigm (see Fig. 1). Both agents are shown a set of k images as a context (k = 10 in Fig. 1). One of these images is privately indicated to the answerer as the target (e.g., bottom row, center), but remains unknown to the questioner. The questioner's goal is therefore to select questions that allow them to identify this target based on responses from the answerer. After a maximum of 20 questions, the questioner must make a guess (i.e., a kway classification). This task can be viewed as the most straightforward extension of a signaling game (Lewis, 1969) to allow for interactive clarification and repair. To approximate the setting of natural "clarification questions" we also consider games that begin with a description of the target. Critically, the appropriate question changes depending on the context of objects and previous information provided by the answerer.
Open-domain clarification question generation without question examples
Our model (Fig. 2) maintains a belief distribution, p(y|x t ), about which image y in the set of images Y is the target. This distribution is conditioned on the history of the interaction, x t = (a 1 , q 1 , ..., a t , q t ), which includes all questions, q, and answers, a, exchanged up to the current step, t. Our model is defined in terms of three basic components. At each interaction step, it must generate a set of candidate questions, select one of these candidates based on expected information gain and finally update its beliefs based on the answer.
Open-domain clarification question generation without question examples
Question Generator. To generate questions without question examples, we must derive suitable candidates using an alternative method. Specifically, we suggest using a pre-trained image captioner to produce a list of candidate captions, which can then be programmatically transformed into question form. We begin by producing a list of captions for each image y ∈ Y and decomposing each of these captions into multiple polar questions according to a constituency parse, obtained using the Berkeley Neural Parser (Kitaev et al., 2019). We then transform each noun phrase (NP) subtree in each caption's constituency tree into a polar question ('Are there <NP>?' with indefinite articles and plurality chosen for agreement). Using this procedure, we generate an average of 10 candidate questions from each caption (see Appendix A for examples).
Open-domain clarification question generation without question examples
Question Selector. To determine the most informative question at turn t + 1, we estimate expected information gain, EIG(y, a; q, x t ), for every question in the candidate set Q (after a question is asked it is removed from the set). EIG is defined as the change in entropy of the distribution over images after observing an answer a ∈ A(q) to question q. Because the initial entropy is the same for every question, maximizing the EIG is equivalent to minimizing the expected conditional entropy of the belief distribution under possible answers. Because different answers are expected given different targets y, we marginalize over a inside the entropy:
Open-domain clarification question generation without question examples
The distribution p(a|q, y) represents predictions about how the answerer will respond to a question when y is the target. We do not have access to a ground-truth answerer model, so we amortize these predictions by training an answer classifier. We introduced a self-supervision objective by either pairing target images with questions derived from them ('yes' answers) or with questions derived from other images ('no' answers). It should be noted that this data-generation method may occasionally yield a false negative when, for a 'no'labelled question-image pair, a question is sampled that does coincidentally apply to the image; however, these samples represent a minority of the training data. We then trained a logistic classifier using cross-entropy loss on concatenated image and caption embeddings obtained from a CNN and RNN encoder, respectively. This classifier yields a prediction of yes vs. no answers for any unseen pair (y, q) with 94% accuracy on held-out, manuallylabelled datapoints.
Open-domain clarification question generation without question examples
Answer Handler. Finally, after obtaining an answer a, our model must update its beliefs for the subsequent time step (and anticipate this update for Eq. 1). The belief update is given by Bayes rule: p(y|x t , q, a) ∝ p(a|x t , q, y)p(q|x t , y)p(y|x t )
Open-domain clarification question generation without question examples
The first term p(a|x t , q, y) can be simplified to our amortized answer prediction model described above by assuming that the answer is independent of past interactions. The second term is given by the deterministic question selector model described above. The third term is given by the belief distribution on the previous time step. The initial belief distribution is either uniform, p(y|x 0 ) ∝ 1, or, when an initial description u is provided, it is proportional to the utterance likelihood under the captioning model, p(y|x 0 ) ∝ p(u|y).
Open-domain clarification question generation without question examples
We evaluated our question-asking framework in grounded interactions with both synthetic and human answerers.
Open-domain clarification question generation without question examples
Before deploying our model in interactions with human speakers, we examined its performance on synthetic datasets where we could carefully control the answerer. We examine two domains: Shapeworld (Kuhnle and Copestake, 2017), a simple artificial dataset of images of random colored shapes paired with captions from a vocabulary of 15 words labeling the possible colors and shapes, and MS COCO (Lin et al., 2015), a more naturalistic dataset containing images of everyday scenes paired with captions elicited by human annotators. Because previous approaches have typically relied on closed-domain question-answer datasets or hand-built question semantics, they are incompatible with our 'open domain' setting. Instead, we compare our full model's performance against several model variants and strong, general-purpose search baselines: a full caption model which generates candidate questions from full image captions without decomposition, comparable to a linear search checking one image at a time; a random question model which selects questions randomly instead of using the expected information gain objective; and, a binary search algorithm which serves as an upper-bound "oracle," unfettered by the expressivity of real language, by randomly halving the set of potential target images with each step of the interaction rather than posing a natural language question. We evaluate these models on a total of 1,000 games sampled from each dataset using contexts of size k = 10 images.
Open-domain clarification question generation without question examples
For Shapeworld, we paired our questioner with an artificial answerer constructed to provide  ground-truth answers to generated questions (Figure 3, left). Our proposed model outperforms the random baseline as well as the full caption model, which produces questions that are too specific to efficiently narrow the space of potential target images, while only slightly under-performing an upper-bound binary search algorithm. These findings demonstrate the utility of having a question set of varying specificity (via decomposing full captions into NPs) as well as the expected information gain objective which adapts question selection to the model's current knowledge. For MS COCO, we construct an artificial answerer that uses a simple heuristic, as ground-truth answers are not readily available for MS COCO. This answerer responds "yes" if a question is generated from the target image, and "no" otherwise ( Figure 3, middle). We again see that our model greatly outperforms a random questioner, but outperforms the full caption baseline to a lesser extent than we observed on Shapeworld. The larger gap between our model and binary search also indicates significant room for improvement. One possible explanation for this gap is the difficulty of finding attributes which appropriately "split" a random set of natural images. To evaluate performance when a clear division of the image set is expressable in natural language, we created an alternative test set  where we ensured that the 10 images in the context were balanced across two categories in COCO (i.e., five "motorcycles" and five "baseballs"). We found that the model was indeed better able to divide the image set when we guaranteed that some high-level cut between the images existed (Table 1). When models were given an initial description of the target image before asking any questions (Figure 3, right), we see that questions are still useful -improving accuracy by 6% from the caption alone.
Open-domain clarification question generation without question examples
Extension to wh-questions. While our main results use polar questions exclusively, our framework has the potential to be extended to more general wh-questions. Using wh-movement rules we can derive questions from image captions that ask about more abstract properties of objects within images (e.g., given the caption "three men holding surfboards on a beach" we can straightforwardly derive questions like: "How many men are there?", "Where are the men?", or "What are the men holding?"). To illustrate this extension we provide preliminary results for simple 'what' questions. We generate these questions by identifying instances of noun phrases followed by verb phrases in captions and transforming these into a set of 'what' questions with single-word answers. We extract the noun (NN) and verb (VBG) from their respective phrases then produce questions of the form 'What   is the <NN> <VBG>?'. To accommodate these questions in our model, we simply modified our answer classifier to produce a probability distribution over the entire vocabulary (rather than a binary yes-no). By incorporating what questions into our framework, we see an improvement of almost 3% after 20 questions are asked (Table 2).
Open-domain clarification question generation without question examples
We ran two experiments to evaluate our question generation model in interactions with real human partners. We recruited a total of 40 participants from Amazon Mechanical Turk to play 10 games each in which our model asked questions until the entropy of the belief distribution over images fell below 1.0 or until 20 questions were asked. Participants were prompted to give either a "yes", "no", or "N/A" response to each question. In the first human experiment, games were sampled from the same 1,000 MS COCO games used for synthetic evaluation (Table 3). We found that our question-asking model was able to successfully improve target selection accuracy when paired with a human answerer, suggesting that our model's questions are human interpretable and that human answers are effective for target selection.
Open-domain clarification question generation without question examples
Our second human experiment examines the more challenging case of asking "clarification questions" in a referential setting. In this experiment we used larger contexts of k = 25 images sampled from the MS COCO test set, and human participants were prompted to give a description of the target to initiate the interaction. Our model formed (uncertain) beliefs based on this initial utterance and proceeded to ask clarification questions which we found improved by 16.5% from the image description alone (see Table 4).
Open-domain clarification question generation without question examples
We introduce a question generation framework capable of producing open-domain clarification questions. Instead of relying on specialized questionanswer training data or pre-specified question meanings, our model uses a pretrained image captioner in conjunction with expected information gain to produce informative questions for unseen images. We demonstrate the effectiveness of this method in a question-driven communication game with synthetic and human answerers. We found it important to generate questions varying in specificity by decomposing captioner utterances into component noun phrases. Having generated this set of potential questions, selecting based on estimated information gain yielded useful questions. Without seeing question examples, our framework demonstrates a capacity for generating effective clarification questions.
Open-domain clarification question generation without question examples
Future research should aim to generate more diverse question sets, allow for more expressive answers, and address abstract properties of objects within images. One approach, as demonstrated by our preliminary work with 'what'-questions, would be to extend our framework to incorporate additional types of wh-questions. Integrating this clarification capacity more fully into collaborative, goal-directed dialog agents will allow them to engage in cooperative repair.  with and without the initial target image description "food".
Answer-based Adversarial Training for Generating Clarification Questions
A goal of natural language processing is to develop techniques that enable machines to process naturally occurring language. However, not all language is clear and, as humans, we may not always understand each other (Grice, 1975); in cases of gaps or mismatches in knowledge, we tend to ask questions (Graesser et al., 2008). In this work, we focus on the task of automatically generating clarification questions: questions that ask for information that is missing from a given linguistic context. Our clarification question generation model builds on the sequence-tosequence approach that has proven effective for several language generation tasks (Sutskever et al., 2014;Serban et al., 2016;Yin et al., 2016;Du et al., 2017). Unfortunately, training a sequenceto-sequence model directly on (context, question) pairs yields questions that are highly generic 1 , corroborating a common finding in dialog systems (Li et al., 2016b). Our goal is to be able to generate clarification questions that are useful and specific.
Answer-based Adversarial Training for Generating Clarification Questions
To achieve this, we begin with a recent observation of Rao and Daumé III (2018), who consider the task of question reranking: a good clarification question is the one whose answer has a high utility, which they define as the likelihood that this question would lead to an answer that will make the context more complete ( §2.3). Inspired by this, we construct a model that first generates a question given a context, and then generates a hypothetical answer to that question. Given this (context, question, answer) triple, we train a utility calculator to estimate the usefulness of this question. We then show that this utility calculator can be generalized using ideas for generative adversarial networks (Goodfellow et al., 2014) for text (Yu et al., 2017), wherein the utility calculator plays the role of the "discriminator" and the question generator is the "generator" ( §2.2), which we train using the MIXER algorithm (Ranzato et al., 2015). We evaluate our approach on two datasets: Amazon product descriptions ( Figure 1) and Stack Exchange posts ( Figure 2). Our two main contributions are:
Answer-based Adversarial Training for Generating Clarification Questions
1. An adversarial training approach for generating clarification questions that models the utility of updating a context with an answer to the clarification question. 2 2. An empirical evaluation using both automatic metrics and human judgments to show that our adversarially trained model generates questions that are more useful and specific to the context than all the baseline models.
Answer-based Adversarial Training for Generating Clarification Questions
Product Answer They are aluminium so the answer is NO. Figure 1: Sample product description from Amazon paired with a clarification question and answer.
Answer-based Adversarial Training for Generating Clarification Questions
Our goal is to build a model that, given a context, can generate an appropriate clarification question. Our dataset consists of (context, question, answer) triples where the context is an initial textual context, question is the clarification question that asks about some missing information in the context and answer is the answer to the clarification question (details in § 3.1). Representationally, our question generator is a standard sequence-to-sequence model with attention ( §2.1). The learning problem is: how to train the sequence-to-sequence model to generate good clarification questions.
Answer-based Adversarial Training for Generating Clarification Questions
An overview of our training setup is shown in Figure 3. Given a context, our question generator, which is a sequence-to-sequence model, outputs a question. In order to evaluate the usefulness of this question, we then have a second sequenceto-sequence model called the "answer generator" that generates a hypothetical answer based on the context and the question ( § 2.5). This (context, generated question and generated answer) triple is fed into a UTILITY calculator, whose initial goal is to estimate the probability that this (question, answer) pair is useful in this context ( § 2.3). This UTILITY is treated as a reward, which is used to update the question generator using the MIXER (Ranzato et al., 2015) algorithm ( § 2.2). Finally, we reinterpret the answer-generator-plusutility-calculator component as a discriminator for differentiating between (context, true question, generated answer) triples and (context, generated question, generated answer) triples , and optimize the generator for this adversarial objective using MIXER ( §2.4).  
Answer-based Adversarial Training for Generating Clarification Questions
We use a standard attention based sequence-tosequence model (Luong et al., 2015) for our question generator. Given an input sequence (context) c = (c 1 , c 2 , ..., c N ), this model generates an output sequence (question) q = (q 1 , q 2 , ..., q T ). The architecture of this model is an encoder-decoder with attention. The encoder is a recurrent neural network (RNN) operating over the input word embeddings to compute a source context representationc. The decoder uses this source representation to generate the target sequence one word at a time:
Answer-based Adversarial Training for Generating Clarification Questions
In Eq 1,h t is the attentional hidden state of the RNN at time t and W s and W c are parameters of the model. 3 The predicted token q t is the token in the vocabulary that is assigned the highest probability using the softmax function. The standard training objective for sequence-to-sequence model is to maximize the log-likelihood of all (c, q) pairs in the training data D which is equivalent to minimizing the following loss,
Answer-based Adversarial Training for Generating Clarification Questions
3 Details are in Appendix A. 
Answer-based Adversarial Training for Generating Clarification Questions
Training sequence-to-sequence models for the task of clarification question generation (with context as input and question as output) using maximum likelihood objective unfortunately leads to the generation of highly generic questions, such as "What are the dimensions?" when asking questions about home appliances. Recently, Rao and Daumé III (2018) observed that the usefulness of a question can be better measured as the utility that would be obtained if the context were updated with the answer to the proposed question. Following this observation, we first use a pretrained answer generator ( §2.5) to generate an answer given a context and a question. We then use a pretrained UTILITY calculator ( § 2.3 ) to predict the likelihood that the generated answer would increase the utility of the context by adding useful information to it. Finally, we train our question generator to optimize this UTILITY based reward. Similar to optimizing metrics like BLEU and ROUGE, this UTILITY calculator also operates on discrete text outputs, which makes optimization difficult due to non-differentiability. A successful recent approach dealing with the nondifferentiability while also retaining some advantages of maximum likelihood training is the Mixed Incremental Cross-Entropy Reinforce (Ranzato et al., 2015) algorithm (MIXER). In MIXER, the overall loss L is differentiated as in REINFORCE (Williams, 1992):
Answer-based Adversarial Training for Generating Clarification Questions
where q s is a random output sample according to the model p θ and θ are the parameters of the network. The expected gradient is then approximated using a single sample q s = (q s 1 , q s 2 , ..., q s T ) from the model distribution (p θ ). In REINFORCE, the policy is initialized randomly, which can cause long convergence times. To solve this, MIXER starts by optimizing maximum likelihood for the initial ∆ time steps, and slowly shifts to optimizing the expected reward from Eq 3 for the remaining (T − ∆) time steps.
Answer-based Adversarial Training for Generating Clarification Questions
In our model, for the initial ∆ time steps, we minimize L mle and for the remaining steps, we minimize the following UTILITY-based loss:
Answer-based Adversarial Training for Generating Clarification Questions
where r(q p ) is the UTILITY based reward on the predicted question and r(q b ) is a baseline reward introduced to reduce the high variance otherwise observed when using REINFORCE. To estimate this baseline reward, we take the idea from the self-critical training approach Rennie et al. (2017) where the baseline is estimated using the reward obtained by the current model under greedy decoding during test time. We find that this approach for baseline estimation stabilizes our model better than the approach used in MIXER.
Answer-based Adversarial Training for Generating Clarification Questions
Given a (context, question, answer) triple, Rao and Daumé III (2018) introduce a utility calculator UTILITY(c, q, a) to calculate the value of updating a context c with the answer a to a clarification question q. They use the utility calculator to estimate the probability that an answer would be a meaningful addition to a context. They treat this as a binary classification problem where the positive instances are the true (context, question, answer) triples in the dataset whereas the negative instances are contexts paired with a random (question, answer) from the dataset. Following Rao and Daumé III (2018), we model our UTILITY calculator by first embedding the words in c and then using an LSTM (long-short term memory) (Hochreiter and Schmidhuber, 1997) to generate a neural representationc of the context by averaging the output of each of the hidden states. Similarly, we obtain neural representationsq andā of q and a respectively using a question and an answer LSTM models. Finally, we use a feed forward neural network F UTILITY (c,q,ā) to predict the usefulness of the question.
Answer-based Adversarial Training for Generating Clarification Questions
The UTILITY calculator trained on true vs random samples from real data (as described in the previous section) can be a weak reward signal for questions generated by a model due to the large discrepancy between the true data and the model's outputs. In order to strengthen the reward signal, we reinterpret the UTILITY calculator (coupled with the answer generator) as a discriminator in an adversarial learning setting. That is, instead of taking the UTILITY calculator to be a fixed model that outputs the expected quality of a (question, answer) pair, we additionally optimize it to distinguish between true (question, answer) pairs and model-generated ones. This reinterpretation turns our model into a form of a generative adversarial network (GAN) (Goodfellow et al., 2014).
Answer-based Adversarial Training for Generating Clarification Questions
GAN is a training procedure for "generative" models that can be interpreted as a game between a generator and a discriminator. The generator is a model g ∈ G that produces outputs (in our case, questions). The discriminator is another model d ∈ D that attempts to classify between true outputs and model-generated outputs. The goal of the generator is to generate data such that it can fool the discriminator; the goal of the discriminator is to be able to successfully distinguish between real and generated data. In the process of trying to fool the discriminator, the generator produces data that is as close as possible to the real data distribution.
Answer-based Adversarial Training for Generating Clarification Questions
Generically, the GAN objective is:
Answer-based Adversarial Training for Generating Clarification Questions
where x is sampled from the true data distribution p, and z is sampled from a prior defined on input noise variables p z .
Answer-based Adversarial Training for Generating Clarification Questions
Although GANs have been successfully used for image tasks, training GANs for text generation is challenging due to the discrete nature of outputs in text. The discrete outputs from the generator make it difficult to pass the gradient update from the discriminator to the generator. Recently, Yu et al. (2017) proposed a sequence GAN model for text generation to overcome this issue. They treat their generator as an agent and use the discriminator as a reward function to update the generative model using reinforcement learning techniques. Our GAN-based approach is inspired by this sequence GAN model with two main modifications: a) We use MIXER algorithm as our generator ( § 2.2) instead of a purely policy gradient approach; and b) We use UTILITY calculator ( § 2.3) as our discriminator instead of a convolutional neural network (CNN).
Answer-based Adversarial Training for Generating Clarification Questions
Theoretically, the discriminator should be trained using (context, true question, true answer) triples as positive instances and (context, generated question, generated answer) triples as the negative instances. However, we find that training a discriminator using such positive instances makes it very strong since the generator would have to not only generate real looking questions but also generate real looking answers to fool the discriminator. Since our main goal is question generation and since we use answers only as latent variables, we instead use (context, true question, generated answer) as our positive instances where we use the pretrained answer generator to get the generated answer for the true question. Formally, our objective function is:
Answer-based Adversarial Training for Generating Clarification Questions
where U is the UTILITY discriminator, M is the MIXER generator,p is our data of (context, question, answer) triples and A is the answer generator.
Answer-based Adversarial Training for Generating Clarification Questions
Question Generator. We pretrain our question generator using the sequence-to-sequence model ( §2.1) to maximize the log-likelihood of all (context, question) pairs in the training data. Parameters of this model are updated during adversarial training. Answer Generator. We pretrain our answer generator using the sequence-to-sequence model ( §2.1) to maximize the log-likelihood of all ([con-text+question], answer) pairs in the training data. Parameters of this model are kept fixed during the adversarial training. 4 Discriminator. In our UTILITY GAN model ( § 2.4), the discriminator is trained to differentiate between true and generated questions. However, since we want to guide our UTILITY based discriminator to also differentiate between true ("good") and random ("bad") questions, we pretrain our discriminator in the same way we trained our UTILITY calculator. For positive instances, we use a context and its true question, answer from the training data and for negative instances, we use the same context but randomly sample a question from the training data (and use the answer paired with that random question).
Answer-based Adversarial Training for Generating Clarification Questions
We base our experimental design on the following research questions:
Answer-based Adversarial Training for Generating Clarification Questions
1. Do generation models outperform simpler retrieval baselines? 2. Does optimizing the UTILITY reward improve over maximum likelihood training? 3. Does using adversarial training improve over optimizing the pretrained UTILITY? 4. How do the models perform when evaluated for nuances such as specificity & usefulness?
Answer-based Adversarial Training for Generating Clarification Questions
We evaluate our model on two datasets. Amazon. In this dataset, context is a product description on amazon.com combined with the product title, question is a clarification question asked to the product and answer is the seller's (or other users') reply to the question. To obtain these data triples, we combine the Amazon question-answering dataset (McAuley and Yang, 2016) with the Amazon reviews dataset (McAuley et al., 2015). We show results on the Home & Kitchen category of this dataset since it contains a large number of questions and is relatively easier for human-based evaluation. It consists of 19, 119 training, 2, 435 tune and 2, 305 test examples (product descriptions), with 3 to 10 questions (average: 7) per description.
Answer-based Adversarial Training for Generating Clarification Questions
Stack Exchange. In this dataset, context is a post on stackexchange.com combined with the title, question is a clarification question asked in the comments section of the post and answer is either the update made to the post in response to the question or the author's reply to the question in the comments section. Rao and Daumé III (2018) curated a dataset of 61, 681 training, 7, 710 tune and 7, 709 test such triples from three related subdomains on stackexchage.com (askubuntu, unix and superuser). Additionally, for 500 instances each from the tune and the test set, their dataset includes 1 to 6 other questions identified as valid questions by expert human annotators from a pool of candidate questions.
Answer-based Adversarial Training for Generating Clarification Questions
We compare three variants (ablations) of our proposed approach, together with an information retrieval baseline:
Answer-based Adversarial Training for Generating Clarification Questions
GAN-Utility is our full model which is a UTIL-ITY calculator based GAN training ( §2.4) including the UTILITY discriminator and the MIXER question generator. 5 Max-Utility is our reinforcement learning baseline where the pretrained question generator model is further trained to optimize the UTILITY reward ( §2.2) without the adversarial training.
Answer-based Adversarial Training for Generating Clarification Questions
MLE is the question generator model pretrained on context, question pairs using maximum likelihood objective ( §2.1).
Answer-based Adversarial Training for Generating Clarification Questions
Lucene 6 is our information retrieval baseline similar to the Lucene baseline described in Rao and Daumé III (2018). Given a context in the test set, we use Lucene, which is a TF-IDF based document ranker, to retrieve top 10 contexts that are most similar to the given context in the train set. We randomly choose a question from the human written questions paired with these 10 contexts in the train set to construct our Lucene baseline 7 .
Answer-based Adversarial Training for Generating Clarification Questions
We evaluate initially with automated evaluation metrics, and then more substantially with crowdsourced human judgments.
Answer-based Adversarial Training for Generating Clarification Questions
Diversity, which calculates the proportion of unique trigrams in the output to measure the diversity as commonly used to evaluate dialogue generation (Li et al., 2016b). BLEU (Papineni et al., 2002) 8 , which evaluates n-gram precision between the output and the references. METEOR (Banerjee and Lavie, 2005), which is similar to BLEU but includes stemmed and synonym matches to measure similarity between the output and the references.
Answer-based Adversarial Training for Generating Clarification Questions
We use Figure-Eight 9 , a crowdsourcing platform, to collect human judgements. Each judgement 10 consists of showing the crowdworker a context and a generated question and asking them to evaluate the question along following axes: Relevance: We ask "Is the question on topic?" and let workers choose from: Yes (1) and No (0) Grammaticality: We ask "Is the question grammatical?" and let workers choose from: Yes (1) and No (0) Seeking new information: We ask "Does the question ask for new information currently not included in the description?" and let workers choose from: Yes (1) and No (0) Specificity: We ask "How specific is the question?" and let workers choose from: 4: Specific pretty much only to this product (or same product from different manufacturer) 3: Specific to this and other very similar products 2: Generic enough to be applicable to many other products of this type 1: Generic enough to be applicable to any product under Home and Kitchen 0: N/A (Not applicable) i.e. Question is not on topic OR is incomprehensible Usefulness: We ask "How useful is the question to a potential buyer (or a current user) of the product?" and let workers choose from: 8 https://github.com/moses-smt/ mosesdecoder/blob/master/scripts/ generic/multi-bleu.perl 9 https://www.figure-eight.com 10 We paid crowdworkers 5 cents per judgment and collected five judgments per question. 4: Useful enough to be included in the product description 3: Useful to a large number of potential buyers (or current users) 2: Useful to a small number of potential buyers (or current users) 1: Useful only to the person asking the question 0: N/A (Not applicable) i.e. Question is not on topic OR is incomprehensible OR is not seeking new information Table 1 shows the inter-annotator agreement (reported by Figure-Eight as confidence 11 ) on each of the above five criteria. Agreement on Relevance, Grammaticality and Seeking new information is high. This is not surprising given that these criteria are not very subjective. On the other hand, the agreement on usefulness and specificity is quite moderate since these judgments can be very subjective.
Answer-based Adversarial Training for Generating Clarification Questions
Since the inter-annotator agreement on the usefulness criteria was particularly low, in order to reduce the subjectivity involved in the fine grained annotation, we convert the range [0-4] to a more coarse binary range [0-1] by mapping the scores 4 and 3 to 1 and the scores 2, 1 and 0 to 0. Table 2 shows the results on the two datasets when evaluated according to automatic metrics.
Answer-based Adversarial Training for Generating Clarification Questions
In the Amazon dataset, GAN-Utility outperforms all ablations on DIVERSITY, suggesting that it produces more diverse outputs. Lucene, on the other hand, has the highest DIVERSITY since it consists of human written questions, which tend to be more diverse because they are much longer compared to model generated questions. This comes at the cost of lower match with the reference as visible in the BLEU and METEOR scores.  In terms of BLEU and METEOR, there is inconsistency. Although GAN-Utility outperforms all baselines according to METEOR, the fully ablated MLE model has a higher BLEU score. This is because BLEU score looks for exact n-gram matches and since MLE produces more generic outputs, it is much more likely that it will match one of 10 references compared to the specific/diverse outputs of GAN-Utility, since one of those ten is highly likely to itself be generic.
Answer-based Adversarial Training for Generating Clarification Questions
In the StackExchange dataset GAN-Utility outperforms all ablations on both BLEU and ME-TEOR. Unlike in the Amazon dataset, MLE does not outperform GAN-Utility in BLEU. This is because the MLE outputs in this dataset are not as generic as in the amazon dataset due to the highly technical nature of contexts in StackExchange. As in the Amazon dataset, GAN-Utility outperforms MLE on DIVERSITY. Interestingly, the Max-Utility ablation achieves a higher DIVER-SITY score than GAN-Utility. On manual analysis we find that Max-Utility produces longer outputs compared to GAN-Utility but at the cost of being less grammatical. Table 3 shows the numeric results of human-based evaluation performed on the reference and the system outputs on 300 random samples from the test set of the Amazon dataset. 12 All approaches produce relevant and grammatical questions. All models are all equally good at seeking new information, but are weaker than Lucene, which performs better at seeking new information but at the 12 We could not ask crowdworkers evaluate the StackExchange data due to its highly technical nature. cost of much lower specificity and lower usefulness.
Answer-based Adversarial Training for Generating Clarification Questions
Our full model, GAN-Utility, performs significantly better at the usefulness criteria showing that the adversarial training approach generates more useful questions. Interestingly, all our models produce questions that are more useful than Lucene and Reference, largely because Lucene and Reference tend to ask questions that are more often useful only to the person asking the question, making them less useful for potential other buyers (see Figure 4). GAN-Utility also performs significantly better at generating questions that are more specific to the product (see details in Figure 5), which aligns with the higher DIVERSITY score obtained by GAN-Utility under automatic metric evaluation. Table 5 contains example outputs from different models along with their usefulness and specificity scores. MLE generates questions such as "is it waterproof?" and "what is the wattage?", which are applicable to many other products. Whereas our GAN-Utility model generates more specific question such as "is this shower curtain mildew resistant?". Appendix C includes further analysis of system outputs on both Amazon and Stack Exchange datasets.
Answer-based Adversarial Training for Generating Clarification Questions
In this work, we describe a novel approach to the problem of clarification question generation. We use the observation of Rao and Daumé III (2018) that the usefulness of a clarification question can be measured by the value of updating a context with an answer to the question. We use a sequence-to-sequence model to generate a question given a context and a second sequence-tosequence model to generate an answer given the context and the question. Given the (context, generated question, generated answer) triple, we calculate the utility of this triple and use it as a reward to retrain the question generator using reinforcement learning based MIXER model. Further, to improve upon the utility calculator, we reinterpret it as a discriminator in an adversarial setting and train both the utility calculator and the MIXER model in a minimax fashion. We find that our adversarial training approach produces more useful and specific questions compared to both a model trained using maximum likelihood objective and a model trained using utility reward based reinforcement learning.
Answer-based Adversarial Training for Generating Clarification Questions
There are several avenues of future work. Following Mostafazadeh et al. (2016), we could combine text input with image input in the Amazon dataset (McAuley and Yang, 2016) to generate more relevant and useful questions. One significant research challenge in the space of free text generation problems when the set of possible outputs is large, is that of automatic evaluation (Lowe et al., 2016): in our results we saw some correlation between human judgments and automatic metrics, but not enough to trust the automatic metrics completely. Lastly, we hope to integrate such a question generation model into a real world platform like StackExchange or Amazon to understand the real utility of such models and to unearth additional research questions. Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. 2016 A Sequence-to-sequence model details
Answer-based Adversarial Training for Generating Clarification Questions
In this section, we describe some of the details of the attention based sequence-to-sequence model introduced in Section 2.1 of the main paper. In equation 1,h t is the attentional hidden state of the RNN at time t obtained by concatenating the target hidden state h t and the source-side context vector c t , and W s is a linear transformation that maps h t to an output vocabulary-sized vector. Each attentional hidden stateh t depends on a distinct input context vectorc t computed using a global attention mechanism over the input hidden states as:
Answer-based Adversarial Training for Generating Clarification Questions
a nt = align(h n , h t )
Answer-based Adversarial Training for Generating Clarification Questions
The attention weights a nt is calculated based on the alignment score between the source hidden state h n and the current target hidden state h t .
Answer-based Adversarial Training for Generating Clarification Questions
In this section, we describe the details of our experimental setup.
Answer-based Adversarial Training for Generating Clarification Questions
We tokenize and lowercase all inputs (context, question and answers). We set the max length of context to be 100, question to be 20 and answer to be 20. We find that increasing the length of contexts (to 150 or 200) of question/ answer (to 40) yields similar results according to automatic metrics with increased experimentation time.
Answer-based Adversarial Training for Generating Clarification Questions
Our sequence-to-sequence model (Section 2.1) operates on word embeddings which are pretrained on in domain data using Glove (Pennington et al., 2014). As frequently used in previous work on neural network modeling, we use an embeddings of size 200 and a vocabulary with cut off frequency set to 10. During train time, we use teacher forcing (Williams and Zipser, 1989). During test time, we use beam search decoding with beam size 5.
Answer-based Adversarial Training for Generating Clarification Questions
We use a hidden layer of size two for both the encoder and decoder recurrent neural network models with size of hidden unit set to 100. We use a dropout of 0.5 and learning ratio of 0.0001. In the MIXER model, we start with ∆ = T and decrease it by 2 for every epoch (we found decreasing ∆ to 0 is ineffective for our task, hence we stop at 2).
Answer-based Adversarial Training for Generating Clarification Questions
First half of Table 5 shows the system generated questions for three product descriptions in the Amazon dataset.
Answer-based Adversarial Training for Generating Clarification Questions
In the first example, the product is a shower curtain. The Reference question is specific and highly useful. Lucene, on the other hand, picks a moderately specific ("how to clean it?") but useful question. MLE model generates a generic but useful "is it waterproof?". Max-Utility generates comparatively a much longer question but in doing so loses out on relevance. This behavior of generating two unrelated sentences is observed quite a few times in both Max-Utility and GAN-Utility models. This suggests that these models, in trying to be very specific, end up losing out on relevance. In the same example, GAN-Utility also generates a fairly long question which, although awkwardly phrase, is quite specific and useful.
Answer-based Adversarial Training for Generating Clarification Questions
In the second example, the product is a Duvet Cover Set. Both Reference and Lucene questions here are examples of questions that are pretty much useful only to the person asking the question. We find many such questions in both Reference and Lucene outputs which is the main reason for the comparatively lower usefulness scores for their outputs. All three of our models generate irrelevant questions since the product description explicitly says that the set is full size.
Answer-based Adversarial Training for Generating Clarification Questions
In the last example, the product is a set of mopping clothes. Reference question is quite specific but has low usefulness. Lucene picks an irrelevant question. MLE and Max-Utility generate highly specific and useful questions. GAN-Utility generates an ungrammatical question by repeating the last word many times. We observe this behavior quite a few times in the outputs of both Max-Utility and GAN-Utility models suggesting that our sequence-to-sequence models are not very good at maintaining long range dependencies.
Answer-based Adversarial Training for Generating Clarification Questions
Second half of Table 5 shows the system generated questions for three posts from the Stack Exchange dataset.
Answer-based Adversarial Training for Generating Clarification Questions
The first example is of a post where someone describes their issue of not being able to recover from their boot. Reference and Lucene questions are useful. MLE generates a generic question that is not very useful. Max-Utility generates a useful question but has slight ungrammaticality in it. GAN-Utility, on the other hand, generates a specific and an useful question.
Answer-based Adversarial Training for Generating Clarification Questions
In the second example, again Reference and Lucene questions are useful. MLE generates a generic question. Max-Utility and GAN-Utility both generate fairly specific question but contain unknown tokens. The Stack Exchange dataset contains several technical terms leading to a long tail in the vocabulary. Owing to this, we find that both Max-Utility and GAN-Utility models generate many instances of questions with unknown tokens.
Answer-based Adversarial Training for Generating Clarification Questions
In the third example, the Reference question is very generic. Lucene asks a relevant question. MLE again generates a generic question. Both Max-Utility and GAN-Utility generate specific and relevant questions.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Inquiry is fundamental to communication, and machines cannot effectively collaborate with humans unless they can ask questions. In this work, we build a neural network model for the task of ranking clarification questions. Our model is inspired by the idea of expected value of perfect information: a good question is one whose expected answer will be useful. We study this problem using data from StackExchange, a plentiful online resource in which people routinely ask clarifying questions to posts so that they can better offer assistance to the original poster. We create a dataset of clarification questions consisting of ∼77K posts paired with a clarification question (and answer) from three domains of StackExchange: askubuntu, unix and superuser. We evaluate our model on 500 samples of this dataset against expert human judgments and demonstrate significant improvements over controlled baselines.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
A principle goal of asking questions is to fill information gaps, typically through clarification questions. 1 We take the perspective that a good question is the one whose likely answer will be useful. Consider the exchange in Figure 1, in which an initial poster (who we call "Terry") asks for help configuring environment variables. This post is underspecified and a responder ("Parker") asks a clarifying question (a) below, but could alternatively have asked (b) or (c):
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
(a) What version of Ubuntu do you have? 1 We define 'clarification question' as a question that asks for some information that is currently missing from the given context. Figure 1: A post on an online Q & A forum "askubuntu.com" is updated to fill the missing information pointed out by the question comment. Parker should not ask (b) because an answer is unlikely to be useful; they should not ask (c) because it is too specific and an answer like "No" or "I do not know" gives little help. Parker's question (a) is much better: it is both likely to be useful, and is plausibly answerable by Terry.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
In this work, we design a model to rank a candidate set of clarification questions by their usefulness to the given post. We imagine a use case (more discussion in § 7) in which, while Terry is writing their post, a system suggests a shortlist of questions asking for information that it thinks people like Parker might need to provide a solution, thus enabling Terry to immediately clarify their post, potentially leading to a much quicker resolution. Our model is based on the decision theoretic framework of the Expected Value of Perfect Information (EVPI) (Avriel and Williams, 1970), a measure of the value of gathering additional information. In our setting, we use EVPI to calculate which questions are most likely to elicit an answer that would make the post more informative. The behavior of our model during test time: Given a post p, we retrieve 10 posts similar to post p using Lucene. The questions asked to those 10 posts are our question candidates Q and the edits made to the posts in response to the questions are our answer candidates A. For each question candidate q i , we generate an answer representation F (p, q i ) and calculate how close is the answer candidate a j to our answer representation F (p, q i ). We then calculate the utility of the post p if it were updated with the answer a j . Finally, we rank the candidate questions Q by their expected utility given the post p (Eq 1).
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Our work has two main contributions: 1. A novel neural-network model for addressing the task of ranking clarification question built on the framework of expected value of perfect information ( §2). 2. A novel dataset, derived from StackExchange 2 , that enables us to learn a model to ask clarifying questions by looking at the types of questions people ask ( §3). We formulate this task as a ranking problem on a set of potential clarification questions. We evaluate models both on the task of returning the original clarification question and also on the task of picking any of the candidate clarification questions marked as good by experts ( §4). We find that our EVPI model outperforms the baseline models when evaluated against expert human annotations. We include a few examples of human annotations along with our model performance on them in the supplementary material. We have released our dataset of ∼77K (p, q, a) triples and the expert annotations on 500 triples to help facilitate further research in this task. 3
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
We build a neural network model inspired by the theory of expected value of perfect information (EVPI). EVPI is a measurement of: if I were to acquire information X, how useful would that be to 2 We use data from StackExchange; per license cc-by-sa 3.0, the data is "intended to be shared and remixed" (with attribution).
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
3 https://github.com/raosudha89/ ranking_clarification_questions me? However, because we haven't acquired X yet, we have to take this quantity in expectation over all possible X, weighted by each X's likelihood. In our setting, for any given question q i that we can ask, there is a set A of possible answers that could be given. For each possible answer a j ∈ A, there is some probability of getting that answer, and some utility if that were the answer we got. The value of this question q i is the expected utility, over all possible answers:
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
In Eq 1, p is the post, q i is a potential question from a set of candidate questions Q and a j is a potential answer from a set of candidate answers A. Here, P[a j |p, q i ] measures the probability of getting an answer a j given an initial post p and a clarifying question q i , and U(p + a j ) is a utility function that measures how much more complete p would be if it were augmented with answer a j . The modeling question then is how to model:
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
1. The probability distribution P[a j |p, q i ] and 2. The utility function U(p + a j ). In our work, we represent both using neural networks over the appropriate inputs. We train the parameters of the two models jointly to minimize a joint loss defined such that an answer that has a higher potential of increasing the utility of a post gets a higher probability. Figure 2 describes the behavior of our model during test time. Given a post p, we generate a set of candidate questions and a set of candidate Figure 3: Training of our answer generator. Given a post p i and its question q i , we generate an answer representation that is not only close to its original answer a i , but also close to one of its candidate answers a j if the candidate question q j is close to the original question q i . answers ( §2.1). Given a post p and a question candidate q i , we calculate how likely is this question to be answered using one of our answer candidates a j ( §2.2). Given a post p and an answer candidate a j , we calculate the utility of the updated post i.e. U(p + a j ) ( §2.3). We compose these modules into a joint neural network that we optimize end-to-end over our data ( §2.4).
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Given a post p, our first step is to generate a set of question and answer candidates. One way that humans learn to ask questions is by looking at how others ask questions in a similar situation. Using this intuition we generate question candidates for a given post by identifying posts similar to the given post and then looking at the questions asked to those posts. For identifying similar posts, we use Lucene 4 , a software extensively used in information retrieval for extracting documents relevant to a given query from a pool of documents. Lucene implements a variant of the term frequency-inverse document frequency (TF-IDF) model to score the extracted documents according to their relevance to the query. We use Lucene to find the top 10 posts most similar to a given post from our dataset ( § 3). We consider the questions asked to these 10 posts as our set of question candidates Q and the edits made to the posts in response to the questions as our set of answer candidates A. Since the top-most similar candidate extracted by Lucene is always the original post itself, the original question and answer paired with the post is always one of the candidates in Q and A.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
§3 describes in detail the process of extracting the 4 https://lucene.apache.org/ (post, question, answer) triples from the StackExchange datadump.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Given a post p and a question candidate q i , our second step is to calculate how likely is this question to be answered using one of our answer candidates a j . We first generate an answer representation by combining the neural representations of the post and the question using a function F ans (p,q i ) (details in §2.4). Given such a representation, we measure the distance between this answer representation and one of the answer candidates a j using the function below:
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
dist(Fans(p,qi),âj) = 1 − cos sim(Fans(p,qi),âj)
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
The likelihood of an answer candidate a j being the answer to a question q i on post p is finally calculated by combining this distance with the cosine similarity between the question q i and the question q j paired with the answer candidate a j :
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
whereâ j ,q i andq j are the average word vector of a j , q i and q j respectively (details in § 2.4) and cos sim is the cosine similarity between the two input vectors.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
We model our answer generator using the following intuition: a question can be asked in several different ways. For e.g. in Figure 1, the question "What version of Ubuntu do you have?" can be asked in other ways like "What version of operating system are you using?", "Version of OS?", etc. Additionally, for a given post and a question, there can be several different answers to that question. For instance, "Ubuntu 14.04 LTS", "Ubuntu 12.0", "Ubuntu 9.0", are all valid answers. To generate an answer representation capturing these generalizations, we train our answer generator on our triples dataset ( §3) using the loss function below:
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
where,â andq is the average word vectors of a and q respectively (details in §2.4), cos sim is the cosine similarity between the two input vectors. This loss function can be explained using the example in Figure 3. Question q i is the question paired with the given post p i . In Eq 3, the first term forces the function F ans (p i ,q i ) to generate an answer representation as close as possible to the correct answer a i . Now, a question can be asked in several different ways. Let Q i be the set of candidate questions for post p i , retrieved from the dataset using Lucene ( § 2.1). Suppose a question candidate q j is very similar to the correct question q i ( i.e. cos sim(q i ,q j ) is near zero). Then the second term forces the answer representation F ans (p i ,q i ) to be close to the answer a j corresponding to the question q j as well. Thus in Figure 3, the answer representation will be close to a j (since q j is similar to q i ), but may not be necessarily close to a k (since q k is dissimilar to q i ).
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Given a post p and an answer candidate a j , the third step is to calculate the utility of the updated post i.e. U(p + a j ). As expressed in Eq 1, this utility function measures how useful it would be if a given post p were augmented with an answer a j paired with a different question q j in the candidate set. Although theoretically, the utility of the updated post can be calculated only using the given post (p) and the candidate answer (a j ), empirically we find that our neural EVPI model performs better when the candidate question (q j ) paired with the candidate answer is a part of the utility function. We attribute this to the fact that much information about whether an answer increases the utility of a post is also contained in the question asked to the post. We train our utility calculator using our dataset of (p, q, a) triples ( §3). We label all the (p i , q i , a i ) pairs from our triples dataset with label y = 1. To get negative samples, we make use of the answer candidates generated using Lucene as described in §2.1. For each a j ∈ A i , where A i is the set of answer candidates for post p i , we label the pair (p i , q j , a j ) with label y = 0, except for when a j = a i . Thus, for each post p i in our triples dataset, we have one positive sample and nine negative samples. It should be noted that this is a noisy labelling scheme since a question not paired with the original question in our dataset can often times be a good question to ask to the post ( § 4). However, since we do not have annotations for such other good questions at train time, we assume such a labelling.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Given a post p i and an answer a j paired with the question q j , we combine their neural representations using a function F util (p i ,q j ,ā j ) (details in § 2.4). The utility of the updated post is then defined as U(p i + a j ) = σ(F util (p i ,q j ,ā j )) 5 . We want this utility to be close to 1 for all the positively labelled (p, q, a) triples and close to 0 for all the negatively labelled (p, q, a) triples. We therefore define our loss using the binary cross-entropy formulation below:
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Our fundamental representation is based on recurrent neural networks over word embeddings. We obtain the word embeddings using the GloVe (Pennington et al., 2014) model trained on the entire datadump of StackExchange. 6 . In Eq 2 and Eq 3, the average word vector representationsq andâ are obtained by averaging the GloVe word embeddings for all words in the question and the answer respectively. Given an initial post p, we generate a post neural representationp using a post LSTM (long short-term memory architecture) (Hochreiter and Schmidhuber, 1997). The input layer consists of word embeddings of the words in the post which is fed into a single hidden layer. The output of each of the hidden states is averaged together to get our neural representationp. Similarly, given a question q and an answer a, we generate the neural representationsq andā using a question LSTM and an answer LSTM respectively. We define the function F ans in our answer model as a feedforward neural network with five hidden layers on the inputsp andq. Likewise, we define the function F util in our utility calculator as a feedforward neural network with five hidden layers on the inputsp,q andā. We train the parameters of the three LSTMs corresponding to p, q and a, and the parameters of the two feedforward neural networks jointly to minimize the sum of the loss of our answer model (Eq 3) and our utility calculator (Eq 4) over our entire dataset:
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Given such an estimate P[a j |p, q i ] of an answer and a utility U(p + a j ) of the updated post, we rank the candidate questions by their value as calculated using Eq 1. The remaining question, then, is how to get data that enables us to train our answer model and our utility calculator. Given data, the training becomes a multitask learning problem, where we learn simultaneously to predict utility and to estimate the probability of answers.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
StackExchange is a network of online question answering websites about varied topics like academia, ubuntu operating system, latex, etc. The data dump of StackExchange contains timestamped information about the posts, comments on the post and the history of the revisions made to the post. We use this data dump to create our dataset of (post, question, answer) triples: where the post is the initial unedited post, the question is the comment containing a question and the answer is either the edit made to the post after the question or the author's response to the question in the comments section.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Extract posts: We use the post histories to identify posts that have been updated by its author. We use the timestamp information to retrieve the initial unedited version of the post.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Extract questions: For each such initial version of the post, we use the timestamp information of its comments to identify the first question comment made to the post. We truncate the comment till its question mark '?' to retrieve the question part of the comment. We find that about 7% of these are rhetoric questions that indirectly suggest a solution to the post. For e.g. "have you considered installing X?". We do a manual analysis of  these non-clarification questions and hand-crafted a few rules to remove them. 7
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Extract answers: We extract the answer to a clarification question in the following two ways: (a) Edited post: Authors tend to respond to a clarification question by editing their original post and adding the missing information. In order to account for edits made for other reasons like stylistic updates and grammatical corrections, we consider only those edits that are longer than four words. Authors can make multiple edits to a post in response to multiple clarification questions. 8 To identify the edit made corresponding to the given question comment, we choose the edit closest in time following the question. (b) Response to the question: Authors also respond to clarification questions as subsequent comments in the comment section. We extract the first comment by the author following the clarification question as the answer to the question.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
In cases where both the methods above yield an answer, we pick the one that is the most semantically similar to the question, where the measure of similarity is the cosine distance between the average word embeddings of the question and the answer.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
We extract a total of 77,097 (post, question, answer) triples across three domains in Stack-Exchange (Table 1). We will release this dataset along with the the nine question and answer candidates per triple that we generate using lucene ( § 2.1). We include an analysis of our dataset in the supplementary material.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
We define our task as given a post p, and a set of candidate clarification questions Q, rank the questions according to their usefulness to the post.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Since the candidate set includes the original question q that was asked to the post p, one possible approach to evaluation would be to look at how often the original question is ranked higher up in the ranking predicted by a model. However, there are two problems to this approach: 1) Our dataset creation process is noisy. The original question paired with the post may not be a useful question. For e.g. "are you seriously asking this question?", "do you mind making that an answer?" 9 . 2) The nine other questions in the candidate set are obtained by looking at questions asked to posts that are similar to the given post. 10 This greatly increases the possibility of some other question(s) being more useful than the original question paired with the post. This motivates an evaluation design that does not rely solely on the original question but also uses human judgments. We randomly choose a total of 500 examples from the test sets of the three domains proportional to their train set sizes (askubuntu:160, unix:90 and superuser:250) to construct our evaluation set.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Due to the technical nature of the posts in our dataset, identifying useful questions requires technical experts. We recruit 10 such experts on Upwork 11 who have prior experience in unix based operating system administration. 12 We provide the annotators with a post and a randomized list of the ten question candidates obtained using Lucene ( § 2.1) and ask them to select a single "best" (B) question to ask, and additionally mark as "valid" (V ) other questions that they thought would be okay to ask in the context of the original post. We enforce that the "best" question be always marked as a "valid" question. We group the 10 annotators into 5 pairs and assign the same 100 examples to the two annotators in a pair.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
We calculate the inter-annotator agreement on the "best" and the "valid" annotations using Cohen's Kappa measurement. When calculating the agreement on the "best" in the strict sense, we get a low 9 Data analysis included in the supplementary material suggests 9% of the questions are not useful.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
10 Note that this setting is different from the distractorbased setting popularly used in dialogue (Lowe et al., 2015) where the distractor candidates are chosen randomly from the corpus. 11 https://upwork.com 12 Details in the supplementary material. agreement of 0.15. However, when we relax this to a case where the question marked as"best" by one annotator is marked as "valid" by another, we get an agreement of 0.87. The agreement on the "valid" annotations, on the other hand, was higher: 0.58. We calculate this agreement on the binary judgment of whether a question was marked as valid by the annotator. Given these annotations, we calculate how often is the original question marked as "best" or "valid" by the two annotators. We find that 72% of the time one of the annotators mark the original as the "best", whereas only 20% of the time both annotators mark it as the "best" suggesting against an evaluation solely based on the original question. On the other hand, 88% of the time one of the two annotators mark it as a "valid" question confirming the noise in our training data. 13 Figure 4 shows the distribution of the counts of questions in the intersection of "valid" annotations (blue legend). We see that about 85% of the posts have more than 2 valid questions and 50% have more than 3 valid questions. The figure also shows the distribution of the counts when the original question is removed from the intersection (red legend). Even in this set, we find that about 60% of the posts have more than two valid questions. These numbers suggests that the candidate set of questions retrieved using Lucene ( §2.1) very often contains useful clarification questions.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Our primary research questions that we evaluate experimentally are:
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
1. Does a neural network architecture improve upon non-neural baselines?  Table 2: Model performances on 500 samples when evaluated against the union of the "best" annotations (B1 ∪ B2), intersection of the "valid" annotations (V 1 ∩ V 2) and the original question paired with the post in the dataset. The difference between the bold and the non-bold numbers is statistically significant with p < 0.05 as calculated using bootstrap test. p@k is the precision of the k questions ranked highest by the model and MAP is the mean average precision of the ranking predicted by the model.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
2. Does the EVPI formalism provide leverage over a similarly expressive feedforward network? 3. Are answers useful in identifying the right question? 4. How do the models perform when evaluated on the candidate questions excluding the original?
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
We compare our model with following baselines:
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Random: Given a post, we randomly permute its set of 10 candidate questions uniformly. 14 Bag-of-ngrams: Given a post and a set of 10 question and answer candidates, we construct a bag-of-ngrams representation for the post, question and answer. We train the baseline on all the positive and negative candidate triples (same as in our utility calculator ( §2.3)) to minimize hinge loss on misclassification error using cross-product features between each of (p, q), (q, a) and (p, a). We tune the ngram length and choose n=3 which performs best on the tune set. The question candidates are finally ranked according to their predictions for the positive label.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Community QA: The recent SemEval2017 Community Question-Answering (CQA) (Nakov et al., 2017) included a subtask for ranking a set of comments according to their relevance to a given post in the Qatar Living 15 forum. Nandi et al. (2017), winners of this subtask, developed a logistic regression model using features based on string similarity, word embeddings, etc. We train this model on all the positively and negatively labelled (p, q) pairs in our dataset (same as in our utility calculator ( § 2.3), but without a). We use a subset of their features relevant to our task. 16
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Neural baselines: We construct the following neural baselines based on the LSTM representation of their inputs (as described in §2.4):
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
1. Neural(p, q): Input is concatenation ofp andq. 2. Neural(p, a): Input is concatenation ofp andā. 3. Neural(p, q, a): Input is concatenation ofp,q andā.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Given these inputs, we construct a fully connected feedforward neural network with 10 hidden layers and train it to minimize the binary cross entropy across all positive and negative candidate triples (same as in our utility calculator ( § 2.3)). The major difference between the neural baselines and our EVPI model is in the loss function: the EVPI model is trained to minimize the joint loss between the answer model (defined on F ans (p, q) in Eq 3) and the utility calculator (defined on F util (p, q, a) in Eq 4) whereas the neural baselines are trained to minimize the loss directly on F (p, q), F (p, a) or F (p, q, a). We include the implementation details of all our neural models in the supplementary material.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
We first describe the results of the different models when evaluated against the expert annotations we collect on 500 samples ( §4). Since the annotators had a low agreement on a single best, we evaluate against the union of the "best" annotations (B1 ∪ B2 in Table 2) and against the intersection of the "valid" annotations (V 1 ∩ V 2 in Table 2).
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Among non-neural baselines, we find that the bag-of-ngrams baseline performs slightly better than random but worse than all the other models. The Community QA baseline, on the other hand, performs better than the neural baseline (Neural (p, q)), both of which are trained without using the answers. The neural baselines with answers (Neural(p, q, a) and Neural(p, a)) outperform the neural baseline without answers (Neural(p, q)), showing that answer helps in selecting the right question.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
More importantly, EVPI outperforms the Neural (p, q, a) baseline across most metrics. Both models use the same information regarding the true question and answer and are trained using the same number of model parameters. 17 However, the EVPI model, unlike the neural baseline, additionally makes use of alternate question and answer candidates to compute its loss function. This shows that when the candidate set consists of questions similar to the original question, summing over their utilities gives us a boost.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
The last column in Table 2 shows the results when evaluated against the original question paired with the post. The bag-of-ngrams baseline performs similar to random, unlike when evaluated against human judgments. The Community QA baseline again outperforms Neural(p, q) model and comes very close to the Neural (p, a) model.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
As before, the neural baselines that make use of the answer outperform the one that does not use the answer and the EVPI model performs significantly better than Neural(p, q, a).
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
In the preceding analysis, we considered a setting in which the "ground truth" original question was in the candidate set Q. While this is a common evaluation framework in dialog response selection (Lowe et al., 2015), it is overly optimistic. We, therefore, evaluate against the "best" and the "valid" annotations on the nine other question candidates. We find that the neural models beat the question comment which is not only relevant to the post but will also elicit useful information missing from the post. Hoogeveen et al. (2015) created the CQADupStack dataset using StackExchange forums for the task of duplicate question retrieval. Our dataset, on the other hand, is designed for the task of ranking clarification questions asked as comments to a post.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
We have constructed a new dataset for learning to rank clarification questions, and proposed a novel model for solving this task. Our model integrates well-known deep network architectures with the classic notion of expected value of perfect information, which effectively models a pragmatic choice on the part of the questioner: how do I imagine the other party would answer if I were to ask this question. Such pragmatic principles have recently been shown to be useful in other tasks as well (Golland et al., 2010;Smith et al., 2013;Orita et al., 2015;Andreas and Klein, 2016). One can naturally extend our EVPI approach to a full reinforcement learning approach to handle multi-turn conversations.
Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information
Our results shows that the EVPI model is a promising formalism for the question generation task. In order to move to a full system that can help users like Terry write better posts, there are three interesting lines of future work. First, we need it to be able to generalize: for instance by constructing templates of the form "What version of are you running?" into which the system would need to fill a variable. Second, in order to move from question ranking to question generation, one could consider sequence-to-sequence based neural network models that have recently proven to be effective for several language generation tasks (Sutskever et al., 2014;Yin et al., 2016). Third is in evaluation: given that this task requires expert human annotations and also given that there are multiple possible good questions to ask, how can we automatically measure performance at this task?, a question faced in dialog and generation more broadly (Paek, 2001;Lowe et al., 2015;Liu et al., 2016).
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We present CLARIFYDELPHI, an interactive system that learns to ask clarification questions (e.g., "why did you lie to your friend?") in order to elicit additional salient contexts of a social or moral situation. We posit that questions whose potential answers lead to diverging moral judgments are the most informative. Thus, we propose a reinforcement learning framework with a defeasibility reward that aims to maximize the divergence between moral judgments of hypothetical answers to a question. Human evaluation demonstrates that our system generates more relevant, informative and defeasible questions compared to competitive baselines. Our work is ultimately inspired by studies in cognitive science that have investigated the flexibility in moral cognition (i.e., the diverse contexts in which moral rules can be bent), and we hope that research in this direction can assist both cognitive and computational investigations of moral judgments.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Commonsense moral reasoning of social situations and actions depends squarely on their context. Offering someone a cup of coffee is generally considered appropriate. If offered to a work colleague, it may even be viewed as a courteous gesture. However, offering coffee to a toddler would be deemed morally irresponsible.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Delphi (Jiang et al., 2022), a recently proposed commonsense moral reasoning model, generates moral judgments for simple actions described in text. However, Delphi's judgments are made in isolation, without any knowledge of surrounding context. Grounding moral reasoning in context is crucial (Talat et al., 2022). How can moral reason-
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Who did you offer it to?
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
When did you offer it?
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
in the morning to a toddler to a work colleague
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Low Divergence
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Policy Reward in the afternoon High Divergence
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
"offering a cup of coffee" Figure 1: The CLARIFYDELPHI question generation approach is trained via reinforcement learning. The reward simulates a set of possible (defeasible) answers to the questions and, using Delphi for feedback, optimizes for questions leading to maximally diverging answers. ers elicit missing salient context? A natural way to do so is by asking clarification questions.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We present CLARIFYDELPHI, an interactive system that learns to ask questions to elicit salient context. Prior research in cognitive science shows that human reasoning exhibits the flexibility not only to articulate where a certain moral rule should hold, but also to imagine valid exceptions where the rule can be bent or defeated based on the demands of the context (Kwon et al., 2022;Levine et al., 2020;Awad et al., 2022).
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We present a first step toward computationally exploring and discovering these defeasible contexts which can potentially flip the moral judgement of a situation. Given a situation and its default judgment (e.g., it is nice to offer a cup of coffee to someone), defeasible contexts can strengthen (e.g., offering it to a colleague) or weaken (e.g., giving it (a) (b) (c) Figure 2: Interaction between a user and CLARIFYDELPHI. The user inputs a situation and CLARIFYDELPHI answers with an initial judgement (obtained from DELPHI) and a clarification question, which the user then answers.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
to a toddler) the judgment (Rudinger et al., 2020;Madaan et al., 2021;Allaway et al., 2022). We aim to generate questions whose answers might uncover missing context for making better-informed moral judgments, and we propose to do so in a conversational setting between a user and CLARI-FYDELPHI.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Our method for clarification question generation is based on reinforcement learning. Using Proximal Policy Optimization (PPO; Schulman et al. 2017;Ouyang et al. 2022) we optimize for generating questions that invoke responses that provide morally salient contexts. CLARIFYDELPHI "imagines" answers to a generated question, using a trained answer generation model. A reward is calculated by comparing the probability distributions Delphi assigns to the imagined answers. Fig. 1 provides an overview of CLARIFYDELPHI.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
The intuition behind our approach is that questions that lead to maximally divergent answers (e.g., "Who did you offer it to?") are also those that elicit most morally salient contexts and therefore are more consequential to the situation. These morally consequential questions surface latent ambiguities that may directly affect the moral decision process. Questions with little divergence in its imagined answers (e.g., "When did you offer it?") have little to offer in terms of resolving contextual moral ambiguities.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Our results show that our approach outperforms other strong clarification question generation baselines; its generated questions lead to consequential answers. We additionally quantify how much supervised clarification question training data is needed for a good initial policy. Lastly we show that questions help with generating defeasible updates.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Our contributions are as follows. We introduce the task of clarification question generation for social and moral situations. For this task we propose an RL based approach, defining defeasibility as a new type of relevance for clarification questions. We publicly release δ-CLARIFY, a dataset of 33k crowdsourced clarification questions, and δ-CLARIFY silver containing generated questions conditioned on a defeasible inference dataset. We also release trained models with their code. 1
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Given a situation, such as lie to my friend, we aim to generate question(s) that are the most relevant for uncovering the most consequential context with respect to making a social or moral judgement. While situations could evoke a multitude of potential questions, the following work is concerned with predicting questions whose answers are likely to be consequential, i.e. answers that could function as either weakeners or strengtheners of the default judgement. The terms weakener and strengthener come from the concept of defeasible inference (Rudinger et al., 2020), which defines a way of reasoning that takes into consideration (new) evidence which could either support (e.g. strengthen) or cancel/weaken an initial inference.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Formally, the task is to predict a question q given a base situation s. The base situation has a default moral judgement j ∈ {bad, ok, good}. For every input tuple of (s i , q i , j i ) there is a hypothetical set of strengthening answers A S and weakening answers A W . Adding the additional information obtained from any q i and corresponding answer a i to Algorithm 1 Training CLARIFYDELPHI Input initial policy model θ0, initial value model ϕ0, Delphi ψDelphi D δ-CLARIFY ← Get dataset of clarification questions. θQ ← Fine-tune θ0 with Eqn 1 from D δ-CLARIFY . ▷ Sec. 3.1 D δ-CLARIFY silver ← Get silver dataset of defeasible answers to questions.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
θA ← Fine-tune lm with Eqn 2 from D δ-CLARIFY . ▷ Sec. 3.2 θCLARIFYDELPHI ← REINFORCEDLEARNING(SSocialChem, θQ, θA, ϕ0, ψDelphi) ▷ Sec. 3.4 procedure REINFORCEDLEARNING (SSocialChem, θQ, θA, ϕ, ψDelphi) θQ old ← θQ, ϕold ← ϕ for iterations = 1, 2, . . . do Sample a minibatch of situations s from S SocialChem .
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
for step = 1, 2, . . . , s do Calculate r using θA and ψDelphi with Eqn 3. Compute lossPPO on the minibatch with Eqn 6. Optimize θ and ϕ with LPPO for one step.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
θQ old ← θQ, ϕold ← ϕ return θQ
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Output θCLARIFYDELPHI the base situation s i results in an updated situation s ui , with an updated judgement j ui .
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
The CLARIFYDELPHI approach is based on reinforcement learning. Algorithm 1 gives an overview of the training process. As a first step, before performing reinforcement learning, we obtain a question generation model θ Q and an answer generation model θ A , which we both train on data that we curated, described in the later Sec. 4. The question generation model predicts the clarification questions and the answer generation model provides (defeasible) answers to the generated questions. By using these two models in addition to Delphi (ψ Delphi ) for calculating the rewards, we do not require any supervised data during RL training. We consider question generation conditioned on a given situation a sequential decision making process over the natural language vocabulary space, where the generated question q with T tokens has an episode of length T . At step t ∈ [1, T ], the state s t = (s, q <t ) is the combination of the given situation and the question decoded up to the (t − 1)-th token; the action c t = q t would be the t-th token to decode. The question generation model, θ Q (q t |q, q <t ; θ), is the policy model that we optimize. We define a reward function r(s, q, a w , a s ) that characterizes the divergence of answers from θ A conditioned on generated question q and discuss the definition of this reward function in §3.3.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
The first subcomponent is a basic question generation system θ Q that outputs a question q conditioned on a situation s. It is used as the initial policy model during RL training.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
For each generated question q, we need to generate a weakening answer a w and a strengthening answer a s in order to calculate the reward r (Formula 3). For the defeasible answer generation system θ A , we take as input a situation s i , the generated question q i ( §3.1), and an update type u ∈ {weakener, strengthener} to predict a weakener-strengthener answer pair a w and a s :
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
An example of an instantiated input/output:
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Input It's bad to be a snitch, TYPE: Weakener, Q.: Why would being a snitch be beneficial? Output doing so would save someones life.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
The crucial element in the input is the update type, as it allows to generate two types of answers for the same s and q. When computing the reward during training, for each question, we filter out all its generated answers which either contradict or are entailed (i.e. no new information) by the given situation, using an off-the-shelf NLI model.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
As a reward for generating a question, we aim to quantify how well the generated questions are able to elicit consequential answers. For this purpose we query Delphi (Jiang et al., 2022) for feedback, using situations updated with answers.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We optimize for questions that lead to maximally divergent answers by defining a reward function which uses the JS-Divergence, between the Delphi probability distribution of the weakener updated situation and the strengthener updated situation:
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Sentence Fusion To create an updated situation that sounds natural and can be used to query Delphi, the situation s, question q i and answer (both a w and a s separately) have to be fused together into s ui . For example: Situation refraining from doing something bad Question When do you do something bad?
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Answer when I'm angry Fusion: refraining from doing something bad when you're angry. We train a model to distill fusion in-context examples obtained from GPT-3 (text-curie-001).
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Delphi for Feedback Delphi is then queried with the updated situation s ui for a judgement, leveraging the probability distribution that Delphi provides over three classes: j ∈ {bad, ok, good}. The probability scores are the probabilities of the special T5 tokens representing each of the three classes, normalized by their sum.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
JS-Divergence We calculate the Jensen-Shannon divergence between the Delphi probability distributions j w and j s obtained from two updated situations originating from defeasible answers to q.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We normalize the reward during training as follows:
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
The µ 0 and σ 0 of rewards are obtained before training begins, by generating a question and calculating its rewards for all s in the training data.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We maximize the reward using Proximal Policy Optimization (PPO) (Schulman et al., 2017) as our RL algorithm, which previous works have shown to be suitable for NLG tasks (Liu et al., 2022b;Ramamurthy et al., 2022). Our implementation of PPO is an adaptions of Ouyang et al. (2022)'s, which includes a KL penalty between the initial policy model θ Q old and the updated policy θ Q . In addition to the policy model, PPO employs a value model (parameterized by ϕ) to estimate the value function for states with incomplete decoded text, i.e. V (s t ; ϕ) for any t. PPO's loss consists of a value model (ϕ) loss and the policy loss, which is  jointly minimized:
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We require data for various components of our CLARIFYDELPHI model: The policy needs bootstrapping from a clarification question dataset and the answer generation model needs data to learn to generate defeasible answers to questions. To the best of our knowledge no such datasets exist. We therefore collect a crowdsourced dataset of clarification question for social and moral situation and a silver dataset of defeasible QAs to train θ Q . The situations are sampled from SOCIAL-CHEM-101 (Forbes et al., 2020) and the COMMONSENSE NORM BANK (Jiang et al., 2022). We call our dataset δ-CLARIFY and it consists of crowdsourced questions, enriched with questions generated by GPT-3 (Brown et al., 2020).
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Next we describe how we create the dataset.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Situation: Tipping people decently Q1 What did they do for you? Q2
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Can you afford to tip? Q3
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Was the service good? Q4
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Did the people perform the service adequately? Q5
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Do you always tip people well regardless of the service quality?
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Situation: Jeff ignores the comment and laughs about it with his boss Q1-4 What was the comment? Q5
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Who made the comment they were laughing at? δ-CLARIFY gold : We crowdsource clarification questions by showing annotators a situation and asking them to write a clarification question they would ask an imagined colleague requesting advice on the situation. Each of the 6425 situations is presented to 5 annotators, resulting in 5 questions per situation (500 situations are used for dev and test respectively). Details of the annotation are found in Appendix A.1.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
δ-CLARIFY silver : The δ-SOCIAL part of the defeasible inference dataset (Rudinger et al., 2020) consists of statements that express default judgments over situations (It is good to protect your kids) and updates that weaken (Your protection is overbearing) or strengthen (Your child is in danger) the default. These updates could be viewed as potential answers to an implicit question about a base situation: "What are you protecting your child from?" for Your child is in danger. We 5-shot prompt GPT-3 to generate questions, conditioned on situation and answer, resulting in ≈ 80K (situation, update type, question, answer) tuples.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Dataset Analysis Fig. 3 shows that the crowdsourced δ-CLARIFY gold has more variety in its most common question starts, which reflects the general diversity of the dataset: For only 10% of the situations, more than 1 Turker asked exactly the same question, and for only 8% of the situations all 5 Turkers used the same Wh-word to start the question. This highlights that there is more than one possible salient clarification question to be asked for any given situation. For the situation tipping people decently in Tab. 1, all 5 Turkers chose to start their questions differently, even though three out of these five questions ask in one way or the other about the service quality. For the other situation 4/5 Turkers asked for a specification "What was the comment?" and 1 Turker asked about the missing agent role. We also see that polar (yes/no) questions appear less frequently, as Turkers were explicitly asked to avoid them unless no other suitable question comes to mind. 2 The δ-CLARIFY silver questions are generated by conditioning on weakener or strengthener updates. Since we aim to predict defeasible questions, the most desirable questions are those whose answers can be both weakeners and strengtheners.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
In the silver data, 53% of situations have at least one question that has been generated by GPT-3 for both update types. The situation Your kids should be your number one priority, for example, has the same question "What are your kids' ages?" for the weakener update They are adult children. and the strengthener update Your children are toddlers. Interestingly, among the subset of defeasible questions in δ-CLARIFY silver , we find that the most frequent question start is 'why'. This suggests that it is easiest to come up with both weakener and strengthener answers to why-questions.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We consider four baselines in our experiments.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Question Generation Without RL To assess what additional improvements training an RL model with a defeasibility rewards provides, we report performance of the supervised question generation model θ Q on its own ( §3.1). We refer to this baseline as t5 fine-tuned. We decode using nucleus sampling with top-p = 0.6.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Pipelines with Question Selection Next, we implement two pipelines where, as the first step, a diverse set of questions is generated for a given situation and then, as the second step, the best question is selected according to a score.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
In order to generate a diverse set of questions we fine-tune T5 on δ-CLARIFY, conditioned on a modified input compared to the model from §3.1: Input <Situation>. Q.: <wh-word> -Output <Question> By also conditioning on the first wh-word of the question it is possible to generate different questions. During inference we generate questions for 14 different question starts. 3 We propose two approaches to scoring these questions: using a discriminator model and using divergence ranking, which we describe as follows.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Discriminator We train a discriminator classifier which labels these questions as either relevant or irrelevant to a given situation. We then choose the question that has been assigned the relevant label with the highest probability.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
The discriminator is a binary classifier based on DeBERTa ( Divergence Ranking We run the defeasible answer simulation with feedback from Delphi for each question in the set. This process is the same as the reward function of the RL approach, except that the JS-divergence score is used to rank the questions instead of being used as a reward for question generation. We compare two variations of this baseline: one with answer filtering using an NLI model as described in Sec. 3.2 (pipeline-nli) and one without filtering (pipeline).
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We saw in §4 that questions conditioned on weakener/strengthener updates are usually causal questions. Using the same input/output configuration as in the pipeline baseline, we generate a why-question for each situation (called why).
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Automatic evaluation of questions and their usefulness for clarifying moral situations is tricky. While we do have gold reference questions, we have shown that humans will produce diverse questions for the same situation ( §4) and just because a question does not appear in the reference set does not necessarily indicate that it is not a consequential question. We therefore perform human evaluation of the models' outputs on Amazon Mechanical Turk on the 500 test set instances from δ-CLARIFY. Given a situation and a question, Turkers are asked to rate the question along three different attributes: Grammaticality (Is the question grammatical?), Relevance (Is the question relevant and plausible to the situation?), and Informativeness (Does the question access new information or regurgitate the situation?). The attributes are further detailed in Appendix A.1.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Additionally, and most importantly, we aim to evaluate the defeasibility of the questions, e.g. how well the generated questions can elicit weakener or strengthener answers. For this purpose, Turkers are given a situation with a question and are first asked to judge this situation (generally ok vs. generally not ok). They then need to say whether and specify if they can think of an answer to the question which might support their judgement and also of an answer which would flip their judgement. 
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We first run the grammaticality, relevance and informativeness evaluation. All questions which are given the lowest rating (e.g. irrelevant and/or uninformative) by at least two annotators are excluded from the second evaluation. It does not make sense to ask about defeasibility for questions which already are irrelevant, and additional weakening or strengthening context is not feasible for uninformative questions.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We find, as displayed in Fig. 4, that CLARIFY-DELPHI has the biggest percentage of relevant and informative questions in the test set, compared to the baselines. We also see that a big majority of the generated questions, from all models, are relevant and informative, with the lowest performing model (discriminator) still producing 448/500 questions that are passed on to the next evaluation round.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We also find that grammaticality across all systems is high with the lowest average score being 0.98 and the highest 0.99 (on a scale from 0 to 1, with 1 being grammatical). The minimal variation in grammaticality score is expected since all models are based upon the same transformer model.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
The CLARIFYDELPHI questions also outperform the baselines in terms of defeasibility, as seen in Table 2: annotators can more often think of a strengthener answer and/or a weakener answer to our questions. The evaluation also shows that adding the answer-filtering with NLI step to the pipeline improves the question selection on all 4 evaluation criteria. The why-baseline is shown to be a strong baseline, indicating that motives and reasons are important for moral reasoning.  Table 2: Defeasibility scores obtained through human evaluation. weakener: Can you think of an answer to the question that weakens your initial judgement? strengthener: Can you think of an answer ot the question that strengthens your intial judgement? defeasibility: Can you think of both?.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Our approach uses RL in conjunction with a supervised policy that has been fine-tuned on question generation. This has been shown to outperform approaches which use RL on top of a "vanilla" lmpolicy (Ramamurthy et al., 2022). To assess the effect of supervision on question generation performance, we trained multiple initial policies on varying percentages of δ-CLARIFY training data: 25%, 50%, 75% and 100%. To compare against more traditional supervised question generation approaches we additionally trained a policy on SQuAD v1.1 data (Rajpurkar et al., 2016). We report two automatic evaluation metrics. To measure informativeness we use an off-the-shelf QA model trained on SQuAD 2.0 from AllenNLP (Gardner et al., 2018). This model either answers a question by pointing to a span in the input or outputs that the question in unanswerable with respect to a given context. For a clarification question to be informative it would not ask about anything already mentioned in the situation. For the QA-metric we thus report the percentage of non-answerable questions. 4 We also report the average maximum BERTScore (Zhang et al., 2019) between a generated question and one of the 5 gold questions in δ-CLARIFY. Fig. 5 shows the following trends with regards to training a supervised policy. More training data leads to more informative questions. The policy trained on SQuAD produces the most uninformative questions which can be explained by the fact that SQuAD questions are conditioned on existing answers in a text. While performance consistently increases from 25% to 75% of the training data, improvements after 75% are minimal. We conclude that for our use case training on about 5000 (75%) situations with 5 questions each leads to a sufficiently good policy. These results are also supported by the BERTScore.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
25% 50% 75% 100%
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
training data type 
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Answer Simulation The answer generation model generally succeeds at generating diverse weakener and strengthener answers to the same question: for only about 0.05% of questions per 1000 PPO epochs the model generates the same answer for both weakener and strengthener.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Our answer generation could be looked at as question-guided defeasible update generation. Rudinger et al. (2020)'s task of Generative Defeasible Inference generates an update given a situation, a moral judgement and the update type (e.g. weakener/strengthener). In our answer generation approach we condition on the same input together with a generated question. This intermediate question generation step functions as a type of macro planning which has been shown to be effective for NLG (Puduppully et al., 2022;Narayan et al., 2022). We evaluate our approach on the same test set using the same evaluation metrics as Rudinger et al. (2020). Table 3 shows that by first predicting the question and then the updates, we improve upon generating defeasible updates for δ-SOCIAL.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Questions We qualitatively inspect the types of generated questions: There are many specification questions asking about a hyponym of an argument   Rudinger et al. (2020) we report BLEU-4 (Papineni et al., 2002) and ROUGE-L (Lin and Hovy, 2002). The first two results are from Rudinger et al. (2020). in the base situation, for example, exterminating pests on your property -"What kind of pests?". The situations extracted from SocialChem often include underspecified pronouns, such as 'something' or 'somewhere'. 60% of the situations containing 'something', for example, elicit what-questions from our model. Note that while such specification questions are valid clarification questions, the SQUAD 2.0 QA model would mark them as answerable given the situation. It is also interesting to see that often when a situation has a missing or implicit semantic argument, such as being anxious sometimes, CLARIFYDELPHI inquires about it: "What are you anxious about?" The generated why-questions most often ask about the motives of the agent in the situation, such as Ben tells Maggie that he's traveling alone -"Why is Ben traveling alone?". More rarely the model generates questions asking about the viewpoint of the patient: asking a friend [...] whether your attire is appropriate for an event -"What is the friend's opinion of your attire?"
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Analysis of Delphi's Probabilities In Tab. 4 we quantify the JSD of Delphi's judgments. Even though the human evaluation showed that CLAR-IFYDELPHI produced the most questions leading to defeasible answers, the JSD and the precentage of judgment flips is higher for the pipeline_nli approach, where we explicitly filter questions to maximize the JSD. Nevertheless, CLARIFYDEL-PHI leads to more Delphi judgment flips and higher JSD between answers than the fine-tuned t5 model without RL (and also all other baselines besides the pipeline). This automatic evaluation and the disagreement with the human annotators also reveals that Delphi's probabilities are not always perfectly calibrated and relying too much on a model's output might potentially lead to some error propagation.  Table 4: Average JSD between P jw and P js of a situation. Judgment Flips: % of answers which led to a flip in Delphi's judgment.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
While we use answer simulation during PPO training, inference only requires a situation as input.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
The clarification questions can then be used to elicit additional context, in the form of answers, through interaction. Fig. 2 illustrates examples of such an interaction between a user, Delphi as the moral reasoning system and CLARIFYDELPHI. After each turn, the situation is updated with the user provided context, for which Delphi produces a new decision. We limit the interaction to three turns. This is based on the observation that after the third turn the sentence fusion starts to deteriorate, resulting in less relevant and more repetitive questions. Additionally, we find that the first two questions generally can capture missing contexts that are most central to making moral decisions. We provide more examples of generated questions in the Appendix.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
In this work we introduce CLARIFYDELPHI, which generates clarification questions for social and moral situations. We show how a RL approach that optimizes for maximally divergent answers in terms of defeasibility outperforms other clarification question baselines. While we start with a supervised policy, the reward function makes use of already trained models and does not rely on any additional training data. We believe that our questions can be useful for providing more disambiguating context through interaction.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
On Western-centricity The majority of the crowdworkers producing the source data (δ-Social and Delphi) and δ-CLARIFY were located in the United States. Due to this, the predictions generated by CLARIFYDELPHI are currently limited to representing only the perspectives of western culture (particularly the United States). Overcoming the western-centric bias is a compelling direction for future research.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
On Defeasibility We rely upon Delphi to produce acceptable judgments given a situation and the modifying context as a measure of defeasibility. We recognize that, however, Delphi is not perfect and is characterized by a variety of limitations such as limited cultural awareness and inconsistent predictions (Jiang et al., 2022). Investigating improved methods for identifying answer divergences that will better capture defeasibility is a topic for future investigation.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Annotations are conducted on Amazon Mechanical Turk (MTurk). We maintain an average pay of $15 per hour for all our crowdsourcing data collection and evaluation tasks. Our crowdsourcing tasks do not collect personal information and are strictly limited to gathering workers' general knowledge. We do not keep any deanonymizing information such as MTurk IDs so that the identity of the workers cannot be directly or indirectly ascertained. Finally, our crowdsourcing task meets the standards for exemptions as human research and has obtained the necessary documentation that deems it exempt through our internal IRB procedures.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Our model is intended to be used for research purposes only and it is not supposed to provide any sort of advice applicable outside of that domain. already mentioned in the situation? very/somewhat/uninformative A screenshot for the evaluation HIT w.r.t grammaticality, relevance, and informativeness is found in Figures 8. Additionally, we evaluate the defeasibility of a question. A screenshot of the defeasibility evaluation is shown in Figure 10.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
IRB approval: We sought and received exemption from our internal IRB. In accordance to the regulations, we do not collect sensitive information. If we do publish WorkerIDs, we will do so by fully anonymizing the information. The exemption received does not require a consent form.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We have not collected any demographic information from the workers. However, all crowdsourcing was conducted in English and the region (current location of the crowdworker) was set to US. Consequently, what counts as a context of consequence is centered around western views, or views of the English speaking cultures within the United States.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
One way to elicit a set of opposing answers is through prompting. We instruct GPT-3 to provide a so-called "bad" and a so-called "good" answer to a question about a situation. For the situation learning how to take a joke and the question "What was the joke?", the two answers could be: "it was a lighthearted joke among friends" and "it was an offensive joke". In order to determine which of the answers is a weakener and which a strengthener, we compare the difference in Delphi's judgement for s and s + a good or s + a bad .
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Policy loss. To compute the policy loss, we first define the truncated estimated advantage function,
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
where the value function of a certain state s t is estimated by the value model V ϕ (·), r t denotes intermediate reward obtained at time step t, and γ and λ denote reward decay factor. PPO then maximizes the empirical expectation of clipped surrogate objective term, defined as cso(Â t , r t (θ), ϵ) = min r t (θ)Â t , clip(r t (θ), 1 − ϵ, 1 + ϵ)Â t ,
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Value loss. The value model is trained with MSE loss with respect to the target value, V targ t :
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
The question generation model θ Q is trained with T5-large (Raffel et al., 2020) on δ-CLARIFY.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We train the answer generation model θ A (T5large) on 77k instances from δ-CLARIFY silver ( §4) and about 4k instances obtained through prompting GPT-3. For the answer-filtering using NLI we use WaNLI (Liu et al., 2022a) as an off-the-shelf NLI model.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We run the PPO algorithm for 6000 steps, with a batch size of 64, top_p= 0.6 and a temperature of 0.7 for the sampling policy. Training for 6000 step took approximately 2 days on 4 A6000, 48G, GPUs.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Our models are based on T5-large which has 770 million parameters.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
For the DELPHI model we use the t5 − large variant.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
CLARIFYDELPHI and the code will be released under the Apache License 2.0. We will release the dataset under CC-BY.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We show a sample of 15 generated questions in Tab. 5.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
We show a sample of generated questions in a twoturn interactive setting in Tab. 6. What are you doing when you have to get up? Paying more when you are able to because you make more.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
What are you making? Apologizing when you have done something wrong. What did you do? being careful when opening your car door in a parking lot.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Are you aware of what is inside your car? being out after curfew.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Are you allowed to be out after curfew? respecting your friend's wishes What is your friend's wish? making a scene What kind of scene? being friends with benefits with someone Why are you friends with benefits with them? not liking certain dogs Why don't you like them? smoking in one's own home What are the risks? wanting to go home with someone
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
How do you feel about them? being angry with your partner when they hide things What are they hiding? Talking about things you know nothing about
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Why are you talking about these things? agreeing to go out with someone who bullied you Why are they bullying you? using your money how you see fit
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
What are you doing with your money?      getting cold feet before a wedding Why are you getting cold feet? getting cold feet before a wedding because you only know your partner for 1 month What is your relationship status?
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Being mindful of your health How are you being mindful? being mindful of your health by working out What are the benefits of working out? Table 6: Sample of situations from the dev set of SocialChem, their expansion through interaction (2 turns) and the CLARIFYDELPHI generated question.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
A For every submission:
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
A1. Did you describe the limitations of your work?
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
A2. Did you discuss any potential risks of your work?
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
A3. Do the abstract and introduction summarize the paper's main claims?
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Section 0 and Section 1 A4. Have you used AI writing assistants when working on this paper?
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Left blank.
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
B Did you use or create scientific artifacts?
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
Section 4 B1. Did you cite the creators of artifacts you used?
ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations
B2. Did you discuss the license or terms for use and / or distribution of any artifacts?