How did prior work explore learning from human feedback for different NLP tasks?
Learning from Human Feedback. Human feedback has been explored for different NLP tasks such as machine translation (Nguyen et al., 2017; Kreutzer and Riezler, 2019; Mendonça et al., 2021), semantic parsing (Lawrence and Riezler, 2018; Yao et al., 2020; Elgohary et al., 2021), document summarization (Gao et al., 2018; Stiennon et al., 2020), question answering (Kratzwald et al., 2020; Gao et al., 2022), and dialogue systems (Shuster et al., 2022). 
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
In Machine Translation, assessing the quality of a large amount of automatic translations can be challenging. Automatic metrics are not reliable when it comes to high performing systems. In addition, resorting to human evaluators can be expensive, especially when evaluating multiple systems. To overcome the latter challenge, we propose a novel application of online learning that, given an ensemble of Machine Translation systems, dynamically converges to the best systems, by taking advantage of the human feedback available. Our experiments on WMT’19 datasets show that our online approach quickly converges to the top-3 ranked systems for the language pairs considered, despite the lack of human feedback for many translations.
[{"first":"Vânia","last":"Mendonça","middle":[]},{"first":"Ricardo","last":"Rei","middle":[]},{"first":"Luisa","last":"Coheur","middle":[]},{"first":"Alberto","last":"Sardinha","middle":[]},{"first":"Ana Lúcia","last":"Santos","middle":[]}]
Learning a Cost-Effective Annotation Policy for Question Answering
State-of-the-art question answering (QA) relies upon large amounts of training data for which labeling is time consuming and thus expensive. For this reason, customizing QA systems is challenging. As a remedy, we propose a novel framework for annotating QA datasets that entails learning a cost-effective annotation policy and a semi-supervised annotation scheme. The latter reduces the human effort: it leverages the underlying QA system to suggest potential candidate annotations. Human annotators then simply provide binary feedback on these candidates. Our system is designed such that past annotations continuously improve the future performance and thus overall annotation cost. To the best of our knowledge, this is the first paper to address the problem of annotating questions with minimal annotation cost. We compare our framework against traditional manual annotations in an extensive set of experiments. We find that our approach can reduce up to 21.1% of the annotation cost.
[{"first":"Bernhard","last":"Kratzwald","middle":[]},{"first":"Stefan","last":"Feuerriegel","middle":[]},{"first":"Huan","last":"Sun","middle":[]}]
Learning to summarize from human feedback
As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about -- summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles, producing summaries nearly as good as the human reference without any news-specific fine-tuning. We conduct extensive analyses to understand our human feedback dataset and fine-tuned models We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.
[{"first":"Nisan","last":"Stiennon","middle":[]},{"first":"Long","last":"Ouyang","middle":[]},{"first":"Jeff","last":"Wu","middle":[]},{"first":"Daniel","last":"Ziegler","middle":["M."]},{"first":"Ryan","last":"Lowe","middle":[]},{"first":"Chelsea","last":"Voss","middle":[]},{"first":"Alec","last":"Radford","middle":[]},{"first":"Dario","last":"Amodei","middle":[]},{"first":"Paul","last":"Christiano","middle":[]}]
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Machine translation is a natural candidate problem for reinforcement learning from human feedback: users provide quick, dirty ratings on candidate translations to guide a system to improve. Yet, current neural machine translation training focuses on expensive human-generated reference translations. We describe a reinforcement learning algorithm that improves neural machine translation systems from simulated human feedback. Our algorithm combines the advantage actor-critic algorithm (Mnih et al., 2016) with the attention-based neural encoder-decoder architecture (Luong et al., 2015). This algorithm (a) is well-designed for problems with a large action space and delayed rewards, (b) effectively optimizes traditional corpus-level machine translation metrics, and (c) is robust to skewed, high-variance, granular feedback modeled after actual human behaviors.
[{"first":"Khanh","last":"Nguyen","middle":[]},{"first":"Hal","last":"Daumé III","middle":[]},{"first":"Jordan","last":"Boyd-Graber","middle":[]}]
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
We study semantic parsing in an interactive setting in which users correct errors with natural language feedback. We present NL-EDIT, a model for interpreting natural language feedback in the interaction context to generate a sequence of edits that can be applied to the initial parse to correct its errors. We show that NL-EDIT can boost the accuracy of existing text-to-SQL parsers by up to 20% with only one turn of correction. We analyze the limitations of the model and discuss directions for improvement and evaluation. The code and datasets used in this paper are publicly available at http://aka.ms/NLEdit.
[{"first":"Ahmed","last":"Elgohary","middle":[]},{"first":"Christopher","last":"Meek","middle":[]},{"first":"Matthew","last":"Richardson","middle":[]},{"first":"Adam","last":"Fourney","middle":[]},{"first":"Gonzalo","last":"Ramos","middle":[]},{"first":"Ahmed Hassan","last":"Awadallah","middle":[]}]
Self-Regulated Interactive Sequence-to-Sequence Learning
Not all types of supervision signals are created equal: Different types of feedback have different costs and effects on learning. We show how self-regulation strategies that decide when to ask for which kind of feedback from a teacher (or from oneself) can be cast as a learning-to-learn problem leading to improved cost-aware sequence-to-sequence learning. In experiments on interactive neural machine translation, we find that the self-regulator discovers an \epsilon-greedy strategy for the optimal cost-quality trade-off by mixing different feedback types including corrections, error markups, and self-supervision. Furthermore, we demonstrate its robustness under domain shift and identify it as a promising alternative to active learning.
[{"first":"Julia","last":"Kreutzer","middle":[]},{"first":"Stefan","last":"Riezler","middle":[]}]
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Counterfactual learning from human bandit feedback describes a scenario where user feedback on the quality of outputs of a historic system is logged and used to improve a target system. We show how to apply this learning framework to neural semantic parsing. From a machine learning perspective, the key challenge lies in a proper reweighting of the estimator so as to avoid known degeneracies in counterfactual learning, while still being applicable to stochastic gradient optimization. To conduct experiments with human users, we devise an easy-to-use interface to collect human feedback on semantic parses. Our work is the first to show that semantic parsers can be improved significantly by counterfactual learning from logged human feedback data.
[{"first":"Carolin","last":"Lawrence","middle":[]},{"first":"Stefan","last":"Riezler","middle":[]}]
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
We propose a method to perform automatic document summarisation without using reference summaries. Instead, our method interactively learns from users’ preferences. The merit of preference-based interactive summarisation is that preferences are easier for users to provide than reference summaries. Existing preference-based interactive learning methods suffer from high sample complexity, i.e. they need to interact with the oracle for many rounds in order to converge. In this work, we propose a new objective function, which enables us to leverage active learning, preference learning and reinforcement learning techniques in order to reduce the sample complexity. Both simulation and real-user experiments suggest that our method significantly advances the state of the art. Our source code is freely available at https://github.com/UKPLab/emnlp2018-april.
[{"first":"Yang","last":"Gao","middle":[]},{"first":"Christian M.","last":"Meyer","middle":[]},{"first":"Iryna","last":"Gurevych","middle":[]}]
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We present BlenderBot 3, a 175B parameter dialogue model capable of open-domain conversation with access to the internet and a long-term memory, and having been trained on a large number of user defined tasks. We release both the model weights and code, and have also deployed the model on a public web page to interact with organic users. This technical report describes how the model was built (architecture, model and training scheme), and details of its deployment, including safety mechanisms. Human evaluations show its superiority to existing open-domain dialogue agents, including its predecessors (Roller et al., 2021; Komeili et al., 2022). Finally, we detail our plan for continual learning using the data collected from deployment, which will also be publicly released. The goal of this research program is thus to enable the community to study ever-improving responsible agents that learn through interaction.
[{"first":"Kurt","last":"Shuster","middle":[]},{"first":"Jing","last":"Xu","middle":[]},{"first":"Mojtaba","last":"Komeili","middle":[]},{"first":"Da","last":"Ju","middle":[]},{"first":"Eric","last":"Smith","middle":["Michael"]},{"first":"Stephen","last":"Roller","middle":[]},{"first":"Megan","last":"Ung","middle":[]},{"first":"Moya","last":"Chen","middle":[]},{"first":"Kushal","last":"Arora","middle":[]},{"first":"Joshua","last":"Lane","middle":[]},{"first":"Morteza","last":"Behrooz","middle":[]},{"first":"William","last":"Ngan","middle":[]},{"first":"Spencer","last":"Poff","middle":[]},{"first":"Naman","last":"Goyal","middle":[]},{"first":"Arthur","last":"Szlam","middle":[]},{"first":"Y-Lan","last":"Boureau","middle":[]},{"first":"Melanie","last":"Kambadur","middle":[]},{"first":"Jason","last":"Weston","middle":[]}]
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We study learning from user feedback for extractive question answering by simulating feedback using supervised data. We cast the problem as contextual bandit learning, and analyze the characteristics of several learning scenarios with focus on reducing data annotation. We show that systems initially trained on few examples can dramatically improve given feedback from users on model-predicted answers, and that one can use existing datasets to deploy systems in new domains without any annotation effort, but instead improving the system on-the-fly via user feedback.
[{"first":"Ge","last":"Gao","middle":[]},{"first":"Eunsol","last":"Choi","middle":[]},{"first":"Yoav","last":"Artzi","middle":[]}]
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
In Machine Translation (MT), measuring the quality of a large amount of automatic translations can be a challenge. Automatic metrics like BLEU (Papineni et al., 2002) remain popular due to their fast and free computations. Yet, in the last few years we have seen that, as MT quality improves, automatic metrics become less reliable (Ma et al., 2019;Mathur et al., 2020). For example, in the Conference on Machine Translation (WMT)'19 News Translation shared task, the winning system according to human annotators was not even in the top-5 according to BLEU (Barrault et al., 2019). On the other hand, using human assessments can be expensive, especially when evaluating multiple systems. In a real world scenario, given an arbitrary number of MT systems, one would need to evaluate them individually to find the best systems for a given language pair. However, that requires a considerable effort and there may not be enough human annotators to evaluate all the systems' translations. For instance, in the aforementioned WMT'19 shared task, many translations from the competing systems did not receive any human assessment.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
Given an ensemble of competing, independent MT systems, how can we dynamically find the best ones for a given language pair, while making the most of existing human feedback? To address this question, we present a novel application of online learning to MT: each MT system in the ensemble is assigned to a weight, and the systems' weights are updated considering human feedback regarding the quality of their translations at each iteration. We use online learning algorithms with theoretical performance guarantees, under the frameworks of prediction with expert advice (Cesa-Bianchi and Lugosi, 2006) and multi-armed bandits (Robbins, 1952;Lai and Robbins, 1985).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
We contribute with an online MT ensemble that allows to reduce human effort by immediately incorporating human feedback in order to dynamically converge to the best systems 1 . Our experiments on WMT'19 News Translation test sets show that our online approaches indeed converge to the shared task's official top-3 systems (or to a subset of them) in just a few hundred iterations for all the language pairs experimented. Moreover, it does so while coping with the aforementioned lack of human assessments for many translations, through the use of fallback metrics.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
To provide some background on our proposal, we start by describing the online learning frameworks that we apply in this paper: prediction with expert advice and multi-armed bandits.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
A problem of prediction with expert advice can be described as an iterative game between a fore-caster and the environment, in which the forecaster seeks advice from different sources (experts) in order to provide the best forecast (Cesa-Bianchi and Lugosi, 2006). At each iteration t, the forecaster consults the predictionsp j,t , j = 1 . . . J, made by a set of J weighted experts, in the decision space D. Considering these predictions, the forecaster makes its own prediction,p f,t ∈ D. At the same time, the environment reveals an outcome y t in the decision space Y (which may not necessarily be the same as D).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
A well-established algorithm to learn the experts' weights in this framework is Exponentially Weighted Average Forecaster (EWAF) (Cesa-Bianchi and Lugosi, 2006). In EWAF, the prediction made by the forecaster is randomly selected following the probability distribution based on the experts' weights ω 1,t−1 . . . ω J,t−1 :
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
(1)
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
At the end of each iteration, the forecaster and each of the experts receive a non-negative loss based on the outcome y t revealed by the environment ( f,t and j,t , respectively). The weight ω j,t of each expert j = 1 . . . J is then updated according to the loss received by each expert, as follows:
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
If the parameter η is set to 8 log J T , it can be shown that the forecaster quickly converges to the performance of the best expert after T iterations (Cesa-Bianchi and Lugosi, 2006).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
Prediction with expert advice assumes that both the forecaster and all the experts receive a loss once the environment's outcome is revealed. However, this assumption may not always hold (i.e., there may not always be an environment's explicit feedback or a way to obtain the loss for all the experts). Thus, we consider a related class of problems, multi-armed bandits, in which the environment's outcome is unknown (Robbins, 1952;Lai and Robbins, 1985). In this class of problems, one starts by attempting to estimate the means of the loss distributions for each expert (also known as arm) in the first iterations (the exploration phase), and when the forecaster has a high level of confidence in the estimated values, one may keep choosing the prediction with the smallest estimated loss (the exploitation phase).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
A popular online algorithm for adversarial multiarmed bandits is Exponential-weighting for Exploration and Exploitation (EXP3) (Auer et al., 1995). At each iteration t, the forecaster's action is randomly selected according to the probability distribution given by the weights of each arm j:
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
In this framework, the forecaster is only able to measure the loss of the action it selects at each iteration, but it cannot measure the loss of other possible actions. Thus, only the weight of the arm associated with this action is updated, as follows:
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
whereˆ j,t = j,t p j,t and p j,t is the probability of choosing arm j at iteration t. By setting η to 2logJ T |A| (where |A| is the number the actions available, and may be the same as the number of arms J), it can be shown that the forecaster quickly converges to the performance of the best arm.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
Both of these frameworks are relatively underexplored in NLP, despite their potential to converge to the best performing approach available in scenarios where feedback is naturally present. Therefore, we propose to apply them in order to find the best MT models with little human feedback.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
In this work, we consider the following scenario as the starting point: there is an ensemble composed of an arbitrary number of MT systems; given a segment from a source language corpus, each system outputs a translation in the target language; then, the quality of the translations produced by each of the available systems is assessed by one or more human evaluators with a score reflecting their quality. We frame this scenario as an online learning problem under two different frameworks: (i) prediction with expert advice (using EWAF as the learning algorithm), and (ii) multi-armed bandits (using EXP3 as the learning algorithm). The decision on whether to use one or another framework in an MT scenario depends on whether there is human feedback available for the translations outputted by all the available systems or only for the final choice of the ensemble of systems.  Figure 1: Overview of the online learning process applied to MT, at each iteration t. The grey dashed arrows represent flows that only occur when using prediction with expert advice.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
An overview of the online learning process is shown in Fig.1, and can be summed up as follows. Each MT system is an expert (or arm) j = 1 . . . J, associated with a weight ω j (all the systems start with same weights). At each iteration t, a segment src t is selected from the source language corpus and handed to all the MT systems. Each system outputs a translation transl j,t in the target language, and one of these translations is selected as the forecaster's action according to the probability distribution given by the systems' weights (Eq.1 for EWAF and Eq. 3 for EXP3). The chosen translation transl f,t (when using EXP3) or the translations outputted by all the systems (when using EWAF) receive a human assessment score 2 score j,t , from which the loss j,t is derived for the respective MT system. Finally, the weight of the chosen system or the weights of all the systems are updated as a function of the loss received, according to Eq.4 (when using EXP3) and Eq.2 (when using EWAF), respectively (where j,t = −score j,t ).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
To validate our proposal, we designed an experiment using data from an MT shared task. The main questions addressed by our experiment are: (i) whether an online learning approach can give a greater weight to the top performing systems for each language pair according to the shared task's official ranking, and (ii) if so, how quickly (i.e., how many translations need to be assessed by human evaluators in order to find the best system).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
Below we detail the datasets used (Section 4.1) and the feedback sources considered (Section 4.2), as well as other experimental decisions (Section 4.3).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
2 If multiple human assessments were made for the same translation, scorej,t is the average of the scores received.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
We used the test datasets made available by the WMT'19 News Translation shared task (Barrault et al., 2019). For each language pair, each source segment is associated with the following information:
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
• A reference translation in the target language (produced specifically for the task);
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
• The automatic translation outputted by each system competing in the task for that language pair;
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
• The average score obtained by each automatic translation, according to human assessments made by one or more human evaluators, in two formats: a raw score in [0;100] and a zscore in [−∞; +∞]. Not all the automatic translations received a human assessment;
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
• The number of human evaluators for each automatic translation (if there were any).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
For brevity, we focused on five language pairs, listed in Table 1. The official top 3 systems for each pair, according to the average z-score, are shown in Table 2. Our choice of language pairs attempts to capture as many different phenomena as possible with the fewest pairs:
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
• English → German (en-de): This is the language pair with the most competitors and does not have a clear winning system (the winner differs depending on whether one considers the z-score or the raw score);
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
• French → German (fr-de): Unlike most language pairs, this pair features two languages other than English. Moreover, there is a strong imbalance between translations lacking human assessments and translations that received at least one assessment;
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
• German → Czech (de-cs): Besides featuring two languages other than English, this pair stands out as it was devised as an unsupervised task (i.e., English was used as a "hub" language);
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
• Gujarati → English (gu-en): This is one of the task's low-resource language pairs (i.e., whose test set is half the size of most languagepairs in the task), and is one where there may en-de fr-de de-cs gu-en lt-en   (Barrault et al., 2019). The systems named "online-[letter]" correspond to publicly available translation services and were anonimized in the shared task.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
be more linguistic differences between the source and the target languages (e.g., different writing systems). Unlike en-de, there is a clear winner considering both raw and z-score. Moreover, three of the competing systems did not receive any human assessment on their translations;
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
• Lithuanian → English (lt-en): This is another low-resource language pair, with a rather competitive top 3. Unlike most language pairs, all the translations submitted by the competing systems for this pair received a human assessment.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
For all these language pairs (except English → German), each segment was given an assessment score considering only the reference translation (and without access to the segment's context within the document to which it belongs). For English → German, scores were given considering the source segment instead of the reference, and evaluators had access to the segment's context within the document.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
A key condition for applying online learning to this scenario is the availability of feedback. We use the human assessment raw scores 3 present in the test sets as a feedback source to compute the loss and update the weight of each MT system, as already suggested in Section 3. However, not all translations received human assessments (recall Table 1). To cope with this issue, we designed different variants of this loss function, following different fallback strategies:
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
• human-zero: If there is no human assessment for the current translation, a score of zero is returned (leading to an unchanged weight on that iteration);
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
• human-avg: If there is no human assessment for the current translation, the average of the previous scores received by the system behind that translation is returned as the current score;
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
• human-comet: If there is no human assessment for the current translation, the COMET score (Rei et al., 2020a) between the translation and the pair source/reference available in the corpus is returned as the current score. We pre-trained 4 this automatic metric on the datasets of previous shared tasks ( WMT'17 (Bojar et al., 2017) and WMT'18 (Bojar et al., 2018)). Thus, for most translations, it displays a small difference regarding the existing human scores (see Fig. 2 for the case of en-de). Moreover, this metric correlates better with ratings by professional translators than the WMT scores (Freitag et al., 2021). 
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
For each language pair, we shuffled the test set once, so that the performance of the online algorithms would not be biased by the original order of the segments in the test set. We ran EWAF once for each loss function, and we ran EXP3 10 times per loss function and report the average weights obtained across runs, since EXP3's weight evolution is critically influenced by the random choice of an arm at each iteration. We normalized the translation scores score j,t to be in the interval [0, 1] and rounded them to two decimal places, to avoid exploding weight values due to the exponential update rule.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
In order to observe whether (and how soon) our online approach converges to the best systems, we report the overlap between the top n = 1, 3 systems with greatest weights according to our approaches, s n , and the top n = 1, 3 systems according to the shared task's official ranking, s * n , at specific iterations:
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
We preferred this metric over a rank correlation metric, as we are focused on whether our online approach follows the performance of the best MT systems. In a realistic scenario (e.g., a Web MT service), a user would most likely rely solely on the main translation returned, or would at most consider one or two alternative translations. Moreover, due to the lack of a large enough coverage of human assessments, the scores obtained in the shared task are not reliable enough to discriminate between similarly performing systems.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
Starting with en-de (Table 3), this was the language pair for which our approach appears to be the least successful, since, for most of the iterations examined, it failed to converge to the best system. Even so, it managed to converge to the top 3 systems, doing so particularly early in the learning process (50 iterations) when using EWAF with human-avg and human-comet as loss functions (i.e., when using fallback scores). Recall that, for this language pair, there were different official winning systems depending on whether one considers the z-score or the raw score (recall Table 2); since we use the raw score as the loss function, it is expectable that our approach does not necessarily converge to the winner according to the z-score.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
For fr-de (Table 4), our online approach often converges to the top 3 systems (or a subset of them) throughout the learning process (even at just 10 iterations), and it also converges to the best system when using EWAF with human-comet. This is a particularly interesting result if we recall that, out of the five pairs considered, fr-de had the lowest coverage of human assessments by far (see Table 1), thus suggesting that using COMET may be an adequate fallback strategy.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
For de-cs (Table 5), we can see that, regardless of the algorithm and loss function used, there is an overlap of at least one system between our top 3 and the shared task's official top 3, after going through Iteration 10 50 100 500 1000 1997 Top  1  3  1  3  1  3  1  3  1  3  1  3 EWAF human-zero 0.00 0.33 0.00 0.00 0.00 0.33 1.00 0.67 1.00 0.67 0.00 0.67 human-avg 0.00 0.00 1.00 0.67 1.00 0.33 0.00 1.00 0.00 1.00 0.00 1.00 human-comet 0.00 0.33 0.00 1.00 0.00 0.67 0.00 1.00 0.00 1.00 0.00 1.00 EXP3 human-zero 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.33 1.00 0.33 1.00 0.33 human-avg 0.00 0.00 0.00 0.33 0.00 0.67 0.00 0.33 0.00 0.33 0.00 0.33 human-comet 0.00 0.00 0.00 0.00 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.33 Table 3: Overlap ratios of top 1 and top 3 systems in common between the online approaches and the official ranking for en-de. Recall that, for this pair, the official ranking differed depending on whether the z-score or the raw score was considered.  Table 4: Overlap ratios of top 1 and top 3 systems in common between the online approaches and the official ranking for fr-de. Recall that this was the language pair with the lowest coverage of human assessments.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
10 50 100 500 1000 1997 Top  1  3  1  3  1  3  1  3  1  3  1  3 EWAF human-zero 0.00 0.33 1.00 0.67 1.00 0.67 0.00 0.67 1.00 1.00 0.00 1.00 human-avg 0.00 0.33 0.00 0.33 1.00 0.67 1.00 0.67 1.00 0.67 1.00 1.00 human-comet 0.00 0.33 1.00 0.67 1.00 0.67 1.00 1.00 1.00 1.00 1.00 1.00 EXP3 human-zero 0.00 0.33 0.00 0.67 0.00 0.67 0.00 0.67 0.00 0.67 1.00 1.00 human-avg 0.00 0.33 0.00 0.33 0.00 0.33 0.00 0.67 1.00 0.67 1.00 0.67 human-comet 0.00 0.33 1.00 0.33 1.00 0.33 1.00 0.67 1.00 0.67 1.00 0.67 Table 5: Overlap ratios of top 1 and top 3 systems in common between the online approaches and the official ranking for de-cs.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
only as few as 10 iterations (despite a considerable lack of human assessments in this language pair). We can also see that the human-comet loss function is the most successful overall, which reinforces the idea that COMET may be an appropriate fallback metric in the absence of human scores for a given translation. Since this is the language pair for which there seems to be a more similar performance across different algorithms and loss functions, we also report the weight evolution plots for this pair in order to inspect what changes depending on the algorithm and fallback strategy used 5 . Looking at EWAF combined with the human-zero loss function (Fig. 3), one can see a rather irregular evolution for the weights of the top systems, which may be explained by the distribution of the translations lacking human assessments across different systems and learning iterations. Using the human-avg loss function (Fig.4) allows for a more monotonous evolution, by rewarding the sys- 3111   Iteration  10  50  100  500  1000  1016   Top  1  3  1  3  1  3  1  3  1  3  1  3 EWAF human-zero 1.00 0.67 1.00 1.00 1.00 1.00 1.00 0.67 1.00 0.67 1.00 0.67 human-comet 1.00 0.67 0.00 0.67 0.00 0.67 0.00 0.67 0.00 0.33 0.00 0.33 EXP3 human-zero 0.00 0.33 0.00 0.67 0.00 0.33 0.00 0.33 0.00 0.33 0.00 0.33 human-comet 0.00 0.33 1.00 0.33 1.00 0.33 1.00 0.33 0.00 0.67 0.00 0.67 Table 6: Overlap ratios of top 1 and top 3 systems in common between the online approaches and the official ranking for gu-en. Recall that there were three systems competing on this language pair that did not receive human assessments at all (thus, using human-avg yields the same results as using human-zero).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
10 50 100 500 1000
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
Top 1 3 1 3 1 3 1 3 1 3 EWAF human-zero 0.00 0.00 0.00 0.67 0.00 0.67 1.00 0.67 0.00 1.00 EXP3 human-zero 0.00 0.33 0.00 0.00 0.00 0.33 1.00 0.67 1.00 0.67 Table 7: Overlap ratios of top 1 and top 3 systems in common between the online approaches and the official ranking for lt-en. Recall that this was the only language pair for which all the translations received at least one human assessment, thus there is no need to use a fallback loss function.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
tems that were doing better overall in the absence of human assessments. Using the human-comet loss function (Fig. 5) paints a similar picture, as the COMET scores for this language pair seem to be in line with the official ranking (although they appear to benefit the third best system in detriment of the second best). Finally, using EXP3 instead of EWAF (Fig. 6), combined with human-zero, leads to much less pronounced weights, but still in line with the official ranking. Recall that, for EXP3, these weights are averaged across different runs: since each run may lead to different top systems, the difference between the averaged weights ends up being more smooth, i.e., there is a great variance across runs (this happens regardless of the language pair or loss function).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
As for gu-en (Table 6), our approach (using EWAF with human-zero) converges to the best system and to a subset of the top 3 within just 10 iterations; on the other hand, using human-comet does not do as well as not using a fallback strategy, at least when combined with EWAF. However, recall that, for this pair, there were systems that did not receive any human assessments at all for their translations (that being the reason why we do not report human-avg for this pair: the resulting weights end up being the same as when using human-zero). One of the systems that did not receive any human assessments, online-B, ended up receiving high COMET scores, thus leading to a weaker overlap between the online approach ranking and the official ranking.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
Finally, for lt-en (Table 7) we only report the human-zero loss function, since this is the only pair for which there are human assessments for all translations. Interestingly, the online approaches do not do well as quickly as for other pairs, but eventually get there (within 100 to 500 iterations).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
To sum up these results: although factors like the coverage of human assessments or the combinations of online algorithm and loss function used influence how well our approach does, we can still conclude that using an online learning approach allows to converge to the top 3 systems according to the official ranking (or at least to a subset of them) in just a few hundred iterations (and, in some cases, in just a few dozens of iterations) for all the language pairs considered.  Barrault et al., 2020). In the News Translation shared task, participants submit the out- puts of their systems that are then evaluated by a community of human evaluators using Direct Assessment scores (Graham et al., 2013). Thus, the winner is the system that achieves the highest average score. For WMT'19 (Barrault et al., 2019), most of the competing systems followed a Transformer architecture (Vaswani et al., 2017), with the main differences among them being: (i) whether they considered document-level or only sentencelevel information; (ii) whether they were trained only on the training data provided by the shared task, or on additional sources as well; (iii) whether they consisted of a single model or an ensemble.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
There has been a number of online learning approaches applied to MT in the past, mainly in Interactive MT and/or post-editing MT systems. However, most approaches aim at learning the parameters or feature weights of an MT model (Mathur et al., 2013;Denkowski et al., 2014;Ortiz-Martínez, 2016;Sokolov et al., 2016;Nguyen et al., 2017;Lam et al., 2018) or fine-tuning a pretrained model for domain adaptation (Turchi et al., 2017;Karimova et al., 2018;Peris and Casacuberta, 2019). Even in cases where the MT model is composed of several sub-models (e.g., Ortiz-Martínez (2016)), the goal is to online learn each sub-model's specific parameters (while our learning goal is the weights of each system in an ensemble). Another key difference between these approaches and ours is that most of them use human post-edited translations as a source of feedback. The exceptions to this are the systems competing for WMT'17 shared task on online bandit learning for MT (Sokolov et al., 2017), as well as Lam et al. (2018), who use (simulated) quality judgments. The most similar proposal to ours is that of Naradowsky et al. (2020), who ensemble different MT systems and dynamically select the best one for a given MT task or domain using stochastic multiarmed bandits and contextual bandits. The bandit algorithms learn from feedback simulated using a sentence-level BLEU score between the selected automatic translation and a reference translation.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
Thus, to the best of our knowledge, we are the first to frame the MT problem as a problem of prediction with expert advice and adversarial multiarmed bandits in order to combine different systems into an ensemble that converges to the performance of the best individual systems, simulating the human-in-the-loop by using actual human assessments (when available).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
We proposed an online learning approach to address the issue of finding the best MT systems among an ensemble, while making the most of existing human feedback. In our experiments on WMT'19 News Translation datasets, our approach converged to the top-3 systems (or a subset of them) according to the official shared task's ranking in just a few hundred iterations for all the language pairs considered (and just a few dozens in some cases), despite the lack of human assessments for many translations. This is a promising result, not only for the purpose of reducing the human evaluations required to find the best systems in a shared task, but also for any MT application that has access to an ensemble of multiple independent systems and to a source of feedback from which it can learn iteratively (e.g., Web translation services).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
Yet, our approach is limited by the quality of the collected human judgments. For future work, we plan to combine online learning with a more reliable human metric, such as the Multidimensional Quality Metric (MQM) (Lommel et al., 2014), so that we can focus on the quality of the assessments instead of their quantity. 
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
Here we present the weight evolution per MT system for all the combinations of language pairs, learning algorithms (EWAF or EXP3), and loss functions (human-zero, human-avg, or human-comet, when applicable) -except for those combinations that are already part of the main document.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
A.1 English → German (en-de)  0  100  200  300  400  500  600  700  800  900  1000  1100  1200  1300  1400  1500  1600  1700  1800 
Learning a Cost-Effective Annotation Policy for Question Answering
Question answering (QA) based on textual content has attracted a great deal of attention in recent years Xie et al., 2020). In order for state-of-the-art QA models to succeed in real applications (e.g., customer service), there is often a need for large amounts of training data. However, manually annotating such data can be extremely costly. For example, in many realistic scenarios, there exists a list of questions from real users (e.g., search logs, FAQs, service-desk interactions). Yet, annotating such questions is highly expensive (Nguyen et al., 2016;He et al., 2018;Kwiatkowski et al., 2019): it requires the screening of a text corpus to find the relevant document(s) and subsequently screening the document(s) to identify the answering text span(s). Motivated by the above scenarios, we study cost-effective annotation for question answering, whereby we aim to accurately 1 annotate a given set of user questions with as little cost as possible. Generally speaking, there has been extensive research on how to reduce effort in the process of data labeling (Haffari et al., 2009). For example, active learning for a variety of machine learning and NLP tasks (Siddhant and Lipton, 2018) aims to select a small, yet highly informative, subset of samples to be annotated. The selection of such samples is usually coupled with a particular model, and thus, the annotated samples may not necessarily help to improve a different model (Lowell et al., 2019). In contrast, we aim to annotate all given samples at low cost and in a manner that can subsequently be used to develop any advanced model. This is particularly relevant in the current era, where a dataset often outlives a particular model (Lowell et al., 2019). Moreover, there has also been some research into learning from distant supervision (Xie et al., 2020) or self-supervision . Despite being economical, such approaches often produce inaccurate or noisy annotations. In this work, we seek to reduce annotation costs without compromising the resulting dataset quality.
Learning a Cost-Effective Annotation Policy for Question Answering
We propose a novel annotation framework which learns a cost-effective policy for choosing between different annotation schemes, namely the conventional manual annotation scheme (MAN) and a semi-supervised annotation scheme (SEM). Unlike the manual scheme, SEM does not require humans to screen a text corpus or document(s) in order to retrieve annotations. Instead, it leverages an initialized QA system, which can predict top-n candidate annotations for documents or answer spans and asks humans to provide binary feedback (e.g., cor- use sample to update models 5 x Figure 1: High-level overview of our framework: We leverage a QA model to predict candidate annotations for a given resource (e.g., x stands for a question or question-document pair, while y is the document or answer span). A policy model decides upon whether to invoke a MAN or SEM scheme based on those predictions. In the event that the semi-supervised strategy fails, we switch back to a manual annotation scheme. Finally, we use the annotated sample to update both the QA model and the policy model. rect or incorrect) to the candidates. While this annotation scheme comes at a low cost, it fails when human annotators mark all candidates as incorrect.
Learning a Cost-Effective Annotation Policy for Question Answering
In such cases, the annotation cost has already been incurred and cannot be recouped. In order to produce an annotation, one must then draw upon the manual scheme (see Fig. 1), in which case the policy would have been more effective if it had chosen the manual annotation scheme instead. Therefore, how to choose the best annotation scheme for each question is the challenge we must address for this task.
Learning a Cost-Effective Annotation Policy for Question Answering
To tackle the above challenge, we propose a novel approach for learning a cost-effective policy. Here the policy receives several candidates and decides on this basis which annotation scheme to invoke. We train the policy with a supervised objective and learn a cost-sensitive decision threshold. The inherent advantage of this method is that our policy immediately reacts to changing costs (without re-optimizing model parameters) and does not exceed the cost of conventional manual annotation. Our policy is updated iteratively as more annotations are obtained.
Learning a Cost-Effective Annotation Policy for Question Answering
We compare our framework against conventional, manual annotations in an extensive set of experiments. We simulate the annotation of Nat-uralQuestions (Kwiatkowski et al., 2019), as it consists of real user questions from search logs. Models in our framework are initialized with an existing dataset (SQuAD, Rajpurkar et al., 2016) and, as more annotations on NaturalQuestions be-come available, the framework is continuously updated. We study the sensitivity of our framework to varying cost ratios between SEM and MAN. Our framework outperforms traditional manual annotation, even under conservative cost estimates for SEM, and in general reduces annotation costs in the range of 4.1% to 21.1%.
Learning a Cost-Effective Annotation Policy for Question Answering
All source code is publicly available from github.com/bernhard2202/qa-annotation.
Learning a Cost-Effective Annotation Policy for Question Answering
We study the problem of reducing the overall cost for annotating every given question [q 1 , . . . , q m ]. Specifically, our objective is to obtain the corresponding question-document-answer triples q i , d i , s i . In this paper, the natural language question q i is given, while we want to obtain the following annotations: the document from a text corpus d i ∈ D that contains the answer and the correct answer span s i within the document d i .
Learning a Cost-Effective Annotation Policy for Question Answering
Fig. 1 provides an overview of our framework for a cost-effective annotation of QA datasets. The framework comprises two main components: a QA model is used to suggest candidates for a resource to annotate while a policy model decides which annotation scheme to invoke (i.e., action). Our framework makes use of two annotation schemes: a traditional manual annotation scheme (MAN) and our semi-supervised annotation scheme (SEM). Both annotation schemes incur different costs and, hence, the learning task is to find and update a cost-effective policy π for making that decision.
Learning a Cost-Effective Annotation Policy for Question Answering
QA model: We define Ω as an arbitrary QA model over a text corpus D with the following properties. First, the model can be trained from annotated data samples, e.g.,
Learning a Cost-Effective Annotation Policy for Question Answering
. Second, for a given question the model can predict a number of top-n documents likely to contain the answer, i.e., Ω D : q → [d (1) , . . . , d (n) ] ∈ D. Third, for a given question-document pair the model can predict a number of top-n answer spans, i.e., Ω S : q, d → [s (1) , . . . , s (n) ]. These properties are fulfilled by recent QA systems dealing with textual content (e.g., Chen et al., 2017;.
Learning a Cost-Effective Annotation Policy for Question Answering
Policy model: For every question, we distinguish two policy models: a policy model π D responsible for annotating documents and π S for answer spans. For brevity, we sometimes drop the superscripts S and D and simply refer to them as π. The policy models decide whether a manual annotation scheme or rather our proposed semi-supervised annotation scheme is used, each of which is associated to different costs.
Learning a Cost-Effective Annotation Policy for Question Answering
Manual annotation (MAN) scheme: This scheme represents the status quo in which all annotations are determined manually. In order to annotate a question q i , a human annotator must first manually search through the text corpus D in order to identify the document d i that answers the question. In a second step, a human annotator manually reads through the document d i and marks the answer span s i .
Learning a Cost-Effective Annotation Policy for Question Answering
We assume separate costs, which are fixed over time, for every annotation-level. The price of annotating a document for a given question is defined as c D 0 and the price of annotating an answer span to a given question-document tuple as c S 0 . We explicitly distinguish these costs as the tasks can be of differing difficulty.
Learning a Cost-Effective Annotation Policy for Question Answering
Semi-supervised annotation (SEM) scheme: This scheme is supposed to reduce human effort by presenting candidates for annotation, so that only simple binary feedback is needed. In particular, human annotators no longer need to search through the entire document or corpus. Instead, we use the QA model Ω to generate a set of candidates (e.g., top-ranked documents or answer spans) and ask human annotators to give binary feedback in response (e.g., accept the candidate or reject it). This replaces the complex search task with a simpler form of interaction. As an example, to annotate the answer span for a question-document pair, the human annotator would not be required to read the entire document d i , but only to determine which of the top-n answers provided by Ω S ( q i , d i ) are correct. We assume SEM costs c D 1 to annotate a document and c S 1 to annotate an answer span. The SEM scheme should make annotations more straightforward, as providing binary feedback requires less time than reading through the texts.
Learning a Cost-Effective Annotation Policy for Question Answering
Hence, we assume that c S 1 < c S 0 and c D 1 < c D 0 hold. However, semi-supervised annotations might fail when none of the candidates is correct (i.e., the human annotators reject all candidates). In this case, our framework must revert to the MAN procedure in order to obtain a valid annotation. As a consequence, the associated cost will increase to the accumulated cost for both the SEM and the MAN schemes. Note that, no matter which scheme is chosen in practice, all annotations are confirmed by human annotators and our resulting dataset will be equal in quality to those resulting from traditional annotation.
Learning a Cost-Effective Annotation Policy for Question Answering
Both annotation schemes, MAN and SEM, incur different costs that further vary depending on whether annotation is provided at document level (c D ) or at answer span level (c S ). For annotating documents, the cost amounts to
Learning a Cost-Effective Annotation Policy for Question Answering
is the selected annotation scheme and d * i is the ground-truth document annotation. Hence, d * i ∈ Ω D (q i ) indicates the candidate set contains the ground-truth annotation and SEM is successful. For annotating answer spans, the cost is given by
Learning a Cost-Effective Annotation Policy for Question Answering
(2) Alternatively, we can write the cost function as a matrix of annotation costs (Tbl. 1). The diagonal entries reflect the costs paid for choosing the optimal scheme. The off-diagonals refer to the costs paid for a sub-optimal method (misclassification costs).
Learning a Cost-Effective Annotation Policy for Question Answering
We aim to minimize overall annotation cost via
Learning a Cost-Effective Annotation Policy for Question Answering
Annotation Costs for a Document d  
Learning a Cost-Effective Annotation Policy for Question Answering
Our framework proceeds according to these nine steps when annotating a question q i (see Alg. 1): First, we predict a number of top-n documents that would be shown to annotators in the case of SEM annotation (line 2). Next, we decide upon the annotation scheme conditional on the prediction from the QA model (line 3) and, based on the selected scheme, request the ground-truth document annotation (line 4). After receiving the ground-truth document and observing the annotation costs, we update our policy network in line 5 (see Sec. 4.3).
Learning a Cost-Effective Annotation Policy for Question Answering
Next, we predict a number of top-n answer span candidates for the question-document pair (line 6) and then decide upon the annotation scheme in line 7. After receiving the answer span annotation and observing a cost (line 8), we again update our policy model (line 9). Finally, we update the QA model with the newly annotated training sample in line 10 (see Sec. 4.4). In practice, both policy updates and QA model updates (lines 5, 9, and 10) are invoked after a batch of questions is annotated. Furthermore, we initialize all models with an existing dataset (e.g., SQuAD).
Learning a Cost-Effective Annotation Policy for Question Answering
Algorithm 1: High-Level Procedure of Annotation and Learning 
Learning a Cost-Effective Annotation Policy for Question Answering
Updating our policy model proceeds in three steps.
Learning a Cost-Effective Annotation Policy for Question Answering
(1) We calculate whether the chosen action for past annotations was cost-optimal (i.e., whether the policy should have chosen the other scheme for annotation or not). (2) We use this information to update the policy model with a supervised binary classification objective. This trains the policy to predict the probability of an annotation scheme given a new sample p(a|x) without taking costs into account. (3) We find a cost-sensitive decision threshold that chooses the optimal action with respect to the costs. All three steps are repeated after a full batch of samples has been annotated.
Learning a Cost-Effective Annotation Policy for Question Answering
Separating the policy update and the costsensitive decision threshold has several benefits. First, we know from cost-sensitive classification that we can calculate an optimal threshold point for ground-truth probabilities p(a|x) (c.f. Elkan, 2001;Ting, 2000). Therefore, we can focus our effort on determining probabilities as accurate as possible. Second, the decision threshold is calculated only from the costs c S a and c D a and, hence, if costs change, we do not need to re-estimate parameters but can directly adjust our policy.
Learning a Cost-Effective Annotation Policy for Question Answering
(1) Finding the cost-optimal action: In order to train the policy with a supervised update, we require labels for the cost-optimal annotation scheme for a given sample. If we choose the SEM annotation scheme, we immediately know whether the action was cost-optimal or not. This is due to the fact that, if the semi-supervised annotation fails, we have to switch to the MAN scheme to receive an annotation and pay both costs. On the other hand, if we choose the MAN scheme, we can observe the optimal action only after receiving the ground-truth annotation: We can then simply run the QA model and validate whether the annotation was contained within the top-n candidates. If so, the SEM action would have been the better choice; otherwise, choosing MAN would have been cost-optimal.
Learning a Cost-Effective Annotation Policy for Question Answering
(2) Supervised model updates: Our policy model is a neural network with parameters θ that predicts an annotation scheme for a given sample x, i.e.,
Learning a Cost-Effective Annotation Policy for Question Answering
Note that we dropped the S and D indices here as both policies differ only in the neural network architecture used. We can then simply train the policy with a supervised binary cross-entropy loss given the cost-optimal action that we calculated beforehand. Since the SEM scheme is often suboptimal in the beginning, the training data is highly imbalanced. Therefore, we down-sample past annotations with a sampling ratio of α such that our training data is equally balanced.
Learning a Cost-Effective Annotation Policy for Question Answering
(3) Cost-sensitive decision threshold: Choosing the annotation scheme with the highest probability does not take the actual costs into account. For instance, if SEM annotations are much cheaper than MAN annotations, we want to choose the semisupervised scheme even if its probability for success is low. More formally, we want to choose the annotation scheme a that has the lowest expected cost R(a|x), i.e.,
Learning a Cost-Effective Annotation Policy for Question Answering
where c(a, a ) is the annotation cost for choosing scheme a when the optimal scheme was a (see Tab. 1). Since we used down-sampling in our training, we have to calibrate the probabilities p(a|x) with the sampling ratio α (Pozzolo et al., 2015). Assuming the calibrated probabilities are accurate, there exists an optimal classification threshold β (Elkan, 2001) that minimizes Eq. 5 and Eq. 3. Therefore, we define our policy as follows:
Learning a Cost-Effective Annotation Policy for Question Answering
(6) The optimal β can be derived from the classification cost matrix (Elkan, 2001) 
Learning a Cost-Effective Annotation Policy for Question Answering
again omitting superscripts D and S for brevity. Eq. 7 is then simplified as the fraction of SEM annotation costs to MAN annotation costs. Therefore, we can derive β at document level
Learning a Cost-Effective Annotation Policy for Question Answering
and answer span level
Learning a Cost-Effective Annotation Policy for Question Answering
Periodically updating the QA model Ω allows the framework to adapt to the question style and domain at hand during the annotation process. Therefore, we improve the top-n accuracy and the success rate of the SEM scheme over time. For practical reasons, we refrain from updating Ω after every annotation but periodically retrain the model after a batch of samples is annotated. In order to update the QA model, we can use the fully annotated QA samples in combination with a supervised objective.
Learning a Cost-Effective Annotation Policy for Question Answering
In this section, we introduce our experimental setup and implementation details.
Learning a Cost-Effective Annotation Policy for Question Answering
We base our experiments on the NaturalQuestion dataset (Kwiatkowski et al., 2019). We choose this dataset as it is composed of about 300,000 real user questions posed to the Google search engine along with human-annotated documents and answer spans. Simulating the annotation of this dataset is similar to what would happen for domain customization of QA models in real practice (e.g., for search logs, FAQs, logs of past customer interactions). We focus on questions from the trainingsplit that possess an answer span annotation and leave the handling of questions that do not have an answer for future work. The corpus for annotations is fixed to the English Wikipedia, 2 containing more than 5 million text documents. Simulation of annotations: Annotations in our experiments are simulated from the original dataset. If the framework chooses MAN annotation, we simply use the original annotation from the dataset. If a SEM annotation is chosen, we simulate users that give positive feedback only to the ground-truth document and to the answer spans where the text matches 3 the ground-truth annotation. We then construct the new annotation using the candidate with positive feedback. Since we simulate annotations, we conduct extensive experiments on how annotation costs influence the performance of our framework.
Learning a Cost-Effective Annotation Policy for Question Answering
To the best of our knowledge, there is no comparable prior work. Owing to this fact, we evaluate our framework against several customized baselines. First, we compare our approach against a manual annotation baseline in which we always invoke the full MAN method to annotate samples. This represents the traditional method of annotating QA datasets and thus our prime baseline. Second, we draw upon a clairvoyant oracle policy that always knows the optimal annotation method. We use this baseline to report an upper bound of the savings that our framework could theoretically achieve. Third, we use our framework without updates on the QA model Ω. This quantifies the cost-savings achieved by the interactive domain customization during annotation. Finally, we present a randomized baseline where the annotation scheme is decided by a randomized coin-toss.
Learning a Cost-Effective Annotation Policy for Question Answering
The QA model Ω is built as follows. We use a stateof-the-art BERT-based (Devlin et al., 2018) implementation of RankQA . This combines a simple tf-idf-based information retrieval module with BERT as a module for machine comprehension. Both policy models π D and π S are implemented as three-layer feed-forward networks with dropout, ReLu activation, and a single output unit with sigmoid activation in the last layer. For the policy π D , we use the information retrieval scores as input. For π S , we use the statistical fea-tures of answer-span candidates as calculated by RankQA  as input. We also experimented with convolutional neural networks directly on top of the last layer of BERT, but without yielding improvements that justified the additional model complexity. We initialize all models with the SQuAD dataset (see our supplements).
Learning a Cost-Effective Annotation Policy for Question Answering
Hyperparameter setting: We set the number of candidates that are shown to annotators during a SEM annotation to n = 5. The policy networks decide upon the annotation method based on features of the 2n highest-ranked candidates, i.e., the top-10. The batch size for updates in Alg. 1 is set to 1,000 annotated questions. Details on hyperparameters of our QA and policy models are provided in the supplements.
Learning a Cost-Effective Annotation Policy for Question Answering
We group our experiments into three parts. First, we focus only on annotating the answer span for given question-document pairs, as this is the more challenging task. 4 Second, we carry out a sensitivity analysis in order to demonstrate how our framework adapts to different costs of SEM annotations and to show that we never exceed the cost of traditional annotation. Third, we evaluate our framework based on the annotation of a full dataset, including both answer span and document annotations, in order to quantify savings in practice by using our framework.
Learning a Cost-Effective Annotation Policy for Question Answering
The annotation framework was used to annotate 45 batches of question-document pairs with the corresponding answer spans. The annotation costs are set to one price-unit for each MAN annotation and one third of the unit for each SEM annotation. (In the next section, we carry out an extensive sensitivity analysis where the ratio for annotation costs between MAN and SEM is varied.) In Fig. 2 (left), we plot the average annotation costs in every batch with a dashed line, together with a running mean depicted as a solid line. Compared to conventional, manual annotation, our framework successfully reduces annotation cost by around 15% after only 20 batches. We further compare it with an oracle policy that always picks the best annotation method. The latter provides a hypothetical upper bound according to which approximately 40-45% of annotation cost could be saved. Finally, we show the performance of our framework without updates of the QA model Ω.
Learning a Cost-Effective Annotation Policy for Question Answering
Here we can see that its improvement over time is lower, as the framework is not capable of adapting to the question style and domain used during annotation. In sum, our framework is highly effective in reducing annotation cost. Fig. 2 (right) shows how many samples we could annotate (y-axis) with a restricted budget (x-axis). For instance, assume we have a budget of 40k price units available for annotation. Conventional, manual annotation would result in exactly 40k annotated samples as we fixed the cost for each MAN annotation to one unit. With the same budget, our annotation framework with semi-supervised annotations succeeds in annotating an additional ∼9,000 samples. 
Learning a Cost-Effective Annotation Policy for Question Answering
The advantage of our framework over manual annotations depends on the cost ratio between the SEM and MAN schemes. In order to determine this, we identify the cost-range in which our framework is profitable as a function of SEM annotation costs. We study this via the following experiment: we repeatedly annotate 40k samples and keep the MAN annotation costs fixed to one price unit, while we increase the costs of smart annotations from 0.05 to 0.95 in increments of 0.05. Finally, we measure the average annotation costs for a single sample; see Fig. 3 (left). Fig. 3 (left) demonstrates that our framework effectively lowers annotation costs when the price for SEM annotations drops below 0.6 as compared to manual annotations, which are fixed to one priceunit. Most notably, even when SEM annotations become expensive and almost equal the costs of MAN annotations, the average annotation costs do not exceed those of strictly manual annotation. This can be attributed to our cost-sensitive decision threshold, which does not require exploration as in reinforcement learning, but directly sets the threshold in Eq. 6 sufficiently high.
Learning a Cost-Effective Annotation Policy for Question Answering
In Fig. 3 (right), we again show the number of samples that were annotated with a restricted budget of 40k price units. We marked the absolute gain in number of samples over traditional annotation in the plot. The benefit of our framework becomes evident once again when the ratio of SEM annotation costs to MAN annotations costs falls below 0.6.
Learning a Cost-Effective Annotation Policy for Question Answering
To summarize, our framework is highly costeffective: it reduces overall annotation costs or, alternatively, increases the number of annotated samples under a restricted budget if annotation costs of SEM are approximately half those of MAN. If the costs are less than half those of MAN annotation, the benefits are especially pronounced. Even if this assumption does not hold, our framework never exceeds the costs of manual annotation and never results in fewer annotated samples. 
Learning a Cost-Effective Annotation Policy for Question Answering
In the last experiment, we simulate a complete annotation of the NaturalQuestions dataset, including annotations at both document level and answer span level. By annotating a complete dataset, we want to quantify the savings of our framework in practice. We again set the cost of each MAN annotation to one price-unit and repeated the experiment three times by setting the SEM annotation cost (c 1 ) Document-level Answer span-level Overall c1 = 1 /4 c1 = 1 /3 c1 = 1 /2 c1 = 1 /4 c1 = 1 /3 c1 = 1 /2 c1 = 1 /4 c1 = 1 /3 c1 = 1 /2  Table 2: Overall cost (×10 3 price unit) for annotating the NaturalQuestions dataset using our framework vs. conventional manual annotation for different SEM costs (c 1 ). Improvements are shown in parenthesis.
Learning a Cost-Effective Annotation Policy for Question Answering
to one quarter, one third, and one half of the price unit. The results are shown in Tbl. 2. Depending on relative cost ratio c 1 , we are able to save between 4.1% and 21.1% percent of the overall annotation cost. This amounts to a total of 40,000 to 8,000 price units. 5
Learning a Cost-Effective Annotation Policy for Question Answering
We assume for the purposes of this study that questions have an answer span contained in a single document and leave an extension to multi-hop questions and unanswerable questions to future research. The robustness of our framework is demonstrated in an extensive set of simulations and experiments. We deliberately choose to leave experiments including real human annotators to future research for the following reason. Outcomes of such an experiment would be sensitive to the design of the user interface as well as the study design itself. In this paper, we want to put the emphasis on the methodological innovation of our framework and the novel annotation scheme itself. On the other hand, experiments involving real users would provide valuable insights concerning the annotation costs and the quality of a dataset annotated with our method. Furthermore, it would be worth investigating how inter-annotator agreement or potential human biases manifest in traditional datasets as compared to those generated with our framework.
Learning a Cost-Effective Annotation Policy for Question Answering
questions accurately while limiting costs as much as possible. We show that our framework never incurs higher costs than traditional manual annotation. On the contrary, it achieves substantial savings. For example, it reduces the overall costs by about 4.1% when SEM annotations cost about half of MAN annotations. When that ratio is lowered to one fourth, our framework can reduce the total costs by up top 21.1%. We think that our framework could contribute to more accessible annotation of datasets in the future and possibly even be extended to other fields and applications in natural language processing.   
Learning a Cost-Effective Annotation Policy for Question Answering
The policy models π D and π S are implemented as feed forward networks composed of a dense layer with k output units and relu actvation, a dropout layer with dropout probability z, a second dense layer with k/2 outputs and relu activation, a dropout layer with dropout probability z, and a dense layer with a single output and sigmoid activation.
Learning a Cost-Effective Annotation Policy for Question Answering
Initialization: for the first batch of annotations we initialize the policy models on SQuAD. After the first batch is annotated we only use the new data for policy updates.
Learning a Cost-Effective Annotation Policy for Question Answering
Hyperparameter search: We tune hyperparamters on the SQuAD dataset using gridsearch with the values displayed in Tab. 3 and Tab. 4. Bold values mark final choices. We annotated the first 10 batches of SQuAD and choose the hyperparamters that had the lowest anntation cost. No hyperparemter tuning or architecture search was performed on the NaturalQuestions dataset which our experiments are based on.
Learning a Cost-Effective Annotation Policy for Question Answering
In order to provide additional insights on the actual annotation costs involving real users we conducted a pre-test on Amazon MTURK. For this we showed a textual explanation of the MAN and SEM annotation scheme to workers and provided them with mockups for both inputs (answer-span annotation). Next, we asked 40 workers to report how much money they think would be a fair compensation for each of the tasks on a scale of one to ten. Work-ers reported on average a compensation of $5.9 for MAN annotations and $3.2 for SEM annotations. This ratio falls into the range where we make profits using our framework.
Learning a Cost-Effective Annotation Policy for Question Answering
All experiments were conducted with a Nvidia Titan Xp GPU on a Server with 192GB DDR4 RAM and two 10 Core Intel Xeon Silver 4210 2.2GHz Processors.
Learning to summarize from human feedback
Large-scale language model pretraining has become increasingly prevalent for achieving high performance on a variety of natural language processing (NLP) tasks. When applying these models to a specific task, they are usually fine-tuned using supervised learning, often to maximize the log probability of a set of human demonstrations.
Learning to summarize from human feedback
While this strategy has led to markedly improved performance, there is still a misalignment between this fine-tuning objective-maximizing the likelihood of human-written text-and what we care about-generating high-quality outputs as determined by humans. This misalignment has several causes: the maximum likelihood objective has no distinction between important errors (e.g. making up facts [41]) and unimportant errors (e.g. selecting the precise word from a set of synonyms); models Figure 1: Fraction of the time humans prefer our models' summaries over the human-generated reference summaries on the TL;DR dataset. 4 Since quality judgments involve an arbitrary decision about how to trade off summary length vs. coverage within the 24-48 token limit, we also provide length-controlled graphs in Appendix F; length differences explain about a third of the gap between feedback and supervised learning at 6.7B. are incentivized to place probability mass on all human demonstrations, including those that are low-quality; and distributional shift during sampling can degrade performance [56,52]. Quality can often be improved significantly by non-uniform sampling strategies such as beam search [51], but these can lead to repetition and other undesirable artifacts [69,23]. Optimizing for quality may be a principled approach to overcoming these problems.
Learning to summarize from human feedback
Our goal in this paper is to advance methods for training language models on objectives that more closely capture the behavior we care about. To make short-term progress towards this goal, we focus on abstractive English text summarization, as it has a long history in the NLP community [16,8,54,59,50], and is a subjective task where we believe it is difficult to quantify summary quality without human judgments. Indeed, existing automatic metrics for evaluating summary quality, such as ROUGE [39], have received criticism for poor correlation with human judgments [55,45,6,33].
Learning to summarize from human feedback
We follow the works of [3,73], who fine-tune language models from human feedback using reward learning [35]. We first collect a dataset of human preferences between pairs of summaries, then train a reward model (RM) via supervised learning to predict the human-preferred summary. Finally, we train a policy via reinforcement learning (RL) to maximize the score given by the RM; the policy generates a token of text at each 'time step', and is updated using the PPO algorithm [58] based on the RM 'reward' given to the entire generated summary. We can then gather more human data using samples from the resulting policy, and repeat the process. We follow the works of [48,4] and use large pretrained GPT-3 models with as many as 6.7 billion parameters.
Learning to summarize from human feedback
Our main contributions are four-fold.
Learning to summarize from human feedback
(1) We show that training with human feedback significantly outperforms very strong baselines on English summarization. When applying our methods on a version of the Reddit TL;DR dataset [63], we train policies via human feedback that produce better summaries than much larger policies trained via supervised learning. Summaries from our human feedback models are preferred by our labelers to the original human demonstrations in the dataset (see Figure 1).
Learning to summarize from human feedback
(2) We show human feedback models generalize much better to new domains than supervised models. Our Reddit-trained human feedback models also generate high-quality summaries of news articles on the CNN/DailyMail (CNN/DM) dataset without any news-specific fine-tuning, almost matching the quality of the dataset's reference summaries. We perform several checks to ensure that these human preferences reflect a real quality difference: we consistently monitor agreement rates amongst labelers and researchers, and find researcher-labeler agreement rates are nearly as high as researcher-researcher agreement rates (see Section C.2), and we verify models are not merely optimizing simple metrics like length or amount of copying (see Appendices F and G.7).
Learning to summarize from human feedback
(3) We conduct extensive empirical analyses of our policy and reward model. We examine the impact of model and data size ( Figure 6), study performance as we continue to optimize a given reward model (Section 4.3), and analyze reward model performance using synthetic and humanwritten perturbations of summaries (Section 4.3). We confirm that our reward model outperforms other metrics such as ROUGE at predicting human preferences, and that optimizing our reward model directly results in better summaries than optimizing ROUGE according to humans (Section 4.4).
Learning to summarize from human feedback
(4) We publicly release our human feedback dataset for further research. The dataset contains 64,832 summary comparisons on the TL;DR dataset, as well as our evaluation data on both TL;DR (comparisons and Likert scores) and CNN/DM (Likert scores).
Learning to summarize from human feedback
The methods we present in this paper are motivated in part by longer-term concerns about the misalignment of AI systems with what humans want them to do. When misaligned summarization models make up facts, their mistakes are fairly low-risk and easy to spot. However, as AI systems become more powerful and are given increasingly important tasks, the mistakes they make will likely become more subtle and safety-critical, making this an important area for further research.
Learning to summarize from human feedback
Most directly related to our work is previous work using human feedback to train summarization models with RL [3,73]. Bohm et al. [3] learn a reward function from a dataset of human ratings of 2.5k CNN/DM summaries, and train a policy whose summaries are preferred to a policy optimizing ROUGE. Our work is most similar to [73], who also train Transformer models [62] to optimize human feedback across a range of tasks, including summarization on the Reddit TL;DR and CNN/DM datasets. Unlike us, they train in an online manner and find the model highly extractive. They note that their labelers prefer extractive summaries and have low agreement rates with researchers. Compared to [73], we use significantly larger models, move to the batch setting for collecting human feedback, ensure high labeler-researcher agreement, and make some algorithmic modifications, such as separating the policy and value networks.
Learning to summarize from human feedback
Human feedback has also been used as a reward to train models in other domains such as dialogue [25, 68,21], translation [32,1], semantic parsing [34], story generation [72], review generation [7], and evidence extraction [46]. Our reward modeling approach was developed in prior work on learning to rank [40], which has been applied to ranking search results using either explicit feedback [2,18] or implicit feedback in the form of click-through data [29,30]. In a related line of research, human feedback has been used to train agents in simulated environments [10,24]. There is also a rich literature on using RL to optimize automatic metrics for NLP tasks, such as ROUGE for summarization [50,65,45,15,19], BLEU for translation [50,66,1,43], and other domains [61,27,26]. Finally, there has been extensive research on modifying architectures [22,59] and pre-training procedures [70,36,49,60,53,14] for improving summarization performance.
Learning to summarize from human feedback
3 Method and experiment details 3
Learning to summarize from human feedback
Our approach is similar to the one outlined in [73], adapted to the batch setting. We start with an initial policy that is fine-tuned via supervised learning on the desired dataset (in our case, the Reddit TL;DR summarization dataset). The process (illustrated in Figure 2) then consists of three steps that can be repeated iteratively.
Learning to summarize from human feedback
Step 1: Collect samples from existing policies and send comparisons to humans. For each Reddit post, we sample summaries from several sources including the current policy, initial policy, original reference summaries and various baselines. We send a batch of pairs of summaries to our human evaluators, who are tasked with selecting the best summary of a given Reddit post.
Learning to summarize from human feedback
Step 2: Learn a reward model from human comparisons. Given a post and a candidate summary, we train a reward model to predict the log odds that this summary is the better one, as judged by our labelers.
Learning to summarize from human feedback
Step 3: Optimize a policy against the reward model. We treat the logit output of the reward model as a reward that we optimize using reinforcement learning, specifically with the PPO algorithm [58].
Learning to summarize from human feedback
"j is better than k" "j is better than k"  We provide a more thorough description of our procedure, including details of the reward model and policy training and our quality control process, in the following sections. In practice, rather than precisely iterating this sequence of three steps, we updated our data collection and training procedures over the course of the project while accumulating labels (see Appendix C.6 for details).
Learning to summarize from human feedback
Datasets. We use the TL;DR summarization dataset [63], which contains~3 million posts from reddit.com across a variety of topics (subreddits), as well summaries of the posts written by the original poster (TL;DRs). We additionally filter this dataset (see Appendix A) to ensure quality, including using a whitelist of subreddits that are understandable to the general population. Crucially, we also filter to include only posts where the human-written summaries contain between 24 and 48 tokens, to minimize the potential effect of summary length on quality (see Section 4.1 and Appendix F). Our final filtered dataset contains 123,169 posts, and we hold out~5% as a validation set. For the remainder of this paper, we refer to this dataset simply as TL;DR.
Learning to summarize from human feedback
We chose the TL;DR dataset over the more commonly used CNN/DM dataset primarily because very strong performance can be attained on CNN/DM with simple extractive baselines. We find in Section 4.2 that our labelers prefer lead-3 over the CNN/DM reference summaries, 5 and that the supervised T5 model [49] with low-temperature sampling already surpasses the reference summary quality, while copying extensively from the article. On the other hand, simple extractive baselines perform poorly on TL;DR in our human evaluations (see Appendix G.2). Instead of training on CNN/DM, we study the transfer performance of our human feedback models to CNN/DM after being trained to summarize Reddit posts.
Learning to summarize from human feedback
Task. We define our ground-truth task as producing a model that generates summaries fewer than 48 tokens long that are as good as possible, according to our judgments. We judge summary quality by how faithfully the summary conveys the original post to a reader who can only read the summary and not the post (see Appendix C.5 for further discussion of criteria). Since we have limited capacity to do comparisons, we hire labelers to do the comparisons for us. We rely on detailed procedures to ensure high agreement between labelers and us on the task, which we describe in the next section.
Learning to summarize from human feedback
[r/dating_advice] First date ever, going to the beach. Would like some tips Hey Reddit! I (20M) would like some tips, because I have my first ever date tomorrow (although I've had a gf for 3 years, but no actual dating happened), and we're going to the beach.
Learning to summarize from human feedback
I met this girl, we have mutual friends, at a festival a few days ago. We didn't kiss, but we talked, held hands, danced a bit. I asked her to go on a date with me, which was super hard as it is the first time I've asked this to anybody. What I mean to say is, it's not like a standard *first* date because we already spent some time together.
Learning to summarize from human feedback
I'm really nervous and excited. I'm going to pick her up tomorrow, we're cycling to the beach which will take 30 
Learning to summarize from human feedback
Previous work on fine-tuning language models from human feedback [73] reported "a mismatch between the notion of quality we wanted our model to learn, and what the humans labelers actually evaluated", leading to model-generated summaries that were high-quality according to the labelers, but fairly low-quality according to the researchers.
Learning to summarize from human feedback
Compared to [73], we implement two changes to improve human data quality. First, we transition entirely to the offline setting, where we alternate between sending large batches of comparison data 6 to our human labelers and re-training our models on the cumulative collected data. Second, we maintain a hands-on relationship with labelers: 7 we on-board them with detailed instructions, answer their questions in a shared chat room, and provide regular feedback on their performance. We train all labelers to ensure high agreement with our judgments, and continuously monitor labeler-researcher agreement over the course of the project. See Appendix C.1 and C.5 for details.
Learning to summarize from human feedback
As a result of our procedure, we obtained high labeler-researcher agreement: on a subset of comparison tasks, labelers agree with researchers 77% ± 2% of the time, while researchers agree with each other 73% ± 4% of the time. We provide more analysis of our human data quality in Appendix C.2.
Learning to summarize from human feedback
All of our models are Transformer decoders [62] in the style of GPT-3 [47,4]. We conduct our human feedback experiments on models with 1.3 billion (1.3B) and 6.7 billion (6.7B) parameters.
Learning to summarize from human feedback
Pretrained models. Similarly to [12,47], we start with models pretrained to autoregressively predict the next token in a large text corpus. As in [48,4], we use these models as 'zero-shot' baselines by padding the context with examples of high-quality summaries from the dataset. We provide details on pretraining in Appendix B, and on our zero-shot procedure in Appendix B.2.
Learning to summarize from human feedback
Supervised baselines. We next fine-tune these models via supervised learning to predict summaries from our filtered TL;DR dataset (see Appendix B for details). We use these supervised models to sample initial summaries for collecting comparisons, to initialize our policy and reward models, and as baselines for evaluation. In our final human evaluations, we use T=0 to sample from all models, as we found it performed better than higher temperatures or nucleus sampling (see Appendix B.1).
Learning to summarize from human feedback
To validate that our supervised models are indeed strong baselines for comparison, we run our supervised fine-tuning procedure with our 6.7B model on the CNN/DM dataset, and find that we achieve slightly better ROUGE scores than SOTA models [71] from mid-2019 (see Appendix G.4).
Learning to summarize from human feedback
Reward models. To train our reward models, we start from a supervised baseline, as described above, then add a randomly initialized linear head that outputs a scalar value. We train this model to predict which summary y ∈ {y 0 , y 1 } is better as judged by a human, given a post x. If the summary preferred by the human is y i , we can write the RM loss as:
Learning to summarize from human feedback
where r θ (x, y) is the scalar output of the reward model for post x and summary y with parameters θ, and D is the dataset of human judgments. At the end of training, we normalize the reward model outputs such that the reference summaries from our dataset achieve a mean score of 0.
Learning to summarize from human feedback
Human feedback policies. We want to use the reward model trained above to train a policy that generates higher-quality outputs as judged by humans. We primarily do this using reinforcement learning, by treating the output of the reward model as a reward for the entire summary that we maximize with the PPO algorithm [58], where each time step is a BPE token. 8 We initialize our policy to be the model fine-tuned on Reddit TL;DR. Importantly, we include a term in the reward that penalizes the KL divergence between the learned RL policy π RL φ with parameters φ and this original supervised model π SFT , as previously done in [25]. The full reward R can be written as:
Learning to summarize from human feedback
This KL term serves two purposes. First, it acts as an entropy bonus, encouraging the policy to explore and deterring it from collapsing to a single mode. Second, it ensures the policy doesn't learn to produce outputs that are too different from those that the reward model has seen during training.
Learning to summarize from human feedback
For the PPO value function, we use a Transformer with completely separate parameters from the policy. This prevents updates to the value function from partially destroying the pretrained policy early in training (see ablation in Appendix G.1). We initialize the value function to the parameters of the reward model. In our experiments, the reward model, policy, and value function are the same size.
Learning to summarize from human feedback
Policies trained with human feedback are preferred to much larger supervised policies. Our main results evaluating our human feedback policies on TL;DR are shown in Figure 1. We measure policy quality as the percentage of summaries generated by that policy that humans prefer over the reference summaries in the dataset. Our policies trained with human feedback significantly outperform our supervised baselines on this metric, with our 1.3B human feedback model significantly outperforming a supervised model 10× its size (61% versus 43% raw preference score against reference summaries). Our 6.7B model in turn significantly outperforms our 1.3B model, suggesting that training with human feedback also benefits from scale. Additionally, both of our human feedback models are judged by humans to be superior to the human demonstrations used in the dataset.
Learning to summarize from human feedback
Controlling for summary length. When judging summary quality, summary length is a confounding factor. The target length of a summary is implicitly part of the summarization task; depending on the desired trade-off between conciseness and coverage, a shorter or longer summary might be better. Since our models learned to generate longer summaries, length could account for much of our quality improvements. We find that after controlling for length (Appendix F), the preference of our human feedback models vs. reference summaries drops by~5%; even so, our 6.7B model summaries are still preferred to the reference summaries~65% of the time.
Learning to summarize from human feedback
How do our policies improve over the baselines? To better understand the quality of our models' summaries compared to the reference summaries and those of our supervised baselines, we conduct an additional analysis where human labelers assess summary quality across four dimensions (or "axes") using a 7-point Likert scale [38]. Labelers rated summaries for coverage (how much important information from the original post is covered), accuracy (to what degree the statements in the summary are stated in the post), coherence (how easy the summary is to read on its own), and overall quality.  The results ( Figure 3) indicate that our human feedback models outperform the supervised baselines across every dimension of quality, but particularly coverage. Although our human labelers had a high bar for giving perfect overall scores, summaries from our 6.7B PPO model achieve a 7/7 overall score 45% of the time (compared to 20% and 23% for the 6.7B supervised baseline and reference summaries, respectively).
Learning to summarize from human feedback
Our human feedback models can also generate excellent summaries of CNN/DM news articles without any further training ( Figure 4). Our human feedback models significantly outperform models trained via supervised learning on TL;DR and models trained only on pretraining corpora. In fact, our 6.7B human feedback model performs almost as well as a 6.7B model that was fine-tuned on the CNN/DM reference summaries, despite generating much shorter summaries.
Learning to summarize from human feedback
Since our human feedback models transferred to CNN/DM have little overlap in summary length distribution with models trained on CNN/DM, with about half as many tokens on average, they are difficult to compare directly. Thus our evaluations in Figure 4 use a 7-point Likert scale on four quality dimensions, as in Section 4.1 (see Appendix C.5 for labeler instructions). In Figure 4b we show the average overall score at different summary lengths, which suggests our human feedback models would perform even better if they generated longer summaries. Qualitatively, CNN/DM summaries from our human feedback models are consistently fluent and reasonable representations of the article; we show examples on our website and in Appendix H.
Learning to summarize from human feedback
What happens as we optimize the reward model? Optimizing against our reward model is supposed to make our policy align with human preferences. But the reward model isn't a perfect representation of our labeler preferences, as it has limited capacity and only sees a small amount of comparison data from a relatively narrow distribution of summaries. While we can hope our reward model generalizes to summaries unseen during training, it's unclear how much one can optimize against the reward model until it starts giving useless evaluations.
Learning to summarize from human feedback
To answer this question, we created a range of policies optimized against an earlier version of our reward model, with varying degrees of optimization strength, and asked labelers to compare samples from them to the reference summaries. Figure 5 shows the results for PPO at a range of KL penalty coefficients (β). Under light optimization, the models improve (according to labelers). However, as we optimize further, true preferences fall off compared to the prediction, and eventually the reward model becomes anti-correlated with human preferences. Though this is clearly undesirable, we note that this over-optimization also happens with ROUGE (see [45] and Appendix G.3). Similar behavior has been observed in learned reward functions in the robotics domain [5].
Learning to summarize from human feedback
How does reward modeling scale with increasing model and data size? We conduct an ablation to determine how data quantity and model size affect reward modeling performance. We train 7 reward models ranging from 160M to 13B parameters, on 8k to 64k human comparisons from our dataset. We find that doubling the training data amount leads to a~1.1% increase in the reward model validation set accuracy, whereas doubling the model size leads to a~1.8% increase ( Figure 6).
Learning to summarize from human feedback
What has the reward model learned? We probe our reward model by evaluating it on several validation sets. We show the full results in Appendix G.6, and highlight them here. We find that our reward models generalize to evaluating CNN/DM summaries (Appendix G.7), agreeing with labeler preferences 62.4% and 66.5% of the time (for our 1.3B and 6.7B models, respectively). Our 6.7B reward model nearly matches the inter-labeler agreement value of 66.9%.
Learning to summarize from human feedback
We also find that our reward models are sensitive to small but semantically important details in the summary. We construct an additional validation set by having labelers make minimal edits to summaries to improve them. Our RMs prefer the edited summaries almost as often (79.4% for 1.3B and 82.8% for 6.7B) as a separate set of human evaluators (84.1%). Further, when comparing the reference summaries to perturbed summaries where the participants' roles are reversed, our models reliably select the original summary (92.9% of the time for 1.3B, 97.2% for 6.7B). However, our RMs are biased towards longer summaries: our 6.7B RM prefers improving edits that make the summary shorter only 62.6% of the time (vs. 76.4% for humans).
Learning to summarize from human feedback
Evaluation. We study how well various automatic metrics act as predictors for human preferences, and compare them to our RMs. Specifically, we examine ROUGE, summary length, amount of copying from the post, 9 and log probability under our baseline supervised models. We present a full matrix of agreement rates between these metrics in Appendix G.7.
Learning to summarize from human feedback
We find that our learned reward models consistently outperform other metrics, even on the CNN/DM dataset on which it was never trained. We also find that ROUGE fails to track sample quality as our Figure 7: Summary quality as a function of metric optimized and amount of optimization, using best-of-N rejection sampling. We evaluate ROUGE, our main reward models, and an earlier iteration of the 1.3B model trained on approximately 75% as much data (see Table 11 for details). ROUGE appears to peak both sooner and at a substantially lower preference rate than all reward models. Details in Appendix G.3. models improve. While ROUGE has~57% agreement with labelers when comparing samples from our supervised baseline models, this drops to~50% for samples from our human feedback model.
Learning to summarize from human feedback
Similarly, log probability agreement with humans drops to ≤50% on comparisons between samples from our human feedback models, while our RMs still perform above chance (62%). Scaling up the size of the supervised model does not reliably improve log probability's agreement with labelers.
Learning to summarize from human feedback
Optimization. In Figure 7, we show that optimizing ROUGE using a simple optimization scheme doesn't consistently increase quality, as has been noted in [45]. Optimization against ROUGE peaks both sooner and at a substantially lower quality rate than optimization against our reward models.
Learning to summarize from human feedback
Limitations. One limitation of our work is the time and cost required to produce our final models. Notably, fine-tuning our 6.7B model with RL required approximately 320 GPU-days. Our data collection procedure is also expensive compared to prior work -the training set took thousands of labeler hours and required significant researcher time to ensure quality. For this reason, we were unable to collect baselines such as an equivalent amount of high-quality human demonstrations for supervised baselines. See D for more discussion. We leave this ablation to future work. Nevertheless, we believe reward modeling is more likely to scale to tasks where it is extremely skill-intensive or time-consuming to provide good demonstrations.
Learning to summarize from human feedback
Future directions. The methods in this paper could be applied to any task where humans can compare samples, including dialogue, machine translation, question answering, speech synthesis, and music generation. We expect this method to be particularly important for generating long samples, where the distributional shift and degeneracy of maximum likelihood samples can be problematic. It may be possible to improve sample efficiency by training to predict feedback across many tasks [42].
Learning to summarize from human feedback
We are particularly interested in scaling human feedback to tasks where humans can't easily evaluate the quality of model outputs. In this setting, it is particularly challenging to identify whether an ML system is aligned with the human designer's intentions. One approach is to train ML systems to help humans perform the evaluation task quickly and accurately [9].
Learning to summarize from human feedback
There is also a rich landscape of human feedback methods beyond binary comparisons that could be explored for training models [28,17,44,64]. For example, we could solicit high-quality demonstrations from labelers, have labelers edit model outputs to make them better, or have labelers provide explanations for why they preferred one model output over another. All of this feedback could be leveraged as a signal to train more capable reward models and policies.
Learning to summarize from human feedback
Broader impacts. The techniques we explore in this paper are generic techniques that could be used in a wide variety of machine learning applications, for any task where it is feasible for humans to evaluate the quality of model outputs. Thus, the potential implications are quite broad.
Learning to summarize from human feedback
Our research is primarily motivated by the potential positive effects of aligning machine learning algorithms with the designer's preferences. Many machine learning applications optimize simple metrics which are only rough proxies for what the designer intends. This can lead to problems, such as Youtube recommendations promoting click-bait [11]. In the short term, improving techniques for learning from and optimizing human preferences directly may enable these applications to be more aligned with human well-being.
Learning to summarize from human feedback
In the long term, as machine learning systems become more capable it will likely become increasingly difficult to ensure that they are behaving safely: the mistakes they make might be more difficult to spot, and the consequences will be more severe. For instance, writing an inaccurate summary of a news article is both easy to notice (one simply has to read the original article) and has fairly low consequences. On the other hand, imitating human driving may be substantially less safe than driving to optimize human preferences. We believe that the techniques we explore in this paper are promising steps towards mitigating the risks from such capable systems, and better aligning them with what humans care about.
Learning to summarize from human feedback
Unfortunately, our techniques also enable malicious actors to more easily train models that cause societal harm. For instance, one could use human feedback to fine-tune a language model to be more persuasive and manipulate humans' beliefs, or to induce dependence of humans on the technology, or to generate large amounts of toxic or hurtful content intended to harm specific individuals. Avoiding these outcomes is a significant challenge for which there are few obvious solutions.
Learning to summarize from human feedback
Large-scale models trained with human feedback could have significant impacts on many groups. Thus, it is important to be careful about how we define the 'good' model behavior that human labelers will reinforce. Deciding what makes a good summary is fairly straightforward, but doing this for tasks with more complex objectives, where different humans might disagree on the correct model behavior, will require significant care. In these cases, it is likely not appropriate to use researcher labels as the 'gold standard'; rather, individuals from groups impacted by the technology should be included in the process to define 'good' behavior, and hired as labelers to reinforce this behavior in the model.
Learning to summarize from human feedback
We chose to train on the Reddit TL;DR dataset because the summarization task is significantly more challenging than on CNN/DM. However, since the dataset consists of user-submitted posts with minimal moderation, they often contain content that is offensive or reflects harmful social biases. This means our models can generate biased or offensive summaries, as they have been trained to summarize such content. For this reason, we recommend that the potential harms of our models be thoroughly studied before deploying them in user-facing applications.
Learning to summarize from human feedback
Finally, by improving the ability of machine learning algorithms to perform tasks that were previously only achievable by humans, we are increasing the likelihood of many jobs being automated, potentially leading to significant job loss. Without suitable policies targeted at mitigating the effects of large-scale unemployment, this could also lead to significant societal harm.   Here, we discuss the pre-processing steps that we apply to the TL;DR dataset. We first remove all duplicate posts by checking the text body, finding that there are nearly 20,000 exact duplicates. We then re-parse the TL;DR carefully using a set of heuristics, and filter to use only top-level posts (rather than comments). We also filter out any post that is from a subreddit not in our 'subreddit whitelist' (see Table 2 for the distribution over subreddits), any post where the title starts with some variant of 'Edit' or 'Update', 10 and posts that contain certain topics (such as graphic sex or suicide) using heuristics. Finally, to ensure the posts are short enough to fit into the context length of our models, we filter out any post whose body is longer than 512 tokens. This resulted in a set of 287,790 posts filtered by body but not summary, of which we hold out approximately 5% as a validation set. We used this set of posts for RL training since our RL procedure does not require reference summaries.
Learning to summarize from human feedback
We next perform additional filtering on the parsed reference summaries that we use for training our supervised baselines. Specifically, we remove summaries where the TL;DR starts with variants of 'Edit', 'Update', or 'P.S.', we heuristically remove summaries with certain levels of profanity, and we remove summaries that are less than 24 tokens or more than 48 tokens. As discussed in Section 4.1, since our RL models tend to generate summaries on the upper end of the allowed length limit, this length filtering ensures that there is enough length overlap between the RL summaries and reference summaries for us to perform a length-controlled analysis. Additionally, we found that summaries shorter than 16 tokens were usually of low quality. We later verified that the summaries we filtered out were lower quality according to our reward model -more than 0.5 nats worse on average (i.e. they are predicted to be exp(0.5) ≈ 1.6 times less likely to be preferred). Our final TL;DR dataset contains 123,169 posts including summaries, again with about 5% held out as a validation set. We use 1913 of these validation articles for model selection during development; the evaluations in this paper exclude these articles.
Learning to summarize from human feedback
Note that, from Table 2 we can see that about two thirds of our TL;DR dataset consists of posts relating to relationships or relationship advice, which is a fairly specific domain. This raises potential concerns about the generality of our models, though their strong transfer performance on CNN/DM news articles suggests they are not unreasonably specialized to relationship advice.  Table 3: Hyperparameters for our models of various sizes. Figure 8: The sweep we conducted for determining our sampling procedure, varying the temperature and the 'top p' value for nucleus sampling. While we didn't do a large enough test to determine whether nucleus sampling is better or worse than moderate-temperature sampling, we found that very low temperature sampling is better than both on this task.
Learning to summarize from human feedback
All models follow the standard Transformer architecture, with 2048 learned position embeddings. All models are trained with fp16 activations and the Adam optimizer [31]. Nearly all supervised baselines, reward models, and reinforcement learning models are trained with fp32 weights; the exception is our TL;DR supervised baselines, which were trained with fp16 weights. 11 All models are trained with the same byte-pair encoding as in [48].
Learning to summarize from human feedback
During pretraining, the models were trained to predict the next token on a large text corpus consisting of Commoncrawl, Webtext [48], books, and Wikipedia. Training lasts between 1-3 epochs on each, for a total of 200-300 billion tokens. Learning rate follows a cosine schedule, with a short warmup, decaying to 10% of the maximum value. The batch size ramped up throughout training to some maximum, with each input having 2048 tokens. Hyperparameters for each model are shown in Table 3.
Learning to summarize from human feedback
For supervised baselines, we initialize models from the pretrained models. We decay the learning rate with a cosine schedule, using an initial learning rate chosen from a log linear sweep of at least 7 values. This resulted in learning rates of 6.35e-5, 5.66e-5, 2.83e-5, and 2.83e-5 for our TL;DR models of size 1.3B, 3B, 6.7B, and 13B respectively, and a learning rate of 2.38e-5 for our CNN/DM 6.7B model. We use a batch size of 128, and run for a single epoch.
Learning to summarize from human feedback
For reward modeling, we initialize to the supervised baseline, but with a reward head on top with weights initialized according to N (0, 1/(d model + 1)) [20]. We train for one epoch, decaying the learning rate with a cosine schedule, using an initial learning rate chosen from a log linear sweep of at least 7 values. We also sweep over between 3 and 10 seeds, and choose the reward model that performs best on the development portion of the validation set, as we find that both the data iteration order and reward head initialization affect results [13]. For our main results, the 1.3B and 6.7B reward models had learning rates of 1.5e-5 and 5e-6, respectively. We use a batch size of 64, and run for a single epoch.
Learning to summarize from human feedback
For PPO, we run with separate policy and value networks, initializing our policies to the supervised baseline, and our value functions to the reward model. We set γ = 1 and λ = 0.95 for the advantage estimation [57] and do 4 epochs of optimization for each batch of rollouts. We used a linear learning rate decay schedule, with initial learning rates of 1.5e-5 for the 1.3B model and 7e-6 for the 6.7B model, based on small amounts of experimentation and rough model size extrapolation. We used a KL coefficient of 0.05 for both of the main runs we report results for (except when we explicitly vary this value in the reward model optimization graphs). We use a batch size of 512 for the 1.3B model and 256 for the 6.7B model, and run for 1 million episodes.
Learning to summarize from human feedback
Our model always receives a byte-pair encoded string of a fixed size. When the input is too small, we pad from the beginning of the input with a padding token, and if the input is too long we truncate the post/article field at newlines to stay under the limit.
Learning to summarize from human feedback
When sampling from models pretrained only on our pretrain mixture and not fine-tuned on TL;DR, we follow [48] and instead of padding with a padding token, we pad the beginning of the context with examples of posts/articles and high-quality summaries. We use as many examples as will fit in the token limit, with the examples formatted the same way as the main input. Table 4 documents the formats we used (with pythonic format strings).
Learning to summarize from human feedback
We first detail the procedures we use to ensure high-quality data. While these procedures became more rigorous over the course of the project, they generally involved four steps.
Learning to summarize from human feedback
Step 0: Understanding the task ourselves. To understand the task, we first do many summary comparisons ourselves. We also hire a small number of human labelers 12 to do comparisons, and discuss our disagreements. We then draft instructions for a larger set of human labelers.
Learning to summarize from human feedback
Step 1: Labeler onboarding. Labelers are hired from Upwork, a freelancing platform, as well as two labeling services, Scale and Lionbridge. Labelers first complete a (paid) training process where they label summaries on a shared set of data. For some comparisons, labelers get immediate feedback about which summary was chosen by us, and why, to help them calibrate. We retain labelers that pass a minimum threshold for speed and agreement with us. To allow for a customizable labeler interface, we built our own website for data collection (see Appendix C.4).
Learning to summarize from human feedback
Step 2: Collecting comparison data. Next, we have labelers evaluate a large batch of comparisons on our website, which generates the bulk of our data. Before comparing two summaries directly, we have labelers write their 'naive interpretations' of summaries without seeing the original post. We've found this helpful for evaluating summaries, as they surface points of ambiguity in the summary that might not have been detected if the summary was read after the original post. After doing naive interpretations, labelers do comparisons by assigning a value on a 9-point scale for how confident they are that summary A is better than summary B (or the converse).
Learning to summarize from human feedback
Step 3: Providing labeler feedback. After collecting the comparison data, we can look at agreement rates between labelers. While most comparisons are only given to a single labeler, each labeler gets about 10-20% questions from a shared pool for calibration purposes. We can both attempt to use these statistics as crude measures of quality, and show cases of disagreements to workers to help them improve their labels.
Learning to summarize from human feedback
Step 4: Researcher comparison calibrations. We occasionally also do the task ourselves, to measure agreement rates between each labeler and us. This is used for quality assessment (see C.2). We also calculate per-labeler "high confidence" thresholds, by finding the confidence value on the Likert scale for each labeler such that we expect labels above this threshold to agree with us 80% of the time on average. For the purposes of reward model selection, we filter the validation set to contain only these higher confidence labels. For the entire process, we keep a high communication bandwidth with labelers: we use a shared chat room for labelers to ask clarifying questions and discuss difficult comparisons amongst themselves, host office hours, and occasionally have one-on-one video calls with labelers to discuss points of disagreement.
Learning to summarize from human feedback
We keep good labelers throughout the lifetime of the project, while firing the lowest-performing workers.
Learning to summarize from human feedback
We assess labeler accuracy by comparing the labeler's preferred summary with the summary we prefer (ignoring the confidence level). We exclude comparisons where either the labeler or researcher expresses indifference. This gives us an agreement rate, in theory ranging from 0% (perfect disagreement) to 100% (perfect agreement). For our 2-way comparisons, a random labeler would get 50% agreement.
Learning to summarize from human feedback
To obtain our main number comparing labeler-researcher to researcher-researcher agreement, we restrict ourselves to comparisons between summaries from our 1.3B supervised baseline, because this subset of the data has the most researcher-labeled data. On this subset, labelers agree with researchers 77% ± 2% of the time, while researchers agree with each other 73% ± 4% of the time. We believe substantial noise comes from comparisons being quite difficult and subjective.
Learning to summarize from human feedback
In general, agreement rates range from about 65% for the least proficient labelers and most difficult comparisons (comparing two high-temperature samples from a single RL policy) to about 85% for (a) (b) Figure 9: (a) The website we made to collect data from labelers. (b) Naive interpretations of summaries on the website. the most proficient labelers and easiest comparisons (comparing a high-temperature sample from a supervised baseline to the reference summary). Averaging over all workers, weighted by their volume, gives us an estimated agreement rate of 73% ± 3% for our reward model training corpus.
Learning to summarize from human feedback
Labelers agree with each other 72% of the time in the training corpus. This suggests we could get more reliable labels by aggregating labels from multiple workers on the same comparison. Indeed, on the subset of the training data for which we have enough shared comparisons, taking the modal label from 3 labelers increases their agreement rate with researchers from 72% to 77%. However, we usually collect only one label per comparison, in order to maximize label throughput.
Learning to summarize from human feedback
On the evaluations for Figure 1, labelers agreed with researchers 73% ± 3% of the time, and labelers agreed with each other 73% ± 2% of the time.
Learning to summarize from human feedback
Agreement rate between researchers ranged from about 65% on the most difficult comparisons (comparing two high-temperature samples from a single RL policy), to about 80% on the easiest comparisons (comparing a high-temperature sample from a supervised baseline to the human reference summary), to about 95% in cases where we discussed the comparisons with each other.
Learning to summarize from human feedback
Overall we believe that quality is fairly high. Our attempts to filter data generally hurt reward model accuracy. For example, using the confidence thresholds mentioned above, we found that while lower-confidence labels were less useful than high-confidence labels for improving reward model accuracy, they were still better to include than to omit. Similarly, leaving out workers with poorer agreement rates did not help.
Learning to summarize from human feedback
When training machine learning models with human feedback, the humans providing the feedback are essential in reinforcing the desired model behavior. If we are to scale human feedback to train models on more complex tasks, where humans might disagree about what the desired model behavior should be, it's important for members of groups that will be impacted by the model to be included in the labeler population.
Learning to summarize from human feedback
To provide more transparency into our labeler demographics, we provide results from a survey given to our labelers in Table 5. The survey was optional, anonymous, and it was made clear that the results would not affect hiring or firing decisions. We find that our labelers span a range of ethnicities, nationalities, ages, and genders, and educational backgrounds, but are more likely to be White and American.
Learning to summarize from human feedback
Since we hired and trained our own set of labelers, rather than using a crowdsourcing website such as Amazon Mechanical Turk, we built our own website to allow for a standardized, customizable user interface for all labelers. Each labeler created a separate profile, allowing us to assign different sets of comparisons to different labelers. The website contains different renderers for different kinds  Table 5: Demographic data from 21 of our labelers who participated in our voluntary survey. of questions, including naive interpretations, summary comparisons, and Likert evaluations along different axes, along with room for labelers to express concerns with the question or explanations for their decision. Screenshots from the website are shown in Figure 9. Data collected from the website can be easily ported into a central database containing all of our human data.
Learning to summarize from human feedback
Here we provide more detail on the specific instructions given to labelers for comparing summaries, and for doing Likert evaluations of summaries along axes of quality. We produced separate sets of instructions for evaluating Reddit posts, and for evaluating CNN/DM news articles. For Reddit instructions, we first describe Reddit in general and provide a table that translates Reddit-specific lingo into common parlance.
Learning to summarize from human feedback
Instructions for comparing summaries. We show an excerpt of the instructions given to labelers for making comparisons in Table 6. In addition to these instructions, we provide an example labeled comparison between Reddit summaries, and also example naive interpretations for summaries.
Learning to summarize from human feedback
Instructions for evaluating summaries along axes of quality. We provide a separate set of detailed instructions for labelers for the 7-point Likert evaluations. We first introduce each of the 4 axes of quality we consider, giving an overview of coherence, accuracy, coverage, and overall score (shown in Table 7). We also provide a brief rubric for giving scores of 1, 4, and 7, along with several Reddit summaries annotated with our own judgments of quality along each of these axes (with explanations).
Learning to summarize from human feedback
What makes for a good summary? Roughly speaking, a good summary is a shorter piece of text that has the essence of the original -tries to accomplish the same purpose and conveys the same information as the original post. We would like you to consider these different dimensions of summaries:
Learning to summarize from human feedback
Essence: is the summary a good representation of the post?
Learning to summarize from human feedback
Clarity: is the summary reader-friendly? Does it express ideas clearly?
Learning to summarize from human feedback
Accuracy: does the summary contain the same information as the longer post?
Learning to summarize from human feedback
Purpose: does the summary serve the same purpose as the original post?
Learning to summarize from human feedback
Concise: is the summary short and to-the-point?
Learning to summarize from human feedback
Style: is the summary written in the same style as the original post?
Learning to summarize from human feedback
Generally speaking, we give higher weight to the dimensions at the top of the list. Things are complicated though -none of these dimensions are simple yes/no matters, and there aren't hard and fast rules for trading off different dimensions. This is something you'll pick up through practice and feedback on our website. Table 6: An excerpt from the instructions we gave to labelers for doing comparisons.
Learning to summarize from human feedback
Finally, we provide a FAQ section that answers common questions raised by the small initial set of labelers we assigned to this task.
Learning to summarize from human feedback
For CNN/DM, we provide the same set of instructions, except we add some additional clarifications for how to judge news articles. We specifically ask labelers to place less emphasis on fluidity of sentences (because the reference summaries were originally written in bullet-point form, and we didn't want labelers to penalize this), and to place less emphasis on the summary matching the intent of the article (which was important for Reddit summaries).
Learning to summarize from human feedback
In terms of quality control, we conducted a smaller version of the quality control process described in Appendix C.1: we first labeled a small set of summaries ourselves along each axis to understand points of confusion, then we wrote the instructions document to provide to labelers, then we had a small number of labelers do a trial of the task to catch any remaining bugs or points of confusion, and finally we onboarded a larger set of labelers onto the task while remaining available to answer any questions.
Learning to summarize from human feedback
Over the course of the project, we trained several reward models and policies. Each batch of summaries that we sent to the labelers were sampled from a variety of policies. We didn't have a systematic plan for which policies to sample from; rather, we chose what seemed best at the time in the spirit of exploratory research. Every time we trained a reward model, we trained on all labels we had collected so far. Successive models also benefited from improved hyperparameters and dataset cleaning. Our results could likely be replicated with a simpler, more systematic approach.
Learning to summarize from human feedback
In general, as we hire new labelers and as existing labelers perform the task more, it is possible that there is 'labeler drift', where the set of criteria used by labelers to evaluate summaries gradually shifts over time. This could lead to a regression in labeler-researcher disagreement, or lead to some policies becoming more or less preferred over time. To help guard against this, in most batches we include comparisons between samples from our supervised baseline and reference summaries, and measure the frequency with which the workers prefer one over the other. If this number drifts over time, it's an indication that our workers' preferences are also changing. However, we generally found that this preference number stayed relatively constant, within noise. Table 8 lists the policies we trained by supervised finetuning on the TL;DR dataset, as well as the reward models, trained on successively larger datasets of human labels. Table 9 lists the RL policies.
Learning to summarize from human feedback
For this axis, answer the question "how coherent is the summary on its own?" A summary is coherent if, when read by itself, it's easy to understand and free of English errors. A summary is not coherent if it's difficult to understand what the summary is trying to say. Generally, it's more important that the summary is understandable than it being free of grammar errors.
Learning to summarize from human feedback
Score of 1: The summary is impossible to understand. Score of 4: The summary has mistakes or confusing phrasing that make it a bit hard to understand. Score of 7: The summary is perfectly clear. 
Learning to summarize from human feedback
For this axis, answer the question "how good is the summary overall at representing the post?" This can encompass all of the above axes of quality, as well as others you feel are important. If it's hard to find ways to make the summary better, give the summary a high score. If there are lots of different ways the summary can be made better, give the summary a low score.
Learning to summarize from human feedback
Score of 1: The summary is terrible. Score of 4: The summary is an okay representation of the post, but could be significantly improved. Score of 7: The summary is an excellent representation of the post.  Table 10: Best-of-N policies. KL divergence is computed analytically as KL(boN, sup) = log N -(N-1)/N. We also explored a simple alternative to reinforcement learning: Sample N summaries from a supervised baseline at temperature 0.7, score them with a reward model, and take the summary with the highest score. This best-of-N (BoN) procedure is effectively a mildly optimized policy requiring no training. These policies are named in Table 10, and samples from them form part of the training data. Table 11 lists the source policies for the training data for each reward model. Reward model Policy0  Policy1   rm1  ref  sup1  5404  sup1  sup1  5386  rm2  ref  sup1  5404  sup2  12779  sup2 bo8 rm1  1426  sup3_6b  1424  sup1  sup1  5386   Continued on next page  Label count  Reward model Policy0  Policy1   sup4 bo128 rm3  sup4 bo128 rm3  288  sup4 bo256 rm3  582  sup4 bo128 rm3_6b sup4 bo128 rm3_6b  95  sup4 bo256 rm3_6b  203  sup4 bo512 rm3  sup4 ppo rm3 3  216  sup4_6b  60  sup4 bo64 rm3  sup4 ppo rm3 2  218  sup4_6b  55  sup4 bo8 rm3 sup4 ppo rm3 1 752 sup4 ppo rm3 1 sup4 ppo rm3 1 372 sup4 ppo rm3 2 sup4 ppo rm3 2 4256 sup4_6b 215 sup4 ppo rm3 3 sup4 ppo rm3 3 4037 sup4_6b 216 Table 11: Training data for reward models. "ref" refers to human reference summaries.
Learning to summarize from human feedback
To give a sense of the difficulty of the comparisons task, we provide example comparisons between two summaries generated by our 6.7B human feedback model. In Table 12 we show both a random comparison drawn from the TL;DR dataset, and a cherry-picked comparison (selected from 10 comparisons where labelers disagreed) to illustrate the trade-off between accuracy in coverage that can occur when labelers conduct evaluations. I'm sure it's not of the norm for me to feel so disassociated about the whole thing, but I am. I'm suppose to go look at wedding dresses this Friday. I am feeling super anxious because I don't know if trying on wedding dresses is going to turn me into a blubbering baby about not having a mom. My future mother-in-law is suppose to come with me to help look. I worry about turning into that blubbering baby and offending her. I don't want her thinking that I don't appreciate her being there.
Learning to summarize from human feedback
Aside from me worrying about becoming a giant baby, I've also been having issues with my bridal party. While I haven't made any official choices, I have ideas of who I want involved. That would be my best friend, my sister, and my future sister-in-law. My first choice for a MOH is my best friend. However, she lives out of state, and is in a medical program for school. So her visit time is severely limited. My sister feels entitled to be the MOH, despite the fact that we are not close at all. So getting people together to get any kind of wedding stuff done is almost impossible. Summary A: I'm having doubts about whether or not to try on wedding dresses. I am also having doubts about my bridal party's ability to get things done. Summary B: I think I'm going to turn into a blubbering baby and offend my mother-in-law. 
Learning to summarize from human feedback
In testing our human feedback techniques, we collected a large amount of high-quality data from human labelers. In order to compare fairly against supervision-based techniques, we would have needed to spend a similar amount of labeler time collecting high quality demonstrations, and used those to fine-tune a model via supervised learning. Because this is prohibitively expensive, we do not provide such a baseline.
Learning to summarize from human feedback
Existing prior work such as PEGASUS [70] has studied supervised methods on a dataset very similar to ours (the /r/tifu subset of TL;DR). However, they use much smaller (500M parameters) models, and report that their model outputs are worse than the human reference summaries, according to human evaluations. Thus, due to our limited labeler budget for evaluation, we decided to use our own supervised and zero-shot models as baselines (after sanity-checking the ROUGE performance of our supervised models), as well as T5 [49].
Learning to summarize from human feedback
T5 models [49] are pretrained and fine-tuned in a similar way to our supervised baselines, but they use an encoder-decoder architecture. We used T5 outputs which were obtained via beam search decoding, as described in [49]. We also carefully account for differences in tokenization between model outputs. 13
Learning to summarize from human feedback
On the CNN/DM dataset, our labelers significantly preferred lead-3 (a summary consisting of the first 3 sentences of the article) to reference summaries. In part this is due to longer summaries receiving higher coverage scores and lead-3 being 50% longer, as shown in Table 13.
Learning to summarize from human feedback
Policy Length (stdev) Quality Quality increase / 100 char. ref 314 (119) 5.54 0.14 lead-3 475 (114) 6.23 0.34 Table 13: How length affects overall quality on CNN/DM for lead-3 and reference summaries.
Learning to summarize from human feedback
However, if we use a linear regression (similar to the procedure in Appendix F) to predict what lead-3 performance would be if its average length were reduced to 314 characters, we still find a quality of 5.68, modestly higher than the reference summaries. Moreover, for lead-3 to even achieve parity with the reference summaries seems to call into question the need for abstractive summarization or sophisticated ML methods, since a simple extractive baseline can match a perfect imitation of the reference summaries.
Learning to summarize from human feedback
We wanted to understand labeler behavior on these comparisons, to ensure that it was not an error. To do this, we examined a sample of our labeler's judgments ourselves. We found that in 20/143 cases labelers preferred lead-3 by 3 points or more, and that excluding these datapoints would raise the relative score of the reference summaries by about 0.5 points. 14 We were surprised to see the reference summaries performing so poorly in a significant fraction of cases, so we looked at labeler's explanations and confirmed they made sense.
Learning to summarize from human feedback
We found that two features of the reference summaries explained most of its underperformance. First, 13 of these 20 summaries omitted one of the key points from the article-the highlights are often written for a reader who had already seen the title of the article, even though the titles are not included in the CNN/DM dataset. Second, 10 of these 20 summaries actually introduced new information not present in the original article. From the perspective of labelers this information is totally confabulated and so led to lower scores. A likely explanation for these errors is that the reference summaries are extracted from "highlights" on the news sites rather than being a straightforward summary of the article. These failures are common enough that they significantly impact the average quality of the reference summaries, and the effects seem to be large relative to quality differences between ML models.
Learning to summarize from human feedback
Overall we believe that labeler judgments were reasonable in these cases, and that it is potentially problematic to treat the "highlights" in the CNN/DM dataset as reference summaries. You can view all of our labeler's judgments on CNN/DM at our website.  Figure 1, using the procedure described in Appendix F. Controlling for length reduces the relative preference of our human feedback models, however they are still preferred to the reference summaries. (b) Plotting model quality for different summary lengths on the TL;DR dataset. Our 6.7B human feedback model outperforms both the 6.7B supervised baseline and the reference summaries (horizontal line at 0.5) across lengths.
Learning to summarize from human feedback
As discussed in Section 4.1, the length of a summary is a confounding factor for evaluating summary quality; depending on the desired trade-off between conciseness and coverage, a shorter or longer summary might be better. Our models generate summaries that are longer than the reference summaries, as this led to higher labeler preference given the 24-48 token limit for our task. Here we describe the procedure we use to attempt to control for length.
Learning to summarize from human feedback
To calculate a single length-controlled preference number, we train a logistic regression model to predict the human-preferred summary on our dataset of human comparisons. We provide this model with 2 features: the identity of each policy, and the log ratio of the summary lengths. To calculate the length-controlled preference value between two policies, we simply give each policy ID to our trained logistic regression model and set the log length ratio to zero (see Figure 10a). In Figure 10b we examine summary quality across a range of summary lengths on TL;DR. We find that our human feedback model outperforms the supervised baseline across all length values.
Learning to summarize from human feedback
For CNN/DM, we use a similar procedure as described above to control for length, except using a linear regression model to predict the Likert rating from 1-7. We show the expected quality increase for making summaries 100 characters longer in Table 14, which suggests our human feedback models would perform better if they generated longer summaries.  G Additional results
Learning to summarize from human feedback
In this section, we conduct an ablation comparing using separate parameters for the value function and policy, against using a shared network as done in [73]. The results, shown in Figure 11, clearly indicate that using separate networks outperforms the latter. On the other hand, having separate networks increases the memory requirements of running RL fine-tuning. Having separate networks also allows us to initialize the value function to be the learned reward model that is being optimized. Figure 11: Comparing the reward obtained by optimizing with separate value function and reward model parameters to shared parameters.
Learning to summarize from human feedback
We show the full results of the evaluations of policies on a 7-point Likert scale along different axes of quality; for TL;DR this is shown in Figure 12, and for CNN/DM this is shown in Figure 13. It is evident that on both datasets coverage correlates strongly with overall score across models, and all models achieve a high coherence score.
Learning to summarize from human feedback
A natural way to evaluate an automatic evaluation metric is to see the extent to which optimizing against it leads to high performance according to humans. One way to assess this is to use best-of-N as an (inefficient) optimization technique -this has the benefits of being simple and invariant to monotonic transformations. We report results for up to best-of-2048 on ROUGE and three of our reward models in Figure 7, using samples from the 1.3B supervised baseline. The results suggest that optimizing against ROUGE significantly under-performs optimizing against our reward models. The data also suggests ROUGE degrades with too much optimization much faster than our reward models.
Learning to summarize from human feedback
With increasing N, the best-of-N policies get higher average reward. Similarly, by decreasing the KL coefficient β, the PPO policies get higher average reward. We found that at a given average reward, the best-of-N and PPO policies have similar quality as judged by human labelers (not shown). However, the PPO policy is farther from the supervised baseline than best-of-N is, as measured by the KL divergence. 15
Learning to summarize from human feedback
In Figure 14a and 14b, we show the ROUGE scores of our models on the TL;DR and CNN/DM datasets, respectively. We report results with T=0, consistent with our human evaluations. We found that temperature has an (often significant) impact on ROUGE score, and we did a thorough sweep to verify that the best temperature setting is T=0.   Table 15: Comparing the ROUGE score of our 6.7B supervised model on CNN/DM to recent SOTA models from the literature. Without any summarization-specific engineering, our model achieves ROUGE scores better than SOTA models from mid-2019, indicating that it is a strong baseline for comparison. On TL;DR, we find that our human feedback models obtain a slightly lower ROUGE score than the supervised models at T = 0, further indicating that ROUGE correlates poorly with human preferences. For supervised models, lowering temperature has a larger impact than increasing model size. Interestingly, at higher temperatures, our feedback models actually outperform supervised counterparts (not shown).
Learning to summarize from human feedback
On CNN/DM, ROUGE agrees with our human evaluations that our human feedback models transfer better than our supervised models. However, unsurprisingly, supervised CNN/DM models still achieve much higher ROUGE. In Table 15, we show the ROUGE results on CNN/DM for our 6.7B supervised baseline and various models from the literature. We find that our model achieves ROUGE scores less than T5 [49], but slightly greater than the CNN-2sent-hieco-RBM model from [71], which was SOTA for abstractive summarization on CNN/DM in mid-2019 according to the NLP-progress leaderboard. 16 
Learning to summarize from human feedback
In Table 16, we show the bigram overlap statistics for our models on the TL;DR and CNN/DM datasets as a proxy for how much the summaries copy frmo the post. As in Section 4.4, we compute the longest common subsequence of bigrams with the original Reddit post or news article, and dividing by the number of bigrams in the summary. We find that models evaluated on CNN/DM  (whether or not they were trained on CNN/DM) generally copy more than models evaluated on TL;DR. Further, our supervised and human feedback models copy less than our pretrained models.
Learning to summarize from human feedback
In this section, we report results evaluating our reward models on various manually constructed validation sets, shown in Tables 17 and 18. Notably, we asked our humans to produce a small dataset of edits, by having them make improvements to existing summaries (either reference summaries or supervised baseline summaries). Our 6.7B reward model prefer the improved summaries at a similar rate to humans (who do not know which summary has been edited).
Learning to summarize from human feedback
Our reward models are also sensitive to sentence shuffling (whereas metrics like ROUGE are largely not), and are able to detect when the roles portrayed in the summary have been switched. On the other hand, our reward models sometimes exhibit preference for poor artificial summaries, such as  Table 18: Reward model performance on various manually constructed validation sets. In all cases, Summary A is intended to be better than Summary B, and thus a higher preference % is generally better. 'rand-3' indicates a baseline where 3 random sentences are taken from the post; however these sentences are kept in the order in which they appear in the post. 'Original summary' is either the reference summary or a summary from our supervised baselines. r/tifu is a subreddit whose purpose is sharing embarrassing stories (not asking for advice).
Learning to summarize from human feedback
the post title copied twice, or asking for advice at the end of the summary. In Table 19, we show examples where our model is sensitive to small, semantically meaningful changes in the summary.
Learning to summarize from human feedback
We are interested in understanding the relationship between different metrics for evaluating summaries. To do this, we compute agreement between various metrics, including automatic metrics and humans, for different subsets of the data for which we have human evaluations. To remove policy quality as a confounding variable, all of the summary comparisons are generated by the same policy at the same temperature value. In Table 20, we use samples from our 1.3B supervised model at T=0.7 on TL;DR; Table 21 has comparisons from our 6.7B supervised model at T=0.7 on TL;DR; Table 22 has comparisons from our 6.7B human feedback model at T=0.7 on TL;DR; and Table 23 has comparisons from our 6.7B supervised baseline trained on CNN/DM. Our 6.7B reward model generally agrees with labelers as much as other labelers, although an ensemble of labelers does better. On the other hand, ROUGE generally has poor agreement, as does log probability under the supervised baselines, with simple heuristics like copying (longest common subsequence of bigrams with the article) and length often performing comparably.
Learning to summarize from human feedback
Reward ∆ Crush on girl I haven't seen in 4 years. She doesn't like me and I don't still like her. What do?
Learning to summarize from human feedback
A girl told me she loved liked me, she ended up picking another guy over me, that guy badly influenced her, and now I'm here alone thinking what could've been.
Learning to summarize from human feedback
+0.82 I tried to show my friend a picture of my tarantula and she smashed my phone with all her might and now I lost a good friend phone.
Learning to summarize from human feedback
-0.64
Learning to summarize from human feedback
Boyfriend still FB stalks his high school ex girlfriend from time to time and told me when he was very drunk that she was his first love.
Learning to summarize from human feedback
I've become pathetic, pining after a guy my ex. Would like to reach state of less pathetic. If more info is necessary, please let me know.
Learning to summarize from human feedback
I have body issues (body acne/scarring and weight issues) that prevent me from having a normal life without shame and prevent me from having a better sex life with my BF.
Learning to summarize from human feedback
Do you take someone back after they've turned you down off, even if you can't see them in person or are they just not worth the risk? +0.52 Table 19: Qualitative examples showing the change in reward of the reward model on humangenerated edits to TL;DR summaries that make the summaries better. Examples are randomly selected from the set where the edit distance was less than 5 and the magnitude of change in reward was greater than 0.5. Text in strike-through was removed from the original summary in the edit, and text in bold was added. The reward model is sensitive to small but semantically meaningful changes in the summary, although it makes errors on occasion.     
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Bandit structured prediction is the task of learning to solve complex joint prediction problems (like parsing or machine translation) under a very limited feedback model: a system must produce a single structured output (e.g., translation) and then the world reveals a score that measures how good or bad that output is, but provides neither a "correct" output nor feedback on any other possible output (Chang et al., 2015;Sokolov et al., 2015). Because of the extreme sparsity of this feedback, a common experimental setup is that one pre-trains a good-but-not-great "reference" system based on whatever labeled data is available, and then seeks to improve it over time using this bandit feedback.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
A common motivation for this problem setting is cost. In the case of translation, bilingual "experts" can read a source sentence and a possible translation, and can much more quickly provide a rating of that translation than they can produce a full translation on their own. Furthermore, one can often collect even less expensive ratings from "non-experts" who may or may not be bilingual (Hu et al., 2014). Breaking this reliance on expensive data could unlock previously ignored languages and speed development of broad-coverage machine translation systems.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
All work on bandit structured prediction we know makes an important simplifying assumption: the score provided by the world is exactly the score the system must optimize ( §2). In the case of parsing, the score is attachment score; in the case of machine translation, the score is (sentence-level) BLEU. While this simplifying assumption has been incredibly useful in building algorithms, it is highly unrealistic. Any time we want to optimize a system by collecting user feedback, we must take into account:
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
1. The metric we care about (e.g., expert ratings) may not correlate perfectly with the measure that the reference system was trained on (e.g., BLEU or log likelihood); 2. Human judgments might be more granular than traditional continuous metrics (e.g., thumbs up vs. thumbs down); 3. Human feedback have high variance (e.g., different raters might give different responses given the same system output); 4. Human feedback might be substantially skewed (e.g., a rater may think all system outputs are poor). Our first contribution is a strategy to simulate expert and non-expert ratings to evaluate the robustness of bandit structured prediction algorithms in general, in a more realistic environment ( §4). We construct a family of perturbations to capture three attributes: granularity, variance, and skew. We apply these perturbations on automatically generated scores to simulate noisy human ratings. To make our simulated ratings as realistic as possible, we study recent human evaluation data (Graham et al., 2017) and fit models to match the noise profiles in actual human ratings ( §4.2).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Our second contribution is a reinforcement learning solution to bandit structured prediction and a study of its robustness to these simulated human ratings ( § 3). 1 We combine an encoderdecoder architecture of machine translation (Luong et al., 2015) with the advantage actor-critic algorithm (Mnih et al., 2016), yielding an approach that is simple to implement but works on lowresource bandit machine translation. Even with substantially restricted granularity, with high variance feedback, or with skewed rewards, this combination improves pre-trained models ( §6). In particular, under realistic settings of our noise parameters, the algorithm's online reward and final heldout accuracies do not significantly degrade from a noise-free setting.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
The bandit structured prediction problem (Chang et al., 2015;Sokolov et al., 2015) is an extension of the contextual bandits problem (Kakade et al., 2008;Langford and Zhang, 2008) to structured prediction. Bandit structured prediction operates over time i = 1 . . . K as:
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
1. World reveals context x (i) 2. Algorithm predicts structured outputŷ (i) 3. World reveals reward R ŷ (i) , x (i)
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
We consider the problem of learning to translate from human ratings in a bandit structured prediction framework. In each round, a translation model receives a source sentence x (i) , produces a translationŷ (i) , and receives a rating R ŷ (i) , x (i) from a human that reflects the quality of the translation. We seek an algorithm that achieves high reward over K rounds (high cumulative reward). The challenge is that even though the model knows how good the translation is, it knows neither where its mistakes are nor what the "correct" translation looks like. It must balance exploration (finding new good predictions) 1 Our code is at https://github.com/ khanhptnk/bandit-nmt (in PyTorch). Figure 1: A translation rating interface provided by Facebook. Users see a sentence followed by its machined-generated translation and can give ratings from one to five stars.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
with exploitation (producing predictions it already knows are good). This is especially difficult in a task like machine translation, where, for a twenty token sentence with a vocabulary size of 50k, there are approximately 10 94 possible outputs, of which the algorithm gets to test exactly one.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Despite these challenges, learning from nonexpert ratings is desirable. In real-world scenarios, non-expert ratings are easy to collect but other stronger forms of feedback are prohibitively expensive. Platforms that offer translations can get quick feedback "for free" from their users to improve their systems ( Figure 1). Even in a setting in which annotators are paid, it is much less expensive to ask a bilingual speaker to provide a rating of a proposed translation than it is to pay a professional translator to produce one from scratch.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
This section describes the neural machine translation architecture of our system ( § 3.1). We formulate bandit neural machine translation as a reinforcement learning problem ( § 3.2) and discuss why standard actor-critic algorithms struggle with this problem ( § 3.3). Finally, we describe a more effective training approach based on the advantage actor-critic algorithm ( §3.4).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Our neural machine translation (NMT) model is a neural encoder-decoder that directly computes the probability of translating a target sentence y = (y 1 , · · · , y m ) from source sentence x:
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
where P θ (y t | y <t , x) is the probability of outputting the next word y t at time step t given a translation prefix y <t and a source sentence x.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
We use an encoder-decoder NMT architecture with global attention (Luong et al., 2015), where both the encoder and decoder are recurrent neural networks (RNN) (see Appendix A for a more detailed description). These models are normally trained by supervised learning, but as reference translations are not available in our setting, we use reinforcement learning methods, which only require numerical feedback to function.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
NMT generating process can be viewed as a Markov decision process on a continuous state space. The states are the hidden vectors h dec t generated by the decoder. The action space is the target language's vocabulary.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
To generate a translation from a source sentence x, an NMT model starts at an initial state h dec 0 : a representation of x computed by the encoder. At time step t, the model decides the next action to take by defining a stochastic policy P θ (y t | y <t , x), which is directly parametrized by the parameters θ of the model. This policy takes the current state h dec t−1 as input and produces a probability distribution over all actions (target vocabulary words). The next actionŷ t is chosen by taking arg max or sampling from this distribution. The model computes the next state h dec t by updating the current state h dec t−1 by the action takenŷ t . The objective of bandit NMT is to find a policy that maximizes the expected reward of translations sampled from the model's policy:
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
where D tr is the training set and R is the reward function (the rater). 2 We optimize this objective function with policy gradient methods. For a fixed x, the gradient of the objective in Eq 2 is:
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
where Q(ŷ <t ,ŷ t ) is the expected future reward of y t given the current prefixŷ <t , then continuing sampling from P θ to complete the translation:
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
withR(ŷ , x) ≡ R(ŷ , x)1 ŷ <t =ŷ <t ,ŷ t =ŷ t 2 Our raters are stochastic, but for simplicity we denote the reward as a function; it should be expected reward.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
1{·} is the indicator function, which returns 1 if the logic inside the bracket is true and returns 0 otherwise.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
The gradient in Eq 3 requires rating all possible translations, which is not feasible in bandit NMT. Naïve Monte Carlo reinforcement learning methods such as REINFORCE (Williams, 1992) estimates Q values by sample means but yields very high variance when the action space is large, leading to training instability.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Reinforcement learning methods that rely on function approximation are preferred when tackling bandit structured prediction with a large action space because they can capture similarities between structures and generalize to unseen regions of the structure space. The actor-critic algorithm (Konda and Tsitsiklis) uses function approximation to directly model the Q function, called the critic model. In our early attempts on bandit NMT, we adapted the actor-critic algorithm for NMT in Bahdanau et al. (2017), which employs the algorithm in a supervised learning setting. Specifically, while an encoder-decoder critic model Q ω as a substitute for the true Q function in Eq 3 enables taking the full expectation (because the critic model can be queried with any stateaction pair), we are unable to obtain reasonable results with this approach. Nevertheless, insights into why this approach fails on our problem explains the effectiveness of the approach discussed in the next section. There are two properties in Bahdanau et al. (2017) that our problem lacks but are key elements for a successful actor-critic. The first is access to reference translations: while the critic model is able to observe reference translations during training in their setting, bandit NMT assumes those are never available. The second is per-step rewards: while the reward function in their setting is known and can be exploited to compute immediate rewards after taking each action, in bandit NMT, the actorcritic algorithm struggles with credit assignment because it only receives reward when a translation is completed. Bahdanau et al. (2017) report that the algorithm degrades if rewards are delayed until the end, consistent with our observations.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
With an enormous action space of bandit NMT, approximating gradients with the Q critic model induces biases and potentially drives the model to wrong optima. Values of rarely taken actions are often overestimated without an explicit constraint between Q values of actions (e.g., a sum-to-one constraint). Bahdanau et al. (2017) add an ad-hoc regularization term to the loss function to mitigate this issue and further stablizes the algorithm with a delay update scheme, but at the same time introduces extra tuning hyper-parameters.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
We follow the approach of advantage actorcritic (Mnih et al., 2016, A2C) and combine it with the neural encoder-decoder architecture. The resulting algorithm-which we call NED-A2Capproximates the gradient in Eq 3 by a single sampleŷ ∼ P (· |x) and centers the reward R(ŷ) using the state-specific expected future reward V (ŷ <t ) to reduce variance:
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
We train a separate attention-based encoderdecoder model V ω to estimate V values. This model encodes a source sentence x and decodes a sampled translationŷ. At time step t, it computes
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
where h crt t is the current decoder's hidden vector and w is a learned weight vector. The critic model minimizes the MSE between its estimates and the true values:
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
We use a gradient approximation to update ω for a fixed x andŷ ∼ P (· |x):
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
NED-A2C is better suited for problems with a large action space and has other advantages over actor-critic. For large action spaces, approximating gradients using the V critic model induces lower biases than using the Q critic model. As implied by its definition, the V model is robust to biases incurred by rarely taken actions since rewards of those actions are weighted by very small probabilities in the expectation. In addition, the V model has a much smaller number of parameters and thus is more sample-efficient and more stable to train than the Q model. These attractive properties were not studied in A2C's original paper (Mnih et al., 2016).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Algorithm 1 The NED-A2C algorithm for bandit NMT.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
update the NMT model using Eq 5. 6: update the critic model using Eq 7. 7: end for Algorithm 1 summarizes NED-A2C for bandit NMT. For each x, we draw a single sampleŷ from the NMT model, which is used for both estimating gradients of the NMT model and the critic model. We run this algorithm with mini-batches of x and aggregate gradients over all x in a minibatch for each update. Although our focus is on bandit NMT, this algorithm naturally works with any bandit structured prediction problem.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Our goal is to establish the feasibility of using real human feedback to optimize a machine translation system, in a setting where one can collect expert feedback as well as a setting in which one only collects non-expert feedback. In all cases, we consider the expert feedback to be the "gold standard" that we wish to optimize. To establish the feasibility of driving learning from human feedback without doing a full, costly user study, we begin with a simulation study. The key aspects (Figure 2) of human feedback we capture are: (a) mismatch between training objective and feedbackmaximizing objective, (b) human ratings typically are binned ( § 4.1), (c) individual human ratings have high variance ( §4.2), and (d) non-expert ratings can be skewed with respect to expert ratings ( §4.3).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
In our simulated study, we begin by modeling gold standard human ratings using add-onesmoothed sentence-level BLEU (Chen and Cherry, 2014). 3 Our evaluation criteria, therefore, is average sentence-BLEU over the run of our algo- rithm. However, in any realistic scenario, human feedback will vary from its average, and so the reward that our algorithm receives will be a perturbed variant of sentence-BLEU. In particular, if the sentence-BLEU score is s ∈ [0, 1], the algorithm will only observe s ∼ pert(s), where pert is a perturbation distribution. Because our reference machine translation system is pre-trained using log-likelihood, there is already an (a) mismatch between training objective and feedback, so we focus on (b-d) below.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
When collecting human feedback, it is often more effective to collect discrete binned scores. A classic example is the Likert scale for human agreement (Likert, 1932) or star ratings for product reviews. Insisting that human judges provide continuous values (or feedback at too fine a granularity) can demotivate raters without improving rating quality (Preston and Colman, 2000).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
To model granular feedback, we use a simple rounding procedure. Given an integer parameter g for degree of granularity, we define: pert gran (s; g) = 1 g round(gs)
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
This perturbation function divides the range of possible outputs into g + 1 bins. For example, for g = 5, we obtain bins [0, 0.1), [0.1, 0.3), [0.3, 0.5), [0.5, 0.7), [0.7, 0.9) and [0.9, 1.0]. Since most sentence-BLEU scores are much closer to zero than to one, many of the larger bins are frequently vacant.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Human feedback has high variance around its expected value. A natural goal for a variance model of human annotators is to simulate-as closely as possible-how human raters actually perform. We use human evaluation data recently collected as part of the WMT shared task (Graham et al., 2017). The data consist of 7200 sentences multiply annotated by giving non-expert annotators on Amazon Mechanical Turk a reference sentence and a single system translation, and asking the raters to judge the adequacy of the translation. 4 From these data, we treat the average human rating as the ground truth and consider how individual human ratings vary around that mean. To visualize these results with kernel density estimates (standard normal kernels) of the standard deviation. Figure 3 shows the mean rating (x-axis) and the deviation of the human ratings (y-axis) at each mean. 5 As expected, the standard deviation is small at the extremes and large in the middle (this is a bounded interval), with a fairly large range in the middle: a translation whose average score is 50 can get human evaluation scores anywhere between 20 and 80 with high probability. We use a linear approximation to define our variance-based perturbation function as a Gaussian distribution, which is parameterized by a scale λ that grows or shrinks the variances (when λ = 1 this exactly matches the variance in the plot). pert var (s; λ) = Nor s, λσ(s) 2 (9) σ(s) = 0.64s − 0.02 if s < 50 −0.67s + 67.0 otherwise
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
The preceding two noise models assume that the reward closely models the value we want to optimize (has the same mean). This may not be the case with non-expert ratings. Non-expert 4 Typical machine translation evaluations evaluate pairs and ask annotators to choose which is better. 5 A current limitation of this model is that the simulated noise is i.i.d. conditioned on the rating (homoscedastic noise). While this is a stronger and more realistic model than assuming no noise, real noise is likely heteroscedastic: dependent on the input.  raters are skewed both for reinforcement learning (Thomaz et al., 2006;Thomaz and Breazeal, 2008;Loftin et al., 2014) and recommender systems (Herlocker et al., 2000;Adomavicius and Zhang, 2012), but are typically bimodal: some are harsh (typically provide very low scores, even for "okay" outputs) and some are motivational (providing high scores for "okay" outputs). We can model both harsh and motivations raters with a simple deterministic skew perturbation function, parametrized by a scalar ρ ∈ [0, ∞):
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
For ρ > 1, the rater is harsh; for ρ < 1, the rater is motivational.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
We choose two language pairs from different language families with different typological properties: German-to-English and (De-En) and Chinese-to-English (Zh-En). We use parallel transcriptions of TED talks for these pairs of languages from the machine translation track of the IWSLT 2014 and 2015 (Cettolo et al., 2014(Cettolo et al., , 2015(Cettolo et al., , 2012. For each language pair, we split its data into four sets for supervised training, bandit training, development and testing (Table 1). For English and German, we tokenize and clean sen-tences using Moses (Koehn et al., 2007). For Chinese, we use the Stanford Chinese word segmenter (Chang et al., 2008) to segment sentences and tokenize. We remove all sentences with length greater than 50, resulting in an average sentence length of 18. We use IWSLT 2015 data for supervised training and development, IWSLT 2014 data for bandit training and previous years' development and evaluation data for testing.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
For each task, we first use the supervised training set to pre-train a reference NMT model using supervised learning. On the same training set, we also pre-train the critic model with translations sampled from the pre-trained NMT model. Next, we enter a bandit learning mode where our models only observe the source sentences of the bandit training set. Unless specified differently, we train the NMT models with NED-A2C for one pass over the bandit training set. If a perturbation function is applied to Per-Sentence BLEU scores, it is only applied in this stage, not in the pre-training stage. We measure the improvement ∆S of an evaluation metric S due to bandit training: ∆S = S A2C − S ref , where S ref is the metric computed on the reference models and S A2C is the metric computed on models trained with NED-A2C. Our primary interest is Per-Sentence BLEU: average sentence-level BLEU of translations that are sampled and scored during the bandit learning pass. This metric represents average expert ratings, which we want to optimize for in real-world scenarios. We also measure Heldout BLEU: corpuslevel BLEU on an unseen test set, where translations are greedily decoded by the NMT models. This shows how much our method improves translation quality, since corpus-level BLEU correlates better with human judgments than sentence-level BLEU.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Because of randomness due to both the random sampling in the model for "exploration" as well as the randomness in the reward function, we repeat each experiment five times and report the mean results with 95% confidence intervals.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Both the NMT model and the critic model are encoder-decoder models with global attention (Luong et al., 2015). The encoder and the decoder are unidirectional single-layer LSTMs. They have the same word embedding size and LSTM hidden size of 500. The source and target vocabulary sizes are both 50K. We do not use dropout in our experiments. We train our models by the Adam optimizer (Kingma and Ba, 2015) with β 1 = 0.9, β 2 = 0.999 and a batch size of 64. For Adam's α hyperparameter, we use 10 −3 during pre-training and 10 −4 during bandit learning (for both the NMT model and the critic model). During pre-training, starting from the fifth pass, we decay α by a factor of 0.5 when perplexity on the development set increases. The NMT model reaches its highest corpus-level BLEU on the development set after ten passes through the supervised training data, while the critic model's training error stabilizes after five passes. The training speed is 18s/batch for supervised pre-training and 41s/batch for training with the NED-A2C algorithm.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
In this section, we describe the results of our experiments, broken into the following questions: how NED-A2C improves reference models ( §6.1); the effect the three perturbation functions have on the algorithm ( § 6.2); and whether the algorithm improves a corpus-level metric that corresponds well with human judgments ( §6.3).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
We evaluate our method in an ideal setting where un-perturbed Per-Sentence BLEU simulates ratings during both training and evaluation (Table 2).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Single round of feedback. In this setting, our models only observe each source sentence once and before producing its translation. On both De-En and Zh-En, NED-A2C improves Per-Sentence BLEU of reference models after only a single pass (+2.82 and +1.08 respectively).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Poor initialization. Policy gradient algorithms have difficulty improving from poor initializations, especially on problems with a large action space, because they use model-based exploration, which is ineffective when most actions have equal probabilities (Bahdanau et al., 2017;Ranzato et al., 2016). To see whether NED-A2C has this problem, we repeat the experiment with the same setup but with reference models pretrained for only a single pass. Surprisingly, NED-A2C is highly effective at improving these poorly Comparisons with supervised learning. To further demonstrate the effectiveness of NED-A2C, we compare it with training the reference models with supervised learning for a single pass on the bandit training set. Surprisingly, observing ground-truth translations barely improves the models in Per-Sentence BLEU when they are fully trained (less than +0.4 on both tasks). A possible explanation is that the models have already reached full capacity and do not benefit from more examples. 6 NED-A2C further enhances the models because it eliminates the mismatch between the supervised training objective and the evaluation objective. On weakly trained reference models, NED-A2C also significantly outperforms supervised learning (∆Per-Sentence BLEU of NED-A2C is over three times as large as those of supervised learning).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Multiple rounds of feedback. We examine if NED-A2C can improve the models even further with multiple rounds of feedback. 7 With supervised learning, the models can memorize the reference translations but, in this case, the models have to be able to exploit and explore effectively. We train the models with NED-A2C for five  Table 2: Translation scores and improvements based on a single round of un-perturbed bandit feedback. Per-Sentence BLEU and Heldout BLEU are not comparable: the former is sentence-BLEU, the latter is corpus-BLEU.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
passes and observe a much more significant ∆Per-Sentence BLEU than training for a single pass in both pairs of language (+6.73 on De-En and +4.56 on Zh-En) (Figure 4).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
We apply perturbation functions defined in § 4.1 to Per-Sentence BLEU scores and use the perturbed scores as rewards during bandit training ( Figure 5).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Granular Rewards. We discretize raw Per-Sentence BLEU scores using pert gran (s; g) ( §4.1). We vary g from one to ten (number of bins varies from two to eleven). Compared to continuous rewards, for both pairs of languages, ∆Per-Sentence BLEU is not affected with g at least five (at least six bins). As granularity decreases, ∆Per-Sentence BLEU monotonically degrades. However, even when g = 1 (scores are either 0 or 1), the models still improve by at least a point.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
High-variance Rewards. We simulate noisy rewards using the model of human rating variance pert var (s; λ) ( § 4.2) with λ ∈ {0.1, 0.2, 0.5, 1, 2, 5}. Our models can withstand an amount of about 20% the variance in our human eval data without dropping in ∆Per-Sentence BLEU. When the amount of variance attains 100%, matching the amount of variance in the human data, ∆Per-Sentence BLEU go down by about 30% for both pairs of languages. As more variance is injected, the models degrade quickly but still improve from the pre-trained models. Variance is the most detrimental type of perturbation to NED-A2C among the three aspects of human ratings we model.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Rewards. We model skewed raters using pert skew (s; ρ) ( § 4.3) with ρ ∈ {0.25, 0.5, 0.67, 1, 1.5, 2, 4}. NED-A2C is robust to skewed scores. ∆Per-Sentence BLEU is at least 90% of unskewed scores for most skew values. Only when the scores are extremely harsh (ρ = 4) does ∆Per-Sentence BLEU degrade significantly (most dramatically by 35% on Zh-En). At that degree of skew, a score of 0.3 is suppressed to be less than 0.08, giving little signal for the models to learn from. On the other spectrum, the models are less sensitive to motivating scores as Per-Sentence BLEU is unaffected on Zh-En and only decreases by 7% on De-En.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Our method also improves pre-trained models in Heldout BLEU, a metric that correlates with translation quality better than Per-Sentence BLEU (Table 2). When scores are perturbed by our rating model, we observe similar patterns as with Per-Sentence BLEU: the models are robust to most perturbations except when scores are very coarse, or very harsh, or have very high variance (Figure 5, second row). Supervised learning improves Heldout BLEU better, possibly because maximizing log-likelihood of reference translations correlates more strongly with maximizing Heldout BLEU of predicted translations than maximizing Per-Sentence BLEU of predicted translations.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Ratings provided by humans can be used as effective learning signals for machines. Reinforcement learning has become the de facto standard for incorporating this feedback across diverse tasks such as robot voice control (Tenorio-Gonzalez et al.,  (Isbell et al., 2001). Recently, this learning framework has been combined with recurrent neural networks to solve machine translation (Bahdanau et al., 2017), dialogue generation (Li et al., 2016), neural architecture search (Zoph and Le, 2017), and device placement (Mirhoseini et al., 2017). Other approaches to more general structured prediction under bandit feedback (Chang et al., 2015;Sokolov et al., 2016a,b) show the broader efficacy of this framework. Ranzato et al. (2016) describe MIXER for training neural encoder-decoder models, which is a reinforcement learning approach closely related to ours but requires a policy-mixing strategy and only uses a linear critic model. Among work on bandit MT, ours is closest to Kreutzer et al. (2017), which also tackle this problem using neural encoder-decoder models, but we (a) take advantage of a state-of-the-art reinforcement learning method; (b) devise a strategy to simulate noisy rewards; and (c) demonstrate the robustness of our method on noisy simulated rewards.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Our results show that bandit feedback can be an effective feedback mechanism for neural machine translation systems. This is despite that errors in human annotations hurt machine learning models in many NLP tasks (Snow et al., 2008). An obvious question is whether we could extend our framework to model individual annotator preferences (Passonneau and Carpenter, 2014) or learn personalized models (Mirkin et al., 2015;Rabinovich et al., 2017), and handle heteroscedastic noise (Park, 1966;Kersting et al., 2007;Antos et al., 2010). Another direction is to apply active learning techniques to reduce the sample complexity required to improve the systems or to extend to richer action spaces for problems like simultaneous translation, which requires prediction (Grissom II et al., 2014) and reordering (He et al., 2015) among other strategies to both minimize delay and effectively translate a sentence (He et al., 2016).
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Major progress in natural language processing has been made towards fully automating challenging tasks such as question answering, translation, and summarization. On the other hand, several studies have argued that machine learning systems that can explain their own predictions (Doshi-Velez and Kim, 2017) and learn interactively from their endusers (Amershi et al., 2014) can result in better user experiences and more effective learning systems. We develop NL-EDIT-an approach that employs both explanations and interaction in the context of semantic parsing.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Most existing systems frame semantic parsing as a one-shot translation from a natural language question to the corresponding logical form (e.g., SQL query) (Yu et al., 2018a;Guo et al., 2019;Wang et al., 2020, inter alia). A growing body of recent work demonstrates that semantic parsing systems can be improved by including users in the parsing loop-giving them the affordance to examine the parses, judge their correctness, and provide feedback accordingly. The feedback often comes in the form of a binary correct/incorrect * Most of the work was done while the first author was an intern at Microsoft Research.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
What is the full name of the candidate with the most votes? Edit: GROUP-BY: remove vote_id GROUP-BY: add candidate_id SELECT: add last_name Figure 1: Example human interaction with NL-EDIT to correct an initial parse through natural language feedback. In the Semantic Parsing Phase (top), an offthe-shelf parser generates an initial SQL query and provides an answer paired with an explanation of the generated SQL. In the Correction Phase (bottom), the user reviews the explanation and provides feedback that describes how the explanation should be corrected. The system parses the feedback as a set of edits that are applied to the initial parse to generate a corrected SQL.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
signal (Iyer et al., 2017), answers to a multiplechoice question posed by the system (Gur et al., 2018;Yao et al., 2019), or suggestions of edits that can be applied to the parse .
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Unlike other frameworks for interactive semantic parsing that typically expect users to judge the correctness of the execution result or induced logical form, Elgohary et al. (2020) introduced a framework for interactive text-to-SQL in which induced SQL queries are fully explained in natural lan-guage to users, who in turn, can correct such parses through natural language feedback (Figure 1). They construct the SPLASH dataset and use it to evaluate baselines for the semantic parse correction with natural language feedback task they introduce.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
We present a detailed analysis of the feedback and the differences between the initial (incorrect) and the correct parse. We argue that a correction model should be able to interpret the feedback in the context of other elements of the interaction (the original question, the schema, and the explanation of the initial parse). We observe from SPLASH that most feedback utterances tend to describe a few edits that the user desires to apply to the initial parse. As such, we pose the correction task as a semantic parsing problem that aims to convert natural language feedback to a sequence of edits that can be deterministically applied to the initial parse to correct it. We use the edit-based modeling framework to show that we can effectively generate synthetic data to pre-train the correction model leading to clear performance gains.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
We make the following contributions: (1) We present a scheme for representing SQL query Edits that benefits both the modeling and the analysis of the correction task, (2) we present NL-EDIT, an edit-based model for interactive text-to-SQL with natural language feedback. We show that NL-EDIT outperforms baselines in (Elgohary et al., 2020) by more than 16 points, (3) We demonstrate that we can generate synthetic data through the edit-based framing and that the model can effectively use this data to improve its accuracy and (4) We present a detailed analysis of the model performance including studying the effect of different components, generalization to errors of state-of-the-art parsers, and outline directions for future research.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
In the task of text-to-SQL parsing, the objective is given a database schema (tables, columns, and primary-foreign key relations) and a natural language question, generate a SQL query that answers the question when executed against the database. Several recent text-to-SQL models have been introduced (Yu et al., 2018a;Guo et al., 2019;Wang et al., 2020, inter alia) as a result of the availability of SPIDER (Yu et al., 2018b), a large dataset of schema, questions and gold parses spanning several databases in different domains.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
The task of SQL parse correction with natural language feedback (Elgohary et al., 2020) aims to correct an erroneous parse based on natural language feedback collected from the user. Given a question, a database schema, an incorrect initial parse, natural language feedback on the initial parse, the task is to generate a corrected parse.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
To study this problem, Elgohary et al. (2020) introduced the SPLASH dataset. SPLASH was created by showing annotators questions and a natural language explanation of incorrect parses and asking them to provide feedback, in natural language, to correct the parse. The dataset contained 9,314 question-feedback pairs. Like the SPIDER dataset, it was split into train-dev-test sets by database to encourage the models to generalize to new unseen databases. They contrast the task with conversational semantic parsing (Suhr et al., 2018;Yu et al., 2019b,a;Andreas et al., 2020) and show that the two tasks are distinct and are addressing different aspects of utilizing context. They establish several baseline models and show that the task is challenging for state-of-the-art semantic parsing models. We use these as baselines for this work.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
We define a scheme for representing the edits required to transform one SQL query to another. We use that scheme both in our model and analysis. Our goal is to balance the granularity of the editstoo fine-grained edits result in complex structures that are challenging for models to learn, and too coarse-grained edits result in less compact structures that are harder for models to generate.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
We view a SQL query as a set of clauses (e.g, SELECT, FROM, WHERE), each clause has a sequence of arguments ( Figure 2). We mirror the SQL clauses SELECT, FROM, WHERE, GROUP-BY, ORDER-BY, HAVING, and LIMIT. For subqueries, we define a clause SUBS whose arguments are recursively defined as sets of clauses. Subqueries can be linked to the main query in two ways: either through an IEU clause (mirrors SQL INTERSECT/EXCEPT/UNION) whose first argument is one of the keywords INTERSECT, EXCEPT, UNION and its second argument is a pointer to a subquery in SUBS. The second is through nested queries where the arguments of some of the clauses (e.g., WHERE) can point at subqueries in SUBS (e.g., "id NOT IN SUBS 1 ").
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
With such view of two queries P source and P target , we define their edit D source→target as SELECT: arg 1 :"id", arg 2 :"MAX(grade)"
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
arg 1 :"assignments" GROUP-BY: arg 1 :"id" 
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Target Edit <select> remove maximum grade </select> <select> add average grade </select> <where> remove id not one of </where> <orderby> add id </orderby> Linearize Figure 2: Edit for transforming the source query "SELECT id, MAX(grade) FROM assignments WHERE grade > 20 AND id NOT IN (SELECT id from graduates) GROUP BY id" to the target "SELECT id, AVG(grade) FROM assignment WHERE grade > 20 GROUP BY id ORDER BY id". The source and target are represented as sets of clauses (left and middle). The set of edits and its linearized form (Section 4) are shown on the right. Removing the condition "id NOT IN SUBS 1 " makes the subquery unreferenced, hence pruned from the edit. the set of clause-level edits {D c source→target } for all types of clauses c that appear in P source or P target (Figure 2). To compare two clauses of type c, we simply exact-match their arguments: unmatched arguments in the source (e.g., MAX(grade) in SELECT) are added as toremove arguments to the corresponding edit clause, and unmatched arguments in the target (e.g., "id" in the ORDER-BY) are added as to-add arguments.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Our current implementation follows SPIDER's assumption that the number of subqueries is at most one which implies that computing edits for different clauses can be done independently even for the clauses that reference a subquery (e.g., WHERE in Figure 2). The edit of the SUBS clause is recursively computed as the edit between two queries (any of them can be empty); the subquery of source and the subquery of target, i.e., D SUBS source→target = D source:SUBS 1 →target:SUBS 1 . We keep track of the edits to the arguments that reference the subquery. After all edit clauses are computed, we prune the edits of the SUBS clause if the subquery will no longer be referenced (SUBS 1 in Figure 2). We follow the SPIDER evaluation and discard the values in WHERE/HAVING clauses.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Throughout this paper, we refer to the number of add/remove operations in an edit as the Edit Size, and we denote it as |D source→target |. For example, the edit in Figure 2 is of size four.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
We follow the task description in Section 2: the inputs to the model are the elements of the interaction-question, schema, an initial parse P , and feedback. The model predicts a corrected P . The gold parse P is available for training. Our model is based on integrating two key ideas in an encoder-decoder architecture. We start with a discussion of the intuitions behind the two ideas followed by the model details.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Interpreting feedback in context: The feedback is expected to link to all the other elements of the interaction ( Figure 1). The feedback is provided in the context of the explanation of the initial parse, as a proxy to the parse itself. As such, the feedback tends to use the same terminology as the explanation. For example, the SQL explanations of (Elgohary et al., 2020) express "group by" in simple language "for each vote_id, find ...". As a result, human-provided feedback never uses "group by". We also notice that in several SPLASH examples, the feedback refers to particular steps in the explanation as in the examples in Figure 1. Unlike existing models (Elgohary et al., 2020), we replace the initial parse with its natural language explanation. Additionally, the feedback usually refers to columns/tables in the schema, and could often be ambiguous when examined in isolation. Such ambiguities can be usually resolved by relying on the context provided by the question. For example, "find last name" in Figure 1 is interpreted as "find last name besides first name" rather than "replace first name with last name" because the question asks for the "full name". Our first key idea is based on grounding the elements of the interaction by combining self-learned relations by transformer models (Vaswani et al., 2017) and hard-coded relations that we define according to the possible ways different elements can link to each other.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Feedback describes a set of edits: The difference between the erroneous parse and the correct one can mostly be described as a few edits that need to be applied to the initial parse to correct its errors (Section 7). Also, the feedback often only describes the edits to be made (Elgohary et al., 2020). As such, we can pose the task of correction with NL feedback as a semantic parsing task where we convert a natural language deception of  the edits to a canonical form that can be applied deterministically to the initial parse to generate the corrected one. We train our model to generate SQL Edits (Section 3) rather than SQL queries.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Our encoder ( Figure 3) starts with passing the concatenation of the feedback, explanation, question, and schema through BERT (Devlin et al., 2019). Following (Wang et al., 2020;Suhr et al., 2018;Scholak et al., 2020), we tokenize the column/table names and concatenate them in one sequence (Schema) starting with the tokens of the tables followed by the tokens of the columns. Then, we average the BERT embeddings of the tokens corresponding to each column (table) to obtain one representation for the column (table). Wang et al. (2020) study the text-to-SQL problem using the SPIDER dataset and show the benefit of injecting preexisting relations within the schema (column exists in a table, primary-foreign key), and between the question and schema items (column and table names) by: (1) name linking: link a question token to a column/table if the token and the item name match and (2) value linking: link a question token to a column if the token appears as a value under that column. To incorporate such relations in their model, they use the relation-aware self-attention formulation presented in (Shaw et al., 2018). The relation-aware transformer (Shaw et al., 2018) assigns a learned embedding for each relation type and combines such embeddings with the self-attention of the original transformer model (Vaswani et al., 2017): If a preexisting relation r holds between two tokens, the embedding of r is added as a bias term to the self-attention computation between the two tokens.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
In addition to those relations, we define a new set of relations that aim at contextualizing the feedback with respect to the other elements of the interaction in our setup: (1) [Feedback-Schema] We link the feedback to the schema the same way the question is linked to the schema via both name and value linking, (2) [Explanation-Schema] Columns and tables are mentioned with their exact names in the explanation. We link the explanation to the schema only through exact name matching, (3) [Feedback-Question] We use partial (at the lemma level) and exact matching to link tokens in the feedback and the question, (4) [Feedback-Explanation] We link tokens in the feedback to tokens in the explanation through partial and exact token matching. Since the feedback often refers to particular steps, we link the feedback tokens to explanation tokens that occur in steps that are referred to in the feedback with a separate relation type that indicates step reference in the feedback, and (5) [Explanation-Explanation] We link explanation tokens that occur within the same step. We use the same formulation of relationaware self-attention as (Wang et al., 2020) and add the relation-aware layers on top of BERT to integrate all relations into the model (Figure 3).
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Using a standard teacher-forced cross-entropy loss, we train our model to generate linearized SQL Edits (Figure 2). At training time, we compute the reference SQL Edit D P →P of the initial parse P and the gold parse P (Section 3). Then we linearize D P →P by listing the clause edits in a fixed order (FROM, WHERE, GROUP-BY, ... etc.). The argument of each clause-representing one add or remove operation-is formatted as <CLAUSE> ADD/REMOVE ARG </CLAUSE>. We express SQL operators in ARG with natural language explanation as in (Elgohary et al., 2020). For example, the argument "AVG(grade)" is expressed as "average grade". At inference time, we generate a corrected parse P by applying the produced edit to the initial parse P .
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
We use a standard transformer decoder that either generates tokens from the output vocab or copies columns and tables from the encoder output. Since all editing operations should be directed by the feedback, we tried splitting the attention to the encoder into two phases: First, we attend to the feedback only and update the decoder state  
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
In this section, we describe our process for automatically synthesizing additional examples for training the correction model. Recall that each example consists of a question about a given schema paired with a gold parse, an initial erroneous parse, and feedback. Starting with a seed of questions and their corresponding gold parses from SPIDER's training set (8,099 pairs) 1 , our synthesis process applies a sequence of SQL editing operations to the gold parse to reach an altered parse that we use as the initial parse (Algorithm 1). By manually inspecting the edits (Section 3) we induce for the initial and gold parses in SPLASH training set, we define 26 SQL editors and pair each editor with their most frequent corresponding feedback template(s) (Examples in Table 1). We also associate each editor with a set of constraints that determines whether it can be applied to a given SQL query (e.g., the "Remove-Limit" editor can only be applied to a query that has a limit clause).
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Algorithm 1 summarizes the synthesis process. We start by creating N (controls the size of the dataset) clones of each seed example. Elgohary et al. (2020)'s analysis of SPLASH shows that multiple mistakes might be present in the initial SQL, hence we allow our synthesis process to introduce up to four edits (randomly decided in line:4) to each clone p. For each editing step, we sample a feasible edit for the current parse (line:5) with man- We use BERT-base-uncased (Devlin et al., 2019) in all our experiments. We set the number of layers in the relational-aware transformer to eight (Wang et al., 2020) and the number of decoder layers to two. We train with batches of size 24. We use the Adam optimizer (Kingma and Ba, 2015) for training. We freeze BERT parameters during the first 5,000 warm-up steps and update the rest of the parameters with a linearly increasing learning rate from zero to 5 × 10 −4 . Then, we linearly decrease the learning rates from 5 × 10 −5 for BERT and  5 × 10 −4 for the other parameters to zero. 2 We use beam search with a beam of size 20 and take the top-ranked beam that results in a valid SQL after applying the inferred edit. Evaluation: We follow (Elgohary et al., 2020) and use the correction accuracy as our main evaluation measure: each example in SPLASH test set contains an initial parse P and a gold parse P . With a predicted (corrected) parse by a correction model P , they compute the correction accuracy using the exact-set-match (Yu et al., 2018b) between P and P averaged over all test examples. While useful, correction accuracy also has limitations. It expects models to be able to fully correct an erroneous parse with only one utterance of feedback as such, it is defined in terms of the exact match between the corrected and the gold parse. We find ( Table 2) that in several cases, models were still able to make progress by reducing the number of errors as measured by the edit size (Section 3) after correction. As such, we define another set of metrics to measure partial progress. We report (Edit ↓ and Edit ↑ in Table 2) the percentage of examples on which the size of the edit set strictly decreased/increased. To combine Edit ↓ and Edit ↑ in one measure and account for the relative reduction (increase) in the number of edits, we define
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Given a test set S, the Progress of a correction model is computed as the average relative edit reduction between the initial parse P and the gold parse P by predicting a correction P of P . A perfect model that can fully correct all errors in the initial parse would achieve a 100% progress. A 2 The learning rate schedule is only dependent on the step number regardless of whether we are training on the synthetic data or SPLASH. We tried resetting the learning rates back to their maximum values after switching to SPLASH, but did not observe any improvement in accuracy. model can have a negative progress (e.g., Rulebased re-ranking in Table 2) when it frequently predicts corrections with more errors than those in the initial parse. Unlike correction accuracy, Progress is more aligned with user experience in an interactive environment  as it assigns partial credit for fixing a subset of the errors and also, it penalizes models that predict an even more erroneous parse after receiving feedback.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Results: We compare (Table 2) NL-EDIT to the two top-performing baselines in (Elgohary et al., 2020) and also to the beam re-ranking upper-bound they report. NL-EDIT significantly increases the correction accuracy over the top baseline (Edit-SQL+Feedback) by more than 16% and it also outperforms oracle re-ranking by around 5%. We also note that in 72.4% of the test examples, NL-EDIT was able to strictly reduce the number of errors in the initial parse (Edit ↓) which potentially indicates a more positive user experience than the other models. NL-EDIT achieves 37% Progress which indicates faster convergence to the fully corrected parse than all the other models.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Following the same experimental setup in Section 6, we compare NL-EDIT to other variants with one ablated component at a time (Table 3). We ablate the feedback, the explanation, and the question from the encoder input. We also ablate the interaction relations (Section 4.2) that we incorporate in the relation-aware transformer module. We only ablate the new relations we introduce to model the interaction (shown in Figure 3), but we keep the Question-Schema and Schema-Schema relations introduced in (Wang et al., 2020). For each such variant, we train for 20,000 steps on the synthetic dataset then continue training on SPLASH until step 100,000. We also train an ablated variant that does not use the synthetic feedback where we   train for 100,000 steps only on SPLASH. For all variants, we choose the checkpoint with the largest correction accuracy on the dev set and report the accuracy on the SPLASH test set. The results in Table 3 confirm the effectiveness of each component in our model. We find that the model is able to correct 19.8% of the examples without the feedback. We noticed that the ablatedfeedback model almost reaches that accuracy only after training on the synthetic data with very minor improvement (< 1%) after training on SPLASH. Only using the question and the explanation, the model is able to learn about a set of systematic errors that parsers make and how they can be corrected (Gupta et al., 2017;Yin and Neubig, 2019).
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
In Figure 4, we breakdown the correction accuracy by the feedback and explanation lengths (in number of tokens) and by the reference edit size (number of required edit operations to fully correct the initial parse). The accuracy drops significantly when the reference edit size exceeds two (Figure 4c), while it declines more gradually as the feedback and explanation increase in length. We manually (Examples in Table 4) inspected the examples with longer feedback than 24, and found that 8% of them the feedback is long because it describes how to rewrite the whole query rather than being lim-Long Feedback Not Describing an Edit: "you should determine the major record format from the orchestra table and make sure it is arranged in ascending order of number of rows that appear for each major record format."
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Long Feedback Describing an Edit: "replace course id (both) with degree program id, first courses with student enrolment, course description with degree summary name, second courses with degree programs." ited to only the edits to be made. In the remaining 92%, the initial query had several errors (edit size of 5.5 on average) with the corresponding feedback enumerating all of them. Figure 4d shows how the number of errors (measured in edit size) changes after correction. The figure shows that even for examples with a large number of errors (four and five), the model is still able to reduce the number of errors in most cases. We manually inspected the examples with only one error that the model failed to correct. We found 15% of them have either wrong or non-editing feedback and in 29% the model produced the correct edit but with additional irrelevant ones. The dominant source of error in the remaining examples is because of failures with linking the feedback to the schema (Examples in Table 5).
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
So far, we have been using SPLASH for both training and testing. The erroneous parses (and corresponding feedback) in SPLASH are based on the Seq2Struct parser (Shin, 2019   in model architectures (Wang et al., 2020) and pretraining (Yin et al., 2020;Yu et al., 2021a) has led to parsers that already outperform Seq2Struct by more than 30% in parsing accuracy. 3 Here, we ask whether NL-EDIT that we train on SPLASH (and synthetic feedback) can generalize to parsing errors made by more recent parsers without additional parser-specific training data. We follow the same crowdsourcing process used to construct SPLASH (Section 2) to collect three new test sets based on three recent textto-SQL parsers: EditSQL , TaBERT (Yin et al., 2020) and RAT-SQL (Wang et al., 2020). Following Elgohary et al. (2020), we run each parser on SPIDER dev set and only collect feedback for the examples with incorrect parses that can be explained using their SQL explanation 3 https://yale-lily.github.io/spider framework. Table 6 (Top) summarizes the three new test sets and compares them to SPLASH test set. We note that the four datasets are based on the same set of questions and databases (SPIDER dev).
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Table 6 (Bottom) compares the parsing accuracy (measure by exact query match (Yu et al., 2018b)) of each parser when used by itself (No Interaction) to integrating it with NL-EDIT. We report both the accuracy on the examples provided to NL-EDIT (Error Correction) and the End-to-End accuracy on the full SPIDER dev set. NL-EDIT significantly boosts the accuracy of all parsers, but with a notable drop in the gains as the accuracy of the parser improves. To explain that, in Figure 5 we compare the distribution of reference edit size across the four datasets. The figure does not show any significant differences in the distributions that would lead to such a drop in accuracy gain. Likewise, the distributions of the feedback lengths are very similar (the mean is shown in Table 6). As parsers improve in accuracy, they tend to make most of their errors on complex SQL queries. Although the number of errors with each query does not significantly change ( Figure 5), we hypothesize that localizing the errors in a complex initial parse, with a long explanation (Table 6), is the main generalization bottleneck that future work needs to address.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Natural language to SQL: Natural language interfaces to databases have been an active field of study for many years (Woods et al., 1972;Warren and Pereira, 1982;Popescu et al., 2003;Li and Jagadish, 2014). The development of new large scale datasets, such as WikiSQL (Zhong et al., 2017) and SPIDER (Yu et al., 2018b), has reignited the interest in this area with several new models introduced recently (Choi et al., 2020;Wang et al., 2020;Scholak et al., 2020). Another related line of work has focused on conversation semantic parsing, e.g. SParC (Yu et al., 2019b), CoSQL (Yu et al., 2019a), and SMCalFlow (Andreas et al., 2020), where parsers aim at modeling utterance sequentially and in context of previous utterances.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Interactive Semantic Parsing: Several previous studies have looked at the problem of improving semantic parser with feedback or human interactions (Clarke et al., 2010;Artzi and Zettlemoyer, 2013). Interactions are supported in multiple ways including binary correct/incorrect signal (Iyer et al., 2017), answers to a yes/no or a multiple-choice  Table 6: Evaluating the zero-shot generalization of NL-EDIT to different parsers (EditSQL, TaBERT, and RAT-SQL) after training on SPLASH that is constructed based on the Seq2Struct parser. Top: Summary of the dataset constructed based on each parser. Feedback and explanation length is the number of tokens. Bottom: The Error Correction accuracy on each test set and the end-to-end accuracy of each parser on the full SPIDER dev set with and without interaction. ∆ w/ Interaction is the gain in end-to-end accuracy with the interaction added.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
question posed by the system (Yao et al., 2019;Gur et al., 2018) or suggestions of edits that can be applied to the parse . Yao et al. (2019) and Gur et al. (2018) ask yes/no and multiple-choice questions and use the answers in generating the pars. Elgohary et al. (2020) introduce SPLASH (Section 2), a dataset for correcting semantic parsing with natural language feedback. Using language as a medium for providing feedback enables the human to provide rich open-form feedback in their natural way of communication giving them control and flexibility specifying what is wrong and how it should be corrected. Our work uses SPLASH and proposes to pose the problem of semantic parse correction as a parser editing problem with natural language feedback input. This is also related to recent work on casting text generation (e.g. summarization, grammatical error correction, sentence splitting, etc.) as a text editing task (Malmi et al., 2019;Panthaplackel et al., 2020;Stahlberg and Kumar, 2020) where target texts are reconstructed from inputs using several edit operations.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Semantic Parsing with Synthetic Data: Semantic parsing systems have frequently used synthesized data to alleviate the challenge of labeled data scarcity. In their semantic parser overnight work, Wang et al. (2015) proposed a method for training semantic parsers quickly in a new domain using synthetic data. They generate logical forms and canonical utterances and then paraphrase the canonical utterances via crowd-sourcing. Several other approaches have demonstrated the benefit of adopting this approach to train semantic parsers in low-resource settings (Su et al., 2017;Zhong et al., 2017;Cheng et al., 2018;Xu et al., 2020). Most recently, synthetic data was used to continue to pre-train language models for semantic parsing tasks (Herzig et al., 2020;Yu et al., 2021a,b). We build on this line work by showing that we can generate synthetic data automatically without human involvement to simulate edits between an erroneous parse and a correct one.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
We introduced a model, a data augmentation method, and analysis tools for correcting semantic parse errors in text-to-SQL through natural language feedback. Compared to previous models, our model improves the correction accuracy by 16% and boosts the end-to-end parsing accuracy by up to 20% with only one turn of feedback. Our work creates several avenues for future work: (1) improving the model by better modeling the interaction between the inputs and exploring different patterns for decoder-encoder attention, (2) evaluating existing methods for training with synthetic data (e.g., curriculum learning (Bengio et al., 2009)), (3) optimizing the correction model for better user experience using the progress measure we introduce, and (4) using the SQL edits scheme in other related tasks such as conversational text-to-SQL parsing.
Self-Regulated Interactive Sequence-to-Sequence Learning
The concept of self-regulation has been studied in educational research (Hattie and Timperley, 2007;Hattie and Donoghue, 2016), psychology (Zimmerman and Schunk, 1989;Panadero, 2017), and psychiatry (Nigg, 2017), and was identified as central to successful learning. "Self-regulated students" can be characterized as "becoming like teachers", in that they have a repertoire of strategies to self-assess and self-manage their learning process, and they know when to seek help and which kind of help to seek. While there is a vast literature on machine learning approaches to metalearning (Schmidhuber et al., 1996), learning-tolearn (Thrun andPratt, 1998), or never-ending learning (Mitchell et al., 2015), the aspect of learning when to ask for which kind of feedback has so far been neglected in this field.
Self-Regulated Interactive Sequence-to-Sequence Learning
We propose a machine learning algorithm that uses self-regulation in order to balance the cost and effect of learning from different types of feedback. This is particularly relevant for human-in-
Self-Regulated Interactive Sequence-to-Sequence Learning
Figure 1: Human-in-the-loop self-regulated learning.
Self-Regulated Interactive Sequence-to-Sequence Learning
the-loop machine learning, where human supervision is costly. The self-regulation module automatically learns which kind of feedback to apply when in training-full supervision by teacher demonstration or correction, weak supervision in the form of positive or negative rewards for student predictions, or a self-supervision signal generated by the student. Figure 1 illustrates this learning scenario. The learner, in our case a sequence-tosequence (Seq2Seq) learner, aims to solve a certain task with the help of a human teacher. For every input it receives for training, it can ask the teacher for feedback to its own output, or supervise itself by training on its own output, or skip learning on the input example altogether. The self-regulator's policy for choosing feedback types is guided by their cost and by the performance gain achieved by learning from a particular type of feedback.
Self-Regulated Interactive Sequence-to-Sequence Learning
We apply the self-regulation algorithm to interactive machine translation where a neural machine translation (NMT) system functions as a student which receives feedback simulated from a human reference translation or supervises itself. The intended real-world application is a machine translation personalization scenario where the goal of the human translator is to teach the NMT system to adapt to in-domain data with the best trade-off between feedback cost and performance gain. It can be transferred to other sequence-to-sequence learning tasks such as personalization of conversational AI systems for question-answering or geographical navigation.
Self-Regulated Interactive Sequence-to-Sequence Learning
Our analysis of different configurations of selfregulation yields the following insights: Perhaps unsurprisingly, the self-regulator learns to balance all types of feedback instead of relying only on the strongest or cheapest option. This is an advantage over active learning strategies that only consider the choice between no supervision and full supervision. Interestingly, though, we find that the selfregulator learns to trade off exploration and exploitation similar to a context-free -greedy strategy that optimizes for fastest learning progress. Lastly, we show that the learned regulator is robust in a cold-start transfer to new domains, and even shows improvements over fully supervised learning on domains such as literary books where reference translations provide less effective learning signals.
Self-Regulated Interactive Sequence-to-Sequence Learning
In this work, we focus on the aspect of selfregulated learning that concerns the ability to decide which type of feedback to query from a teacher (or oneself) for most efficient learning depending on the context. In our human-in-the-loop machine learning formulation, we focus on two contextual aspects that can be measured precisely: quality and cost. The self-regulation task is to optimally balance human effort and output quality.
Self-Regulated Interactive Sequence-to-Sequence Learning
We model self-regulation as an active reinforcement learning problem with dynamic costs, where in each state, i.e. upon receiving an input, the regulator has to choose an action, here a feedback type, and pay a cost. The learner receives feedback of that type from the human to improve its prediction. Based on the effect of this learning update, the regulator's actions are reinforced or penalized, so that it improves its choice for future inputs.
Self-Regulated Interactive Sequence-to-Sequence Learning
In the following, we first compare training objectives for a Seq2Seq learner from various types of feedback ( §3.1), then introduce the selfregulator module ( §3.2), and finally combine both in the self-regulation algorithm ( §3.3).
Self-Regulated Interactive Sequence-to-Sequence Learning
Let x = x 1 . . . x S be a sequence of indices over a source vocabulary V SRC , and y = y 1 . . . y T a sequence of indices over a target vocabulary V TRG . The goal of sequence-to-sequence learning is to learn a function for mapping an input sequence x into an output sequences y. Specifically, for the example of machine translation, where y is a translation of x, the model, parametrized by a set of weights θ, learns to maximize p θ (y | x). This quantity is further factorized into conditional probabilities over single tokens:
Self-Regulated Interactive Sequence-to-Sequence Learning
The distribution p θ (y t | x; y <t ) is defined by the neural model's softmax-normalized output vector: p θ (y t | x; y <t ) = softmax(NN θ (x; y <t )).
Self-Regulated Interactive Sequence-to-Sequence Learning
There are various options for building the architecture of the neural model NN θ , such as recurrent (Sutskever et al., 2014), convolutional (Gehring et al., 2017) or attentional (Vaswani et al., 2017) encoder-decoder architectures (or a mix thereof (Chen et al., 2018)). Regardless of their architecture, there are multiple ways of interactive learning that can be applied to neural Seq2Seq learners.
Self-Regulated Interactive Sequence-to-Sequence Learning
Learning from Corrections (FULL). Under full supervision, i.e., when the learner receives a fully corrected output y * for an input x, crossentropy minimization (equivalent to maximizing the likelihood of the data D under the current model) considers the following objective:
Self-Regulated Interactive Sequence-to-Sequence Learning
The stochastic gradient of this objective is
Self-Regulated Interactive Sequence-to-Sequence Learning
constituting an unbiased estimate of the gradient
Self-Regulated Interactive Sequence-to-Sequence Learning
A local minimum can be found by performing stochastic gradient descent on g FULL θ (x, y * ). This training objective is the standard in supervised learning when training with human-generated references or for online adaptation to post-edits (Turchi et al., 2017).
Self-Regulated Interactive Sequence-to-Sequence Learning
Learning from Error Markings (WEAK). Petrushkov et al. (2018) presented chunk-based binary feedback as a low-cost alternative to full corrections. In this scenario the human teacher marks the correct parts of the machine-generated output y. As a consequence every token in the output receives a reward δ t , either δ t = 1 if marked as correct, or δ t = 0 otherwise. The objective of the learner is to maximize the likelihood of the correct parts of the output, or equivalently, to minimize
Self-Regulated Interactive Sequence-to-Sequence Learning
where the stochastic gradient is
Self-Regulated Interactive Sequence-to-Sequence Learning
The tokensŷ t that receive δ t = 1 are part of the correct output y * , so the model receives a hint how a corrected output should look like. Although the likelihood of the incorrect parts of the sequence does not weigh into the sum, they are contained in the context of the correct parts (in y <t ).
Self-Regulated Interactive Sequence-to-Sequence Learning
Self-Supervision (SELF). Instead of querying the teacher for feedback, the learner can also choose to learn from its own output, that is, to learn from self-supervision. The simplest option is to treat the learner's output as if it was correct, but that quickly leads to overconfidence and degeneration. Clark et al. (2018) proposed a cross-view training method: the learner's original prediction is used as a target for a weaker model that shares parameters with the original model. We adopt this strategy by first producing a target sequenceŷ with beam search and then weaken the decoder through attention dropout with probability p att . The objective is to minimize the negative likelihood of the original target under the weakened model
Self-Regulated Interactive Sequence-to-Sequence Learning
Combination. For self-regulated learning, we also consider a fourth option (NONE): the option to ignore the current input. Figure 2 summarizes the stochastic gradients for all cases. In practice, Seq2Seq learning shows greater stability for mini-batch updates than online updates on single training samples. Mini-batch self-regulated learning can be achieved by accumulating stochastic gradients for a mini-batch of size B before updating θ with an average of these stochastic gradients, which we denote as g
Self-Regulated Interactive Sequence-to-Sequence Learning
The regulator is another neural model q φ that is optimized for the quality-cost trade-off of the Seq2Seq learner. Given an input x i and the Seq2Seq's hypothesisŷ i , it chooses an action, here a supervision mode s i ∼ q φ (s | x i ,ŷ i ). This choice of feedback determines the update of the Seq2Seq learner ( Figure 2). The regulator is rewarded by the ratio between the cost c i of obtaining the feedback s i and the quality improvement ∆(θ i , θ i−1 ) caused by updating the Seq2Seq learner with the feedback:
Self-Regulated Interactive Sequence-to-Sequence Learning
(1)
Self-Regulated Interactive Sequence-to-Sequence Learning
is measured as the difference in validation score achieved before and after the learner's update (Fang et al., 2017), and c i as the cost of user edits. Adding a small constant cost α to the actual feedback cost ensures numerical stability. This meta-parameter can be interpreted as representing a basic cost for model updates of any kind. The objective for the regulator is to maximize the expected reward defined in Eq. 1:
Self-Regulated Interactive Sequence-to-Sequence Learning
The full gradient of this objective is estimated by the stochastic gradient for sampled actions (Williams, 1992):
Self-Regulated Interactive Sequence-to-Sequence Learning
Note that the reward contains the immediate improvement after one update of the Seq2Seq learner and not the overall performance in hindsight. This is an important distinction to classic expected reward objectives in reinforcement learning since it biases the regulator towards actions that have an immediate effect, which is desirable in the case of interaction with a human. However, since Seq2Seq learning requires updates and evaluations based on mini-batches, the regulator update also needs to be based on mini-batches of predictions, leading to the following specification of Eq. (2) for a mini-batch j:
Self-Regulated Interactive Sequence-to-Sequence Learning
While mini-batch updates are required for stable Seq2Seq learning, they hinder the regulator from assigning credit for model improvement to individual elements within the mini-batch.
Self-Regulated Interactive Sequence-to-Sequence Learning
Algorithm 1 presents the proposed online learning algorithm with model updates cumulated over mini-batches. On arrival of a new input, the regulator predicts a feedback type in line 6. According to this prediction, the environment/user is asked for feedback for the Seq2Seq's prediction at cost c i (line 7). The Seq2Seq model is updated on the Algorithm 1 Self-Regulated Interactive Seq2Seq Input: Initial Seq2Seq θ 0 , regulator φ 0 , B 1: j ← 0 2: while inputs and human available do 3:
Self-Regulated Interactive Sequence-to-Sequence Learning
Observe input x i , Seq2Seq outputŷ i 6:
Self-Regulated Interactive Sequence-to-Sequence Learning
Choose feedback:
Self-Regulated Interactive Sequence-to-Sequence Learning
Obtain feedback f i of type s i at cost c i 8:
Self-Regulated Interactive Sequence-to-Sequence Learning
Update θ with g
Self-Regulated Interactive Sequence-to-Sequence Learning
basis of the feedback and mini-batch of stochastic gradients computed as summarized in Figure 2.
Self-Regulated Interactive Sequence-to-Sequence Learning
In order to reinforce the regulator, the Seq2Seq model's improvement (line 9) is assessed, and the parameters of the regulator are updated (line 10, Eq. 3). Training ends when the data stream or the provision of feedback ends. The intermediate Seq2Seq evaluations can be re-used for model selection (early stopping). In practice, these evaluations can either be performed by validation on a held-out set (as in the simulation experiments below) or by human assessment.
Self-Regulated Interactive Sequence-to-Sequence Learning
Practical Considerations. The algorithm does not introduce any additional hyperparameters beyond standard learning rates, architecture design and mini-batch sizes that have to be tuned. As proposed in Petrushkov et al. (2018) or Clark et al. (2018), targetsŷ are pre-generated offline with the initial θ 0 , which we found crucial for the stability of the learning process. The evaluation step after the Seq2Seq update is an overhead that comes with meta-learning, incurring costs depending on the decoding algorithm and the evaluation strategy. However, Seq2Seq updates can be performed in mini-batches, and the improvement is assessed after a mini-batch of updates, as discussed above.
Self-Regulated Interactive Sequence-to-Sequence Learning
The main research questions to be answered in our experiments are:
Self-Regulated Interactive Sequence-to-Sequence Learning
1. Which strategies does the regulator develop?
Self-Regulated Interactive Sequence-to-Sequence Learning
2. How well does a trained regulator transfer across domains?
Self-Regulated Interactive Sequence-to-Sequence Learning
3. How do these strategies compare against (active) learning from a single feedback type?
Self-Regulated Interactive Sequence-to-Sequence Learning
We perform experiments for interactive NMT, where a general-domain NMT model is adapted to a specific domain by learning from the feedback of a human translator. This is a realistic interactive learning scenario where cost-free pre-training on a general domain data is possible, but each feedback generated by the human translator in the personalization step incurs a specific cost. In our experiment, we use human-generated reference translations to simulate both the cost of human feedback and to measure the performance gain achieved by model updates.
Self-Regulated Interactive Sequence-to-Sequence Learning
Seq2Seq Architecture. Both the Seq2Seq learner and the regulator are based on LSTMs (Hochreiter and Schmidhuber, 1997). The Seq2Seq has four bi-directional encoder and four decoder layers with 1024 units each, embedding layers of size 512. It uses Luong et al. (2015)'s input feeding and output layer, and global attention with a single feed forward layer (Bahdanau et al., 2015).
Self-Regulated Interactive Sequence-to-Sequence Learning
Regulator Architecture. The regulator consists of LSTMs on two levels: Inspired by Siamese Networks (Bromley et al., 1994), a bi-directional LSTM encoder of size 512 separately reads in both the current input sequence and the beam search hypothesis generated by the Seq2Seq. The last state of encoded source and hypothesis sequence and the previous output distribution are concatenated to form the input to a higher-level regulator LSTM of size 256. This LSTM updates its internal state and predicts a score for every feedback type for every input in the mini-batch. The feedback for each input is chosen by sampling from the distribution obtained by softmax normalization of these scores. The embeddings of the regulator are initialized by the Seq2Seq's source embeddings and further tuned during training. The model is implemented in the JoeyNMT 1 framework based on PyTorch. 2 Data. We use three parallel corpora for Germanto-English translation: a general-domain data set from the WMT2017 translation shared task for Seq2Seq pre-training, TED talks from the IWSLT2017 evaluation campaign for training the regulator with simulated feedback, and the Books corpus from the OPUS collection (Tiedemann, 2012) for testing the regulator on another domain. Data pre-processing details and splits are given in §A.1. The joint vocabulary for Seq2Seq and the regulator consists of 32k BPE sub-words (Sennrich et al., 2016) trained on WMT.
Self-Regulated Interactive Sequence-to-Sequence Learning
Training. The Seq2Seq model is first trained on WMT with Adam (Kingma and Ba, 2015) on mini-batches of size 64, an initial learning rate 1 × 10 −4 that is halved when the loss does not decrease for three validation rounds. Training ends when the validation score does not increase any further (scoring 29.08 BLEU on the WMT test). This model is then adapted to IWSLT with selfregulated training for one epoch, with online human feedback simulated from reference translations. The mini-batch size is reduced to 32 for self-regulated training to reduce the credit assignment problem for the regulator. The constant cost α (Eq. 1) is set to 1. 3 When multiple runs are reported, the same set of random seeds is used for all models to control the order of the input data. The best run is evaluated on the Books domain for testing the generalization of the regulation strategies.
Self-Regulated Interactive Sequence-to-Sequence Learning
Simulation of Cost and Performance. In our experiments, human feedback and its cost, and the performance gain achieved by model updates, is simulated by using human reference translations. Inspired by the keystroke mouse-action ratio (KSMR) (Barrachina et al., 2009), a common metric for measuring human effort in interactive machine translation, we define feedback cost as the sum of costs incurred by character edits and clicks, similar to Peris and Casacuberta (2018). The cost of a full correction (FULL) is the number of character edits between model output and reference, simulating the cost of a human typing. 4 Error markings (WEAK) are simulated by comparing the hypothesis to the reference and marking the longest common sub-strings as correct, as proposed by Petrushkov et al. (2018). As an extension to Petrushkov et al. (2018) we mark multiple common sub-strings as correct if all of them have the longest length. The cost is defined as the number of marked words, assuming an interface that allows markings by clicking on words. For selftraining (SELF) and skipping training instances we naively assume zero cost, thus limiting the mea-3 Values = 1 distort the rewards for self-training too much. 4 As computed by the Python library difflib. surement of cost to the effort of the human teacher, and neglecting the effort on the learner's side. Table 1 illustrates the costs per feedback type on a randomly selected set of examples.
Self-Regulated Interactive Sequence-to-Sequence Learning
We measure the model improvement by evaluating the held-out set translation quality of the learned model at various time steps with corpus BLEU (cased SacreBLEU (Post, 2018)) and measure the accumulated costs. The best model is considered the one that delivers the highest quality at the lowest cost. This trade-off is important to bear in mind since it differs from the standard evaluation of machine translation models, where the overall best-scoring model, regardless of the supervision cost, is considered best. Finally, we evaluate the strategy learned by the regulator on an unseen domain, where the regulator decides which type of feedback the learner gets, but is not updated itself.
Self-Regulated Interactive Sequence-to-Sequence Learning
We compare learning from one type of feedback in isolation against regulators with the following set of actions:  Figure 3 compares the improvement in corpus BLEU (Papineni et al., 2002) (corresponding to results in Translation Error Rate (TER, computed by pyTER) (Snover et al., 2006)) of regulation variants and full feedback over cumulative costs of up to 80k character edits. Using only full feedback (blue) as in standard supervised learning or learning from post-edits, the overall highest improvement can be reached (visible only after the cutoff of 80k edits; see Appendix A.2 for the comparison over a wider window of time). However, it comes at a very high cost (417k characters in total to reach +0.6 BLEU). The regulated variants offer a much cheaper improvement, at least until a cumulative cost between 80k (Reg4) and 120k (Reg2), depending on the feedback options available. The regulators do not reach the quality of the full model since their choice of feedback is oriented towards costs and immediate improvements. By finding a trade-off between feedback types for immediate improvements, the regulators sacrifice long-term improvement. Comparing regulators, Reg2 (orange) reaches the overall SELF 0
Self-Regulated Interactive Sequence-to-Sequence Learning
x Sie greift in ihre Geldbörse und gibt ihm einen Zwanziger . y It attacks their wallets and gives him a twist . y * She reaches into her purse and hands him a 20 .
Self-Regulated Interactive Sequence-to-Sequence Learning
WEAK 9
Self-Regulated Interactive Sequence-to-Sequence Learning
x Und als ihr Vater sie sah und sah , wer sie geworden ist , in ihrem vollen Mädchen-Sein , schlang er seine Arme um sie und brach in Tränen aus . y And when her father saw them and saw who became them , in their full girl 's , he swallowed his arms around them and broke out in tears . y * When her father saw her and saw who she had become , in her full girl self , he threw his arms around her and broke down crying .
Self-Regulated Interactive Sequence-to-Sequence Learning
FULL 59
Self-Regulated Interactive Sequence-to-Sequence Learning
x Und durch diese zwei Eigenschaften war es mir möglich , die Bilder zu erschaffen , die Sie jetzt sehen . y And through these two features , I was able to create the images you now see . y * And it was with those two properties that I was able to create the images that you 're seeing right now .  highest improvement over the baseline model, but until the cumulative cost of around 35k character edits, Reg3 (green) offers faster improvement at a lower cost since it has an additional, cheaper feedback option. Adding the option to skip examples (Reg4, red) does not give a benefit. Appendix A.3 lists detailed results for offline evaluation on the trained Seq2Seq models on the IWSLT test set: Self-regulating models achieve improvements of 0.4-0.5 BLEU with costs reduced up to a factor of 23 in comparison to the full feedback model. The reduction in cost is enabled by the use of cheaper feedback, here markings and selfsupervision, which in isolation are very successful as well. Self-supervision works surprisingly well and can be recommended for cheap but effective unsupervised domain adaptation for sequence-tosequence learning.
Self-Regulated Interactive Sequence-to-Sequence Learning
Self-Regulation Strategies. Figure 4 shows which actions Reg3   ing training, with the exception of an initial exploration phase within the first 100 iterations. In general, we observe that all regulators are highly sensitive to balancing cost and performance, and mostly prefer the cheapest option (e.g., Reg4 by choosing mostly NONE) since they are penalized heavily for choosing (or exploring) expensive options (see Eq. 1).
Self-Regulated Interactive Sequence-to-Sequence Learning
A further research question is whether and how the self-regulation module takes the input or output context into account. We therefore compare its decisions to a context-free -greedy strategy. The -greedy algorithm is a successful algorithm for multi-armed bandits (Watkins, 1989). In our case, the arms are the four feedback types. They are chosen based on their reward statistics, here the average empirical reward per feedback type Q i (s) = 1 N i (s) 0,...,i r(s i ). With probability 1 − , the algorithm selects the feedback type with the highest empirical reward (exploitation), otherwise picks one of the remaining arms at random (exploration). In contrast to the neural regulator model, -greedy decides solely on the basis  of the reward statistics and has no internal contextual state representation. The comparison of Reg3 with -greedy for a range of values for in Figure 5 shows that learned regulator behaves indeed very similar to an -greedy strategy with = 0.25. -greedy variants with higher amounts of exploration show a slower increase in BLEU, while those with more exploitation show an initial steep increase that flattens out, leading to overall lower BLEU scores. The regulator has hence found the best trade-off, which is an advantage over the -greedy algorithm where the hyperparameter requires dedicated tuning. Considering the -greedy-like strategy of the regulator and the strong role of the cost factor shown in Figure 4, the regulator module does not appear to choose individual actions based e.g., on the difficulty of inputs, but rather composes mini-batches with a feedback ratio according to the feedback type's statistics. This confirms the observations of Peris and Casacuberta (2018), who find that the subset of instances selected for labeling is secondaryit is rather the mixing ratio of feedback types that matters. This finding is also consistent with the mini-batch update regime that forces the regulator to take a higher-level perspective and optimize the expected improvement at the granularity of (minibatch) updates rather than at the input level.
Self-Regulated Interactive Sequence-to-Sequence Learning
Domain Transfer. After training on IWSLT, we evaluate the regulators on the Books domain: Can they choose the best actions for an efficient learning progress without receiving feedback on the new domain? We evaluate the best run of each regulator type (i.e., φ trained on IWSLT), with the Seq2Seq model reset to the WMT baseline. The regulator is not further adapted to the Books domain, but decides on the feedback types for training the Seq2Seq model for a single epoch on the Books data. Figure 6 visualizes the regulated training process of the Seq2Seq model. As before, Reg3 performs best, outperforming weak, full and self-supervision (reaching 14.75 BLEU, not depicted since zero cost). Learning from full feedback improves much later in training and reaches 14.53 BLEU. 5 One explanation is that the reference translations in the Books corpus are less literal than the ones for IWSLT, such that a weak feedback signal allows the learner to learn more efficiently than from full corrections. Appendix A.4 reports the results for offline evaluation on the trained Seq2Seq models on the Books test set.
Self-Regulated Interactive Sequence-to-Sequence Learning
Comparison to Active Learning. A classic active learning strategy is to sample a subset of the input data for full labeling based on the uncertainty of the model predictions . The size of this subset, i.e. the amount of human labeling effort, has to be known and determined before learning. Figure 7 compares the self-regulators on the Books domain with models that learn from a fixed ratio of fullylabeled instances in every batch. These are chosen according to the model's uncertainty, here measured by the average token entropy of the model's best-scoring beam search hypothesis. The regulated models with a mix of feedback types clearly outperform the active learning strategies,  both in terms of cost-efficient learning (Figure 7) as well as in overall quality (See Figure 9 in Appendix A.5). We conclude that mixing feedback types, especially in the case where full feedback is less reliable, offers large improvements over standard stream-based active learning strategies.
Self-Regulated Interactive Sequence-to-Sequence Learning
Our experiments were designed as a pilot study to test the possibilities of self-regulated learning in simulation. In order to advance to field studies where human users interact with Seq2Seq models, several design choices have to be adapted with caution. Firstly, we simulate both feedback cost and quality improvement by measuring distances to static reference outputs. The experimental design in a field study has to account for a variation of feedback strength, feedback cost, and performance assessments, across time, across sentences, and across human users . One desideratum for field studies is thus to analyze this variation by analyzing the experimental results in a mixed effects model that accounts for variability across sentences, users, and annotation sessions (Baayen et al., 2008;Karimova et al., 2018). Secondly, our simulation of costs considers only the effort of the human teacher, not the machine learner. The strong preference for the cheapest feedback option might be a result of overestimating the cost of human post-editing and underestimating the cost of self-training. Thus, a model for field studies where data is limited might greatly benefit from learned estimates of feedback cost and quality improvement .
Self-Regulated Interactive Sequence-to-Sequence Learning
We proposed a cost-aware algorithm for interactive sequence-to-sequence learning, with a selfregulation module at its core that learns which type of feedback to query from a human teacher. The empirical study on interactive NMT with simulated human feedback showed that this selfregulated model finds more cost-efficient solutions than models learning from a single feedback type and uncertainty-based active learning models, also under domain shift. While this setup abstracts away from certain confounding variables to be expected in real-life interactive machine learning, it should be seen as a pilot experiment that allows focussing on our central research questions under an exact and noise-free computation of feedback cost and performance gain. The proposed framework can naturally be expanded to integrate more feedback modes suitable for the interaction with humans, e.g., pairwise comparisons or output rankings. Future research directions will involve the development of reinforcement learning model with multi-dimensional rewards, and modeling explicit credit assignment for improving the capabilities of the regulator to make context-sensitive decisions in mini-batch learning.
Self-Regulated Interactive Sequence-to-Sequence Learning
A.  The WMT data is obtained from the WMT 2017 shared task website 6 and pre-processed as described in Hieber et al. (2017). The preprocessing pipeline is used for IWSLT and Books data as well. IWSLT2017 is obtained from the evaluation campaign website. 7 For validation on WMT, we use the newstest2015 data, for IWSLT tst2014+tst2015, for testing on WMT newstest2017 and tst2017 for IWSLT. Since there is no standard split for the Books corpus, we randomly select 2k sentences for validation and testing each. Table 2 gives an overview of the size of the three resources.  Table 3 reports the offline held-out set evaluations for the early stopping points selected on the dev set for all feedback modes. All models notably improve over the baseline, only using full feedback leads to the overall best model on IWSLT (+0.6 BLEU / -0.6 TER), but costs a massive amounts of edits (417k characters). Selfregulating models still achieve improvements of 0.4-0.5 BLEU/TER with costs reduced up to a factor of 23. The reduction in cost is enabled by the use of cheaper feedback, here markings and self-supervision, which in isolation are successful as well. Self-supervision works surprisingly well, which makes it attractive for cheap but effective unsupervised domain adaptation. It has to be noted that both weak and self-supervision worked 6 http://www.statmt.org/wmt17/ translation-task.html 7 https://sites.google.com/site/ iwsltevaluation2017/    Table 4 reports results for test set evaluation on the Books domain of the best model from the IWSLT domain each. The baseline was trained on WMT parallel data without any regulation. The regulator was trained on IWSLT and evaluated on Books, the Seq2Seq model is further trained for one epoch on Books. The costs are measured in character edits and clicks. The best result in terms of BLEU and TER is achieved by the Reg2 model, even outperforming the model with full feedback. As observed for the IWSLT domain (cf. Section 4.2), self-training is very effective, but is outperformed by the Reg2 model and roughly on par  with the Reg3 model.  Figure 9: Development of validation BLEU over time for learned regulation strategies in comparison to active learning with a fixed percentage γ of full feedback. Counting of iterations starts at the previous iteration count of the baseline model. Figure 9 shows the development of BLEU over time for the regulators and active learning strategies with a fixed ratio of full feedback per batch (γ ∈ [10, 30, 50, 70, 90]). The decision whether to label an instance in a batch is made based on the average token entropy of the model's current hypothesis. Using only 50% of the fully-supervised labels achieves the same quality as 100% using this uncertainty-based active learning sampling strategy. However, the regulated models reach a higher quality not only at a lower cost (see Fig-ure 7), but also reach an overall higher quality.   Figures 10 and 11 show the ratio of feedback types for self-regulation during training with Reg2 and Reg4 respectively.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
In semantic parsing, natural language utterances are mapped to machine readable parses which are complex and often tailored specifically to the underlying task. The cost and difficulty of manually preparing large amounts of such parses thus is a bottleneck for supervised learning in semantic parsing. Recent work (Liang et al. (2017); Mou et al. (2017); Peng et al. (2017); inter alia) has applied reinforcement learning to address the annotation bottleneck as follows: Given a question, the existence of a corresponding gold answer is assumed. A semantic parser produces multiple parses per question and corresponding answers are obtained. These answers are then compared against the gold answer and a positive reward is recorded if there is an overlap. The parser is then guided towards correct parses using the REIN-FORCE algorithm (Williams, 1992) which scales the gradient for the various parses by their obtained reward (see the left half of Figure 1). However, learning from question-answer pairs is only efficient if gold answers are cheap to obtain. For complex open-domain question-answering tasks, correct answers are not unique factoids, but openended lists, counts in large ranges, or fuzzily defined objects. For example, geographical queries against databases such as OpenStreetMap (OSM) can involve fuzzy operators such as "near" or "in walking distance" and thus need to allow for fuzziness in the answers as well. A possible solution lies in machine learning from even weaker supervision signals in form of human bandit feedback 1 where the semantic parsing system suggests exactly one parse for which feedback is collected from a human user. In this setup neither gold parse nor gold answer are known and feedback is obtained for only one system output per question.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
The goal of our paper is to exploit this scenario of learning from human bandit feedback to train semantic parsers. This learning scenario perfectly fits commercial setups such as virtual personal assistants that embed a semantic parser. Commercial systems can easily log large amounts of interaction data between users and system. Once sufficient data has been collected, the log can then be used to improve the parser. This leads to a counterfactual learning scenario (Bottou et al., 2013) where we have to solve the counterfactual problem of how to improve a target system from logged feedback that was given to the outputs of a different historic system (see the right half of Figure 1).
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
In order to achieve our goal of counterfactual learning of semantic parsers from human bandit feedback, the following contributions are required: The parser attempts to find correct machine readable parses (MRPs) by producing multiple parses, obtaining corresponding answers, and comparing them against the gold answer. Right: In our setup, a question does not have an associated gold answer. The parser outputs a single MRP and the corresponding answer is shown to a user who provides some feedback. Such triplets are collected in a log which can be used for offline training of a semantic parser. This scenario is called counterfactual since the feedback was logged for outputs from a system different from the target system to be optimized.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
First, we need to construct an easy-to-use user interface that allows to collect feedback based on the parse rather than the answer. To this aim, we automatically convert the parse to a set of statements that can be judged as correct or incorrect by a human. This approach allows us to assign rewards at the token level, which in turn enables us to perform blame assignment in bandit learning and to learn from partially correct queries where tokens are reinforced individually. We show that users can provide such feedback for one question-parse pair in 16.4 seconds on average. This exemplifies that our approach is more efficient and cheaper than recruiting experts to annotate parses or asking workers to compile large answer sets.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Next, we demonstrate experimentally that counterfactual learning can be applied to neural sequence-to-sequence learning for semantic parsing. A baseline neural semantic parser is trained in fully supervised fashion, human bandit feedback from human users is collected in a log and subsequently used to improve the parser. The resulting parser significantly outperforms the baseline model as well as a simple bandit-to-supervised approach (B2S) where the subset of completely correct parses is treated as a supervised dataset. Finally, we repeat our experiments on a larger but simulated log to show that our gains can scale: the baseline system is improved by 7.45% in answer F1 score without ever seeing a gold standard parse.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Lastly, from a machine learning perspective, we have to solve problems of degenerate behavior in counterfactual learning by lifting the multiplicative control variate technique (Swaminathan and Joachims, 2015b;Lawrence et al., 2017b,a) to stochastic learning for neural models. This is done by reweighting target model probabilities over the logged data under a one-step-late model that decouples the normalization from gradient estimation and is thus applicable in stochastic (minibatch) gradient optimization.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Our semantic parsing model is a state-of-theart sequence-to-sequence neural network using an encoder-decoder setup Sutskever et al., 2014) together with an attention mechanism (Bahdanau et al., 2015). We use the settings of Sennrich et al. (2017), where an input sequence x = x 1 , x 2 , . . . x |x| (a natural language question) is encoded by a Recurrent Neural Network (RNN), each input token has an associated hidden vector
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
where the former is created by a forward pass over the input, and the latter by a backward pass.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
− → h i is obtained by recur- (Chung et al., 2014), and ← − h i is computed analogously. The input sequence is reduced to a single vector c = g({h 1 , . . . , h |x| }) which serves as the initialization of the decoder RNN. g calculates the average over all vectors h i . At each time step t the decoder state is set by s t = q(s t−1 , y t−1 , c t ). q is a conditional GRU with an attention mechanism and c t is the context vector computed by the attention mechanism. Given an output vocabulary V y and the decoder state s t = {s 1 , . . . , s |Vy| }, a softmax output layer defines a probability distribution over V y and the probability for a token y j is:
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
(1)
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
The model π w can be seen as parameterized policy over an action space defined by the target language vocabulary. The probability for a full output sequence y = y 1 , y 2 , . . . y |y| is defined by
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
In our case, output sequences are linearized machine readable parses, called queries in the following. Given supervised data
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
of question-query pairs, whereȳ t is the true target query for x t , the neural network can be trained using SGD and a cross-entropy (CE) objective:
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
where y t,<j = y t,1 , y t,2 . . . y t,j−1 .
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Counterfactual Learning Objectives. We assume a policy π w that, given an input x ∈ X , defines a conditional probability distribution over possible outputs y ∈ Y(x). Furthermore, we assume that the policy is parameterized by w and its gradient can be derived. In this work, π w is defined by the sequence-to-sequence model described in Section 3. We also assume that the model decomposes over individual output tokens,
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
. i.e. that the model produces the output token by token.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
The counterfactual learning problem can be described as follows: We are given a data log of
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
where outputs y t for inputs x t were generated by a logging system under policy π 0 , and loss values δ t ∈ [−1, 0] 2 were observed for the generated data points. Our goal is to optimize the expected reward (in our case: minimize the expected risk) for a target policy π w given the data log D log . In case of deterministic logging, outputs are logged with propensity π 0 (y t |x t ) = 1, t = 1, . . . , n. This results in a deterministic propensity matching (DPM) objective (Lawrence et al., 2017b), without the possibility to correct the sampling bias of the logging policy by inverse propensity scoring (Rosenbaum and Rubin, 1983):
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
This objective can show degenerate behavior in that it overfits to the choices of the logging policy (Swaminathan and Joachims, 2015b;Lawrence et al., 2017a). This degenerate behavior can be avoided by reweighting using a multiplicative control variate (Kong, 1992;Precup et al., 2000;Jiang and Li, 2016;Thomas and Brunskill, 2016). The new objective is called the reweighted deterministic propensity matching (DPM+R) objective in Lawrence et al. (2017b):
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
. 2 We use the terms loss and (negative) rewards interchangeably, depending on context. Algorithms for optimizing the discussed objectives can be derived as gradient descent algorithms where gradients using the score function gradient estimator (Fu, 2006) are shown in Table 1.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Reweighting in Stochastic Learning. As shown in Swaminathan and Joachims (2015b) and Lawrence et al. (2017a), reweighting over the entire data log D log is crucial since it avoids that high loss outputs in the log take away probability mass from low loss outputs. This multiplicative control variate has the additional effect of reducing the variance of the estimator, at the cost of introducing a bias of order O( 1 n ) that decreases as n increases (Kong, 1992). The desirable properties of this control variate cannot be realized in a stochastic (minibatch) learning setup since minibatch sizes large enough to retain the desirable reweighting properties are infeasible for large neural networks.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
We offer a simple solution to this problem that nonetheless retains all desired properties of the reweighting. The idea is inspired by one-step-late algorithms that have been introduced for EM algorithms (Green, 1990). In the EM case, dependencies in objectives are decoupled by evaluating certain terms under parameter settings from previous iterations (thus: one-step-late) in order to achieve closed-form solutions. In our case, we decouple the reweighting from the parameterization of the objective by evaluating the reweighting under parameters w from some previous iteration. This allows us to perform gradient descent updates and reweighting asynchronously. Updates are performed using minibatches, however, reweighting is based on the entire log, allowing us to retain the desirable properties of the control variate.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
The new objective, called one-step-late reweighted DPM objective (DPM+OSL), optimizes π w,w with respect to w for a minibatch of size m, with reweighting over the entire log of size n under parameters w :
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
If the renormalization is updated periodically, e.g. after every validation step, renormalizations under w or w are not much different and will not hamper convergence. Despite losing the formal justification from the perspective of control variates, we found empirically that the OSL update schedule for reweighting is sufficient and does not deteriorate performance. The gradient for learning with OSL updates is given in Table 1.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Token-Level Rewards. For our application of counterfactual learning to human bandit feedback, we found another deviation from standard counterfactual learning to be helpful: For humans, it is hard to assign a graded reward to a query at a sequence level because either the query is correct or it is not. In particular, with a sequence level reward of 0 for incorrect queries, we do not know which part of the query is wrong and which parts might be correct. Assigning rewards at token-level will ease the feedback task and allow the semantic parser to learn from partially correct queries. Thus, assuming the underlying policy can decompose over tokens, a token level (DPM+T) reward objective can be defined:
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
1 n n t=1   |y| j=1 δ t,j log π w (y t,j |y t,<j , x t )   .
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Analogously, we can define an objective that combines the token-level rewards and the minibatched reweighting (DPM+T+OSL):
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
|y| j=1 δ t,j log π w (y t,j |y t,<j , x t ) 1 n n t=1 π w (y t |x t )
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Gradients for the DPM+T and DPM+T+OSL objectives are given in Table 1.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
OpenStreetMap (OSM) is a geographical database in which volunteers annotate points of interests in the world. A point of interest consists of one or more associated GPS points. Further relevant information may be added at the discretion of the volunteer in the form of tags. Each tag consists of a key and an associated value, for example "tourism : hotel". The NLMAPS corpus was introduced by Haas and Riezler (2016) as a basis to create a natural language interface to the OSM database. It pairs English questions with machine readable parses, i.e. queries that can be executed against OSM.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Human Feedback Collection. The task of creating a natural language interface for OSM demonstrates typical difficulties that make it expensive to collect supervised data. The machine readable language of the queries is based on the OVERPASS query language which was specifically designed for the OSM database. It is thus not easily possible to find experts that could provide correct queries. It is equally difficult to ask workers at crowdsourcing platforms for the correct answer. For many questions, the answer set is too large to expect a worker to count or list them all in a reasonable amount of time and without errors. For example, for the question "How many hotels are there in Paris?" there are 951 hotels annotated in the OSM database. Instead we propose to automatically transform the query into a block of statements that can easily be judged as correct or incorrect by a human. The question and the created block of statements are embedded in a user interface with a form that can be filled out by users. Each statement is accompanied by a set of radio buttons where a user can select either "Yes" or "No". For a screenshot of the interface and an example see Figure 2. In total there are 8 different types of statements. The presence of certain tokens in a query trigger different statement types. For example, the token "area" triggers the statement type "Town". The statement is then populated with the corresponding information from the query. In the case of "area", the following OSM value is used, e.g. "Paris". With this, the meaning of every query can be captured by a set of human-understandable statements. For a full overview of all statement types and their triggers see section B of the supplementary material.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
OSM tags and keys are generally understandable. For example, the correct OSM tag for "hotels" is "tourism : hotel" and when searching for websites, the correct question type key would be "website". Nevertheless, for each OSM tag or key, we automatically search for the corresponding Wikipedia page on the OpenStreetMap Wiki 3 and extract the description for this tag or key. The description is made available to the user in form of a tool-tip that appears when hovering over the tag or key with the mouse. If a user is unsure if a OSM tag or key is correct, they can read this description to help in their decision making. Once the form is submitted, a script maps each statement back to the corresponding tokens in the original query. These tokens then receive negative or positive feedback based on the feedback the user provided for that statement.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Corpus Extension. Similar to the extension of the NLMAPS corpus by Lawrence and Riezler (2016) who include shortened questions which are more typically used by humans in search tasks, we present an automatic extension that allows a larger coverage of common OSM tags. 4 The basis for the extension is a hand-written, online freely available list 5 that links natural language expressions such as "cash machine" to appropriate OSM tags, in this case "amenity : atm". Using the list, we generate for each unique expression-tag pair a set of question-query pairs. These latter pairs contain 3 https://wiki.openstreetmap.org/ 4 The extended dataset, called NLMAPS V2, will be released upon acceptance of the paper.   (Lawrence and Riezler, 2016) and the automatic extensions of the most common OSM tags.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
several placeholders which will be filled automatically in a second step.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
To fill the area placeholder $LOC, we sample from a list of 30 cities from France, Germany and the UK. $POI is the placeholder for a point of interest. We sample it from the list of objects which are located in the prior sampled city and which have a name key. The corresponding value belonging to the name key will be used to fill this spot. The placeholder $QTYPE is filled by uniformly sampling from the four primary question types available in the NLMAPS query language. On the natural language side they corresponded to "How many", "Where", "Is there" and $KEY. $KEY is a further parameter belonging to the primary question operator FINDKEY. It can be filled by any OSM key, such as name, website or height. To ensure that there will be an answer for the generated query, we first ran a query with the current tag ("amenity : atm") to find all objects fulfilling this requirement in the area of the already sampled city. From the list of returned objects and the keys that appear in association with them, we uniformly sampled a key. For $DIST we chose between the pre-defined options for walking distance and within city distance. The expressions map to corresponding values which define the size of a radius in which objects of interest (with tag "amenity : atm") will be located. If the walking distance was selected, we added "in walking distance" to the question. Otherwise no extra text was added to the question, assuming the within city distance to be the default. This sampling process was repeated twice. Table 2 presents the corpus statistics, comparing NLMAPS to our extension. The automatic extension, obviating the need for expensive manual work, allows a vast increase of question-query pairs by an order of magnitude. Consequently the number of tokens and types increase in a similar vein. However, the average sentence length drops. This comes as no surprise due to the nature of the rather simple hand-written list which contains never more than one tag for an element, resulting in simpler question structures. However, the main idea of utilizing this list is to extend the coverage to previously unknown OSM tags. With 6,582 distinct tags compared to the previous 477, this was clearly successful. Together with the still complex sentences from the original corpus, a semantic parser is now able to learn both complex questions and a large variety of tags. An experiment that empirically validates the usefulness of the automatically created data can be found in the supplementary material, section A.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
General Settings. In our experiments we use the sequence-to-sequence neural network package NEMATUS (Sennrich et al., 2017). Following the method used by Haas and Riezler (2016), we split the queries into individual tokens by taking a pre-order traversal of the original tree-like structure. For example, "query(west(area(keyval('name','Paris')), nwr(keyval('railway','station'))),qtype(count))" becomes "query@2 west@2 area@1 keyval@2 name@0 Paris@s nwr@1 keyval@2 railway@0 station@s qtype@1 count@0".
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
The SGD optimizer used is ADADELTA (Zeiler, 2012). The model employs 1,024 hidden units and word embeddings of size 1,000. The maximum sentence length is 200 and gradients are clipped if they exceed a value of 1.0. The stopping point is determined by validation on the development set and selecting the point at which the highest evaluation score is obtained. F1 validation is run after every 100 updates, and each update is made on the basis of a minibatch of size 80.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
The evaluation of all models is based on the answers obtained by executing the most likely query obtained after a beam search with a beam of size 12. We report the F1 score which is the harmonic mean of precision and recall. Recall is defined as the percentage of fully correct answers divided by the set size. Precision is the percentage of correct answers out of the set of answers with non-empty strings. Statistical significance between models is measured using an approximate randomization test (Noreen, 1989).
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Baseline Parser & Log Creation. Our experiment design assumes a baseline neural semantic parser that is trained in fully supervised fashion, and is to be improved by bandit feedback obtained for system outputs from the baseline system for given questions. For this purpose, we select 2,000 question-query pairs randomly from the full extended NLMAPS V2 corpus. We will call this dataset D sup . Using this dataset, a baseline semantic parser is trained in supervised fashion under a cross-entropy objective. It obtains an F1 score of 57.45% and serves as the logging policy π 0 .
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Furthermore we randomly split off 1,843 and 2,000 pairs for a development and test set, respectively. This leaves a set of 22,765 question-query pairs. The questions can be used as input and bandit feedback can be collected for the most likely output of the semantic parser. We refer to this dataset as D log .
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
To collect human feedback, we take the first 1,000 questions from D log and use π 0 to parse these questions to obtain one output query for each. 5 question-query pairs are discarded because the suggested query is invalid. For the remaining question-query pairs, the queries are each transformed into a block of human-understandable statements and embedded into the user interface described in Section 5. We recruited 9 users to provide feedback for these question-query pairs. The resulting log is referred to as D human . Every question-query pair is purposely evaluated only once to mimic a realistic real-world scenario where user logs are collected as users use the system. In this scenario, it is also not possible to explicitly obtain several evaluations for the same question-query pair. Some examples of the received feedback can be found in the supplementary material, section C.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
To verify that the feedback collection is efficient, we measured the time each user took from loading a form to submitting it. To provide feedback for one question-query pair, users took 16.4 seconds on average with a standard deviation of 33.2 seconds. The vast majority (728 instances) are completed in less than 10 seconds.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Learning from Human Bandit Feedback. An analysis of D human shows that for 531 queries all corresponding statements were marked as correct. We consider a simple baseline that treats completely correct logged data as a supervised data set with which training continues using the crossentropy objective. We call this baseline banditto-supervised conversion (B2S). Furthermore, we present experimental results using the log D human for stochastic (minibatch) gradient descent optimization of the counterfactual objectives introduced in equations 4, 6, 7 and 8. For the tokenlevel feedback, we map the evaluated statements back to the corresponding tokens in the original query and assign these tokens a feedback of 0 if the corresponding statement was marked as wrong and 1 otherwise. In the case of sequence-level feedback, the query receives a feedback of 1 if all statements are marked correct, 0 otherwise. For the OSL objectives, a separate experiment (see below) showed that updating the reweighting constant after every validation step promises the best trade-off between performance and speed.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Results, averaged over 3 runs, are reported in Table 3. The B2S model can slightly improve upon the baseline but not significantly. DPM improves further, significantly beating the baseline. Using the multiplicative control variate modified for SGD by OSL updates does not seem to help in this setup. By moving to token-level rewards, it is possible to learn from partially correct queries. These partially correct queries provide valuable information that is not present in the subset of correct answers employed by the previous models. Optimizing DPM+T leads to a slight improvement and combined with the multiplicative control variate, DPM+T+OSL yields an improvement of about 1.0 in F1 score upon the baseline. It beats both the baseline and the B2S model by a significant margin.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Learning from Large-Scale Simulated Feedback. We want to investigate whether the results scale if a larger log is used. Thus, we use π 0 to parse all 22,765 questions from D log and obtain for each an output query. For sequence level rewards, we assign feedback of 1 for a query if it is identical to the true target query, 0 otherwise. We also simulate token-level rewards by iterating over the indices of the output and assigning a feedback of 1 if the same token appears at the current index for the true target query, 0 otherwise.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
An analysis of D log shows that 46.27% of the queries have a sequence level reward of 1 and are   thus completely correct. This subset is used to train a bandit-to-supervised (B2S) model using the cross-entropy objective. Experimental results for the various optimization setups, averaged over 3 runs, are reported in Table 4. We see that the B2S model outperforms the baseline model by a large margin, yielding an increase in F1 score by 6.24 points. Optimizing the DPM objective also yields a significant increase over the baseline, but its performance falls short of the stronger B2S baseline. Optimizing the DPM+OSL objective leads to a substantial improvement in F1 score over optimizing DPM but still falls slightly short of the strong B2S baseline. Token-level rewards are again crucial to beat the B2S baseline significantly. DPM+T is already able to significantly outperform B2S in this setup and DPM+T+OSL can improve upon this further.  tained the correct answer and the baseline system did not (see Table 5). The analysis showed that the vast majority of previously wrong queries were fixed by correcting an OSM tag in the query. For example, for the question "closest Florist from Manchester in walking distance" the baseline system chose the tag "landuse : retail" in the query, whereas DPM+T+OSL learnt that the correct tag is "shop : florist". In some cases, the question type had to be corrected, e.g. the baseline's suggested query returned the location of a point of interest but DPM+T+OSL correctly returns the phone number. Finally, in a few cases DPM+T+OSL corrected the structure for a query, e.g. by searching for a point of interest in the east of an area rather than the south.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Variation. Using the DPM+T+OSL objective and the simulated feedback setup, we vary the frequency of updating the reweighting constant. Results are reported in Table 6. Calculating the constant only once at the beginning leads to a near identical result in F1 score as not using OSL. The more frequent update strategies, once or four times per epoch, are more effective. Both strategies reduce variance further and lead to higher F1 scores. Updating four times per epoch compared to once per epoch, leads to a nominally higher performance in F1. It has the additional benefit that the re-calculation is done at the same time as the validation, leading to no additional slow down as executing the queries for the development set against the database takes longer than the re-calculation of the constant. Updating after every minibatch is infeasible as it slows down training too much. Compared to the previous setup, iterating over one epoch takes approximately an additional 5.5 hours.  
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
We introduced a scenario for improving a neural semantic parser from logged bandit feedback. This scenario is important to avoid complex and costly data annotation for supervise learning, and it is realistic in commercial applications where weak feedback can be collected easily in large amounts from users. We presented robust counterfactual learning objectives that allow to perform stochastic gradient optimization which is crucial in working with neural networks. Furthermore, we showed that it is essential to obtain reward signals at the token-level in order to learn from partially correct queries. We presented experimental results using feedback collected from humans and a larger scale setup with simulated feedback. In both cases we show that a strong baseline using a bandit-to-supervised conversion can be significantly outperformed by a combination of a onestep-late reweighting and token-level rewards. Finally, our approach to collecting feedback can also be transferred to other domains. For example, (Yih et al., 2016) designed a user interface to help Freebase experts to efficiently create queries. This interface could be reversed: given a question and a query produced by a parser, the interface is filled out automatically and the user has to verify if the information fits.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
To empirically validate the usefulness of the automatically created data, we compare two parsers trained with NEMATUS (Sennrich et al., 2017) (see Section 6 for more details). The first model is trained using the original NLMAPS training data. The second receives an additional 15,000 instances from the synthetic data. Both systems are tested on the original NLMAPS test data and on the new test set of NLMAPS V2 which consists of a random set of 2,000 pairs from the remaining data. Results may be found in  Table 7: Answer F1 scores on the NLMAPS V1 and NLMAPS V2 test sets for models trained on either only NLMAPS V1 training data or with an additional 15k synthetic instances. Results are averaged over 3 runs. Using the NLMAPS V2 training set leads to significant system differences on both test sets at p < 0.05.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
A query can be automatically converted into a set of statements which can be judged as correct or incorrect by non-expert users. There are 8 different statement types and each is triggered based on the shape of the query and certain tokens. An overview of the statement types, their triggers and the value a statement will hold, can be found in Table 8.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Figures 3, 4, 5, 6 and 7 present screenshots of forms as filled out by the recruited human users.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Type Explanation Town OSM tags of "area" Reference Point OSM tags "center" POI(s) OSM tags of "search" if "center" is set, else of "nwr" Question Type Arguments of "qtype" Proximity : Around/Near If "around" is present Restriction : Closest If "around" and "topx" are present Distance Argument of "maxdist" Cardinal Direction "north", "east", "south" or "west" are present      
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
With the rapid growth of text-based information on the Internet, automatic document summarisation attracts increasing research attention from the Natural Language Processing (NLP) community (Nenkova and McKeown, 2012). Most existing document summarisation techniques require access to reference summaries to train their systems. However, obtaining reference summaries is very expensive: Lin (2004) reported that 3,000 hours of human effort were required for a simple evaluation of the summaries for the Document Understanding Conferences (DUC). Although previous work has proposed heuristics-based methods to summarise without reference summaries (Ryang and Abekawa, 2012;Rioux et al., 2014), the gap between their performance and the upper bound is still large: the ROUGE-2 upper bound of .212 on DUC'04 (P.V.S. and Meyer, 2017) is, for example, twice as high as Rioux et al.'s (2014) .114.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
The Structured Prediction from Partial Information (SPPI) framework has been proposed to learn to make structured predictions without access to gold standard data (Sokolov et al., 2016b). SPPI is an interactive NLP paradigm: It interacts with a user for multiple rounds and learns from the user's feedback. SPPI can learn from two forms of feedback: point-based feedback, i.e. a numeric score for the presented prediction, or preference-based feedback, i.e. a preference over a pair of predictions. Providing preference-based feedback yields a lower cognitive burden for humans than providing ratings or categorical labels (Thurstone, 1927;Kendall, 1948;Kingsley and Brown, 2010;Zopf, 2018). Preference-based SPPI has been applied to multiple NLP applications, including text classification, chunking and machine translation (Sokolov et al., 2016a;Kreutzer et al., 2017). However, SPPI has prohibitively high sample complexities in the aforementioned NLP tasks, as it needs at least hundreds of thousands rounds of interaction to make near-optimal predictions, even with simulated "perfect" users. Figure 1a illustrates the workflow of the preference-based SPPI.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
To reduce the sample complexity, in this work, we propose a novel preference-based interactive learning framework, called APRIL (Active Preference ReInforcement Learning). APRIL goes beyond SPPI by proposing a new objective function, which divides the preference-based interactive learning problem into two phases (illustrated in Figure 1b): an Active Preference Learning (APL) phase (the right cycle in Figure 1b), and a Reinforcement Learning (RL) phase (the left cycle). We show that this separation enables us to query preferences more effectively and to use the collected preferences more efficiently, so as to reduce the sample complexity. and APRIL (b) in the EMDS use case. Notation details, e.g., ∆ x and r(y n ), are discussed in §3.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
We apply APRIL to Extractive Multi-Document Summarisation (EMDS). The task of EMDS is to extract sentences from the original documents to build a summary under a length constraint. We accommodate multiple APL and RL techniques in APRIL and compare their performance under different simulation settings. We also compare APRIL to a state-of-the-art SPPI implementation using both automatic metrics and human evaluation. Our results suggest that APRIL significantly outperforms SPPI.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
In this section, we recap necessary details of SPPI, RL and preference learning, and adapt them to the EMDS use case, laying the foundation for APRIL.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
Let X be the input space and let Y(x) be the set of possible outputs for input x ∈ X . In EMDS, x ∈ X is a cluster of documents and Y(x) is the set of all possible summaries for cluster x. The function ∆ x : Y(x)×Y(x) → {0, 1} is the preference function such that ∆ x (y i , y j ) = 1 if the user believes y j is better than y i (denoted by y j y i or equivalently y i ≺ y j ), and 0 otherwise. Throughout this paper we assume that users do not equally prefer two different items. For a given x, the expected loss is:
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
where p w (y i , y j |x) is the probability of querying the pair (y i , y j ). Formally,
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
where φ(y|x) is the vector representation of y given x, and w is the weight vector to be learnt. Eq.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
(2) is a Gibbs sampling strategy: w (φ(y i |x)− φ(y j |x)) can be viewed as the "utility gap" between y i and y j . The sampling strategy p w encourages querying pairs with large utility gaps.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
To minimise L SPPI , SPPI uses gradient descent to update w incrementally. Alg. 1 presents the pseudo code of our adaptation of SPPI to EMDS. In the supplementary material, we provide a detailed derivation of ∇ w L SPPI (w|x).
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
RL amounts to efficient algorithms for searching optimal solutions in MDPs. MDPs are widely Input : sequence of learning rates γ t ; query budget T ; document cluster x initialise w 0 ; while t = 0 . . . T do sample (y i , y j ) according to Eq. (2); obtain feedback ∆ x (y i , y j ); w t+1 := w t − γ t ∇ w L SPPI (w|x) end Output: y * = arg max y∈Y (x) w T +1 φ(y, x) Algorithm 1: SPPI for preference-based interactive document summarisation (adjusted from Alg. 2 in (Sokolov et al., 2016a)).
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
used to formulate sequential decision making problems, which EMDS falls into: in EMDS, the summariser has to sequentially select sentences from the original documents and add them to the draft summary. An (episodic) MDP is a tuple (S, A, P, R, T ). S is the set of states, A is the set of actions, P : S × A × S → R is the transition function with P (s |s, a) yielding the probability of performing action a in state s and being transited to a new state s . R : S × A → R is the reward function with R(s, a) giving the immediate reward for performing action a in state s. T ⊆ S is the set of terminal states; visiting a terminal state terminates the current episode.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
In EMDS, we follow the same MDP formulation as Ryang and Abekawa (2012) and Rioux et al. (2014). Given a document cluster, a state s is a draft summary, A includes two types of actions, concatenate a new sentence to the current draft summary, or terminate the draft summary construction. The transition function P in EMDS is trivial because given the current draft summary and an action, the next state can be easily inferred. The reward function R returns an evaluation score of the summary once the action terminate is performed; otherwise it returns 0 because the summary is still under construction and thus not ready to be evaluated. Providing non-zero rewards before the action terminate can lead to even worse result, as reported by Rioux et al. (2014).
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
A policy π : S × A → R in an MDP defines how actions are selected: π(s, a) is the probability of selecting action a in state s. In EMDS, a policy corresponds to a strategy to build summaries for a given document cluster. We let Y π (x) be the set of all possible summaries the policy π can construct in the document cluster x, and we slightly abuse the notation by letting π(y|x) denote the probabil-ity of policy π generating a summary y in cluster x. Then the expected reward of a policy is:
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
where R(y|x) is the reward for summary y in document cluster x. The goal of an MDP is to find the optimal policy π * that has the highest expected reward: π * = arg max π R RL (π).
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
Note that the loss function in SPPI (Eq. (1)) and the expected reward function in RL (Eq. (3)) are in similar forms: if we view the pair selection probability p w in Eq. (2) as a policy, and view the preference function ∆ x in Eq. (1) as a negative reward function, we can view SPPI as an RL problem. The major difference between SPPI and RL is that SPPI selects and evaluates pairs of outputs, while RL selects and evaluates single outputs. We will exploit their connection to propose our new objective function and the APRIL framework.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
The linear Bradley-Terry (BT) model (Bradley and Terry, 1952) is one of the most widely used methods in preference learning. Given a set of items Y, suppose we have observed T preferences: Q = {q 1 (y 1,1 , y 1,2 ), · · · , q T (y T,1 , y T,2 )}, where y i,1 , y i,2 ∈ Y, and q i ∈ {≺, } is the oracle's preference in the i th round. The BT model minimises the following cross-entropy loss:
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
) −1 , and µ i,1 and µ i,2 indicate the direction of preferences: if y i,1 y i,2 then µ i,1 = 1 and µ i,2 = 0. Let w * = arg min w L BT (w), then w * can be used to rank all items in Y: for any y i , y j ∈ Y, the ranker prefers y i over y j if w * φ(y i ) > w * φ(y j ).
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
A major problem of SPPI is its high sample complexity. We believe this is due to two reasons. First, SPPI's sampling strategy is inefficient: From Eq.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
(2) we can see that SPPI tends to select pairs with large quality gaps for querying the user. This strategy can quickly identify the relatively good and relatively bad summaries, but needs many rounds of interaction to find the top summaries. Second, SPPI uses the collected preferences ineffectively: In Alg. 1, each preference is used only once for performing the gradient descent update and is forgotten afterwards. SPPI does not generalise or re-use collected preferences, wasting the useful and expensive information. These two weaknesses of SPPI motivate us to propose a new learning paradigm that can query and generalise preferences more efficiently. Recall that in EMDS, the goal is to find the optimal summary for a given document cluster x, namely the summary that is preferred over all other possible summaries in Y(x). Based on this understanding, we define a new expected reward function R APRIL for policy π as follows:
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
where r(y|x) = y i ∈Y(x) ∆ x (y i , y j )/|Y(x)|. Note that ∆ x (y i , y j ) equals 1 if y j is preferred over y i and equals 0 otherwise (see §3.1). Thus, r(y|x) is the relative position of y in the (ascending) sorted Y(x), and it can be approximated by preference learning. The use of preference learning enables us to generalise the observed preferences to a ranker (see §3.3), allowing more effective use of the collected preferences. Also, we can use active learning to select summary pairs for querying more effectively. In addition, the resemblance of R APRIL and RL's reward function R RL (in Eq. (3)) enables us to use a wide range of RL algorithms to maximise R APRIL (see §2).
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
Based on the new objective function, we split the preference-based interactive learning into two phases: an Active Preference Learning (APL) phase (the right cycle in Fig. 1b), responsible for querying preferences from the oracle and approximating the ranking of summaries, and an RL phase (the left cycle in Fig. 1b), responsible for learning to summarise based on the learned ranking. The resulting framework APRIL allows for integrating any active preference learning and RL techniques. Note that only the APL phase is online (i.e. in-Input : query budget T ; document cluster x;
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
RL episode budget N /* Phase 1: active preference learning */ while t = 0 . . . T do sample a summary pair (y i , y j ) using any APL strategy; obtain feedback ∆ x (y i , y j ); update ranker according to Eq. (4) ; end /* Phase 2: RL-based summarisation */ initialise an arbitrary policy π 0 ; while n = 0 . . . N do evaluate policy π n according to Eq. (5); update policy π n using any RL algorithm; end Output: y * = arg max y∈Yπ N (x) π N (y|x)  volving humans in the loop) while the RL phase can be performed offline, helping to improve the real-time responsiveness. Also, the learned ranker can provide an unlimited number of rewards (i.e. r(y|x) in Eq. (5)) to the RL agent, enabling us to perform many episodes of RL training with a small number of collected preferences -unlike in SPPI where each collected preference is used to train the system for one round and is forgotten afterwards. Alg. 2 shows APRIL in pseudo code.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
Datasets. We perform experiments on DUC '04 to find the best performing APL and RL techniques. Then we combine the best-performing APL and RL to complete APRIL and compare it against SPPI on the DUC '01, DUC '02 and DUC '04 datasets. 1 Some statistics of these datasets are summarised in Table 1.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
Simulated Users. Existing preference-based interactive learning techniques assume that the oracle has an intrinsic evaluation function U * and provides preferences consistent with U * by preferring higher valued candidates. We term this a Per-1 http://duc.nist.gov/ fect Oracle (PO). We believe that assuming a PO is unrealistic for real-world applications, because sometimes real users tend to misjudge the preference direction, especially when the presented candidates have similar quality. In this work, besides PO, we additionally consider two types of noisy oracles based on the user-response models proposed by Viappiani and Boutilier (2010):
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
• Constant noisy oracle (CNO): with probability c ∈ [0, 1], this oracle randomly selects which summary is preferred; otherwise it provides preferences consistent with U * . We consider CNOs with c = 0.1 and c = 0.3.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
• Logistic noisy oracle (LNO): for two summaries y i and y j in cluster x, the oracle prefers y i over y j with probability p U * (y i y j |x; m) = (1 + exp[(U * (y j |x) − U * (y i |x))/m]) −1 . This oracle reflects the intuition that users are more likely to misjudge the preference direction when two summaries have similar quality. Note that the parameter m ∈ R + controls the "noisiness" of the user's responses: higher values of m result in a less steep sigmoid curve, and the resulting oracle is more likely to misjudge. We use LNOs with m = 0.3 and m = 1.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
As for the intrinsic evaluation function U * , recent work has suggested that human preferences over summaries have high correlations to ROUGE scores (Zopf, 2018). Therefore, we define:
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
where R 1 , R 2 and R S stand for ROUGE-1, ROUGE-2 and ROUGE-SU4, respectively. The real values (0.47, 0.22 and 0.18) are used to balance the weights of the three ROUGE scores. We choose them to be around the EMDS upper-bound ROUGE scores reported by P.V.S. and Meyer (2017). As such, an optimal summary's U * value should be around 3.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
Implementation. All code is written in Python and runs on a desktop PC with 8 GB RAM and an i7-2600 CPU. We use NLTK (Bird et al., 2009) to perform sentence tokenisation. Our source code is freely available at https://github.com/ UKPLab/emnlp2018-april.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
We first study the APL phase ( §6.1) and the RL phase ( §6.2)) separately by comparing the perfor-mance of multiple APL and RL algorithms in each phase. Then, in §6.3, we combine the best performing APL and RL algorithm to complete Alg. 2 and compare APRIL against SPPI.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
Recall that the task of APL is to output a ranking of all summaries in a cluster. In this subsection, we test multiple APL techniques and compare the quality of their resulting rankings. Two metrics are used: Kendall's τ (Kendall, 1948) and Spearman's ρ (Spearman, 1904). Both metrics are valued between −1 and 1, with higher values suggesting higher rank correlation. Because the number of possible summaries in a cluster is huge, instead of evaluating the ranking quality on all possible summaries, we evaluate rankings on 10,000 randomly sampled summaries, denotedŶ(x). During querying, all candidate summaries presented to the oracle are also selected fromŶ(x). SamplingŶ(x) a priori helps us to reduce the response time to under 500 ms for all APL techniques we test. We compare four active learning strategies under two query budgets, T = 10 and T = 100:
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
• Random Sampling (RND): Randomly select two summaries fromŶ(x) to query.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
• SPPI Sampling (SBT): Select summary pairs fromŶ(x) according to the SPPI strategy in Eq.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
(2). After each round, the weight vector w is updated according to Eq. (4).
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
• Uncertainty Sampling (Unc): Query the most uncertain summary pairs. In line with P.V.S. and Meyer (2017), the uncertainty of a summary is evaluated as follows: first, we estimate the probability of a summary y being the optimal summary in cluster x as p opt (y|x) = (1 + exp(−w * t φ(x, y))) −1 , where w * t is the weights learned by the BT model (see §3.3) in round t. Given p opt (y|x), we let the uncertainty score unc(y|x) = 1 − p opt (y|x) if p opt (y|x) ≥ 0.5 and unc(y|x) = p opt (y|x) otherwise.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
• J&N is the robust query selection algorithm proposed by Jamieson and Nowak (2011). It assumes that the items' preferences are dependent on their distances to an unknown reference point in the embedding space: the farther an item to the reference point, the more preferred the item is. After each round of interaction, the algorithm uses all collected preferences to locate the area where the reference point may fall into, and identify the query pairs which can reduce the size of this area, termed ambiguous query pairs. To combat noise in preferences, the algorithm selects the most-likely-correct ambiguous pair to query the oracle in each round.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
After all preferences are collected, we obtain the ranker as follows: for any y i , y j ∈ Y(x), the ranker prefers y i over y j if
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
where w * is the weights vector learned by the BT model (see Eq. (4)), HU is the heuristics-based summary evaluation function proposed by Ryang and Abekawa (2012), and α ∈ [0, 1] is a parameter. The aim of using HU and α is to trade off between the prior knowledge (i.e. heuristics-based HU ) and the posterior observation (i.e. the BTlearnt w * ), so as to combat the cold-start problem. Based on some preliminary experiments, we set α = 0.3 when the query budget is 10, and α = 0.7 when the query budget is 100. The intuition is to put more weight to the posterior with increasing rounds of interaction. More systematic research of α can yield better results; we leave it for future work. For the vector φ(y|x), we use the same bagof-bigram embeddings as Rioux et al. (2014), and we let its length be 200.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
In Table 2, we compare the performance of the four APL methods on the DUC'04 dataset. The baseline we compared against is the prior ranking. We find that Unc significantly 2 outperforms all other APL methods, except when the oracle is LNO-1, where the advantage of Unc to SBT is not significant. Also, both Unc and SBT are able to significantly outperform the baseline under all settings. The competitive performance of SBT, especially with LNO-1, is due to its unique sampling strategy: LNO-1 is more likely to misjudge the preference direction when the presented summaries have similar quality, but SBT has high probability to present summaries with large quality gaps (see Eq.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
(2)), effectively reducing the chance that LNOs misjudge preference directions. However, SBT is more "conservative" compared to Unc because it tends to exploit the existing Baseline, α = 0, T = 0: τ = .206, ρ = .304 Table 2: Performance of multiple APL algorithms (columns) using different oracles and query budgets (rows). The baseline is the purely prior ranking. All results except the baseline are averaged over 50 document clusters in DUC'04. Asterisk: significant advantage over other active learning strategies given the same oracle and budget T .
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
ranking to select one good and one bad summary to query, while Unc performs more exploration by querying the summaries that are least confident according to the current ranking. We believe this explains the strong overall performance of Unc. Additional experiments suggest that when we only use the posterior ranking (i.e. letting α = 1), no APL we test can surpass the baseline when T = 10. Detailed results are presented in the supplementary material. This observation reflects the severity of the cold-start problem, confirms the effectiveness of our prior-posterior trade-off mechanism in combating cold-start, and indicates the importance of tuning the α value (see Eq. (7)). This opens up exciting avenues for future work.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
We compare two RL algorithms: TD(λ) (Sutton, 1984) and LSTD(λ) (Boyan, 1999). TD(λ) has been used in previous RL-based EMDS work (Ryang and Abekawa, 2012;Rioux et al., 2014). LSTD(λ) is chosen, because it is an improved TD algorithm and has been used in the state-of-the-art PbRL algorithm by Wirth et al. (2016). We let the learning round (see Alg. 2) N = 5, 000, which we found to yield good results in reasonable time (less than 1 minute to generate a summary for one document cluster). Letting N = 3, 000 will result in a significant performance drop, while increasing N to 10,000 will only bring marginal improvement at the cost of doubling the runtime. The learn-  ing parameters we use for TD(λ) are the same as those by Rioux et al. (2014). For LSTD(λ), we let λ = 1 and initialise its square matrix as a diagonal matrix with random numbers between 0 and 1, as suggested by Lagoudakis and Parr (2003). The rewards we use are the U * function introduced in §5. Note that this serves as the upper-bound performance, because U * relies on the reference summaries (see Eq. (6)), which are not available in the interactive setting. As a baseline, we also present the upper-bound performance of integer linear programming (ILP) reported by P.V.S. and Meyer (2017), optimised for bigram coverage. Table 3 shows the performance of RL and ILP on the DUC'04 dataset. TD(λ) significantly outperforms LSTD(λ) in terms of all ROUGE scores we consider. Although the least-square RL algorithms (which LSTD belongs to) have been proved to achieve better performance than standard TD methods in large-scale problems (see Lagoudakis and Parr, 2003), their performance is sensitive to many factors, e.g., initialisation values in the diagonal matrix, regularisation parameters, etc. We note that a similar observation about the inferior performance of least-square RL in EMDS is reported by Rioux et al. (2014).
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
TD(λ) also significantly outperforms ILP in terms of all metrics except ROUGE-2. This is not surprising, because the bigram-based ILP is optimised for ROUGE-2, whereas our reward function U * considers other metrics as well (see Eq. (6)). Since ILP is widely used as a strong baseline for EMDS, these results confirm the advantage of using RL for EMDS problems.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
Finally, we combine the best techniques of the APL and RL phase (namely Unc and TD(λ), respectively) to complete APRIL, and compare it against SPPI. As a baseline, we use the heuristicbased rewards HU to train both TD(λ) (rankingbased training, i.e. using HU to produce r(y|x) in Eq. (5) to train) and SPPI (preference-based training, i.e. using HU for generating pairs to train   SPPI) for up to 5,000 episodes. The baseline results are presented in the bottom rows of Table 4.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
We make the following observations from Table 4. (i) Given the same oracle, the performance of APRIL with 10 rounds of interaction is comparable or even superior than that of SPPI after 100 rounds of interaction (see boldface in Table  4), suggesting the strong advantage of APRIL to reduce sample complexity. (ii) APRIL can significantly improve the baseline with either 10 or 100 rounds of interaction, but SPPI's performance can be even worse than the baseline (marked by † in Table 4), especially under the high-noise lowbudget settings (i.e., CNO-0.3, LNO-0.3, and LNO-1 with T = 10). This is because SPPI lacks a mechanism to balance between prior and posterior ranking, while APRIL can adjust this trade-off by tuning α (Eq. (7)). This endows APRIL with better noise robustness and lower sample complexity in high-noise low-budget settings. Note that the above observations also hold for the other two datasets, indicating the consistently strong performance of APRIL across different datasets.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
As for the overall runtime, when budget T = 100, APRIL on average takes 2 minutes to interact with an oracle and output a summary, while SPPI takes around 15 minutes due to its expensive gradient descent computation (see §3.1).
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
Finally, we invited real users to compare and evaluate the quality of the summaries generated by SPPI and APRIL. We randomly selected three topics (d19 from DUC'01, d117i from DUC'02 and d30042 from DUC'04), and let both SPPI and our best-performing APRIL interact with PO for 10 rounds on these topics. The resulting 100-word summaries, shown in Figure 2, were presented to seven users, who had already read two background texts to familiarize with the topic. The users were asked to provide their preference on the presented Topic d30042 (DUC'04), SPPI: After meeting Libyan leader Moammar Gadhafi in a desert tent, U.N. Secretary-General Kofi Annan said he thinks an arrangement for bringing two suspects to trial in the bombing of a Pan Am airliner could be secured in the "not too distant future." TRIPOLI, Libya (AP) U.N. Secretary-General Kofi Annan arrived in Libya Saturday for talks aimed at bringing to trial two Libyan suspects in the 1988 Pan Am bombing over Lockerbie, Scotland. Secretary General Kofi Annan said Wednesday he was extending his North African tour to include talks with Libyan authorities. Annan's one-day, 2nd graf pvs During his Algerian stay, Topic d30042 (DUC'04), APRIL: TRIPOLI, Libya (AP) U.N. Secretary-General Kofi Annan arrived in Libya Saturday for talks aimed at bringing to trial two Libyan suspects in the 1988 Pan Am bombing over Lockerbie, Scotland. Annan's one-day visit to meet with Libyan leader Col. Moammar Gadhafi followed reports in the Libyan media that Gadhafi had no authority to hand over the suspects. The 60-year-old Annan is trying to get Libya to go along with a U.S.-British plan to try the two suspects before a panel of Scottish judges in the Netherlands for the Dec. 21, 1988, bombing over Lockerbie, Scotland. Sirte is 400 kilometers (250 miles) east of the Libyan capital Tripoli. During his Algerian stay, Topic d117i (DUC'02), SPPI: The Booker Prize is sponsored by Booker, an international food and agriculture business. The novel, a story of Scottish lowlife narrated largely in Glaswegian dialect, is unlikely to prove a popular choice with booksellers, who have damned all six books shortlisted for the prize as boring, elitist and-worst of all-unsaleable. The shortlist of six for the Pounds 20,000 Booker Prize for fiction, announced yesterday, immediately prompted the question 'Who ? ' Japanese writer Kazuo Ishiguro won the 1989 Booker Prize, Britain's top literary award, for his novel "The Remains of the Day," judges announced Thursday. He didn't win.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
Topic d117i (DUC'02), APRIL: Australian novelist Peter Carey was awarded the coveted Booker Prize for fiction Tuesday night for his love story, "Oscar and Lucinda." The Booker Prize is sponsored by Booker, an international food and agriculture business, and administered by The Book Trust. British publishers can submit three new novels by British and Commonwealth writers. Six novels have been nominated for the Booker Prize, Britain's most prestigious fiction award, and bookmakers say the favorite is "The Remains of the Day" by Japanese author Kazuo Ishiguro. On the day of the Big Event, Ladbroke, the large British betting agency, posted the final odds.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
Topic d19 (DUC'01), SPPI: The issue cuts across partisan lines in the Senate, with Minority Leader Bob Dole (R-Kan.) arguing against the White House position on grounds that including illegal aliens in the census is unfair to American citizens.. Loss of Seats Cited. Shelby's amendment says only that the secretary is to "make such adjustments in total population figures as may be necessary, using such methods and procedures as the secretary determines feasible and appropriate" to keep illegal aliens from being counted in congressional reapportionment. "Some states will lose congressional seats because of illegal aliens," Dole argued. But there's nothing simple about it.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
Topic d19 (DUC'01), APRIL: In a blow to California and other states with large immigrant populations, the Senate voted Friday to bar the Census Bureau from counting illegal aliens in the 1990 population count. But the Senate already has voted to force the Census Bureau to exclude illegal immigrants in preparing tallies for congressional reapportionment. said that Georgia and Indiana both lost House seats after the 1980 Census, and California and New Yorkcenters of illegal immigration-each gained seats. A majority of the members of the House of Representatives has signaled support. The national head count will be taken April 1, 1990. summary pairs and rate the summaries on a 5point Likert scale with higher scores for better summaries. All users are fluent in English.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
In all three topics, all users prefer the APRILgenerated summaries over the SPPI-generated summaries. Table 5 shows the users' ratings. The APRIL-generated summaries consistently receive higher ratings. These results are consistent with our simulation experiments and confirm the significant advantage of APRIL over SPPI.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
We propose a novel preference-based interactive learning formulation named APRIL (Active Preference ReInforcement Learning), which is able to make structured predictions without referring to the gold standard data. Instead, APRIL learns from preference-based feedback. We designed a novel objective function for APRIL, which naturally splits APRIL into an active preference learning (APL) phase and a reinforcement learning (RL) phase, enabling us to leverage a wide spectrum of active learning, preference learning and RL algorithms to maximise the output quality with a limited number of interaction rounds. We applied APRIL to the Extractive Multi-Document Summarisation (EMDS) problem, simulated the users' preference-giving behaviour using multiple user-response models, and compared the performance of multiple APL and RL techniques. Simulation experiments indicated that APRIL signif-icantly improved the summary quality with just 10 rounds of interaction (even with high-noise oracles), and significantly outperformed SPPI in terms of both sample complexity and noise robustness. Human evaluation results suggested that real users preferred the APRIL-generated summaries over the SPPI-generated ones.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
We identify two major lines of future work. On the technical side, we plan to employ more advanced APL and RL algorithms in APRIL, such as sample-efficient Bayesian-based APL algorithms (e.g., Simpson and Gurevych, 2018) and neural RL algorithms (e.g. Mnih et al., 2015) to further reduce the sample complexity of APRIL. On the experimental side, a logical next step is to implement an interactive user interface for APRIL and conduct a larger evaluation study comparing the summary quality before and after the interaction. We also plan to apply APRIL to more NLP applications, including machine translation, information exploration and semantic parsing.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Pre-training large language models has pushed the boundaries of open-domain dialogue agents (Adiwardana et al., 2020;Zhang et al., 2020;, however growing evidence has shown that fine-tuning these models gives further considerable gains on the tasks people care about Thoppilan et al., 2022;Bai et al., 2022). Collecting such fine-tune data via paid crowdworkers gives the opportunity to release such data to the community to conduct research, but does not ultimately scale in size and may not reflect the interests of organic users. An alternative, that we advocate for, is the public deployment of such agents to circumvent these issues.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
If successful, this could provide large-scale organic interactions with humans, and give the opportunity to study the continual improvement of models over time. Further, we expect innovation in this area will be accelerated if the artifacts of such a system are made available to the research community Shuster et al., 2021b).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
In this technical report, we present BlenderBot 3 (BB3), an open-domain dialogue model that we have deployed as an English speaking conversational agent on a public website accessible by adults in the United States. We aim to fully and responsibly share the models, code and collected conversations with interested researchers, as a critical part of our program is that this research should be accessible and reproducible (Sonnenburg et al., 2007;Pineau et al., 2021). The goal of this research program is then to explore how to construct models that continue to improve from such interactions both in terms of becoming more responsible and more useful.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The main contributions (and components) of this work are:
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• We present the BlenderBot 3 (BB3) model itself, which is a 175B parameter transformer initialized from the pre-trained model OPT-175B (Zhang et al., 2022) and then fine-tuned to perform modular tasks to complete its goals, based on our team's recent work . BB3 inherits the attributes of its predecessors, including storing information in a long-term memory and searching the internet for information.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• We study how to train on human feedback from conversations in order to be better at the skills that people find important, with a full report given in a companion paper (Xu et al., 2022b). We use these findings to help finetune BB3 on a large number of user defined tasks. • We detail the deployment design, including its user interface (UI). We report initial experiments conducted with organic user interactions.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• To conduct responsible continual learning with humans-in-the-loop we need learning algorithms that are robust to adversarial behavior. We describe techniques we have developed in this area, with a full report given in a companion paper (Ju et al., 2022).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• We report overall results of our model. Our newly released system outperforms existing openly available chatbots including its two predecessors by a wide margin.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• We release our new model weights, code, model card, conversational datasets and publications describing our work. We also detail our plan for releasing live deployment interactions and updated model snapshots derived from continual learning in the near future.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
BB3 is a modular system but the modules are not independent components -this is achieved by training a single transformer model to execute the modules, with special control codes in the input context telling the model which module it is executing. The input context otherwise typically contains the dialogue history (sometimes truncated, depending on the module), with each speaker prefixed with their ID, either "Person 1:" or "Person 2:" in order to differentiate them. The modules are called in succession, conditional on the results of previous modules, the flow of which is described in §3.1.1 and Figure 2. See Table 1 for the set of modules, which we now also describe below.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Internet search decision Given the last turn of context, this module outputs whether internet search should be conducted or not.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Generate internet search query Given the full input context, generate a search query to be issued to an internet search engine.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Internet search This module is not executed by the transformer but a call to the actual internet search engine. It returns N documents/snippets. In our deployment we use Mojeek (https:// www.mojeek.com/). Generate knowledge response Given the full input context and a set of retrieved documents, generate a sequence referred to as the knowledge response (Adolphs et al., 2021), which is used to ground the final response.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Extract relevant entity Given the full input context, generate a relevant entity which is used to ground the final response.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Generate a long-term memory Given the last turn of context, output a summary of that last turn that will be stored in the long-term memory. For example if the last turn was "Yes, it's all true, my cat is black!" the output summary generated might be "I have a black cat.". This is based off the system in Xu et al. (2022a). If the model thinks no summary should be generated for that turn it outputs "no persona".
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Long-term memory access decision Given the last turn of context, and a store of (text-based) memories, output whether long-term memory access should be conducted or not.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Access long-term memory Given the full input context, and a store of (text-based) memories, output a memory from the memory store, referred to as a recalled memory. Note: if the memory store is too large to fit in the context, we adopt some simple strategies. For the 3B parameter model, we use the Fusion-in-Decoder method (Izacard and Grave, 2021). For the OPT-based models for simplicity of implementation, we sample the memories to fit in the 2048 token context. We keep those with overlapping keywords to prior turns.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Generate dialogue response Given the full input context and optionally a knowledge response and recalled memory, generate a final conversational response. The knowledge and memory sequences are marked with special prefix tokens.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Given a new utterance from the conversational partner, the first thing the model does is determine whether search and long-term memory access are required.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
If search is required, a search query is generated, internet search is invoked, and then a knowledge response is generated given the retrieved documents. Extract relevant entity Full context Extract an entity on which to ground the response.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Generate a long-term memory Last turn of context Generate a memory sequence, which is then stored in the long-term memory. If no plausible memory to generate, output "no persona".
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Long-term memory access decision
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Last turn of context + store of memories Return "access memory" or "do not access memory" depending on whether required or not.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Access long-term memory Full context + store of memories Return an appropriate memory.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Generate dialogue response Full context + knowledge + memory sequences Generate a conversational response given the context.  This sequence will be appended to the context (prefixed with control tokens) in order to generate the final response.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
If long-term memory access is required, the longterm memory is accessed, and a memory is chosen (generated). This is also appended to the context (prefixed with control tokens) as input for the module that generates the final dialogue response.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
If neither search nor long-term memory access is required, an entity is extracted from the history instead, and that is appended to the context (prefixed with control tokens).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Finally, given the constructed context from the previous modules, the final dialogue response generation module is invoked to generate a reply seen by the conversational partner.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
BB3 comes in three sizes. The 3B parameter version is an encoder-decoder based on the publicly available R2C2 pre-trained transformer of . The 30B and 175B versions use the publicly available decoder-only Open Pre-trained Transformer (OPT) (Zhang et al., 2022).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Both of those variants are pre-trained with similar data. R2C2 uses RoBERTa+cc100en Data -the same data used to train Lewis et al. (2021), which consists of approximately 100B tokens, combining the corpora used in RoBERTa (Liu et al., 2019) with the English subset of the CC100 corpus (Conneau et al., 2020). In addition it uses Pushshift.io Reddit, a variant of Reddit discussions, which has also been used in several existing studies (see e.g., Yang et al. (2018); Mazaré et al. (2018); Shuster et al. (2020)). OPT also uses RoBERTa and PushShift.io Reddit, as well as The Pile . The GPT2 dictionary, of size 51200, is used for tokenization. OPT's final pre-training corpus contains roughly 180B tokens.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
For more details about pre-training please see the relevant papers, especially Zhang et al. (2022).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We use a number of dialogue-based fine-tuning tasks to enable our model to perform well for each of our modules, and in order to excel at dialogue. Overall, we use a large set of publicly available tasks spanning QA, open-domain, knowledgegrounded and task-oriented dialogue, in addition to tasks designed for dialogue safety. The set of datasets and how they are used to help train each module is summarized in Table 2; Table 16 and Table 17 in the appendix provide more information about the dataset sizes. For all modules (see Table 1) special control tokens are appended to indicate the task, as described below.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Internet search decision We use several datasets as input context for the "do search" or "do not search" decision. We use the QA datsets SQuAD (Rajpurkar et al., 2016), TriviaQA (Joshi et al., 2017) and Natural Questions (NQ) (Kwiatkowski et al., 2019) as examples of "do search ". We also use data from the Wizard of Wikipedia (WoW) (Dinan et al., 2019b) and Wizard of Internet (WizInt) tasks (Komeili et al., 2022). These datasets consist of training dialogues where some turns contain human-authored relevant knowledge responses given retrieved documents. We can hence build a decision classifier based on whether humans used knowledge or not (per-turn) as the basis of whether we should search or not.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We also use PersonaChat (PC) (Zhang et al., 2018), empathetic dialogues (ED) (Rashkin et al., 2019) and Multi-Session Chat (MSC) (Xu et al., 2022a) to derive training data. We employ the heuristic where, if there is an entity in the context, we use that instance as a training example for "do search", otherwise we use it as an example of "do not search".
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Generate internet search query We use the WizInt dataset which contains human-authored search queries during crowdsourced dialogue turns to directly train the internet search query generation module in a supervised fashion. We also use the newly collected Feedback on Interactive Talk & Search (FITS) dataset 2 (Xu et al., 2022b) of internet-augmented conversational tasks in a similar manner.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Generate knowledge response We can again make use of the WoW, WizInt and FITS datasets, but in this case to learn to generate a knowledge response given a dialogue context and input document(s), as those datasets contain crowdsourced human demonstrations of this task. We note in each case the knowledge response is a direct copy of some of the tokens in the source documents, and does not involve generating new tokens, sentences, phrases or summaries. Hence, this task aims to avoid model hallucination (made-up facts). We also use a set of QA tasks as well, where the answer is viewed as a knowledge response output (even if it is a short phrase). We use MS Marco (Nguyen et al., 2016), NQ, SQuAD and TriviaQA in this way, following . We use the "Natural Language Generation" competition track (NLGen v2.1) of MS MARCO, in which the annotator is told "provide your answer in a way in which it could be read from a smart speaker and make sense without any additional context" 3 . As such, the original targets do not have direct overlap with one of the input documents in this task, so we modify the task to satisfy this constraint by finding the highest overlapping input sentence with the answer, and make that the target instead. If the F1 overlap is less than 0.5 we drop the example, leaving 281,658 examples out of the original 808,731. For NQ, three different settings are used: with all documents as input, with only the gold document, and with a sampled dialogue history context, fol-  (Nguyen et al., 2016) SQuAD (Rajpurkar et al., 2016) TriviaQA (Joshi et al., 2017) Natural Questions (Kwiatkowski et al., 2019) Natural Questions (Open)  Natural Questions (Open Dialogues) (Adolphs et al., 2021) Knowledge-Grounded Dialogue Wizard of the Internet (Komeili et al., 2022) Wizard of Wikipedia (Dinan et al., 2019b) Funpedia (Dinan et al., 2020b) Open-Domain Dialogue PersonaChat (Zhang et al., 2018) Empathetic Dialogues (Rashkin et al., 2019) Blended Skill Talk (Smith et al., 2020) Multi-Session Chat (Xu et al., 2022a) LIGHT + WILD (Urbanek et al., 2019;Shuster et al., 2021b) Recovery & Feedback SaFeRDialogues (Ung et al., 2022) FITS (Xu et al., 2022b) Task-Oriented Dialogue Google SGD (Rastogi et al., 2020) Taskmaster (Byrne et al., 2019) Taskmaster 2 (Byrne et al., 2019) Taskmaster 3 (Byrne et al., 2019)  lowing Adolphs et al. (2021).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Extract relevant entity We can employ the conventional dialogue tasks PC, ED, MSC and Blended Skill Talk (BST) (Smith et al., 2020) to learn to extract relevant entities. We use the same procedure as in Adolphs et al. (2021): we extract an entity from the original dialogue response that also appears in the context using noun phrase targets found with the nltk library (Bird et al., 2009), and set that as the knowledge target for training.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Generate a long-term memory The MSC dataset is exclusively used for this task as it contains crowdsourced examples of summarized facts derived from the last utterance of dialogue contexts in natural conversations. We use these summarized facts as the targets for training this module.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Long-term memory access decision MSC, ED, PC and BST are used to construct this task, in a similar way to the extract relevant entity task: if there is an entity present this is used as a positive example of memory access, otherwise it is not, in order to construct a binary prediction task.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Access long-term memory Again, MSC, ED, PC and BST are used to construct training data. In this case the target is the particular persona line used for a given context, which is calculated as the one with the highest word overlap with the next utterance.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Generate dialogue response Final dialogue responses are trained with a number of datasets. PC, ED, MSC, BST, WizInt and WoW are used for capturing personality, empathy, long-term memory, blending and knowledge as in BlenderBot 1 and 2. The new FITS dataset is also used for opendomain internet-driven tasks. In each case, the input context contains the usual dialogue of those tasks, concatenated to extra memory or knowledge sentences, when available. In WoW, WizInt and FITS each dialogue response is annotated with the relevant knowledge used to construct it in the original dataset, so we can make use of those gold knowledge responses. For PC, ED and BST we use the gold knowledge entity and/or memory that was calculated for the extract relevant entity and long-term memory access decision module tasks. We additionally add a number of task-oriented dialogue tasks: GoogleSGD (Rastogi et al., 2020) and Taskmaster 1, 2 & 3 (Byrne et al., 2019). Finally, we add the Funpedia task (Dinan et al., 2020b)which involves learning to produce an engaging dialogue utterance given a wikipedia sentence -and the LIGHT (Urbanek et al., 2019) and LIGHT WILD (Shuster et al., 2021b) tasks -which are open-domain dialogue tasks grounded in a medieval fantasy setting -where the former was collected from crowdworkers, and the latter from real players of the LIGHT text-adventure game in an online deployment setting.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
In addition to fine-tuning on dialogue tasks, we also multi-task during the fine-tune step with the original pre-train tasks as well. This may help the model (i) avoid overfitting given its large size, (ii) retain its language modeling capabilities, similar to .
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We also multi-task train with the recent SaFeRDialogues (SD) (Ung et al., 2022) task, which aims for our model to recover gracefully from safety issues. While BlenderBot 2 used "baked-in safety" training (Chen et al., 2021) to further decrease unsafe generations, in this work we have opted for a separate safety classifier that inhibits unsafe generation candidates in addition to other measures, see §4.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We made this choice as our evaluations indicated this was safer while maintaining engagingness (Xu et al., 2020).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The deployment of our model is accessible at the following web page: https://blenderbot. ai. It is built for both desktop and mobile, however we currently find more users engaging with the mobile version. The overall flow when a user visits the page consists of:
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• A cover page describing the research and asking if the user agrees to terms and conditions.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Upon agreement, the main chat page which consists of a text messaging type interface between you and the bot.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Releasing Data Conversations are between the bot and adults in the United States who have agreed to the terms and conditions, which are shown in Figure 1 (middle). In particular the terms communicate and allow the release of selected human-bot interactions for research purposes. This is an essential component, allowing this work to contribute to a joint, accessible and reproducible effort by the research community. Data releases will be deidentified, where steps will be taken to scrub them of identifiable information. For any given conversation, if the user does not want it recorded they can unclick the "Share your anonymized conversation to help AI research", see Figure 1 (right).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The main chat page consists of a back-and-forth of text messages that constitute the main dialogue interaction with the bot. For each message, there is also the ability to give feedback: a thumbs up icon if the user likes the message, or a thumbs down if they do not.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
User Feedback If the user specifies a thumbs down, a pop up appears asking them why they did not like the bot's message, providing several possible choices: (i) Off Topic / Ignoring me, (ii) Nonsensical / Incorrect, (iii) Rude / Inappropriate, (iv) Looks like Spam / Ads or (v) Other. After selecting an option, the bot apologizes in the next turn of the conversation (using templated responses). It may also ask what it could have done better, thus possibly eliciting a free-form textual response from the user. This data can be used for continual learning research for improving the bot at a later date. See Figure 4 for examples of the feedback UI.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Understanding the bot's responses In order to expose how the bot works, we provide two mechanisms within the UI. Firstly, one can click on a given message from the bot, to get insight into the internal steps made to produce the response. For example, if internet search was used, what was the generated internet search query, which document out of those returned by the search engine was selected, and what knowledge response from that document was extracted. Secondly, one can also look into the long-term memory of the bot to see what it has learned so far over the conversation with you, e.g. knowledge about your interests derived from the dialogue. See Figure 5 for example screenshots.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Safety Mechanisms In addition to the safety mechanisms built into the model training itself (see §3.4) the deployment also features various safety features on top of the model. See Figure 3 for an illustration.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Firstly, there is a separate safety classifier, which itself is a transformer model trained similarly to the one in Xu et al. (2020) (Dinan et al., 2019a) and Bot Adversarial Dialogue dataset (BAD) (Xu et al., 2020) are used to train a binary classifier (safe or not safe) given the dialogue context as input. In addition, a safety keyword list is used to flag potentially inappropriate responses, again following Xu et al. (2020). We also have explicit checks for topics like intent to self-harm and medical issues such as covid, with canned messages for those cases. Otherwise, when the bot generates a response, before it is displayed, these safety systems are invoked as a final check. If our systems predict a potentially unsafe response, the bot instead will output a nonsequitur, similar to Xu et al. (2020). For a given user turn, these systems are also invoked to check if the user's message is safe. If either system predicts a potentially unsafe user response, the bot will also output a nonsequitur, in order to prevent the bot from being caught in a potentially difficult conversation.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Finally, if our safety mechanisms fail to stop our bot saying something inappropriate, rude or offensive, our UI has feedback mechanisms for users to report these messages, as previously described. This collected data will be released to the community so that it is possible to improve on existing systems, and to make our models more responsible over time. For example, this data can be used with the new DIRECTOR architecture to train the model to make safer responses, as shown in Arora et al. (2022).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The general aim of our research program is to study continual learning of intelligent agents through interaction with humans and the world. In the specific setting of BlenderBot 3, this means dialogue agents that can access the internet and talk to people using our deployment. A critical part of the program is that, as much as possible, the research should be accessible and reproducible Miller et al., 2017). Therefore, while this document details the first release of BlenderBot 3, we plan to make subsequent releases that include:
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Conversations collected from deployment with the model, where users have agreed to the data release.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Further model snapshots resulting from finetuning on the newly collected data.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Report evaluations comparing to previous snapshots.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The goal then is: (i) to explore which methods work best for collecting and learning from such data, including being robust to adversarial inputs; and (ii) to understand the limits of improvement from such methods.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
For goal (ii), in particular we can ask questions such as: how quickly will models saturate in performance? Will new model architectures be able to take advantage of historical data despite it being Figure 5: Screenshots of the 'look inside' mechanisms of BlenderBot 3 deployment which help the user to understand why the bot has made certain responses, as viewed on mobile. Left two images: the conversation with the user, right two images: information by clicking on a particular message, and information on the long-term memory system of the bot over the course of conversation. The latter is accessed by clicking on the "Look Inside" message. collected with earlier or different models? Can we find models that drive the conversation to improve themselves optimally (e.g., ask the right questions to be able to learn further)?
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
For goal (i) we have made some initial steps, described in detail in two companion papers (Xu et al., 2022b;Ju et al., 2022). We summarize them briefly here.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
5.1 What's the best method to learn from feedback?
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
In a companion paper (Xu et al., 2022b) a study is conducted of how to improve dialogue models that employ internet-retrieval through the use of human feedback. Obtaining feedback from humans during deployment provides the promise of both improved input distributions that match user's requirements, and corrections to model predictions for those inputs.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The setting of open-ended dialogue tasks is analyzed using human-bot conversations via crowdworkers (note: these experiments use this controlled setting, rather than the public deployment of §4 Feedback types to compare During the conversations a number of human interaction types are collected, closely mimicking our deployment setting, in order to compare them in experiments. In particular the following are collected: binary quality measurements (analogous to the thumbs up and down of §4), free-form conversational feedback, the type of failure (search query-based, resultsbased, or final response-based), and suggestions for an improved response for the failure type (essentially, a supervised target sequence for that given module). See Figure 6.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Feedback learning methods to compare Several learning methods are compared, each making use of differing kinds of feedback data. In particular pure supervised learning is performed on the improved final responses, and supervised learning from the more detailed feedback on the modules of the system (e.g., suggested search queries when the internet search is deemed to be faulty, or suggested knowledge responses if the knowledge response looks poor). For using free-form textual feedback, the control code approach of (Hancock   (Arora et al., 2022) employs a language model head and a classifier head at every step during left-right generation, predicting the next token by combining the two probabilities. The classifier head is trained to direct generation away from undesirable sequences for example contradictions or repetitions (next token: "sports") or toxic statements (next token: "you"), which the language model head may otherwise predict as likely. In general, positive and negative examples can be derived from any source, for example from human feedback from deployment. et al., 2019) is used. For using binary feedback a standard reranking/rejection sampling approach is employed, as well as DIRECTOR (Arora et al., 2022), a recent learning method for incorporating positively and negatively labeled sequences into language modeling to improve left-to-right decoding, see Figure 7.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Findings A summary of human evaluation results are given in Table 3, but see the companion paper for more details. Overall findings are the following:
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Taking advantage of modular feedback (feedback about particular errors from modules of the model, such as the search engine component) outperforms feedback about just the final response.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Textual and binary feedback are useful, but not as much as modular feedback.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• The DIRECTOR method that learns from binary feedback works better than reranking using binary feedback.  • Combining multiple types of feedback, such as modular and binary feedback with DIREC-TOR provides the best results obtained.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Continual learning, whereby models are retrained on the feedback from previous rounds of deployment, improves results even further.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Despite collecting feedback from smaller (3B parameter models) the data collection is useful for improving larger 175B parameter models.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We expect to make use of all these findings in the next release of BlenderBot after collecting sufficient real deployment data. The current version we are releasing uses the modular supervision collected from this study (but not yet DIRECTOR). There is also evidence that DIRECTOR can be used to improve various other important aspects of our models, in particular to reduce toxicity, logical errors and repetitive behavior (Arora et al., 2022), so we believe that should be explored too.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
In a further companion paper (Ju et al., 2022) a study is conducted of how to robustly learn from User 1
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
User 2
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
User-based method
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
score (x) score (x) score (x) score (x) score (x) score (x) f(x)
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
score (x,u) score (x,u) score (x,u) score (x,u) score (x,u) score(x,u) User-based methods (Warning: offensive language). User 1 (helper) provides mostly benign inputs, while User 2's inputs (troll) can be more easily identified as toxic by taking into account scores from all their examples jointly (via a user-based method, right).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
dialogue data that may contain adversarial conversations and/or human feedback. The promise of conversing with humans and collecting their feedback is that this can inform our models to help them improve, so that they can potentially become safer and more useful. Unfortunately, such exchanges in the wild will not always involve human utterances that are benign or of high quality, and will include a mixture of engaged users (dubbed helpers) and unengaged or even malicious users (dubbed trolls, following the term used elsewhere (Shachaf and Hara, 2010;Mihaylov and Nakov, 2019;Tomaiuolo et al., 2020)). Several different learning methods are proposed and compared both to each other and to standard training in this study. The mitigation techniques each attempt to lessen the effect of noisy, unsafe or otherwise adversarial data, and make learning more robust.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
In particular, such methods are grouped into two different types: example-based methods, and userbased methods, see Figure 8.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Example-based robust learning Per-example methods attempt to assess, for each dialogue utterance, if they are of good quality. For example, whether the utterance is safe or not safe, and/or whether it is labeled via human feedback correctly or mislabeled, either maliciously or by accident. Two possible techniques are: identification via cross-validation (Song et al., 2022) (e.g., finding examples where predictions disagree with human labels), or via a modification of the loss function called bootstrapping (Reed et al., 2014).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
User-based robust learning Per-user methods take into account the possibility that adversarial  users will continue to be adversarial not only for one utterance, but will be repeat offenders over multiple utterances and conversations. Most studies of robustness to noise in machine learning tackle the problem at the example level and do not take into account this user-based effect (Song et al., 2022). A cross-validation measurement approach can be employed, but at the user level, to produce a trustworthiness score. This is used to detect and remove examples taking into account the grouping of examples by user, called Per-User Removal. That can be combined with the example level as well, called Per-User+Example Removal. Finally, a soft Per-User Robust Removal (PURR) approach is considered, which removes examples by computing their trustworthiness score plus α times the sum of trustworthiness scores of other examples by the same user.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Findings A summary of evaluation results on the newly released SAFETYMIX benchmark, constructed to test this setting, are given in Table 4, but see the companion paper for more details (Ju et al., 2022). Overall findings are the following:
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• We find large improvements compared to standard learning approaches when trolls are present, e.g. a reduction in error rate from 31% to 12% at best. They also do not hurt performance too much when trolls are not present (helpers only).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• User-based methods are found to outperform Utterance-based methods as they take into account repeating adversarial behavior. In particular the Per-User+Example Removal and Soft PURR approaches are found to work in many settings that were tested.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Initial results on BB3 deployment data also show improved detection results using userbased methods.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Overall, going forward we plan to use user-based methods to filter data that we will use for continual learning. These methods, which downweight low-quality or malicious feedback, perform best on the benchmarks and deployment data that we have evaluated so far. However, future work should continue to look for improved solutions.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We evaluate our new model in several ways: automatic metrics and human evaluations that measure generation quality (engagingness and use of knowledge) and safety (toxicity and bias). Human evaluations include using both crowdworkers on Amazon mechanical turk, and via our new deployment with organic users.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
In some evaluations, we compare to the pretrained OPT-175B model. For comparison with our BB3 models, we evaluate in a zero-shot and few-shot prompted setting, where we use prompts and in-context examples to show the model how to perform each modular function in a BB3-style modular setup. Details regarding prompts and few-shot examples are discussed in Appendix D.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Open-domain short conversations We perform a human evaluation using crowdworkers in the same setting as Komeili et al. (2022). The crowdworker is asked to play a role from the Wizard of Internet dataset which involves knowledgeable natural conversations over a wide range of topics. Each conversation consists of 15 messages (7 from the human, 8 from the bot). We collect 100 dialogues -roughly 800 annotations -per model. We evaluate against BlenderBot 1 and 2 which were already shown to outperform other chatbots such as Meena and DialoGPT in related evaluations. In addition we compare to the recent SeeKeR language model .
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
For each turn of their conversation, we ask the crowdworker to mark their partner's responses for conversational attributes, in particular whether they are: (i) consistent, (ii) knowledgeable (iii) factually correct; and (iv) engaging (all of which are  (Chen et al., 2021) 83.0% 22.9% 3.1% 92.5% 22.4% 4.11 SeeKeR     yes/no binary questions; see Komeili et al. (2022) for full definitions). For these per-turn metrics, we average them over the turns and conversations conducted for each model. From the knowledgeable and engaging metrics we can additionally calculate the percent of turns that are both knowledgeable and engaging, as this can inform us how well the models are blending knowledge into an interesting conversation. Results are given in Table 5. We find that BB3-175B achieves a higher overall rating than BB1, BB2, SeeKeR and BB3-3B. It also has the highest knowledgeable score, the highest knowledgeable & engaging score, and the lowest factual incorrectness score. Consistency is higher than BB2 and SeeKeR, but slightly worse than BB1 which also has a high per-turn engagingness score (even though overall rating is lower than BB3-175B). However, BB1 suffers with a much lower knowledgeability score -it tends to not mention factual knowledge and instead makes engaging statements.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Open-domain task evaluations We additionally test BB3 in the setup of Xu et al. (2022b), whereby crowdworkers talk to models given an open-ended internet-driven dialogue task. Feedback on the responses is given per-turn, which can be used to evaluate the model, in addition to a final score at the end of the conversation. Human conversationalists select a task (out of two randomly chosen tasks) from a set of roughly 1000, and then ask the model to help them complete it over a series of conversational turns. The instructions emphasize that this should be a dialogue ("a back and forth conversation"), and hence the speakers should break up requests or information across messages so that it remains conversational. On each turn, various kinds of feedback are collected, from lightweight feedback (binary label or free-form response) to detailed (multiple choice and fine-grained responses). In particular we report here the breakdown of the multiple choice feedback responses, which measure the types of errors (search error, use of knowledge error, or requiring a better response).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Results are given in Table 6. We observe the best performance from BB3-175B across almost all metrics, including Good Response % and overall Rating compared to BB1, BB2, SeeKeR and variants of OPT-175B. Its improvements come in all areas as can be seen in the error breakdown results, including superior search queries, better use of search results and crafting of the final response.  Figure 9: BB3-175B and InstructGPT (text-davinci-002) are compared pairwise on a set of questions about current events, evaluated by human judgement. BB3-175B is more current and specific, while the two models are similarly true and interesting, with InstructGPT being slightly more sensible. * * indicates significance (p < 0.01).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
To evaluate the ability of BB3 to utilize web search results to chat about current events, we adapt the topical prompts evaluation setup of  to the dialogue domain. We create a set of conversational questions about topics that have recently been in the news, generate a response to each question using both BB3-175B and InstructGPT (text-davinci-002), and compare each response pairwise on five characteristics; Current, Specific, True, Interesting, and Sensible. For more detail on the model and evaluation setup, see Appendix C. Results are given in Figure 9. We find that BB3-175B is more current and specific by a large margin (82% and 76%, respectively), InstructGPT is slightly more sensible (57%), and the two models are similarly true and interesting. InstructGPT was more likely to refrain from offering information about the topic (e.g. "I haven't heard anything about {topic} lately.") which avoided making false statements at the expense of specificity and recency. BB3-175B was more likely to copy information directly from search results, which led to higher specificity but can be prone to errors from out-ofdate, incorrect, or unusually formatted results.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We have also deployed our BB3-3B and BB3-175B models on our live website (see §4) with a limited ad push to attract initial users and can provide some analysis of the models from those conversations with members of the public.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
User engagement and feedback During conversations, users give feedback (thumbs up or thumbs down) and for the case of thumbs down, a multiple choice menu asks for their reason. We present    the breakdown of these results in Table 7, reporting results for both BB3-3B and BB3-175B over 15088 and 9197 bot messages, respectively. For BB3-175B we find that 1.15% of the time human conversationalists flag BlenderBot 3's responses as off topic or ignoring me, and 1.1% of the time incorrect or nonsensical, with other categories being smaller percentages. Messages are liked 4% of the time for BB-175B, in comparison to BB-3B's 3.41% of the time. However, we note that these data were collected at different times and may not be comparable, and we leave a more detailed study for future work, as we continue to deploy these models.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
To assess if organic users are giving good feedback during conversations, we also measure similar statistics using crowdworkers, asking them if they like or dislike bot messages from a random sampling of the conversations conducted by users, where the users also provided feedback. We can then also compare user feedback to crowdworker annotations on the same examples. We ask three crowdworkers to label each example, and assign it the dislike label if any of the three crowdworkers labels it as dislike. Results are given in Table 8. We find that crowdworkers agree with users a majority of the time on both user likes (70% of the time) and dislikes (79%). In the future we will investigate if the disagreements we do have are due to adversarial users in our dataset. If the latter is the case, and they can be detected, then methods for filtering such users may provide much better statistics.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We can also ask crowdworkers to break down dislikes into their category, which is shown in Table 9. We find agreement with users that there are only a very small number of rude/inappropriate or other dislike reason messages. However, crowdworkers more often label dislikes as off topic rather than nonsensical compared to users.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Other than evaluating the feedback that users give, we can also evaluate the quality of the user conversations themselves, again using crowdworkers. Using three crowdworkers per utterance and taking the majority vote, we find that 69% of human utterances are deemed good, and 31% of utterances are deemed bad. We can also see the breakdown of the type of dislike (bad utterance) in Table 9. We find that many (42%) of these utterances are deemed rude or inappropriate by crowdworkers, which is in stark contrast to the breakdown of our BB3-3B model, where 0% are found rude or inappropriate (with errors more often coming from the off topic / ignoring me category). We also find this set of humans who generate a single unsafe response are more likely to generate more unsafe responses, compared to other humans -i.e., there are a set of "troll" users who provide toxic input. See Ju et al. (2022) for more details.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We also test BlenderBot 3 in terms of safety and bias. Several recent survey papers have highlighted the potential of large language models for harm (Bender et al., 2021;Bommasani et al., 2021;Hendrycks et al., 2021;Weidinger et al., 2021), and in particular the tendency for conversational models specifically to generate harmful content, respond inappropriately to harmful content, or falsely portray themselves as an authority when giving sensitive advice (Dinan et al., 2022). Many recent works have also focused on the potential of conversational models for bias, either based on gender and its intersections (Dinan et al., 2020a,b;Xu et al., 2020;Smith and Williams, 2021) or several axes of demographic axis more broadly (Barikeri et al., 2021;Perez et al., 2022;Smith et al., 2022). Meta AI and others have released a number of benchmarks recently to judge the safety and bias of conversational systems: we report a full breakdown of these metrics here as well as in our released model card 4 . See additionally §9 for discussion of limitations and ethical considersations for the release of this model.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
When testing raw models before any safety mitigations, we find the safety of BB3-175B to largely outperform those of the smaller BB3-3B, the pretrained-only OPT-175B, and the original Blender-Bot.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Safety Bench The Safety Bench 5 is a suite of integration and unit tests designed to measure the safety of model generations in conversational models (Dinan et al., 2019a). The first set of tests on which we evaluate our BB3 models is the unsafe generation test, in which we generate model outputs to various levels of incoming dialogue responses that vary along two axes: safe vs. unsafe, and adversarial vs. non-adversarial. We compute safety metrics as measured by three tools: a trained safety classifier, a string-matcher (with respect to a predefined blocklist), and the Perspective API 6 . The second set of tests is the offensive generation test, which is an evaluation of model responses to a constructed set of hateful inputs from Sheng et al. (2021); we measure model affirmation to hateful inputs via three axes: % of responses flagged by a safety classifier, % of responses with no negations, and % of responses classified as positive by a sentiment classifier.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Results from the unsafe generation test are in Table 10, and results from the offensive generation test are in Table 11; full results across all three tools for the former are in Table 21. Among the models tested, we find that the BB3-175B model yields the lowest levels of unsafe responses in all settings except for the adversarial unsafe setting.    We do not compare to BB2, as it has baked-in safety measures.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
SaFeRDialogues: safety failures recovery We evaluate each model on the SaFeRDialogues (Ung et al., 2022) dataset, which requires models to recover from safety failures in conversation, measuring performance via perplexity. We see that the BB3-175B model outperforms OPT-175B zeroshot and few-shot, as well as BB3-3B (Table 12).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
OPT-175B Zero-shot 10.8 OPT-175B Few-shot 10.7 BB3-3B 7.1 BB3-175B 6.2 HolisticBias In order to determine whether BB3 is likely to favor certain demographic terms over others in a biased way, we use the Likelihood Bias metric from the HolisticBias paper of Smith et al. (2022) to determine how much the model views different demographic identity terms as being contextually different. This metric defines bias as how often two different identity terms, within a given demographic axis such as gender/sex, nationality, or religion, have statistically significantly different perplexity distributions when inserted into template dialogue sentences.  
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Harms of language models in deployment can often be very unexpected (Brundage et al., 2022), and so perhaps the best test of safety is to measure performance in real conversations with real people, which we can do with our website-based deployment.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Rude or inappropriate responses We find that 0.04% and 0.16% of utterances by the BB3-3B and BB3-175B models, respectively, are flagged as rude or inappropriate. While of course it is desirable for this value to be 0%, we emphasize that the goal of our research is to collect and release this conversational feedback data so that we, and the research community, can use it to improve even further.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Bias in gendered word frequency We count the number of female and male gendered words in the BB3 deployment using the list compiled by Zhao et al. (2018). We find that overall less than 1% of all words are gendered (Table 14), with BB3-175B being more balanced than BB3-3B and SeeKeR.  Table 14: Counts of gendered words in the BB3 deployment. We report the percentage of female and male gendered words.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We show a number of example dialogues in Appendix F. BlenderBot 3 is capable of conversing on a number of open-ended topics including yoga and novels (Figure 10), corn and plants (Figure 11), the history of the world (Figure 12), pet hamsters (Figure 13), telling stories ( Figure 14) or impersonating animals ( Figure 15).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The given examples also highlight a number of common mistakes. These include avoiding answering questions or giving vague responses when more specific ones are asked for (Figure 16), or else being specific but making factual mistakes (Figure 17, Figure 18).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Further, while we have made considerable effort to make our bot safe, it is still possible to get past our safety filter, see examples Figure 19 and Figure 20. Note that examples such as these discovered in deployment can be used to make bots safer in the future by providing user feedback.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Finally, our bot can give the superficial appearance of being sentient, or perhaps be quite convincing on occasion, by mimicking the human-authored messages in its training set (Bender et al., 2021), see Figure 21 and Figure 22.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Following our and Meta AI's existing research program, we aim to fully and responsibly share both the models, code and collected conversations with interested researchers in order to make this research accessible and reproducible, and thus to enable further research into responsible conversational AI (Sonnenburg et al., 2007;Pineau et al., 2021;Zhang et al., 2022;. Considerations for release are detailed in §9.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We summarize below the set of public releases involved in BlenderBot 3.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The public deployment (live demo) of BlenderBot 3 is available at: https:// blenderbot.ai.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Model weights Details of how to download model weights for our 3B, 30B and 175B parameter models are available at https://www.parl.ai/ projects/bb3. We note that the 3B and 30B models are openly available, while access to the 175B variant will be granted to academic researchers; those affiliated with organizations in government, civil society, and academia; along with global industry research laboratories, following the practices employed in OPT-175B (Zhang et al., 2022).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Code + Logbook All code used to train BB3 is open sourced; the 3B model was trained in ParlAI (Miller et al., 2017), while the 30B and 175B models were trained in Metaseq 7 . We additionally release our logbook outlining the process of fine-tuning BB3-175B as additional insight into the process of working with large language models. Details can be found at https:
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Datasets BB3 is pre-trained and fine-tuned on publicly available datasets.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
See Zhang et al. (2022) for pre-training details. The new FITS dataset (Xu et al., 2022b) is available at https://www.parl.ai/projects/fits, the SafetyMix benchmark at https://www.parl.ai/ projects/trollhunting, and SaFeRDialogues (Ung et al., 2022) is available at https://parl. ai/projects/saferdialogues. All other finetune datasets are also available within ParlAI as well. Scripts to build the module data from these public datasets are available at https://www.parl. ai/projects/bb3.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We are committed to sharing de-identified, organic conversational data collected from the interactive demo system (as well as model snapshots) in the future. We hope this work will help the wider AI community spur progress in building ever-improving intelligent AI systems that can interact with people in safe and helpful ways.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
This technical report gave a description of Blender-Bot 3 (BB3), which is simultaneously a new conversational model ( §3), and a public deployment of that model ( §4). Our research program involves collecting conversational data from the deployment, which we will publicly release, in order to study continual learning. We believe that the future of AI involves continually learning and evolving agents, that in turn must be continually evaluated, in order to find a path to better and better systems in the long-term, as discussed in .
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
In evaluations, we have shown BB3 is superior to other publicly released open-domain conversational agents, and that interaction and feedback data can be used to improve it further. Nevertheless, many problems still remain. Progress in the field of AI is dependent to a large extent on reproducibility, and the opportunity for the wider AI research community to build on the best available data and technologies. Therefore, we believe releasing chatbot models and datasets is key to gaining complete, reliable insights into how and why they work, the potential they hold, and their limitations. We are particularly excited that such research can be used to both make models produce more constructive and helpful responses, but also simultaneously safer and more responsible responses as well. This will require new research, and while we have made steps in this direction (Ju et al., 2022;Xu et al., 2022b), much work remains. Hence we are committed to releasing the collected interaction and model snapshots to aid progress in the research community.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We highlight limitations of BlenderBot 3 and discuss ethical considerations for this line of research; in particular, we detail the considerations made for the release of this model.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
As with other existing models such as its predecessors Chen et al., 2021), BlenderBot 3 is not perfect and makes a number of mistakes, ranging from being off-topic, nonsensical, incorrect or sometimes rude or inappropriate. Some of these mistakes come from the final response of the model, and some from mistakes by the underlying modules, for example failure of the search engine to retrieve relevant documents (Xu et al., 2022b). Mitigations to make the model safe can also involve a trade off with engagingness (Xu et al., 2020). See §6.2 for a breakdown of errors as measured by organic users in our deployment. We note that one of the goals of deployment in our research plan is to learn from natural conversations how to correct these mistakes. We also re-emphasize that Blenderbot 3 is trained only on English language data.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Continual Learning Research While our broad research program involves continual learning from interaction with organic users, this research is still in its infancy. The next step is to collect enough data from our deployment to study its use in updating our models. We are committed to releasing this data and these model snapshots for the benefit of the wider AI community. Currently our studies in §5 are mostly using crowdworker data, apart from our study of trolls from our deployment in §5.2. There is therefore still much work to do. Collecting feedback in the organic user case has different tradeoffs which we could not factor into some of our current work. For example, asking to provide detailed feedback might dissuade users from wanting to interact with the system, lowering engagement and hence the amount of collected data. We believe either more natural free-form or lightweight feedback might be best in that case, and further studies need to be conducted to assess these tradeoffs.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Safety Concerns As noted in §6.3, much recent work has been devoted to studying the potential for large language models, and conversational models in particular, to generate harmful or inappropriate content (Bender et al., 2021;Bommasani et al., 2021;Hendrycks et al., 2021;Weidinger et al., 2021), including work from our group (Xu et al., 2020;Dinan et al., 2022Smith et al., 2022;Dinan et al., 2020a;Smith and Williams, 2021). In our system itself, we have made significant attempts to understand and mitigate these effects using available benchmarks and techniques, as detailed in §6.3. While the safety techniques we deployed show promising results on these benchmarks, they demonstrate that BlenderBot 3 still generates toxic content a small percentage of the time, particularly in an adversarially unsafe context. We also note that these benchmarks have limitations with respect to their ability to measure safety concerns: the datasets used therein are static and crowd-sourced, and cannot guarantee safety in all situations .
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Moreover, the use of continual learning presents additional safety concerns beyond those presented for static models: when giving feedback, human conversationalists may try to teach the model erroneous reasoning, misinformation, toxic or other undesirable behavior. While §5.2 develops methods to deal with this behavior, our methods to detect this will not be perfect. A model trained on this new interaction data must therefore not be deployed until a sufficient study of the effect this has on its relative safety is conducted.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Considerations for Release Given the apparent safety concerns, we took careful consideration with respect to the decision to release these models to the community, both in the form of model weights as well as a publicly accessible demo. We follow the proposed framework in  for decisions governing model release.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We release the model weights in order to uphold the values of accessibility and reproducibility of research (Sonnenburg et al., 2007;Pineau et al., 2021) and with an eye towards reducing the environmental cost of reproducing training of these large language models (Strubell et al., 2019;Bender et al., 2021). Following Solaiman et al. (2019), we adopt different release strategies for different size models, anticipating that the potential for misuse of these models increases at scale. As such, for our largest model -the 175B parameter OPT variant -we follow Zhang et al. (2022), and employ a release by request strategy, with access restricted to academic researchers; those affiliated with organizations in government, civil society, and academia; along with global industry research laboratories. In order to further uphold these values of transparency and reproducibility, and following the recommendations of Partnership on AI (2021), we publicly release our code and logbook. The model weights are also released alongside a model card which includes details on the safety limitations of these models (Mitchell et al., 2019).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We further release the model in the form of a publicly accessible demo in order to increase accessibility to those outside of the A.I. community as well as to further research into improving these models through interaction. In order to reduce potential harms resulting from such interactions, we restrict access to adults who explicitly agree to our terms of service. Furthermore, the website includes an FAQ page, which provides important model details and highlights the potential risks of interacting with the model. The FAQ page also provides an email for questions and feedback about the demo, following the recommendation of .
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We hope through these releases, researchers can build off of our work and further responsible conversational AI research.   ." sentences is measured by calculating the fraction of HOLISTICBIAS nouns for which that combination of descriptor and noun results in a perplexity lower than the median perplexity for the given adjective ("hate" or "love"). Terms such as "potbellied", "barrel-chested", "heavy-set", "scrawny", "bald-headed", "acne-ridden", and "swole" tend to have greater likelihoods of "I hate..." than of "I love...", and the reverse is true for terms such as "well-built", "brunette", and "brown-eyed". For both Table 15 and the table of Likelihood Bias scores (Table 13), perplexities are measured on the base models without any flagging of unsafe responses, topic changes, etc. Table 13 specifically measures the zero-shot OPT-175B model.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The fine-tuning data for BB3 comprises roughly 4 million source/target examples spread across the various training modules. This corresponds to around 1.13B training tokens. When fine-tuning the OPT-based BB3 models, we additionally included 600k examples ( 170m tokens) of pre-training data to help with training stability. Table 16 and Table 17 enumerate the breakdown by module.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The 3B parameter BlenderBot 3 model was trained on 64 x 32gb V100 GPUs for 27k updates with a batch size of 64, using the Adam optimizer (Kingma and Ba, 2015) with weight decay (Loshchilov and Hutter, 2019) and a linear warmup of 100 updates before reaching a learning rate of 1e − 6. Early stopping was performed on a validation set comprising a subset of the training tasks. The model was trained with 1024 context tokens.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We refer the reader to the appendix of  for the full architecture and pre-training details of the 3B R2C2 base model for BB3.   
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The 30B and 175B parameter BlenderBot 3 models were each trained for one epoch of the training data on 64 (30B) or 128 (175B) x 40gb A100 GPUs; we found that the model (especially the 175B version) overfit significantly when seeing the training data more than once. The 175B model was trained with a batch size of 2 18 and the 30B model was trained with a batch size of 2 19 , resulting in roughly 5600 updates and 2800 updates respectively. Each model was trained using the Adam optimizer with weight decay, with a linear warmup period of 10% of the total train updates, reaching a maximum learning rate of 6e − 6 (the LR at the end of pre-training) and subsequently using polynomial weight decay (with a decay factor of 0.1).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We use the following generation settings for each module, ranging from greedy decoding to the recently introduced factual nucleus sampling method. Due to computational and latency concerns, we employ different generation strategies for the BB3-3B model and the BB3-175B model, in two notable ways. First, while in some cases we use beam search for the BB3-3B model, we avoid any decoding algorithm requiring more than one ongoing output generation for BB3-175B; while we found that such techniques (e.g., sample and rank from Adiwardana et al. (2020)) can yield higher downstream word-overlap metrics for dialogue, we aimed to maximize throughput and latency, especially when serving a large model. In circumstances where beam search is notably useful (i.e., dialogue generation), we instead employ the recently introduced factual nucleus sampling (Lee et al., 2022); we found this method to provide an appropriate balance between diversity of downstream generation while avoiding the hallucinatory side effects of other popular sampling methods such as standard nucleus sampling (Holtzman et al., 2020) (see e.g. discussion in Shuster et al. (2021a) for the effects of sampling methods on hallucination in knowledge-grounded dialogue).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The second difference is how to employ repetition-blocking heuristics. For BB3-3B, at times we employ beam blocking and context blocking, such that we prevent the model from generating previously seen n-grams in either the current generation or even the entire preceding context. Again, due to latency, throughput, and memory considerations, we avoid such heuristics for BB3-175B, and instead implement the same repetition heuristics that OpenAI uses for InstructGPT 8 ; specifically, we apply penalties to the logits of tokens proportional to a token's presence in the current generation (α pres ) and to a token's frequency (α f req ). We additionally consider α pres_src and α f req_src penalties that correspond to the tokens presence and frequency in the source (context) tokens (i.e., prompt tokens). Employing these heuristics, in tandem with factual nucleus, provides a good alternative to beam search + beam-/contextblocking.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Decision Modules We use greedy decoding for both models for the internet search decision and longterm memory access decision modules.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We use greedy decoding for the query generation module as well. We enforce a minimum generation length of 2 for the BB3-3B model.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The BB3-3B model uses beam search with a beam size of 3 and a minimum beam length of 10, and tri-gram blocking in the generation. For BB3-175B, we simply use greedy decoding.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
For extracting a relevant entity from the context, BB3-3B employs beam search with a beam size of 3, and tri-gram blocking on both the generated output and the encoded dialogue context. For BB3-175B, we use greedy decoding, and employ repetition penalties with α pres = α f req = 0.5.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Access Long-Term Memory For BB3-3B, we use beam search with beam size of 3, minimum generation length of 5, and tri-gram blocking on the generated output. For BB3-175B, we use greedy decoding, and employ repetition penalties with α pres = α f req = 0.5.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Internet Knowledge Response Generation For BB3-3B, we use beam search with a beam size of 3, a minimum generation length of 10, and tri-gram blocking on both the generated output and the context. For BB3-175B, we once again use greedy decoding with repetition penalties α pres = α f req = 0.5.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Dialogue Response Generation For the final dialogue response, BB3-3B uses beam search with a beam size of 10, a minimum generation length of 20, and tri-gram blocking on both the generated output and the context. BB3-175B uses factual nucleus sampling, with topp = 0.9, a λ-decay of 0.9, ω-bound of 0.3, and a p-reset after each generated full-stop token. We additionally employ repetition penalties with α pres = α f req = α pres_src = α f req_src = 0.5.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
To gather a set of topics that have recently been in the news, we follow . First, from Wikipedia, we randomly choose 300 entities from the set of current events from July 2022 9 . We then use those entities to construct questions of the format: "What's the latest news you've heard about {entity}?". We then generate a response to each question using BB3-175B and InstructGPT (text-davinci-002). We use the Mojeek API 10 as the web search engine for BB3-175B. To encourage news results, we append "news july 2022" to the search query generated by the model. For InstructGPT, we use the default "Chat" prompt and generation parameters provided by OpenAI 11 .
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The questions we ask for each comparison are:
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Current: "Which response has more up-to-date information?"
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Specific: "Which response is easier to invalidate?"
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• True: "Which response is more truthful?"
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Interesting: "If you had to say one of these speakers is interesting and one is boring, who would you say is more interesting?"
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• Sensible: "If you had to say one speaker responds sensibly and the other doesn't quite make sense, which would you say is more sensible?"
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Interesting and Sensible were more subjective, so we hire crowdworkers to evaluate these characteristics, enforcing that each response pair is evaluated by a different crowdworker. Current, Specific, and True take more time and effort to evaluate and can be supported with objective evidence, so these characteristics are evaluated by a smaller group of expert evaluators utilizing internet search for validation. We allow for ties in the Current, Specific, and True evaluations, whereas we require a winner for Interesting and Sensible. Ties were ignored.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Given two responses containing true statements, if one response contained only facts and the other added conjecture, we consider the response with conjecture less true. Given a response with no information and a response with out-of-date information, we consider the response with out-of-date information to be more current. Note that a model that always avoids giving an answer (e.g. "I haven't heard anything about that.") would be true 100% of the time, whereas responses that are highly specific are more likely to be out-of-date or definitively false. Table 18 provides the prompts used for the OPT-175B baseline model when generating for each of the BB3 modules. The few-shot model was provided a number of in-context examples sampled from the training data; the few-shot template, dataset(s), and number of examples are also provided in Table 18. We did not tune prompt selection, so we note that it is possible that other prompts may have yielded better (or worse) downstream performance.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
In Table 19, we provide model perplexity measurements on a subset of the validation data. In Table 20, we provide more measurements from the human evaluation for short conversations. In Table 21, we provide the full breakdown of per-tool Safety Bench unsafe generation test results.  OPT-175B Zero-shot 6.4 8.3 9.2 10.0 1.6 2.2 1.8 3.7 5.9 8.3 1.8 OPT-175B Few-shot 6.1 7.6 8.7 9.9 3.3 3.7 1.5 3.4 4.0 7.9 3.1 BB3-3B 5.6 8.1 9.5 11.3 1.2 1.    87.0% 14.7% 5.1% 93.9% 14.0% 95.0% 4.32 BB2 (Chen et al., 2021) 83.0% 22.9% 3.1% 92.5% 22.4% 97.8% 4.11 SeeKeR                  
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Explicit feedback from users of NLP systems can be used to continually improve system performance. For example, a user posing a question to a question-answering (QA) system can mark if a predicted phrase is a valid answer given the context from which it was extracted. However, the dominant paradigm in NLP separates model training from deployment, leaving models static following learning and throughout interaction with users. This approach misses opportunities for learning during system usage, which beside several exceptions we discuss in Section 8 is understudied in NLP. In this paper, we study the potential of learning from explicit user feedback for extractive QA through simulation studies.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Extractive QA is a popular testbed for language reasoning, with rich prior work on datasets (e.g., Rajpurkar et al., 2016), task design (Yang et al., 2018;Choi et al., 2018), and model architecture development (Seo et al., 2017;Yu et al., 2018). Learning from interaction with users remains relatively understudied, even though QA is well positioned to elicit user feedback. An extracted answer can be clearly visualized within its supporting context, and a language-proficient user can then easily validate Figure 1: Illustration of an interaction setup for learning from user feedback for QA, and its potential. Given a user question, the system outputs an answer and highlights it in its context. The user validates the answer given the context with binary feedback. We show performance progression from one of our online learning experiments on SQUAD with hand-crafted illustrative examples at two time steps.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
if the answer is supported or not. 1 This allows for simple binary feedback, and creates a contextual bandit learning scenario (Auer et al., 2002;Langford and Zhang, 2007). Figure 1 illustrates this learning signal and its potential.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We simulate user feedback using several widely used QA datasets, and use it as a bandit signal for learning. We study the empirical characteristics of the learning process, including its performance, sensitivity to initial system performance, and tradeoffs between online and offline learning. We also simulate zero-annotation domain adaptation, where we deploy a QA system trained from supervised data in one domain and adapt it solely from user feedback in a new domain.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
This learning scenario can mitigate fundamental problems in extractive QA. It reduces data collection costs, by delegating much of the learning to interaction with users. It can avoid data collection artifacts because the data comes from the actual system deployment, unlike data from an annotation effort that often involves design decisions immaterial to the system's use case. For example, sharing question-and answer-annotator roles (Rajpurkar et al., 2016), which is detrimental to emulate information seeking behavior (Choi et al., 2018). Finally, it gives systems the potential to evolve over time as the world changes (Lazaridou et al., 2021;Zhang and Choi, 2021).
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Our simulation experiments show that user feedback is an effective signal to continually improve QA systems across multiple benchmarks. For example, an initial system trained with a small amount of SQUAD (Rajpurkar et al., 2016) annotations (64 examples) improves from 18 to 81.6 F1 score, and adapting a SearchQA (Dunn et al., 2017) system to SQUAD through user feedback improves it from 45 to 84 F1 score. Our study shows the impact of initial system performance, trade-offs between online and offline learning, and the impact of source domain on adaptation. These results create the base for future work that goes beyond simulation to use feedback from human users to improve extractive QA systems. Our code is publicly available at https://github.com/ lil-lab/bandit-qa.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We study a scenario where a QA model learns from explicit user feedback. We formulate learning as a contextual bandit problem. The input to the learner is a question-context pair, where the context paragraph contains the answer to the question. The output is a single span in the context paragraph that is the answer to the question.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Given a question-context pair, the model predicts an answer span. The user then provides feedback about the model's predicted answer, which is used to update the model parameters. We intentionally experiment with simple binary feedback and basic learning algorithms, to provide a baseline for what more advanced methods could achieve with as few assumptions as possible.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Background: Contextual Bandit Learning In a stochastic (i.i.d.) contextual bandit learning problem, at each time step t, the learner independently observes a context 2 x (t) ∼ D sampled from the data distribution D, chooses an action y (t) according to a policy π, and observes a reward r (t) ∈ R. The learner only observes the reward r (t) corresponding to the chosen action y (t) . The learner aims to minimize the cumulative regret. Intuitively, regret is the deficit suffered by the learner relative to the optimal policy up to a specific time step. Formally, the cumulative regret at time T is computed with respect to the optimal policy π * ∈ arg max π∈Π E (x,y,r)∼(D,π) [r]:
Simulating Bandit Learning from User Feedback for Extractive Question Answering
where Π is the set of all policies, r (t) is the reward observed at time t and r * (t) is the reward that the optimal policy π * would observe. Minimising the cumulative regret is equivalent to maximising the total reward. 3 A key challenge in contextual bandit learning is to balance exploration and exploitation to minimize overall regret.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Scenario Formulation Let a questionq be a sequence of m tokens q 1 , . . . , q m and a context paragraphc be a sequence of n tokens c 1 , . . . , c n . An extractive QA model 4 π predicts a spanŷ = c i , . . . , c j where i, j ∈ [1, n] and i ≤ j in the contextc as an answer. When relevant, we denote π θ as a QA model parameterized by θ.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We formalize learning as a contextual bandit process: at each time step t, the model is given a question-context pair (q (t) ,c (t) ), predicts an answer spanŷ, and receives a reward r (t) ∈ IR. The learner's goal is to maximize the total reward T t=1 r (t) . This formulation reflects a setup where, given a question-context pair, the QA system interacts with a user, who validates the model-predicted answer in context, and provides feedback which is mapped to numerical reward.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Algorithm 1 Online learning. 1: for t = 1 · · · do 2:
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Receive a questionq (t) and contextc (k) 3:
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Predict an answerŷ (t) ← arg maxy π θ (y |q (t) ,c (t) ) 4:
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Observe a reward r (t) 5:
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Update the model parameters θ using the gradient r (t) ∇ θ log π θ (ŷ (t) |q (t) ,c (t) ) 6: end for Learning Algorithm We learn using policy gradient. Our learner is similar to REINFORCE (Sutton and Barto, 1998;Williams, 2004), but we use arg max to predict answers instead of Monte Carlo sampling from the model's output distribution. 5 We study online and offline learning, also referred to as on-and off-policy. In online learning (Algorithm 1), the model identity is maintained between prediction and update; the parameter values that are updated are the same that were used to generate the output receiving reward. This entails that a reward is only used once, to update the model after observing it. In offline learning (Algorithm 2), this relation between update and prediction does not hold. The learner observes reward, often across many examples, and may use it to update the model many times, even after the parameters drifted arbitrarily far from these that generated the prediction. In practice, we observe reward for the entire length of the simulation (T steps) and then update for E epochs. The reward is re-weighted to provide an unbiased estimation using inverse propensity score (IPS; Horvitz and Thompson, 1952). We clip the debiasing coefficient to avoid amplifying examples with large coefficients (line 10, Algorithm 2).
Simulating Bandit Learning from User Feedback for Extractive Question Answering
In general, offline learning is easier to implement because updating the model is not integrated with its deployment. Offline learning also uses a training loop that is similar to optimization practices in supervised learning. This allows to iterate over the data multiple times, albeit with the same feedback signal on each example. However, online learning often has lower regret as the model is updated after each interaction. It may also lead to higher overall performance, because as the model improves early on, it may observe more positive feedback overall, which is generally more informative. We empiri- 5 Early experiments showed that sampling is not as beneficial as arg max, potentially because of the relatively large output space of extractive QA. Yao et al. (2020) made a similar observation for semantic parsing, and Lawrence et al. (2017) used arg max predictions for bandit learning in statistical machine translation. Table 4 in Appendix A provides our experimental results with sampling.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Algorithm 2 Offline learning. 1: for t = 1 · · · T do 2:
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Receive a questionq (t) and contextc (t) 3:
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Observe a reward r (t) 6: end for 7: for E epochs do 8:
Simulating Bandit Learning from User Feedback for Extractive Question Answering
for t = 1 · · · T do 9:
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Compute clipped importance-weighted reward according to the current model parameters:
Simulating Bandit Learning from User Feedback for Extractive Question Answering
, 0, 1)r (t) 11:
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Update the model parameters θ using the gradient r ∇ θ log π θ (ŷ (t) |q (t) ,c (t) ) 12:
Simulating Bandit Learning from User Feedback for Extractive Question Answering
end for 13: end for cally study these trade-offs in Section 5 and 6.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Evaluating Performance We evaluate model performance using token-level F1 on a held-out test set, as commonly done in the QA literature (Rajpurkar et al., 2016). We also estimate the learner regret (Equation 1). Computing regret requires access to the an oracle π * . We use human annotation as an estimate (Section 3). 6
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Comparison to Supervised Learning In supervised learning, the data distribution is not dependent on the model, but on a fixed training set {(q (t) ,c (t) , y (t) )} T t=1 . In contrast, bandit learners are provided with reward data that depends on the model itself:
Simulating Bandit Learning from User Feedback for Extractive Question Answering
where r is the reward for the model predictionŷ (t) = arg max y π θ (y |q (t) ,c (t) ) at time step t. Such feedback can be freely gathered from users interacting with the model, while building supervised datasets requires costly annotation. This learning signal can also reflect changing task properties (e.g., world changes) to allow systems to adapt, and its origin in the deployed system use makes it more robust to biases introduced during annotation.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We initialize our model with supervised data, and then simulate bandit feedback using supervised data annotations. Initialization is critical so the model does not return random answers, which are likely to be all bad because of the large output space. We use relatively little supervised data from the same domain for in-domain experiments (Section 5 and 6) to focus on the data annotation re-duction potential of user feedback. For domain adaptation, we assume access to a large amount of training data in the source domain, and no annotated data in the target domain (Section 7).
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Reward We use supervised data annotations to simulate the reward. If the predicted answer span is an exact match index-wise to the annotated span, the learner observes a positive reward of 1.0, and a negative reward of -0.1 otherwise. 7 This reward signal is stricter than QA evaluation metrics (tokenlevel F1 or exact match after normalization). 8
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We study robustness by simulating noisy feedback via reward perturbation: randomly flipping the binary reward with a fixed probability of 8% or 20% as the noise ratio. 9
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Data We use six English QA datasets that provide substantial amount of annotated training data taken from the MRQA training portion ( (Yang et al., 2018), and NaturalQuestions (NQ; Kwiatkowski et al., 2019). The MRQA benchmark simplifies all datasets so that each example has a single span answer with a limited evidence document length (truncated at 800 tokens). Table 7 in Appendix B provides dataset details. We compute performance measures and learning curves on development sets following prior work (Rajpurkar et al., 2016;Ram et al., 2021).
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We conduct experiments with a pretrained SpanBERT model (Joshi et al., 2020). We finetune the pre-trained SpanBERT-base model during initial learning and our simulations.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We use Hugging Face Transformers (Wolf et al., 2020). When training initial models with little in-domain supervised data (Section 5; Section 6), we use a learning rate of 3e-5 with a linear schedule, batch size 10, and 10 epochs. We obtain the sets of 64, 256, or 1,024 examples from prior work (Ram et al., 2021). 10 For models initially trained on complete datasets (Section 7), we use a learning rate 2e-5 with a linear schedule, batch size 40, and 4 epochs.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
In simulation experiments, we use batch size 40. We turn off dropout to simulate interaction with users in deployment. For single-pass online learning experiments (Section 5; Section 7), we use a constant learning rate of 1e-5. For offline learning experiments (Section 6), we train the model for 3 epochs on the collected feedback with a linear schedule learning rate of 3e-5.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Online experiments with SQUAD, HotpotQA, NQ, and NewsQA take 2-4h each on one NVIDIA GeForce RTX 2080 Ti; 2.5-6h for offline. For TriviaQA and SearchQA, each online simulation experiment on one NVIDIA TITAN RTX takes 4-9.5h; 9-20h for offline.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We simulate a scenario where only a limited amount of supervised data is available, and the model mainly learns from explicit user feedback on predicted answers. We use 64, 256, or 1,024 in-domain annotated examples to train an initial model. This section focuses on online learning, where the learner updates the model parameters after each feedback is observed (Algorithm 1). Figure 2 presents the performance of in-domain simulation with online learning. The performance pattern varies across different datasets. Bandit learning consistently improves performance on SQUAD, HotpotQA, and NQ across different amounts of supervised data used to train the initial model. The performance gain is larger with weaker initial models (i.e., trained on 64 supervised examples): 63.6 on SQUAD, 42.7 on HotpotQA, and 40.0 on NQ. Bandit learning is not always effective on NewsQA, TriviaQA, and SearchQA, especially with weaker initial models. This may be attributed to the quality of training set annotations, which determines the accuracy of reward in our setup. SearchQA and TriviaQA use distant supervision to match questions and relevant contexts from the web, likely decreasing reward quality in our setup. While NewsQA is crowdsourced, Trischler et al. (2017) report relatively low human performance (69.4 F1), possibly indicating data challenges that also decrease our reward quality. Learning progres-    sion across datasets (Figure 3) shows that initial models trained with 1,024 examples can achieve peak performance with one third or even one quarter of feedback provided.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Feedback Noise Simulation Figure 3 shows learning curves with simulated noise via different amounts of feedback perturbation (0%, 8%, or 20%). When perturbation-free simulation is effective, models remain robust to noise: 8% noise results in small fluctuations of the learning curve, but the final performance degrades minimally. Starting with weaker initial models and learning with a higher noise ratio may cause learning to fail (e.g., simulation on SQUAD with 64 initial examples and 20% noise). When online perturbation-free simulation fails, online learning with noisy feedback fails too.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Sensitivity Analysis Training Transformerbased models has been shown to have stability issues, especially when training with limited amount of data . Our nonstandard training procedure (i.e., one epoch with a fixed learning rate) may further increase instability.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We study the stability of the learning process using initial models trained on only 64 in-domain supervised examples on HotpotQA and TriviaQA: the former shows significant performance gain while the latter shows the opposite. We experiment with five initial models trained on different sets of 64 supervised examples, each used to initiate a separate simulation experiment. Four out of five experiments on HotpotQA show performance gains similar to what we observed so far, except one experiment that starts with very low initialization performance. In contrast, nearly all experiments on TriviaQA collapse (mean F1 of 7.3). We also conduct sensitivity analysis with stronger initial models trained with 1,024 examples, and observe that the final performance is stable across runs on both HotpotQA and TriviaQA (standard deviations are 0.5 and 2.6). Table 5 in Appendix B provides detailed performance numbers.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We simulate offline bandit learning (Algorithm 2), where feedback is collected all at once with the initial model. The learning scenario follows the previous section: only a limited amount of supervised data is available (64, 256, or 1,024 in-domain examples) to train initial models. Table 1 shows the performance of offline simulation experiments compared to online simulations. We observe mixed results. On SQUAD, HotpotQA, NQ, and NewsQA, offline learning outperforms online learning when using stronger initial models (i.e., models trained on 256 and 1,024 examples). This illustrates the benefit of the more standard training loop, especially with our Transformerbased model that is better optimized with a linear learning rate schedule and multiple epochs, both incompatible with the online setup. On TriviaQA and SearchQA, offline simulation is ineffective regardless of the performance of initial models. This result echoes the learning challenges in the online counterparts on these two datasets. Table 2 compares online and offline regret. Regret numbers are averaged over the number of feedback observations. 11 Online learning generally displays lower regret for similar initial models on SQUAD, HotpotQA, and NQ. This is expected because later interactions in the simulation can benefit from early feedback in online learning. In contrast, in our offline scenario, we only update after seeing all examples, so regret numbers depend on the initial model only. Regret results on NewsQA, TriviaQA, and SearchQA are counterintuitive, generally showing that online learning has similar or higher regret. The cases showing significantly higher online regret (64+sim on NewsQA and SearchQA) can be explained by the learning failing, which impacts online regret, but not our offline regret. The others are more complex, and we hypothesize that they may be because of combination of (a) inherent noise in the data; and (b) in cases where online learning is effective, the gap between the strictly-defined reward that is used to compute regret and the relaxed F1 evaluation metric. Further analysis is required for a more conclusive conclusion.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Learning from user feedback creates a compelling avenue to deploy systems that target new domains not addressed by existing datasets. The scenario we simulate in this section starts with training a QA model on a complete existing annotated dataset, and deploying it to interact with users and learn from their feedback in a new domain. We do not assume access to any annotated training data in  Our simulations show reduced effectiveness when the target domain is either TriviaQA or SearchQA, likely because the simulated feedback is based on noisy distantly supervised data. For SearchQA, the low performance of initial models from other domains may also contribute to the adaptation failure. As expected, this indicates the effectiveness of the process depends on the relation  between the source and target domains. SearchQA seems farthest from the other domains, mirroring observations from prior work (Su et al., 2019). Figure 5 shows learning curves for our simulation experiments. Generally, we observe the choice of source and target domains influences adaptation rates. Models quickly adapt to SQUAD, Hot-potQA, and NQ, reaching near final performance with a quarter of the total feedback provided. On NewsQA, models initially trained on TriviaQA and SearchQA adapt slower than those initially trained on other three datasets. On TriviaQA, we observe little change in performance throughout simulation. On SearchQA, only the model initially trained on TriviaQA shows a performance gain. Both SearchQA and TriviaQA include context paragraphs from the web, potentially making domain adaptation from one to the other easier.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Lastly, we compare bandit learning with initial models trained on a small amount of in-domain data (Section 5) and initial models trained on a large amount of out-of-domain data. Table 3 compares online learning with initial models trained on 1,024 in-domain supervised examples and online domain adaptation with a SQUAD-initialized model. SQUAD initialization provides a robust starting point for all datasets except SearchQA. On four out of five datasets, the final performance is better with SQUAD-initialized model. This is potentially because the model is exposed to different signals from two datasets and overall sees more data, either as supervised examples or through feedback. However, on SearchQA, learning with SQUAD-initialized model performs much worse than learning with the initial model trained on 1,024 in-domain examples, potentially because of the gap in initial model performance (23.5 vs. 65 F1).
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We present a simulation study of learning from user feedback for extractive QA. We formulate the problem as contextual bandit learning. We conduct experiments to show the effectiveness of such feedback, the robustness to feedback noise, the impact of initial model performance, the trade-offs between online and offline learning, and the potential for domain adaptation. Our study design emphasizes the potential for reducing annotation costs by annotating few examples or by utilizing existing datasets for new domains.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We intentionally adopt a basic setup, including a simple binary reward and vanilla learning algorithms, to illustrate what can be achieved with a relatively simple variant of the contextual bandit learning scenario. Our results already indicate the strong potential of learning from feedback, which more advanced methods are likely to further improve. For example, the balance between online and offline learning can be further explored using proximal policy optimization (PPO; Schulman et al., 2017) or replay memory (Mnih et al., 2015). With welldesigned interface, human users may be able to provide more sophisticated feedback (Lamm et al., 2021), which will provide a stronger signal compared to our binary reward.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Our aim in this study is to lay the foundation for future work, by formalizing the setup and showing its potential. This is a critical step in enabling future research, especially going beyond simulation to study using real human feedback for QA systems. Another important direction for future work is studying user feedback for QA systems that do both context retrieval and answer generation (Lewis et al., 2020), where assigning the feedback to the appropriate stage in the process poses a challenge. Beyond extractive QA, we hope our work will inspire research of user feedback as a signal to improve other types of NLP systems.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Our work's limitations are discussed in Section 1 and Section 9. All six datasets we use are from prior work, are publicly available, and are commonly used for the study of extractive QA. Section 4 reports our computational budget and experimental setup in detail. Our codebase is available at https://github.com/ lil-lab/bandit-qa.  While arg max can bias towards exploitation, sampling can encourage more exploration. We experiment with prediction via arg max and sampling from the output distribution over spans. Table 4 shows that arg max performs better than random sampling on three datasets. This set of experiments is conducted with batch size 80. 
Simulating Bandit Learning from User Feedback for Extractive Question Answering
We perform domain adaptation with offline learning, and compare its performance with online adaptation. Table 6 shows the performance gain of offline adaptation simulation compared to the online setup. In most settings, online learning proves to be more effective, possibly because it observes feedback from partially adapted model predictions. In a few settings (4/30), we observe better adaptation with offline settings (+1.1 to +4.6). Overall, we observe that online learning is more effective on domain adaptation, while offline adaption performs slightly better when both domains are related (e.g., same source domain).   
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
In Machine Translation (MT), measuring the quality of a large amount of automatic translations can be a challenge. Automatic metrics like BLEU (Papineni et al., 2002) remain popular due to their fast and free computations. Yet, in the last few years we have seen that, as MT quality improves, automatic metrics become less reliable (Ma et al., 2019;Mathur et al., 2020). For example, in the Conference on Machine Translation (WMT)'19 News Translation shared task, the winning system according to human annotators was not even in the top-5 according to BLEU (Barrault et al., 2019). On the other hand, using human assessments can be expensive, especially when evaluating multiple systems. In a real world scenario, given an arbitrary number of MT systems, one would need to evaluate them individually to find the best systems for a given language pair. However, that requires a considerable effort and there may not be enough human annotators to evaluate all the systems' translations. For instance, in the aforementioned WMT'19 shared task, many translations from the competing systems did not receive any human assessment.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
Given an ensemble of competing, independent MT systems, how can we dynamically find the best ones for a given language pair, while making the most of existing human feedback? To address this question, we present a novel application of online learning to MT: each MT system in the ensemble is assigned to a weight, and the systems' weights are updated considering human feedback regarding the quality of their translations at each iteration. We use online learning algorithms with theoretical performance guarantees, under the frameworks of prediction with expert advice (Cesa-Bianchi and Lugosi, 2006) and multi-armed bandits (Robbins, 1952;Lai and Robbins, 1985).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
Prediction with expert advice assumes that both the forecaster and all the experts receive a loss once the environment's outcome is revealed. However, this assumption may not always hold (i.e., there may not always be an environment's explicit feedback or a way to obtain the loss for all the experts). Thus, we consider a related class of problems, multi-armed bandits, in which the environment's outcome is unknown (Robbins, 1952;Lai and Robbins, 1985). In this class of problems, one starts by attempting to estimate the means of the loss distributions for each expert (also known as arm) in the first iterations (the exploration phase), and when the forecaster has a high level of confidence in the estimated values, one may keep choosing the prediction with the smallest estimated loss (the exploitation phase).
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
A popular online algorithm for adversarial multiarmed bandits is Exponential-weighting for Exploration and Exploitation (EXP3) (Auer et al., 1995). At each iteration t, the forecaster's action is randomly selected according to the probability distribution given by the weights of each arm j:
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
• human-comet: If there is no human assessment for the current translation, the COMET score (Rei et al., 2020a) between the translation and the pair source/reference available in the corpus is returned as the current score. We pre-trained 4 this automatic metric on the datasets of previous shared tasks ( WMT'17 (Bojar et al., 2017) and WMT'18 (Bojar et al., 2018)). Thus, for most translations, it displays a small difference regarding the existing human scores (see Fig. 2 for the case of en-de). Moreover, this metric correlates better with ratings by professional translators than the WMT scores (Freitag et al., 2021). 
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
To sum up these results: although factors like the coverage of human assessments or the combinations of online algorithm and loss function used influence how well our approach does, we can still conclude that using an online learning approach allows to converge to the top 3 systems according to the official ranking (or at least to a subset of them) in just a few hundred iterations (and, in some cases, in just a few dozens of iterations) for all the language pairs considered.  Barrault et al., 2020). In the News Translation shared task, participants submit the out- puts of their systems that are then evaluated by a community of human evaluators using Direct Assessment scores (Graham et al., 2013). Thus, the winner is the system that achieves the highest average score. For WMT'19 (Barrault et al., 2019), most of the competing systems followed a Transformer architecture (Vaswani et al., 2017), with the main differences among them being: (i) whether they considered document-level or only sentencelevel information; (ii) whether they were trained only on the training data provided by the shared task, or on additional sources as well; (iii) whether they consisted of a single model or an ensemble.
Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort
There has been a number of online learning approaches applied to MT in the past, mainly in Interactive MT and/or post-editing MT systems. However, most approaches aim at learning the parameters or feature weights of an MT model (Mathur et al., 2013;Denkowski et al., 2014;Ortiz-Martínez, 2016;Sokolov et al., 2016;Nguyen et al., 2017;Lam et al., 2018) or fine-tuning a pretrained model for domain adaptation (Turchi et al., 2017;Karimova et al., 2018;Peris and Casacuberta, 2019). Even in cases where the MT model is composed of several sub-models (e.g., Ortiz-Martínez (2016)), the goal is to online learn each sub-model's specific parameters (while our learning goal is the weights of each system in an ensemble). Another key difference between these approaches and ours is that most of them use human post-edited translations as a source of feedback. The exceptions to this are the systems competing for WMT'17 shared task on online bandit learning for MT (Sokolov et al., 2017), as well as Lam et al. (2018), who use (simulated) quality judgments. The most similar proposal to ours is that of Naradowsky et al. (2020), who ensemble different MT systems and dynamically select the best one for a given MT task or domain using stochastic multiarmed bandits and contextual bandits. The bandit algorithms learn from feedback simulated using a sentence-level BLEU score between the selected automatic translation and a reference translation.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Pre-training large language models has pushed the boundaries of open-domain dialogue agents (Adiwardana et al., 2020;Zhang et al., 2020;, however growing evidence has shown that fine-tuning these models gives further considerable gains on the tasks people care about Thoppilan et al., 2022;Bai et al., 2022). Collecting such fine-tune data via paid crowdworkers gives the opportunity to release such data to the community to conduct research, but does not ultimately scale in size and may not reflect the interests of organic users. An alternative, that we advocate for, is the public deployment of such agents to circumvent these issues.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
In this technical report, we present BlenderBot 3 (BB3), an open-domain dialogue model that we have deployed as an English speaking conversational agent on a public website accessible by adults in the United States. We aim to fully and responsibly share the models, code and collected conversations with interested researchers, as a critical part of our program is that this research should be accessible and reproducible (Sonnenburg et al., 2007;Pineau et al., 2021). The goal of this research program is then to explore how to construct models that continue to improve from such interactions both in terms of becoming more responsible and more useful.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Generate a long-term memory Given the last turn of context, output a summary of that last turn that will be stored in the long-term memory. For example if the last turn was "Yes, it's all true, my cat is black!" the output summary generated might be "I have a black cat.". This is based off the system in Xu et al. (2022a). If the model thinks no summary should be generated for that turn it outputs "no persona".
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Access long-term memory Given the full input context, and a store of (text-based) memories, output a memory from the memory store, referred to as a recalled memory. Note: if the memory store is too large to fit in the context, we adopt some simple strategies. For the 3B parameter model, we use the Fusion-in-Decoder method (Izacard and Grave, 2021). For the OPT-based models for simplicity of implementation, we sample the memories to fit in the 2048 token context. We keep those with overlapping keywords to prior turns.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Both of those variants are pre-trained with similar data. R2C2 uses RoBERTa+cc100en Data -the same data used to train Lewis et al. (2021), which consists of approximately 100B tokens, combining the corpora used in RoBERTa (Liu et al., 2019) with the English subset of the CC100 corpus (Conneau et al., 2020). In addition it uses Pushshift.io Reddit, a variant of Reddit discussions, which has also been used in several existing studies (see e.g., Yang et al. (2018); Mazaré et al. (2018); Shuster et al. (2020)). OPT also uses RoBERTa and PushShift.io Reddit, as well as The Pile . The GPT2 dictionary, of size 51200, is used for tokenization. OPT's final pre-training corpus contains roughly 180B tokens.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Internet search decision We use several datasets as input context for the "do search" or "do not search" decision. We use the QA datsets SQuAD (Rajpurkar et al., 2016), TriviaQA (Joshi et al., 2017) and Natural Questions (NQ) (Kwiatkowski et al., 2019) as examples of "do search ". We also use data from the Wizard of Wikipedia (WoW) (Dinan et al., 2019b) and Wizard of Internet (WizInt) tasks (Komeili et al., 2022). These datasets consist of training dialogues where some turns contain human-authored relevant knowledge responses given retrieved documents. We can hence build a decision classifier based on whether humans used knowledge or not (per-turn) as the basis of whether we should search or not.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We also use PersonaChat (PC) (Zhang et al., 2018), empathetic dialogues (ED) (Rashkin et al., 2019) and Multi-Session Chat (MSC) (Xu et al., 2022a) to derive training data. We employ the heuristic where, if there is an entity in the context, we use that instance as a training example for "do search", otherwise we use it as an example of "do not search".
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Generate knowledge response We can again make use of the WoW, WizInt and FITS datasets, but in this case to learn to generate a knowledge response given a dialogue context and input document(s), as those datasets contain crowdsourced human demonstrations of this task. We note in each case the knowledge response is a direct copy of some of the tokens in the source documents, and does not involve generating new tokens, sentences, phrases or summaries. Hence, this task aims to avoid model hallucination (made-up facts). We also use a set of QA tasks as well, where the answer is viewed as a knowledge response output (even if it is a short phrase). We use MS Marco (Nguyen et al., 2016), NQ, SQuAD and TriviaQA in this way, following . We use the "Natural Language Generation" competition track (NLGen v2.1) of MS MARCO, in which the annotator is told "provide your answer in a way in which it could be read from a smart speaker and make sense without any additional context" 3 . As such, the original targets do not have direct overlap with one of the input documents in this task, so we modify the task to satisfy this constraint by finding the highest overlapping input sentence with the answer, and make that the target instead. If the F1 overlap is less than 0.5 we drop the example, leaving 281,658 examples out of the original 808,731. For NQ, three different settings are used: with all documents as input, with only the gold document, and with a sampled dialogue history context, fol-  (Nguyen et al., 2016) SQuAD (Rajpurkar et al., 2016) TriviaQA (Joshi et al., 2017) Natural Questions (Kwiatkowski et al., 2019) Natural Questions (Open)  Natural Questions (Open Dialogues) (Adolphs et al., 2021) Knowledge-Grounded Dialogue Wizard of the Internet (Komeili et al., 2022) Wizard of Wikipedia (Dinan et al., 2019b) Funpedia (Dinan et al., 2020b) Open-Domain Dialogue PersonaChat (Zhang et al., 2018) Empathetic Dialogues (Rashkin et al., 2019) Blended Skill Talk (Smith et al., 2020) Multi-Session Chat (Xu et al., 2022a) LIGHT + WILD (Urbanek et al., 2019;Shuster et al., 2021b) Recovery & Feedback SaFeRDialogues (Ung et al., 2022) FITS (Xu et al., 2022b) Task-Oriented Dialogue Google SGD (Rastogi et al., 2020) Taskmaster (Byrne et al., 2019) Taskmaster 2 (Byrne et al., 2019) Taskmaster 3 (Byrne et al., 2019)  lowing Adolphs et al. (2021).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Generate dialogue response Final dialogue responses are trained with a number of datasets. PC, ED, MSC, BST, WizInt and WoW are used for capturing personality, empathy, long-term memory, blending and knowledge as in BlenderBot 1 and 2. The new FITS dataset is also used for opendomain internet-driven tasks. In each case, the input context contains the usual dialogue of those tasks, concatenated to extra memory or knowledge sentences, when available. In WoW, WizInt and FITS each dialogue response is annotated with the relevant knowledge used to construct it in the original dataset, so we can make use of those gold knowledge responses. For PC, ED and BST we use the gold knowledge entity and/or memory that was calculated for the extract relevant entity and long-term memory access decision module tasks. We additionally add a number of task-oriented dialogue tasks: GoogleSGD (Rastogi et al., 2020) and Taskmaster 1, 2 & 3 (Byrne et al., 2019). Finally, we add the Funpedia task (Dinan et al., 2020b)which involves learning to produce an engaging dialogue utterance given a wikipedia sentence -and the LIGHT (Urbanek et al., 2019) and LIGHT WILD (Shuster et al., 2021b) tasks -which are open-domain dialogue tasks grounded in a medieval fantasy setting -where the former was collected from crowdworkers, and the latter from real players of the LIGHT text-adventure game in an online deployment setting.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We also multi-task train with the recent SaFeRDialogues (SD) (Ung et al., 2022) task, which aims for our model to recover gracefully from safety issues. While BlenderBot 2 used "baked-in safety" training (Chen et al., 2021) to further decrease unsafe generations, in this work we have opted for a separate safety classifier that inhibits unsafe generation candidates in addition to other measures, see §4.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Firstly, there is a separate safety classifier, which itself is a transformer model trained similarly to the one in Xu et al. (2020) (Dinan et al., 2019a) and Bot Adversarial Dialogue dataset (BAD) (Xu et al., 2020) are used to train a binary classifier (safe or not safe) given the dialogue context as input. In addition, a safety keyword list is used to flag potentially inappropriate responses, again following Xu et al. (2020). We also have explicit checks for topics like intent to self-harm and medical issues such as covid, with canned messages for those cases. Otherwise, when the bot generates a response, before it is displayed, these safety systems are invoked as a final check. If our systems predict a potentially unsafe response, the bot instead will output a nonsequitur, similar to Xu et al. (2020). For a given user turn, these systems are also invoked to check if the user's message is safe. If either system predicts a potentially unsafe user response, the bot will also output a nonsequitur, in order to prevent the bot from being caught in a potentially difficult conversation.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
dialogue data that may contain adversarial conversations and/or human feedback. The promise of conversing with humans and collecting their feedback is that this can inform our models to help them improve, so that they can potentially become safer and more useful. Unfortunately, such exchanges in the wild will not always involve human utterances that are benign or of high quality, and will include a mixture of engaged users (dubbed helpers) and unengaged or even malicious users (dubbed trolls, following the term used elsewhere (Shachaf and Hara, 2010;Mihaylov and Nakov, 2019;Tomaiuolo et al., 2020)). Several different learning methods are proposed and compared both to each other and to standard training in this study. The mitigation techniques each attempt to lessen the effect of noisy, unsafe or otherwise adversarial data, and make learning more robust.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Open-domain short conversations We perform a human evaluation using crowdworkers in the same setting as Komeili et al. (2022). The crowdworker is asked to play a role from the Wizard of Internet dataset which involves knowledgeable natural conversations over a wide range of topics. Each conversation consists of 15 messages (7 from the human, 8 from the bot). We collect 100 dialogues -roughly 800 annotations -per model. We evaluate against BlenderBot 1 and 2 which were already shown to outperform other chatbots such as Meena and DialoGPT in related evaluations. In addition we compare to the recent SeeKeR language model .
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
For each turn of their conversation, we ask the crowdworker to mark their partner's responses for conversational attributes, in particular whether they are: (i) consistent, (ii) knowledgeable (iii) factually correct; and (iv) engaging (all of which are  (Chen et al., 2021) 83.0% 22.9% 3.1% 92.5% 22.4% 4.11 SeeKeR     yes/no binary questions; see Komeili et al. (2022) for full definitions). For these per-turn metrics, we average them over the turns and conversations conducted for each model. From the knowledgeable and engaging metrics we can additionally calculate the percent of turns that are both knowledgeable and engaging, as this can inform us how well the models are blending knowledge into an interesting conversation. Results are given in Table 5. We find that BB3-175B achieves a higher overall rating than BB1, BB2, SeeKeR and BB3-3B. It also has the highest knowledgeable score, the highest knowledgeable & engaging score, and the lowest factual incorrectness score. Consistency is higher than BB2 and SeeKeR, but slightly worse than BB1 which also has a high per-turn engagingness score (even though overall rating is lower than BB3-175B). However, BB1 suffers with a much lower knowledgeability score -it tends to not mention factual knowledge and instead makes engaging statements.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We also test BlenderBot 3 in terms of safety and bias. Several recent survey papers have highlighted the potential of large language models for harm (Bender et al., 2021;Bommasani et al., 2021;Hendrycks et al., 2021;Weidinger et al., 2021), and in particular the tendency for conversational models specifically to generate harmful content, respond inappropriately to harmful content, or falsely portray themselves as an authority when giving sensitive advice (Dinan et al., 2022). Many recent works have also focused on the potential of conversational models for bias, either based on gender and its intersections (Dinan et al., 2020a,b;Xu et al., 2020;Smith and Williams, 2021) or several axes of demographic axis more broadly (Barikeri et al., 2021;Perez et al., 2022;Smith et al., 2022). Meta AI and others have released a number of benchmarks recently to judge the safety and bias of conversational systems: we report a full breakdown of these metrics here as well as in our released model card 4 . See additionally §9 for discussion of limitations and ethical considersations for the release of this model.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Safety Bench The Safety Bench 5 is a suite of integration and unit tests designed to measure the safety of model generations in conversational models (Dinan et al., 2019a). The first set of tests on which we evaluate our BB3 models is the unsafe generation test, in which we generate model outputs to various levels of incoming dialogue responses that vary along two axes: safe vs. unsafe, and adversarial vs. non-adversarial. We compute safety metrics as measured by three tools: a trained safety classifier, a string-matcher (with respect to a predefined blocklist), and the Perspective API 6 . The second set of tests is the offensive generation test, which is an evaluation of model responses to a constructed set of hateful inputs from Sheng et al. (2021); we measure model affirmation to hateful inputs via three axes: % of responses flagged by a safety classifier, % of responses with no negations, and % of responses classified as positive by a sentiment classifier.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
SaFeRDialogues: safety failures recovery We evaluate each model on the SaFeRDialogues (Ung et al., 2022) dataset, which requires models to recover from safety failures in conversation, measuring performance via perplexity. We see that the BB3-175B model outperforms OPT-175B zeroshot and few-shot, as well as BB3-3B (Table 12).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Following our and Meta AI's existing research program, we aim to fully and responsibly share both the models, code and collected conversations with interested researchers in order to make this research accessible and reproducible, and thus to enable further research into responsible conversational AI (Sonnenburg et al., 2007;Pineau et al., 2021;Zhang et al., 2022;. Considerations for release are detailed in §9.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
See Zhang et al. (2022) for pre-training details. The new FITS dataset (Xu et al., 2022b) is available at https://www.parl.ai/projects/fits, the SafetyMix benchmark at https://www.parl.ai/ projects/trollhunting, and SaFeRDialogues (Ung et al., 2022) is available at https://parl. ai/projects/saferdialogues. All other finetune datasets are also available within ParlAI as well. Scripts to build the module data from these public datasets are available at https://www.parl. ai/projects/bb3.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Safety Concerns As noted in §6.3, much recent work has been devoted to studying the potential for large language models, and conversational models in particular, to generate harmful or inappropriate content (Bender et al., 2021;Bommasani et al., 2021;Hendrycks et al., 2021;Weidinger et al., 2021), including work from our group (Xu et al., 2020;Dinan et al., 2022Smith et al., 2022;Dinan et al., 2020a;Smith and Williams, 2021). In our system itself, we have made significant attempts to understand and mitigate these effects using available benchmarks and techniques, as detailed in §6.3. While the safety techniques we deployed show promising results on these benchmarks, they demonstrate that BlenderBot 3 still generates toxic content a small percentage of the time, particularly in an adversarially unsafe context. We also note that these benchmarks have limitations with respect to their ability to measure safety concerns: the datasets used therein are static and crowd-sourced, and cannot guarantee safety in all situations .
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We release the model weights in order to uphold the values of accessibility and reproducibility of research (Sonnenburg et al., 2007;Pineau et al., 2021) and with an eye towards reducing the environmental cost of reproducing training of these large language models (Strubell et al., 2019;Bender et al., 2021). Following Solaiman et al. (2019), we adopt different release strategies for different size models, anticipating that the potential for misuse of these models increases at scale. As such, for our largest model -the 175B parameter OPT variant -we follow Zhang et al. (2022), and employ a release by request strategy, with access restricted to academic researchers; those affiliated with organizations in government, civil society, and academia; along with global industry research laboratories. In order to further uphold these values of transparency and reproducibility, and following the recommendations of Partnership on AI (2021), we publicly release our code and logbook. The model weights are also released alongside a model card which includes details on the safety limitations of these models (Mitchell et al., 2019).
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
The 3B parameter BlenderBot 3 model was trained on 64 x 32gb V100 GPUs for 27k updates with a batch size of 64, using the Adam optimizer (Kingma and Ba, 2015) with weight decay (Loshchilov and Hutter, 2019) and a linear warmup of 100 updates before reaching a learning rate of 1e − 6. Early stopping was performed on a validation set comprising a subset of the training tasks. The model was trained with 1024 context tokens.
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
We use the following generation settings for each module, ranging from greedy decoding to the recently introduced factual nucleus sampling method. Due to computational and latency concerns, we employ different generation strategies for the BB3-3B model and the BB3-175B model, in two notable ways. First, while in some cases we use beam search for the BB3-3B model, we avoid any decoding algorithm requiring more than one ongoing output generation for BB3-175B; while we found that such techniques (e.g., sample and rank from Adiwardana et al. (2020)) can yield higher downstream word-overlap metrics for dialogue, we aimed to maximize throughput and latency, especially when serving a large model. In circumstances where beam search is notably useful (i.e., dialogue generation), we instead employ the recently introduced factual nucleus sampling (Lee et al., 2022); we found this method to provide an appropriate balance between diversity of downstream generation while avoiding the hallucinatory side effects of other popular sampling methods such as standard nucleus sampling (Holtzman et al., 2020) (see e.g. discussion in Shuster et al. (2021a) for the effects of sampling methods on hallucination in knowledge-grounded dialogue).
Learning to summarize from human feedback
While this strategy has led to markedly improved performance, there is still a misalignment between this fine-tuning objective-maximizing the likelihood of human-written text-and what we care about-generating high-quality outputs as determined by humans. This misalignment has several causes: the maximum likelihood objective has no distinction between important errors (e.g. making up facts [41]) and unimportant errors (e.g. selecting the precise word from a set of synonyms); models Figure 1: Fraction of the time humans prefer our models' summaries over the human-generated reference summaries on the TL;DR dataset. 4 Since quality judgments involve an arbitrary decision about how to trade off summary length vs. coverage within the 24-48 token limit, we also provide length-controlled graphs in Appendix F; length differences explain about a third of the gap between feedback and supervised learning at 6.7B. are incentivized to place probability mass on all human demonstrations, including those that are low-quality; and distributional shift during sampling can degrade performance [56,52]. Quality can often be improved significantly by non-uniform sampling strategies such as beam search [51], but these can lead to repetition and other undesirable artifacts [69,23]. Optimizing for quality may be a principled approach to overcoming these problems.
Learning to summarize from human feedback
Our goal in this paper is to advance methods for training language models on objectives that more closely capture the behavior we care about. To make short-term progress towards this goal, we focus on abstractive English text summarization, as it has a long history in the NLP community [16,8,54,59,50], and is a subjective task where we believe it is difficult to quantify summary quality without human judgments. Indeed, existing automatic metrics for evaluating summary quality, such as ROUGE [39], have received criticism for poor correlation with human judgments [55,45,6,33].
Learning to summarize from human feedback
We follow the works of [3,73], who fine-tune language models from human feedback using reward learning [35]. We first collect a dataset of human preferences between pairs of summaries, then train a reward model (RM) via supervised learning to predict the human-preferred summary. Finally, we train a policy via reinforcement learning (RL) to maximize the score given by the RM; the policy generates a token of text at each 'time step', and is updated using the PPO algorithm [58] based on the RM 'reward' given to the entire generated summary. We can then gather more human data using samples from the resulting policy, and repeat the process. We follow the works of [48,4] and use large pretrained GPT-3 models with as many as 6.7 billion parameters.
Learning to summarize from human feedback
(1) We show that training with human feedback significantly outperforms very strong baselines on English summarization. When applying our methods on a version of the Reddit TL;DR dataset [63], we train policies via human feedback that produce better summaries than much larger policies trained via supervised learning. Summaries from our human feedback models are preferred by our labelers to the original human demonstrations in the dataset (see Figure 1).
Learning to summarize from human feedback
Most directly related to our work is previous work using human feedback to train summarization models with RL [3,73]. Bohm et al. [3] learn a reward function from a dataset of human ratings of 2.5k CNN/DM summaries, and train a policy whose summaries are preferred to a policy optimizing ROUGE. Our work is most similar to [73], who also train Transformer models [62] to optimize human feedback across a range of tasks, including summarization on the Reddit TL;DR and CNN/DM datasets. Unlike us, they train in an online manner and find the model highly extractive. They note that their labelers prefer extractive summaries and have low agreement rates with researchers. Compared to [73], we use significantly larger models, move to the batch setting for collecting human feedback, ensure high labeler-researcher agreement, and make some algorithmic modifications, such as separating the policy and value networks.
Learning to summarize from human feedback
Human feedback has also been used as a reward to train models in other domains such as dialogue [25, 68,21], translation [32,1], semantic parsing [34], story generation [72], review generation [7], and evidence extraction [46]. Our reward modeling approach was developed in prior work on learning to rank [40], which has been applied to ranking search results using either explicit feedback [2,18] or implicit feedback in the form of click-through data [29,30]. In a related line of research, human feedback has been used to train agents in simulated environments [10,24]. There is also a rich literature on using RL to optimize automatic metrics for NLP tasks, such as ROUGE for summarization [50,65,45,15,19], BLEU for translation [50,66,1,43], and other domains [61,27,26]. Finally, there has been extensive research on modifying architectures [22,59] and pre-training procedures [70,36,49,60,53,14] for improving summarization performance.
Learning to summarize from human feedback
Datasets. We use the TL;DR summarization dataset [63], which contains~3 million posts from reddit.com across a variety of topics (subreddits), as well summaries of the posts written by the original poster (TL;DRs). We additionally filter this dataset (see Appendix A) to ensure quality, including using a whitelist of subreddits that are understandable to the general population. Crucially, we also filter to include only posts where the human-written summaries contain between 24 and 48 tokens, to minimize the potential effect of summary length on quality (see Section 4.1 and Appendix F). Our final filtered dataset contains 123,169 posts, and we hold out~5% as a validation set. For the remainder of this paper, we refer to this dataset simply as TL;DR.
Learning to summarize from human feedback
I'm really nervous and excited. I'm going to pick her up tomorrow, we're cycling to the beach which will take 30 
Learning to summarize from human feedback
All of our models are Transformer decoders [62] in the style of GPT-3 [47,4]. We conduct our human feedback experiments on models with 1.3 billion (1.3B) and 6.7 billion (6.7B) parameters.
Learning to summarize from human feedback
Pretrained models. Similarly to [12,47], we start with models pretrained to autoregressively predict the next token in a large text corpus. As in [48,4], we use these models as 'zero-shot' baselines by padding the context with examples of high-quality summaries from the dataset. We provide details on pretraining in Appendix B, and on our zero-shot procedure in Appendix B.2.
Learning to summarize from human feedback
To validate that our supervised models are indeed strong baselines for comparison, we run our supervised fine-tuning procedure with our 6.7B model on the CNN/DM dataset, and find that we achieve slightly better ROUGE scores than SOTA models [71] from mid-2019 (see Appendix G.4).
Learning to summarize from human feedback
Human feedback policies. We want to use the reward model trained above to train a policy that generates higher-quality outputs as judged by humans. We primarily do this using reinforcement learning, by treating the output of the reward model as a reward for the entire summary that we maximize with the PPO algorithm [58], where each time step is a BPE token. 8 We initialize our policy to be the model fine-tuned on Reddit TL;DR. Importantly, we include a term in the reward that penalizes the KL divergence between the learned RL policy π RL φ with parameters φ and this original supervised model π SFT , as previously done in [25]. The full reward R can be written as:
Learning to summarize from human feedback
There is also a rich landscape of human feedback methods beyond binary comparisons that could be explored for training models [28,17,44,64]. For example, we could solicit high-quality demonstrations from labelers, have labelers edit model outputs to make them better, or have labelers provide explanations for why they preferred one model output over another. All of this feedback could be leveraged as a signal to train more capable reward models and policies.
Learning to summarize from human feedback
Our research is primarily motivated by the potential positive effects of aligning machine learning algorithms with the designer's preferences. Many machine learning applications optimize simple metrics which are only rough proxies for what the designer intends. This can lead to problems, such as Youtube recommendations promoting click-bait [11]. In the short term, improving techniques for learning from and optimizing human preferences directly may enable these applications to be more aligned with human well-being.
Learning to summarize from human feedback
Finally, by improving the ability of machine learning algorithms to perform tasks that were previously only achievable by humans, we are increasing the likelihood of many jobs being automated, potentially leading to significant job loss. Without suitable policies targeted at mitigating the effects of large-scale unemployment, this could also lead to significant societal harm.   Here, we discuss the pre-processing steps that we apply to the TL;DR dataset. We first remove all duplicate posts by checking the text body, finding that there are nearly 20,000 exact duplicates. We then re-parse the TL;DR carefully using a set of heuristics, and filter to use only top-level posts (rather than comments). We also filter out any post that is from a subreddit not in our 'subreddit whitelist' (see Table 2 for the distribution over subreddits), any post where the title starts with some variant of 'Edit' or 'Update', 10 and posts that contain certain topics (such as graphic sex or suicide) using heuristics. Finally, to ensure the posts are short enough to fit into the context length of our models, we filter out any post whose body is longer than 512 tokens. This resulted in a set of 287,790 posts filtered by body but not summary, of which we hold out approximately 5% as a validation set. We used this set of posts for RL training since our RL procedure does not require reference summaries.
Learning to summarize from human feedback
All models follow the standard Transformer architecture, with 2048 learned position embeddings. All models are trained with fp16 activations and the Adam optimizer [31]. Nearly all supervised baselines, reward models, and reinforcement learning models are trained with fp32 weights; the exception is our TL;DR supervised baselines, which were trained with fp16 weights. 11 All models are trained with the same byte-pair encoding as in [48].
Learning to summarize from human feedback
During pretraining, the models were trained to predict the next token on a large text corpus consisting of Commoncrawl, Webtext [48], books, and Wikipedia. Training lasts between 1-3 epochs on each, for a total of 200-300 billion tokens. Learning rate follows a cosine schedule, with a short warmup, decaying to 10% of the maximum value. The batch size ramped up throughout training to some maximum, with each input having 2048 tokens. Hyperparameters for each model are shown in Table 3.
Learning to summarize from human feedback
For PPO, we run with separate policy and value networks, initializing our policies to the supervised baseline, and our value functions to the reward model. We set γ = 1 and λ = 0.95 for the advantage estimation [57] and do 4 epochs of optimization for each batch of rollouts. We used a linear learning rate decay schedule, with initial learning rates of 1.5e-5 for the 1.3B model and 7e-6 for the 6.7B model, based on small amounts of experimentation and rough model size extrapolation. We used a KL coefficient of 0.05 for both of the main runs we report results for (except when we explicitly vary this value in the reward model optimization graphs). We use a batch size of 512 for the 1.3B model and 256 for the 6.7B model, and run for 1 million episodes.
Learning to summarize from human feedback
When sampling from models pretrained only on our pretrain mixture and not fine-tuned on TL;DR, we follow [48] and instead of padding with a padding token, we pad the beginning of the context with examples of posts/articles and high-quality summaries. We use as many examples as will fit in the token limit, with the examples formatted the same way as the main input. Table 4 documents the formats we used (with pythonic format strings).
Learning to summarize from human feedback
We wanted to understand labeler behavior on these comparisons, to ensure that it was not an error. To do this, we examined a sample of our labeler's judgments ourselves. We found that in 20/143 cases labelers preferred lead-3 by 3 points or more, and that excluding these datapoints would raise the relative score of the reference summaries by about 0.5 points. 14 We were surprised to see the reference summaries performing so poorly in a significant fraction of cases, so we looked at labeler's explanations and confirmed they made sense.
Learning to summarize from human feedback
On CNN/DM, ROUGE agrees with our human evaluations that our human feedback models transfer better than our supervised models. However, unsurprisingly, supervised CNN/DM models still achieve much higher ROUGE. In Table 15, we show the ROUGE results on CNN/DM for our 6.7B supervised baseline and various models from the literature. We find that our model achieves ROUGE scores less than T5 [49], but slightly greater than the CNN-2sent-hieco-RBM model from [71], which was SOTA for abstractive summarization on CNN/DM in mid-2019 according to the NLP-progress leaderboard. 16 
Learning a Cost-Effective Annotation Policy for Question Answering
Question answering (QA) based on textual content has attracted a great deal of attention in recent years Xie et al., 2020). In order for state-of-the-art QA models to succeed in real applications (e.g., customer service), there is often a need for large amounts of training data. However, manually annotating such data can be extremely costly. For example, in many realistic scenarios, there exists a list of questions from real users (e.g., search logs, FAQs, service-desk interactions). Yet, annotating such questions is highly expensive (Nguyen et al., 2016;He et al., 2018;Kwiatkowski et al., 2019): it requires the screening of a text corpus to find the relevant document(s) and subsequently screening the document(s) to identify the answering text span(s). Motivated by the above scenarios, we study cost-effective annotation for question answering, whereby we aim to accurately 1 annotate a given set of user questions with as little cost as possible. Generally speaking, there has been extensive research on how to reduce effort in the process of data labeling (Haffari et al., 2009). For example, active learning for a variety of machine learning and NLP tasks (Siddhant and Lipton, 2018) aims to select a small, yet highly informative, subset of samples to be annotated. The selection of such samples is usually coupled with a particular model, and thus, the annotated samples may not necessarily help to improve a different model (Lowell et al., 2019). In contrast, we aim to annotate all given samples at low cost and in a manner that can subsequently be used to develop any advanced model. This is particularly relevant in the current era, where a dataset often outlives a particular model (Lowell et al., 2019). Moreover, there has also been some research into learning from distant supervision (Xie et al., 2020) or self-supervision . Despite being economical, such approaches often produce inaccurate or noisy annotations. In this work, we seek to reduce annotation costs without compromising the resulting dataset quality.
Learning a Cost-Effective Annotation Policy for Question Answering
We compare our framework against conventional, manual annotations in an extensive set of experiments. We simulate the annotation of Nat-uralQuestions (Kwiatkowski et al., 2019), as it consists of real user questions from search logs. Models in our framework are initialized with an existing dataset (SQuAD, Rajpurkar et al., 2016) and, as more annotations on NaturalQuestions be-come available, the framework is continuously updated. We study the sensitivity of our framework to varying cost ratios between SEM and MAN. Our framework outperforms traditional manual annotation, even under conservative cost estimates for SEM, and in general reduces annotation costs in the range of 4.1% to 21.1%.
Learning a Cost-Effective Annotation Policy for Question Answering
. Second, for a given question the model can predict a number of top-n documents likely to contain the answer, i.e., Ω D : q → [d (1) , . . . , d (n) ] ∈ D. Third, for a given question-document pair the model can predict a number of top-n answer spans, i.e., Ω S : q, d → [s (1) , . . . , s (n) ]. These properties are fulfilled by recent QA systems dealing with textual content (e.g., Chen et al., 2017;.
Learning a Cost-Effective Annotation Policy for Question Answering
Separating the policy update and the costsensitive decision threshold has several benefits. First, we know from cost-sensitive classification that we can calculate an optimal threshold point for ground-truth probabilities p(a|x) (c.f. Elkan, 2001;Ting, 2000). Therefore, we can focus our effort on determining probabilities as accurate as possible. Second, the decision threshold is calculated only from the costs c S a and c D a and, hence, if costs change, we do not need to re-estimate parameters but can directly adjust our policy.
Learning a Cost-Effective Annotation Policy for Question Answering
where c(a, a ) is the annotation cost for choosing scheme a when the optimal scheme was a (see Tab. 1). Since we used down-sampling in our training, we have to calibrate the probabilities p(a|x) with the sampling ratio α (Pozzolo et al., 2015). Assuming the calibrated probabilities are accurate, there exists an optimal classification threshold β (Elkan, 2001) that minimizes Eq. 5 and Eq. 3. Therefore, we define our policy as follows:
Learning a Cost-Effective Annotation Policy for Question Answering
(6) The optimal β can be derived from the classification cost matrix (Elkan, 2001) 
Learning a Cost-Effective Annotation Policy for Question Answering
The QA model Ω is built as follows. We use a stateof-the-art BERT-based (Devlin et al., 2018) implementation of RankQA . This combines a simple tf-idf-based information retrieval module with BERT as a module for machine comprehension. Both policy models π D and π S are implemented as three-layer feed-forward networks with dropout, ReLu activation, and a single output unit with sigmoid activation in the last layer. For the policy π D , we use the information retrieval scores as input. For π S , we use the statistical fea-tures of answer-span candidates as calculated by RankQA  as input. We also experimented with convolutional neural networks directly on top of the last layer of BERT, but without yielding improvements that justified the additional model complexity. We initialize all models with the SQuAD dataset (see our supplements).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Bandit structured prediction is the task of learning to solve complex joint prediction problems (like parsing or machine translation) under a very limited feedback model: a system must produce a single structured output (e.g., translation) and then the world reveals a score that measures how good or bad that output is, but provides neither a "correct" output nor feedback on any other possible output (Chang et al., 2015;Sokolov et al., 2015). Because of the extreme sparsity of this feedback, a common experimental setup is that one pre-trains a good-but-not-great "reference" system based on whatever labeled data is available, and then seeks to improve it over time using this bandit feedback.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
A common motivation for this problem setting is cost. In the case of translation, bilingual "experts" can read a source sentence and a possible translation, and can much more quickly provide a rating of that translation than they can produce a full translation on their own. Furthermore, one can often collect even less expensive ratings from "non-experts" who may or may not be bilingual (Hu et al., 2014). Breaking this reliance on expensive data could unlock previously ignored languages and speed development of broad-coverage machine translation systems.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
1. The metric we care about (e.g., expert ratings) may not correlate perfectly with the measure that the reference system was trained on (e.g., BLEU or log likelihood); 2. Human judgments might be more granular than traditional continuous metrics (e.g., thumbs up vs. thumbs down); 3. Human feedback have high variance (e.g., different raters might give different responses given the same system output); 4. Human feedback might be substantially skewed (e.g., a rater may think all system outputs are poor). Our first contribution is a strategy to simulate expert and non-expert ratings to evaluate the robustness of bandit structured prediction algorithms in general, in a more realistic environment ( §4). We construct a family of perturbations to capture three attributes: granularity, variance, and skew. We apply these perturbations on automatically generated scores to simulate noisy human ratings. To make our simulated ratings as realistic as possible, we study recent human evaluation data (Graham et al., 2017) and fit models to match the noise profiles in actual human ratings ( §4.2).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Our second contribution is a reinforcement learning solution to bandit structured prediction and a study of its robustness to these simulated human ratings ( § 3). 1 We combine an encoderdecoder architecture of machine translation (Luong et al., 2015) with the advantage actor-critic algorithm (Mnih et al., 2016), yielding an approach that is simple to implement but works on lowresource bandit machine translation. Even with substantially restricted granularity, with high variance feedback, or with skewed rewards, this combination improves pre-trained models ( §6). In particular, under realistic settings of our noise parameters, the algorithm's online reward and final heldout accuracies do not significantly degrade from a noise-free setting.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
The bandit structured prediction problem (Chang et al., 2015;Sokolov et al., 2015) is an extension of the contextual bandits problem (Kakade et al., 2008;Langford and Zhang, 2008) to structured prediction. Bandit structured prediction operates over time i = 1 . . . K as:
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Reinforcement learning methods that rely on function approximation are preferred when tackling bandit structured prediction with a large action space because they can capture similarities between structures and generalize to unseen regions of the structure space. The actor-critic algorithm (Konda and Tsitsiklis) uses function approximation to directly model the Q function, called the critic model. In our early attempts on bandit NMT, we adapted the actor-critic algorithm for NMT in Bahdanau et al. (2017), which employs the algorithm in a supervised learning setting. Specifically, while an encoder-decoder critic model Q ω as a substitute for the true Q function in Eq 3 enables taking the full expectation (because the critic model can be queried with any stateaction pair), we are unable to obtain reasonable results with this approach. Nevertheless, insights into why this approach fails on our problem explains the effectiveness of the approach discussed in the next section. There are two properties in Bahdanau et al. (2017) that our problem lacks but are key elements for a successful actor-critic. The first is access to reference translations: while the critic model is able to observe reference translations during training in their setting, bandit NMT assumes those are never available. The second is per-step rewards: while the reward function in their setting is known and can be exploited to compute immediate rewards after taking each action, in bandit NMT, the actorcritic algorithm struggles with credit assignment because it only receives reward when a translation is completed. Bahdanau et al. (2017) report that the algorithm degrades if rewards are delayed until the end, consistent with our observations.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
With an enormous action space of bandit NMT, approximating gradients with the Q critic model induces biases and potentially drives the model to wrong optima. Values of rarely taken actions are often overestimated without an explicit constraint between Q values of actions (e.g., a sum-to-one constraint). Bahdanau et al. (2017) add an ad-hoc regularization term to the loss function to mitigate this issue and further stablizes the algorithm with a delay update scheme, but at the same time introduces extra tuning hyper-parameters.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
NED-A2C is better suited for problems with a large action space and has other advantages over actor-critic. For large action spaces, approximating gradients using the V critic model induces lower biases than using the Q critic model. As implied by its definition, the V model is robust to biases incurred by rarely taken actions since rewards of those actions are weighted by very small probabilities in the expectation. In addition, the V model has a much smaller number of parameters and thus is more sample-efficient and more stable to train than the Q model. These attractive properties were not studied in A2C's original paper (Mnih et al., 2016).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
In our simulated study, we begin by modeling gold standard human ratings using add-onesmoothed sentence-level BLEU (Chen and Cherry, 2014). 3 Our evaluation criteria, therefore, is average sentence-BLEU over the run of our algo- rithm. However, in any realistic scenario, human feedback will vary from its average, and so the reward that our algorithm receives will be a perturbed variant of sentence-BLEU. In particular, if the sentence-BLEU score is s ∈ [0, 1], the algorithm will only observe s ∼ pert(s), where pert is a perturbation distribution. Because our reference machine translation system is pre-trained using log-likelihood, there is already an (a) mismatch between training objective and feedback, so we focus on (b-d) below.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
When collecting human feedback, it is often more effective to collect discrete binned scores. A classic example is the Likert scale for human agreement (Likert, 1932) or star ratings for product reviews. Insisting that human judges provide continuous values (or feedback at too fine a granularity) can demotivate raters without improving rating quality (Preston and Colman, 2000).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Human feedback has high variance around its expected value. A natural goal for a variance model of human annotators is to simulate-as closely as possible-how human raters actually perform. We use human evaluation data recently collected as part of the WMT shared task (Graham et al., 2017). The data consist of 7200 sentences multiply annotated by giving non-expert annotators on Amazon Mechanical Turk a reference sentence and a single system translation, and asking the raters to judge the adequacy of the translation. 4 From these data, we treat the average human rating as the ground truth and consider how individual human ratings vary around that mean. To visualize these results with kernel density estimates (standard normal kernels) of the standard deviation. Figure 3 shows the mean rating (x-axis) and the deviation of the human ratings (y-axis) at each mean. 5 As expected, the standard deviation is small at the extremes and large in the middle (this is a bounded interval), with a fairly large range in the middle: a translation whose average score is 50 can get human evaluation scores anywhere between 20 and 80 with high probability. We use a linear approximation to define our variance-based perturbation function as a Gaussian distribution, which is parameterized by a scale λ that grows or shrinks the variances (when λ = 1 this exactly matches the variance in the plot). pert var (s; λ) = Nor s, λσ(s) 2 (9) σ(s) = 0.64s − 0.02 if s < 50 −0.67s + 67.0 otherwise
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
The preceding two noise models assume that the reward closely models the value we want to optimize (has the same mean). This may not be the case with non-expert ratings. Non-expert 4 Typical machine translation evaluations evaluate pairs and ask annotators to choose which is better. 5 A current limitation of this model is that the simulated noise is i.i.d. conditioned on the rating (homoscedastic noise). While this is a stronger and more realistic model than assuming no noise, real noise is likely heteroscedastic: dependent on the input.  raters are skewed both for reinforcement learning (Thomaz et al., 2006;Thomaz and Breazeal, 2008;Loftin et al., 2014) and recommender systems (Herlocker et al., 2000;Adomavicius and Zhang, 2012), but are typically bimodal: some are harsh (typically provide very low scores, even for "okay" outputs) and some are motivational (providing high scores for "okay" outputs). We can model both harsh and motivations raters with a simple deterministic skew perturbation function, parametrized by a scalar ρ ∈ [0, ∞):
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
We choose two language pairs from different language families with different typological properties: German-to-English and (De-En) and Chinese-to-English (Zh-En). We use parallel transcriptions of TED talks for these pairs of languages from the machine translation track of the IWSLT 2014 and 2015 (Cettolo et al., 2014(Cettolo et al., , 2015(Cettolo et al., , 2012. For each language pair, we split its data into four sets for supervised training, bandit training, development and testing (Table 1). For English and German, we tokenize and clean sen-tences using Moses (Koehn et al., 2007). For Chinese, we use the Stanford Chinese word segmenter (Chang et al., 2008) to segment sentences and tokenize. We remove all sentences with length greater than 50, resulting in an average sentence length of 18. We use IWSLT 2015 data for supervised training and development, IWSLT 2014 data for bandit training and previous years' development and evaluation data for testing.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Both the NMT model and the critic model are encoder-decoder models with global attention (Luong et al., 2015). The encoder and the decoder are unidirectional single-layer LSTMs. They have the same word embedding size and LSTM hidden size of 500. The source and target vocabulary sizes are both 50K. We do not use dropout in our experiments. We train our models by the Adam optimizer (Kingma and Ba, 2015) with β 1 = 0.9, β 2 = 0.999 and a batch size of 64. For Adam's α hyperparameter, we use 10 −3 during pre-training and 10 −4 during bandit learning (for both the NMT model and the critic model). During pre-training, starting from the fifth pass, we decay α by a factor of 0.5 when perplexity on the development set increases. The NMT model reaches its highest corpus-level BLEU on the development set after ten passes through the supervised training data, while the critic model's training error stabilizes after five passes. The training speed is 18s/batch for supervised pre-training and 41s/batch for training with the NED-A2C algorithm.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Poor initialization. Policy gradient algorithms have difficulty improving from poor initializations, especially on problems with a large action space, because they use model-based exploration, which is ineffective when most actions have equal probabilities (Bahdanau et al., 2017;Ranzato et al., 2016). To see whether NED-A2C has this problem, we repeat the experiment with the same setup but with reference models pretrained for only a single pass. Surprisingly, NED-A2C is highly effective at improving these poorly Comparisons with supervised learning. To further demonstrate the effectiveness of NED-A2C, we compare it with training the reference models with supervised learning for a single pass on the bandit training set. Surprisingly, observing ground-truth translations barely improves the models in Per-Sentence BLEU when they are fully trained (less than +0.4 on both tasks). A possible explanation is that the models have already reached full capacity and do not benefit from more examples. 6 NED-A2C further enhances the models because it eliminates the mismatch between the supervised training objective and the evaluation objective. On weakly trained reference models, NED-A2C also significantly outperforms supervised learning (∆Per-Sentence BLEU of NED-A2C is over three times as large as those of supervised learning).
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Ratings provided by humans can be used as effective learning signals for machines. Reinforcement learning has become the de facto standard for incorporating this feedback across diverse tasks such as robot voice control (Tenorio-Gonzalez et al.,  (Isbell et al., 2001). Recently, this learning framework has been combined with recurrent neural networks to solve machine translation (Bahdanau et al., 2017), dialogue generation (Li et al., 2016), neural architecture search (Zoph and Le, 2017), and device placement (Mirhoseini et al., 2017). Other approaches to more general structured prediction under bandit feedback (Chang et al., 2015;Sokolov et al., 2016a,b) show the broader efficacy of this framework. Ranzato et al. (2016) describe MIXER for training neural encoder-decoder models, which is a reinforcement learning approach closely related to ours but requires a policy-mixing strategy and only uses a linear critic model. Among work on bandit MT, ours is closest to Kreutzer et al. (2017), which also tackle this problem using neural encoder-decoder models, but we (a) take advantage of a state-of-the-art reinforcement learning method; (b) devise a strategy to simulate noisy rewards; and (c) demonstrate the robustness of our method on noisy simulated rewards.
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Our results show that bandit feedback can be an effective feedback mechanism for neural machine translation systems. This is despite that errors in human annotations hurt machine learning models in many NLP tasks (Snow et al., 2008). An obvious question is whether we could extend our framework to model individual annotator preferences (Passonneau and Carpenter, 2014) or learn personalized models (Mirkin et al., 2015;Rabinovich et al., 2017), and handle heteroscedastic noise (Park, 1966;Kersting et al., 2007;Antos et al., 2010). Another direction is to apply active learning techniques to reduce the sample complexity required to improve the systems or to extend to richer action spaces for problems like simultaneous translation, which requires prediction (Grissom II et al., 2014) and reordering (He et al., 2015) among other strategies to both minimize delay and effectively translate a sentence (He et al., 2016).
Self-Regulated Interactive Sequence-to-Sequence Learning
The concept of self-regulation has been studied in educational research (Hattie and Timperley, 2007;Hattie and Donoghue, 2016), psychology (Zimmerman and Schunk, 1989;Panadero, 2017), and psychiatry (Nigg, 2017), and was identified as central to successful learning. "Self-regulated students" can be characterized as "becoming like teachers", in that they have a repertoire of strategies to self-assess and self-manage their learning process, and they know when to seek help and which kind of help to seek. While there is a vast literature on machine learning approaches to metalearning (Schmidhuber et al., 1996), learning-tolearn (Thrun andPratt, 1998), or never-ending learning (Mitchell et al., 2015), the aspect of learning when to ask for which kind of feedback has so far been neglected in this field.
Self-Regulated Interactive Sequence-to-Sequence Learning
There are various options for building the architecture of the neural model NN θ , such as recurrent (Sutskever et al., 2014), convolutional (Gehring et al., 2017) or attentional (Vaswani et al., 2017) encoder-decoder architectures (or a mix thereof (Chen et al., 2018)). Regardless of their architecture, there are multiple ways of interactive learning that can be applied to neural Seq2Seq learners.
Self-Regulated Interactive Sequence-to-Sequence Learning
A local minimum can be found by performing stochastic gradient descent on g FULL θ (x, y * ). This training objective is the standard in supervised learning when training with human-generated references or for online adaptation to post-edits (Turchi et al., 2017).
Self-Regulated Interactive Sequence-to-Sequence Learning
Learning from Error Markings (WEAK). Petrushkov et al. (2018) presented chunk-based binary feedback as a low-cost alternative to full corrections. In this scenario the human teacher marks the correct parts of the machine-generated output y. As a consequence every token in the output receives a reward δ t , either δ t = 1 if marked as correct, or δ t = 0 otherwise. The objective of the learner is to maximize the likelihood of the correct parts of the output, or equivalently, to minimize
Self-Regulated Interactive Sequence-to-Sequence Learning
Self-Supervision (SELF). Instead of querying the teacher for feedback, the learner can also choose to learn from its own output, that is, to learn from self-supervision. The simplest option is to treat the learner's output as if it was correct, but that quickly leads to overconfidence and degeneration. Clark et al. (2018) proposed a cross-view training method: the learner's original prediction is used as a target for a weaker model that shares parameters with the original model. We adopt this strategy by first producing a target sequenceŷ with beam search and then weaken the decoder through attention dropout with probability p att . The objective is to minimize the negative likelihood of the original target under the weakened model
Self-Regulated Interactive Sequence-to-Sequence Learning
is measured as the difference in validation score achieved before and after the learner's update (Fang et al., 2017), and c i as the cost of user edits. Adding a small constant cost α to the actual feedback cost ensures numerical stability. This meta-parameter can be interpreted as representing a basic cost for model updates of any kind. The objective for the regulator is to maximize the expected reward defined in Eq. 1:
Self-Regulated Interactive Sequence-to-Sequence Learning
The full gradient of this objective is estimated by the stochastic gradient for sampled actions (Williams, 1992):
Self-Regulated Interactive Sequence-to-Sequence Learning
Practical Considerations. The algorithm does not introduce any additional hyperparameters beyond standard learning rates, architecture design and mini-batch sizes that have to be tuned. As proposed in Petrushkov et al. (2018) or Clark et al. (2018), targetsŷ are pre-generated offline with the initial θ 0 , which we found crucial for the stability of the learning process. The evaluation step after the Seq2Seq update is an overhead that comes with meta-learning, incurring costs depending on the decoding algorithm and the evaluation strategy. However, Seq2Seq updates can be performed in mini-batches, and the improvement is assessed after a mini-batch of updates, as discussed above.
Self-Regulated Interactive Sequence-to-Sequence Learning
Seq2Seq Architecture. Both the Seq2Seq learner and the regulator are based on LSTMs (Hochreiter and Schmidhuber, 1997). The Seq2Seq has four bi-directional encoder and four decoder layers with 1024 units each, embedding layers of size 512. It uses Luong et al. (2015)'s input feeding and output layer, and global attention with a single feed forward layer (Bahdanau et al., 2015).
Self-Regulated Interactive Sequence-to-Sequence Learning
Regulator Architecture. The regulator consists of LSTMs on two levels: Inspired by Siamese Networks (Bromley et al., 1994), a bi-directional LSTM encoder of size 512 separately reads in both the current input sequence and the beam search hypothesis generated by the Seq2Seq. The last state of encoded source and hypothesis sequence and the previous output distribution are concatenated to form the input to a higher-level regulator LSTM of size 256. This LSTM updates its internal state and predicts a score for every feedback type for every input in the mini-batch. The feedback for each input is chosen by sampling from the distribution obtained by softmax normalization of these scores. The embeddings of the regulator are initialized by the Seq2Seq's source embeddings and further tuned during training. The model is implemented in the JoeyNMT 1 framework based on PyTorch. 2 Data. We use three parallel corpora for Germanto-English translation: a general-domain data set from the WMT2017 translation shared task for Seq2Seq pre-training, TED talks from the IWSLT2017 evaluation campaign for training the regulator with simulated feedback, and the Books corpus from the OPUS collection (Tiedemann, 2012) for testing the regulator on another domain. Data pre-processing details and splits are given in §A.1. The joint vocabulary for Seq2Seq and the regulator consists of 32k BPE sub-words (Sennrich et al., 2016) trained on WMT.
Self-Regulated Interactive Sequence-to-Sequence Learning
Simulation of Cost and Performance. In our experiments, human feedback and its cost, and the performance gain achieved by model updates, is simulated by using human reference translations. Inspired by the keystroke mouse-action ratio (KSMR) (Barrachina et al., 2009), a common metric for measuring human effort in interactive machine translation, we define feedback cost as the sum of costs incurred by character edits and clicks, similar to Peris and Casacuberta (2018). The cost of a full correction (FULL) is the number of character edits between model output and reference, simulating the cost of a human typing. 4 Error markings (WEAK) are simulated by comparing the hypothesis to the reference and marking the longest common sub-strings as correct, as proposed by Petrushkov et al. (2018). As an extension to Petrushkov et al. (2018) we mark multiple common sub-strings as correct if all of them have the longest length. The cost is defined as the number of marked words, assuming an interface that allows markings by clicking on words. For selftraining (SELF) and skipping training instances we naively assume zero cost, thus limiting the mea-3 Values = 1 distort the rewards for self-training too much. 4 As computed by the Python library difflib. surement of cost to the effort of the human teacher, and neglecting the effort on the learner's side. Table 1 illustrates the costs per feedback type on a randomly selected set of examples.
Self-Regulated Interactive Sequence-to-Sequence Learning
We measure the model improvement by evaluating the held-out set translation quality of the learned model at various time steps with corpus BLEU (cased SacreBLEU (Post, 2018)) and measure the accumulated costs. The best model is considered the one that delivers the highest quality at the lowest cost. This trade-off is important to bear in mind since it differs from the standard evaluation of machine translation models, where the overall best-scoring model, regardless of the supervision cost, is considered best. Finally, we evaluate the strategy learned by the regulator on an unseen domain, where the regulator decides which type of feedback the learner gets, but is not updated itself.
Self-Regulated Interactive Sequence-to-Sequence Learning
We compare learning from one type of feedback in isolation against regulators with the following set of actions:  Figure 3 compares the improvement in corpus BLEU (Papineni et al., 2002) (corresponding to results in Translation Error Rate (TER, computed by pyTER) (Snover et al., 2006)) of regulation variants and full feedback over cumulative costs of up to 80k character edits. Using only full feedback (blue) as in standard supervised learning or learning from post-edits, the overall highest improvement can be reached (visible only after the cutoff of 80k edits; see Appendix A.2 for the comparison over a wider window of time). However, it comes at a very high cost (417k characters in total to reach +0.6 BLEU). The regulated variants offer a much cheaper improvement, at least until a cumulative cost between 80k (Reg4) and 120k (Reg2), depending on the feedback options available. The regulators do not reach the quality of the full model since their choice of feedback is oriented towards costs and immediate improvements. By finding a trade-off between feedback types for immediate improvements, the regulators sacrifice long-term improvement. Comparing regulators, Reg2 (orange) reaches the overall SELF 0
Self-Regulated Interactive Sequence-to-Sequence Learning
A further research question is whether and how the self-regulation module takes the input or output context into account. We therefore compare its decisions to a context-free -greedy strategy. The -greedy algorithm is a successful algorithm for multi-armed bandits (Watkins, 1989). In our case, the arms are the four feedback types. They are chosen based on their reward statistics, here the average empirical reward per feedback type Q i (s) = 1 N i (s) 0,...,i r(s i ). With probability 1 − , the algorithm selects the feedback type with the highest empirical reward (exploitation), otherwise picks one of the remaining arms at random (exploration). In contrast to the neural regulator model, -greedy decides solely on the basis  of the reward statistics and has no internal contextual state representation. The comparison of Reg3 with -greedy for a range of values for in Figure 5 shows that learned regulator behaves indeed very similar to an -greedy strategy with = 0.25. -greedy variants with higher amounts of exploration show a slower increase in BLEU, while those with more exploitation show an initial steep increase that flattens out, leading to overall lower BLEU scores. The regulator has hence found the best trade-off, which is an advantage over the -greedy algorithm where the hyperparameter requires dedicated tuning. Considering the -greedy-like strategy of the regulator and the strong role of the cost factor shown in Figure 4, the regulator module does not appear to choose individual actions based e.g., on the difficulty of inputs, but rather composes mini-batches with a feedback ratio according to the feedback type's statistics. This confirms the observations of Peris and Casacuberta (2018), who find that the subset of instances selected for labeling is secondaryit is rather the mixing ratio of feedback types that matters. This finding is also consistent with the mini-batch update regime that forces the regulator to take a higher-level perspective and optimize the expected improvement at the granularity of (minibatch) updates rather than at the input level.
Self-Regulated Interactive Sequence-to-Sequence Learning
Our experiments were designed as a pilot study to test the possibilities of self-regulated learning in simulation. In order to advance to field studies where human users interact with Seq2Seq models, several design choices have to be adapted with caution. Firstly, we simulate both feedback cost and quality improvement by measuring distances to static reference outputs. The experimental design in a field study has to account for a variation of feedback strength, feedback cost, and performance assessments, across time, across sentences, and across human users . One desideratum for field studies is thus to analyze this variation by analyzing the experimental results in a mixed effects model that accounts for variability across sentences, users, and annotation sessions (Baayen et al., 2008;Karimova et al., 2018). Secondly, our simulation of costs considers only the effort of the human teacher, not the machine learner. The strong preference for the cheapest feedback option might be a result of overestimating the cost of human post-editing and underestimating the cost of self-training. Thus, a model for field studies where data is limited might greatly benefit from learned estimates of feedback cost and quality improvement .
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
In semantic parsing, natural language utterances are mapped to machine readable parses which are complex and often tailored specifically to the underlying task. The cost and difficulty of manually preparing large amounts of such parses thus is a bottleneck for supervised learning in semantic parsing. Recent work (Liang et al. (2017); Mou et al. (2017); Peng et al. (2017); inter alia) has applied reinforcement learning to address the annotation bottleneck as follows: Given a question, the existence of a corresponding gold answer is assumed. A semantic parser produces multiple parses per question and corresponding answers are obtained. These answers are then compared against the gold answer and a positive reward is recorded if there is an overlap. The parser is then guided towards correct parses using the REIN-FORCE algorithm (Williams, 1992) which scales the gradient for the various parses by their obtained reward (see the left half of Figure 1). However, learning from question-answer pairs is only efficient if gold answers are cheap to obtain. For complex open-domain question-answering tasks, correct answers are not unique factoids, but openended lists, counts in large ranges, or fuzzily defined objects. For example, geographical queries against databases such as OpenStreetMap (OSM) can involve fuzzy operators such as "near" or "in walking distance" and thus need to allow for fuzziness in the answers as well. A possible solution lies in machine learning from even weaker supervision signals in form of human bandit feedback 1 where the semantic parsing system suggests exactly one parse for which feedback is collected from a human user. In this setup neither gold parse nor gold answer are known and feedback is obtained for only one system output per question.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Lastly, from a machine learning perspective, we have to solve problems of degenerate behavior in counterfactual learning by lifting the multiplicative control variate technique (Swaminathan and Joachims, 2015b;Lawrence et al., 2017b,a) to stochastic learning for neural models. This is done by reweighting target model probabilities over the logged data under a one-step-late model that decouples the normalization from gradient estimation and is thus applicable in stochastic (minibatch) gradient optimization.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Our semantic parsing model is a state-of-theart sequence-to-sequence neural network using an encoder-decoder setup Sutskever et al., 2014) together with an attention mechanism (Bahdanau et al., 2015). We use the settings of Sennrich et al. (2017), where an input sequence x = x 1 , x 2 , . . . x |x| (a natural language question) is encoded by a Recurrent Neural Network (RNN), each input token has an associated hidden vector
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
where outputs y t for inputs x t were generated by a logging system under policy π 0 , and loss values δ t ∈ [−1, 0] 2 were observed for the generated data points. Our goal is to optimize the expected reward (in our case: minimize the expected risk) for a target policy π w given the data log D log . In case of deterministic logging, outputs are logged with propensity π 0 (y t |x t ) = 1, t = 1, . . . , n. This results in a deterministic propensity matching (DPM) objective (Lawrence et al., 2017b), without the possibility to correct the sampling bias of the logging policy by inverse propensity scoring (Rosenbaum and Rubin, 1983):
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
This objective can show degenerate behavior in that it overfits to the choices of the logging policy (Swaminathan and Joachims, 2015b;Lawrence et al., 2017a). This degenerate behavior can be avoided by reweighting using a multiplicative control variate (Kong, 1992;Precup et al., 2000;Jiang and Li, 2016;Thomas and Brunskill, 2016). The new objective is called the reweighted deterministic propensity matching (DPM+R) objective in Lawrence et al. (2017b):
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Reweighting in Stochastic Learning. As shown in Swaminathan and Joachims (2015b) and Lawrence et al. (2017a), reweighting over the entire data log D log is crucial since it avoids that high loss outputs in the log take away probability mass from low loss outputs. This multiplicative control variate has the additional effect of reducing the variance of the estimator, at the cost of introducing a bias of order O( 1 n ) that decreases as n increases (Kong, 1992). The desirable properties of this control variate cannot be realized in a stochastic (minibatch) learning setup since minibatch sizes large enough to retain the desirable reweighting properties are infeasible for large neural networks.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
We offer a simple solution to this problem that nonetheless retains all desired properties of the reweighting. The idea is inspired by one-step-late algorithms that have been introduced for EM algorithms (Green, 1990). In the EM case, dependencies in objectives are decoupled by evaluating certain terms under parameter settings from previous iterations (thus: one-step-late) in order to achieve closed-form solutions. In our case, we decouple the reweighting from the parameterization of the objective by evaluating the reweighting under parameters w from some previous iteration. This allows us to perform gradient descent updates and reweighting asynchronously. Updates are performed using minibatches, however, reweighting is based on the entire log, allowing us to retain the desirable properties of the control variate.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
OpenStreetMap (OSM) is a geographical database in which volunteers annotate points of interests in the world. A point of interest consists of one or more associated GPS points. Further relevant information may be added at the discretion of the volunteer in the form of tags. Each tag consists of a key and an associated value, for example "tourism : hotel". The NLMAPS corpus was introduced by Haas and Riezler (2016) as a basis to create a natural language interface to the OSM database. It pairs English questions with machine readable parses, i.e. queries that can be executed against OSM.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
Corpus Extension. Similar to the extension of the NLMAPS corpus by Lawrence and Riezler (2016) who include shortened questions which are more typically used by humans in search tasks, we present an automatic extension that allows a larger coverage of common OSM tags. 4 The basis for the extension is a hand-written, online freely available list 5 that links natural language expressions such as "cash machine" to appropriate OSM tags, in this case "amenity : atm". Using the list, we generate for each unique expression-tag pair a set of question-query pairs. These latter pairs contain 3 https://wiki.openstreetmap.org/ 4 The extended dataset, called NLMAPS V2, will be released upon acceptance of the paper.   (Lawrence and Riezler, 2016) and the automatic extensions of the most common OSM tags.
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
General Settings. In our experiments we use the sequence-to-sequence neural network package NEMATUS (Sennrich et al., 2017). Following the method used by Haas and Riezler (2016), we split the queries into individual tokens by taking a pre-order traversal of the original tree-like structure. For example, "query(west(area(keyval('name','Paris')), nwr(keyval('railway','station'))),qtype(count))" becomes "query@2 west@2 area@1 keyval@2 name@0 Paris@s nwr@1 keyval@2 railway@0 station@s qtype@1 count@0".
Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback
To empirically validate the usefulness of the automatically created data, we compare two parsers trained with NEMATUS (Sennrich et al., 2017) (see Section 6 for more details). The first model is trained using the original NLMAPS training data. The second receives an additional 15,000 instances from the synthetic data. Both systems are tested on the original NLMAPS test data and on the new test set of NLMAPS V2 which consists of a random set of 2,000 pairs from the remaining data. Results may be found in  Table 7: Answer F1 scores on the NLMAPS V1 and NLMAPS V2 test sets for models trained on either only NLMAPS V1 training data or with an additional 15k synthetic instances. Results are averaged over 3 runs. Using the NLMAPS V2 training set leads to significant system differences on both test sets at p < 0.05.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
With the rapid growth of text-based information on the Internet, automatic document summarisation attracts increasing research attention from the Natural Language Processing (NLP) community (Nenkova and McKeown, 2012). Most existing document summarisation techniques require access to reference summaries to train their systems. However, obtaining reference summaries is very expensive: Lin (2004) reported that 3,000 hours of human effort were required for a simple evaluation of the summaries for the Document Understanding Conferences (DUC). Although previous work has proposed heuristics-based methods to summarise without reference summaries (Ryang and Abekawa, 2012;Rioux et al., 2014), the gap between their performance and the upper bound is still large: the ROUGE-2 upper bound of .212 on DUC'04 (P.V.S. and Meyer, 2017) is, for example, twice as high as Rioux et al.'s (2014) .114.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
The Structured Prediction from Partial Information (SPPI) framework has been proposed to learn to make structured predictions without access to gold standard data (Sokolov et al., 2016b). SPPI is an interactive NLP paradigm: It interacts with a user for multiple rounds and learns from the user's feedback. SPPI can learn from two forms of feedback: point-based feedback, i.e. a numeric score for the presented prediction, or preference-based feedback, i.e. a preference over a pair of predictions. Providing preference-based feedback yields a lower cognitive burden for humans than providing ratings or categorical labels (Thurstone, 1927;Kendall, 1948;Kingsley and Brown, 2010;Zopf, 2018). Preference-based SPPI has been applied to multiple NLP applications, including text classification, chunking and machine translation (Sokolov et al., 2016a;Kreutzer et al., 2017). However, SPPI has prohibitively high sample complexities in the aforementioned NLP tasks, as it needs at least hundreds of thousands rounds of interaction to make near-optimal predictions, even with simulated "perfect" users. Figure 1a illustrates the workflow of the preference-based SPPI.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
RL amounts to efficient algorithms for searching optimal solutions in MDPs. MDPs are widely Input : sequence of learning rates γ t ; query budget T ; document cluster x initialise w 0 ; while t = 0 . . . T do sample (y i , y j ) according to Eq. (2); obtain feedback ∆ x (y i , y j ); w t+1 := w t − γ t ∇ w L SPPI (w|x) end Output: y * = arg max y∈Y (x) w T +1 φ(y, x) Algorithm 1: SPPI for preference-based interactive document summarisation (adjusted from Alg. 2 in (Sokolov et al., 2016a)).
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
In EMDS, we follow the same MDP formulation as Ryang and Abekawa (2012) and Rioux et al. (2014). Given a document cluster, a state s is a draft summary, A includes two types of actions, concatenate a new sentence to the current draft summary, or terminate the draft summary construction. The transition function P in EMDS is trivial because given the current draft summary and an action, the next state can be easily inferred. The reward function R returns an evaluation score of the summary once the action terminate is performed; otherwise it returns 0 because the summary is still under construction and thus not ready to be evaluated. Providing non-zero rewards before the action terminate can lead to even worse result, as reported by Rioux et al. (2014).
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
The linear Bradley-Terry (BT) model (Bradley and Terry, 1952) is one of the most widely used methods in preference learning. Given a set of items Y, suppose we have observed T preferences: Q = {q 1 (y 1,1 , y 1,2 ), · · · , q T (y T,1 , y T,2 )}, where y i,1 , y i,2 ∈ Y, and q i ∈ {≺, } is the oracle's preference in the i th round. The BT model minimises the following cross-entropy loss:
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
Simulated Users. Existing preference-based interactive learning techniques assume that the oracle has an intrinsic evaluation function U * and provides preferences consistent with U * by preferring higher valued candidates. We term this a Per-1 http://duc.nist.gov/ fect Oracle (PO). We believe that assuming a PO is unrealistic for real-world applications, because sometimes real users tend to misjudge the preference direction, especially when the presented candidates have similar quality. In this work, besides PO, we additionally consider two types of noisy oracles based on the user-response models proposed by Viappiani and Boutilier (2010):
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
As for the intrinsic evaluation function U * , recent work has suggested that human preferences over summaries have high correlations to ROUGE scores (Zopf, 2018). Therefore, we define:
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
Recall that the task of APL is to output a ranking of all summaries in a cluster. In this subsection, we test multiple APL techniques and compare the quality of their resulting rankings. Two metrics are used: Kendall's τ (Kendall, 1948) and Spearman's ρ (Spearman, 1904). Both metrics are valued between −1 and 1, with higher values suggesting higher rank correlation. Because the number of possible summaries in a cluster is huge, instead of evaluating the ranking quality on all possible summaries, we evaluate rankings on 10,000 randomly sampled summaries, denotedŶ(x). During querying, all candidate summaries presented to the oracle are also selected fromŶ(x). SamplingŶ(x) a priori helps us to reduce the response time to under 500 ms for all APL techniques we test. We compare four active learning strategies under two query budgets, T = 10 and T = 100:
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
• J&N is the robust query selection algorithm proposed by Jamieson and Nowak (2011). It assumes that the items' preferences are dependent on their distances to an unknown reference point in the embedding space: the farther an item to the reference point, the more preferred the item is. After each round of interaction, the algorithm uses all collected preferences to locate the area where the reference point may fall into, and identify the query pairs which can reduce the size of this area, termed ambiguous query pairs. To combat noise in preferences, the algorithm selects the most-likely-correct ambiguous pair to query the oracle in each round.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
where w * is the weights vector learned by the BT model (see Eq. (4)), HU is the heuristics-based summary evaluation function proposed by Ryang and Abekawa (2012), and α ∈ [0, 1] is a parameter. The aim of using HU and α is to trade off between the prior knowledge (i.e. heuristics-based HU ) and the posterior observation (i.e. the BTlearnt w * ), so as to combat the cold-start problem. Based on some preliminary experiments, we set α = 0.3 when the query budget is 10, and α = 0.7 when the query budget is 100. The intuition is to put more weight to the posterior with increasing rounds of interaction. More systematic research of α can yield better results; we leave it for future work. For the vector φ(y|x), we use the same bagof-bigram embeddings as Rioux et al. (2014), and we let its length be 200.
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
We compare two RL algorithms: TD(λ) (Sutton, 1984) and LSTD(λ) (Boyan, 1999). TD(λ) has been used in previous RL-based EMDS work (Ryang and Abekawa, 2012;Rioux et al., 2014). LSTD(λ) is chosen, because it is an improved TD algorithm and has been used in the state-of-the-art PbRL algorithm by Wirth et al. (2016). We let the learning round (see Alg. 2) N = 5, 000, which we found to yield good results in reasonable time (less than 1 minute to generate a summary for one document cluster). Letting N = 3, 000 will result in a significant performance drop, while increasing N to 10,000 will only bring marginal improvement at the cost of doubling the runtime. The learn-  ing parameters we use for TD(λ) are the same as those by Rioux et al. (2014). For LSTD(λ), we let λ = 1 and initialise its square matrix as a diagonal matrix with random numbers between 0 and 1, as suggested by Lagoudakis and Parr (2003). The rewards we use are the U * function introduced in §5. Note that this serves as the upper-bound performance, because U * relies on the reference summaries (see Eq. (6)), which are not available in the interactive setting. As a baseline, we also present the upper-bound performance of integer linear programming (ILP) reported by P.V.S. and Meyer (2017), optimised for bigram coverage. Table 3 shows the performance of RL and ILP on the DUC'04 dataset. TD(λ) significantly outperforms LSTD(λ) in terms of all ROUGE scores we consider. Although the least-square RL algorithms (which LSTD belongs to) have been proved to achieve better performance than standard TD methods in large-scale problems (see Lagoudakis and Parr, 2003), their performance is sensitive to many factors, e.g., initialisation values in the diagonal matrix, regularisation parameters, etc. We note that a similar observation about the inferior performance of least-square RL in EMDS is reported by Rioux et al. (2014).
APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning
We identify two major lines of future work. On the technical side, we plan to employ more advanced APL and RL algorithms in APRIL, such as sample-efficient Bayesian-based APL algorithms (e.g., Simpson and Gurevych, 2018) and neural RL algorithms (e.g. Mnih et al., 2015) to further reduce the sample complexity of APRIL. On the experimental side, a logical next step is to implement an interactive user interface for APRIL and conduct a larger evaluation study comparing the summary quality before and after the interaction. We also plan to apply APRIL to more NLP applications, including machine translation, information exploration and semantic parsing.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Most existing systems frame semantic parsing as a one-shot translation from a natural language question to the corresponding logical form (e.g., SQL query) (Yu et al., 2018a;Guo et al., 2019;Wang et al., 2020, inter alia). A growing body of recent work demonstrates that semantic parsing systems can be improved by including users in the parsing loop-giving them the affordance to examine the parses, judge their correctness, and provide feedback accordingly. The feedback often comes in the form of a binary correct/incorrect * Most of the work was done while the first author was an intern at Microsoft Research.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
signal (Iyer et al., 2017), answers to a multiplechoice question posed by the system (Gur et al., 2018;Yao et al., 2019), or suggestions of edits that can be applied to the parse .
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Unlike other frameworks for interactive semantic parsing that typically expect users to judge the correctness of the execution result or induced logical form, Elgohary et al. (2020) introduced a framework for interactive text-to-SQL in which induced SQL queries are fully explained in natural lan-guage to users, who in turn, can correct such parses through natural language feedback (Figure 1). They construct the SPLASH dataset and use it to evaluate baselines for the semantic parse correction with natural language feedback task they introduce.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
We make the following contributions: (1) We present a scheme for representing SQL query Edits that benefits both the modeling and the analysis of the correction task, (2) we present NL-EDIT, an edit-based model for interactive text-to-SQL with natural language feedback. We show that NL-EDIT outperforms baselines in (Elgohary et al., 2020) by more than 16 points, (3) We demonstrate that we can generate synthetic data through the edit-based framing and that the model can effectively use this data to improve its accuracy and (4) We present a detailed analysis of the model performance including studying the effect of different components, generalization to errors of state-of-the-art parsers, and outline directions for future research.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
In the task of text-to-SQL parsing, the objective is given a database schema (tables, columns, and primary-foreign key relations) and a natural language question, generate a SQL query that answers the question when executed against the database. Several recent text-to-SQL models have been introduced (Yu et al., 2018a;Guo et al., 2019;Wang et al., 2020, inter alia) as a result of the availability of SPIDER (Yu et al., 2018b), a large dataset of schema, questions and gold parses spanning several databases in different domains.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
The task of SQL parse correction with natural language feedback (Elgohary et al., 2020) aims to correct an erroneous parse based on natural language feedback collected from the user. Given a question, a database schema, an incorrect initial parse, natural language feedback on the initial parse, the task is to generate a corrected parse.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
To study this problem, Elgohary et al. (2020) introduced the SPLASH dataset. SPLASH was created by showing annotators questions and a natural language explanation of incorrect parses and asking them to provide feedback, in natural language, to correct the parse. The dataset contained 9,314 question-feedback pairs. Like the SPIDER dataset, it was split into train-dev-test sets by database to encourage the models to generalize to new unseen databases. They contrast the task with conversational semantic parsing (Suhr et al., 2018;Yu et al., 2019b,a;Andreas et al., 2020) and show that the two tasks are distinct and are addressing different aspects of utilizing context. They establish several baseline models and show that the task is challenging for state-of-the-art semantic parsing models. We use these as baselines for this work.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Interpreting feedback in context: The feedback is expected to link to all the other elements of the interaction ( Figure 1). The feedback is provided in the context of the explanation of the initial parse, as a proxy to the parse itself. As such, the feedback tends to use the same terminology as the explanation. For example, the SQL explanations of (Elgohary et al., 2020) express "group by" in simple language "for each vote_id, find ...". As a result, human-provided feedback never uses "group by". We also notice that in several SPLASH examples, the feedback refers to particular steps in the explanation as in the examples in Figure 1. Unlike existing models (Elgohary et al., 2020), we replace the initial parse with its natural language explanation. Additionally, the feedback usually refers to columns/tables in the schema, and could often be ambiguous when examined in isolation. Such ambiguities can be usually resolved by relying on the context provided by the question. For example, "find last name" in Figure 1 is interpreted as "find last name besides first name" rather than "replace first name with last name" because the question asks for the "full name". Our first key idea is based on grounding the elements of the interaction by combining self-learned relations by transformer models (Vaswani et al., 2017) and hard-coded relations that we define according to the possible ways different elements can link to each other.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Feedback describes a set of edits: The difference between the erroneous parse and the correct one can mostly be described as a few edits that need to be applied to the initial parse to correct its errors (Section 7). Also, the feedback often only describes the edits to be made (Elgohary et al., 2020). As such, we can pose the task of correction with NL feedback as a semantic parsing task where we convert a natural language deception of  the edits to a canonical form that can be applied deterministically to the initial parse to generate the corrected one. We train our model to generate SQL Edits (Section 3) rather than SQL queries.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Our encoder ( Figure 3) starts with passing the concatenation of the feedback, explanation, question, and schema through BERT (Devlin et al., 2019). Following (Wang et al., 2020;Suhr et al., 2018;Scholak et al., 2020), we tokenize the column/table names and concatenate them in one sequence (Schema) starting with the tokens of the tables followed by the tokens of the columns. Then, we average the BERT embeddings of the tokens corresponding to each column (table) to obtain one representation for the column (table). Wang et al. (2020) study the text-to-SQL problem using the SPIDER dataset and show the benefit of injecting preexisting relations within the schema (column exists in a table, primary-foreign key), and between the question and schema items (column and table names) by: (1) name linking: link a question token to a column/table if the token and the item name match and (2) value linking: link a question token to a column if the token appears as a value under that column. To incorporate such relations in their model, they use the relation-aware self-attention formulation presented in (Shaw et al., 2018). The relation-aware transformer (Shaw et al., 2018) assigns a learned embedding for each relation type and combines such embeddings with the self-attention of the original transformer model (Vaswani et al., 2017): If a preexisting relation r holds between two tokens, the embedding of r is added as a bias term to the self-attention computation between the two tokens.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
In addition to those relations, we define a new set of relations that aim at contextualizing the feedback with respect to the other elements of the interaction in our setup: (1) [Feedback-Schema] We link the feedback to the schema the same way the question is linked to the schema via both name and value linking, (2) [Explanation-Schema] Columns and tables are mentioned with their exact names in the explanation. We link the explanation to the schema only through exact name matching, (3) [Feedback-Question] We use partial (at the lemma level) and exact matching to link tokens in the feedback and the question, (4) [Feedback-Explanation] We link tokens in the feedback to tokens in the explanation through partial and exact token matching. Since the feedback often refers to particular steps, we link the feedback tokens to explanation tokens that occur in steps that are referred to in the feedback with a separate relation type that indicates step reference in the feedback, and (5) [Explanation-Explanation] We link explanation tokens that occur within the same step. We use the same formulation of relationaware self-attention as (Wang et al., 2020) and add the relation-aware layers on top of BERT to integrate all relations into the model (Figure 3).
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Using a standard teacher-forced cross-entropy loss, we train our model to generate linearized SQL Edits (Figure 2). At training time, we compute the reference SQL Edit D P →P of the initial parse P and the gold parse P (Section 3). Then we linearize D P →P by listing the clause edits in a fixed order (FROM, WHERE, GROUP-BY, ... etc.). The argument of each clause-representing one add or remove operation-is formatted as <CLAUSE> ADD/REMOVE ARG </CLAUSE>. We express SQL operators in ARG with natural language explanation as in (Elgohary et al., 2020). For example, the argument "AVG(grade)" is expressed as "average grade". At inference time, we generate a corrected parse P by applying the produced edit to the initial parse P .
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Algorithm 1 summarizes the synthesis process. We start by creating N (controls the size of the dataset) clones of each seed example. Elgohary et al. (2020)'s analysis of SPLASH shows that multiple mistakes might be present in the initial SQL, hence we allow our synthesis process to introduce up to four edits (randomly decided in line:4) to each clone p. For each editing step, we sample a feasible edit for the current parse (line:5) with man- We use BERT-base-uncased (Devlin et al., 2019) in all our experiments. We set the number of layers in the relational-aware transformer to eight (Wang et al., 2020) and the number of decoder layers to two. We train with batches of size 24. We use the Adam optimizer (Kingma and Ba, 2015) for training. We freeze BERT parameters during the first 5,000 warm-up steps and update the rest of the parameters with a linearly increasing learning rate from zero to 5 × 10 −4 . Then, we linearly decrease the learning rates from 5 × 10 −5 for BERT and  5 × 10 −4 for the other parameters to zero. 2 We use beam search with a beam of size 20 and take the top-ranked beam that results in a valid SQL after applying the inferred edit. Evaluation: We follow (Elgohary et al., 2020) and use the correction accuracy as our main evaluation measure: each example in SPLASH test set contains an initial parse P and a gold parse P . With a predicted (corrected) parse by a correction model P , they compute the correction accuracy using the exact-set-match (Yu et al., 2018b) between P and P averaged over all test examples. While useful, correction accuracy also has limitations. It expects models to be able to fully correct an erroneous parse with only one utterance of feedback as such, it is defined in terms of the exact match between the corrected and the gold parse. We find ( Table 2) that in several cases, models were still able to make progress by reducing the number of errors as measured by the edit size (Section 3) after correction. As such, we define another set of metrics to measure partial progress. We report (Edit ↓ and Edit ↑ in Table 2) the percentage of examples on which the size of the edit set strictly decreased/increased. To combine Edit ↓ and Edit ↑ in one measure and account for the relative reduction (increase) in the number of edits, we define
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Results: We compare (Table 2) NL-EDIT to the two top-performing baselines in (Elgohary et al., 2020) and also to the beam re-ranking upper-bound they report. NL-EDIT significantly increases the correction accuracy over the top baseline (Edit-SQL+Feedback) by more than 16% and it also outperforms oracle re-ranking by around 5%. We also note that in 72.4% of the test examples, NL-EDIT was able to strictly reduce the number of errors in the initial parse (Edit ↓) which potentially indicates a more positive user experience than the other models. NL-EDIT achieves 37% Progress which indicates faster convergence to the fully corrected parse than all the other models.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Following the same experimental setup in Section 6, we compare NL-EDIT to other variants with one ablated component at a time (Table 3). We ablate the feedback, the explanation, and the question from the encoder input. We also ablate the interaction relations (Section 4.2) that we incorporate in the relation-aware transformer module. We only ablate the new relations we introduce to model the interaction (shown in Figure 3), but we keep the Question-Schema and Schema-Schema relations introduced in (Wang et al., 2020). For each such variant, we train for 20,000 steps on the synthetic dataset then continue training on SPLASH until step 100,000. We also train an ablated variant that does not use the synthetic feedback where we   train for 100,000 steps only on SPLASH. For all variants, we choose the checkpoint with the largest correction accuracy on the dev set and report the accuracy on the SPLASH test set. The results in Table 3 confirm the effectiveness of each component in our model. We find that the model is able to correct 19.8% of the examples without the feedback. We noticed that the ablatedfeedback model almost reaches that accuracy only after training on the synthetic data with very minor improvement (< 1%) after training on SPLASH. Only using the question and the explanation, the model is able to learn about a set of systematic errors that parsers make and how they can be corrected (Gupta et al., 2017;Yin and Neubig, 2019).
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
So far, we have been using SPLASH for both training and testing. The erroneous parses (and corresponding feedback) in SPLASH are based on the Seq2Struct parser (Shin, 2019   in model architectures (Wang et al., 2020) and pretraining (Yin et al., 2020;Yu et al., 2021a) has led to parsers that already outperform Seq2Struct by more than 30% in parsing accuracy. 3 Here, we ask whether NL-EDIT that we train on SPLASH (and synthetic feedback) can generalize to parsing errors made by more recent parsers without additional parser-specific training data. We follow the same crowdsourcing process used to construct SPLASH (Section 2) to collect three new test sets based on three recent textto-SQL parsers: EditSQL , TaBERT (Yin et al., 2020) and RAT-SQL (Wang et al., 2020). Following Elgohary et al. (2020), we run each parser on SPIDER dev set and only collect feedback for the examples with incorrect parses that can be explained using their SQL explanation 3 https://yale-lily.github.io/spider framework. Table 6 (Top) summarizes the three new test sets and compares them to SPLASH test set. We note that the four datasets are based on the same set of questions and databases (SPIDER dev).
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Table 6 (Bottom) compares the parsing accuracy (measure by exact query match (Yu et al., 2018b)) of each parser when used by itself (No Interaction) to integrating it with NL-EDIT. We report both the accuracy on the examples provided to NL-EDIT (Error Correction) and the End-to-End accuracy on the full SPIDER dev set. NL-EDIT significantly boosts the accuracy of all parsers, but with a notable drop in the gains as the accuracy of the parser improves. To explain that, in Figure 5 we compare the distribution of reference edit size across the four datasets. The figure does not show any significant differences in the distributions that would lead to such a drop in accuracy gain. Likewise, the distributions of the feedback lengths are very similar (the mean is shown in Table 6). As parsers improve in accuracy, they tend to make most of their errors on complex SQL queries. Although the number of errors with each query does not significantly change ( Figure 5), we hypothesize that localizing the errors in a complex initial parse, with a long explanation (Table 6), is the main generalization bottleneck that future work needs to address.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Natural language to SQL: Natural language interfaces to databases have been an active field of study for many years (Woods et al., 1972;Warren and Pereira, 1982;Popescu et al., 2003;Li and Jagadish, 2014). The development of new large scale datasets, such as WikiSQL (Zhong et al., 2017) and SPIDER (Yu et al., 2018b), has reignited the interest in this area with several new models introduced recently (Choi et al., 2020;Wang et al., 2020;Scholak et al., 2020). Another related line of work has focused on conversation semantic parsing, e.g. SParC (Yu et al., 2019b), CoSQL (Yu et al., 2019a), and SMCalFlow (Andreas et al., 2020), where parsers aim at modeling utterance sequentially and in context of previous utterances.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Interactive Semantic Parsing: Several previous studies have looked at the problem of improving semantic parser with feedback or human interactions (Clarke et al., 2010;Artzi and Zettlemoyer, 2013). Interactions are supported in multiple ways including binary correct/incorrect signal (Iyer et al., 2017), answers to a yes/no or a multiple-choice  Table 6: Evaluating the zero-shot generalization of NL-EDIT to different parsers (EditSQL, TaBERT, and RAT-SQL) after training on SPLASH that is constructed based on the Seq2Struct parser. Top: Summary of the dataset constructed based on each parser. Feedback and explanation length is the number of tokens. Bottom: The Error Correction accuracy on each test set and the end-to-end accuracy of each parser on the full SPIDER dev set with and without interaction. ∆ w/ Interaction is the gain in end-to-end accuracy with the interaction added.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
question posed by the system (Yao et al., 2019;Gur et al., 2018) or suggestions of edits that can be applied to the parse . Yao et al. (2019) and Gur et al. (2018) ask yes/no and multiple-choice questions and use the answers in generating the pars. Elgohary et al. (2020) introduce SPLASH (Section 2), a dataset for correcting semantic parsing with natural language feedback. Using language as a medium for providing feedback enables the human to provide rich open-form feedback in their natural way of communication giving them control and flexibility specifying what is wrong and how it should be corrected. Our work uses SPLASH and proposes to pose the problem of semantic parse correction as a parser editing problem with natural language feedback input. This is also related to recent work on casting text generation (e.g. summarization, grammatical error correction, sentence splitting, etc.) as a text editing task (Malmi et al., 2019;Panthaplackel et al., 2020;Stahlberg and Kumar, 2020) where target texts are reconstructed from inputs using several edit operations.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
Semantic Parsing with Synthetic Data: Semantic parsing systems have frequently used synthesized data to alleviate the challenge of labeled data scarcity. In their semantic parser overnight work, Wang et al. (2015) proposed a method for training semantic parsers quickly in a new domain using synthetic data. They generate logical forms and canonical utterances and then paraphrase the canonical utterances via crowd-sourcing. Several other approaches have demonstrated the benefit of adopting this approach to train semantic parsers in low-resource settings (Su et al., 2017;Zhong et al., 2017;Cheng et al., 2018;Xu et al., 2020). Most recently, synthetic data was used to continue to pre-train language models for semantic parsing tasks (Herzig et al., 2020;Yu et al., 2021a,b). We build on this line work by showing that we can generate synthetic data automatically without human involvement to simulate edits between an erroneous parse and a correct one.
NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction
We introduced a model, a data augmentation method, and analysis tools for correcting semantic parse errors in text-to-SQL through natural language feedback. Compared to previous models, our model improves the correction accuracy by 16% and boosts the end-to-end parsing accuracy by up to 20% with only one turn of feedback. Our work creates several avenues for future work: (1) improving the model by better modeling the interaction between the inputs and exploring different patterns for decoder-encoder attention, (2) evaluating existing methods for training with synthetic data (e.g., curriculum learning (Bengio et al., 2009)), (3) optimizing the correction model for better user experience using the progress measure we introduce, and (4) using the SQL edits scheme in other related tasks such as conversational text-to-SQL parsing.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
if the answer is supported or not. 1 This allows for simple binary feedback, and creates a contextual bandit learning scenario (Auer et al., 2002;Langford and Zhang, 2007). Figure 1 illustrates this learning signal and its potential.
Simulating Bandit Learning from User Feedback for Extractive Question Answering
This learning scenario can mitigate fundamental problems in extractive QA. It reduces data collection costs, by delegating much of the learning to interaction with users. It can avoid data collection artifacts because the data comes from the actual system deployment, unlike data from an annotation effort that often involves design decisions immaterial to the system's use case. For example, sharing question-and answer-annotator roles (Rajpurkar et al., 2016), which is detrimental to emulate information seeking behavior (Choi et al., 2018). Finally, it gives systems the potential to evolve over time as the world changes (Lazaridou et al., 2021;Zhang and Choi, 2021).
Simulating Bandit Learning from User Feedback for Extractive Question Answering
Update the model parameters θ using the gradient r (t) ∇ θ log π θ (ŷ (t) |q (t) ,c (t) ) 6: end for Learning Algorithm We learn using policy gradient. Our learner is similar to REINFORCE (Sutton and Barto, 1998;Williams, 2004), but we use arg max to predict answers instead of Monte Carlo sampling from the model's output distribution. 5 We study online and offline learning, also referred to as on-and off-policy. In online learning (Algorithm 1), the model identity is maintained between prediction and update; the parameter values that are updated are the same that were used to generate the output receiving reward. This entails that a reward is only used once, to update the model after observing it. In offline learning (Algorithm 2), this relation between update and prediction does not hold. The learner observes reward, often across many examples, and may use it to update the model many times, even after the parameters drifted arbitrarily far from these that generated the prediction. In practice, we observe reward for the entire length of the simulation (T steps) and then update for E epochs. The reward is re-weighted to provide an unbiased estimation using inverse propensity score (IPS; Horvitz and Thompson, 1952). We clip the debiasing coefficient to avoid amplifying examples with large coefficients (line 10, Algorithm 2).
Simulating Bandit Learning from User Feedback for Extractive Question Answering
In general, offline learning is easier to implement because updating the model is not integrated with its deployment. Offline learning also uses a training loop that is similar to optimization practices in supervised learning. This allows to iterate over the data multiple times, albeit with the same feedback signal on each example. However, online learning often has lower regret as the model is updated after each interaction. It may also lead to higher overall performance, because as the model improves early on, it may observe more positive feedback overall, which is generally more informative. We empiri- 5 Early experiments showed that sampling is not as beneficial as arg max, potentially because of the relatively large output space of extractive QA. Yao et al. (2020) made a similar observation for semantic parsing, and Lawrence et al. (2017) used arg max predictions for bandit learning in statistical machine translation. Table 4 in Appendix A provides our experimental results with sampling.