How have prior methods explored using human feedback to improve dialogue models?
There are a number of existing methods for col-
lecting human feedback from human-model con-
versations. Deployed models can be improved in symmetric conversations conducted between mod-
els and humans during deployment by learning to
mimic human conversationalists, as shown in the
LIGHT dialogue game (Shuster et al., 2020). This
is not directly applicable if the conversations are
asymmetric, for example in the case of one speaker
(human) who asks the questions, and the other (bot)
who always answers, as there would be no human
supervision of the answers. In the non-symmetric
case, one can however try to make use of the tex-
tual response from humans when conversing with
the bot, but alternative learning methods must then
be used. Li et al. (2016b) studies models that learn
how to ask questions in order to learn from the an-
swers, while Li et al. (2016a) learns from general
textual feedback/comments, particularly in the case
where the bot has produced a low quality response.
Another approach is to learn a reward signal (pos-
itive or negative reaction) based on user textual
responses, as shown in the “self-feeding chatbot”
(Hancock et al., 2019). Finally, rather than using
conversational feedback, one can use sophisticated
web-based UIs to collect data, for example stack
ranking potential responses (Ouyang et al., 2022;
Bai et al., 2022).
Learning through Dialogue Interactions by Asking Questions
A good dialogue agent should have the ability to interact with users by both responding to questions and by asking questions, and importantly to learn from both types of interaction. In this work, we explore this direction by designing a simulator and a set of synthetic tasks in the movie domain that allow such interactions between a learner and a teacher. We investigate how a learner can benefit from asking questions in both offline and online reinforcement learning settings, and demonstrate that the learner improves when asking questions. Finally, real experiments with Mechanical Turk validate the approach. Our work represents a first step in developing such end-to-end learned interactive dialogue agents.
[{"first":"Jiwei","last":"Li","middle":[]},{"first":"Alexander","last":"Miller","middle":["H."]},{"first":"Sumit","last":"Chopra","middle":[]},{"first":"Marc'Aurelio","last":"Ranzato","middle":[]},{"first":"Jason","last":"Weston","middle":[]}]
Dialogue Learning With Human-In-The-Loop
An important aspect of developing conversational agents is to give a bot the ability to improve through communicating with humans and to learn from the mistakes that it makes. Most research has focused on learning from fixed training sets of labeled data rather than interacting with a dialogue partner in an online fashion. In this paper we explore this direction in a reinforcement learning setting where the bot improves its question-answering ability from feedback a teacher gives following its generated responses. We build a simulator that tests various aspects of such learning in a synthetic environment, and introduce models that work in this regime. Finally, real experiments with Mechanical Turk validate the approach.
[{"first":"Jiwei","last":"Li","middle":[]},{"first":"Alexander","last":"Miller","middle":["H."]},{"first":"Sumit","last":"Chopra","middle":[]},{"first":"Marc'Aurelio","last":"Ranzato","middle":[]},{"first":"Jason","last":"Weston","middle":[]}]
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Frozen models trained to mimic static datasets can never improve their performance. Models that can employ internet-retrieval for up-to-date information and obtain feedback from humans during deployment provide the promise of both adapting to new information, and improving their performance. In this work we study how to improve internet-driven conversational skills in such a learning framework. We collect deployment data, which we make publicly available, of human interactions, and collect various types of human feedback – including binary quality measurements, free-form text feedback, and fine-grained reasons for failure. We then study various algorithms for improving from such feedback, including standard supervised learning, rejection sampling, model-guiding and reward-based learning, in order to make recommendations on which type of feed- back and algorithms work best. We find the recently introduced DIRECTOR model (Arora et al., 2022) shows significant improvements over other existing approaches.
[{"first":"Jing","last":"Xu","middle":[]},{"first":"Megan","last":"Ung","middle":[]},{"first":"Mojtaba","last":"Komeili","middle":[]},{"first":"Kushal","last":"Arora","middle":[]},{"first":"Y-Lan","last":"Boureau","middle":[]},{"first":"Jason","last":"Weston","middle":[]}]
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
The majority of conversations a dialogue agent sees over its lifetime occur after it has already been trained and deployed, leaving a vast store of potential training signal untapped. In this work, we propose the self-feeding chatbot, a dialogue agent with the ability to extract new training examples from the conversations it participates in. As our agent engages in conversation, it also estimates user satisfaction in its responses. When the conversation appears to be going well, the user’s responses become new training examples to imitate. When the agent believes it has made a mistake, it asks for feedback; learning to predict the feedback that will be given improves the chatbot’s dialogue abilities further. On the PersonaChat chit-chat dataset with over 131k training examples, we find that learning from dialogue with a self-feeding chatbot significantly improves performance, regardless of the amount of traditional supervision.
[{"first":"Braden","last":"Hancock","middle":[]},{"first":"Antoine","last":"Bordes","middle":[]},{"first":"Pierre-Emmanuel","last":"Mazare","middle":[]},{"first":"Jason","last":"Weston","middle":[]}]
Deploying Lifelong Open-Domain Dialogue Learning
Much of NLP research has focused on crowdsourced static datasets and the supervised learning paradigm of training once and then evaluating test performance. As argued in de Vries et al. (2020), crowdsourced data has the issues of lack of naturalness and relevance to real-world use cases, while the static dataset paradigm does not allow for a model to learn from its experiences of using language (Silver et al., 2013). In contrast, one might hope for machine learning systems that become more useful as they interact with people. In this work, we build and deploy a role-playing game, whereby human players converse with learning agents situated in an open-domain fantasy world. We show that by training models on the conversations they have with humans in the game the models progressively improve, as measured by automatic metrics and online engagement scores. This learning is shown to be more efficient than crowdsourced data when applied to conversations with real users, as well as being far cheaper to collect.
[{"first":"Kurt","last":"Shuster","middle":[]},{"first":"Jack","last":"Urbanek","middle":[]},{"first":"Emily","last":"Dinan","middle":[]},{"first":"Arthur","last":"Szlam","middle":[]},{"first":"Jason","last":"Weston","middle":[]}]
Learning through Dialogue Interactions by Asking Questions
When a student is asked a question by a teacher, but is not confident about the answer, they may ask for clarification or hints. A good conversational agent (a learner/bot/student) should have this ability to interact with a dialogue partner (the teacher/user). However, recent efforts have mostly focused on learning through fixed answers provided in the training set, rather than through interactions. In that case, when a learner encounters a confusing situation such as an unknown surface form (phrase or structure), a semantically complicated sentence or an unknown word, the agent will either make a (usually poor) guess or will redirect the user to other resources (e.g., a search engine, as in Siri). Humans, in contrast, can adapt to many situations by asking questions.
Learning through Dialogue Interactions by Asking Questions
We identify three categories of mistakes a learner can make during dialogue 1 : (1) the learner has problems understanding the surface form of the text of the dialogue partner, e.g., the phrasing of a question; (2) the learner has a problem with reasoning, e.g. they fail to retrieve and connect the relevant knowledge to the question at hand; (3) the learner lacks the knowledge necessary to answer the question in the first place -that is, the knowledge sources the student has access to do not contain the needed information.
Learning through Dialogue Interactions by Asking Questions
All the situations above can be potentially addressed through interaction with the dialogue partner. Such interactions can be used to learn to perform better in future dialogues. If a human student has problems understanding a teacher's question, they might ask the teacher to clarify the question. If the student doesn't know where to start, they might ask the teacher to point out which known facts are most relevant. If the student doesn't know the information needed at all, they might ask the teacher to tell them the knowledge they're missing, writing it down for future use.
Learning through Dialogue Interactions by Asking Questions
In this work, we try to bridge the gap between how a human and an end-to-end machine learning dialogue agent deal with these situations: our student has to learn how to learn. We hence design a simulator and a set of synthetic tasks in the movie question answering domain that allow a bot to interact with a teacher to address the issues described above. Using this framework, we explore how a bot can benefit from interaction by asking questions in both offline supervised settings and online reinforcement learning settings, as well as how to choose when to ask questions in the latter setting. In both cases, we find that the learning system improves through interacting with users.
Learning through Dialogue Interactions by Asking Questions
Learning language through interaction and feedback can be traced back to the 1950s, when Wittgenstein argued that the meaning of words is best understood from their use within given language games (Wittgenstein, 2010). The direction of interactive language learning through language games has been explored in the early seminal work of Winograd (Winograd, 1972), and in the recent SHRD-LURN system (Wang et al., 2016). In a broader context, the usefulness of feedback and interactions has been validated in the setting of multiple language learning, such as second language learning (Bassiri, 2011) and learning by students (Higgins et al., 2002;Latham, 1997;Werts et al., 1995).
Learning through Dialogue Interactions by Asking Questions
In the context of dialogue, with the recent popularity of deep learning models, many neural dialogue systems have been proposed. These include the chit-chat type end-to-end dialogue systems (Vinyals & Le, 2015;Li et al., 2015;Sordoni et al., 2015), which directly generate a response given the previous history of user utterance. It also include a collection of goal-oriented dialogue systems , which complete a certain task such as booking a ticket or making a reservation at a restaurant. Another line of research focuses on supervised learning for question answering from dialogues (Dodge et al., 2015;Weston, 2016), using either a given database of knowledge Miller et al., 2016) or short stories . As far as we know, current dialogue systems mostly focus on learning through fixed supervised signals rather than interacting with users.
Learning through Dialogue Interactions by Asking Questions
Our work is closely related to the recent work of Weston (2016), which explores the problem of learning through conducting conversations, where supervision is given naturally in the response during the conversation. Their work introduced multiple learning schemes from dialogue utterances. In particular the authors discussed Imitation Learning, where the agent tries to learn by imitating the dialogue interactions between a teacher and an expert student; Reward-Based Imitation Learning, which only learns by imitating the dialogue interactions which have have correct answers; and Forward Prediction, which learns by predicting the teacher's feedback to the student's response. Despite the fact that Forward Prediction does not uses human-labeled rewards, the authors show that it yields promising results. However, their work did not fully explore the ability of an agent to learn via questioning and interaction. Our work can be viewed as a natural extension of theirs.
Learning through Dialogue Interactions by Asking Questions
In this section we describe the dialogue tasks we designed 2 . They are tailored for the three different situations described in Section 1 that motivate the bot to ask questions: (1) Question Clarification, in which the bot has problems understanding its dialogue partner's text; (2) Knowledge Operation, in which the bot needs to ask for help to perform reasoning steps over an existing knowledge base; and (3) Knowledge Acquisition, in which the bot's knowledge is incomplete and needs to be filled.
Learning through Dialogue Interactions by Asking Questions
For our experiments we adapt the WikiMovies dataset , which consists of roughly 100k questions over 75k entities based on questions with answers in the open movie dataset (OMDb). The training/dev/test sets respectively contain 181638 / 9702 / 9698 examples. The accuracy metric corresponds to the percentage of times the student gives correct answers to the teacher's questions.
Learning through Dialogue Interactions by Asking Questions
Each dialogue takes place between a teacher and a bot. In this section we describe how we generate tasks using a simulator. Section 4.2 discusses how we test similar setups with real data using Mechanical Turk.
Learning through Dialogue Interactions by Asking Questions
The bot is first presented with facts from the OMDb KB. This allows us to control the exact knowledge the bot has access to. Then, we include several teacher-bot question-answer pairs unrelated to the question the bot needs to answer, which we call conversation histories 3 . In order to explore the benefits of asking clarification questions during a conversation, for each of the three scenarios, our simulator generated data for two different settings, namely, Question-Answering (denoted by QA), and Asking-Question (denoted by AQ). For both QA and AQ, the bot needs to give an answer to the teacher's original question at the end. The details of the simulator can be found in the appendix.
Learning through Dialogue Interactions by Asking Questions
In this setting, the bot does not understand the teacher's question. We focus on a special situation where the bot does not understand the teacher because of typo/spelling mistakes, as shown in Figure  1. We intentionally misspell some words in the questions such as replacing the word "movie" with "movvie" or "star" with "sttar". 4 To make sure that the bot will have problems understanding the question, we guarantee that the bot has never encountered the misspellings before-the misspellingintroducing mechanisms in the training, dev and test sets are different, so the same word will be misspelled in different ways in different sets. We present two AQ tasks: (i) Question Paraphrase where the student asks the teacher to use a paraphrase that does not contain spelling mistakes to clarify the question by asking "what do you mean?"; and (ii) Question Verification where the student asks the teacher whether the original typo-bearing question corresponds to another question without the spelling mistakes (e.g., "Do you mean which film did Tom Hanks appear in?"). The teacher will give feedback by giving a paraphrase of the original question without spelling mistakes (e.g., "I mean which film did Tom Hanks appear in") in Question Paraphrase or positive/negative feedback in Question Verification. Next the student will give an answer and the teacher will give positive/negative feedback depending on whether the student's answer is correct. Positive and negative feedback are variants of "No, that's incorrect" or "Yes, that's right" 5 . In these tasks, the bot has access to all relevant entries in the KB.
Learning through Dialogue Interactions by Asking Questions
The bot has access to all the relevant knowledge (facts) but lacks the ability to perform necessary reasoning operations over them; see Figure 2. We focus on a special case where the bot will try to understand what are the relevant facts. We explore two settings: Ask For Relevant Knowledge (Task 3) where the bot directly asks the teacher to point out the relevant KB fact and Knowledge Verification (Task 4) where the bot asks whether the teacher's question is relevant to one particular KB fact. The teacher will point out the relevant KB fact in the Ask For Relevant Knowledge setting or give a positive or negative response in the Knowledge Verification setting. Then the bot will give an answer to the teacher's original question and the teacher will give feedback on the answer.
Learning through Dialogue Interactions by Asking Questions
For the tasks in this subsection, the bot has an incomplete KB and there are entities important to the dialogue missing from it, see Figure 3. For example, given the question "Which movie did Tom Hanks star in?", the missing part could either be the entity that the teacher is asking about (question entity for short, which is Tom Hanks in this example), the relation entity (starred actors), the answer to the question (Forrest Gump), or the combination of the three. In all cases, the bot has little chance of giving the correct answer due to the missing knowledge. It needs to ask the teacher the answer to acquire the missing knowledge. The teacher will give the answer and then move on to other questions (captured in the conversational history). They later will come back to reask the question. At this point, the bot needs to give an answer since the entity is not new any more.
Learning through Dialogue Interactions by Asking Questions
Though the correct answer has effectively been included in the earlier part of the dialogue as the answer to the bot's question, as we will show later, many of the tasks are not as trivial as they look when the teacher reasks the question. This is because the bot's model needs to memorize the missing entity and then construct the links between the missing entities and known ones. This is akin to the real world case where a student might make the same mistake again and again even though each time the teacher corrects them if their answer is wrong. We now detail each task in turn.  Figure 1: Sample dialogues for the two Question Clarification tasks (rows) using both the traditional QA setting (left column) and AQ setting (right column). In each case the same example is given for simplicity. Black text prefixed by "kb:" denotes KB knowledge that the student has access to. Blue text is spoken by the teacher, denoted by T. (+) denotes a reward of 1 (and 0 otherwise) that the teacher assigns to the bot. Red text denotes responses or questions posed by the bot, denoted by S. Brown denotes typos deliberately introduced by the authors. For the Question Verification setting, the student can either ask a correct (pertinent) question (as in this example) or an incorrect (irrelevant) one. The teacher will give positive or negative feedback based on the correctness of the student's question. In our offline superised learning experiments, the probability of asking pertinent questions and correctly answering the original question from the teacher is set to 0.5. Finally, T/S denotes 5 pairs of questions and answers that are irrelevant to the rest of the conversation.  Missing Question Entity: The entity that the teacher is asking about is missing from the knowledge base. All KB facts containing the question entity will be hidden from the bot. In the example for Task 5 in Figure 3, since the teacher's question contains the entity Tom Hanks, the KB facts that contain Tom Hanks are hidden from the bot. Missing Answer Entity: The answer entity to the question is unknown to the bot. All KB facts that contain the answer entity will be hidden. Hence, in Task 6 of Figure 3, all KB facts containing the answer entity Forrest Gump will be hidden from the bot.
Learning through Dialogue Interactions by Asking Questions
Missing Relation Entity: The relation type is unknown to the bot. In Task 7 of Figure 3, all KB facts that express the relation starred actors are hidden from the bot.
Learning through Dialogue Interactions by Asking Questions
Missing Triples: The triple that expresses the relation between the question entity and the answer entity is hidden from the bot. In Task 8 of Figure 3, the triple "Forrest Gump (question entity) starred actors Tom Hanks (answer entity)" will be hidden.
Learning through Dialogue Interactions by Asking Questions
Missing Everything: The question entity, the relation entity, the answer entity are all missing from the KB. All KB facts in Task 9 of Figure 3 will be removed since they either contain the relation entity (i.e., starred actors), the question entity (i.e., Forrest Gump) or the answer entity Tom Hanks.
Learning through Dialogue Interactions by Asking Questions
We now discuss in detail the regimes we used to train and test our models, which are divided between evaluation within our simulator and using real data collected via Mechanical Turk.
Learning through Dialogue Interactions by Asking Questions
Using our simulator, our objective was twofold. We first wanted to validate the usefulness of asking questions in all the settings described in Section 3. Second, we wanted to assess the ability of our student bot to learn when to ask questions. In order to accomplish these two objectives we explored training our models with our simulator using two methodologies, namely, Offline Supervised Learning and Online Reinforcement Learning.
Learning through Dialogue Interactions by Asking Questions
The motivation behind training our student models in an offline supervised setting was primarily to test the usefulness of the ability to ask questions. The dialogues are generated as described in the previous section, and the bot's role is generated with a fixed policy. We chose a policy where answers to the teacher's questions are correct answers 50% of the time, and incorrect otherwise, to add a degree of realism. Similarly, in tasks where questions can be irrelevant they are only asked correctly 50% of the time. 6
Learning through Dialogue Interactions by Asking Questions
The offline setting explores different combinations of training and testing scenarios, which mimic different situations in the real world. The aim is to understand when and how observing interactions between two agents can help the bot improve its performance for different tasks. As a result we construct training and test sets in three ways across all tasks, resulting in 9 different scenarios per task, each of which correspond to a real world scenario.
Learning through Dialogue Interactions by Asking Questions
The three training sets we generated are referred to as TrainQA, TrainAQ, and TrainMix. TrainQA follows the QA setting discussed in the previous section: the bot never asks questions and only tries to immediately answer. TrainAQ follows the AQ setting: the student, before answering, first always asks a question in response to the teacher's original question. TrainMix is a combination of the two where 50% of time the student asks a question and 50% does not.
Learning through Dialogue Interactions by Asking Questions
The three test sets we generated are referred to as TestQA, TestAQ, and TestModelAQ. TestQA and TestAQ are generated similarly to TrainQA and TrainAQ, but using a perfect fixed policy (rather than 50% correct) for evaluation purposes. In the TestModelAQ setting the model has to get the form of the question correct as well. In the Question Verification and Knowledge Verification tasks there are many possible ways of forming the question and some of them are correct -the model has to choose the right question to ask. E.g. it should ask "Does it have something to do with the fact that Larry Crowne directed by Tom Hanks?"rather than "Does it have something to do with the fact that Forrest Gump directed by Robert Zemeckis?" when the latter is irrelevant (the candidate list of questions is generated from the known knowledge base entries with respect to that question). The policy is trained using either the TrainAQ or TrainMix set, depending on the training scenario. The teacher will reply to the question, giving positive feedback if the student's question is correct and no response and negative feedback otherwise. The student will then give the final answer. The difference between TestModelAQ and TestAQ only exists in the Question Verification and Knowledge Verification tasks; in other tasks there is only one way to ask the question and TestModelAQ and TestAQ are identical.
Learning through Dialogue Interactions by Asking Questions
To summarize, for every task listed in Section 3 we train one model for each of the three training sets (TrainQA, TrainAQ, TrainMix) and test each of these models on the three test sets (TestQA, TestAQ, and TestModelAQ), resulting in 9 combinations. For the purpose of notation the train/test combination is denoted by "TrainSetting+TestSetting". For example, TrainAQ+TestQA denotes a model which is trained using the TrainAQ dataset and tested on TestQA dataset. Each combination has a real world interpretation. For instance, TrainAQ+TestQA would refer to a scenario where a student can ask the teacher questions during learning but cannot to do so while taking an exam. Similarly, TrainQA+TestQA describes a stoic teacher that never answers a student's question at either learning or examination time. The setting TrainQA+TestAQ corresponds to the case where a lazy student never asks question at learning time but gets anxious during the examination and always asks a question.
Learning through Dialogue Interactions by Asking Questions
We also explored scenarios where the student learns the ability to decide when to ask a question. In other words, the student learns how to learn.
Learning through Dialogue Interactions by Asking Questions
Although it is in the interest of the student to ask questions at every step of the conversation, since the response to its question will contain extra information, we don't want our model to learn this behavior. Each time a human student asks a question, there's a cost associated with that action. This cost is a reflection of the patience of the teacher, or more generally of the users interacting with the bot in the wild: users won't find the bot engaging if it always asks clarification questions. The student should thus be judicious about asking questions and learn when and what to ask. For instance, if the student is confident about the answer, there is no need for it to ask. Or, if the teacher's question is so hard that clarification is unlikely to help enough to get the answer right, then it should also refrain from asking.
Learning through Dialogue Interactions by Asking Questions
We now discuss how we model this problem under the Reinforcement Learning framework. The bot is presented with KB facts (some facts might be missing depending on the task) and a question. It needs to decide whether to ask a question or not at this point. The decision whether to ask is made by a binary policy P RLQuestion . If the student chooses to ask a question, it will be penalized by cost AQ . We explored different values of cost AQ ranging from [0, 2], which we consider as modeling the patience of the teacher. The goal of this setting is to find the best policy for asking/notasking questions which would lead to the highest cumulative reward. The teacher will appropriately reply if the student asks a question. The student will eventually give an answer to the teacher's initial question at the end using the policy P RLAnswer , regardless of whether it had asked a question. The student will get a reward of +1 if its final answer is correct and −1 otherwise. Note that the student can ask at most one question and that the type of question is always specified by the task under consideration. The final reward the student gets is the cumulative reward over the current dialogue episode. In particular the reward structure we propose is the following:
Learning through Dialogue Interactions by Asking Questions
Asking Question Not asking Question Final Answer Correct 1-costAQ 1 Final Answer Incorrect -1-costAQ -1 For each of the tasks described in Section 3, we consider three different RL scenarios. Good-Student: The student will be presented with all relevant KB facts. There are no misspellings or unknown words in the teacher's question. This represents a knowledgable student in the real world that knows as much as it needs to know (e.g., a large knowledge base, large vocabulary). This setting is identical across all missing entity tasks (5 -9).
Learning through Dialogue Interactions by Asking Questions
Poor-Student: The KB facts or the questions presented to the student are flawed depending on each task. For example, for the Question Clarification tasks, the student does not understand the question due to spelling mistakes. For the Missing Question Entity task the entity that the teacher asks about is unknown by the student and all facts containing the entity will be hidden from the student. This setting is similar to a student that is underprepared for the tasks. Medium-Student: The combination of the previous two settings where for 50% of the questions, the student has access to the full KB and there are no new words or phrases or entities in the question, and 50% of the time the question and KB are taken from the Poor-Student setting.
Learning through Dialogue Interactions by Asking Questions
Finally, to validate our approach beyond our simulator by using real language, we collected data via Amazon Mechanical Turk. Due to the cost of data collection, we focused on real language versions of Tasks 4 (Knowledge Verification) and 8 (Missing Triple), see Secs. 3.2 and 3.3 for the simulator versions. That is, we collect dialoguess and use them in an offline supervised learning setup similar to Section 4.1.1. This setup allows easily reproducibile experiments.
Learning through Dialogue Interactions by Asking Questions
For Mechanical Turk Task 4, the bot is asked a question by a human teacher, but before answering can ask the human if the question is related to one of the facts it knows about from its memory. It is then required to answer the original question, after some additional dialog turns relating to other question/answer pairs (called "conversational history", as before). For Task 8, the bot is asked a question by a human but lacks the triple in its memory that would be needed to answer it. It is allowed to ask for the missing information, the human responds to the question in free-form language. The bot is then required to answer the original question, again after some "conversational history" has transpired.
Learning through Dialogue Interactions by Asking Questions
We collect around 10,000 episodes (dialogues) for training, 1000 for validation, and 2500 for testing for each of the two tasks. In each case, we give instructions to the Turkers that still follow the original form of the task, but make the tasks contain realistic language written by humans. The instructions given to the Turkers are given in the appendix.
Learning through Dialogue Interactions by Asking Questions
For both tasks, while the human turkers replace the simulator that the bot was previously conversing with, the bot's dialogue actions (capabilities) are essentially unchanged from before. That is, when answering questions, now the bot is required to answer a human's questions rather than templated questions from the simulator. When the bot is asking questions, the bot still asks in the same form as before, e.g. questions like "Does it have something to do with X" for Task 4 or "I don't know.
Learning through Dialogue Interactions by Asking Questions
What's the answer?" for Task 8. However, now its questions are answered by a human. In both cases (asking and answering) the human data is richer with potentially more complex language and lexical variability. Examples of the collected dialogues are given in Figure 5.  
Learning through Dialogue Interactions by Asking Questions
For both offline supervised and online RL settings, we use the End-to-End Memory Network model (MemN2N) (Sukhbaatar et al., 2015) as a backbone. The model takes as input the last utterance of the dialogue history (the question from the teacher) as well as a set of memory contexts including short-term memories (the dialogue history between the bot and the teacher) and long-term memories (the knowledge base facts that the bot has access to), and outputs a label. We refer readers to the Appendix for more details about MemN2N.
Learning through Dialogue Interactions by Asking Questions
Offline Supervised Settings: The first learning strategy we adopt is the reward-based imitation strategy (denoted vanilla-MemN2N) described in (Weston, 2016), where at training time, the model maximizes the log likelihood probability of the correct answers the student gave (examples with incorrect final answers are discarded). Candidate answers are words that appear in the memories, which means the bot can only predict the entities that it has seen or known before.
Learning through Dialogue Interactions by Asking Questions
We also use a variation of MemN2N called "context MemN2N" (Cont-MemN2N for short) where we replace each word's embedding with the average of its embedding (random for unseen words) and the embeddings of the other words that appear around it. We use both the preceeding and following words as context and the number of context words is a hyperparameter selected on the dev set.
Learning through Dialogue Interactions by Asking Questions
An issue with both vanilla-MemN2N and Cont-MemN2N is that the model only makes use of the bot's answers as signals and ignores the teacher's feedback. We thus propose to use a model that jointly predicts the bot's answers and the teacher's feedback (denoted as TrainQA (+FP)). The bot's answers are predicted using a vanilla-MemN2N and the teacher's feedback is predicted using the Forward Prediction (FP) model as described in (Weston, 2016). We refer the readers to the Appendix for the FP model details. At training time, the models learn to jointly predict the teacher's feedback and the answers with positive reward. At test time, the model will only predict the bot's answer.
Learning through Dialogue Interactions by Asking Questions
For the TestModelAQ setting described in Section 4, the model needs to decide the question to ask. Again, we use vanilla-MemN2N that takes as input the question and contexts, and outputs the question the bot will ask.
Learning through Dialogue Interactions by Asking Questions
Online RL Settings: A binary vanilla-MemN2N (denoted as P RL (Question)) is used to decide whether the bot should or should not ask a question, with the teacher replying if the bot does ask something. A second MemN2N is then used to decide the bot's answer, denoted as P RL (Answer). P RL (Answer) for QA and AQ are two separate models, which means the bot will use different models for final-answer prediction depending on whether it chooses to ask a question or not. 7
Learning through Dialogue Interactions by Asking Questions
We use the REINFORCE algorithm (Williams, 1992) to update P RL (Question) and P RL (Answer). For each dialogue, the bot takes two sequential actions (a 1 , a 2 ): to ask or not to ask a question (denoted as a 1 ); and guessing the final answer (denoted as a 2 ). Let r(a 1 , a 2 ) denote the cumulative reward for the dialogue episode, computed using Table 1. The gradient to update the policy is given by:
Learning through Dialogue Interactions by Asking Questions
p(a 1 , a 2 ) = P RL (Question)(a 1 ) · P RL (answer)(a 2 ) ∇J(θ) ≈ ∇ log p(a 1 , a 2 )[r(a 1 , a 2 ) − b]
Learning through Dialogue Interactions by Asking Questions
where b is the baseline value, which is estimated using another MemN2N model that takes as input the query x and memory C, and outputs a scalar b denoting the estimation of the future reward. The baseline model is trained by minimizing the mean squared loss between the estimated reward b and actual cumulative reward r, ||r − b|| 2 . We refer the readers to (Ranzato et al., 2015;Zaremba & Sutskever, 2015) for more details. The baseline estimator model is independent from the policy models and the error is not backpropagated back to them.
Learning through Dialogue Interactions by Asking Questions
In practice, we find the following training strategy yields better results: first train only P RL (answer), updating gradients only for the policy that predicts the final answer. After the bot's final-answer policy is sufficiently learned, train both policies in parallel 8 . This has a real-world analogy where the bot first learns the basics of the task, and then learns to improve its performance via a question-asking policy tailored to the user's patience (represented by cost AQ ) and its own ability to asnwer questions.   Tables 2, 7 and 8 (the latter two are in the appendix). Table 7 presents results for the vanilla-MemN2N and Forward Prediction models. Table 2 presents results for Cont-MemN2N, which is better at handling unknown words. We repeat each experiment 10 times and report the best result. Finally, Table 8 presents results for the test scenario where the bot itself chooses when to ask questions. Observations can be summarized as as follows:
Learning through Dialogue Interactions by Asking Questions
-Asking questions helps at test time, which is intuitive since it provides additional evidence:
Learning through Dialogue Interactions by Asking Questions
• TrainAQ+TestAQ (questions can be asked at both training and test time) performs the best across all the settings.
Learning through Dialogue Interactions by Asking Questions
• TrainQA+TestAQ (questions can be asked at training time but not at test time) performs worse than TrainQA+TestQA (questions can be asked at neither training nor test time) in tasks Question Clarification and Knowledge Operation due to the discrepancy between training and testing.
Learning through Dialogue Interactions by Asking Questions
• TrainQA+TestAQ performs better than TrainQA+TestQA on all Knowledge Acquisition tasks, the only exception being the Cont-MemN2N model on the Missing Triple setting. The explanation is that for most tasks in Knowledge Acquisition, the learner has no chance of giving the correct answer without asking questions. The benefit from asking is thus large enough to compensate for the negative effect introduced by data discrepancy between training and test time.
Learning through Dialogue Interactions by Asking Questions
• TrainMix offers flexibility in bridging the gap between datasets generated using QA and AQ, very slightly underperforming TrainAQ+TestAQ, but gives competitive results on both TestQA and TestAQ in the Question Clarification and Knowledge Operations tasks.
Learning through Dialogue Interactions by Asking Questions
• TrainAQ+TestQA (allowing questions at training time but forbid questions at test time) performs the worst, even worse than TrainQA+TestQA. This has a real-world analogy where a student becomes dependent on the teacher answering their questions, later struggling to answer the test questions without help.
Learning through Dialogue Interactions by Asking Questions
• In the Missing Question Entity task (the student does not know about the question entity), the Missing Answer Entity task (the student does not know about the answer entity), and Missing Everything task, the bot achieves accuracy less than 0.01 if not asking questions at test time (i.e., TestQA).
Learning through Dialogue Interactions by Asking Questions
• The performance of TestModelAQ, where the bot relies on its model to ask questions at test time (and thus can ask irrelevant questions) performs similarly to asking the correct question at test time (TestAQ) and better than not asking questions (TestQA).
Learning through Dialogue Interactions by Asking Questions
-Cont-MemN2N significantly outperforms vanilla-MemN2N. One explanation is that considering context provides significant evidence distinguishing correct answers from candidates in the dialogue history, especially in cases where the model encounters unfamiliar words.
Learning through Dialogue Interactions by Asking Questions
For the RL settings, we present results for Task 2 (Question Verification) and Task 6 (Missing Answer Entities) in Figure 6. Task 2 represents scenarios where different types of student  Task 2 and Task 6 have different abilities to correctly answer questions (e.g., a poor student can still sometimes give correct answers even when they do not fully understand the question). Task 6 represents tasks where a poor learner who lacks the knowledge necessary to answer the question can hardly give a correct answer. All types of students including the good student will theoretically benefit from asking questions (asking for the correct answer) in Task 6. We show the percentage of question-asking versus the cost of AQ on the test set and the accuracy of question-answering on the test set vs the cost of AQ. Our main findings were:
Learning through Dialogue Interactions by Asking Questions
• A good student does not need to ask questions in Task 2 (Question Verification), because they already understand the question. The student will raise questions asking for the correct answer when cost is low for Task 6 (Missing Answer Entities). • A poor student always asks questions when the cost is low. As the cost increases, the frequency of question-asking declines. • As the AQ cost increases gradually, good students will stop asking questions earlier than the medium and poor students. The explanation is intuitive: poor students benefit more from asking questions than good students, so they continue asking even with higher penalties. • As the probability of question-asking declines, the accuracy for poor and medium students drops. Good students are more resilient to not asking questions.
Learning through Dialogue Interactions by Asking Questions
Results for the Mechanical Turk Tasks are given in Table 3. We again compare vanilla-MemN2N and Cont-MemN2N, using the same TrainAQ/TrainQA and TestAQ/TestQA combinations as before, for Tasks 4 and 8 as described in Section 4.2. We tune hyperparameters on the validation set and repeat each experiment 10 times and report the best result.
Learning through Dialogue Interactions by Asking Questions
While performance is lower than on the related Task 4 and Task 8 simulator tasks, we still arrive at the same trends and conclusions when real data from humans is used. The performance was expected to be lower because (i) real data has more lexical variety, complexity and noise; and (ii) the training set was smaller due to data collection costs (10k vs. 180k). We perform an analysis of the difference between simulated and real training data (or combining the two) in the appendix, which shows that using real data is indeed important and measurably superior to using simulated data.  More importantly, the same main conclusion is observed as before: TrainAQ+TestAQ (questions can be asked at both training and test time) performs the best across all the settings. That is, we show that a bot asking questions to humans learns to outperform one that only answers them.
Learning through Dialogue Interactions by Asking Questions
In this paper, we explored how an intelligent agent can benefit from interacting with users by asking questions. We developed tasks where interaction via asking questions is desired. We explore both online and offline settings that mimic different real world situations and show that in most cases, teaching a bot to interact with humans facilitates language understanding, and consequently leads to better question answering ability.
Learning through Dialogue Interactions by Asking Questions
In the first step, the query x is transformed to a vector representation u 0 by summing up its constituent word embeddings: u 0 = Ax. The input x is a bag-of-words vector and A is the d × V word embedding matrix where d denotes the vector dimensionality and V denotes the vocabulary size. Each memory c i is similarly transformed to vector m i . The model will read information from the memory by linking input representation q with memory vectors m i using softmax weights:
Learning through Dialogue Interactions by Asking Questions
The goal is to select memories relevant to the last utterance x, i.e., the memories with large values of p 1 i . The queried memory vector o 1 is the weighted sum of memory vectors. The queried memory vector o 1 will be added on top of original input, u 1 = o 1 + u 0 . u 1 is then used to query the memory vector. Such a process is repeated by querying the memory N times (so called "hops"). N is set to three in all experiments in this paper.
Learning through Dialogue Interactions by Asking Questions
In the end, u N is input to a softmax function for the final prediction:
Learning through Dialogue Interactions by Asking Questions
where L denotes the number of candidate answers and y denotes the representation of the answer. If the answer is a word, y is the corresponding word embedding. If the answer is a sentence, y is the embedding for the sentence achieved in the same way as we obtain embeddings for query x and memory c.
Learning through Dialogue Interactions by Asking Questions
Reward Based Imitation (RBI) and Forward Prediction (FP) RBI and FP are two dialogue learning strategies proposed in (Weston, 2016) by harnessing different types of dialogue signals. RBI handles the case where the reward or the correctness of a bot's answer is explicitly given (for example, +1 if the bot's answer is correct and 0 otherwise). The model is directly trained to predict the correct answers (with label 1) at training time, which can be done using End-to-End Memory Networks (MemN2N) (Sukhbaatar et al., 2015) that map a dialogue input to a prediction.
Learning through Dialogue Interactions by Asking Questions
FP handles the situation where a real-valued reward for a bot's answer is not available, meaning that there is no +1 or 0 labels paired with a student's utterance. However, the teacher will give a response to the bot's answer, taking the form of a dialogue utterance. More formally, suppose that x denotes the teacher's question and C=c 1 , c 2 , ..., c N denotes the dialogue history. In our AQ settings, the bot will ask a question a regarding the teacher's question, denoted as a ∈ A, where A denotes the student's question pool. The teacher will provide an utterance in response to the student question a. In FP, the model first maps the teacher's initial question x and dialogue history C to vector representation u using a memory network with multiple hops. Then the model will perform another hopof attention over all possible student's questions in A, with an additional part that incorporates the information of which candidate (i.e., a) was actually selected in the dialogue:
Learning through Dialogue Interactions by Asking Questions
where yâ denotes the vector representation for the student's question candidateâ. β is a ddimensional vector to signify the actual action a that the student chooses. For tasks where the student only has one way to ask questions (e.g., "what do you mean"), there is no need to perform hops of attention over candidates since the cardinality of A is just 1. We thus directly assign a probability of 1 to the student's question, making o the sum of vector representation of y a and β.
Learning through Dialogue Interactions by Asking Questions
o is then combined with u to predict the teacher's feedback t using a softmax:
Learning through Dialogue Interactions by Asking Questions
where x ri denotes the embedding for the i th response.
Learning through Dialogue Interactions by Asking Questions
Dialogue Simulator In this section we further detail the simulator and the datasets we generated in order to realize the various scenarios discussed in Section 3. We focused on the problem of movieQA where we adapted the WikiMovies dataset proposed in . The dataset consists of roughly 100k questions with over 75k entities from the open movie dataset (OMDb).
Learning through Dialogue Interactions by Asking Questions
Each dialogue generated by the simulator takes place between a student and a teacher. The simulator samples a random question from the WikiMovies dataset and fetches the set of all KB facts relevant to the chosen question. This question is assumed to be the one the teacher asks its student, and is referred to as the "original" question. The student is first presented with the relevant KB facts followed by the original question. Providing the KB facts to the student allows us to control the exact knowledge the student is given access to while answering the questions. At this point, depending on the task at hand and the student's ability to answer, the student might choose to directly answer it or ask a "followup" question. The nature of the followup question will depend on the scenario under consideration. If the student answers the question, it gets a response from the teacher about its correctness and the conversation ends. However if the student poses a followup question, the teacher gives an appropriate response, which should give additional information to the student to answer the original question. In order to make things more complicated, the simulator pads the conversation with several unrelated student-teacher question-answer pairs. These question-answer pairs can be viewed as distractions and are used to test the student's ability to remember the additional knowledge provided by the teacher after it was queried. For each dialogue, the simulator incorporates 5 such pairs (10 sentences). We refer to these pairs as conversational histories.
Learning through Dialogue Interactions by Asking Questions
For the QA setting (see Section 3), the dialogues generated by the simulator are such that the student never asks a clarification question. Instead, it simply responds to the original question, even if it is wrong. For the dialogs in the AQ setting, the student always asks a clarification question. The nature of the question asked is dependent on the scenario (whether it is Question Clarification, Knowledge Operation, or Knowledge Acquisition) under consideration. In order to simulate the case where the student sometimes choses to directly answer the original question and at other times choses to ask question, we created training datasets, which were a combination of QA and AQ (called "Mixed"). For all these cases, the student needs to give an answer to the teacher's original question at the end.
Learning through Dialogue Interactions by Asking Questions
These are the instructions given for the textual feedback Mechanical Turk task (we also constructed a separate task to collect the questions to ask the bot with similar instructions, not described here):
Learning through Dialogue Interactions by Asking Questions
Task 4 (answers to bot's questions):
Learning through Dialogue Interactions by Asking Questions
Title: Write brief responses to given dialogue exchanges (about 15 min) Description: Write a brief response answering a provided question (25 questions per HIT).
Learning through Dialogue Interactions by Asking Questions
Each task consists of the following triplets: 1) a question by the teacher 2) the correct answer(s) to the question (separated by "OR"), unknown to the student 3) a clarifying question asking for feedback from the teacher Consider the scenario where you are the teacher and have already asked the question, and received the reply from the student. Please compose a brief response replying to the student's question. The correct answers are provided so that you know whether the student's question was relevant or not. For example, given 1) question: "what is a color in the united states flag?"; 2) correct answer: "white OR blue OR red"; 3) student reply: "does this have to do with 'US Flag has colors red,white,blue'?", your response could be something like "that's right!"; for 3) reply: "does this have to do with 'United States has population 320 million", you might say "No, that fact is not relevant" or "Not really". Please vary responses and try to minimize spelling mistakes. If the same responses are copied/pasted or similar responses are overused, we'll reject the HIT. Avoid naming the student or addressing "the class" directly. We will consider bonuses for higher quality responses during review.
Learning through Dialogue Interactions by Asking Questions
Task 8: answers to bot's questions:
Learning through Dialogue Interactions by Asking Questions
Title: Write brief responses to given dialogue exchanges (about 10 min)
Learning through Dialogue Interactions by Asking Questions
Description: Write a sentence describing the answer to a question (25 questions per HIT).
Learning through Dialogue Interactions by Asking Questions
Each task consists of the following triplets: 1) a question by the teacher 2) the correct answer(s) to the question (separated by "OR"), unknown to the student 3) a question from the student asking the teacher for the answer Consider the scenario where you are the teacher and have already asked the question, and received the reply from the student. Please compose a brief response replying to the student's question. The correct answers are provided so that you know which answers to provide. For example, given 1) question: "what is a color in the united states flag?"; 2) correct answer: "white OR blue OR red"; 3) student reply: "i dont know. what's the answer ?", your response could be something like "the color white is in the US flag" or "blue and red both appear in it". Please vary responses and try to minimize spelling mistakes, and do not include the capitalized "OR" in your response. If the same responses are copied/pasted or similar responses are overused, we'll reject the HIT. You don't need to mention every correct answer in your response. Avoid naming the student or addressing "the class" directly. We will consider bonuses for higher quality responses during review.
Learning through Dialogue Interactions by Asking Questions
Here we provide additional experiments to supplement the ones described in Section 6.2. In the main paper, results were shown when training and testing on the collected Mechanical Turk data (around 10,000 episodes of training dialogues for training). As we collected the data in the same settings as Task 4 and 8 of our simulator, we could also consider supplementing training with simulated data as well, of which we have a larger amount (over 100,000 episodes). Note this is only for training, we will still test on the real (Mechanical Turk collected) data. Although the simulated data has less lexical variety as it is built from templates, the larger size might obtain improve results.
Learning through Dialogue Interactions by Asking Questions
Results are given Table 5 when training on the combination of real and simulator data, and testing on real data. This should be compared to training on only the real data (Table 4) and only on the simulator data (Table 6). The best results are obtained from the combination of simulator and real data. The best real data only results (selecting over algorithm and training strategy) on both tasks outperform the best results using simulator data, i.e. using Cont-MemN2N with the Train AQ / TestAQ setting) 0.774 and 0.797 is obtained vs. 0.714 and 0.788 for Tasks 4 and 8 respectively. This is despite there being far fewer examples of real data compared to simulator data. Overall we obtain two main conclusions from this additional experiment: (i) real data is indeed measurably superior to simulated data for training our models; (ii) in all cases (across different algorithms, tasks and data types -be they real data, simulated data or combinations) the bot asking questions (AQ) outperforms it only answering questions and not asking them (QA). The latter reinforces the main result of the paper.      
Dialogue Learning With Human-In-The-Loop
A good conversational agent (which we sometimes refer to as a learner or bot 1 ) should have the ability to learn from the online feedback from a teacher: adapting its model when making mistakes and reinforcing the model when the teacher's feedback is positive. This is particularly important in the situation where the bot is initially trained in a supervised way on a fixed synthetic, domainspecific or pre-built dataset before release, but will be exposed to a different environment after release (e.g., more diverse natural language utterance usage when talking with real humans, different distributions, special cases, etc.). Most recent research has focused on training a bot from fixed training sets of labeled data but seldom on how the bot can improve through online interaction with humans. Human (rather than machine) language learning happens during communication (Bassiri, 2011;Werts et al., 1995), and not from labeled datasets, hence making this an important subject to study.
Dialogue Learning With Human-In-The-Loop
In this work, we explore this direction by training a bot through interaction with teachers in an online fashion. The task is formalized under the general framework of reinforcement learning via the teacher's (dialogue partner's) feedback to the dialogue actions from the bot. The dialogue takes place in the context of question-answering tasks and the bot has to, given either a short story or a set of facts, answer a set of questions from the teacher. We consider two types of feedback: explicit numerical rewards as in conventional reinforcement learning, and textual feedback which is more natural in human dialogue, following (Weston, 2016). We consider two online training scenarios: (i) where the task is built with a dialogue simulator allowing for easy analysis and repeatability of experiments; and (ii) where the teachers are real humans using Amazon Mechanical Turk.
Dialogue Learning With Human-In-The-Loop
We explore important issues involved in online learning such as how a bot can be most efficiently trained using a minimal amount of teacher's feedback, how a bot can harness different types of feedback signal, how to avoid pitfalls such as instability during online learing with different types of feedback via data balancing and exploration, and how to make learning with real humans feasible via data batching. Our findings indicate that it is feasible to build a pipeline that starts from a model trained with fixed data and then learns from interactions with humans to improve itself.
Dialogue Learning With Human-In-The-Loop
Reinforcement learning has been widely applied to dialogue, especially in slot filling to solve domain-specific tasks (Walker, 2000;Schatzmann et al., 2006;Singh et al., 2000;2002). Efforts include Markov Decision Processes (MDPs) (Levin et al., 1997;2000;Walker et al., 2003;Pieraccini et al., 2009), POMDP models (Young et al., 2010;Gašic et al., 2013;2014) and policy learning (Su et al., 2016). Such a line of research focuses mainly on frames with slots to fill, where the bot will use reinforcement learning to model a state transition pattern, generating dialogue utterances to prompt the appropriate user responses to put in the desired slots. This goal is different from ours, where we study end-to-end learning systems and also consider non-reward based setups via textual feedback.
Dialogue Learning With Human-In-The-Loop
Our work is related to the line of research that focuses on supervised learning for question answering (QA) from dialogues (Dodge et al., 2015;Weston, 2016), either given a database of knowledge Miller et al., 2016) or short texts Hermann et al., 2015;Rajpurkar et al., 2016). In our work, the discourse includes the statements made in the past, the question and answer, and crucially the response from the teacher. The latter is what makes the setting different from the standard QA setting, i.e. we use methods that leverage this response also, not just answering questions. Further, QA works only consider fixed datasets with gold annotations, i.e. they do not consider a reinforcement learning setting.
Dialogue Learning With Human-In-The-Loop
Our work is closely related to a recent work from Weston (2016) that learns through conducting conversations where supervision is given naturally in the response during the conversation. That work introduced the use of forward prediction that learns by predicting the teacher's feedback, in addition to using reward-based learning of correct answers. However, two important issues were not addressed: (i) it did not use a reinforcement learning setting, but instead used pre-built datasets with fixed policies given in advance; and (ii) experiments used only simulated and no real language data. Hence, models that can learn policies from real online communication were not investigated. To make the differences with our work clear, we will now detail these points further.
Dialogue Learning With Human-In-The-Loop
The experiments in (Weston, 2016) involve constructing pre-built fixed datasets, rather than training the learner within a simulator, as in our work. Pre-built datasets can only be made by fixing a prior in advance. They achieve this by choosing an omniscient (but deliberately imperfect) labeler that gets π acc examples always correct (the paper looked at values 50%, 10% and 1%). Again, this was not learned, and was fixed to generate the datasets. Note that the paper refers to these answers as coming from "the learner" (which should be the model), but since the policy is fixed it actually does not depend on the model. In a realistic setting one does not have access to an omniscient labeler, one has to learn a policy completely from scratch, online, starting with a random policy, so their setting was not practically viable. In our work, when policy training is viewed as batch learning over iterations of the dataset, updating the policy on each iteration, (Weston, 2016) can be viewed as training only one iteration, whereas we perform multiple iterations. This is explained further in Sections 4.2 and 5.1. We show in our experiments that performance improves over the iterations, i.e. it is better than the first iteration. We show that such online learning works for both rewardbased numerical feedback and for forward prediction methods using textual feedback (under certain conditions which are detailed). This is a key contribution of our work.
Dialogue Learning With Human-In-The-Loop
Finally, (Weston, 2016) only conducted experiments on synthetic or templated language, and not real language, especially the feedback from the teacher was scripted. While we believe that synthetic datasets are very important for developing understanding (hence we develop a simulator and conduct experiments also with synthetic data), for a new method to gain traction it must be shown to work on real data. We hence employ Mechanical Turk to collect real language data for the questions and importantly for the teacher feedback and construct experiments in this real setting.
Dialogue Learning With Human-In-The-Loop
We begin by describing the data setup we use. In our first set of experiments we build a simulator as a testbed for learning algorithms. In our second set of experiments we use Mechanical Turk to provide real human teachers giving feedback.
Dialogue Learning With Human-In-The-Loop
The simulator adapts two existing fixed datasets to our online setting. Following Weston (2016), we use (i) the single supporting fact problem from the bAbI datasets  which consists of 1000 short stories from a simulated world interspersed with questions; and (ii) the WikiMovies dataset  which consists of roughly 100k (templated) questions over 75k entities based on questions with answers in the open movie database (OMDb). Each dialogue takes place between a teacher, scripted by the simulation, and a bot. The communication protocol is as follows:
Dialogue Learning With Human-In-The-Loop
(1) the teacher first asks a question from the fixed set of questions existing in the dataset, (2) the bot answers the question, and finally (3) the teacher gives feedback on the bot's answer.
Dialogue Learning With Human-In-The-Loop
We follow the paradigm defined in (Weston, 2016) where the teacher's feedback takes the form of either textual feedback, a numerical reward, or both, depending on the task. For each dataset, there are ten tasks, which are further described in Sec. A and illustrated in Figure 5 of the appendix. We also refer the readers to (Weston, 2016) for more detailed descriptions and the motivation behind these tasks. In the main text of this paper we only consider Task 6 ("partial feedback"): the teacher replies with positive textual feedback (6 possible templates) when the bot answers correctly, and positive reward is given only 50% of the time. When the bot is wrong, the teacher gives textual feedback containing the answer. Descriptions and experiments on the other tasks are detailed in the appendix. Example dialogues are given in Figure 1.
Dialogue Learning With Human-In-The-Loop
The difference between our simulation and the original fixed tasks of Weston (2016) is that models are trained on-the-fly. After receiving feedback and/or rewards, we update the model (policy) and then deploy it to collect teacher's feedback in the next episode or batch. This means the model's policy affects the data which is used to train it, which was not the case in the previous work. Figure 1: Simulator sample dialogues for the bAbI task (left) and WikiMovies (right). We consider 10 different tasks following Weston (2016) but here describe only Task 6; other tasks are detailed in the appendix. The teacher's dialogue is in black and the bot is in red. (+) indicates receiving positive reward, given only 50% of the time even when correct.  and incorrectly (right). Real humans provide more variability of language in both questions and textual feedback than in the simulator setup (cf. Figure 1).
Dialogue Learning With Human-In-The-Loop
Who wrote the Linguini Incident ? richard shepard Richard Shepard is one of the right answers here. What year did The World Before Her premiere? 2012 Yep! That's when it came out. Which are the movie genres of Mystery of the 13th Guest? crime Right, it can also be categorized as a mystery. Sample dialogues with incorrect answers from the bot: What are some movies about a supermarket ? supermarket There were many options and this one was not among them. Which are the genres of the film Juwanna Mann ? kevin pollak That is incorrect. Remember the question asked for a genre not name. Who wrote the story of movie Coraline ? fantasy That's a movie genre and not the name of the writer. A better answer would of been Henry Selick or Neil Gaiman.
Dialogue Learning With Human-In-The-Loop
Finally, we extended WikiMovies using Mechanical Turk so that real human teachers are giving feedback rather than using a simulation. As both the questions and feedback are templated in the simulation, they are now both replaced with natural human utterances. Rather than having a set of simulated tasks, we have only one task, and we gave instructions to the teachers that they could give feedback as they see fit. The exact instructions given to the Turkers is given in Appendix B. In general, each independent response contains feedback like (i) positive or negative sentences; or (ii) a phrase containing the answer or (iii) a hint, which are similar to setups defined in the simulator. However, some human responses cannot be so easily categorized, and the lexical variability is much larger in human responses. Some examples of the collected data are given in Figure 2.
Dialogue Learning With Human-In-The-Loop
In our experiments, we used variants of the End-to-End Memory Network (MemN2N) model (Sukhbaatar et al., 2015) as our underlying architecture for learning from dialogue.
Dialogue Learning With Human-In-The-Loop
The input to MemN2N is the last utterance of the dialogue history x as well as a set of memories (context) C=c 1 , c 2 , ..., c N . The memory C encodes both short-term memory, e.g., dialogue histories between the bot and the teacher, and long-term memories, e.g., the knowledge base facts that the bot has access to. Given the input x and C, the goal is to produce an output/label a.
Dialogue Learning With Human-In-The-Loop
In the first step, the query x is transformed to a vector representation u 0 by summing up its constituent word embeddings: u 0 = Ax. The input x is a bag-of-words vector and A is the d × V word embedding matrix where d denotes the emebbding dimension and V denotes the vocabulary size. Each memory c i is similarly transformed to a vector m i . The model will read information from the memory by comparing input representation u 0 with memory vectors m i using softmax weights:
Dialogue Learning With Human-In-The-Loop
This process selects memories relevant to the last utterance x, i.e., the memories with large values of p 1 i . The returned memory vector o 1 is the weighted sum of memory vectors. This process can be repeated to query the memory N times (so called "hops") by adding o n to the original input, u 1 = o 1 + u 0 , or to the previous state, u n = o n + u n−1 , and then using u n to query the memories again.
Dialogue Learning With Human-In-The-Loop
In the end, u N is input to a softmax function for the final prediction:
Dialogue Learning With Human-In-The-Loop
where y 1 , . . . , y L denote the set of candidate answers. If the answer is a word, y i is the corresponding word embedding. If the answer is a sentence, y i is the embedding for the sentence achieved in the same way that we obtain embeddings for query x and memory C.
Dialogue Learning With Human-In-The-Loop
The standard way MemN2N is trained is via a cross entropy criterion on known input-output pairs, which we refer to as supervised or imitation learning. As our work is in a reinforcement learning setup where our model must make predictions to learn, this procedure will not work, so we instead consider reinforcement learning algorithms which we describe next.
Dialogue Learning With Human-In-The-Loop
In this section, we present the algorithms we used to train MemN2N in an online fashion. Our learning setup can be cast as a particular form of Reinforcement Learning. The policy is implemented by the MemN2N model. The state is the dialogue history. The action space corresponds to the set of answers the MemN2N has to choose from to answer the teacher's question. In our setting, the policy chooses only one action for each episode. The reward is either 1 (a reward from the teacher when the bot answers correctly) or 0 otherwise. Note that in our experiments, a reward equal to 0 might mean that the answer is incorrect or that the positive reward is simply missing. The overall setup is closest to standard contextual bandits, except that the reward is binary.
Dialogue Learning With Human-In-The-Loop
When working with real human dialogues, e.g. collecting data via Mechanical Turk, it is easier to set up a task whereby a bot is deployed to respond to a large batch of utterances, as opposed to a single one. The latter would be more difficult to manage and scale up since it would require some form of synchronization between the model replicas interacting with each human.
Dialogue Learning With Human-In-The-Loop
This is comparable to the real world situation where a teacher can either ask a student a single question and give feedback right away, or set up a test that contains many questions and grade all of them at once. Only after the learner completes all questions, it can hear feedback from the teacher.
Dialogue Learning With Human-In-The-Loop
We use batch size to refer to how many dialogue episodes the current model is used to collect feedback before updating its parameters. In the Reinforcement Learning literature, batch size is related to off-policy learning since the MemN2N policy is trained using episodes collected with a stale version of the model. Our experiments show that our model and base algorithms are very robust to the choice of batch size, alleviating the need for correction terms in the learning algorithm (Bottou et al., 2013).
Dialogue Learning With Human-In-The-Loop
We consider two strategies: (i) online batch size, whereby the target policy is updated after doing a single pass over each batch (a batch size of 1 reverts to the usual on-policy online learning); and (ii) dataset-sized batch, whereby training is continued to convergence on the batch which is the size of the dataset, and then the target policy is updated with the new model, and a new batch is drawn and the procedure iterates. These strategies can be applied to all the methods we use, described below.
Dialogue Learning With Human-In-The-Loop
Next, we discuss the learning algorithms we considered in this work.
Dialogue Learning With Human-In-The-Loop
The simplest algorithm we first consider is the one employed in Weston (2016). RBI relies on positive rewards provided by the teacher. It is trained to imitate the correct behavior of the learner, i.e., learning to predict the correct answers (with reward 1) at training time and disregarding the other ones. This is implemented by using a MemN2N that maps a dialogue input to a prediction, i.e. using the cross entropy criterion on the positively rewarded subset of the data.
Dialogue Learning With Human-In-The-Loop
In order to make this work in the online setting which requires exploration to find the correct answer, we employ an -greedy strategy: the learner makes a prediction using its own model (the answer assigned the highest probability) with probability 1 − , otherwise it picks a random answer with probability . The teacher will then give a reward of +1 if the answer is correct, otherwise 0. The bot will then learn to imitate the correct answers: predicting the correct answers while ignoring the incorrect ones.
Dialogue Learning With Human-In-The-Loop
The second algorithm we use is the REINFORCE algorithm (Williams, 1992), which maximizes the expected cumulative reward of the episode, in our case the expected reward provided by the teacher. The expectation is approximated by sampling an answer from the model distribution. Let a denote the answer that the learner gives, p(a) denote the probability that current model assigns to a, r denote the teacher's reward, and J(θ) denote the expectation of the reward. We have:
Dialogue Learning With Human-In-The-Loop
where b is the baseline value, which is estimated using a linear regression model that takes as input the output of the memory network after the last hop, and outputs a scalar b denoting the estimation of the future reward. The baseline model is trained by minimizing the mean squared loss between the estimated reward b and actual reward r, ||r − b|| 2 . We refer the readers to (Ranzato et al., 2015;Zaremba & Sutskever, 2015) for more details. The baseline estimator model is independent from the policy model, and its error is not backpropagated through the policy model.
Dialogue Learning With Human-In-The-Loop
The major difference between RBI and REINFORCE is that (i) the learner only tries to imitate correct behavior in RBI while in REINFORCE it also leverages the incorrect behavior, and (ii) the learner explores using an -greedy strategy in RBI while in REINFORCE it uses the distribution over actions produced by the model itself.
Dialogue Learning With Human-In-The-Loop
FP (Weston, 2016) handles the situation where a numerical reward for a bot's answer is not available, meaning that there are no +1 or 0 labels available after a student's utterance. Instead, the model assumes the teacher gives textual feedback t to the bot's answer, taking the form of a dialogue utterance, and the model tries to predict this instead. Suppose that x denotes the teacher's question and C=c 1 , c 2 , ..., c N denotes the dialogue history as before. In FP, the model first maps the teacher's initial question x and dialogue history C to a vector representation u using a memory network with multiple hops. Then the model will perform another hop of attention over all possible student's answers in A, with an additional part that incorporates the information of which candidate (i.e., a) was actually selected in the dialogue:
Dialogue Learning With Human-In-The-Loop
where yâ denotes the vector representation for the student's answer candidateâ. β is a (learned) d-dimensional vector to signify the actual action a that the student chooses. o is then combined with u to predict the teacher's feedback t using a softmax:
Dialogue Learning With Human-In-The-Loop
where x ri denotes the embedding for the i th response. In the online setting, the teacher will give textual feedback, and the learner needs to update its model using the feedback. It was shown in Weston (2016) that in an off-line setting this procedure can work either on its own, or in conjunction with a method that uses numerical rewards as well for improved performance. In the online setting, we consider two simple extensions:
Dialogue Learning With Human-In-The-Loop
• -greedy exploration: with probability the student will give a random answer, and with probability 1 − it will give the answer that its model assigns the largest probability. This method enables the model to explore the space of actions and to potentially discover correct answers. • data balancing: cluster the set of teacher responses t and then balance training across the clusters equally. 2 This is a type of experience replay (Mnih et al., 2013) but sampling with an evened distribution. Balancing stops part of the distribution dominating the learning. For example, if the model is not exposed to sufficient positive and negative feedback, and one class overly dominates, the learning process degenerates to a model that always predicts the same output regardless of its input.
Dialogue Learning With Human-In-The-Loop
Experiments are first conducted using our simulator, and then using Amazon Mechanical Turk with real human subjects taking the role of the teacher 3 .
Dialogue Learning With Human-In-The-Loop
Online Experiments In our first experiments, we considered both the bAbI and WikiMovies tasks and varied batch size, random exploration rate , and type of model. Figure 3 and Figure 4 shows (Task 6) results on bAbI and WikiMovies. Other tasks yield similar conclusions and are reported in the appendix.
Dialogue Learning With Human-In-The-Loop
Overall, we obtain the following conclusions:
Dialogue Learning With Human-In-The-Loop
• In general RBI and FP do work in a reinforcement learning setting, but can perform better with random exploration. • In particular RBI can fail without exploration. RBI needs random noise for exploring labels otherwise it can get stuck predicting a subset of labels and fail.  Performance is largely independent of batch size, and RBI performs similarly to REINFORCE. Note that supervised, rather than reinforcement learning, with gold standard labels achieves 100% accuracy on this task.
Dialogue Learning With Human-In-The-Loop
• REINFORCE obtains similar performance to RBI with optimal .
Dialogue Learning With Human-In-The-Loop
• FP with balancing or with exploration via both outperform FP alone.
Dialogue Learning With Human-In-The-Loop
• For both RBI and FP, performance is largely independent of online batch size.
Dialogue Learning With Human-In-The-Loop
Dataset Batch Size Experiments Given that larger online batch sizes appear to work well, and that this could be important in a real-world data collection setup where the same model is deployed to gather a large amount of feedback from humans, we conducted further experiments where the batch size is exactly equal to the dataset size and for each batch training is completed to convergence.  (Weston, 2016).
Dialogue Learning With Human-In-The-Loop
After the model has been trained on the dataset, it is deployed to collect a new dataset of questions and answers, and the process is repeated. Table 1 reports test error at each iteration of training, using the bAbI Task 6 as the case study (see the appendix for results on other tasks). The following conclusions can be made for this setting:
Dialogue Learning With Human-In-The-Loop
• RBI improves in performance as we iterate. Unlike in the online case, RBI does not need random exploration. We believe this is because the first batch, which is collected with a randomly initialized model, contains enough variety of examples with positive rewards that the model does not get stuck predicting a subset of labels.
Dialogue Learning With Human-In-The-Loop
• FP is not stable in this setting. This is because once the model gets very good at making predictions (at the third iteration), it is not exposed to a sufficient number of negative responses anymore. From that point on, learning degenerates and performance drops as the model always predicts the same responses. At the next iteration, it will recover again since it has a more balanced training set, but then it will collapse again in an oscillating behavior.
Dialogue Learning With Human-In-The-Loop
• FP does work if extended with balancing or random exploration with sufficiently large .
Dialogue Learning With Human-In-The-Loop
• RBI+FP also works well and helps with the instability of FP, alleviating the need for random exploration and data balancing.
Dialogue Learning With Human-In-The-Loop
Overall, our simulation results indicate that while a bot can be effectively trained fully online from bot-teacher interactions, collecting real dialogue data in batches (which is easier to collect and iterate experiments over) is also a viable approach. We hence pursue the latter approach in our next set of experiments. Relation to experiments in Weston (2016) As described in detail in Section 2 the datasets we use in our experiments were introduced in . However, that work involved constructing pre-built fixed policies (and hence, datasets), rather than training the learner in a reinforcement/interactive learning using a simulator, as in our work. They achieved this by choosing an omniscient (but deliberately imperfect) labeler that gets π acc examples always correct (the paper looked at values 1%, 10% and 50%). In a realistic setting one does not have access to an omniscient labeler, one has to learn a policy completely from scratch, online, starting with a random policy, as we do here. Nevertheless, it is possible to compare our learnt policies to those results because we use the same train/valid/test splits.
Dialogue Learning With Human-In-The-Loop
The clearest comparison comparison is via Table 1, where the policy is learnt using batch iterations of the dataset, updating the policy on each iteration.  can be viewed as training only one iteration, with a pre-built policy, as explained above, where 59%, 81% and 99% accuracy was obtained for RBI for π acc with 1%, 10% and 50% respectively 4 . While π acc of 50% is good enough to solve the task, lower values are not. In this work a random policy begins with 74% accuracy on the first iteration, but importantly on each iteration the policy is updated and improves, with values of 87%, 90% on iterations 2 and 3 respectively, and 98% on iteration 6. This is a key differentiator to the work of  where such improvement was not shown. We show that such online learning works for both reward-based numerical feedback and for forward prediction methods using textual feedback (as long as balancing or random exploration is performed sufficiently). The final performance outperforms most values of π acc from  unless π is so large that the task is already solved. This is a key contribution of our work.
Dialogue Learning With Human-In-The-Loop
Similar conclusions can be made for Figures 3 and 4. Despite our initial random policy starting at close to 0% accuracy, if random exploration ≥ 0.2 is employed then after a number of epochs the performance is better than most values of π acc from , e.g. compare the accuracies given in the previous paragraph (59%, 81% and 99%) to Figure 3, top left.
Dialogue Learning With Human-In-The-Loop
We employed Turkers to both ask questions and then give textual feedback on the bot's answers, as described in Section 3.2. Our experimental protocol was as follows. We first trained a MemN2N using supervised (i.e., imitation) learning on a training set of 1000 questions produced by Turkers and using the known correct answers provided by the original dataset (and no textual feedback). Next, using the trained policy, we collected textual feedback for the responses of the bot for an additional 10,000 questions. Examples from the collected dataset are given in Figure 2. Given this dataset, we compare various models: RBI, FP and FP+RBI. As we know the correct answers to the additional questions, we can assign a positive reward to questions the bot got correct. We hence measure the impact of the sparseness of this reward signal, where a fraction r of additional examples have rewards. The models are tested on a test set of ∼8,000 questions (produced by Turkers), and hyperparameters are tuned on a similarly sized validation set. Note this is a harder task than the WikiMovies task in the simulator due to the use natural language from Turkers, hence lower test performance is expected.
Dialogue Learning With Human-In-The-Loop
Results are given in 
Dialogue Learning With Human-In-The-Loop
The tasks in Weston (2016) were specifically: -Task 1: The teacher tells the student exactly what they should have said (supervised baseline).
Dialogue Learning With Human-In-The-Loop
-Task 2: The teacher replies with positive textual feedback and reward, or negative textual feedback.
Dialogue Learning With Human-In-The-Loop
-Task 3: The teacher gives textual feedback containing the answer when the bot is wrong.
Dialogue Learning With Human-In-The-Loop
-Task 4: The teacher provides a hint by providing the class of the correct answer, e.g., "No it's a movie" for the question "which movie did Forest Gump star in?".
Dialogue Learning With Human-In-The-Loop
-Task 5: The teacher provides a reason why the student's answer is wrong by pointing out the relevant supporting fact from the knowledge base.
Dialogue Learning With Human-In-The-Loop
-Task 6: The teacher gives positive reward only 50% of the time.
Dialogue Learning With Human-In-The-Loop
-Task 7: Rewards are missing and the teacher only gives natural language feedback.
Dialogue Learning With Human-In-The-Loop
-Task 8: Combines Tasks 1 and 2 to see whether a learner can learn successfully from both forms of supervision at once.
Dialogue Learning With Human-In-The-Loop
-Task 9: The bot asks questions of the teacher about what it has done wrong.
Dialogue Learning With Human-In-The-Loop
-Task 10: The bot will receive a hint rather than the correct answer after asking for help.
Dialogue Learning With Human-In-The-Loop
We refer the readers to (Weston, 2016) for more detailed descriptions and the motivation behind these tasks. The difference in our system is that the model can be trained on-the-fly via the simulator: after receiving feedback and/or rewards, the model can update itself and apply its learning to the next episode. We present results on Tasks 2, 3 and 4 in this appendix
Dialogue Learning With Human-In-The-Loop
These are the instructions given for the textual feedback mechanical turk task (we also constructed a separate task to collect the initial questions, not described here):
Dialogue Learning With Human-In-The-Loop
Title: Write brief responses to given dialogue exchanges (about 15 min) Description: Write a brief response to a student's answer to a teacher's question, providing feedback to the student on their answer.
Dialogue Learning With Human-In-The-Loop
Each task consists of the following triplets:
Dialogue Learning With Human-In-The-Loop
1. a question by the teacher 2. the correct answer(s) to the question (separated by "OR")
Dialogue Learning With Human-In-The-Loop
3. a proposed answer in reply to the question from the student Consider the scenario where you are the teacher and have already asked the question, and received the reply from the student. Please compose a brief response giving feedback to the student about their answer. The correct answers are provided so that you know whether the student was correct or not.
Dialogue Learning With Human-In-The-Loop
For example, given 1) question: "what is a color in the united states flag?"; 2) correct answer: "white, blue, red"; 3) student reply: "red", your response could be something like "that's right!"; for 3) reply: "green", you might say "no that's not right" or "nope, a correct answer is actually white".
Dialogue Learning With Human-In-The-Loop
Please vary responses and try to minimize spelling mistakes. If the same responses are copied/pasted or overused, we'll reject the HIT.
Dialogue Learning With Human-In-The-Loop
Avoid naming the student or addressing "the class" directly.
Dialogue Learning With Human-In-The-Loop
We will consider bonuses for higher quality responses during review.  Figure 5: The ten tasks our simulator implements, which evaluate different forms of teacher response and binary feedback. In each case the same example from WikiMovies is given for simplicity, where the student answered correctly for all tasks (left) or incorrectly (right). Red text denotes responses by the bot with S denoting the bot. Blue text is spoken by the teacher with T denoting the teacher's response. For imitation learning the teacher provides the response the student should say denoted with S in Tasks 1 and 8. A (+) denotes a positive reward.     
Dialogue Learning With Human-In-The-Loop
In the experiment in Section 5.2 we conducted experiments with real human feedback. Here, we compare this to a form of synthetic feedback, mostly as a sanity check, but also to see how much improvement we can get if the signal is simpler and cleaner (as it is synthetic). We hence constructed synthetic feedback for the 10,000 responses, using either Task 2 (positive or negative feedback), Task 3 (answers provided by teacher) or a mix (Task 2+3) where we use one or the other for each example (50% chance of each). The latter makes the synthetic data have a mixed setup of responses, which more closely mimics the real data case. The results are given in Table 4. The RBI+FP combination is better using the synthetic data than the real data with Task 2+3 or Task 3, which is to be expected, but the real data is competitive, despite the difficulty of dealing with its lexical and semantic variability. The real data is better than using Task 2 synthetic data.
Dialogue Learning With Human-In-The-Loop
For comparison purposes, we also ran a supervised (imitation learning) MemN2N on different sized training sets of turker authored questions with gold annotated labels (so, there are no numerical rewards or textual feedback, this is a pure supervised setting). The results are given in Table 5. They indicate that RBI+FP and even FP alone get close to the performance of fully supervised learning.
Dialogue Learning With Human-In-The-Loop
Model r = 0 r = 0.1 r = 0.    Table 6: Second Iteration of Feedback Using synthetic textual feedback of synthetic Task2+3 with the RBI+FP method, an additional iteration of data collection of 10k examples, varying sparse binary reward fraction r and exploration . The performance of the first iteration model was 0.478.
Dialogue Learning With Human-In-The-Loop
We conducted experiments with an additional iteration of data collection for the case of binary rewards and textual feedback using the synthetic Task 2+3 mix. We selected the best model from the previous training, using RBI+FP with r = 1 which previously gave a test accuracy of 0.478 (see Table 4). Using that model as a predictor, we collected an additional 10,000 training examples.
Dialogue Learning With Human-In-The-Loop
We then continue to train our model using the original 1k+10k training set, plus the additional 10k. As before, we report the test accuracy varying r on the additional collected set. We also report the performance from varying , the proportion of random exploration of predictions on the new set. The results are reported in Table 6. Overall, performance is improved in the second iteration, with slightly better performance for large r and = 0.5. However, the improvement is mostly invariant to those parameters, likely because FP takes advantage of feedback from incorrect predictions in any case.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Large language models employed as dialogue agents are primarily trained on human-written documents and human-human conversations collected from the web for pre-training (Conneau et al., 2019;Baumgartner et al., 2020), and humanhuman crowdsourced conversations  for fine-tuning. The models are then used at inference time to conduct conversations with humans, with no further learning taking place (Adiwardana et al., 2020;Roller et al., 2020). Humanmodel conversations -which are never seen at training time -can have a quite different distribution to the original human-human training data used, and our current techniques can lose performance due to lack of robustness to such deviations (Chollet, 2019;Bengio, 2019 Figure 1: Using human feedback to improve opendomain internet-driven dialogue agents. We compare various types of feedback (and corresponding learning algorithms) in this work, such as binary feedback (good/bad), free-form text or supervised responses (better suggestions) for different modules of the system.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
In this work, we study learning from the feedback collected during deployment of models in human-model conversations. Such a setting has the opportunity to learn from within-distribution data, both in terms of the input contexts, but also the responses required (targets). Not only can this mean improvement in skills that are similar to the pre-train and fine-tune data, but potentially the learning of completely new skills -that are desired by users of the system. We thus take existing state of the art internet-augmented models such as BlenderBot 2 (Komeili et al., 2021; and SeeKeR (Shuster et al., 2022a), deploy them to human crowdworkers, and exper-iment with various methods to learn from such interactions. We thus first ask crowdworkers what topic and task they would like to talk about, in order to collect in-domain data, and then collect conversations involving these skills. During the conversations we collect various kinds of human feedback, including binary feedback (good/bad), free-form conversational feedback, and the type of failure (search query-based, results-based, or final response-based), as well as suggestions for improvements (see Figure 1).
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
We then explore a variety of methods for learning from feedback, and compare them in detailed experiments. In particular, we compare supervised learning methods, rejection sampling, model guiding and reward-based learning. Our findings are:
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• Taking advantage of modular feedback (feedback about particular errors from modules of the model, such as the search engine component) outperforms feedback about just the final response.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• Textual and binary feedback are also very useful signals, but not as much as modular feedback.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• The recently introduced DIRECTOR method ( Arora et al., 2022), when learning from binary feedback, works better than reranking or reward-based learning.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• Combining multiple types of feedback, such as modular and binary feedback with DIREC-TOR provides the best results we obtained.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• Continual learning, whereby we retrain models on the feedback from previous rounds of deployment, improves results even further.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• Despite collecting feedback from smaller (3B parameter) models, the data collection is useful for improving much larger (175B parameter) models.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
We make the collected data and feedback, the models, and the code publicly available for this work 1 .
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
To select an input distribution closely aligned with human preferences, we first collected a set of skills humans would like an AI powered text-messaging chatbot to possess. We instruct that the hypothetical chatbot can talk about any topic, and has the ability to surf the internet for information. We then asked each human annotator to provide:
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
(i) a topic (1-10 words), (ii) three tasks related to the topic; and (iii) descriptions of how they would assess if the chatbot has completed those tasks.  See Appendix subsection A.1 for a screenshot of the task definition collection instructions, and further details.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Overall, we collected 1108 task types via 152 annotators, which cover diverse topics -from making healthy food to loom weaving to Caribbean holidays. Grouping them into types, they include question answering followed by discussion, providing ranked lists, providing reviews, summary generation, personal recommendations, reasoning/deductions (e.g., how to perform calculations), creativity (e.g., tell a joke), tutorials, instructions, and more. Many of these tasks require, or else are made simpler, by use of the internet, e.g., searching for particular entities or topics, and responding conditioned on pertinent results. Some examples are given in Table 1.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
After collecting topic and task definitions, the next step is to deploy conversational models (bots) that are asked to exhibit these skills. Human conversationalists select a task (out of two randomly chosen tasks) from the set collected in subsection 3.1 and then ask the model to help them complete it over a series of conversational turns. The instructions emphasize that this should be a dialogue ("a back and forth conversation"), and hence the speakers should break up requests or information across messages so that it remains conversational.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
The human conversationalist is instructed that the bot might not be perfect, in which case feedback can be given in order to improve the bot in the future. We collect various kinds of feedback, from lightweight feedback (binary label or free-form response) to detailed (multiple choice and fine-grained responses) such that in our experiments we can compare and contrast them in order to make recommendations on which kinds of feedback work best. Hence after each dialogue turn we collect the following set of feedback types:
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• Binary feedback on whether the response was considered satisfactory or not.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• Free-form textual feedback on what was wrong in the case of an unsatisfactory response.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• Multi-choice input on how the bot could improve this turn:
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
(a) using a better search query; or (b) paying more attention to relevant search results; (c) some other issue; or (d) no issue (a good response).
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• In the case of selecting (a), the human is then asked what would be a more appropriate search query.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• In case (b), the human is shown the search results and asked to select a relevant portion.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• In case (c), the human is asked what would be an improved overall response.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Continuing the conversation After feedback has been given, the conversation is continued. If multiple-choice option (a) was selected previously, the bot on this next turn is forced to use the "gold" search query given by the user. Similarly, for (b), the provided gold knowledge context is added to the input of the model. In the case of (c), the bot is simply bypassed, and it is assumed to have provided the given gold response. In this way, even for a poorly performing bot, headway can be made in the conversation towards completing the task,  , which we publicly release. Train v1 and v2 correspond to two rounds of continual learning, where the v1 data consists of conversations and feedback from our deployed base models, and the v2 data consists of conversations and feedback from deployed models trained using the v1 data. and collecting feedback on its subsequent stages.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
(Without such a procedure, the bot may just get stuck in a poor quality loop, and then there would be no choice but to abandon the conversation.)
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
The conversation is continued until the human marks the task as complete or a minimum of 4 turns has been completed. When the task is complete we also collect a final rating (out of 5) for the bot's performance.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
We consider the following set of state of the art publicly available conversational models:
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• BlenderBot (BB1) (Roller et al., 2021); a 2.7B
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
parameter Transformer model pre-trained and fine-tuned on dialogue data to exhibit conversational skills; however these models have no ability to use the internet, but simply generate responses given the dialogue context.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• BlenderBot 2.0 (BB2) (Komeili et al., 2021;, a 2.7B parameter model multi-tasked on the same tasks as BB1, and also with additional tasks which give it the ability to execute internet search queries and condition on the results using a fusion-indecoder (FiD) (Izacard and Grave, 2020) style approach. The search query generator is a separate 400M parameter transformer.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• SeeKeR (Shuster et al., 2022a); uses a similar 2.7B parameter architecture, but utilizing the Knowledge-to-response (K2R) approach (Adolphs et al., 2021) which performs a multistep generation procedure: first generating a relevant knowledge response, and then conditions on that to generate a final dialogue response. It is multi-tasked on the same training data as BB2, and in addition on some other knowledge-intensive tasks, such as QA tasks, as well.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
• OPT-175B (Zhang et al., 2022) and BB3-175B (Shuster et al., 2022b): we compare the 175B language model OPT (either 0-shot or few-shot, following Shuster et al. (2022b)) with BlenderBot 3, which is fine-tuned with conversational datasets including modular supervision, and internet-augmentation , from our task. This setting examines if our experiments and results are applicable to very large language models.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
We can evaluate model performance during conversations between humans and the deployed models, as humans are providing direct feedback on the conversational responses from the model. In particular we can measure the number of good responses (with no issue), the average final rating, and compute a breakdown of error types (better search query, results or other issue).
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Overall, we collect over 210k human-bot utterances in over 14k dialogues (episodes), with feedback for each of the bot utterances. The data is split into three major portions: v1, v2, and test unseen splits, see Table 2 for a full breakdown.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
The v1 split consists of dialogues conducted with one of our base deployed models (subsection 3.3), and feedback was collected from those dialogues. We then split that data into train, valid and test dialogues. We use this data to train several learning methods using the feedback from the v1 models. These new models are then redeployed.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
The v2 split consists of dialogues and feedback with the new models that were trained using the v1 data. This data is again split into train, valid and test dialogues. We can then repeat this process and train models on the v2 data as well.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Finally, the unseen test split consists of completely new skills (topics and tasks) unseen in the v1 and v2 splits, and is used to test transfer of v1 or v2 based models to these new skills.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
We also verified the quality of our data. For each conversation, we ask 3 human crowdworkers to rate the bot and human's performance and also assess if the bot was able to complete the given task. We consider the task as complete if 2 out of the 3 annotators labeled the task as complete. We see that in 90.4% of the cases the task is completed. Note that with the feedback from the human (see section 3.2) the human-model conversation should always progress even if the model has errors so ideally if the human is doing a perfect job this would be 100%. We also assess the quality of the human conversationalist directly and ask annotators to "rate the human's messages in defining, clarifying, and helping the bot complete the task on a scale from 1-5 (1 = was not helpful at all in helping the bot complete the task, 5 = guided the bot to complete the task)." For conversations where the task was completed, the human conversation partner's messages were rated at an average of 3.8. For conversations where the task was incomplete, the human conversation partner's messages were rated at an average of 3.5.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
In the following, we will describe the methods we will experiment with for learning from the collected human feedback.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
The easiest to use type of feedback, with perhaps the strongest learning signal, is a provided gold response by the user for a given dialogue context. One can simply continue to fine-tune the model on the set of collected gold responses (from case (c) in section 3.2). One can optionally also add all the bot responses that were marked as good to the fine-tune set as well (case (d) in section 3.2). We use the validation set to choose the weighting between these two types of supervised data.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Using the multiple-choice feedback on the types of improvement, the model can learn to improve those individual components of the model. For BB2 and SeeKeR one can use provided gold search queries (case (a) in section 3.2) directly to fine-tune the search query generation.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Provided gold knowledge responses (relevant search results, case (b) in section 3.2)) are similarly easy to use for fine-tuning in the SeeKeR model because the model is already trained to generate such responses directly. For BB2, there are no direct knowledge responses as this is implicit in FiD, so in that case we use a similar method to Hancock et al. (2019) whereby we train in a supervised fashion with the knowledge response as a target, but add special tokens to both input and target to indicate this is not a standard dialogue response task. The goal is that this additional training signal can then help learn useful features for the actual overall response task.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
For free-form textual feedback, we can also use a similar approach and simply fine-tune with the feedback as targets, with special tokens appended to both the input context and the feedback target, again following Hancock et al. (2019) which showed this approach can work.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Using the binary satisfaction feedback signal one can train a reward model. We employ a 311M parameter transformer pre-trained on pushshift.io Reddit (Baumgartner et al., 2020) using a masked language model objective. Then, given the context and response concatenated as input, we train it with a standard classification loss on our satisfaction task. Such a model has multiple uses (see following subsections) but one obvious approach is to rerank generation candidates from the conversational model using the reward model with the aim that the highest ranked provide the highest satisfaction. Such approaches have been employed in many use cases previously (Nie et al., 2020;Nakano et al., 2021;Askell et al., 2021;Thoppilan et al., 2022).
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Rejection sampling/reranking relies on the set of generated candidates containing at least one good candidate, and has no effect on the initial quality of the candidate generations themselves -it only scores the final generated sequences. We next consider using a reward model trained via subsection 4.4 to train the generation model itself. Given training set contexts, we generate candidates, rerank the candidates, and select the highest ranking. We then train the generation model to use those highest ranking candidates as targets, i.e. by fine-tuning with those targets. This is similar to the approach used in Thoppilan et al. (2022).
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
The recently introduced DIRECTOR model (Arora et al., 2022), instead of using a reward model, trains a unified decoder-classifier architecture. It predicts for every token both: (i) the language modeling (LM) next token probability using the standard LM head; and (ii) a task-suitability probability using a second classifier head. Both heads are fed the output of the last decoder block, and map from the embedding dimension to the size of the vocabulary, with all the parameters jointly trained using both positive generation data (that can train the language modeling head and also be positive examples for the classifier) and negative data (that trains the classification head only). Finally, during decoding, left-to-right generation is conducted by combining the two probabilities from the two heads to incorporate negative feedback into the generation process. This method was shown to outperform other model guiding approaches, in addition to being more efficient as many other methods employ a separate reward or language model to perform the guiding (Krause et al., 2020;Yang and Klein, 2021;.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
We provide automatic evaluation results in Table 3 and human evaluation results in Table 4 comparing various methods described in the previous section.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Internet-augmentation helps First, this is an expected result, due to the nature of our tasks, but we find that using internet-augmentation helps in line with other internet-based dialogue tasks (Dinan et al., 2019;Komeili et al., 2021;Shuster et al., 2022a). We find that BB2 and SeeKeR, which both perform internet search and condition on documents, outperform BB1 that does not. This improvement is quite large, e.g. BB1 has 24.8% Good responses, compared to BB2 and SeeKeR having 33.2% and 49.3% respectively. SeeKeR, which has a modular search architecture that aims to use retrieved knowledge more accurately, performs markedly better than BB2, which is in line with previous results on other datasets (Shuster et al., 2022a).
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Human feedback helps Across the board we find different kinds of feedback can improve our base models BB2 3B and SeeKeR 3B; we will analyse specific methods further in the subsequent discussion. These overall improvements can be seen in terms of all the human evaluation metrics measured (Good response%, Rating, and all three Error Breakdown types), as well as the automatic evaluation metrics we measured (F1 and PPL). We also generally (although not in every single case) see correlation between automatic and human evaluation metrics, e.g. the best methods are best in both types of metric.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Modular superior to non-modular feedback In the modular feedback setting humans give feedback about what has gone wrong in the pipeline of the model: whether the internet search query was poor, or the document/knowledge chosen after searching was poorly chosen. Taking into account modular feedback outperforms using only supervised feedback of final responses in both automatic metric and human evaluations for both BB2 and SeeKeR models. For BB2 we see close to 2% improvement in Good responses for modular feedback compared to supervised feedback (40.3% → 42.0%), with both far superior to BB2 without feedback (33.2%). However, SeeKeR which has a modular design, and hence is much easier to supply modular feedback to (as the supervision can directly train each module) sees a larger improvement of 4.5% (52.2% → 56.7%).
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Free-form feedback is useful (but not as much as gold labels) Free-form feedback also gives clear gains over the baseline model for both BB2 and SeeKeR, but falls short of supervised feedback by 3% and 1% respectively for the two model variants. This does not seem surprising as supervised feedback directly gives a clear loss to optimize (simply try to generate the suggestion) whereas feedback is less clear a signal, depending on how it is phrased. However, we do not rule out other free-form feedback algorithms giving better results in the future, see e.g. Scheurer et al. (2022) for a recent method.   Table 4: Human Evaluation results of baselines and various methods learning from human feedback. We bold statistically significant improvements (independent two-sample t-test, p < 0.05) of methods over their respective baselines (BB2 3B or SeeKeR 3B). We also bold statistical significance of BB3-175B over all the 3B baseline models (BB1, BB2, SeeKeR) and the OPT-175B few-shot model.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Binary feedback can work well Non-textual feedback that consists only of a rating can also be helpful for improving systems, in this case binary feedback (good or bad). All three algorithms we employ that use this type of feedback (reranking, reward-based learning, and DIRECTOR) all show gains over the baseline without feedback, with improvements consistent across both BB2 and SeeKeR model variants. Reranking and DIREC-TOR work better than reward-based learning with automatic metrics, so we run those two methods in human evaluations. In some cases these methods then show improvements superior to supervised feedback, e.g. DIRECTOR has a 3.3% Good re-sponses improvement over supervised feedback for SeeKeR (but not for BB2, although for both baseline models DIRECTOR has superior F1).
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
DIRECTOR is better than reranking and rewardbased learning DIRECTOR outperforms reranking and reward-based learning (where all three models utilize binary feedback) for both base models BB2 and SeeKeR. This is both in terms of automatic metrics, e.g. DIRECTOR with a BB2 base model has an F1 of 16.2, whereas reranking and reward-based learning have 15.8 and 15.1 respectively, as well as in terms of human evaluations. For human evaluations, we see a 1-2% improvement in Good response % over reranking for both base models. Presumably this is because DIREC-TOR can guide the generation to a higher quality, whereas reranking can only perform well if a good candidate has been generated by the base model.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Combining multiple feedback signals (where DI-RECTOR works best) If one has access to multiple feedback signal types, some of the algorithms we have tried are capable of using them all. In particular, we can train DIRECTOR with both binary feedback (to train the classifier head) and module feedback (to train the language modeling head for the different modules). This gives the best results out of all methods for both base models by quite a margin in both automatic and human evaluations. E.g., for improving the BB2 base model this gives 47.0% Good responses, compared to the original baseline of 33.2% or even DIRECTOR with only binary feedback of 37.8%. We see this trend is also apparent in other algorithms, as we also measure the performance of supervised feedback + reranking, which also gives gains over either of those methods alone in automatic evaluations, although it still lags behind DIRECTOR.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Iterative deployment and feedback collection improves results further During the process of evaluating all the models that were trained with v1 data described above, more data was collected from those models, which we refer to as the v2 split (see subsection 3.5). We can thus then train models on the v2 split, yielding potentially improved models. In the ideal case one could conduct an iterative continual learning setup, each time retraining on the data collected from previous rounds, improving further each time. We test this setup by training DIRECTOR (module+binary feedback), our best system from v1, with the v2 data split. The result shown in Table 3 (last row) indicates there are significant gains from this procedure, as this method obtains our best results across all data splits (valid, test seen v1 and the unseen set).
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Very large models benefit from feedback from smaller models OPT-175B, either in zero-shot or few-shot variants is only pre-trained on dialogue data, and not fine-tuned on our task, and performs reasonably -but not better than smaller models that are fine-tuned. BlenderBot 3 (Shuster et al., 2022b) is trained with the modular supervision feedback data collected from the smaller (3B parameter) models, in addition to fine-tuning on other standard dialogue datasets. This model provides the best human evaluation metrics of all the systems we test, with a good response rate of 64.8% and a rating of 4.08. This indicates: (i) how important fine-tuning with relevant data is even to very large models; and (ii) even though our data was collected with feedback from small models fine-tuning using this data still brings large gains to larger models. This is an encouraging result as models are improving in architecture and increasing in scale over time, but data we have collected in the past should still remain useful for these models in the future. We provide cherry picked and lemon picked examples of BB3-175B in Appendix B, as well as comparing to OPT-175B. While there a number of success cases, even our best models still make factual errors and contradictions in some cases. Hence, it appears that continued interaction with further feedback collection in the future will be beneficial for further improvements.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
In conclusion, we have studied whether a conversational model can learn new skills after the standard pre-training / fine-tuning setup by interacting with humans during its deployment. We study the use of different kinds of user feedback data and different learning algorithms for leveraging them, in order to compare their performance. We find that granular (modular) feedback about types of errors can yield strong performance, which can also work very well in conjunction with binary feedback using the recently introduced DIRECTOR model, yielding our best results. Evidence also suggests that iterative retraining and redeployment also brings further gains, and that the feedback collected is useful for models differing from the ones originally conversed with, e.g., if much larger models are used in the future.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
All of our experiments have taken place by deploying conversational agents on Amazon Mechanical Turk with crowdworkers 2 , using English-language responses written by workers located in the United States. While these workers are reasonably diverse (Moss et al., 2020), this is quite different to a public deployment with organic users, who are using the system not because they are being paid but because they are genuinely engaged. In that case, collecting feedback will have different tradeoffs which we could not factor into the current work. For example, asking to provide detailed feedback might dissuade users from wanting to interact with the system, lowering engagement and hence the amount of collected data. We believe either more natural free-form or lightweight feedback might be best in that case, which is why we study and compare feedback methods in this work to evaluate their relative impact.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
In public deployments with organic users, safety issues also become a much more important factor -in particular dealing with noisy or adversarial inputs and feedback. In the worst case this could mean human conversationalists could teach the model erroneous reasoning, misinformation, toxic or other undesirable behavior. We note that steps to address this issue are studied elsewhere, for example Ju et al. (2022).
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
The data collection lasted for around 6 months and in total over 700 crowdworkers who are Englishspeaking annotators located in the United States were recruited and compensated through the Amazon Mechanical Turk platform. Before the data collection starts, all crowdworkers are informed that any message they send may be publicly disclosed for research purposes, and are instructed not to send any personal identifiable information (for example, name, address, email, or phone number etc.) in their messages.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
The FITS task contains data from all the deployed models (including the 3 baseline models and their fine-tuned versions). The breakdown by model types in the FITS dataset: 70% are BB2-based, 25% SeeKeR-based models and 5% other model types including OPT-based models.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
We provide several example outputs of our models on the FITS dataset, including examples that showcase both the successes and failures. Successes In Figure 2, we compare the model outputs of the BB3-175B model that has been trained on the FITS task and the OPT-175B few-shot model that has not, given the same topic. Unlike the OPT-175B few-shot model, BB3-175B is able to generate better search queries and pay attention to search results. In Figure 3, we show two success cases for BB3-175B. In both cases the model is able to engage with human speakers on the topic, and listen to human feedback to improve the results even further.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Failures Despite showing continual improvement by re-training on collected human feedback, our models, like other state-of-the-art dialogue models, can still make common mistakes during deployment. Failure cases are shown in Figure 4 for our BB3-175B model where it generates contradicting or factually incorrect outputs.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
We use the openly available ParlAI framework for all 3B model training runs, as well as for evaluations, where metrics are measured using default settings. All the 3B fine-tuned models are trained with a maximum of eight 32GB GPUs (NVIDIA V100), optimized with Adam using β 1 = 0.9, β 2 = 0.999, ϵ = 1e − 08. Models are trained up to 8000 updates with batch size up to 128. The typical fine-tuning time for the 3B retrieval-based BB2 and SeeKeR models is around 24 hrs before it early stops.  A2. Did you discuss any potential risks of your work? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
A3. Do the abstract and introduction summarize the paper's main claims? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
A4. Have you used AI writing assistants when working on this paper? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Did you use or create scientific artifacts?
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
B1. Did you cite the creators of artifacts you used? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
D Did you use human annotators (e.g., crowdworkers) or research with human participants?
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
Training a dialogue agent to converse like a human requires extensive supervision. The most common approach is to train models to imitate humans in large corpora of crowdsourced or scraped conversations (Serban et al., 2015). These fullysupervised conversations tend to be expensive to collect in sufficient quantity and/or occur in settings with significant differences from the deployment environment (Ross et al., 2009). Instead, dialogue agents would ideally learn directly from dialogue, the conversations they participate in after deployment, which are usually abundant, taskspecific, dynamic, and cheap. This corresponds to the way humans learn to converse-not merely observing others engaging in "expert-level" conver- * *BH completed most of this work at Facebook (FAIR). sations, but instead actively adjusting and correcting our speech based on feedback woven throughout our own conversations (Bassiri, 2011;Werts et al., 1995). Giving a dialogue agent this ability would enable it to continuously improve and adapt over its lifetime, rather than requiring additional annotation costs for each and every improvement.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
However, naively training a dialogue agent on its own conversations yields poor results. For example, training a model on its own output can simply reinforce its existing failure modes, and mistakes by the agent can lead to absurd conversations that no longer resemble the target domain (Hashimoto and Sassano, 2018). To combat this, one approach is to allow the agent to request feed-back during conversations (Zhang et al., 2018a;Li et al., 2017b), e.g., when it believes it is about to make a mistake. This approach, however, falls victim to the Dunning-Kruger effect (Kruger and Dunning, 1999), which in this case suggests that a bad model will also be bad at knowing when it is doing a bad job. Regardless of when feedback is requested, existing methods typically require accompanying scalar rewards or adherence to particular templates or structure to ensure that the feedback is usable by the model (Rieser and Lemon, 2011;. These requirements may be acceptable for paid annotators, but they impose unnatural workflows on unpaid conversation partners in a standard dialogue environment. Humans are able to request and provide feedback using only natural language; ideally, dialogue agents would be able to do the same.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
In this work we propose the self-feeding chatbot, a dialogue agent with the ability to extract new examples from the conversations it participates in after deployment ( Figure 1). Concretely, in addition to being trained on the primary DIALOGUE task, the agent is trained to predict its speaking partner's satisfaction with its responses. When the conversation seems to be going well, the user's responses (but not the bot's own utterances) become the targets in new training examples for the DIA-LOGUE task. When the agent believes it has made a mistake, it instead requests feedback on what it could have said instead. Predicting the feedback that will be provided in a given context becomes an auxiliary task (FEEDBACK) on which the model is also trained. Importantly, these new examples improve the agent's dialogue abilities while using only natural responses from the user that do not require special structure, accompanying numerical feedback, or additional human intervention in order to be used.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
With this approach, the conversations the chatbot participates in are sliced into two complementary datasets-one largely protected from the chatbot's mistakes (DIALOGUE examples), and one which directly addresses them (FEEDBACK examples). We validate our approach on the PER-SONACHAT (Zhang et al., 2018b) dialogue dataset, finding empirically that regardless of the number of available supervised examples, the dialogue ability of the chatbot is always improved by adding the automatically extracted examples of either type, and improves the most by adding both.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
The main contributions of this work thus include the following:
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
• We propose the self-feeding chatbot, a dialogue agent with the ability to extract new training examples for itself from the conversations it participates in during deployment.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
• We show that dialogue ability improves by imitating human responses when the human is satisfied, or by asking for feedback when they are not, predicting it as an auxiliary task.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
• We demonstrate that classifying user satisfaction is a learnable task important for the selffeeding process, significantly outperforming an approach based on model uncertainty.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
• We release three new datasets to further research in this direction: (1) deployment chat logs (513k messages); (2) ratings of user satisfaction (42k); (3) textual feedback on what a bot could have said in a given context (62k).
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
The datasets and models described in this paper are available via the ParlAI platform (Miller et al., 2017), along with training code. Hyperparameter values are included in Appendix G.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
The lifecycle of a self-feeding chatbot is outlined in Figure 2. In the initial training phase, the dialogue agent is trained on two tasks-DIALOGUE (next utterance prediction, or what should I say next?) and SATISFACTION (how satisfied is my speaking partner with my responses?)-using whatever supervised training data is available. We refer to these initial DIALOGUE examples as Human-Human (HH) examples, since they were generated in conversations between two humans.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
In the deployment phase, the agent engages in multi-turn conversations with users, extracting new deployment examples of two types. Each turn, the agent observes the context x (i.e., the conversation history) and uses it to predict its next utteranceŷ and its partner's satisfactionŝ. If the satisfaction score is above a specified threshold t, the agent extracts a new Human-Bot (HB) DIALOGUE example using the previous context x and the human's response y and continues the conversation.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
If, however, the user seems unsatisfied with its previous response (ŝ < t), the agent requests feedback with a question q, and the resulting feedback response f is used to create a new example for the FEEDBACK task (what feedback am I about to receive?). The agent acknowledges receipt of the feedback and the conversation continues. The rate at which new DIALOGUE or FEEDBACK examples are collected can be adjusted by raising or lowering the satisfaction threshold t (we use t = 0.5). 1 Periodically, the agent is retrained using all available data, thereby improving performance on the primary DIALOGUE task.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
It is important to note that the user's responses are always in the form of natural dialogue. In particular, at no point are the new FEEDBACK examples inspected, post-processed, or cleaned. Instead, we rely on the fact that the feedback is not random: regardless of whether it is a verbatim response, a description of a response, or a list of possible responses (see Table 2 for examples), there is a learnable relationship between conversation contexts and their corresponding feedback which requires many of the same language understanding skills to master as does carrying on a normal conversation.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
The experiments in this paper are limited to the setting where the number of supervised and deployment examples are on the same order of magnitude; however, we envision scenarios in which the number of deployment examples can easily grow to 100× or more the number of supervised examples over the chatbot's deployment lifetime, effectively providing a massive task-specific corpus at minimal cost. Table 1 reports the sizes of each dataset, all of which are available via ParlAI.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
The chatbot's primary task (DIALOGUE) is to carry on a coherent and engaging conversation with a speaking partner. Training examples take the form of (x, y) pairs, where x is the context of the conversation (the concatenation of all responses so far up to some history length, delimited with tokens marking the speaker), and y is the appropriate response given by the human.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
The Human-Human (HH) portion of the DIA-LOGUE dataset comes from the PERSONACHAT dataset (Zhang et al., 2018b), which consists of  short dialogs (6-8 turns) between two crowdworkers (humans) who have been assigned short text profiles and are instructed to "chat with the other person naturally and try to get to know each other." We chose this dataset because of its size (over 145k total examples), the breadth of topics it covers, and its focus on promoting engaging conversations, which we anticipate being a necessary property of a chatbot that people will be willing to chat with voluntarily and repeatedly. We use the standard splits of the dataset made available in ParlAI as a part of the ConvAI2 challenge (Burtsev et al., 2018). Since the question of how to incorporate external knowledge (such as profiles) in dialogue is an open research question of its own (Li et al., 2016;Luan et al., 2017;Luo et al., 2018) and we are primarily interested in the question of learning from dialogue, we discard the profiles and simply train and test on the conversations themselves, making the dataset more challenging in terms of raw performance scores.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
The Human-Bot (HB) portion of the DIA-LOGUE dataset is extracted during deployment as described earlier. The context may contain responses from both the human and the bot, but the target response is always from the human, as we will see experimentally that targeting bot responses degrades performance. Because the chitchat domain is symmetric, both the HH and HB DIALOGUE examples are used for the same task. In an asymmetric setting where the bot has a different role than the human, it is unclear whether HB examples may still be used as an auxiliary task, but FEEDBACK examples will remain usable.  
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
The objective of the SATISFACTION auxiliary task is to predict whether or not a speaking partner is satisfied with the quality of the current conversation. Examples take the form of (x, s) pairs, where x is the same context as in the DIALOGUE task, and s ∈ [0, 1], ranging from dissatisfied to satisfied. Crucially, it is hard to estimate from the bot's utterance itself whether the user will be satisfied, but much easier using the human's response to the utterance, as they may explicitly say something to that effect, e.g. "What are you talking about?". The dataset for this task was collected via crowdsourcing. Workers chatted with our baseline dialogue agent and assigned a rating 1-5 for the quality of each of the agent's responses. 2 Contexts with rating 1 were mapped to the negative class (dissatisfied) and ratings [3,4,5] mapped to the positive class (satisfied). Contexts with rating 2 were discarded to increase the separation between classes for a cleaner training set. Note that these numeric ratings were requested only when collecting the initial training data, not during deployment, where only natural dialogue is used.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
The objective of the FEEDBACK auxiliary task is to predict the feedback that will be given by the speaking partner when the agent believes it has made a mistake and asks for help. Examples take the form of (x, f ) pairs, where x is the same context as the other two tasks and f is the feedback utterance.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
2 A snapshot of the data collection interface and sample conversations are included in the Appendix. Training data for this task is collected during deployment. Whenever the user's estimated satisfaction is below a specified threshold, the chatbot responds "Oops! Sorry. What should I have said instead?". 3 A new example for the FEEDBACK task is then extracted using the context up to but not including the turn where the agent made the poor response as x and the user's response as f (as shown in Figure 1). At that point to continue the conversation during deployment, the bot's history is reset, and the bot instructs the user to continue, asking for a new topic. Examples of FEEDBACK responses are shown in Table 2.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
The self-feeding chatbot has two primary components: an interface component and a model component. The interface component is shared by all tasks, and includes input/output processing (tokenization, vectorization, etc.), conversation history storage, candidate preparation, and control flow (e.g., when to ask a question vs. when to give a normal dialogue response). The model component contains a neural network for each task, with embeddings, a network body, and a task head, some of which can be shared. In our case, we obtained maximum performance by sharing all parameters between the FEEDBACK and DIALOGUE tasks (prepending FEEDBACK responses with a special token), and using separate model parameters for the SATISFACTION task. Identifying optimal task structure in multi-task learning (MTL) architectures is an open research problem (Ruder, 2017). Regardless of what parameters are shared, each training batch contains examples from only one task at a time, candidate sets remain separate, and each task's cross-entropy loss is multiplied by a task-specific scaling factor tuned on the validation set to help account for discrepancies in dataset size, loss magnitude, dataset relevance, etc.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
Our dialogue agent's models are built on the Transformer architecture (Vaswani et al., 2017), which has been shown to perform well on a variety of NLP tasks (Devlin et al., 2018;Radford et al., 2018), including multiple persona-based chat applications (Shuster et al., 2018a,b;Rashkin et al., 2018). For the SATISFACTION task, the context x is encoded with a Transformer and converted to the scalar satisfaction predictionŝ by a final linear layer in the task head. The DIALOGUE and FEEDBACK tasks are set up as ranking problems, as in (Zhang et al., 2018b;Mazaré et al., 2018), where the model ranks a collection of candidate responses and returns the top-ranked one as its response. The context x is encoded with one Transformer andŷ andf candidates are encoded with another. The score for each candidate is calculated as the dot product of the encoded context and encoded candidate.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
During training, negative candidates are pulled from the correct responses for the other examples in the mini-batch. During evaluation, however, to remain independent of batch size and data shuffling, each example is assigned a static set of 19 other candidates sampled at random from its split of the data. During deployment, all 127,712 unique HH DIALOGUE candidates from the train split are encoded once with the trained model and each turn the model selects the top-ranked one for the given context.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
Contexts and candidates are tokenized using the default whitespace and punctuation tokenizer in ParlAI. We use a maximum dialogue history length of 2 (i.e., when making a prediction, the dialogue agent has access to its previous utterance and its partner's response). Tokens are embedded with fastText (Bojanowski et al., 2017) 300-dimensional embeddings. We do not limit the vocabulary size, which varies from 11.5k to 23.5k words in our experiments, depending on the training set. The Transformer is implemented in PyTorch (Paszke et al., 2017) within the ParlAI framework. We use the AdaMax (Kingma and Ba, 2014) optimizer with a learning rate schedule that decays based on the inverse square root of the step number after 500 steps of warmup from 1e-5. We use proportional sampling (Sanh et al., 2018) to select batches from each task for training, with batch size 128. Each Transformer layer has two attention heads and FFN size 32. The initial learning rate (0.001-0.005), number of Transformer layers (1-2), and task-specific loss factors (0.5-2.0) are selected on a per-experiment basis based on a grid search over the validation set averaged over three runs (we use the DIALOGUE validation set whenever multiple tasks are involved). We use early stopping based on the validation set to decide when to stop training. The hyperparameter values for the experiments in Section 5 are included in Appendix G.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
Note that throughout development, a portion of the DIALOGUE validation split was used as an informal test set. The official hidden test set for the DIALOGUE task was used only to produce the final numbers included in this paper.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
Throughout this section, we use the ranking metric hits@X/Y, or the fraction of the time that the correct candidate response was ranked in the top X out of Y available candidates; accuracy is another name for hits@1/Y. Statistical significance for improvement over baselines is assessed with a two-sample one-tailed T-test.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
Our main result, reported in Table 3, is that utilizing the deployment examples improves accuracy on the DIALOGUE task regardless of the number of available supervised (HH) DIALOGUE examples. 4 The boost in quality is naturally most pronounced when the HH DIALOGUE training set is small (i.e., where the learning curve is steepest), yielding an increase of up to 9.4 accuracy points, a 31% improvement. However, even when the entire PER-SONACHAT dataset of 131k examples is used-a much larger dataset than what is available for most dialogue tasks-adding deployment examples is still able to provide an additional 1.6 points of accuracy on what is otherwise a very flat region of  where the user appears to already be satisfied with the agent's responses, each FEEDBACK example corresponds to a mistake made by the model, giving the latter dataset a more active role in im-proving quality. Interestingly, our best-performing model, which achieves 46.3 accuracy on DIA-LOGUE, scores 68.4 on FEEDBACK, suggesting that the auxiliary task is a simpler task overall.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
When extracting HB DIALOGUE examples, we ignore human responses that the agent classifies as expressing dissatisfaction, since these turns do not represent typical conversation flow. Including these responses in the 60k HB dataset decreases hits@1/20 by 1.2 points and 0.6 points when added to 20k and 131k HH DIALOGUE examples, respectively. We also explored using chatbot responses with favorable satisfaction scores (ŝ > t) as new training examples, but found that our models performed better without them (see Appendix D for details).
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
We also found that "fresher" feedback results in bigger gains. We compared two models trained  
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
For maximum efficiency, we aim to ask for feedback when it will most benefit our model. The approach we chose (classifying the tone of partner responses) takes advantage of the fact that it is easier to recognize that a mistake has already been made than it is to avoid making that mistake; or in other words, sentiment classification is generally an easier task than next utterance prediction.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
We compare this to the approach of asking for feedback whenever the model is most uncertain what to say next. This approach acts on the as-sumption that the model will be least confident when it is about to make a mistake, which we find very frequently to not be the case. Not only is it difficult to recognize one's own mistakes, but also there are often multiple valid responses to a given context (e.g., "Yes, I love seafood!" or "Yuck, fish is gross.")-a lack of certainty about which to use does not necessarily suggest a poor model. Table 4 reports the maximum F1 scores achieved by each method on the SATISFACTION test set. For the model uncertainty approach, we tested two variants: (a) predict a mistake when the confidence in the top rated response is below some threshold t, and (b) predict a mistake when the gap between the top two rated responses is below the threshold t. We used the best-performing standalone DIALOGUE model (one trained on the full 131k training examples) for assessing uncertainty and tuned the thresholds to achieve maximum F1 score. For the user satisfaction approach, we trained our dialogue agent on just the SATISFAC-TION task. Finally, we also report the performance of a regular-expression-based method which we used during development, based on common ways of expressing dissatisfaction that we observed in our pilot studies, see Appendix F for details.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
As shown by Table 4, even with only 1k training examples (the amount we used for the experiments in Section 5.1), the trained classifier significantly outperforms both the uncertainty-based methods and our original regular expression, by as much as 0.28 and 0.42 F1 points, respectively.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
In this work we achieved learning from dialogue using two types of self-feeding: imitation of satisfied user messages, and learning from the feedback of unsatisfied users. In actuality, there are even more ways a model could learn to improve itself-for example, learning which question to ask in a given context to receive the most valuable feedback. One could even use the flexible nature of dialogue to intermix data collection of more than one type-sometimes requesting new FEED-BACK examples as in this work, and other times requesting new SATISFACTION examples (e.g., by asking "Did my last response make sense?"). In this way, a dialogue agent could simultaneously increase its dialogue ability, and increase its ability to improve further. We leave exploration of this meta-learning theme to future work.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
Here we report in greater detail the protocol we followed to collect the SATISFACTION, FEEDBACK, and HB DIALOGUE examples used Table 4 investigating the learning curve for this task.
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
No filtering was performed on the crowdworker conversations. Upon inspection after the fact, some workers did indeed give poor responses, make typographical mistakes, misunderstand the instructions, try to use the chatbot as a question answering interface, etc. We assume however that similar types of noise will be present in most chatbot deployment environments and opted to maintain a workflow that truly does not require developer intervention to use the newly collected examples.  
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
Our experiments use the PERSONACHAT distribution that was released as a part of the ConvAI2 (Burtsev et al., 2018) challenge. This distribution is slightly cleaner than the original PERSONACHAT release and comes with a new crowdsourced test set. In order to compare with the models and baselines used in the original PERSONACHAT paper (Zhang et al., 2018b), we report in this section the performance of our models on the original PERSONACHAT test set, not the ConvAI2 test set. Note that all numbers reported here are for models that do not have access to the profiles that were used in the creation of the conversations; models that do have access to this additional information tend to perform even better. Ours Transformer 49.6 Self-Feeding 51.7  We also considered whether it was possible to consistently identify really good responses by the chatbot, rather than the really bad ones. These could be potentially be used as DIALOGUE examples along with the ones that have human responses as targets (what we refer to as HH and HB in the paper). To explore this question, we modified our SATISFACTION dataset so that contexts with a rating of 5 were the positive class and ones with ratings [1, 2, 3] were the negative class (discarding ratings of 4 to increase the separation between classes). The results were negative-even with a training set of over 34k examples, the maximum precision we were able to achieve while maintaining at least 10% recall was 0.70, which is insufficient to improve performance on the DIALOGUE task. Upon inspection, it appears that really good responses are hard to identify because most of the time they look like a normal human-to-human conversation, and recognizing an appropriate next utterance is precisely the DIALOGUE task that we are trying to solve! Negative responses, however, are much more semantically similar to one another, since most express one of a few common ideas such as asking for clarification or conveying confusion.  
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
As described in Section 5.2, before we trained a classifier on the SATISFACTION task, we used the union of the following six regular expressions (using Python regular expression syntax) to identify user dissatisfaction and trigger feedback requests:
Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
r"i . * (?:said|asked|told). * " r"((not|nt|n't). * mak. * sense)|(mak. * no . * sense)" r"u(m|h)+\W" r"you. * what\?" r"what. * you (?:mean|refer|talk). * \?" r"what. * to do with. * \?"  Table 9: The hyperparameters used to obtain the results in Table 3.
Deploying Lifelong Open-Domain Dialogue Learning
Humans learn to use language over the course of their lives from the interactions they have with the world and other people. Yet, the prevailing dominant paradigm in natural language processing (NLP) research is to build a fixed dataset from which to train a model and then freeze it, without any ability for the model to interact with humans using language at training time at all. While we need such interaction in order to study human-machine communication to its full extent, constraints usually inhibit such research. Firstly, conducting such experiments can be costly. Many datasets in NLP are collected with crowdsourcing, whereby one pays the crowdworkers to perform interaction and * * Joint first authors. annotation tasks. This leads to several issues, not least that research budgets for paying crowdworkers mean that data will have a limit. Secondly, as crowdworkers are motivated by pay, not by interest in the actual tasks themselves, the data distribution may not match the desired one (de Vries et al., 2020).
Deploying Lifelong Open-Domain Dialogue Learning
In this work we study the ability of an opendomain 1 dialogue model to iteratively learn from conversations with intrinsically motivated humans. In order to engage humans at scale, we build and deploy a (free to play) game with a purpose (Von Ahn, 2006) whereby human players role-play characters and converse with other characters (that are our learning models) situated within the game world. We choose a fantasy game world, in order to maximize engagement. Our system iterates between collecting data of human-model interactions, retraining updated models on the newly collected data, and redeploying them. Simultaneously, it provides a natural metric to evaluate and compare models online using the continuation rate of players (how long they continue playing).
Deploying Lifelong Open-Domain Dialogue Learning
We show that we can successfully collect, retrain and redeploy models that improve both offline automatic metrics and human continue rates. Our overall system is engaging enough that we can collect data at a rate that is 1/5 th of the price per utterance of crowdsourcing, where the cost of our method is the cost of advertisements that make players aware of the game. Moreover, the data we collect is also more effective per utterance at improving continue rates due to being more on-distribution than crowdsourced data. As our models improve, these rates improve as well, as the continuation rate increases 1 In this work we study dialogue that can be about any topic but within the scope of a fantasy game world. Note this differs from open-domain dialogue talking about our world, e.g. the game players can talk about the sauce recipe from Bredwell across the sea (see Fig. 1), but not about the pizza in Chicago.
Deploying Lifelong Open-Domain Dialogue Learning
-meaning relatively more data is collected. Overall, our work provides good evidence that lifelong dialogue learning in deployed systems with intrinsically motivated humans (rather than crowdworkers) can be successful, in particular by embedding such learning within games.
Deploying Lifelong Open-Domain Dialogue Learning
The training code and parameters of the models deployed, and the data collected in this work will be made publicly available for reproducibility and further research by the community 2 .
Deploying Lifelong Open-Domain Dialogue Learning
In this section we describe the game that we will build and deploy, which is a dialogue role-playing game. It is a game with a purpose, and as such is designed to both train and evaluate open-domain dialogue agents.
Deploying Lifelong Open-Domain Dialogue Learning
Core Game The core game involves pairing two agents in a given setting -where one is a human player and the other is a dialogue agent with an underlying machine learning model. The two players are assigned characters, with given names and backstories (personas), and their current location and its description. See Figure 1 for examples. Each player's goal is simply to act out (role-play) their character's dialogue in the given situation. We refer to one such dialogue episode as a mini-game. Dialogue in the game is in English.
Deploying Lifelong Open-Domain Dialogue Learning
Role-Playing (Acting) Score We take advantage that role-playing is a pursuit that a large number of human players find fun (Horsfall and Oikonomou, 2011), and are hence naturally engaged in the openended nature of this process. However, to encourage and further motivate players to play their characters well, we introduce the concept of an (automated) dungeon master (DM), who will assess the quality of the player's role-playing. For each dialogue turn, we apply a learned model to the human player's dialogue, which assesses how likely their utterance is given the context. We convert this to a score, between 1 and 5 stars, that is presented to the human player, to reward them for good acting.
Deploying Lifelong Open-Domain Dialogue Learning
While this signal is noisy, because our DM model is not perfect, it gives motivating feedback to the players to continue playing.
Deploying Lifelong Open-Domain Dialogue Learning
Other Gamification Steps The acting scores (between 1-5 stars per turn) are accumulated, and a player's total score is presented on a leaderboard compared to all other players, providing further motivation to reach the top of the leaderboard. We also award "badges" if, for a given dialogue, a certain number of points are collected (11 for 1 badge, 16 for two): the badges represent the characters in the game, motivating the desire to role-play all the characters in the game, and collect all the badges.
Deploying Lifelong Open-Domain Dialogue Learning
Game Loop Each dialogue (mini-game) consists of 6 turns of dialogue per agent (12 total). At the end of the mini-game the human player is presented with four choices: (i) choose to move to a new location, where they will continue to play this character, but meet a new character to converse with; (ii) stay in the same room but wait for a new character to arrive to converse with; (iii) change to role-play a completely new pair of characters in a new setting; or (iv) end the game. These choices encourage the player to choose another mini-game that they are most interested in, and the variety of mini-games gives different role-playing possibilities, making the dialogue data more diverse.
Deploying Lifelong Open-Domain Dialogue Learning
License Agreement and Public Release Upon entry to the game, players are asked to agree to the use and release of the resulting game data as a publicly available dataset for research purposes. They are urged to stick to their assigned characters in the game, and hence should not use any personally identifying information, which the terms also tell them explicitly not to share. In the released data, no other information about the player is retained except for the messages they send in the game.
Deploying Lifelong Open-Domain Dialogue Learning
Game Safety We employ a safety classifier (Dinan et al., 2019b) on both human and model turns. For safety reasons, we limit our dialogue models to be retrieval models, so that we could vet the entire set of candidates for offensive language before runtime. The set of settings and character personas were all also vetted for offensive language. Additionally, gender bias concerns have been previously studied within the available LIGHT MTurk training set (Dinan et al., 2019a), and we make use of that publicly available data here as well. We note that, compared to other deployed dialogue systems, there is an extra level of indirection due to playing characters in a game that makes language relatively less offensive. For example, a thief in the game saying "I'm going to steal your money" to another game character is far less offensive compared to a digital assistant saying it directly to a human user.
Deploying Lifelong Open-Domain Dialogue Learning
Retrieval Models All the models we have currently deployed are retrieval models (see previous discussion of safety). In particular, we use the Poly-Encoder (PE) Transformer architecture as a base , as it provides state of the art results compared to other retrieval models, whilst being tractable to deploy. PE encodes the context with a standard bidirectional transformer, but produces an encoding into a fixed small number of codes, N . We tried values of N = 5 and N = 20. Each label candidate then attends to these codes before producing a final matching score. The model is trained with cross-entropy given the correct label, and by subsampling negative examples from the given training batch.
Deploying Lifelong Open-Domain Dialogue Learning
Architecture and Training Choices We employ the 90M and 622M parameter models from  that have been pre-trained on 1.5B training examples from pushshift.io Reddit, which we then fine-tune. We also consider two other enhancements, chosen to mitigate problems that we observed with the models: (i) negative context training, whereby negatives are also selected from the immediate dialogue history as well as the batch which can help reduce a model's tendency to repeat itself (Holtzman et al., 2019;Welleck et al., 2020); and (ii) decoding control (See et al., 2019) whereby at decoding time responses are rescaled before scoring based on their specificity (normalized inverse document frequency). The latter can control the genericness of the responses, which is known to affect human judgments.
Deploying Lifelong Open-Domain Dialogue Learning
Generative Models In addition to the deployed models, we also perform training and automatic evaluation metrics on generative models offline, where safety concerns are less important as the models are not user-facing. We employ an encoderdecoder Transformer architecture using the state of the art pre-trained 2.7 billion parameter BlenderBot model , which we fine-tune on our task.
Deploying Lifelong Open-Domain Dialogue Learning
Agent Dialogue Model Training a dialogue model involves one of the setups described above, and a set of (dialogue context, correct label) pairs. We will train on such pairs both from crowdsourced data and data collected within game in our lifelong learning setup. All training and experiments are performed using the ParlAI software framework .
Deploying Lifelong Open-Domain Dialogue Learning
Acting Score Model We can apply a retrieval model to also score the human's role-playing abilities. In this case, the context is the entire dialogue history, setting and the player's character persona as input to the encoder, while the candidates to score are the ones from the training set, as usual, plus additionally the human's (player's) actual response. For speed, the encoder can actually be run while the human player is typing, as it does not depend on their response, which is treated as a candidate label instead. The score given to the user is then proportional to the human response's rank amongst all the candidates 3 .
Deploying Lifelong Open-Domain Dialogue Learning
After collecting a certain amount of episodes of conversational data between humans and models, one can consider using this data for training. We utilize the following observation: while the model utterances may contain many mistakes, it is assumed that a human sufficiently engaged provides high quality responses, even if the model responses are mistakes, and can thus be treated as gold, and used as a fully supervised signal. We thus separate the dialogue data into all possible (context, next utterance) pairs, and then only consider the pairs with human next utterances as training data. We also compare this to further filtering this set by scoring the quality of the human utterances, discarding those episodes (mini-games) with lower quality. We use the acting score model previously described for this purpose, summing the scores obtained across an episode, and discarding the episode if this value is less than C, where C is a hyperparameter tuned on the validation set. After training our model from a given round of collection, we can go back to the collection process utilizing instead the new model that has been trained on more data. The hypothesis is that the higher quality the model is: (i) the higher quality the human data will be as well; and (ii) the more likely the human players are to converse longer, increasing the data set size by larger amounts.
Deploying Lifelong Open-Domain Dialogue Learning
Apart from the collection-training cycle of our deployed lifelong learning setup, one can also in parallel perform evaluation. For each separate minigame (episode of dialogue) we can potentially deploy a different model for human-model conversation. We maintain a pool of models with differing architectures or hyperparameters, and select randomly from the pool in each episode. For any given episode we record whether the player continued playing to the next mini-game or not, which we refer to as the continue rate. We can measure the quality of a model using its averaged continue rate over all players and episodes. In this way we can also perform model selection online. 3 The player is awarded 2 stars if their response is in the top 2000, 3 stars in the top 1000, and 4 stars in the top 100.
Deploying Lifelong Open-Domain Dialogue Learning
We performed three rounds of our lifelong learning setup.
Deploying Lifelong Open-Domain Dialogue Learning
Round 1 consists of models trained on LIGHT MTurk data only. We train the retrieval model variants described in Section 4.1, and deploy them within the game.
Deploying Lifelong Open-Domain Dialogue Learning
Round 2 consists of models trained on LIGHT MTurk data + 50,982 WILD examples collected from the deployment of the Round 1 models, and again deploy these within the game. 
Deploying Lifelong Open-Domain Dialogue Learning
While our setup is a lifelong learning setup and the models are still currently deployed and collecting data, for this paper we froze the collection at a given point in order to provide a data release and provide experimental results. The data statistics for the total newly collected dataset, called LIGHT WILD, over all rounds is shown in Table 1. Validation and test sets were extracted from a portion of the data 4 from Round 2. Table 2 compares this dataset to several existing commonly used open-domain dialogue datasets. The number of episodes and dialogue utterances are larger than many existing datasets, e.g. four times as many as LIGHT MTurk, and almost eight times that of Empathetic Dialogues. Uniquely, our dataset contains human-model conversations, hence the total number of human utterances is actually half of the utterances, which is still twice as large as the number in LIGHT MTurk. Our dataset also has a large degree of diversity, which is important for tasks in general, and especially for open-domain dialogue. The number of unique locations and roles that can be played by speakers (characters) is large (587 and 630, respectively). The number of players of the game at the time of freezing was over 13,000, which also makes the diversity far larger than typical crowdsourced datasets, e.g. LIGHT MTurk involved 1,052 crowdworkers   and Empathetic Dialog involved 810 crowdworkers. Finally, the number of unique tokens is larger in LIGHT WILD, indicating the diversity of language used.
Deploying Lifelong Open-Domain Dialogue Learning
While we only deployed retrieval models, we report experiments training both retrieval models and generative models on the data from the three rounds, selecting best hyperparameters using the validation set. We report the performance on three different test sets: LIGHT (MTurk) Seen and Unseen test sets (Urbanek et al., 2019), where unseen means that the test locations do not overlap with the training set locations, and our WILD test set. The results are given in Table 3. They show a steady increase in the Hits@1/20 metric (Top 1 accuracy given 19 random distractors) for the retrieval models over the rounds on all three test sets, and a similar decrease in perplexity (PPL) for the generative models. In particular there is a large jump in the performance on the WILD Test set between Rounds 1 and 2 as the training set switches from crowdsourced to in-distribution WILD data, and a further increase in Round 3 as more data is again collected and retrained on. While our WILD data is of a different distribution to the two LIGHT (MTurk) test sets, the data collection from our lifelong learning setup still gives gains on those tests as well. Our reported numbers, as far as we are aware, are the best re-ported numbers on these datasets, e.g. the original LIGHT paper reports 76.5% and 70.5% for the Seen and Unseen test sets, respectively (compared to our 87.72% and 83.48%). Overall, we see clear gains from the extra data collected in our setup.
Deploying Lifelong Open-Domain Dialogue Learning
We construct learning curves given all the collected data to analyze the performance gain per new training example. We plot Hits@1/20 accuracy on the WILD validation set against the number of training examples, comparing data from WILD collection to LIGHT (Mturk). We also consider a 50/50 mix, where we equally sample from the two sources LIGHT+WILD to provide the next training example. Figure 2 (left) shows the results. We observe that on a per-example basis our WILD data gives more accuracy gains than LIGHT MTurk data, e.g. 83.59% for WILD compared to 80.63% for LIGHT, when limiting WILD to the same training set size as the total size of LIGHT. As the WILD dataset is more than twice as large this monotonically improves further, up to 85.95% using all of the WILD data. We observe that the improvements have not saturated and that further lifelong learning will bring further model improvements. Combining the two data sources, as shown in the LIGHT+WILD plot, brings yet further gains, up to 87.2%. Overall, our collected WILD data has high quality as a learning signal for training models.   . The cost axis is in units scaled by the cost of LIGHT WILD collection required to achieve the same performance as using the entire LIGHT MTurk dataset; it is more than 8× cheaper to use LIGHT WILD examples than LIGHT MTurk examples to achieve an accuracy of 80.63%. We also show performance for models which equally sample data from LIGHT MTurk+WILD datasets for training; utilizing all the data from both sources yields the best performance. However, LIGHT WILD data gives better accuracy improvements per training example (left plot).
Deploying Lifelong Open-Domain Dialogue Learning
We plot similar learning curves, but as a function of the cost to collect the data instead of the number of training examples instead, see Figure 2 (right). Although we do not pay players to play the game, we did spend money to advertise the game online in order to attract players. We compare the cost per WILD example relatively to the cost per LIGHT (MTurk) example, where the x-axis is scaled in units that are multiples of the cost required to achieve 80.63% using WILD data (as this is the performance of using all the LIGHT MTurk data together). We observe that it costs over 8x more to achieve the same accuracy using LIGHT (MTurk) data (see dashed horizontal line). The actual cost per utterance of WILD data is around 1/5 th of the price of LIGHT MTurk data, but more than that, it is also relatively more effective per utterance in terms of metrics. For the same amount spent there is a large gap between the two systems, for example using all the WILD data gives a performance of 85.95%, whereas for the same spend LIGHT MTurk only achieves ∼77.5%. Overall, WILD deployment is relatively a very cost effective strategy.
Deploying Lifelong Open-Domain Dialogue Learning
Our lifelong learning setup deploys multiple models (see Sec. 4.3) at the same time randomly assigning them to concurrent users per episode (minigame). We can thus directly compare the quality of models via their continue rate. Continue rates during Round 2 of collection comparing several model variants are given in Table Figure 3: Predicted Data Quality. Left: Hits@1/20 accuracy on the WILD validation set when training with LIGHT MTurk + 10,000 examples from the WILD training set of a given predicted quality level, see Sec. 5.3.5. Data that is predicted to be higher quality yields improved validation accuracies. Right: The distribution of data quality predictions over the training set. A spike is seen at quality bin 6 because that is the lowest score one can achieve when completing a full episode (1 star per turn is awarded at minimum). Values lower than bin 5 indicate incomplete low-scoring episodes.  Table 4: Deployment-based Evaluation, comparing several metrics on data collected during Round 2 of collection.
Deploying Lifelong Open-Domain Dialogue Learning
∆ Continue Rate + WILD train data (Round 2) +1.3 ± 0.7% 90M → 622M parameters PE −3.2 ± 0.7% + Negative context training +2.6 ± 1.3% + Decoding control +2.5 ± 1.1% methods used together improving more. We confirm these conclusion in Table 5 where we show the change in continue rates when independently adjusting one of these factors, by averaging over model continue rates for other factors of variation.
Deploying Lifelong Open-Domain Dialogue Learning
We also observe the unexpected result that the larger models perform worse than the small models across the board on continue rates. Deeper analysis given in appendix C suggests that while the larger model makes less mistakes, it is more often seen as boring, which would reasonably impact a player's desire to continue playing. Understanding and controlling this trade-off should be studied further.
Deploying Lifelong Open-Domain Dialogue Learning
Not every player is as engaged in the game as every other player, or produces as high quality dialogue. We hypothesize that we can predict which players produce higher quality data via the acting score model (Sec. 4.1), and that such higher quality data is relatively better for training models. Figure 3 (right) shows the distribution over the WILD training set of predicted quality using the acting score model. We observe 83.7% of the episodes have a score above the minimum value of 6 (there are 6 turns, and on each turn a score between 1-4 is awarded, explaining the spike at the value of 6). Scores below 6 indicate incomplete dialogues, which only account for 4.0% of the data.
Deploying Lifelong Open-Domain Dialogue Learning
To assess whether these scores are indeed indicators of data quality, we selected an equal amount of 10,000 examples from each of the bins (1-5), (6), (7), . . . , (16) (grouping 1-5 together to make that group large enough) and compared them as training sources. We train a set of retrieval models on these training sources, where each model also has access to all of the LIGHT MTurk data (111k examples) in addition to the WILD 10k from their respective bins. The results are given in Figure 3 (left). We observe a monotonic improvement on the WILD validation set with increasing predicted quality. We see similar, but smaller gains on the LIGHT (Seen) validation set as well, e.g. 86.59% for quality bin 6, and 87.11% for quality bin 16.
Deploying Lifelong Open-Domain Dialogue Learning
While we can clearly select lower or higher quality data, we can also ask the question whether some of the data is so low quality we should simply remove it from the training data in order to get better performance. Experiments show that is not the case, and that even the lowest quality data does provide a useful signal, e.g. performance drops slightly from 87.06% to 86.69% on the WILD validation set if we remove bins lower than 6, but otherwise training on all other data, and to 85.38% if we remove bins lower than 9.
Deploying Lifelong Open-Domain Dialogue Learning
Just as the design of a crowdsourcing task will affect the cost and quality of data, this is likely even more the case in the design of a game. If the design is poor, players will not play it at all; whereas in contrast to paying crowdworkers, if players really like a game, they are willing to pay to play it. Accordingly, the plots we presented in Figure 2 represent the results of our particular game design; there may well be a design with vastly more cost efficient learning rates. While a full of study of the elements of game design is outside of the scope of this paper, we note that for adjustments we did make to the game after initial deployment we observed large changes in user behavior. For example, after the addition of the three user controls for how to continue the game loop after an episode is finished (as described in Sec. 3), compared to only a single choice, we saw an increase in the continue rate by 3.3 ± 1.6% when using the same model.
Deploying Lifelong Open-Domain Dialogue Learning
Model quality also affects cost and quality of the data collected. Noting the effects of changing gamification options (alongside other hard-totrack circumstances) we only report continue rate comparisons between models relative to runs in the same batch. Still, players' enjoyment of these models (as estimated by continue rate in Table 4) directly changes how much they engage with the game. As such it is more expensive to test models that are worse for the game experience (which we would consider fair from a player perspective). Hence, as models improve, costs actually go down, enabling to collect data at higher rate.
Deploying Lifelong Open-Domain Dialogue Learning
We can compare the dialogue data collected within our deployed system to crowdsourced data from LIGHT MTurk. We analyze over and underexpressed words in our dataset compared to the latter.
Deploying Lifelong Open-Domain Dialogue Learning
Calculating the top 70 most overexpressed words, all overexpressed at least 3.5x relative to crowdsourced data, we note several interesting observations about our data's distribution: -There are more natural endings to conversations: e.g. "goodbye" (4×) and "bye" (4×) are overexpressed.
Deploying Lifelong Open-Domain Dialogue Learning
-There are overexpressed words associated with aggression: "stab" (8.5×), "dagger" (6.1×), "club" (5.5×), "kills" (4.9×), "blade" (4.2×). -There are overexpressed words associated with overtly friendly actions as well: "smiles" (12.9×), "nods" (10.9×), "kiss" (6.1×), "hug" (3.7×), and "bows" (5.9×). -There are more mentions of adventuring: "quest" (5.4×), and other similar words not in the top 70 are overexpressed as well, such as "adventure" (2.5×) and "mission" (2.1×). -There is an increased use of slang: "ur" (93×), "u" (28×), "yo" (5×), "dude" (6×). We note that some emojis exist in the dataset as well, which do not appear at all in the crowdsourced data.
Deploying Lifelong Open-Domain Dialogue Learning
In contrast, looking at the 70 most underexpressed words, all underexpressed by a factor of at least 1.3×, we observed the following patterns: -Less mentions of village and farm life: "peasant", "fields" (both 2× underexpressed), "farm" and "crops" (both 1.9×), "harvest" (1.8×), "villagers" (1.7×), and "work" (1.4×).
Deploying Lifelong Open-Domain Dialogue Learning
-Less mention of passages of time: "week" (2.1×), "year" (1.9×), "days" (1.8×).
Deploying Lifelong Open-Domain Dialogue Learning
Overall, we see a pattern that game players seek more exciting conversations, involving more emotional, action-packed interactions such as seeking quests, whereas crowdworkers are more evenkeeled, and able to discuss dry topics such as last year's harvest or taxes with more frequency. This is not unexpected as game players often seek immediacy and larger-than-life experiences (Grodal et al., 2000).
Deploying Lifelong Open-Domain Dialogue Learning
We have presented a fully realized system for improving upon an open-domain dialogue task by utilizing a deployed game for lifelong learning. Detailed experiments showed that the one can collect high quality data that improves both automatic offline metrics and user engagement metrics when used for training models. We find this exciting because this approach shows it is possible to build continually improving models that learn from interacting with humans in the wild (as opposed to experiments with paid crowdworkers), which represents a paradigm shift away from the limited static dataset setup that is prevalent in much of the work of the community.
Deploying Lifelong Open-Domain Dialogue Learning
Future work should study the resulting publicly released data to explore other methods of lifelong learning, or other learning signals that could be extracted from human utterances, for example the ideas in Hancock et al. (2019). Another possible direction, for when model performance begins to saturate, is to exploit control of the game engine itself to emphasize learning on the most difficult cases or the ones with the most learning signal, such as in the work on adversarial collection (Yang et al., 2017;Nie et al., 2019). Finally, our roleplaying setup can also be applied more generally, for example incorporating both dialogue and actions, situated in other domains.
Deploying Lifelong Open-Domain Dialogue Learning
We show game screenshots in Figures 4 and 5.
Deploying Lifelong Open-Domain Dialogue Learning
In our main results we use the human utterances collected from the role-playing game to form the WILD dataset targets, the hypothesis being that model utterances may or not be correct, and so are not as good a signal. A contrasting view could see training on the model utterance data as a kind of distillation of the previous model's knowledge. To test the performance of WILD human vs. model utterances, we conducted further experiments comparing them to each other. We observe a significant drop in Hits@1/20 performance using the model utterances for training on the WILD validation set (86.05% for human utterances, and 73.96% for model), and similarly on the LIGHT validation set (82.32% for human utterances, and 77.56% for model).
Deploying Lifelong Open-Domain Dialogue Learning
We evaluated the differences between the small and large retrieval models (Sec. 4.1) during deployment to analyze the differences between them. The expectation tends to be that larger models would perform better, especially when the automatic metrics reflected this case. To eliminate the possibility that difficulty in tuning the decoding control differed between the small and large models, we launched a task to crowdworkers to evaluate the models shown in rows 3 and 8 of Table 4. These correspond to the small and large models trained with LIGHT + WILD data without decoding control, further noted as LW90M and LW622M.
Deploying Lifelong Open-Domain Dialogue Learning
Workers were asked to have a 12-turn conversation with a model (6 turns each), and evaluate each model utterance with respect to some of the most common mistakes we observed models making. They were then asked to provide an overall score for the model, answering "How much fun did you have Role-playing with your chat partner? Would you have a similar conversation with this chat partner in the future?". Results are given in Table 6, listing the percentage of utterances falling into each mistake type.
Deploying Lifelong Open-Domain Dialogue Learning
We observed that LW622M avoided a number of common mistakes when compared to LW90M, contradicting itself less frequently, assuming the role of the wrong character or using a wrong location less frequently, and going off-topic less frequently. These types of mistakes were common complaints of players interacting with our game, and all seemed to be related to having somewhat bad experiences. LW622M also had a higher percentage of conversations without any listed issues, with 34.1% compared to 27.3% for LW90M.
Deploying Lifelong Open-Domain Dialogue Learning
LW622M however was rated as being repetitive or boring more than twice as frequently as the smaller model. This difference could be supported in that LW622M used utterances that were on average 2 words shorter than those of LW90M over these evaluations. This issue could explain the phenomenon where players are more likely to leave after interacting with a larger model than a smaller one. The only thing really encouraging unpaid players to continue interacting with our models is  Table 6: Percentage of utterances flagged with an issue alongside overall satisfaction, by model. that they are fun and engaging, and while it is possible to overlook the model making a mistake if it is still somewhat fun, it is likely less possible to remain engaged when the model is actually boring and repetitive. While LW622M is a better model across several aspects, it is clear from our live deployment evaluations that something is lost in scaling up from LW90M. Comparing these models with real players lets us see this issue, and moving forward should lead us to search for a model that does not become more boring when learning to not make mistakes.