How have prior work incorporated personality attributes to train personalized dialogue generation models?
Personalized Dialogue Generation Open-
domain dialogue has been studied in depth for a
long time (Koehn et al., 2003; Ni et al., 2021),
and under the influence of the psychological
theory, personality has been incorporated into
the requirements for dialogue generation. Per-
sonalized dialogue generation has three typical
approaches: (1) Using well-defined sparse persona
attributes (e.g., gender, age), the model can utilize
different attributes efficiently and interpretably,
and knowledge-enhanced dialogue generation
approaches can be borrowed (Zhang et al.,
2018a; Song et al., 2019; Wolf et al., 2019; Liu
et al., 2020; Bao et al., 2020; Song et al., 2021).
However, sparse attributes can only provide little
persona information without complex semantics.
(2) Mining information from dense textual persona
descriptions, which contain rich and deep persona
information but are very noisy (Qian et al., 2018;
Song et al., 2020; Zheng et al., 2020; Song et al.,
2021). (3) Implicitly modeling persona profiles
from historical dialogue query (Li et al., 2016b;
Ma et al., 2021; Zhong et al., 2022). This approach
does not rely on additional persona information,
but it is difficult to acquire personality implicitly
from dialogue history without reference objects.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Maintaining a consistent persona is essential for dialogue agents. Although tremendous advancements have been brought, the limited-scale of annotated personalized dialogue datasets is still a barrier towards training robust and consistent persona-based dialogue models. This work shows how this challenge can be addressed by disentangling persona-based dialogue generation into two sub-tasks with a novel BERT-over-BERT (BoB) model. Specifically, the model consists of a BERT-based encoder and two BERT-based decoders, where one decoder is for response generation, and another is for consistency understanding. In particular, to learn the ability of consistency understanding from large-scale non-dialogue inference data, we train the second decoder in an unlikelihood manner. Under different limited data settings, both automatic and human evaluations demonstrate that the proposed model outperforms strong baselines in response quality and persona consistency.
[{"first":"Haoyu","last":"Song","middle":[]},{"first":"Yan","last":"Wang","middle":[]},{"first":"Kaiyan","last":"Zhang","middle":[]},{"first":"Wei-Nan","last":"Zhang","middle":[]},{"first":"Ting","last":"Liu","middle":[]}]
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
.
[{"first":"Qiao","last":"Qian","middle":[]},{"first":"Minlie","last":"Huang","middle":[]},{"first":"Haizhou","last":"Zhao","middle":[]},{"first":"Jingfang","last":"Xu","middle":[]},{"first":"Xiaoyan","last":"Zhu","middle":[]}]
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Personalized chatbots focus on endowing chatbots with a consistent personality to behave like real users, give more informative responses, and further act as personal assistants. Existing personalized approaches tried to incorporate several text descriptions as explicit user profiles. However, the acquisition of such explicit profiles is expensive and time-consuming, thus being impractical for large-scale real-world applications. Moreover, the restricted predefined profile neglects the language behavior of a real user and cannot be automatically updated together with the change of user interests. In this paper, we propose to learn implicit user profiles automatically from large-scale user dialogue history for building personalized chatbots. Specifically, leveraging the benefits of Transformer on language understanding, we train a personalized language model to construct a general user profile from the user's historical responses. To highlight the relevant historical responses to the input post, we further establish a key-value memory network of historical post-response pairs, and build a dynamic post-aware user profile. The dynamic profile mainly describes what and how the user has responded to similar posts in history. To explicitly utilize users' frequently used words, we design a personalized decoder to fuse two decoding strategies, including generating a word from the generic vocabulary and copying one word from the user's personalized vocabulary. Experiments on two real-world datasets show the significant improvement of our model compared with existing methods. Our code is available at https://github.com/zhengyima/DHAP
[{"first":"Zhengyi","last":"Ma","middle":[]},{"first":"Zhicheng","last":"Dou","middle":[]},{"first":"Yutao","last":"Zhu","middle":[]},{"first":"Hanxun","last":"Zhong","middle":[]},{"first":"Ji-Rong","last":"Wen","middle":[]}]
Exploiting Persona Information for Diverse Generation of Conversational Responses
In human conversations, due to their personalities in mind, people can easily carry out and maintain the conversations. Giving conversational context with persona information to a chatbot, how to exploit the information to generate diverse and sustainable conversations is still a non-trivial task. Previous work on persona-based conversational models successfully make use of predefined persona information and have shown great promise in delivering more realistic responses. And they all learn with the assumption that given a source input, there is only one target response. However, in human conversations, there are massive appropriate responses to a given input message. In this paper, we propose a memory-augmented architecture to exploit persona information from context and incorporate a conditional variational autoencoder model together to generate diverse and sustainable conversations. We evaluate the proposed model on a benchmark persona-chat dataset. Both automatic and human evaluations show that our model can deliver more diverse and more engaging persona-based responses than baseline approaches.
[{"first":"Haoyu","last":"Song","middle":[]},{"first":"Wei-Nan","last":"Zhang","middle":[]},{"first":"Yiming","last":"Cui","middle":[]},{"first":"Dong","last":"Wang","middle":[]},{"first":"Ting","last":"Liu","middle":[]}]
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Endowing dialogue systems with personas is essential to deliver more human-like conversations. However, this problem is still far from well explored due to the difficulties of both embodying personalities in natural languages and the persona sparsity issue observed in most dialogue corpora. This paper proposes a pre-training based personalized dialogue model that can generate coherent responses using persona-sparse dialogue data. In this method, a pre-trained language model is used to initialize an encoder and decoder, and personal attribute embeddings are devised to model richer dialogue contexts by encoding speakers' personas together with dialogue histories. Further, to incorporate the target persona in the decoding process and to balance its contribution, an attention routing structure is devised in the decoder to merge features extracted from the target persona and dialogue contexts using dynamically predicted weights. Our model can utilize persona-sparse dialogues in a unified manner during the training process, and can also control the amount of persona-related features to exhibit during the inference process. Both automatic and manual evaluation demonstrates that the proposed model outperforms state-of-the-art methods for generating more coherent and persona consistent responses with persona-sparse data.
[{"first":"Yinhe","last":"Zheng","middle":[]},{"first":"Rongsheng","last":"Zhang","middle":[]},{"first":"Xiaoxi","last":"Mao","middle":[]},{"first":"Minlie","last":"Huang","middle":[]}]
A Persona-Based Neural Conversation Model
We present persona-based models for handling the issue of speaker consistency in neural response generation. A speaker model encodes personas in distributed embeddings that capture individual characteristics such as background information and speaking style. A dyadic speaker-addressee model captures properties of interactions between two interlocutors. Our models yield qualitative performance improvements in both perplexity and BLEU scores over baseline sequence-to-sequence models, with similar gains in speaker consistency as measured by human judges.
[{"first":"Jiwei","last":"Li","middle":[]},{"first":"Michel","last":"Galley","middle":[]},{"first":"Chris","last":"Brockett","middle":[]},{"first":"Georgios","last":"Spithourakis","middle":["P."]},{"first":"Jianfeng","last":"Gao","middle":[]},{"first":"Bill","last":"Dolan","middle":[]}]
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
We introduce a new approach to generative data-driven dialogue systems (e.g. chatbots) called TransferTransfo which is a combination of a Transfer learning based training scheme and a high-capacity Transformer model. Fine-tuning is performed by using a multi-task objective which combines several unsupervised prediction tasks. The resulting fine-tuned model shows strong improvements over the current state-of-the-art end-to-end conversational models like memory augmented seq2seq and information-retrieval models. On the privately held PERSONA-CHAT dataset of the Conversational Intelligence Challenge 2, this approach obtains a new state-of-the-art, with respective perplexity, Hits@1 and F1 metrics of 16.28 (45 % absolute improvement), 80.7 (46 % absolute improvement) and 19.5 (20 % absolute improvement).
[{"first":"Thomas","last":"Wolf","middle":[]},{"first":"Victor","last":"Sanh","middle":[]},{"first":"Julien","last":"Chaumond","middle":[]},{"first":"Clement","last":"Delangue","middle":[]}]
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Chit-chat models are known to have several problems: they lack specificity, do not display a consistent personality and are often not very captivating. In this work we present the task of making chit-chat more engaging by conditioning on profile information. We collect data and train models to (i)condition on their given profile information; and (ii) information about the person they are talking to, resulting in improved dialogues, as measured by next utterance prediction. Since (ii) is initially unknown our model is trained to engage its partner with personal topics, and we show the resulting dialogue can be used to predict profile information about the interlocutors.
[{"first":"Saizheng","last":"Zhang","middle":[]},{"first":"Emily","last":"Dinan","middle":[]},{"first":"Jack","last":"Urbanek","middle":[]},{"first":"Arthur","last":"Szlam","middle":[]},{"first":"Douwe","last":"Kiela","middle":[]},{"first":"Jason","last":"Weston","middle":[]}]
Profile Consistency Identification for Open-domain Dialogue Agents
Maintaining a consistent attribute profile is crucial for dialogue agents to naturally converse with humans. Existing studies on improving attribute consistency mainly explored how to incorporate attribute information in the responses, but few efforts have been made to identify the consistency relations between response and attribute profile. To facilitate the study of profile consistency identification, we create a large-scale human-annotated dataset with over 110K single-turn conversations and their key-value attribute profiles. Explicit relation between response and profile is manually labeled. We also propose a key-value structure information enriched BERT model to identify the profile consistency, and it gained improvements over strong baselines. Further evaluations on downstream tasks demonstrate that the profile consistency identification model is conducive for improving dialogue consistency.
[{"first":"Haoyu","last":"Song","middle":[]},{"first":"Yan","last":"Wang","middle":[]},{"first":"Wei-Nan","last":"Zhang","middle":[]},{"first":"Zhengyu","last":"Zhao","middle":[]},{"first":"Ting","last":"Liu","middle":[]},{"first":"Xiaojiang","last":"Liu","middle":[]}]
You Impress Me: Dialogue Generation via Mutual Persona Perception
Despite the continuing efforts to improve the engagingness and consistency of chit-chat dialogue systems, the majority of current work simply focus on mimicking human-like responses, leaving understudied the aspects of modeling understanding between interlocutors. The research in cognitive science, instead, suggests that understanding is an essential signal for a high-quality chit-chat conversation. Motivated by this, we propose Pˆ2 Bot, a transmitter-receiver based framework with the aim of explicitly modeling understanding. Specifically, Pˆ2 Bot incorporates mutual persona perception to enhance the quality of personalized dialogue generation. Experiments on a large public dataset, Persona-Chat, demonstrate the effectiveness of our approach, with a considerable boost over the state-of-the-art baselines across both automatic metrics and human evaluations.
[{"first":"Qian","last":"Liu","middle":[]},{"first":"Yihong","last":"Chen","middle":[]},{"first":"Bei","last":"Chen","middle":[]},{"first":"Jian-Guang","last":"LOU","middle":[]},{"first":"Zixuan","last":"Chen","middle":[]},{"first":"Bin","last":"Zhou","middle":[]},{"first":"Dongmei","last":"Zhang","middle":[]}]
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
Pre-training models have been proved effective for a wide range of natural language processing tasks. Inspired by this, we propose a novel dialogue generation pre-training framework to support various kinds of conversations, including chit-chat, knowledge grounded dialogues, and conversational question answering. In this framework, we adopt flexible attention mechanisms to fully leverage the bi-directional context and the uni-directional characteristic of language generation. We also introduce discrete latent variables to tackle the inherent one-to-many mapping problem in response generation. Two reciprocal tasks of response generation and latent act recognition are designed and carried out simultaneously within a shared network. Comprehensive experiments on three publicly available datasets verify the effectiveness and superiority of the proposed framework.
[{"first":"Siqi","last":"Bao","middle":[]},{"first":"Huang","last":"He","middle":[]},{"first":"Fan","last":"Wang","middle":[]},{"first":"Hua","last":"Wu","middle":[]},{"first":"Haifeng","last":"Wang","middle":[]}]
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Various approaches have been explored to introduce explicit personas in dialogue models (Qian et al., 2018;Zheng et al., 2020;. The PERSONA can be defined as a composite of elements of identity, such as profiles and background personal facts. In persona-based dialogues, the generated responses are conditioned not only on the dialogue context but also on some predefined personas, so the presenting personality could be more consistent.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Existing persona-based dialogue models heavily utilize a set of persona-related dialogue data Golovanov et al., 2019), such as the PersonaChat (Zhang et al., 2018). This kind of crowd-sourced dataset covers rich persona features, * Wei-Nan Zhang is the corresponding author.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Persona: I've a son who is in junior high school Query:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
You have any children?
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
GPT-2: No kids. I work at home depot so I'm busy. namely "persona-dense". Nevertheless, the scale of such crowd-sourced datasets is limited by the expensive costs: two annotators are asked to act the part of a given provided persona and chat naturally to get to know each other during the conversation. On the other hand, conversations in daily life are not always persona-related. According to Twitter content analysis, less than 10% messages on Twitter reveal personal anecdote or activities at home or work and even less for personally identifiable information (Naaman et al., 2010;Humphreys et al., 2014). As a result, the large-scale data collected from social media would only contain a limited amount of persona-related dialogues, which is "persona-sparse". The limited-scale of crowd-sourced data and the persona-sparsity in large-scale data present one common challenge: a model trained on limited personalized data cannot sufficiently understand persona consistency. As shown in Figure 1, a 12-layer GPT2 (Radford et al., 2019) finetuned on the PersonaChat dataset still shows a lack of consistency.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
After rethinking the essence of persona-based dialogue generation, we can find that it requires the dialogue agent to own the capabilities to 1) understand the persona-response consistency and 2) generate a persona-related response given the dialogue context. Obviously, an ideal dataset that satisfies both features are difficult to annotate. However, once we disentangle persona-based dialogue generation into two sub-tasks: consistency understanding and dialogue generation, it is easy to find abundant data resources for them. For consistency understanding, we may leverage large-scale nondialogue inference data, such as SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) as the training data. As for dialogue generation, we already have various large-scale persona-sparse datasets.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Inspired by the aforementioned motivation, in this work, we explore to learn a consistent personabased dialogue model from limited personalized dialogues, with the assistance of large-scale nondialogue inference data. Specifically, the proposed model consists of an encoder E, an auto-regressive decoder D 1 for response generation, and a bidirectional decoder D 2 for consistency understanding. Given personas P and dialogue query Q, the E and D 1 jointly work in an encoder-decoder manner to capture a typical query to response mapping F G (S|Q, P ), and generate a coarse response representation R 1 . Then R 1 and personas P are fed into the bidirectional decoder D 2 to map R 1 to final response representations R 2 : F U (R 2 |S, P ). Since the consistency understanding part F U (R|S, P ) is independent of the dialogue query Q, it can be learned on non-dialogue inference datasets. Here an unlikelihood training objective (Welleck et al., 2019a) is applied to make contradicted cases in the inference data less likely so that D 2 could acquire the ability of consistency understanding.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
We initialize all modules from BERT (Devlin et al., 2019) and name the proposed model BERTover-BERT (BoB). To verify the effectiveness of our model, we experiment on two limited data scenarios: 1) a persona-dense scenario (Zhang et al., 2018) with low-resource settings (Zhao et al., 2019), and 2) a persona-sparse scenario (Zheng et al., 2019). Both automatic and human evaluations indicate that our model generalizes well under different settings and outperforms strong baselines on most metrics, especially on persona consistency.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Contributions in this work are three-fold:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
• We disentangled the task of persona-based dialogue generation into two sub-tasks: consistency understanding and dialogue generation.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
• A BERT-based generative framework, BoB, was proposed for training persona-based dialogue models from limited data.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
• An unlikelihood training method with nondialogue inference data was introduced to enhance persona consistency understanding.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
In this work, our goal is to learn a persona-based dialogue model from limited personalized data. To address the challenges of consistency understanding brought by limited data, we leverage large-scale non-dialogue inference data in our model.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Formally, let Q = q 1 , q 2 , ..., q n denote the dialogue query, R = r 1 , r 2 , ..., r m denote the target response, and P denote the personas. In addition, let N denote the non-dialogue inference data, which consists of premise, hypothesis, and their label. The premise and hypothesis are both natural sentences. Note that in the following sections, we use fonts to distinguish between sentences (P, Q, R) and their vector representations (P , Q, R 1 , R 2 ).
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
The task of the proposed model M is to generate a persona consistent responseR =r 1 ,r 2 , ...,r m , based on both persona P and query Q, i.e.,R = M(Q, P). As shown in Figure 2, the proposed model M consists of three BERT-based submodules: an encoder E, a response decoder D 1 , and a consistency understanding decoder D 2 . More concretely, E encodes the embeddings of persona and query, i.e., P and Q, into hidden states H. D 1 performs cross-attention on H in a typical encoderdecoder manner, and generate a coarse representation R 1 . D 2 learns consistency understanding from non-dialogue inference data N and further converts P and R 1 into final representations R 2 . At last, a consistent responseR could be generated from R 2 .
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
For response generation, a typical persona-based dialogue model needs persona P and dialogue query Q to generate a response. For consistency understanding, a model needs persona P, response R, and the consistency labels between P and R. However, if we entangle generation and understanding, it is not easy to obtain sufficient annotated data that satisfy the format of {P, Q, R, Label}.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Instead, in our model, we design the decoder D 2 to disentangle generation and understanding, where D 2 maps R 1 , rather than Q, to R 2 . The key to "disentangling" is we can get R 1 without the participation of Q, as R 1 is the representation of R. As a result, the mapping from R 1 to R 2 could be independent of Q. In this way, it becomes possible to 1) learn persona-based dialogue generation from {P, Q, R}, i.e., the personalized data, and 2) learn consistency understanding from {P, R, Label}. Moreover, considering the limited amount of such annotated data, we could approximate {P, R, Label} by the abundant non-dialogue inference data N ={Premise, Hypothesis, Label}, where P and R corresponds to the Premise and Hypothesis.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Given data P and R, suppose D 2 understands persona consistency, it should maximize the likelihood of generating R if R is not contradicted to P. Otherwise, it should minimize the likelihood of generating R. Motivated by this observation, we choose to apply unlikelihood training on D 2 to make it understand consistency. The detailed training objectives will be provided in Sec 3.4.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
The encoder E works like a standard BERT model, which bidirectionally encodes the input embeddings to a sequence of hidden vectors, from which the downstream tasks will be performed on.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
In our model, the input consists of persona P and dialogue query Q. For persona, whether P is personal facts (e.g., "I have two dogs") or profiles (e.g., "location: Seattle"), we could always convert it into a sequence of words. A special token is placed between persona sequence and dialogue query, and the input is formated as:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Then the embedding layer will convert input into representations. Following usual practice, the input representations are the sum of the corresponding token, type, and position embeddings, where the type embedding is 0 and 1 for persona and query, respectively. P and Q can also get their independent representations. The resulted representations are P and Q, which could be jointly denoted as emb = e p 1 , e p 2 , ..., e q l , where l is the maximum length of the input.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Once we get the input representations, encoder E will perform multi-head attetnion (Vaswani et al., 2017) on the emb to transform the embeddings into a sequence of hidden vectors H. The multi-head attetnion could be denoted as MultiHead(query, key, value), where scaled dot-product attention is performed on query, key, and value. There are N identical layers in E, for each layer:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
where h 0 = emb, and FNN is a fully connected feed-forward network containing two linear transformations with a ReLU activation in between. h N is the final output of encoder E, i.e., H.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
The response generation decoder D 1 is initialized from BERT to inherit its robust language model but works in an auto-regressive decoder manner. First, a cross-attention is inserted between E and D 1 to pass the context information. Second, a leftto-right mask is applied to D 1 to preserve the autoregressive generation property.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
As the cross-attention does not exist in the BERT model, it is randomly initialized and updated during training. In the cross-attention, the query comes from the previous layer of D 1 , and the key and value come from H:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
This attention is similar to the typical encoderdecoder attention mechanism in sequence to sequence models (Bahdanau et al., 2015), which attends to all positions in the context representations H according to the variations of r 1 . In training, r 0 1 is initialized from the embeddings of the target response. At each generation step, future tokens in the target response should not be considered. Therefore, as shown in Figure 2, a left-to-right mask is applied to D 1 to ensure that the predictions can only depend on the known outputs.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
D 1 also has N identical layers. And the output of the last layer r N 1 , i.e., R 1 , is further fed to D 2 .
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Like E and D 1 , the consistency understanding decoder D 2 is also initialized from BERT, from where D 2 initializes a good semantic representation for understanding tasks.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
In each layer of D 2 , the multi-head attention is performed twice:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
The resulted r i+1 2 in each layer thus fuses information from both P and R 1 . The output of the last layer of D 2 is the final representations R 2 . With an output layer, e.g. linear layers, upon the R 2 , we can get the generated responseR.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
We employ negative log-likelihood (NLL) loss and unlikelihood loss for dialogue generation and consistency understanding. A brief illustration is shown in the last column of Figure 2 and detailed descriptions will be provided in this section.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Response Generation In our model, the widely adopted negative log-likelihood loss is applied in the training. For E and D 1 , they read the persona P and dialogue query Q to predict the target response R, which yields the raw representations R 1 :
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
log(p θ (r i |P, Q, R <i )).
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
The generation part in D 2 is also trained by NLL. D 2 reads persona embeddings P and raw representations R 1 to predict the target response R:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
log(p γ (r i |P, R 1 , R <i )).
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Unlikelihood Training Given large-scale nondialogue inference dataset, we collect positive data D + from the entailed category and collect negative data D − from the contradicted category:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
whereP andR are premise and hypothesis from the non-dialogue inference data, and their representations in our model are denoted asP andR. For data from D + , we still apply the NLL loss:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
For data from D − , we apply the unlikelihood objective to minimize the likelihood of contradictions:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
which penalizes every token in the contradicted target. Therefore, the loss L D − 2 U L makes generating contradicted responses less likely.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Training Procedure The training steps can be summarized as follows:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
1) Response Generation. Given P, Q, and R from personalized dialogue data, we calculate the response generation loss L 1 = L D 1 N LL + αL D 2 N LL ; 2) Consistency Understanding. Given D + and D − from non-dialogue inference data, we calculate the unlikelihood loss L 2 = βL
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
3) Optimization. Sum up L 1 and L 2 . Update parameters with back-propagation. We initialize our model from the publicly available BERT base model, with 12 layers and hidden size 768. We employ an Adam optimizer with a learning rate of varying from 5e-6 to 5e-5. Empirically, we set α to 5e-3 and β to 0.1. The training of the proposed model was done on an Nvidia Telsa V100 32G GPU. Other details please refer to the released projects.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
To evaluate the performance of the proposed model, we carried out persona-based dialogue generation experiments in a persona-dense scenario and a persona-sparse scenario with two publicly available datasets:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
• PersonaChat (Zhang et al., 2018) is a crowdsourced dataset covering rich persona features. The dialogues in this dataset are grounded on specific personal facts. Here we use the Con-vAI2 PersonaChat (Dinan et al., 2019), so the results are comparable to existing methods.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
• PersonalDialog (Zheng et al., 2019) is a large-scale persona-sparse dataset, which is collected from Chinese social media Weibo. This dataset provides persona profiles and dialogues, but the majority of the dialogues are not persona-related. Two testsets are provided: a random testset, which is identically distributed as the training data, and a biased testset, which is manually selected to cover persona-related features.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
We summarize the key statistics of two personalized dialogue datasets in Tabel 1. As aforementioned, we leverage non-dialogue inference data to address the consistency understanding issue brought by limited personalized data. Here we use the non-dialogue inference dataset MNLI (Williams et al., 2018) and its Chinese version CMNLI  as our auxiliary data. Moreover, to better compare models' performance on persona consistency, we leverage two dialogue inference datasets, DNLI (Welleck et al., 2019b) and KvPI (Song et al., 2020b), for evaluations. The statistics 1 of these inference datasets are summarized in Table2.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
The following models, including both nonpretrained and pretrained ones, have been compared in the experiments.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Baselines. Vanilla Transformer (Vaswani et al., 2017) is employed as baselines for the experiments on both PersonaChat and PersonalDialog. Personas are concatenated to the dialogue queries.  Non-Pretrained Models. Meta-learning has recently been explored in addressing the limited personalized data issue. CMAML (Song et al., 2020c) is a meta-learning based method that learns from few shot personas by customizing the model structures. Besides the meta-learning methods, GDR (Song et al., 2020a) introduces inference ability on the PersonaChat with a generate-refine framework. However, the two models are elaborately designed for the persona-dense dataset and not appliable for the persona-sparse scenario. Thus we only employ them for experiments on PersonaChat.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Pre-training Models. In the ConvAI2 challenge (Dinan et al., 2019), which utilizes Per-sonaChat as the competition dataset, LIC (Golovanov et al., 2019) is the best performing model. Thus we compare this model in the experiments on both PersonaChat and PersonalDialog. Atten-tionRouting (Zheng et al., 2020) is a pre-training method specially designed for the persona-sparse dataset, and it is also the latest model on Personal-Dialog. We also finetune a GPT2 (Radford et al., 2019) for a thorough comparison on PersonaChat.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
We focus on two main aspects of the persona-based dialogues: response quality and persona consistency. To compare different models, we employ both automatic metrics and human evaluations.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Automatic Metrics For dialogue quality, we employ perplexity (PPL.) and distinct 1/2 (Dist.1/2) following common practice (Zhang et al., 2018;Zheng et al., 2020). Lower perplexity means better language modeling. Distinct 1/2 (Li et al., 2016a) are the ratio of distinct uni-grams / bi-grams, and higher distinct means better reponse diversity.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
For persona consistency, we employ two metrics. The first is Consistency Score (C.Score) (Madotto et al., 2019), which leverages a referee model to predict consistency and can be defined as:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
NLI(r, p i ).
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
(11) Here the NLI is a pre-trained RoBERTa model  finetuned with the dialogue inference datasets, i.e., DNLI and KvPI, as descriped in Table 2. The RoBERT model achieves testset accuracy of 89.3% and 88.9% on DNLI and KvPI, which is aligned to the reported 88.20% (Welleck et al., 2019b) and 88.0% (Song et al., 2020b).
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
The second metric is Delta Perplexity (∆P), which evaluates consistency from model's internal distributions. Li et al. (2020) first calculates the perplexity of entailed (p.Ent) and contradicted (p.Ctd) dialogues in the inference dataset. A dialogue model with good understanding ability should assign lower perplexity to the entailed dialogues while higher perplexity to the contradictions. From this intuition, the ∆P can be defined as:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
where a larger ∆P means the model has a better ability to distinguish entailment from contradiction. In our experiments, we get entailed and contradicted {persona, query, response} tuples from the dialogue inference datasets DNLI and KvPI.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Human Evaluations We recruit two teams (one for English and another for Chinese), each consists of five professional annotators, from a third-party company. These annotators are proficient in language tasks but know nothing about the models. We sample 100 {persona, query, response} tuples for each model's evaluation under every setting.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Human annotators are asked to evaluate dialogue quality from three conventional criteria: fluency (Flue.), informativeness (Info.), and relevance (Relv.). Each criterion is rated on a fivescale, where 1, 3, and 5 indicate unacceptable, moderate, and perfect performance, respectively. The annotators are also instructed to label the consistency (Per.C.) between persona and response, where 1 means persona-related and consistent, 0 means irrelevant, and -1 means contradicted.   
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Full PersonaChat We first report the full Per-sonaChat experimental results in Table 3. Our method achieves better performance consistently across all automatic and human evaluation metrics, which shows the effectiveness of our model. Among all the metrics, our model obtains significant improvements on PPL and ∆P. The lowest testset PPL means our model has learned a good language model fitting this dataset. Moreover, the highest ∆P shows that our model could more effectively distinguish entailment from contradiction than other baselines, which indicates our model has a better understanding of persona consistency.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Less Personalized Data Now that our model achieves better performance with a large margin on the full PersonaChat dataset, we want to test our model by simulating a low-resource scenario (Zhao et al., 2019), where we gradually reduce the number of examples by halving the training set. We report the low-resource settings' results in Table 4.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
As we can see, our model can outperform most of the baselines' best results even by using only 1/8 of the training data. The performance gains largely benefit from the powerful language model of the backbone BERT model. Furthermore, due to the disentangling of generation and understanding, our model presents a stable performance on ∆P regardless of the size of the training set. This is in line with our expectations because the proposed model learns consistency understanding from the non-dialogue inference data rather than the personadense dialogue data. We observe that the method also improves fluency and informativeness. It is mainly due to the introduction of the non-dialogue inference data in the training procedure, which potentially enriches the dialogue language model.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
We further validate our model on a persona-sparse scenario. To have a more intuitive understanding of "sparsity", we recruit the same annotation team to annotate whether the dataset response is personarelated in the sampled random and biased test data. Results show that only 1% responses are personarelated in the random test data and 28% in the biased test data. We calculate the Fleiss' Kappa among the five annotators and obtain a kappa of 0.774, which means substantial agreement (Landis and Koch, 1977). We report the evaluation results on both random and biased testsets in Table 5.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
On the random test set, experimental results demonstrate that our model has some advantages over other methods, but no method can consistently outperform the others. One possible reason is that the task has degenerated into the ordinary dialogue generation in the random test set, so our model's advantages can not be effectively leveraged. In contrast, on the biased test set, our model achieves the best performance on most metrics. The good performance on the metrics C.Score and Per.C. indicates that our model can be effectively trained from a dataset with limited personalized dialogues.  
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
In addition to the good performance of the BoB model, we are also curious about Q1: what is the key to the BoB model's understanding ability? Q2: can the pre-trained models understand persona consistency just through finetuning on the personalized dialogues? And Q3: does the extremely low PPL come from the initialization of the BERT model or the architecture of the proposed BoB model? To better answer the above questions, we ablate the BoB model in the following three ways: 1) w/o UL. It removes the unlikelihood objective. 2) E+D 1 . It removes the unlikelihood objective and the second decoder D 2 . 3) E. It removes the unlikelihood objective and both decoders and thus degenerates into a vanilla BERT model. We report the ablation results on PersonalDialog in Table 5 and full PersonaChat in Table 6. From these results:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Answer to Q1:
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
The key to our model's understanding is the unlikelihood training. In training, our model assigns large perplexity to the contradictions. In generating, the non-contradicted responses are more likely to be generated as they are with much smaller losses. Table 7 shows an example. And as presented in the results, after removing the unlikelihood objective, all ablated models suffer from significant performance degradations in consistency-related metrics, such as Per.C. and ∆P.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Persona I've a son who is in junior high school Query You have any children?
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
GPT2 No kids. I work at home depot so I'm busy.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Ours Yes, I have a son in the 8th grade. Answer to Q2: Pretrained models barely understand consistency from personalized dialogues. According to the poor performances on ∆P, the three BERT-based ablated models can hardly distinguish contradiction from entailment. Although their Per.C. metric still looks good, it may come from just mimicking and copying words rather than understanding. A similar phenomenon also occurs to the pre-trained GPT2, as shown in Table 3. It is also this phenomenon that motivates us to introduce the unlikelihood training into the BoB model.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
Answer to Q3: D 2 in the BoB architecture contributes most to the PPL. As shown in both datasets' ablation results, the PPL decreases the most after removing D 2 . We can also see an apparent gap between the models with D 2 and the vanilla BERT on PPL. Nevertheless, the BERT model still offers a good initialization for the BoB model to achieve the best performance on different metrics.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
The implementation for the BoB model is released at https://github.com/songhaoyu/BoB.
BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data
In this work, we propose a novel BERT-based dialogue model to learn from limited personalized data by disentangling response generation and consistency understanding. Unlikelihood training with non-dialogue inference data is introduced to en-hance the model's understanding ability. Experiments on two publicly available datasets demonstrate that our model can be trained with limited personalized dialogue data while still obtain significant improvements over strong methods.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
Generating human-level conversations by machine has been a long-term goal of AI since the Turing Test [Turing, 1950]. However, as argued by [Vinyals and Le, 2015], the current conversational systems are still unable to deliver realistic conversations to pass the Test. Amongst the many limitations, the lack of a coherent personality is one of the most challenging difficulties. Though personality is a well-defined concept in psychology [Norman, 1963;Gosling et al., 2003], while in this paper, the personality of a chatbot refers to the character that the bot plays or performs during conversational interactions. In this scenario, personality settings include age, gender, language, speaking style [Walker et al., 1997], level of knowledge, areas of expertise, and other explicit and implicit cues that may portray character [Shum et al., 2018] 
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
A chatbot needs to present a coherent personality to gain confidence and trust from the user [Yu et al., 2016]. Person- ality can make the chatbot easier to communicate with, more predictable and trustable, and therefore helps to establish an emotional connection with the user [Shum et al., 2018]. Thus, generating responses that reflect a coherent personality is important for the chatbot [Güzeldere and Franchi, 1995].
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
However, existing generation models are unable to demonstrate coherent personality, as exemplified in Figure 1. Recent works proposed in  and [Al-Rfou et al., 2016] can handle implicit personality using user embeddings in response generation (projecting each user into a vector). However, such models can not assign an explicit profile to generate coherent responses. Moreover, these models need to be trained on dialogue data from many different users, which is expensive and has a sparsity issue: some users have very few dialogue data.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
In this paper we define personality as a set of profile keys and values 2 and propose a model consisting of three key modules: a profile detector which detects whether a profile key and which key should be addressed, a bidirectional decoder that generates a response backward and forward from a selected profile value, and a position detector which predicts a proper word position at which a profile value can be replaced during the training of the decoder. The contributions of this paper are two-fold: First, we address a novel problem of endowing a chatbot with an explicit personality or profile, which allows system developers to control the profile of a chatting machine specifically. Second, instead of just learning subtle, implicit personality from speaker-tagged dialogue data, our model deals with specific, explicit personality from generic dialogue data. Our model does not depend on speaker-tagged dialogue data which are more expensive and sparse.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
The task can be formally defined as follows: given a post X = x 1 x 2 · · · x n , and an explicit profile defined as a set of key-value pairs {< k i , v i > |i = 1, 2, · · · , K}, the task aims to generate a response Y = y 1 y 2 · · · y m that is coherent to the profile. We consider several common attributes in personality including name, gender, age, weight, location, and constellation. The generation process can be briefly stated as below:
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
where P (z|X) is the probability of using the profile given post X, which is computed by the Profile Detector; P f r (Y |X) = m t=1 P f r (y t |Y <t , X) is given by a general forward decoder, the same as [Sutskever et al., 2014], and P bi (Y |X, {< k i , v i >}) is given by a Bidirectional Decoder.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
Note that post/response pair < X, Y > is collected from social media, and the profile value may not occur in the response Y at all. This leads to the discrepancy between training and test, as described in the Position Detector section. Our model works as follows (see Figure 1): given a post, the profile detector will predict whether the profile should be used. If not, a general seq2seq decoder will be used to generate the response; otherwise, the profile detector will further select an appropriate profile key and its value. Starting from the selected profile value, a response will be generated forward and backward by the bidirectional decoder. To train the bidirectional decoder on generic dialogue data (see Figure 2), the position detector predicts a word position from which decoding should start given the selected profile value. Note that the position detector will not be used during test.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
The encoder aims to encode a post to a vector representation. Given a post X = x 1 x 2 · · · x n , the hidden states of the post Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence  h 1 , h 2 , · · ·, h n are obtained by a gated recurrent unit (GRU) [Chung et al., 2014], as follows:
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
where x t is the embedding of the t-th word x t .
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
The profile detector has two roles: first to detect whether the post should be responded with the profile, and second to select a specific < key, value > to be addressed in the response. The first role of the profile detector is defined by the probability P (z|X) (z ∈ {0, 1}) where z = 1 means the profile should be used. For instance, if the post is "how are you today", P (z = 1|X) ≈ 0, while if the post is "how old are you", P (z = 1|X) ≈ 1. P (z|X) is a binary classifier trained on labeled data. More formally, the probability is computed as follows:
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
where W p is the parameter of the classifier and h = j h j , simply the sum of all hidden states, but other elaborated methods such as attention-based models are also applicable.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
The second role of the profile detector is to decide which profile value should be addressed in a generated response. This is implemented as follows:
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
where W is the weight and k i /v i is the embedding of a profile key/value respectively. h = j h j is the representation of the post. β is a probability distribution over profile keys.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
The optimal profile value is selected with the maximal probability: v = v j where j = argmax i (β i ). As long as a profile value v is obtained, the decoding process will be determined by the bidirectional decoder, as follows:
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
This decoder aims to generate a response in which a profile value will be mentioned. Inspired by [Mou et al., 2016], we design a bidirectional decoder which consists of a backward decoder and a forward decoder, but with a key difference that a position detector is employed to predict a start decoding position.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
where v is a selected profile value. The bidirectional decoder will generate Y b in a backward direction and Y f forward. The backward decoder (P b ) generates Y b from the given profile value v to the start of the response. The forward decoder (P f ) 3 generates Y f from v to the end of the response, but takes as input the already generated first half, Y b . The process is defined formally as 3 Note that this decoder is different from P f r (yt|Y<t, X). follows:
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
In order to encode more contexts in the forward decoder, the first half of a generated response (Y b ), along with the profile value ( v), serves as initial input to the forward decoder. The probability P b and P f is calculated via
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
is the state of the corresponding decoder, c ( * ) j is the context vector, and * ∈ {b, f } where b indicates the backward decoder and f the forward decoder. The vectors are updated as follows:
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
can be viewed as the similarity between decoder state s ( * ) j+l and encoder hidden state h t , l = 1 when * = b (backward), and l = −1 when * = f (forward). And these MLPs have the same form as Eq. 4, but with different parameters. The position detector is designed to provide more supervision to the bidirectional decoder. It is designed to provide a Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence  start decoding position to the decoder during training. For instance, given a post X ="你(you)-1 有(have)-2 什么(what)-3 特长(speciality)-4 ？-5 (what's your speciality?) 4 " and a response Y ="我(I)-1 非常(very)-2 擅长(good at)-3 小提 琴(violin)-4(I am good at playing violin)", and a profile key value pair "<特长, 钢琴> (< hobby, piano >)", the position detector will predict that "小提琴(violin)-4" in the response can be replaced by the profile value "钢琴(piano)" to ensure grammaticality. The predicted position "小提琴(violin)-4" is then passed to the decoder (see Eq. 6) to signal the start decoding position.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
As mentioned, the bidirectional decoder starts from a profile value to generate the entire sequence at the test stage. However, in our training dataset, the profile values may be rarely mentioned in the responses. For instance, given the profile key value pair <爱好, 冰球> (< hobby, hockey >), the value 冰球(hockey) rarely occurs in the training corpus. In other words, even though we have a training instance (X, Y, < k, v >), the value (v) may not occur in Y at all. Hence, the bidirectional decoder is not aware from which word decoding should start. This leads to the discrepancy between training and test: during training, the decoder is unaware of the start decoding position but during test, the start decoding word is specified.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
This issue makes our work differ substantially from previous approaches where supervision is directly observable either between post and response [Gu et al., 2016] or between response and knowledge base [Yin et al., 2015]. Experiments also show that the position detector contributes much to the performance improvement than a random position picking strategy [Mou et al., 2016].
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
In order to find an appropriate position at which the profile value can be replaced, we need to estimate the probability: P (j|y 1 y 2 · · · y m , < k, v >)), 1 ≤ j ≤ m which indicates how likely the word y j can be replaced by the profile value v. We apply a simple technique to approximate the probability: a word can be replaced by a given profile value if the word has the maximal similarity.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
where cos(y j , v) denotes the cosine similarity between a word in a response and a profile value. More elaborated techniques, for instance, language models, will be studied as future work.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
Two loss functions are defined: one on the generation probability and the other on the profile detector. The first loss is defined as below: 4 The number indicates the position of each word.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
The first term is the negative log likelihood of observing D (c) and the second term for D (pr) . v is a word in Y whose position is predicted by the position detector during training. D (pr) consists of pairs where a post is related to a profile key and its response gives a meaningful reaction to the post, and D (c) has only general post-response pairs. The two decoders (P f r and P bi ) have no shared parameters. Since the number of instances in D (c) is much larger than that of D (pr) , we apply a two-stage training strategy: D (c) will be used to train P bi at the early stage for several epoches, where v is a randomly chosen word in a response, and then D (pr) for further training at the later stage.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
The above formulation generally adopts the hard form of P (z|X) (see Eq. 3): P (z = 1|X) = 1 for profile-related pairs and P (z = 1|X) = 0 for others. In order to better supervise the learning of the profile detector, we define the second loss and add it to the first one with a weight α as the overall loss (i.e., L = L 1 + αL 2 ):
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
where the first term is for binary prediction of using profile or not, and the second for profile key selection. k is the profile key whose value should be addressed, K is the total number of keys, β is the predicted distribution over profile keys as defined by Eq. 4, and β is one-hot representation of the gold distribution over keys. < X, Y, z > is obtained by manual annotation while (X, Y, k) is obtained by matching the keywords and synonyms in the profile with the post, which is noisy. This works well in practice and reduces manual labors largely.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
We prepared several datasets 5 : Weibo Dataset (WD) -D: We collected 9,697,651 postresponse pairs from Weibo. The dataset is used for training P f r (Y |X) and P bi (Y |X, v) at the early stage and 10,000 pairs are used for validation to make early stop. Profile Binary Subset (PB -D (pb) ∈ D): We extracted 76,930 pairs from WD for 6 profile keys ({name, gender, age, location, weight, constellation}) with about 200 regular expression patterns. The dataset is annotated by 13 annotators. Each pair is manually labeled to positive if a post is asking for a profile value and the response is a logic reaction to the post, or negative otherwise.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
This dataset is used to train the binary classifier (P (z|X)) (see D (pb) in Eq. 11). 3,000 pairs are used for test and the remainder for training. Profile Related Subset (PR -D (pr) ∈ D (pb) ): This dataset only contains pairs whose posts are positive in PB. In total, we have 42,193 such pairs. This dataset is used to train the bidirectional decoder. Manual Dataset (MD): This dataset has 600 posts written by 4 human curators, including 50 negative and 50 positive posts for each key. A positive post for a profile key (e.g., how old are you?) means that it should be responded by a profile value, while a negative post (e.g., how are you today?) should not. This dataset is used to test the performance on real conversation data rather than social media data.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
In our experiments 6 , the encoder and decoders are all have 4 layers of GRUs with a 512-dimensional hidden state. The dimension of word embedding is set to 100. The vocabulary size is limited to 40, 000. The word embeddings are pre-trained on an unlabeled corpus (about 60, 000, 000 Weibo posts) with word2vec. And the other parameters are initialized by sampling from a uniform distribution U (−sqrt(3/n), sqrt(3/n)), where n is the dimension of parameters. Training is conducted by stochastic gradient descent (SGD) with a mini-batch of 128 pairs. The learning rate is initialized with 0.5 and the decay factor is 0.99.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
We evaluated our model at both post and session level. At the post level, three metrics (naturalness, logic, and correctness) are defined to evaluate the response generated by each model. At the session level, the models are evaluated from the aspects of consistency and variety to justify the performance in the real conversational setting. We named our model Personality-Coherent Conversation Machine (PCCM) and compared it with several baselines: Seq2Seq: a general sequence to sequence model [Sutskever et al., 2014]. Seq2Seq + Profile Value (+PV): if the profile detector decides that a profile value should be used (P (z|X) > 0.5), the response is simply the value of the key decided by the profile detector (see Eq. 4); otherwise, a general seq2seq decoder will be used. Seq2Seq + Profile Value Decoding (+PVD): the response is generated by a general seq2seq decoder which starts decoding forwardly from the value of the selected key. PCCM-Pos: Instead of using a predicted position obtained by the position detector to start the decoding process, the bidirectional decoder in this setting randomly picks a word in a response during training, the same as [Mou et al., 2016].
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
To conduct post-level evaluation, we used 600 posts from MD, 50 positive/negative posts respectively for each key. Each post is input to all the models to get the corresponding responses. Thus, each post has 5 responses and these responses are randomly shuffled and then presented to two curators. Post-response pairs are annotated according to the following metrics, based on a 1/0 scoring schema: Naturalness (Nat.) measures the fluency and grammaticality of a response. Too short responses will be judged as lack of naturalness. Logic measures whether the response is a logical reaction to a post. For instance, for post "how old are you", a logical response could be "I am 3 years old" or "I do not know". Correctness (Cor.) measures whether the response provides a correct answer to a post given the profile. For instance, for post "how old are you", if the profile has a key value pair like < age, 3 >, responses like "I am 18" will be judged as wrong.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
Each response is judged by two curators. The Cohen's Kappa statistics are 0.46, 0.75 and 0.82 for naturalness, logic, and correctness respectively. Naturalness has a rather lower Kapp because it is more difficult to judge.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
Nat    Table 2 support the following statements: First, our model is better than all other baselines in all metrics, indicating that our model can generate more natural, logical, and correct responses; Second, in comparison to simply responding with a profile value (Seq2Seq+PV) where the responses are generally too short, our model can generate more natural responses; Third, the position detection contributes to better generation, in comparison to a random position (PCCM vs. PCCM-Pos).
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
In order to compare these models in real conversation sessions, we randomly generated sessions based on MD. For each profile key, we randomly chose 3 positive posts 7 from MD, generated responses to the 3 posts for each model, and obtained a session of 3 post-response pairs. In this way, 240 sessions are generated, and each key has 40 sessions. The sessions are manually checked with the following metrics: Consistency measures whether there is a response contradictory to the given profile. Score 1 indicates that all the three responses are consistent to the profile, and score 0 otherwise.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
Variety measures the language variety of the three responses in a session. Score 1 indicates that the linguistic patterns and wordings are different between any two of them, and score 0 otherwise.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
Results are shown in Table 4 and we presented some session examples in Table 3. We can clearly see the following observations: 1) Our model is remarkably better than all the baselines w.r.t both metrics. Results of our model against Seq2Seq+PVD indicate that the bidirectional decoder can generate responses of much richer language variety. The results of PCCM-Pos show that the position detector improves consistency and variety remarkably.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
2) if simply responding with a profile value (Seq2Seq+PV), the model can obtain good consistency but very bad language variety, which is in line with the intuition.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
3) The general Seq2Seq model is too weak to generate consistent or linguistically various responses.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
Consistency  
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
We also presented results of automatic evaluation for the profile detector and position detector.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
The profile detector is evaluated from two aspects: whether a profile should be used or not (P (z = 1|X)), and whether a profile key is correctly chosen. Note that the prediction of profile key selection is cascaded on that of P (z = 1|X).
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
Dataset (# samples) Binary profile Key selection PB (3000) 85.1% 74.8% MD (600) 82.0% 70.5%  Table 6: Accuracy for predicting the start decoding position.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
As mentioned previously, the position detector plays a key role in improving the naturalness, logic, and correctness of responses (see PCCM vs. PCCM-pos in Table 2), and the consistency and variety of conversational sessions (see Table  4). Thus, it is necessary to evaluate the performance of this module separately. We randomly sampled 200 post-response pairs from PR for each key (1200 pairs in total), and then manually annotated the optimal position from which decoding should start. The results are shown in Table 6. The position for most keys can be estimated accurately while for name the prediction is bad. This is because the value of the key rarely occurs in our corpus, and the embeddings of such values are not fully trained. Nevertheless, the results are better than a random word picking strategy (see PCCM vs. PCCM-Pos in Table 2).
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
The effectiveness of our model is verified on six profile keys, but manual labors are required. We will show the extensibility of the model by evaluating it on four additional keys: hobby, idol, speciality, and employer.  Firstly, for the 4 keys, we extracted 16,332 post-response pairs from WD with 79 hand-crafted patterns and each pair is noisily mapped to one of the keys with these patterns. These new pairs, along with the old pairs on the six pairs, are used to retrain the model. Secondly, we constructed a test dataset consisting of 400 posts, 50 positive and 50 negative humanwritten posts for each key. Responses from our model and Seq2Seq are obtained and then evaluated. The manual labor exists only in hand-crafting the 79 patterns.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
Results show that our model has a relative 10% drop on the new keys with respect to logic and correctness, and remains unchanged in naturalness. Nevertheless, our model is still much better than the Seq2Seq model. The baseline has no drop in naturalness and logic because this model does not deal with any pre-specified profile.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
We present a model that can generate responses that are coherent to a pre-specified, explicit personality or profile. Instead of learning implicit personality from speaker-tagged dialogue data, our work allows system developers to control chatbots' profile explicitly using generic dialogue data. Extensive results show that our model is effective to deliver more coherent and diversified conversations.
Assigning Personality/Proﬁle to a Chatting Machine for Coherent Conversation Generation
Our work moves toward endowing a chatbot with controllable personality. As a preliminary attempt, we only experimented with a few attributes of personality. Obviously, there are much more to be explored in this direction such as speaking styles, linguistic cues for extrovert or introvert, and many other subtle traits. More interestingly, this task raises more challenging problems such as semantic reasoning: if you ask a ten-year-old boy chatbot with "are you married?" or "do you play women football?", it requires to make reasoning with commonsense knowledge.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Faced with extensive information available on the Internet, it is very appealing to have an intelligent assistant that can provide the most relevant information [16,46,47], collaborate with us on dayto-day problems [14,45], or even act as our agent for some specific tasks [37]. Towards this ultimate goal, in the dialogue system area, building digital agents has attracted more and more attention and had some preliminary applications in our daily life [14,24,45]. In the future, some chit-chat between humans will inevitably be completed by digital agents. In this paper, we carry out a preliminary study toward this goal and focus on the problem of developing personalized chatbots. A personalized chatbot aims at leveraging personalized information (e.g., a predefined persona [13,41]) to provide personalized responses when communicating with others. Such personalized information can help chatbots generate more consistent and informative replies. More importantly, if the personalized information can be well captured, the chatbots can perform similar behaviors like real users (e.g., serving as an agent of the user and give similar responses to others when the user is busy, as shown in Figure 1), thus having the potential to be an intelligent agent with that specific personality [37].
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Many models have been proposed for improving the chatbot's capability to generate personalized responses. Early studies tried to integrate the user ID embeddings to a sequence-to-sequence (Seq2Seq) model for identifying the user and generating user-related responses [3,4,9]. Recently, some studies proposed to assign predefined personas to chatbots so as to generate more personalized responses [23,31,41]. They assumed that personality can be described in several sentences or attributes and help chatbots generate more persona-related responses. Different from existing studies, in this work, we propose letting the chatbot learn the implicit user profile automatically from the user dialogue history and generate personalized responses based on the learned user profile. In this way, the chatbot can be personalized by user's (e.g., user A) historical data, behave like this user, act as this user's agent, and chat with any other users (e.g., user B, C, etc. If you enjoy programming, you can consider the latest MacBook.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Busy Figure 1: An example of the personalized chatbot serving as an intelligent agent for user Tom when Tom is busy.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Our idea is motivated by: (1) Contrary to explicit persona descriptions, user dialogue history is easier to be gathered on the user's client devices. It is evident that obtaining explicit descriptions for massive users is impractical in real applications [4]: First, users may be lazy to set their profile before using the chatbot [6]. Second, manually collecting user profiles is costly and time-consuming. Furthermore, even if the user profile is collected, it cannot be updated with the change of user interests, thus may be ineffective over time. Finally, a fixed set of properties is not suitable for describing all users. (2) The user dialogue history contains massive personalized information, which is suitable for learning user profiles automatically. As shown in Figure 1, the dialogue history of a user includes their historical responses, and the corresponding posts issued by other users. Intuitively, users' historical responses can often reflect their language style, background knowledge, frequently used words, and even their interests. For example, there may be many electronic device names that appeared in the historical responses of an electronic hobbyist (e.g., MacBook in Figure 1). Besides, the interaction content and style between a specific user and others can be captured from the historical post-response pairs. When faced with a new input post, the chatbot can look for the historical data, check how the user has responded to a similar post before, and apply similar interactions to generate a suitable response. In addition, user profiles learned from historical data can be gradually updated with more data being collected. In summary, the user dialogue history is easy to obtain and appropriate for building user profile.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
To achieve our idea, we propose a model DHAP for personalized chatbots, which focuses on learning implicit user profile from user Dialogue History Automatically and generating Personalized responses. In our model, a general user profile representation is firstly constructed from the user's historical responses to capture the general information, including user interest, background knowledge, and speaking style. This is implemented by a personalized language model based on Transformer. Then, we design a personalized post encoder to construct the personalized post representation. The general user profile is utilized in the post encoder to better capture the semantic information of the input post. Next, we build a key-value memory network to store the user's historical post-response pairs. Based on this history memory, the dynamic post-aware user profile is built by highlighting the historical responses relevant to the current input post. Finally, we design a personalized decoder to fuse the learned user profile into the response generation process. The personalized decoder can switch between generating a word from a generic vocabulary and copying a word from the user's personalized vocabulary. Experimental results on two large-scale datasets show that our proposed DHAP significantly outperforms existing response generation models in various evaluation metrics.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Our contributions are three-fold: (1) We learn the implicit user profile from user's dialogue history automatically for generating personalized responses. By this means, our method can be applied without additional annotations on user profiles, and create personalized chatbots as user's digital agents. (2) We build two kinds of user profiles from the dialogue history, including the general user profile reflecting the user's general information and the dynamic post-aware user profile to apply similar interactions in the historical data for the current input. (3) We design a personalized decoder to coordinate two personalized decoding strategies, including generating a word from the generic vocabulary and copying a word from the personalized vocabulary to leverage the user's word preference.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Open-domain Chatbots and Response Generation. Open-domain chatbots have attracted more and more attention, due to their broad application in real applications, such as Microsoft XiaoIce [45]. Typical methods can be categorized into two groups: retrieval-based and generation-based. Retrieval-based methods aim to select a suitable response from a large repository [38,49,50], while generationbased methods aim at generating a response from scratch [26,28,32,48]. In this study, we focus on the response generation problem.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Some early studies treat the response generation task as a statistical machine translation problem because of its end-to-end and data-driven features [25,34]. More recently, with the progress of deep learning, Seq2Seq methods have been applied to response generation and achieve great performance [8,26,27,34]. Many Seq2Seq-based extensions have been applied to tackle the "safe response" problem [8]; to incorporate external knowledge [44]; to generate responses with emotions or personas [9,23,43]; and to model the hierarchical structure of the dialogue context [26,27].
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Personalized Chatbots. Endowing chatbots with a coherent personality is a challenging but necessary step to achieve the ultimate goal of building an intelligent assistant. With personality, a chatbot can generate more informative and user-specific responses [9,41], and has the potential to perform similar behaviors as real humans.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Traditional personalized chatbots focus on modeling the user's psychological behavior such as the "Big Five" of speakers [17]. In recent years, deep-learning based methods were proposed to learn the persona information directly from large-scale dialogue datasets via end-to-end neural networks [4,9,23,41]. Some researchers first tried to input the user ID embeddings into the decoder of a Seq2Seq model to generate more personalized responses [1,3,4,9]. Despite users can be identified by their IDs, the personalized performance is limited because no user-related information is used in the model. Therefore, another group of researchers proposed assigning explicit profiles (personas) for chatbots to generate personalized responses. For example, Zhang et al. [41] published the PERSONA-CHAT dataset, in which each user is assigned with several persona description sentences and the conversations between users are collected. With explicit profile descriptions, chatbots can learn to generate more consistent and informative dialogues. On this dataset, many methods have achieved encouraging performance, such as variational autoencoders [30], pre-trained language models [29,36], and multi-task modeling [29,31,35]. These explicit persona-based methods enjoy the high quality of predefined personas, but the acquisition of these persona data is expensive and even impossible when applied to real-world systems [4].
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
In this work, we propose using the implicit user profile to drive the response generation. Such user profile is implicitly learned with neural models from user dialogue history rather than predefined explicitly. Due to the fact that the user dialogue history contains abundant user-specific information and is easily accessible in realworld applications, our method is more applicable in practice.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
In this section, we first provide an overview of our proposed model DHAP. The details of each component are provided later, and the model training is introduced finally.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Suppose that for a user , we have their dialogue history including a series of responses issued by and the corresponding posts:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
= (( 1 , 1 ), · · · , ( , )), where is the number of historical post-response pairs. Note that the posts ( 1 , · · · , ) here can be issued by different users, but the responses ( 1 , · · · , ) are all issued by the same user . We call them historical posts and historical responses respectively in the following sections. Under the singleturn setting, given an input post = ( 1 , · · · , ) and the user dialogue history , with sequence-to-sequence modeling, our task is to generate a personalized response = ( 1 , · · · , ) as:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
where denotes the word generated at the -th step, and < denotes the previous generated words ( 1 , · · · , −1 ). It is worth noting that the dialogue history here includes dialogues between the user and several other users, thus the history is not a multiturn dialogue between two fixed interlocutors.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
To compute the probability ( | , ), we design a model called DHAP, which stands for learning user profiles from Dialogue History Automatically for Personalized chatbots. The structure of DHAP is shown in Figure 2. We briefly introduce the key components of DHAP as follows. The number of the modules corresponds to the mark in the figure. In general, DHAP considers personalized information in both the encoder and the decoder side.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
3.1.1 Encoder. DHAP has two different encoders, which encode the input post and user's historical responses, respectively:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Part (1): History encoder and general user profile. Since abundant personalized information (e.g., background knowledge and speaking style) is often hidden in user's historical responses, DHAP firstly establishes a Transformer-based personalized language model to encode historical responses ( 1 , · · · , ), then summarizes the general user profile e based on the historical responses. As e does not depend on the input , we call it general user profile. "General" does not mean it is general among all users or cannot be updated together with time. In contrast, every user has their own profile, which can be updated once they issue a new response.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Part (2): Personalized post encoder. DHAP also has an encoder for the input post, which is implemented by a bidirectional GRU (BiGRU). To make the post encoder aware of the user's personalized information, we use the general user profile e for initialization. Consequently, the post = ( 1 , · · · , ) is represented by hidden state sequence h = (h 1 , · · · , h ). These representations will be dynamically aggregated as a personalized post representation c by an attention mechanism at the decoding step .
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
In the decoder side, DHAP incorporates the personalized information in two perspectives:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Part (3): User history memory and dynamic post-aware user profile. In the general user profile, all historical responses are considered in a general view. However, for a specific input post , the historical responses may play different roles. Intuitively, the historical responses relevant to are more valuable than the irrelevant ones. To highlight the historical responses relevant to the input post, DHAP builds a key-value memory of historical postresponse pairs (( 1 , 1 ), · · · , ( , )). Given c as a personalized representation of , DHAP builds a dynamic post-aware user profile e by selecting and aggregating the historical responses from the user history memory. As e is dynamic with different input posts and summarizes related user history information, we call it dynamic post-aware user profile.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Part (4): Personalized decoder. Finally, the personalized post representation c , general user profile e , and dynamic post-aware user profile e are fused together to decode the response sequentially. Inspired by CopyNet [7], DHAP can switch between generating a word from a generic vocabulary ( ( | )) and copying a word from a user's personalized vocabulary ( ( | )) as:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
where ( ) and ( ) are computed by our designed decoding switcher. ( | ) and ( | ) are calculated based on same inputs(e.g., c , e and e ) with different functions.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
In the remaining part of this section, we will introduce these four components of DHAP in detail.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Based on our observation, there is a large amount of personalized information in the user's historical responses. For example, a fan of cricket may talk a lot about cricket topics with others. Furthermore, different users can hold different speaking styles, such as enjoying speaking slang. Therefore, our first idea is to devise a model for learning such personalized language information in a general view.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Inspired by the strong ability of Transformer [33] to aggregate context and model sequences, we use a Transformer encoder to learn contextual representations of historical responses. In particular, we first add special tokens and concatenates all historical responses as = [⟨CLS⟩; 1 ; ⟨SEP⟩; · · · ; ; ⟨SEP⟩], where each response = ( 1 , · · · , ) contains words, and [;] is the concatenation operation. A "⟨SEP⟩" token is added at the tail of each response for segmentation, while a "⟨CLS⟩" token is added at the sequence head for summary. Then we map all words and special tokens into embeddings, and learn their contextual representations:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
where e ,fuse is the sum of the word embedding, segment embedding, and position embedding of the -th word. These three embeddings are used in a similar way like BERT [5]. The representation of "⟨CLS⟩" token (e ) summarizes the information in the entire historical responses, thus we call it the general user profile. E contains contextual representations of words in historical responses. For , we denote its contextual representations as (e 1 , · · · , e ).
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Transformer (·) is a -layer bidirectional Transformer encoder identical to the original implementation described in [33].
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
With the history encoder, we obtain: (1) a general user profile e , which summarizes all historical responses and contains personalized information of the user; and (2) the contextual representations of words in the historical responses E . These representations will be used to build the dynamic post-aware user profile in Section 3.4.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
In real-world applications, the posts are often very short, even ambiguous. Thus, building accurate encoding of the input post is difficult, which further leads to poor quality of the generated responses [9,28]. Fortunately, with personalized background knowledge, the chatbot can get more input information and is promising to better capture the semantic information of the post. Let us use an example to explain this: given a post "The new MAC is so beautiful", different users may have different understandings. For a programmer, "MAC" may refer to the Apple's laptop; but for a fashion girl, she may associate "MAC" with the lipstick. Therefore, the user's history can help distinguish the word "MAC" and provide more background knowledge for the input post, so that the post representation can be significantly enhanced. We call this encoder "personalized post encoder" -the encoded representations of the same post can be different for various users, because these users may have different profiles and different understandings of the same post.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Specifically, DHAP employs a BiGRU to encode the input post. Here we choose RNN-based architectures because they are better at capturing local correlations and encoding positional information than Transformers for short texts [19,39]. To facilitate the encoder with personalized information, DHAP uses the general user profile e to initialize hidden states of BiGRU. Given the post = ( 1 , · · · , ), its representations (h 1 , · · · , h ) are built as:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
where x is the embedding obtained by the embedding table.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
These hidden states draw personalized information from the general user profile, and in the decoding phase, they are aggregated by the decoding hidden state h through an attention mechanism:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
The detailed calculation and updating scheme of the decoding state h at time step will be described in Section 3.5.4. Based on the attention mechanism, the personalized post representation c can dynamically focus on some important words of the post according to the current decoding state. In the next section, DHAP will use c to build the dynamic post-aware user profile.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
With the general user profile, DHAP can capture personalized information of a user in a general view. However, when faced with a new input post, the historical data may play different roles. For example, when a crazy fan of cricket meets a post about cricket, they will be talkative and post a lot of things. But when they face other daily topics, they may behave much more gently. Hence, it is valuable to dynamically select the information that is most relevant to the input, and the chatbot can behave differently as the input varies. Following this idea, we propose to dynamically aggregate historical responses that are highly relevant to the current input post, and leverage them as a reference to drive the response generation. To achieve this, DHAP uses a key-value memory network [18] to store the user's historical post-response pairs. Then the personalized post representation is used as the query to select highly related keys (historical posts) from the user history memory, and their corresponding values (historical responses) are aggregated as the dynamic post-aware user profile.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
We firstly transform the historical post-response pairs into key-value pairs and build the memory.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
As we discussed earlier, the historical posts are usually issued by different users. Thus the language style and topic of them may be various. Under this circumstance, it is more reasonable to treat them independently, so DHAP applies a BiGRU to represent each historical post, respectively. In our implementation, this BiGRU shares parameters with the personalized post encoder (introduced in Section 3.3). Consider the -th historical post , its representation is computed by a summing pooling over the word dimension as p = =1 h , where h is the hidden state of the BiGRU for the -th word in . For all historical posts ( 1 , · · · , ), their representations are denoted as (p 1 , · · · , p ). Similarly, the representation of historical responses is also computed by the same pooling strategy as r = =1 e , where e is the contextual representation of the -th word in . Different from historical posts, the contextual word representations of historical responses are obtained by the history encoder in Equation (3). Thus, all historical responses are represented as (r 1 , · · · , r ).
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Finally, we build the user history memory by using the post representations as key and the corresponding response representations as value, i.e., k 1 : v 1 , · · · , k : v = p 1 : r 1 , · · · , p : r . 
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
where , is the attention weight of the -th historical response based on the personalized post representation c in a similar way like Equation (7). Note that the dynamic post-aware user profile is computed at each decoding time step . The most relevant information hidden in historical responses can thus be selected to help response generation.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Finally, the generation probability of responses can be calculated by the personalized post representation c , the general user profile e , and the dynamic post-aware user profile e . Inspired by CopyNet [7], in addition to leveraging the personalized information captured by the implicit user profile, we construct a personalized vocabulary so that the model is allowed to directly select personalized words that the user frequently used in history. Specifically, the probability of the word generated in the personalized decoder is computed as Equation (2), where is the probability of general decoding mode and ( ) is the probability of copy decoding mode. They are computed by our proposed decoding switcher.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
| and ( | ) are the probabilities of generating under two modes, respectively.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
It is worth noting that: (1) The switching probability and ( ) are both in [0, 1]. Thus, it is a "soft" decoding switcher. (2) The generic vocabulary also contains words in the personalized vocabulary. The generation probability of is obtained by the sum of probabilities under two decoding modes. Therefore, DHAP is just biased to the personalized words rather than lost in it.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
3.5.1 Decoding switcher. The decoding switcher determines the probability of the two decoding modes, i.e., predicting a word from the generic vocabulary to maintain sentence fluency, or copying a word directly from the personalized vocabulary to make the response more informative and personalized. Specifically, DHAP computes the switching probability based on the matching degree between the decoder state and the concatenation of the personalized post representation, general user profile, and dynamic post-aware user profile, and further calculates two decoding mode probabilities.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
where h is the decoding hidden state at step , and d ∈ R 2 is the matching degree vector to estimate the two mode probabilities. The softmax function guarantees that ( ) + ( ) = 1. 
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
3.5.3 Personalized copy decoding. The personalized vocabulary of user is composed of the words that appear in their historical responses. DHAP can directly select a word from this vocabulary to generate a more personalized response. Inspired by copy mechanism [7], the probability of selecting a word is computed as:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
where , is the attention weight calculated by the personalized post representation c attentively reading the representation of historical responses E with the same attention process in Equation (6).
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
3.5.4 Decoder state updating. DHAP applies a GRU as the decoder.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
The hidden state at decoding step is calculated as:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
where y −1 is the embedding vector of the last generated word. The decoding states are initialized by the last hidden state of the personalized post encoder:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Our goal is to maximize the generation probability of the target response given the input post and user's dialogue history. A length penalty is applied as [9] to alleviate the generation of meaningless responses. As a result, the loss function of DHAP is defined as:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
where is a hyper-parameter to control the associated length penalty weight. ( | < , , ) is the generation probability of word based on the given input post and user's history, which is computed in Equation (2). All parameters are optimized by the loss function and the whole model is trained in an end-to-end manner.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Although there are many public datasets for response generation, as far as we know, none of them contain user identification. To collect each user's dialogue history and evaluate the effectiveness of our model, we use two datasets extracted from two open online chatting forums, i.e., Weibo and Reddit. The two datasets contain massive dialogue utterances (i.e., post-response pairs) and user identification information, thus we can sample the data by users.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
To guarantee enough personalized information, we retrieve users with more than ten utterances to maintain an effective dialogue history. Each utterance is used as the target response for generation, while its former responses and the corresponding posts are treated as the dialogue history. We divide the utterances by users into 8:1:1 as training, validation, and test set respectively in time order. Besides, given a user, we ensure that the time of its records in the validation set and test set are behind the records in the training set. Weibo dataset is a subset of PChatbotW [22], which is collected from Weibo for the one-year period beginning from Sept. 10, 2018. On Weibo, a user can post short messages visible to the public, which will be referred to as posts. Other users can make comments on a published post, which will be referred to as response. For data cleaning, we remove hashtags, URLs, emoticons, and duplicate text as [22]. We also remove the utterances whose length is less than five words or more than 100 words. We use comparable scales of samples with [4] to conduct our experiments. It comprises 300,000 users and 8,618,374 utterances, in total 31M words.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Reddit dataset is extracted from comment chains scraped from Reddit from Dec. 1, 2015 to Oct. 30, 2018 [42]. Since the Reddit discussions can be naturally expanded as tree-structured reply chains, we pair the parent node with all its child nodes respectively, and construct multiple post-response pairs. we treat the parent node and the child node as the post and response, respectively. As a result, a parent node can be a submission or a comment, while a child node only refers to a comment. For each submission, we use its title as the post text. We clean the raw data by removing instances containing word repetitions, offensive words, or multi-language sentences. It contains 315,340 users and 24,162,464 utterances, in total 55M words.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
We evaluate the performance of our approach by comparing it with four groups of highly related and strong baseline methods:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
(1) Non-personalized response generation models. Seq2SeqWA [2] is a standard GRU-based Seq2Seq model with attention mechanism. MMI [8] is a Seq2SeqWA using Maximum Mutual Information as loss function to improve diversity.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
(2) Personalized models using user ID embeddings. Speaker [9] is also based on Seq2SeqWA but using user ID embeddings as additional input to the decoder. PersonaWAE [4] is built on an augmented Wasserstein autoencoder. It utilizes user ID embeddings for building a personalization Gaussian mixture distribution, and fuses personalization in the decoder.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
(3) Personalized models using explicit user profiles. Since no explicit user profiles are given in our datasets, we use the historical responses of users as their persona texts. GPMN [41] enhances the Seq2SeqWA with a memory module, which encodes each piece of persona description as an individual memory representation. It uses the input message as the query to aggregate and incorporate the memory representations for response generation. PerCVAE [30] uses the user profile descriptions as conditions and applies a conditional variational autoencoder to generate diverse responses.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
(4) Personalized models using implicit user profiles. Since no existing methods consider mining user profiles from dialogue history implicitly, we adapt several state-of-the-art multi-turn response generation models to personalized response generation. We replace the dialogue context in the original models by the user's historical post-response pairs. VHRED-P [27] extends the hierarchical recurrent encoder-decoder with a latent variable to model the complex dependencies among multiple utterances in the context. ReCoSa-P [40] applies a self-attention mechanism to measure the relevance between the response and each utterance in the context.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
Automatic Evaluation: We consider several automatic metrics in different perspectives to jointly evaluate the generated responses.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
(1) We use BLEU-1, BLEU-2 [20], and ROUGE-L [12] to measure word overlaps between the generated response and ground truth. A higher value of these metrics indicates a higher word-level similarity between the generated response and the golden response. Table 1: Automatic evaluation results of all models. All models are categorized into four groups: (1) non-personalized; (2) using user ID; (3) using explicit user profile; and (4) using dialogue history. " †" denotes the result is significantly worse than our method DHAP in t-test with < 0.05 level. The best results are in bold and the second best results are underlined.  (2) Following [8], we employ Dist-1 and Dist-2 to evaluate the diversity of the generated response. Responses with more distinct unigrams/bigrams will have higher Dist-1/Dist-2.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
(3) As suggested by [4], we use three embedding-based metrics to measure the semantic relevance between the generated response and the ground-truth response. Concretely, we use the bag-of-words embeddings to represent both the generated and ground-truth response, and calculate their average similarity (Ave.), greedy similarity (Gre.), and extrema similarity (Ext.). The pre-trained word embeddings for Weibo and Reddit corpus are offered by Li et al. [10] and Pennington et al. [21], respectively. (4) Furthermore, since the goal of our model is to leverage user history for personalization, we evaluate the personalized performance by measuring how much information in the dialogue history is reflected in the generated response. Following [11,15], we use Persona F1 (P-F1) to measure the unigram F1 between the generated response and user's historical responses. Thus, the more historical words the generated response contains, the higher P-F1 we will get. Since the importance of the shared words can be different, following [30], we further use Persona Coverage (P-Cover) to measure the IDF-weighted word overlap between generated response and dialogue history. Specifically, for historical responses { 1 , · · · , } and the generated response , P-Cover is defined as:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
where is the set of shared words between and . Human Evaluation: The automatic evaluation metrics can measure the quality of the generated response with respect to the ground-truth. However, due to the diversity of human dialogues, a response different from the ground-truth may also be acceptable. Thus, we randomly sample 100 test samples to conduct human evaluations. We present the generated responses, the corresponding post, and the user's historical post-response pairs to three welleducated annotators. The annotators will evaluate the quality of the generated responses in a double-blind fashion. Following [4], the evaluation criterion includes: (1) Readability, which measures the grammatical correctness and smoothness of generated responses;
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
(2) Informativeness, which measures whether the responses are informative or trivial; and (3) Personalization, which measures if the response can reflect personalized information (sharing some information with the history of the user). For the former two perspectives, we use a score 1/2/3 for bad/normal/good quality. For personalization, we use the score 0/1 to judge whether a response Table 3: Performance of ablation models on Weibo dataset. " †" denotes the result is significantly worse than our method in t-test with < 0.05 level. The best results are denoted in bold font. 
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
To determine the parameters of the model, we conducted multiple sets of experiments. The final parameters are selected as follows.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
For all datasets, we use 512 as the hidden size of GRU, 0.001 as the learning rate. The hidden size and number of heads of Transformer are 256 and 8. The number of Transformer layers = 6. The history length is set to 25. The vocabulary size is limited to 40,000. The word embedding dimension is 300/100 for Weibo/Reddit datasets, respectively. We use the Adam optimizer with a batch size of 256. We train all models for 10 epochs and select the best model based on the validation results on BLEU-1.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
4.5.1 Automatic Evaluation. All evaluation results under automatic metrics are reported in Table 1. We can observe that:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
(1) Among all models, DHAP achieves the best results in terms of all evaluation metrics. DHAP improves performance with a large margin over two strongest baselines VHRED-P and ReCoSa-P, which can also learn implicit user profile. Concretely, DHAP significantly outperforms ReCoSa-P by 28.4%/12.2% improvements in BLEU-1 on Weibo/Reddit dataset. The reason for the improvement reduction on Reddit set is that it has a larger scale and more varied conversations, which leads to more noise. Besides, for the embedding similarity metrics, DHAP also outperforms the best baselines. These results demonstrate that DHAP can generate more semantically relevant responses to the ground-truth by leveraging user's history. Furthermore, DHAP has dramatic improvements of Dist-1/2, indicating DHAP can generate more informative and diverse responses based on the personalized information. All these results prove that learning implicit user profiles from user's dialogue history can improve the quality of generated responses.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
(2) All personalized methods outperform non-personalized methods, indicating that personalization is helpful for generating more informative and relevant responses. Seq2SeqWA generally has the lowest performance, reflecting that the semantic information in the post is insufficient for generating an informative response. MMI improves the diversity performance significantly, but loses some ability on modeling semantic relevance, as it changes the training objective. Speakers and PersonaWAE use user ID embeddings to identify different users for personalization, and outperform the non-personalized methods. The explicit persona-based model GPMN and PerCVAE show comparable performance to the user ID embedding based baselines. A potential reason is that they are designed for leveraging explicit user profile, which is usually of high quality. In our case, they are only provided with the user's historical responses, which are much noisy. Therefore, existing personalized methods for explicit user profile is not appropriate for dealing with the implicit user profile contained in the user history.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
(3) Among all personalized methods, the ones using implicit user profile perform better. VHRED-P and ReCoSa-P show better performance on most metrics, confirming that the dialogue history can be used to mine implicit user profile for a specific user. However, these two methods are originally proposed for multi-turn dialogue generation. Their performance in personalized tasks is limited because the dialogue history covers far more aspects than the context in multi-turn dialogue. On the contrary, our DHAP models the personalized information in both encoder and decoder side, and consider the implicit user profile in both general and dynamic style. Hence, DHAP can achieve significant improvements compared with existing personalized baselines.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
We also conduct a human evaluation for all models on Weibo dataset. The results are shown in Table 2. Generally, DHAP achieves significant improvements in terms of all perspectives, which is consistent with the results on automatic metrics. In particular, we find that DHAP is much better than ReCoSa-P in terms of personalization. This is because DHAP learns the implicit user profile more comprehensively and enhances the influence of personalized words directly in the decoder. DHAP also performs better than other baselines in terms of readability, which shows that DHAP is better at language understanding with the help of user history. Besides, the informativeness of responses generated by DHAP is also improved. This demonstrates that leveraging personalized information is effective to generate more meaningful responses.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
In summary, the automatic and human evaluation results strongly verify that dialogue history is suitable to build the user profile implicitly, and leveraging implicit user profiles is effective to generate meaningful and personalized responses, further achieving a personalized chatbot. 
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
We further analyze the influence of different modules (Section 4.6.1) and the performance over different history lengths (Section 4.6.2). Both of these experiments are performed on Weibo dataset.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
4.6.1 Ablation Study. DHAP learns several personalized user profiles based on the dialogue history and designs a decoding switcher and two decoding strategies in the personalized decoder. We remove one of them once a time to analyze its contribution. The experiment results on Weibo dataset are shown in Table 3.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
The ablation on personalized representations. Three settings are considered: (1) without G: the general user profile e is not used; (2) without D: the dynamic post-aware user profile e is not used; and (3) without PC: the post encoder is non-personalized, namely initializing the post encoder with random states rather than the general user profile.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
The results show that all of the personalized representations are useful. Specifically, removing the general user profile causes the most decline in all metrics, which confirms the necessity and contribution of it on summarizing personalized information in a general view. The performance degradation caused by removing the dynamic post-aware user profile shows that selecting historical responses relevant to the input post contributes to the further enhancement of user modeling. The influence of removing personalization in post encoder is relatively smaller. It proves that using the user profile to enhance the understanding of the current post is effective but limited, since such information is only provided at the beginning step and decreases with the hidden state update.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
The ablation of components in the personalized decoder. We test the following variants of our model: (1) without GEN: the general decoding is banned; (2) without COP: the copying mode is banned; (3) with FIX: the probability of two modes are fixed. Specifically, the general decoding probability is set as 0.8 and the copy decoding probability is 0.2, these probabilities are set according to the best results of DHAP in our preliminary experiments.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
It can be seen that the results of three variants all underperform the whole framework. Without general decoding, the performance of DHAP drops sharply in terms of all metrics except personalization metrics. Specifically, it drops 46.55% in terms of BLEU-1. This indicates that only copying words from personalized vocabulary is unable to generate a suitable response, because there are lots of noises irrelevant to the current post and some general words may not be contained in the vocabulary. The reason for its improvement on personalized metrics is that all of the generated words are copied from the history, regardless of the significant hurt on relevance and diversity. Thus, general decoding considering both the post, decoding states, and personalized information is necessary. However, only using general decoding also hurts the performance, which indicates the words reflecting user personalized information is also very valuable. To combine the two decoding strategies, DHAP calculates the possibilities of two decoding strategies dynamically. It works well in DHAP yet using fixed probabilities has lower performance. 4.6.2 Performance across Various Lengths of Dialogue History. As we leverage user's dialogue history for personalization, the length of history may affect the model's performance. To investigate the influence of history length, we test the performance of DHAP by using different numbers of historical post-response pairs. The results of BLEU-1 on Weibo set are illustrated in Figure 3. We find:
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
(1) In general, DHAP performs better when a user has a longer dialogue history. This is consistent with our speculation as a longer dialogue history can provide more personalized information. DHAP achieves the best performance with the history length around 25. Unfortunately, when more than 30 historical pairs are used, the performance of DHAP becomes unstable. The potential reason is that more historical data may bring more noise and increase the difficulty of building an effective user profile. (2) When the history is less than 5, ReCoSa-P performs better than DHAP without general user profile. This is because the persona information is extremely limited. Under this circumstance, the more complex architecture of ReCoSa shows its superiority. Nevertheless, our DHAP still performs best, showing its scalability for various history lengths.
One Chatbot Per Person: Creating Personalized Chatbots based on Implicit User Profiles
In this work, we implemented response generation of personalized chatbots in an alternative way. Different from existing personalized methods, we propose the personalized model DHAP, which learns the implicit user profile automatically from large-scale user dialogue history. We design a personalized language model to capture the user's general interest from their historical responses and summarize the general user profile. To further highlight the historical responses which are relevant and valuable to the current input post, we build a history memory and construct the dynamic postaware user profile. We build a personalized decoder to coordinate two personalized decoding strategies. Experimental results confirm the effectiveness of our model on generating informativeness and personalized responses.
Exploiting Persona Information for Diverse Generation of Conversational Responses
Building human-like dialogue agents to pass the Turing Test [Turing, 1950] has been a long-term goal of artificial intelligence. Open-domain conversations need to be diversified [Li et al., 2015] and sustained [Li et al., 2016b], as the primary goal of an open-domain chatbot is to establish a connection with the user and to accompany the user over a long period of time [Shum et al., 2018]. Because of the vast quantity of rational responses for an open-domain conversational message, sequence-to-sequence (Seq2Seq) model [Sutskever et al., 2014] has been widely used for conversation modeling [Vinyals and Le, 2015;Shang et al., 2015;Serban et al., 2016]. Seq2Seq model enables the incorporation of rich context when mapping between consecutive dialogue turns and is trained by predicting one target response in a given dialogue context using the maximum-likelihood estimation (MLE) objective function.  Despite the successful application of Seq2Seq in dialogue modeling, it is still quite impossible for current dialogue agents to pass the Turing Test, while one of the reasons is the lack of a coherent persona [Vinyals and Le, 2015]. Endowing an open-domain chatbot with persona is challenging, yet impactful to deliver more realistic and engaging conversations. First, presenting a coherent persona is crucial for a chatbot to gain confidence and trust from users [Li et al., 2016a;Qian et al., 2017]. Second, grounded on predefined persona, a chatbot can be trained to both ask and answer questions about personal topics, thus making users more engaged [Zhang et al., 2018]. Due to these reasons, there has been a growing interest in modeling persona within the Seq2Seq model. However, human dialogue has a unique property that many other learning tasks do not: given a dialogue context, there may exist more than one valid response 1 Rajendran et al., 2018]. As illustrated in Table 1, the potential candidate responses cover different aspects and are non-repetitive. Such responses can not only provide more candidates for different dialogue strategies  but also enrich the diversity. Nevertheless, the one-to-many property is not taken into account in the present persona dialogue models. Although recent Seq2Seq-based persona models were successful, they still suffer from the low diversity problem [Jiang and de Rijke, 2018] brought by Seq2Seq. They tend to generate repetitive and non-specific responses like "I don't know" [Li et al., 2015;Li et al., 2016b], regard-less of the conversational context.
Exploiting Persona Information for Diverse Generation of Conversational Responses
In this paper, we follow the definition of persona in Zhang et al. [2018] (as exemplified in Table 1), and work on the explicit textual persona. This is because the unstructured persona data is easier to collect and more authentic 2 . We propose a memory-augmented architecture to exploit the explicit persona texts and incorporate a latent variable that can capture response variations under the conditional variational autoencoder model, rather than learn to predict only one response using MLE. To generate diverse persona-based responses, our model draws samples from the latent variable and combines two different decoding strategies to generate words or copy words directly from persona texts. We evaluate our model on the ConvAI2 persona-chat dataset and yield promising results in (i) incorporating persona information into responses and (ii) generating diverse and engaging responses. In summary, the contributions of this paper are three-fold: First, we propose to address the one-to-many property in endto-end model. To the best of our knowledge, this is the first work that jointly models persona text and response diversity. Second, we present a novel end-to-end model that incorporates persona texts into diverse responses (Section 3). Third, experimental results show that our model can generate persona-based responses as well as deliver more diverse and more engaging responses than baselines (Section 4).
Exploiting Persona Information for Diverse Generation of Conversational Responses
In this section, we describe the components of the proposed Persona-CVAE model in detail. As illustrated in Figure 1 (b), the conditional graphical model shows the core idea of proposed Persona-CVAE. We assume that the generation of z depends on x, y and p. y relies on x, p and z. The main objective is to learn the conditional probability q ϕ (z|x, p) and p θ (y|x, p, z), with parameters ϕ and θ. The variational parameters ϕ are learned jointly with the generative model parameters θ, as described in Section 3.2.
Exploiting Persona Information for Diverse Generation of Conversational Responses
The task can be formally defined as: given an input message X and a set of persona texts P = {P 1 , P 2 , ..., P n }, the goal is to generate diverse responsesŶ = {Ŷ 1 ,Ŷ 2 , ...,Ŷ m }, based on both input message and persona texts. The diverse in responses means that the utterances inŶ are non-repetitive, cover different aspects and differ in words.
Exploiting Persona Information for Diverse Generation of Conversational Responses
The proposed model uses four random variables to represent each dyadic conversation: the dialogue context X, the target response Y , the set of persona texts P and a latent variable z. z is used to capture the latent distribution over the valid responses. Further, X is composed of both input messages  and responses in dialogue history and can be denoted as Figure 2 demonstrates an overview of our model. The sentence encoder is a bidirectional recurrent neural network (BRNN) [Schuster and Paliwal, 1997] and is used to encode a single sentence (e.g., the target response). We concatenate the last hidden state of both forward and backward RNN to capture the semantic information from both sides, formulized
Exploiting Persona Information for Diverse Generation of Conversational Responses
where T is the sequence length. For the context encoder, we use a hierarchical encoding strategy. First, each sentence X i in context X is encoded by the sentence encoder to get a latent representation h senti . Then a single layer forward RNN is used to encode the sentence representations into a final state h context .
Exploiting Persona Information for Diverse Generation of Conversational Responses
Persona differs from dialogue context in two main aspects: (i) it keeps unchanged throughout the conversation, and (ii) it is only the unilateral information (here is for the chatbot), while two sides share the same dialogue context. Therefore, we conjecture that modeling persona differently from dialogue context, with a memory augmented architecture, will be beneficial for the model's performance. The details will be discussed in Section 3.3.
Exploiting Persona Information for Diverse Generation of Conversational Responses
Then we define the conditional distribution over the above random variables. As mentioned, we define the generation process as a conditional distribution p(y, z|x, p) = p(y|x, p, z)p(z|x, p) and our objective is to approximate p(y|x, p, z) and p(z|x, p) with deep neural networks. Following the definition in , we refer to p(y|x, p, z) as the response decoder and p θ (z|x, p) as the prior network. Further, to approximate the true posterior distribution, we refer to q ϕ (z|x, y, p) as the recognition network.
Exploiting Persona Information for Diverse Generation of Conversational Responses
CVAE is trained to maximize the conditional loglikelihood, but this involves an intractable marginalization over the latent variable. Previous works [Sohn et al., 2015;Yan et al., 2016] have shown that CVAE can be efficiently trained with the Stochastic Gradient Variational Bayes (SGVB) [Kingma and Welling, 2013] by maximizing the variational lower bound of the conditional log-likelihood.
Exploiting Persona Information for Diverse Generation of Conversational Responses
As proposed in , we also assume the latent variable z follows multivariate Gaussian distribution with a diagonal covariance matrix. And the variational lower bound of Persona-CVAE can be written as:
Exploiting Persona Information for Diverse Generation of Conversational Responses
Since we assume latent variable z follows isotropic Gaussian distribution, the recognition network q ϕ (z|x, y, p) ∼ N (µ, σ 2 I) and the prior network p θ (z|x, p) ∼ N (µ , σ 2 I).
Exploiting Persona Information for Diverse Generation of Conversational Responses
We sample z either from N (µ, σ 2 I) in training or N (µ , σ 2 I) in testing. To make the sampling operation differentiable, we use the reparametrization trick [Kingma and Welling, 2013] and we have:
Exploiting Persona Information for Diverse Generation of Conversational Responses
We project the concatenation of x,p and z to a vector, to serve as the initial state of response decoder. Then the response decoder predicts words under the Decoding Strategies (Section 3.4).
Exploiting Persona Information for Diverse Generation of Conversational Responses
The persona memory module has two roles: first is to encode the persona texts P = {p 1 , p 2 , ..., p k } into a dense vector (persona memory) and sencond is to select a persona p i to be expressed in the response. Notice that persona is not always needed in the response, and thus model selects persona from {p 1 , p 2 , ..., p k } ∪ {None}.
Exploiting Persona Information for Diverse Generation of Conversational Responses
The first role of the persona memory module is defined by multi-hops attention over persona texts and dialogue context. We convert the persona texts into memory vectors {m 1 , m 2 , ..., m k } of dimension d, computed by embedding each p i in a continuous space using an embedding matrix A of size d×V . We use the embedded dialogue context h context (with the same dimensions d) as an input. We compute the match between context vector h context and each memory vector m i in the embedding space by:
Exploiting Persona Information for Diverse Generation of Conversational Responses
where softmax(x i ) = e xi / j e xj . Meanwhile, each persona p i has an output vector c i (by another embedding matrix C). A single hop output o is computed by a sum over the output vector c i , weighted by the probability from equation (4):
Exploiting Persona Information for Diverse Generation of Conversational Responses
And k-hops attention are stacked in the following way:
Exploiting Persona Information for Diverse Generation of Conversational Responses
where u 0 = h context . For the embedding matrix, we use an adjacent strategy, i.e. A k+1 = C k . Finally, the u 3 is used as our persona memory 3 . The second role of the persona memory module is to decide which persona should be expressed in the generated response. This is implemented as:
Exploiting Persona Information for Diverse Generation of Conversational Responses
where W p is a weight matrix, and u 3 is the persona memory. z is the sampled latent variable, as described in Section 3.2. The latent variable captures some information that is not presented in the dialogue context and is beneficial for selecting the optimal persona. As aforementioned, in some cases, there is no need to incorporate persona information, so we need an MLP here to make the decision. The model selects a persona with the maximal probability, i.e.p = p j , where j = argmax i (α i ). Then the words from selected persona are further used in response decoder.
Exploiting Persona Information for Diverse Generation of Conversational Responses
We propose two decoding strategies (general purpose and special cases) to better express persona information.
Exploiting Persona Information for Diverse Generation of Conversational Responses
The soft decoding strategy assumes that at each decoding step t, there is a distribution α t over the two types {persona, other}. Similarly, according to the selected persona words, the entire vocabulary set is also divided into two parts, i.e. {persona words, other words}. At time step t, the response decoder first estimates word generation probabilities over the two vocabulary sets independently, denoted as P t per and P t other :
Exploiting Persona Information for Diverse Generation of Conversational Responses
and then computes the type distributions:
Exploiting Persona Information for Diverse Generation of Conversational Responses
where s t is the hidden state of decoder RNN cell. The final probability of generating a word is a mixture of type-specific generation distributions where the coefficients are type probabilities: P(y t |Y <t , X, P ) = α t per P t per ∪ α t other P t other (11) 3 In our experiments, k = 3 performs better than k = 1 or 2 , but there is no significant performance change when k = 4 or 5.
Exploiting Persona Information for Diverse Generation of Conversational Responses
We propose force decoding strategy from the observation that persona texts can directly serve as a valid response under some circumstances. In the decoding process, if the last words of the partially decoded sequence are the same as the first words of selected persona, the input words to decoder RNN cell will come from the selected persona words successively in next few steps:
Exploiting Persona Information for Diverse Generation of Conversational Responses
where p i is from the selected persona. At time step t, p i is also the output word. The decoder uses FDS only once in a decoding process. Though this strategy is simple, we found it works well in some situations.
Exploiting Persona Information for Diverse Generation of Conversational Responses
The Persona-CVAE is trained to maximizing the variational lower bound of the conditional log-likelihood in Formula (1). Persona memory module and type distribution are trained by two standard cross-entropy loss function. As addressed in Bowman et al. [2015], the KL annealing trick is necessary for the training of RNN-based CVAE. Besides, another essential technique for the model to work is the bag-of-word loss . Finally, the Persona-CVAE model is trained with the sum of these losses and optimized through backpropagation.
Exploiting Persona Information for Diverse Generation of Conversational Responses
We perform experiments on the recently released ConvAI2 benchmark dataset, which is an extended version (with a new test set) of persona-chat dataset [Zhang et al., 2018]. The conversations are obtained from crowdworkers who were randomly paired and asked to act the part of a given persona.
Exploiting Persona Information for Diverse Generation of Conversational Responses
This dataset contains 164,356 utterances in over 10,981 dialogues and has a set of 1,155 personas, each consisting of at least four profile texts. The testing set contains 1,016 dialogues and 200 never seen before personas. We set aside 800 dialogues together with its profile texts from the training set for validation. The final data have 9,181/800/1,016 dialogues for train/validate/test.
Exploiting Persona Information for Diverse Generation of Conversational Responses
To train the persona memory module, we label each utterance with its corresponding persona.We first compute word inverse document frequency: idf i = 1/(1 + log(1 + tf i )), where tf i is from the GloVe index via Zipf's law [Zhang et al., 2018]. If the utterance has the highest tf-idf similarity with a profile sentence and the similarity is higher than a threshold, then we label the utterance with this profile sentence. Otherwise, the utterance's persona label is none. We also label the position that shares a word with profile sentence to learn the type distribution.
Exploiting Persona Information for Diverse Generation of Conversational Responses
We compared the proposed Persona-CVAE model with five state-of-the-art generative baseline models. The five models fall into two categories, i.e., persona-free model and personabased model:  Table 2: Automatic evaluation results in diversity (Dtinct-1&2) and persona integration (P. Cover). N is the number of generated responses in each turn. Numbers with the mean that the result is statistically significant over all baselines (2-tailed t-test, p-value<0.01).
Exploiting Persona Information for Diverse Generation of Conversational Responses
Seq2Seq: a general persona-free model with context attention mechanism [Shang et al., 2015]. Dialogue CVAE: a persona-free model that uses a latent variable to learn a distribution over potential conversational intents and improves the discourse-level diversity of responses . Oracle Seq2Seq+Copy (Oracle. Copy): a persona-based model with copy mechanism. This model is similar to Persona-Seq2Seq, but it selects the persona text whose tokens have the highest unigram tf-idf similarity to the ground truth response (the Oracle). It achieves the best results in a variety of copy-based persona models [Yavuz et al., 2018]. The aim of this experiment is to have a better comparison with the copy-based persona models, although using the ground truth for persona selection is not fair. Among these baselines, we fine-tuned Seq2Seq, Dialogue CVAE and Oracle Seq2Seq+Copy on the ConvAI2 personachat dataset, while the other two methods are using the latest released models from Zhang et al. [2018] in ParlAI. Moreover, to make different models comparable, we generate N responses from the Seq2Seq based models by sampling from the softmax. For the latent variable based models, we sample N times from the latent z to generate responses.
Exploiting Persona Information for Diverse Generation of Conversational Responses
In our experiments 4 , the RNN is two-layer GRU with a 500dimensional hidden state. The dimension of word embedding is set to 300, and thus the persona memory size is also 300. The vocabulary size is limited to 20,000. The latent variable size is set to 100. KL annealing steps are set to 10,000. We train the model with a minibatch size of 32 and use Adam optimizer with an initial learning rate of 0.001. All parameters are initialized by sampling from a uniform distribution.
Exploiting Persona Information for Diverse Generation of Conversational Responses
Automatically evaluating an open-domain generative dialogue model is still an unsolved research challenge. Although metrics such as BLEU and perplexity have been 4 Code available at: https://github.com/vsharecodes/percvae . used for dialogue quality evaluation [Vinyals and Le, 2015;Serban et al., 2016], recent work has found that BLEU shows very weak correlation with human judgment . Further, perplexity is not computed on a per-response basis. Since the goal of the proposed model is not to predict only one target response, but rather diverse persona-based responses, we do not employ BLEU or perplexity for evaluation. Following the one-to-many property, we use two metrics to evaluate how well our model enhancing diversity and incorporating persona information: Distinct-K (Dtinct-K): this metric calculates the number of distinct k-grams in generated responses and is scaled by the total number of generated tokens to avoid favoring long responses [Li et al., 2015]. The Distinct-1 and Distinct-2 are thus the token ratios for unigrams and bigrams. This metric is an indicator of word-level diversity for generated responses. Persona Coverage (P. Cover): we propose this metric to evaluate how well persona information is expressed. We assume that there are M predefined profile sentences {p 1 , p 2 , ..., p M } and model generates N hypothesis responses {ŷ 1 ,ŷ 2 , ...,ŷ N }. The response-level persona coverage is defined as:
Exploiting Persona Information for Diverse Generation of Conversational Responses
where S(ŷ i , p j ) is the counts of shared words weighted by words' idf (as described in Section 4.1). Assume that the words' set shared byŷ i and p j are W and the inverse document frequency for w k ∈ W is f k , then we have:
Exploiting Persona Information for Diverse Generation of Conversational Responses
and |W | denotes the number of words in W . The final score is averaged over the entire test dataset, and we report the performance in Table 2. As can be seen, even if N =1, our model still outperforms all the baselines in diversity but is inferior to the Oracle model in the Persona Coverage. When N =5 and 10, the proposed model obtains the best performance in all metrics compared with all the baseline models. Due to the number of persona texts in each dialogue (generally 4 or 5), generating too many responses leads to a small decline in the Persona Coverage.
Exploiting Persona Information for Diverse Generation of Conversational Responses
The results show that under the different number of N , our model always generates diverse responses as well as effectively incorporates persona information into multiple responses. To better evaluate the diversity of generated responses, the following experiments are all based on N =5.   
Exploiting Persona Information for Diverse Generation of Conversational Responses
We also explore two settings for human evaluation. In both settings, we present persona texts, input message, as well as N generated responses.
Exploiting Persona Information for Diverse Generation of Conversational Responses
In the first setting, we employ judges to evaluate a random sample of 100 items per model according to three metrics, based on a 1/0 scoring schema, similar as Qian et al. [2017]: Engagingness (Engage.): a response is engaging only when it is appropriate, interesting and easy to answer. This is the overall score of N generated responses. Variety: this metric measures the variety of N generated responses. Score 1 indicates that there is a significant difference in the linguistic patterns and wordings and score 0 otherwise. Persona Detection: we measure the model's ability to incorporate persona information by displaying two possible persona texts: one is the true persona that the model used and another is randomly sampled from the rest of persona texts. Then the judges will determine which is more likely to be the persona used by model according to N generated responses.
Exploiting Persona Information for Diverse Generation of Conversational Responses
We calculated the Fleiss' kappa to measure inter-rater consistency. Fleiss' kappa for Engagingness, Variety and Persona Detection is 0.5149, 0.8281 and 0.6656, indicating "moderate agreement", "almost perfect" and "substantial agreement" respectively. Results in Table 3 support: First, Our model performs better than all baselines in all subjective metrics. Second, Our model successfully incorporates persona information. Third, Our model delivers diverse and engaging responses.
Exploiting Persona Information for Diverse Generation of Conversational Responses
In addition, we performed a Preference Test in three persona-based models. In this setting, the judges are presented with pairwise items and are asked to decide which of the two N responses is of higher quality. And a partial ordering relation about the overall quality is given: best response quality > response variety > persona coverage. The judges first need to do a re-rank in mind and decide which model offers a better top response. If no difference then considering variety in responses, otherwise whether persona text is expressed. As shown in Table 4, the proposed model is significantly (2-tailed t-test, p < 0.01) preferred than the other two baselines. In the pairwise setting, the diverse persona-   based responses are more attractive to users. We present some generated examples in Table 5.
Exploiting Persona Information for Diverse Generation of Conversational Responses
In order to investigate the influence of different decoding strategies, we also conducted ablation tests where one or two decoding strategies were removed from Persona-CVAE. The results are shown in Table 6. As we can see, after removing the SDS, the Persona Coverage decreases the most, indicating the SDS contributes to a higher persona information coverage. However, SDS also shows a negative effect on Distinct-1. This may be because the SDS encourages the use of words from profile sentences, which leads to lowfrequency words more difficult to appear.
Exploiting Persona Information for Diverse Generation of Conversational Responses
In this paper, we focus on the diverse generation of conversational responses based on chatbot's persona and propose a memory-augmented architecture named Persona-CVAE. We conduct experiments on the ConvAI2 dataset. Experimental results show that our model outperforms the state-of-the-art methods, especially in terms of diversity and persona integration. For future work, we will explore modeling the persona information of users in open-domain conversations.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Building a "human-like" conversation system has been an important topic in artificial intelligence, where one of the major challenges is to present a consistent persona so that the system can interact with users in a more natural way to gain users' confidence and trust. The user engagement of a dialogue agent increases when the agent is conditioned on various persona settings, including age, gender, language, location, or even a proper accent (Shum, He, and Li 2018;Wang et al. 2018;Huang, Zhu, and Gao 2019;Zhou et al. 2018b). Various approaches have been explored to personalize a dialogue system (Li et al. 2016b;Qian et al. 2018;Kottur, Wang, and Carvalho 2017). Recent advances in pre-training methods have led to stateof-the-art results in a range of natural language processing tasks (Peters et al. 2018;Devlin et al. 2019;Radford et al. 2019;Ke et al. 2019). Promising results are also obtained by applying these approaches in personalized dialogue generation models, such as fine-tuning a pre-trained model on a small set of persona-related dialogues (e.g. PERSONA-CHAT (Zhang et al. 2018)) (Mazaré et al. 2018;Wolf et al. 2018;Golovanov et al. 2019). However, the dialogue data used in the fine-tuning stage of these methods are usually crowd-sourced, where speakers are required to exchange their personas within limited turns of conversation. This data collection scheme is guaranteed to yield dialogues that cover rich persona related features (i.e., "personadense") and thus facilitate fine-tuning directly. However, such a setting is expensive and can only produce a limited amount of dialogues. Further, models fine-tuned using these data tend to over-fit to the routine that persona-related features should be exhibited in every response. This does not meet the common practice observed in our daily communication.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
As a matter of fact, most speakers in our daily conversations are not aiming to exhibit their personas within limited turns of interactions, namely, real-world dialogues are not always persona-related. For example, as shown in the dialogue session of Figure 1, speaker A and B only reveal their personas in the first turn of the conversation, while the rest part of this dialogue is not persona-related. Therefore, we argue that data collected from real-world conversations only contain a limited amount of dialogues that relate to speakers' persona. In other words, real dialogue data are "persona-sparse". Directly fine-tuning on these real-world conversations may mislead the model to focus on dialogues that are not persona-related, since these dialogues are in the majority. Further, speakers' personas may be regarded as the noises and tend to be ignored by the dialogue model since they are not related to most responses.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
To address the above issues, we propose a pre-training based method that can utilize persona-sparse data to build a personalized dialogue agent. Specifically, the dialogue data we use come from the daily conversations on social media, where speakers are not asked to reveal their personas intentionally. This differs from previous pre-training based approaches that utilize crowdsourced dialog data such as PERSONA-CHAT (Zhang et al. 2018), which is personadense and thus a direct fine-tuning process will suffice for the pre-trained dialogue model to capture persona related features (Wolf et al. 2018;Golovanov et al. 2019).
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
In this paper, we adopt the encoder-decoder framework and use a pre-trained language model to initialize an encoder and decoder. Attribute embeddings are added in the encoder to capture rich persona related features when modeling dialogue histories, and an attention routing mechanism is proposed in the decoder to incorporate the target persona in the decoding process. Three attention routes are used in this study and each route models a certain source of features, i.e., features extracted from the target persona, dialogue histories, and previously decoded tokens. A dynamic weight predictor is built to weigh the output of each route, so that the contribution of the target persona in the final output can be balanced. In this manner, we can leverage persona-sparse dialogue data in the training stage and control the amount of persona information to exhibit in the inference stage. Automatic and manual evaluation indicates that our method can produce dialogue responses that are more coherent and contain richer persona features.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Our main contributions can be summarized as follows:
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
1. We propose a pre-training based method that can utilize persona-sparse data to build personalized dialogue models. Our method can take full advantage of the pretrained model for generating diverse and coherent dialogues, while effectively leveraging real-world data that are persona-sparse to produce persona-related responses.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
2. We propose an attention routing mechanism to weigh persona features dynamically in the decoder. It allows us to utilize persona-sparse dialogue data in a unified manner during the training process and to control the amount of persona-related features to exhibit in the decoded responses.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
3. Both automatic and manual evaluation shows that our method outperforms previous methods in producing more coherent and persona-related responses.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Our study aims at generating a fluent response Y that is coherent with a given dialogue context C and an explicitly represented target persona T of the responder:  Specifically, the persona T can be regarded as a set of attributes (such as gender, location, or personal interest) T ={t 1 , t 2 , ..., t N } and each attribute is represented as a key-value pair
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
contains several turns of conversations (i.e., utterances U i ) and the persona T i of the associated speaker. Figure 2 shows an overview of our model. The encoder and decoder used in our study follow the Transformer framework (Vaswani et al. 2017), and share the same set of weights. The encoder is used to encode the dialogue context C into a context encoding E C and the target persona T into a persona encoding E T , independently. Attribute embeddings are added when producing E C . The decoder takes as input E C and E T and decodes the output in an auto-regressive way. An attention routing mechanism is proposed by extending the original multi-head attention module and introducing a dynamic weight predictor. Outputs of these attention routes are merged using the dynamically predicted weight.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
We introduce attribute embeddings in the encoder to model each persona T i , (i = 1, 2, ..., n) that is involved in the dialogue context C. Specifically, we first concatenate all the utterances in C with a special token " SPE" and map each attribute t j in T i to an embedding representation. The input embedding for the Transformer encoder at each time step is constructed by adding the word embedding, positional embedding and attribute embeddings together ( Figure 3). The proposed attribute embeddings enhance the dialogue context encoding E C since the persona of every speaker is modeled in E C . This differs from the previous work of Golovanov et al. (2019), where only word embeddings are used.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
More precisely, three attributes are modeled in this study: Gender, Location, and Interest Tags. The embedding of Gen-
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Speaker 2
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
How are you _SEP Fine _SEP Word Emb I am P 1 P 2 P 3 P 4 P 5 P 6 Positional Emb P 5 P 5
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Tag 1 Tag 2 Tag 1 Tag 1 Tag 1 Tag Emb Tag 2 Tag 2 Tag 2 Figure 3: Input representation of the dialogue context. The input embedding for each token is the sum of a word embedding, a positional embedding, and attribute embeddings. Three kinds of attribute embeddings are modeled, i.e., gender embedding, location embedding, and tag embedding. The tag embedding of a speaker is calculated as the average of all the tag representations since each speaker may have several interest tags. der and Location can be obtained simply utilizing look-up tables since these attributes only have one unique value for each speaker, while the embedding of Interest Tags is computed as the average of all the tag embeddings for a speaker since each speaker may have several different interest tags.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Moreover, for the target persona T that the generated response should be coherent to, we pack all the key-value pairs in T into a word sequence and feed the corresponding word embeddings to the encoder to obtain E T .
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
In order to effectively utilize the persona-sparse dialogue data in a unified manner, it is expected to involve little or no persona features in the decoding process when training on non-persona-related dialogues, whereas to incorporate abundant persona features when modeling persona-related dialogues. In this study, we devise an attention routing mecha-nism in the decoder to control the contribution of the target persona E T in the decoding process.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Specifically, the vanilla multi-head attention module in the original Transformer block is extended to model the encodings of the target persona E T , the dialogue context E C and previously decoded tokens E prev , respectively. We name each set of the multi-head attention operation as an attention route since it routes to different input features. More specifically, The three attention routes in our study use features extracted from previously decoded tokens E prev as the query, and attend to E T , E C and E prev , respectively, i.e.,
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Here, we use unmasked bi-directional self-attention in Eq.2 and 3 to facilitate more efficient interactions, and use masked self-attention in Eq. 4 to avoid feeding the "golden truth" token. The outputs of each attention route O T , O C and O prev are averaged using a persona weight α ∈ [0, 1]:
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
where a large α value indicates that more persona information will flow to the final outputs, and thus the generated responses are expected to exhibit more persona-related features. Note that Eq. 5 ensures that features extracted from the dialogue context O C and previous decoded tokens O prev can always be incorporated in the decoder. Ideally, the value of α should be annotated based on whether the training dialogue is persona-related or not. However, this would be impractical for a large scale dialogue dataset. In this study, we design a dynamic weight predictor to calculate α automatically in the training stage. Specifically, the predictor is modeled as a binary classifier P θ (r|E c ) that takes as input the dialogue context encoding E C and predicts whether the training dialogue is persona related (r = 1) or not (r = 0). The confidence of this binary classifier is used as the predicted weight:
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
More precisely, we model the weight predictor using a neural network that is parameterized by θ, and develop a heuristic script to produce labels for the training dialogue to optimize θ. This script applies manually crafted rules such as word matching to decide whether a given dialogue is persona-related or not. The weight predictor is jointly trained with the dialogue model by optimizing the following cross entropy loss on these script-generated noisy labels:
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Note that we can also directly use the heuristic script as the weight predictor, i.e., set α to either 1 (the input dialogue is persona-related) or 0 (otherwise) in the training process. However, these hard decisions may bring bias introduced by the script to our model and thus lead to sub-optimal results.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
On the contrary, our neural-based predictor models a soft approximation of the prior knowledge provided by the heuristic script, and the joint training approach also guide the encoder to capture more persona related features in the context encoding E C . Our experiments also verify that the soft weights produced by our predictor lead to better results compared to the hard weights produced by the heuristic script.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
A pre-training process is employed in this study. Specifically, we collect a large set of text data and follow the scheme introduced by (Radford et al. 2018) to train a language model by optimizing the standard maximum log likelihood loss:
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
where φ is the parameter set of the language model, k is the size of the context window, and u i−k , . . . , u i−1 , u i is a sequence of tokens that is sampled from the training corpus. Once pretrained, the parameter set φ is used to initialize the encoder and decoder of our model, and a fine-tuning process is employed to adapt φ to the dialogue dataset. We optimize the following loss for the dialogue generation task:
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
where E C and E T is the dialogue context and target persona encoding, respectively, and u i−k , . . . , u i−1 , u i is a sequence of tokens from the dialogue response.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Further, in order to bridge the gap between the data used in the pre-training and the fine-tuning stage, we also optimize the language model loss (i.e., Eq. 7) that is evaluated using utterances sampled from the dialogue contexts in the fine-tuning process. This is in line with the prior work (Radford et al. 2018), in which performance improvements are observed when incorporating such an auxiliary loss. Specifically, L LM (φ) is optimized in the pre-training stage and the following loss is optimized in the fine-tuning stage:
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
where λ 1 and λ 2 are hyper-parameters to balance each loss.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
The dialogue data used in this study were sampled from the PersonalDialog dataset (Zheng et al. 2019), which were collected from a Chinese social media Weibo. Each dialogue in this dataset is composed of a Weibo post and its following replies. A structured personal profile of each speaker was also provided in this dataset, and three persona attributes (i.e., "Gender", "Location" and "Interest Tags") were approached in our study. Figure 1 shows a sampled dialogue and Table 1 shows a basic statistic of the data used in this study. About 0.88M dialogues are labeled as persona-related by our heuristic script. We randomly sampled 10K sessions of dialogues as the validation set, and constructed two test sets, i.e., a random test set and a biased test set, to test the behavior of our model in different contexts. Specifically, the random test set contained 10K sessions of dialogues that were randomly sampled. Most of these dialogues did not contain persona-related features since common Weibo users are not required to exhibit their personas intentionally on Weibo. Therefore, the contexts provided by the random test set are persona-sparse. The biased test set was deliberately chosen to provide us different contexts under which speakers tend to reveal their personas. For example, the dialogue "Are you a boy or a girl?" and "I am a girl" is biased since the speaker reveals her gender in response to the gender-related post. We manually labeled 521 biased dialogues in this study. The contexts provided by the biased test set are persona-dense. It will be interesting to see if our model can produce more persona consistent responses under these biased contexts.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
In order to better evaluate whether the generated dialogue responses carry rich persona-related features, we built a binary classifier that takes as input a dialogue response R and a persona T in the form of a concatenated token sequence, and predicts whether T is exhibited in R. Specifically, we randomly sampled a batch of response-persona pairs, and manually labeled 1,044 positive pairs such that the persona is exhibited in the response. We also labeled the same number of negative pairs such that the response do not carry persona related features. We split these data into the train, validation, and test set with the ratio of 8:1:1 and fine-tuned a classifier based on the BERT-base model (Devlin et al. 2019). The accuracy of this classifier on the test set achieved 75.5%. Similar approach was also used by Zhou et al. (2018a) and Zheng et al. (2019).
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Our pre-training stage used a dataset that was collected from a set of Chinese novels, which covered a variety of genres (including Comedy, Romance, Mystery). The final pretraining corpus contains about 0.5 billion tokens and we trained a character-level language model with a vocabulary size of 13,084. The encoder and decoder contained 12 Transformer blocks, and 12 attention heads were used. Token embeddings had the size of 768 and the context window was of size 512. The dynamic weight predictor was implemented as a multilayer perceptron after an average pooling layer on E C . The value of λ 1 and λ 2 in Eq. 9 was set to 0.2 and 0.5, respec-tively. The pretraining stage lasted for 70 epochs, and we fine-tuned our model for another 30 epochs.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
We chose several baselines:
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
• Att+PAB: A RNN based model that encodes the input persona into a representation vector using a persona fusion module, and decodes personalized responses utilizing a persona-aware bias (Zheng et al. 2019).
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
• Trans.: A Transformer model (Vaswani et al. 2017) that takes concatenated dialogue histories as input and produces the corresponding dialogue response without using persona-related features.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
• TTransfo: The TransferTransfo model as introduced by Wolf et al. (2018). This model fine-tunes the pretrained model directly on the persona-sparse dialogues.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
• TTransfo + P: The same as the TransferTransfo model but this model is fine-tuned using only dialogues that are labeled as persona-related by our heuristic script, i.e., noisy persona dense dialogue data.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
• LConv: The multi-input model 1 proposed in (Golovanov et al. 2019). This model uses a pre-trained encoder and decoder and is fine-tuned directly on the persona-sparse dialogues without using the dynamic weight predictor.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
• LConv + P: The same as the multi-input model but it is fine-tuned using only dialogues that are labeled as persona-related by our heuristic script.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
All the baselines that utilize the Transformer architecture used the same set of hyper-parameters, and the same pretraining approach is employed. Further, we performed several ablation tests with our model to validate the effectiveness of each component. Specifically, the following variants were tested: (1) without the pre-training process (w/o PreT); (2) without the attribute embedding in the encoder (w/o AEmb); (3) without the dynamic weight predictor (w/o DWP), i.e., λ 2 in Eq. 9 was set to 0 and the outputs from all the attention routes were equally averaged. In order to further demonstrate the effectiveness of the proposed dynamic weighting scheme, we also tested the performance of our model using heuristic weights (+ HW), i.e., λ 2 in Eq. 9 was set to 0 and the weight α in Eq. 5 was set to either 1 or 0 based on the results of the heuristic script during the training.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Moreover, we also tried to generate different responses by setting the weight α in Eq. 5 to different values in the inference stage. Specifically, we set α to 0 (no persona), 1 (full persona), and the value predicted by the dynamic weight predictor, respectively.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Metrics We evaluated the models with the following automatic metrics: (1) Persona Accuracy (Acc.) was computed   (Li et al. 2016a) was used to measure the proportion of unique n-gram in the generated responses (n=1,2). (5) Perplexity (ppl.) was used to measure how the model fits the test data.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
The performance on the random and biased test set is shown in Table 2 and Table 3, respectively. It can be seen that our model outperforms all the baselines on all the metrics except for the perplexity. This indicates that our proposed model can produce diversified dialogue responses carrying rich persona-related features. We can further observe that: 1) Generating personalized dialogue responses hurts perplexity scores since persona-related words are relative rare and more biased generation of such words will lead to higher perplexity. Though the baseline models fit the test data well (with lower perplexity), they fail to produce more persona-related responses (with lower persona accuracy scores) compared to our model. This observation is in line with the results reported in the ConvAI2 competition (Dinan et al. 2019); 2) Ablation tests show that the pre-training stage significantly boost the model performance, and the proposed attribute embedding and dynamic weight predictor also help to improve the performance of our model;
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
3) The weight α in Eq. 5 can be used to control the amount of persona-related features to exhibit in the decoding process. Higher α values lead to more persona-consistent responses; 4) Larger performance gaps between our model and the baselines are obtained on the biased test set. This shows that our model can generate more persona-consistent responses in biased contexts.  Moreover, the effect of the persona weight α on the generated responses was further evaluated. Specifically, we computed the scores of persona accuracy, BLEU, F1, and distinct corresponding to different α values (Figure 4). It is interesting to observe that: 1) The persona accuracy increases rapidly with α ( Figure 4a). This shows that more persona-related features will be incorporated in the decoded responses when α is larger. 2) The scores for BLEU, F1 and distinct on the random test set decrease when α increases (dashed lines in Figure 4b). This is because the dialogues in the random test set are persona-sparse and less overlap between model-produced and human-generated responses will be observed if more persona-related features are incorporated. 3) A clear increasing trend for BLEU, F1 and distinct is observed on the biased test set, but a performance drop is observed when α reaches 1 (solid lines in Figure 4b). This indicates that generating more persona-related responses lead to better performance on the persona-dense contexts, but merely pursuing persona consistency may hurt the performance on other dimensions. This is in line with the manual evaluation results shown in Table 4. 
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Metrics For a given dialogue context and a target persona, we generated responses using all the transformer-based baselines and our model. Three individual annotators were employed to rate the model-generated responses together with the human-generated responses (Gold Resp) from three aspects: 1) Utterance Fluency: whether the responses are fluent and could plausibly have been produced by a human; 2) Persona Consistency: whether the responses are consistent with the target persona; 3) Context Coherency: whether the responses are coherent with the dialogue context. The rating scale of each measure is of (0, 1, 2), in which rating 0 means worst and 2 best.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
Results 200 dialogue sessions were sampled from each of these two test sets, respectively, and 3.2K responses were generated. The inter-rater annotation agreement was measured using the Fleiss's kappa κ (Randolph 2005). Particularly, the κ value for Utterance Fluency, Persona Consistency, and Context Coherency was 0.81, 0.70, 0.52, respectively on the random test set, and 0.82, 0.73, 0.49, respectively on the biased test set. This indicates substantial annotation agreement for fluency and persona consistency, and moderate agreement for context coherency. Table 4 shows the manual evaluation results. Our model outperforms all the baselines in all the measures. Particularly for persona consistency, our full persona model (i.e., α=1) significantly outperforms all the baselines with a large margin. This indicates that our model can generate more persona-consistent responses that are fluent and contextcoherent. Further observations also show that: 1) Exhibiting too many persona-related features (i.e., obtaining higher persona consistency) hurts response fluency and context coherency. This is in line with the trade-off between the persona accuracy and perplexity as observed in the automatic evaluation results. Moreover, our dynamic weight predictor provides a better balance between the persona-consistency and context coherency, especially on the biased test set; 2) The persona consistency of our full persona model (i.e., Figure 5: Sample responses generated by baselines and our model. α=1) even surpasses the human-generated response on the random test set. This further proves that our model can incorporate richer persona-related features in the generated responses. 3) Although directly fine-tuning on the noisy persona dense data (i.e., TTransfo+P and LConv+P) helps to produce more persona-consistent responses, our model still surpasses these baselines significantly. This verifies the effects of the proposed dynamic weighting scheme. This observation is also in line with the automatic evaluation results shown in Table 2 and 3. Figure 5 shows a sampled case, in which our model can generate coherent responses that reveal rich persona features, while responses produced by the baselines either do not exhibit persona-related features or are not grammatically fluent. This case also shows that the persona weight α can be effectively used to control whether to exhibit persona-related features or not. Specifically, our model with the full persona (α = 1) reveals the location attribute in the response while our model without persona (α = 0) does not exhibit persona related features. See the supplementary file for more cases.
A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
In this paper, we present a pre-training based dialogue generation model that can produce coherent persona-consistent responses conditioned on explicitly represented personas. Our method can effectively utilize persona-sparse dialogue data in the fine-tuning stage. We add attribute embeddings in the encoder to model the persona of each speaker involved in the dialogue context and devise a dynamic weighting scheme in the decoder to balance the amount of persona-related features to exhibit in the decoded responses. Automatic and manual evaluation results show that our model can incorporate richer persona-related features in the generated responses compared to state-of-the-art baselines when the dialogues available at the fine-tuning stage are persona-sparse.
A Persona-Based Neural Conversation Model
As conversational agents gain traction as user interfaces, there has been growing research interest in training naturalistic conversation systems from large volumes of human-to-human interactions (Ritter et al., 2011;Li et al., 2016). One major issue for these data-driven systems is their propensity to select the response with greatest likelihood-in effect a consensus response of the humans represented in the training data. Outputs are frequently vague or non-committal (Li et al., 2016), and when not, they can be wildly inconsistent, as illustrated in Table 1.
A Persona-Based Neural Conversation Model
In this paper, we address the challenge of consistency and how to endow data-driven systems with the coherent "persona" needed to model humanlike behavior, whether as personal assistants, per-* The entirety of this work was conducted at Microsoft.  sonalized avatar-like agents, or game characters. 1 For present purposes, we will define PERSONA as the character that an artificial agent, as actor, plays or performs during conversational interactions. A persona can be viewed as a composite of elements of identity (background facts or user profile), language behavior, and interaction style. A persona is also adaptive, since an agent may need to present different facets to different human interlocutors depending on the interaction.
A Persona-Based Neural Conversation Model
Fortunately, neural models of conversation generation Shang et al., 2015;Li et al., 2016) provide a straightforward mechanism for incorporating personas as embeddings. We therefore explore two per-sona models, a single-speaker SPEAKER MODEL and a dyadic SPEAKER-ADDRESSEE MODEL, within a sequence-to-sequence (SEQ2SEQ) framework (Sutskever et al., 2014). The Speaker Model integrates a speaker-level vector representation into the target part of the SEQ2SEQ model. Analogously, the Speaker-Addressee model encodes the interaction patterns of two interlocutors by constructing an interaction representation from their individual embeddings and incorporating it into the SEQ2SEQ model. These persona vectors are trained on human-human conversation data and used at test time to generate personalized responses. Our experiments on an open-domain corpus of Twitter conversations and dialog datasets comprising TV series scripts show that leveraging persona vectors can improve relative performance up to 20% in BLEU score and 12% in perplexity, with a commensurate gain in consistency as judged by human annotators.
A Persona-Based Neural Conversation Model
Given a sequence of inputs X = {x 1 , x 2 , ..., x n X }, an LSTM associates each time step with an input gate, a memory gate and an output gate, respectively denoted as i t , f t and o t . We distinguish e and h where e t denotes the vector for an individual text unit (for example, a word or sentence) at time step t while h t denotes the vector computed by the LSTM model at time t by combining e t and h t−1 . c t is the cell state vector at time t, and σ denotes the sigmoid function. Then, the vector representation h t for each time step t is given by:
A Persona-Based Neural Conversation Model
where
A Persona-Based Neural Conversation Model
In SEQ2SEQ generation tasks, each input X is paired with a sequence of outputs to predict: Y = {y 1 , y 2 , ..., y n Y }. The LSTM defines a distribution over outputs and sequentially predicts tokens using a softmax function:
A Persona-Based Neural Conversation Model
where f (h t−1 , e yt ) denotes the activation function between h t−1 and e yt . Each sentence terminates with a special end-of-sentence symbol EOS. In keeping with common practices, inputs and outputs use different LSTMs with separate parameters to capture different compositional patterns.
A Persona-Based Neural Conversation Model
During decoding, the algorithm terminates when an EOS token is predicted. At each time step, either a greedy approach or beam search can be adopted for word prediction.
A Persona-Based Neural Conversation Model
Our work introduces two persona-based models: the Speaker Model, which models the personality of the respondent, and the Speaker-Addressee Model which models the way the respondent adapts their speech to a given addressee -a linguistic phenomenon known as lexical entrainment (Deutsch and Pechmann, 1982).
A Persona-Based Neural Conversation Model
For the response generation task, let M denote the input word sequence (message) M = {m 1 , m 2 , ..., m I }. R denotes the word sequence in response to M , where R = {r 1 , r 2 , ..., r J , EOS} and J is the length of the response (terminated by an EOS token). r t denotes a word token that is associated with a K dimensional distinct word embedding e t . V is the vocabulary size.
A Persona-Based Neural Conversation Model
Our first model is the Speaker Model, which models the respondent alone. This model represents each individual speaker as a vector or embedding, which encodes speaker-specific information (e.g., dialect, register, age, gender, personal information) that influences the content and style of her responses. Note that these attributes are not explicitly annotated, which would be tremendously expensive for our datasets. Instead, our model manages to cluster users along some of these traits (e.g., age, country of residence) based on the responses alone. Figure 1 gives a brief illustration of the Speaker Model. Each speaker i ∈ [1, N ] is associated with a user-level representation v i ∈ R K×1 . As in standard SEQ2SEQ models, we first encode message S into a vector representation h S using the source LSTM. Then for each step in the target side, hidden units are obtained by combining the representation produced by the target LSTM at the previous time step, the word representations at the current time step, and the speaker embedding v i : 
A Persona-Based Neural Conversation Model
where W ∈ R 4K×3K . In this way, speaker information is encoded and injected into the hidden layer at each time step and thus helps predict personalized responses throughout the generation process. The Speaker embedding {v i } is shared across all conversations that involve speaker i. {v i } are learned by back propagating word prediction errors to each neural component during training.
A Persona-Based Neural Conversation Model
Another useful property of this model is that it helps infer answers to questions even if the evidence is not readily present in the training set. This is important as the training data does not contain explicit information about every attribute of each user (e.g., gender, age, country of residence). The model learns speaker representations based on conversational content produced by different speakers, and speakers producing similar responses tend to have similar embeddings, occupying nearby positions in the vector space. This way, the training data of speakers nearby in vector space help increase the generalization capability of the speaker model. For example, consider two speakers i and j who sound distinctly British, and who are therefore close in speaker embedding space. Now, suppose that, in the training data, speaker i was asked Where do you live? and responded in the UK. Even if speaker j was never asked the same question, this answer can help influence a good response from speaker j, and this without explicitly labeled geo-location information.
A Persona-Based Neural Conversation Model
Word embeddings (50k Figure 1: Illustrative example of the Speaker Model introduced in this work. Speaker IDs close in embedding space tend to respond in the same manner. These speaker embeddings are learned jointly with word embeddings and all other parameters of the neural model via backpropagation. In this example, say Rob is a speaker clustered with people who often mention England in the training data, then the generation of the token 'england' at time t = 2 would be much more likely than that of 'u.s.'. A non-persona model would prefer generating in the u.s. if 'u.s.' is more represented in the training data across all speakers.
A Persona-Based Neural Conversation Model
A natural extension of the Speaker Model is a model that is sensitive to speaker-addressee interaction patterns within the conversation. Indeed, speaking style, register, and content does not vary only with the identity of the speaker, but also with that of the addressee. For example, in scripts for the TV series Friends used in some of our experiments, the character Ross often talks differently to his sister Monica than to Rachel, with whom he is engaged in an on-again off-again relationship throughout the series. The proposed Speaker-Addressee Model operates as follows: We wish to predict how speaker i would respond to a message produced by speaker j.
A Persona-Based Neural Conversation Model
Similarly to the Speaker model, we associate each speaker with a K dimensional speaker-level representation, namely v i for user i and v j for user j. We obtain an interactive representation V i,j ∈ R K×1 by linearly combining user vectors v i and v j in an attempt to model the interactive style of user i towards user j,
A Persona-Based Neural Conversation Model
where W 1 , W 2 ∈ R K×K . V i,j is then linearly incorporated into LSTM models at each step in the target:
A Persona-Based Neural Conversation Model
V i,j depends on both speaker and addressee and the same speaker will thus respond differently to a message from different interlocutors. One potential issue with Speaker-Addressee modelling is the difficulty involved in collecting a large-scale training dataset in which each speaker is involved in conversation with a wide variety of people. Like the Speaker Model, however, the Speaker-Addressee Model derives generalization capabilities from speaker embeddings. Even if the two speakers at test time (i and j) were never involved in the same conversation in the training data, two speakers i and j who are respectively close in embeddings may have been, and this can help modelling how i should respond to j.
A Persona-Based Neural Conversation Model
For decoding, the N-best lists are generated using the decoder with beam size B = 200. We set a maximum length of 20 for the generated candidates. Decoding operates as follows: At each time step, we first examine all B × B possible next-word candidates, and add all hypothesis ending with an EOS token to the N-best list. We then preserve the top-B unfinished hypotheses and move to the next word position.
A Persona-Based Neural Conversation Model
To deal with the issue that SEQ2SEQ models tend to generate generic and commonplace responses such as I don't know, we follow Li et al. (2016) by reranking the generated N-best list using a scoring function that linearly combines a length penalty and the log likelihood of the source given the target:
A Persona-Based Neural Conversation Model
log p(R|M, v) + λ log p(M |R) + γ|R| (11) where p(R|M, v) denotes the probability of the generated response given the message M and the respondent's speaker ID. |R| denotes the length of the target and γ denotes the associated penalty weight. We optimize γ and λ on N-best lists of response candidates generated from the development set using MERT (Och, 2003) by optimizing BLEU. To compute p(M |R), we train an inverse SEQ2SEQ model by swapping messages and responses. We trained standard SEQ2SEQ models for p(M |R) with no speaker information considered.
A Persona-Based Neural Conversation Model
Data Collection Training data for the Speaker Model was extracted from the Twitter FireHose for the six-month period beginning January 1, 2012. We limited the sequences to those where the responders had engaged in at least 60 (and at most 300) 3-turn conversational interactions during the period, in other words, users who reasonably frequently engaged in conversation. This yielded a set of 74,003 users who took part in a minimum of 60 and a maximum of 164 conversational turns (average: 92.24, median: 90). The dataset extracted using responses by these "conversationalists" contained 24,725,711 3-turn sliding-window (context-message-response) conversational sequences.
A Persona-Based Neural Conversation Model
In addition, we sampled 12000 3-turn conversations from the same user set from the Twitter Fire-Hose for the three-month period beginning July 1, 2012, and set these aside as development, validation, and test sets (4000 conversational sequences each). Note that development, validation, and test sets for this data are single-reference, which is by design. Multiple reference responses would typically require acquiring responses from different people, which would confound different personas.
A Persona-Based Neural Conversation Model
Training Protocols We trained four-layer SEQ2SEQ models on the Twitter corpus following the approach of (Sutskever et al., 2014). Details are as follows:
A Persona-Based Neural Conversation Model
• 4 layer LSTM models with 1,000 hidden cells for each layer. • Batch size is set to 128.
A Persona-Based Neural Conversation Model
• Learning rate is set to 1.0. • Parameters are initialized by sampling from the uniform distribution [−0.1, 0.1]. • Gradients are clipped to avoid gradient explosion with a threshold of 5. • Vocabulary size is limited to 50,000.
A Persona-Based Neural Conversation Model
• Dropout rate is set to 0.2. Source and target LSTMs use different sets of parameters. We ran 14 epochs, and training took roughly a month to finish on a Tesla K40 GPU machine.
A Persona-Based Neural Conversation Model
As only speaker IDs of responses were specified when compiling the Twitter dataset, experiments on this dataset were limited to the Speaker Model.
A Persona-Based Neural Conversation Model
The Twitter Persona Dataset was collected for this paper for experiments with speaker ID information. To obtain a point of comparison with prior state-of-the-art work Li et al., 2016), we measure our baseline (non-persona) LSTM model against prior work on the dataset of , which we call the Twitter Sordoni Dataset. We only use its test-set portion, which contains responses for 2114 context and messages. It is important to note that the Sordoni dataset offers up to 10 references per message, while the Twitter Persona dataset has only 1 reference per message. Thus BLEU scores cannot be compared across the two Twitter datasets (BLEU scores on 10 references are generally much higher than with 1 reference). Details of this dataset are in .
A Persona-Based Neural Conversation Model
Data Collection For the dyadic Speaker-Addressee Model we used scripts from the American television comedies Friends 2 and The Big Bang Theory, 3 available from Internet Movie Script Database (IMSDb). 4 We collected 13 main characters from the two series in a corpus of 69,565 turns. We split the corpus into training/development/testing sets, with development and testing sets each of about 2,000 turns.
A Persona-Based Neural Conversation Model
Training Since the relatively small size of the dataset does not allow for training an open domain dialog model, we adopted a domain adaption strategy where we first trained a standard SEQ2SEQ System BLEU MT baseline (Ritter et al., 2011) 3.60% Standard LSTM MMI (Li et al., 2016) 5.26% Standard LSTM MMI (our system) 5.82% Human 6.08% Table 2: BLEU on the Twitter Sordoni dataset (10 references). We contrast our baseline against an SMT baseline (Ritter et al., 2011), and the best result (Li et al., 2016) on the established dataset of . The last result is for a human oracle, but it is not directly comparable as the oracle BLEU is computed in a leave-one-out fashion, having one less reference available. We nevertheless provide this result to give a sense that these BLEU scores of 5-6% are not unreasonable. models using a much larger OpenSubtitles (OSDb) dataset (Tiedemann, 2009), and then adapting the pre-trained model to the TV series dataset.
A Persona-Based Neural Conversation Model
The OSDb dataset is a large, noisy, open-domain dataset containing roughly 60M-70M scripted lines spoken by movie characters. This dataset does not specify which character speaks each subtitle line, which prevents us from inferring speaker turns. Following , we make the simplifying assumption that each line of subtitle constitutes a full speaker turn. 5 We trained standard SEQ2SEQ models on OSDb dataset, following the protocols already described in Section 5.1. We run 10 iterations over the training set.
A Persona-Based Neural Conversation Model
We initialize word embeddings and LSTM parameters in the Speaker Model and the Speaker-Addressee model using parameters learned from OpenSubtitles datasets. User embeddings are randomly initialized from [−0.1, 0.1]. We then ran 5 additional epochs until the perplexity on the development set stabilized.
A Persona-Based Neural Conversation Model
Following Li et al., 2016) we used BLEU (Papineni et al., 2002) for parameter tuning and evaluation. BLEU has been shown to correlate well with human judgment on the response generation task, as demonstrated in . Besides BLEU scores, we also report perplexity as an indicator of model capability.
A Persona-Based Neural Conversation Model
Since our main experiments are with a new dataset (the Twitter Persona Dataset), we first show that our LSTM baseline is competitive with the state-of-   the-art (Li et al., 2016) on an established dataset, the Twitter Sordoni Dataset . Our baseline is simply our implementation of the LSTM-MMI of (Li et al., 2016), so results should be relatively close to their reported results. Table 2 summarizes our results against prior work. We see that our system actually does better than (Li et al., 2016), and we attribute the improvement to a larger training corpus, the use of dropout during training, and possibly to the "conversationalist" nature of our corpus.
A Persona-Based Neural Conversation Model
We first report performance on the Twitter Persona dataset. Perplexity is reported in Table 3. We observe about a 10% decrease in perplexity for the Speaker model over the standard SEQ2SEQ model. In terms of BLEU scores (Table 4), a significant performance boost is observed for the Speaker model over the standard SEQ2SEQ model, yielding an increase of 21% in the maximum likelihood (MLE) setting and 11.7% for mutual information setting (MMI). In line with findings in (Li et al., 2016), we observe a consistent performance boost introduced by the MMI objective function over a standard SEQ2SEQ model based on the MLE objective function. It is worth noting that our persona models are more beneficial to the MLE models than to the MMI models. This result is intuitive as the persona models help make Standard LSTM MLE outputs more informative and less bland, and thus make the use of MMI less critical. For the TV Series dataset, perplexity and BLEU scores are respectively reported in Table 5 and Table 6. As can be seen, the Speaker and Speaker-Addressee models respectively achieve perplexity values of 25.4 and 25.0 on the TV-series dataset,   7.0% and 8.4% percent lower than the correspondent standard SEQ2SEQ models. In terms of BLEU score, we observe a similar performance boost as on the Twitter dataset, in which the Speaker model and the Speaker-Addressee model outperform the standard SEQ2SEQ model by 13.7% and 10.6%. By comparing the Speaker-Addressee model against the Speaker model on the TV Series dataset, we do not observe a significant difference. We suspect that this is primarily due to the relatively small size of the dataset where the interactive patterns might not be fully captured. Smaller values of perplexity are observed for the Television Series dataset than the Twitter dataset, the perplexity of which is over 40, presumably due to the noisier nature of Twitter dialogues.
A Persona-Based Neural Conversation Model
Diverse Responses by Different Speakers Table 7 represents responses generated by persona models in response to three different input questions. We randomly selected 10 speakers (without cherry-picking) from the original Twitter dataset. We collected their user level representations from a speaker look-up table and integrated them into the decoding models. The model tends to generate specific responses for different people in response to the factual questions. 6 Table 8 shows responses generated from the Speaker-Addressee Model using the TV-series dataset. Interestingly, we regularly observe that this model is sensitive to the identity of the addressee, generating words specifically targeted at that addressee (e.g., her name). For example, the model produces Of course, I love you, Emily in response to an input from Emily. Also, the model generates Of course I love you. ( kisses him), where the pronoun "him" accurately identifies the gender of the addressee.  Table 7: Responses generated by the baseline (LSTM-MMI) and the Speaker Model for ten randomly selected users, without cherry picking. # indicates poor-quality responses produced by the system.
A Persona-Based Neural Conversation Model
We conducted a human evaluation of outputs from the Speaker Model, using a crowdsourcing service. Since we cannot expect crowdsourced human judges to know or attempt to learn the ground truth of Twitter users who are not well-known public figures, we designed our experiment to evaluate the consistency of outputs associated with the speaker IDs. To this end, we collected 24 pairs of questions for which we would expect  responses to be consistent if the persona model is coherent. For example, responses to the questions What country do you live in? and What city do you live in? would be considered consistent if the answers were England and London respectively, but not if they were UK and Chicago. Similarly, the responses to Are you vegan or vegetarian? and Do you eat beef? are consistent if the answers generated are vegan and absolutely not, but not if they are vegan and I love beef. We collected 20 pairs of outputs for randomly-selected personas provided by the Speaker Model for each question pair (480 response pairs total). We also obtained the corresponding outputs from the baseline MMI-enhanced SEQ2SEQ system.
A Persona-Based Neural Conversation Model
Since our purpose is to measure the gain in consistency over the baseline system, we presented the pairs of answers system-pairwise, i.e., 4 responses, 2 from each system, displayed on the screen, and asked judges to decide which of the two systems was more consistent. The position in which the system pairs were presented on the screen was randomized. The two systems were judged on 5-point zero-sum scale, assigning a score of 2 (-2) if one system was judged more (less) consistent than the other, and 1 (-1) if one was rated "somewhat" more (less) consistent. Ties were assigned a score of zero. Five judges rated each pair and their scores were averaged and remapped into 5 equal-width bins. After discarding ties, we found the persona model was judged either "more consistent" or "somewhat more consistent" in 56.7% of cases. If we ignore the "somewhat more consistent" judgments, the persona model wins in 6.1% of cases, compared  Table 9: Examples of speaker consistency and inconsistency generated by the Speaker Model with only 1.6% for the baseline model. It should be emphasized that the baseline model is a strong baseline, since it represents the consensus of all 70K Twitter users in the dataset 7 . Table 9 illustrates how consistency is an emergent property of two arbitrarily selected users. The model is capable of discovering the relations between different categories of location such as London and the UK, Jakarta and Indonesia. However, the model also makes inconsistent response decisions, generating different answers in the second example in response to questions asking about age or major. Our proposed persona models integrate user embeddings into the LSTM, and thus can be viewed as encapsulating a trade-off between a persona-specific generation model and a general conversational model.
A Persona-Based Neural Conversation Model
We have presented two persona-based response generation models for open-domain conversation generation. There are many other dimensions of speaker behavior, such as mood and emotion, that are beyond the scope of the current paper and must be left to future work.
A Persona-Based Neural Conversation Model
Although the gains presented by our new models are not spectacular, the systems outperform our baseline SEQ2SEQ systems in terms of BLEU, perplexity, and human judgments of speaker consistency. We have demonstrated that by encoding personas in distributed representations, we are able to capture personal characteristics such as speaking style and background information. In the Speaker-Addressee model, moreover, the evidence suggests that there is benefit in capturing dyadic interactions.
A Persona-Based Neural Conversation Model
Our ultimate goal is to be able to take the profile of an arbitrary individual whose identity is not known in advance, and generate conversations that accurately emulate that individual's persona in terms of linguistic response behavior and other salient characteristics. Such a capability will dramatically change the ways in which we interact with dialog agents of all kinds, opening up rich new possibilities for user interfaces. Given a sufficiently large training corpus in which a sufficiently rich variety of speakers is represented, this objective does not seem too far-fetched.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
Non-goal-oriented dialogue systems (chatbots) are an interesting test-bed for interactive Natural Language Processing (NLP) systems and are also directly useful in a wide range of applications ranging from technical support services to entertainment. However, building intelligent conversational agents remains an unsolved problem in artificial intelligence research. Recently, recurrent neural network based models with sufficient capacity and access to large datasets attracted a large interest when first attempted. Vinyals and Le (2015) showed that they were capable of generating meaningful responses in some chit-chat settings. Still, further inquiries in the capabilities of these neural network architectures and developments Miao, Yu, and Blunsom, 2015;Sordoni et al., 2015;Serban et al., 2017;Li, Monroe, and Jurafsky, 2016;Li et al., 2017) indicated that they were limited which made communicating with them a rather unsatisfying experience for human beings.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
The main issues with these architectures can be summarized as: • (i) the wildly inconsistent outputs and the lack of a consistent personality , • (ii) the absence of a long-term memory as these models have difficulties to take into account more than the last dialogue utterance; and
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
Copyright c 2019, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
• (iii) a tendency to produce consensual and generic responses (e.g. I dont know) which are vague and not engaging for humans (Li, Monroe, and Jurafsky, 2016).
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
In this work, we make a step toward more consistent and relevant data-driven conversational agents by proposing a model architecture, associated training and generation algorithms which are able to significantly improve over the traditional seq-2-seq and information-retrieval baselines in terms of (i) relevance of the answer (ii) coherence with a predefined personality and dialog history, and (iii) grammaticality and fluency as evaluated by automatic metrics.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
An interesting challenge to evaluate the quality of opendomain conversation agent is the Conversational Intelligence Challenge 2 1 (ConvAI2) that was held during the NIPS 2018 conference and which we shortly present here with its associated dataset.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
ConvAI2 is based on the PERSONA-CHAT dataset (Zhang et al., 2018), a crowd-sourced dialogue dataset in which each speaker was asked to condition its utterances on a predefined profile comprising a few sentences defining a personality as illustrated on figure 1. Paired workers were asked to chat naturally and to get to know each other during the conversation. This produced an interesting dataset with rapid turns of topics as it can be seen on the example we reproduce on table 1.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
As automatic evaluation is still an open question in dialogue systems (Liu et al., 2016), the PERSONA-CHAT dataset comes with three automated metrics on its evaluation set. The ConvAI2 challenge further evaluated these metrics on a privately held portion of PERSONA-CHAT combined with human evaluation.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
The automatic metrics involves three tasks defined on the same dataset which are (i) a language modeling task where the metric is the perplexity of gold utterance tokens as computed from the model's next token probability predictions (denoted PPL) (ii) a next utterance retrieval task where the associated metric is the accuracy of retrieving a gold next utterance among 19 random distractor responses sampled from other dialogues (denoted Hits@1) and (iii) a generation task which consists in generating a response in the dialog setting  and where the metric is the F1 (precision and recall) of the content words of a gold dialog utterance in the predicted utterances (denoted F1).
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
Human evaluations are based on a combination of four metrics: fluency, consistency, engagingness (each evaluated as a grade between 1 and 5) and whether the human could guess the persona used by the bot (selection between two possible personas).
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
The generative model used in TransferTransfo is a multilayer Transformer encoder based on the Generative Pretrained Transformer of Radford et al.. This model largely follows the original transformer work of Vaswani et al.. For more details on this recent model architecture which has become ubiquitous in Natural Language Processing, we refer readers to the detailed guide recently published by the Harvard SEAS natural-language processing group: The Annotated Transformer. 2 We used a 12-layer decoder-only transformer with masked self-attention heads (768 dimensional states and 12 attention heads). By masked attention, we mean that the Transformer uses constrained self-attention where every token can only attend to its left context. In the literature this version of the Transformer is often referred to as a Transformer decoder since it is similar to the decoder part of the original encoder-decoder Transformer of Vaswani et al. (2017).
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
This model is similar to the large Transformer model recently used in several works leading to impressive results on several down-stream NLP tasks (Radford et al., 2018;Devlin et al., 2018). Our model is based on a recently published PyTorch adaptation by the HuggingFace team which can be found at: https://github.com/huggingface/ pytorch-openai-transformer-lm.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
Following Radford et al.; Devlin et al. the model uses learned positional embeddings with supported sequence lengths up to 512 tokens. The input sentences are preprocessed and tokenized using bytepair encoding (BPE) vocabulary with 40,000 merges (Sennrich, Haddow, and Birch, 2015).
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
Following the work of Radford et al., the model is pretrained on the BooksCorpus dataset (Zhu et al., 2015) which contains over 7,000 unpublished books (about 800M words) from a variety of genres (Adventure, Fantasy, Romance...). The critical choice for this pre-training dataset is to use a document-level corpus rather than a shuffled sentence-level corpus to take advantage of long contiguous sequences and paragraphs and learn to condition on long-range information. This is not possible with shuffled sentence-level corpora such as the Billion Word Benchmark (Chelba et al., 2013) used for instance in ELMo (Peters et al., 2018). We used the pre-trained model weights open-sourced by Radford et al..
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
After the pre-training step, the model is fine-tuned on the PERSONA-CHAT dataset using an augmented input representation and a multi-task learning scheme that we will now describe in greater details.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
Input representation We adapt the input representation of the model to be able to switch from a single (or unannotated) speaker setting like the one of the BookCorpus dataset to a two-speakers settings plus personality sentences like the one of the PERSONA-CHAT dataset. More precisely, a sequence of input tokens for the model is constructed for each utterance by concatenating all the persona sentences of the current speaker (usually 4 to 6 sentences in the PERSONA-CHAT dataset) with a history of the dialog's previous utterances (typically 3 to 5 previous utterances).
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
From this sequence of input tokens, a sequence of input embeddings for the Transformer is constructed as follows. The word and positional embeddings learned during the pretraining phase are augmented with a set of dialog-state embeddings illustrated on figure 1.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
This set of additional embeddings is used to indicate whether the current token is part of (i) a personality sentence, (ii) an utterance from PERSON1 or (iii) an utterance from PERSON2. These additional embeddings are learned on the PERSONA-CHAT dataset during the fine-tuning phase.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
The input of the self-attention block of the Transformer model is then the sum of the three types of embeddings (word, dialog-state and positional) for each word.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
Separation tokens may also be added to further separate each utterances of the dialog as it is commonly done for Transformer's inputs (Radford et al., 2018;Devlin et al., 2018).
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
Another simple adaptation from pre-training to finetuning is to promote an invariance to personality sentence ordering by reusing the same positional embeddings for each personality sentences. This is similar in spirit to the Set Transformer recently proposed in Lee et al.. Self-attention model are inherently insensitive to position and ordering and this feature can be conveniently harnessed to bias toward positional invariance. One interesting invariance that can be observed in conditional dialog datasets like the PERSONA-CHAT dataset is the invariance of the predicted utterances with respect to various orders of the personality sentences conditioning the dialog. A similar effect can be obtained by augmenting the training dataset with copies of the dialogs wherein the personality sentences are shuffled.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
Multi-task learning Fine-tuning is done by optimizing a combination of two loss functions: (i) a next-utterance classification loss, and (ii) a language modeling loss.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
The next-utterance classification loss is illustrated on figure 2 and bears similarities with the Next Sentence Prediction task developed in a parallel work by Devlin et al.. It consists in training a classifier to distinguish a correct next utterance appended to the input sequence from a set of randomly sampled distractors (in practice between 2 and 6 randomly sampled utterances). The classifier is a linear layer taking as input the last hidden state of the self-attention model and computing a score. For classification a special token [CLS] is added at the sentence illustrated in blue on figure 2, the last hidden state used for the classifier thus corresponds to the hidden-state associated to this termination special token. The computed scores are passed through a softmax layer to obtain classification probabilities. The parameters of the Transformer and the next-utterance classifier layer are finetuned jointly to maximize the log-probability of the correct label.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
The language modeling loss is the commonly used crossentropy loss where the final hidden state of the self-attention model is fed into an output softmax over the vocabulary to obtain next token probabilities. These probabilities are then scored using a negative log-likelihood loss where the gold next tokens are taken as labels.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
Fine-tuning details We fine-tuned the model with a batch size of 32 sequences having an average of 250 tokens depending on the batch for 200,000 steps, which is approximately 2 epochs over the PERSONA-CHAT training dataset (32 sequences * 250 tokens = 8,000 tokens/batch). We used Adam with a learning rate of 6.25e-5, β 1 = 0.9, β 2 = 0.999, L2 weight decay of 0,01 and a coefficient of 2 on the Language Modeling loss when summing with the next-utterance classification loss losses. The learning rate was linearly decayed to zero over the course of the training. We use a dropout probability of 0.1 on all layers. Following Radford et al. we use a relu activation function. Fine-tuning the model took about 10h on four K80 GPUs.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
Decoding details Generation was performed using beam search with sampling and a small beam size of 4. Simple n-grams filtering is used to ensure the model doesn't directly copy from the personality sentences (forbidden by the ConvAI2 rules) as well as older utterances. The final beams are ranked according to a scalar combination of the length-normalized utterance probability and the nextutterance classification score. Increasing the importance of the next-utterance classification score results in utterances that stick more closely to the provided personality sentences but also reduce the diversity of the dialog.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
Results on the public evaluation split and the privately-held test splits of the PERSONA-CHAT dataset are illustrated on   More importantly, while the model's hyper-parameters were tuned on the validation set, the performance improvements translate to the private test set as scored by the Con-vAI2 evaluation server with a 45% absolute improvement in perplexity (PPL), 46% absolute improvement in Hits@1 and 20% improvement in F1.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
The perplexity is noticeably low for an open-domain language modeling task which may be in-part due to a few repetitive portions of the dataset like the introductory utterances at the beginning of each dialog ("Hello, how are you?") and the copy mechanisms from the personality sentences.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
Transfer learning from language models have been recently shown to bring strong empirical improvements in discriminative language understanding tasks. In the present work, we show that such improvements can be extended to generative tasks such as open-domain dialog generation which combine many linguistics aspects such as co-reference resolution, common-sense knowledge and long-range dependency modeling among others. We offer hints as to what kind of multi-task fine-tuning setups can be effective in these setups and illustrate the effectiveness of this approach on a recent dialog task. Important future work is still needed to understand the most optimal settings and models.
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
PPL Hits@1 F1 PPL Hits@1 F1 Generative Profile Memory (Zhang et al., 2018) 34.54 12.5 ----Retrieval KV Profile Memory (Zhang et al., 2018) -51.  
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Despite much recent success in natural language processing and dialogue research, communication between a human and a machine is still in its infancy. It is only recently that neural models have had sufficient capacity and access to sufficiently large datasets that they appear to generate meaningful responses in a chit-chat setting. Still, conversing with such generic chit-chat models for even a short amount of time quickly exposes their weaknesses Vinyals and Le, 2015).
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Common issues with chit-chat models include: (i) the lack of a consistent personality (Li et al., 2016a) as they are typically trained over many dialogs each with different speakers, (ii) the lack of an explicit long-term memory as they are typically trained to produce an utterance given only the recent dialogue history (Vinyals and Le, 2015); 1 Work done while at Facebook AI Research. and (iii) a tendency to produce non-specific answers like "I don't know" (Li et al., 2015). Those three problems combine to produce an unsatisfying overall experience for a human to engage with. We believe some of those problems are due to there being no good publicly available dataset for general chit-chat.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Because of the low quality of current conversational models, and because of the difficulty in evaluating these models, chit-chat is often ignored as an end-application. Instead, the research community has focused on taskoriented communication, such as airline or restaurant booking (Bordes et al., 2016), or else singleturn information seeking, i.e. question answering (Rajpurkar et al., 2016). Despite the success of the latter, simpler, domain, it is well-known that a large quantity of human dialogue centers on socialization, personal interests and chit-chat (Dunbar et al., 1997). For example, less than 5% of posts on Twitter are questions, whereas around 80% are about personal emotional state, thoughts or activities, authored by so called "Meformers" (Naaman et al., 2010).
Personalizing Dialogue Agents: I have a dog, do you have pets too?
In this work we make a step towards more engaging chit-chat dialogue agents by endowing them with a configurable, but persistent persona, encoded by multiple sentences of textual description, termed a profile. This profile can be stored in a memory-augmented neural network and then used to produce more personal, specific, consistent and engaging responses than a persona-free model, thus alleviating some of the common issues in chit-chat models. Using the same mechanism, any existing information about the persona of the dialogue partner can also be used in the same way. Our models are thus trained to both ask and answer questions about personal topics, and the resulting dialogue can be used to build a model of the persona of the speaking partner.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
To support the training of such models, we present the PERSONA-CHAT dataset, a new dialogue dataset consisting of 162,064 utterances between crowdworkers who were randomly paired and each asked to act the part of a given provided persona (randomly assigned, and created by another set of crowdworkers). The paired workers were asked to chat naturally and to get to know each other during the conversation. This produces interesting and engaging conversations that our agents can try to learn to mimic.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Studying the next utterance prediction task during dialogue, we compare a range of models: both generative and ranking models, including Seq2Seq models and Memory Networks (Sukhbaatar et al., 2015) as well as other standard retrieval baselines. We show experimentally that in either the generative or ranking case conditioning the agent with persona information gives improved prediction of the next dialogue utterance. The PERSONA-CHAT dataset is designed to facilitate research into alleviating some of the issues that traditional chitchat models face, and with the aim of making such models more consistent and engaging, by endowing them with a persona. By comparing against chit-chat models built using the OpenSubtitles and Twitter datasets, human evaluations show that our dataset provides more engaging models, that are simultaneously capable of being fluent and consistent via conditioning on a persistent, recognizable profile.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
The aim of this work is to facilitate more engaging and more personal chit-chat dialogue. The PERSONA-CHAT dataset is a crowd-sourced dataset, collected via Amazon Mechanical Turk, where each of the pair of speakers condition their dialogue on a given profile, which is provided.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
The data collection consists of three stages: (i) Personas: we crowdsource a set of 1155 possible personas, each consisting of at least 5 profile sentences, setting aside 100 never seen before personas for validation, and 100 for test.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
(ii) Revised personas: to avoid modeling that takes advantage of trivial word overlap, we crowdsource additional rewritten sets of the same 1155 personas, with related sentences that are rephrases, generalizations or specializations, rendering the task much more challenging.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
(iii) Persona chat: we pair two Turkers and assign them each a random (original) persona from the pool, and ask them to chat. This resulted in a dataset of 162,064 utterances over 10,907 dialogs, 15,602 utterances (1000 dialogs) of which are set aside for validation, and 15,024 utterances (968 dialogs) for test.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
The final dataset and its corresponding data collection source code, as well as models trained on the data, are all available open source in ParlAI 2 .
Personalizing Dialogue Agents: I have a dog, do you have pets too?
In the following, we describe each data collection stage and the resulting tasks in more detail.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
We asked the crowdsourced workers to create a character (persona) description using 5 sentences, providing them only a single example: 2 http://parl.ai "I am a vegetarian. I like swimming. My father used to work for Ford. My favorite band is Ma-roon5. I got a new job last month, which is about advertising design."
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Our aim was to create profiles that are natural and descriptive, and contain typical topics of human interest that the speaker can bring up in conversation. Because the personas are not the real profiles of the Turkers, the dataset does not contain personal information (and they are told specifically not to use any). We asked the workers to make each sentence short, with a maximum of 15 words per sentence. This is advantageous both for humans and machines: if they are too long, crowdsourced workers are likely to lose interest, and for machines the task could become more difficult.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Some examples of the personas collected are given in Table 1 (left).
Personalizing Dialogue Agents: I have a dog, do you have pets too?
A difficulty when constructing dialogue datasets, or text datasets in general, is that in order to encourage research progress, the task must be carefully constructed so that is neither too easy nor too difficult for the current technology (Voorhees et al., 1999). One issue with conditioning on textual personas is that there is a danger that humans will, even if asked not to, unwittingly repeat profile information either verbatim or with significant word overlap. This may make any subsequent machine learning tasks less challenging, and the solutions will not generalize to more difficult tasks. This has been a problem in some recent datasets: for example, the dataset curation technique used for the well-known SQuAD dataset suffers from this word overlap problem to a certain extent (Chen et al., 2017).
Personalizing Dialogue Agents: I have a dog, do you have pets too?
To alleviate this problem, we presented the original personas we collected to a new set of crowdworkers and asked them to rewrite the sentences so that a new sentence is about "a related characteristic that the same person may have", hence the revisions could be rephrases, generalizations or specializations. For example "I like basketball" can be revised as "I am a big fan of Michael Jordan" not because they mean the same thing but because the same persona could contain both.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
In the revision task, workers are instructed not to trivially rephrase the sentence by copying the original words. However, during the entry stage if a non-stop word is copied we issue a warning, I also went to school to work with technology. My mother is a medical doctor
Personalizing Dialogue Agents: I have a dog, do you have pets too?
The woman who gave birth to me is a physician. I am very shy.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
I am not a social person. I like to build model spaceships.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
I enjoy working with my hands.  
Personalizing Dialogue Agents: I have a dog, do you have pets too?
After collecting personas, we then collected the dialogues themselves, conditioned on the personas. For each dialogue, we paired two random crowdworkers, and gave them the instruction that they will chit-chat with another worker, while playing the part of a given character. We then provide them with a randomly chosen persona from our pool, different to their partners. The instructions are on purpose quite terse and simply ask them to "chat with the other person naturally and try to get to know each other". In an early study we noticed the crowdworkers tending to talk about themselves (their own persona) too much, so we also added the instructions "both ask questions and answer questions of your chat partner" which seemed to help. We also gave a bonus for high quality dialogs. The dialog is turn-based, with a maximum of 15 words per message. We again gave instructions to not trivially copy the character descriptions into the messages, but also wrote explicit code sending them an error if they tried to do so, using simple string matching. We define a minimum dialogue length which is randomly between 6 and 8 turns each for each dialogue. An example dialogue from the dataset is given in Table 2.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
We focus on the standard dialogue task of predicting the next utterance given the dialogue history, but consider this task both with and without the profile information being given to the learning agent. Our goal is to enable interesting directions for future research, where chatbots can for instance have personalities, or imputed personas could be used to make dialogue more engaging to the user. We consider this in four possible scenarios: conditioning on no persona, your own persona, their persona, or both. These scenarios can be tried using either the original personas, or the revised ones. We then evaluate the task using three metrics: (i) the log likelihood of the correct sequence, measured via perplexity, (ii) F1 score, and (iii) next utterance classification loss, following . The latter consists of choosing N random distractor responses from other dialogues (in our setting, N =19) and the model selecting the best response among them, resulting in a score of one if the model chooses the correct response, and zero otherwise (called hits@1 in the experiments).
Personalizing Dialogue Agents: I have a dog, do you have pets too?
We consider two classes of model for next utterance prediction: ranking models and generative models. Ranking models produce a next utterance by considering any utterance in the training set as a possible candidate reply. Generative models generate novel sentences by conditioning on the dialogue history (and possibly, the persona), and then generating the response word-by-word. Note one can still evaluate the latter as ranking models by computing the probability of generating a given candidate, and ranking candidates by those scores.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
We first consider two baseline models, an IR baseline (Sordoni et al., 2015) and a supervised embedding model, Starspace (Wu et al., 2017) 3 . While there are many IR variants, we adopt the simplest one: find the most similar message in the (training) dataset and output the response from that exchange. Similarity is measured by the tf-idf weighted cosine similarity between the bags of words. Starspace is a recent model that also performs information retrieval but by learning the similarity between the dialog and the next utterance by optimizing the embeddings directly for that task using the margin ranking loss and k-negative sampling. The similarity function sim(q, c ′ ) is the cosine similarity of the sum of word embeddings of the query q and candidate c ′ . Denoting the dictionary of D word embeddings as W which is a D × d matrix, where W i indexes the i th word (row), yielding its d-dimensional embedding, it embeds the sequences q and c ′ .
Personalizing Dialogue Agents: I have a dog, do you have pets too?
In both methods, IR and StarSpace, to incorporate the profile we simply concatenate it to the query vector bag of words.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Both the previous models use the profile information by combining it with the dialogue history, which means those models cannot differentiate between the two when deciding on the next utterance. In this model we instead use a memory network with the dialogue history as input, which then performs attention over the profile to find relevant lines from the profile to combine with the input, and then finally predicts the next utterance. We use the same representation and loss as in the Starspace model, so without the profile, the two models are identical. When the profile is available attention is performed by computing the similarity of the input q with the profile sentences p i , computing the softmax, and taking the weighted sum:
Personalizing Dialogue Agents: I have a dog, do you have pets too?
where Softmax(z i ) = e z i / j e z j . One can then rank the candidates c ′ using sim(q + , c ′ ). One can also perform multiple "hops" of attention over the profile rather than one, as shown here, although that did not bring significant gains in our parameter sweeps.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
The key-value (KV) memory network (Miller et al., 2016) was proposed as an improvement to the memory network by performing attention over keys and outputting the values (instead of the same keys as in the original), which can outperform memory networks dependent on the task and definition of the key-value pairs. Here, we apply this model to dialogue, and consider the keys as dialog histories (from the training set), and the values as the next dialogue utterances, i.e., the replies from the speaking partner. This allows the model to have a memory of past dialogues that it can directly use to help influence its prediction for the current conversation. The model we choose is identical to the profile memory network just described in the first hop over profiles, while in the second hop, q + is used to attend over the keys and output a weighted sum of values as before, producing q ++ . This is then used to rank the candidates c ′ using sim(q ++ , c ′ ) as before. As the set of (key-value) pairs is large this would make training very slow.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
In our experiments we simply trained the profile memory network and used the same weights from that model and applied this architecture at test time instead. Training the model directly would presumably give better results, however this heuristic already proved beneficial compared to the original network.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
The input sequence x is encoded by applying h e t = LST M enc (x t | h e t−1 ). We use GloVe (Pennington et al., 2014) for our word embeddings. The final hidden state, h e t , is fed into the decoder LST M dec as the initial state h d 0 . For each time step t, the decoder then produces the probability of a word j occurring in that place via the softmax, i.e., p(y t,j = 1 | y t−1 , . . . ,
Personalizing Dialogue Agents: I have a dog, do you have pets too?
.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
The model is trained via negative log likelihood. The basic model can be extended to include persona information, in which case we simply prepend it to the input sequence x, i.e., x = ∀p ∈ P || x, where || denotes concatenation. For the OpenSubtitles and Twitter datasets trained in Section 5.2 we found training a language model (LM), essentially just the decoder part of this model, worked better and we report that instead.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Finally, we introduce a generative model that encodes each of the profile entries as individual memory representations in a memory network. As before, the dialogue history is encoded via LST M enc , the final state of which is used as the initial hidden state of the decoder. Each entry p i = p i,1 , . . . , p i,n ∈ P is then encoded via f (p i ) = |p i | j α i p i,j . That is, we weight words by their inverse term frequency: α i = 1/(1 + log(1 + tf)) where tf is computed from the GloVe index via Zipf's law 4 . Let F be the set of encoded memories. The decoder now attends over the encoded profile entries, i.e., we compute the mask a t , context c t and next inputx t as:
Personalizing Dialogue Agents: I have a dog, do you have pets too?
If the model has no profile information, and hence no memory, it becomes equivalent to the Seq2Seq model.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
We first report results using automated evaluation metrics, and subsequently perform an extrinsic evaluation where crowdsourced workers perform a human evaluation of our models.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
The main results are reported in Table 3. Overall, the results show the following key points: Persona Conditioning Most models improve significantly when conditioning prediction on their own persona at least for the original (non-revised) versions, which is an easier task than the revised ones which have no word overlap. For example, the Profile Memory generation model has improved perplexity and hits@1 compared to Seq2Seq, and all the ranking algorithms (IR baseline, Starspace and Profile Memory Networks) obtain improved hits@1.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Ranking vs. Generative. Ranking models are far better than generative models at ranking. This is perhaps obvious as that is the metric they are optimizing, but still the performance difference is quite stark. It may be that the word-based probability which generative models use works well, but is not calibrated well enough to give a sentencebased probability which ranking requires. Human evaluation is also used to compare these methods, which we perform in Sec. 5.2.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Ranking Models. For the ranking models, the IR baseline is outperformed by Starspace due to its learnt similarity metric, which in turn is outperformed by Profile Memory networks due to the attention mechanism over the profiles (as all other parts of the models are the same). Finally KV Profile Memory networks outperform Profile Memory Networks in the no persona case due to the ability   Table 4: Human Evaluation of various PERSONA-CHAT models, along with a comparison to human performance, and Twitter and OpenSubtitles based models (last 4 rows), standard deviation in parenthesis.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
to consider neighboring dialogue history and next utterance pairs in the training set that are similar to the current dialogue, however when using persona information the performance is similar. Revised Personas. Revised personas are much harder to use. We do however still see some gain for the Profile Memory networks compared to none (0.354 vs. 0.318 hits@1). We also tried two variants of training: with the original personas in the training set or the revised ones, a comparison of which is shown in Table 6 of the Appendix. Training on revised personas helps, both for test examples that are in original form or revised form, likely due to the model be forced to learn more than simple word overlap, forcing the model to generalize more (i.e., learn semantic similarity of differing phrases).
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Their Persona. We can also condition a model on the other speaker's persona, or both personas at once, the results of which are in Tables 5 and 6 in the Appendix. Using "Their persona" has less impact on this dataset. We believe this is because most speakers tend to focus on themselves when it comes to their interests. It would be interesting how often this is the case in other datasets. Certainly this is skewed by the particular instructions one could give to the crowdworkers. For example if we gave the instructions "try not to talk about yourself, but about the other's interests' likely these metrics would change.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
As automated metrics are notoriously poor for evaluating dialogue (Liu et al., 2016) we also perform human evaluation using crowdsourced workers. The procedure is as follows. We perform almost exactly the same setup as in the dataset collection process itself as in Section 3.3. In that setup, we paired two Turkers and assigned them each a random (original) persona from the collected pool, and asked them to chat. Here, from the Turker's point of view everything looks the same except instead of being paired with a Turker they are paired with one of our models instead (they do not know this). In this setting, for both the Turker and the model, the personas come from the test set pool.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
After the dialogue, we then ask the Turker some additional questions in order to evaluate the quality of the model. We ask them to evaluate fluency, engagingness and consistency (scored between 1-5). Finally, we measure the ability to detect the other speaker's profile by displaying two possible profiles, and ask which is more likely to be the profile of the person the Turker just spoke to. More details of these measures are given in the Appendix.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
The results are reported in Table 4 for the best performing generative and ranking models, in both the No Persona and Self Persona categories, 100 dialogues each. We also evaluate the scores of human performance by replacing the chatbot with a human (another Turker). This effectively gives us upper bound scores which we can aim for with our models. Finally, and importantly, we compare our models trained on PERSONA-CHAT with chit-chat models trained with the Twitter and OpenSubtitles datasets (2009 and 2018 versions) instead, following Vinyals and Le (2015). Example chats from a few of the models are shown in the Appendix in Tables 7, 8 , 9, 10, 11 and 12. Firstly, we see a difference in fluency, engagingness and consistency between all PERSONA-CHAT models and the models trained on OpenSubtitles and Twitter. PERSONA-CHAT is a resource that is particularly strong at providing training data for the beginning of conversations, when the two speakers do not know each other, focusing on asking and answering questions, in contrast to other resources. We also see suggestions of more subtle differences between the models, although these differences are obscured by the high variance of the human raters' evaluations. For example, in both the generative and ranking model cases, models endowed with a persona can be detected by the human conversation partner, as evidenced by the persona detection accuracies, whilst maintaining fluency and consistency compared to their nonpersona driven counterparts.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Finding the balance between fluency, engagement, consistency, and a persistent persona remains a strong challenge for future research.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Two tasks could naturally be considered using PERSONACHAT: (1) next utterance prediction during dialogue, and (2) profile prediction given dialogue history. The main study of this work has been Task 1, where we have shown the use of profile information. Task 2, however, can be used to extract such information. While a full study is beyond the scope of this paper, we conducted some preliminary experiments, the details of which are in Appendix D. They show (i) human speaker's profiles can be predicted from their dialogue with high accuracy (94.3%, similar to human performance in Table 4) or even from the model's dialogue (23% with KV Profile Memory) showing the model is paying attention to the human's interests. Further, the accuracies clearly improve with further dialogue, as shown in Table 14. Combining Task 1 and Task 2 into a full system is an exciting area of future research.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
In this work we have introduced the PERSONA-CHAT dataset, which consists of crowd-sourced dialogues where each participant plays the part of an assigned persona; and each (crowd-sourced) persona has a word-distinct paraphrase. We test various baseline models on this dataset, and show that models that have access to their own personas in addition to the state of the dialogue are scored as more consistent by annotators, although not more engaging. On the other hand, we show that models trained on PERSONA-CHAT (with or without personas) are more engaging than models trained on dialogue from other resources (movies, Twitter).
Personalizing Dialogue Agents: I have a dog, do you have pets too?
We believe PERSONA-CHAT will be a useful resource for training components of future dialogue systems. Because we have paired human generated profiles and conversations, the data aids the construction of agents that have consistent per-sonalities and viewpoints. Furthermore, predicting the profiles from a conversation moves chitchat tasks in the direction of goal-directed dialogue, which has metrics for success. Because we collect paraphrases of the profiles, they cannot be trivially matched; indeed, we believe the original and rephrased profiles are interesting as a semantic similarity dataset in their own right. We hope that the data will aid training agents that can ask questions about users' profiles, remember the answers, and use them naturally in conversation.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
In Table 5 and Table 6 we show additional results on next utterance prediction. In particular we give results for conditioning on the other person's persona ("Their") instead of one's own ("Self") or the combination ("Both"). In particular, we see that, knowing the other's persona does not help precision on this data with these models. Finally, we show in Table 6 for the ranking models the performance difference from training on original personas versus revised ones. Revised personas give improved results, perhaps because the models are forced to learn more than trivial word overlap (i.e. semantic similarity of differing phrases).
Personalizing Dialogue Agents: I have a dog, do you have pets too?
In Tables 7, 8 , 9, 11, 12 we show example dialogues between Turkers and various models that were collected as part of the human evaluation of Section 5.2.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
After dialogues between humans and a model, we then ask the Turker some additional questions in order to evaluate the quality of the model. They are, in order:
Personalizing Dialogue Agents: I have a dog, do you have pets too?
• Fluency: We ask them to judge the fluency of the other speaker as a score from 1 to 5, where 1 is "not fluent at all", 5 is "extremely fluent", and 3 is "OK".
Personalizing Dialogue Agents: I have a dog, do you have pets too?
• Engagingness: We ask them to judge the engagingness of the other speaker disregarding fluency from 1-5, where 1 is "not engaging at all", 5 is "extremely engaging", and 3 is "OK".
Personalizing Dialogue Agents: I have a dog, do you have pets too?
• Consistency: We ask them to judge the consistency of the persona of the other speaker, where we give the example that "I have a dog" followed by "I have no pets" is not consistent. The score is again from 1-5.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
• Profile Detection: Finally, we display two possible profiles, and ask which is more likely to be the profile of the person the Turker just spoke to. One profile is chosen at random, and the other is the true persona given to the model.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
While the main study of this work is the ability to improve next utterance classification by conditioning on a persona, one could naturally consider two tasks: (1) next utterance prediction during dialogue, and (2) profile prediction given dialogue history. In the main paper we show that Task 1 can be improved by using profile information. Task 2, however, can be used to extract such information.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
In this section we conduct a preliminary study of the ability to predict the persona of a speaker given a set of dialogue utterances. We consider the dialogues between humans (PERSON 0) and our best performing model, the retrieval-based Key-Value Profile Memory Network (PERSON 1) from Section 5.2. We tested the ability to predict the profile information of the two speakers from the dialogue utterances of each speaker, considering all four combinations. We employ the same IR baseline model used in the main paper to predict profiles: it ranks profile candidates, either at the entire profile level (considering all the sentences that make up the profile as a bag) or at the sentence level (each sentence individually). We consider 100 negative profile candidates for each positive profile, and compute the error rate of predicting the true profile averaged over all dialogues and candidates. The results are given in Table 13, both for the model conditioned on profile information, and the same KV Memory model that is not. The results indicate the following:
Personalizing Dialogue Agents: I have a dog, do you have pets too?
• It is possible to predict the humans profile from their dialogue utterances (PERSON 0, Profile 0) with high accuracy at both the profile and sentence level, independent of the model they speaking to.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
• Similarly the model's profile can be predicted with high accuracy from its utterances (PER-SON 1, Profile 1) when it is conditioned on the profile, otherwise this is chance level (w/o Profile).
Personalizing Dialogue Agents: I have a dog, do you have pets too?
• It is possible to predict the model's profile from the human's dialogue, but with a lower accuracy (PERSON 0, Profile 1) as long as the model is conditioned on its own profile. This indicates the human responds to the model's utterances and pays attention to the model's interests.  Table 6: Evaluation of dialog utterance prediction with ranking models using hits@1 in four settings: conditioned on the speakers persona ("self persona"), the dialogue partner's persona ("their persona"), both or none. The personas are either the original source given to Turkers to condition the dialogue, or the rewritten personas that do not have word overlap, explaining the poor performance of IR in that case.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
• Similarly, the human's profile can be predicted from the model's dialogue, but with lower accuracy. Interestingly, the model without profile conditioning is better at this, perhaps because it does not concentrate on talking about itself, and pays more attention to responding to the human's interests. There appears to be a tradeoff that needs to be explored and understood here.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
We also study the performance of profile prediction as the dialogue progresses, by computing error rates for dialogue lengths 1 to 8 (the longest length we consider in this work). The results, given in Table 14, show the error rate of predicting the persona decreases in all cases as dialogue length increases.
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Overall, the results in this section show that it is plausible to predict profiles given dialogue utterances, which is an important extraction task. Note that better results could likely be achieved with more sophisticated models.   Table 13: Profile Prediction. Error rates are given for predicting either the persona of speaker 0 (Profile 0) or of speaker 1 (Profile 1) given the dialogue utterances of speaker 0 (PERSON 0) or speaker 1 (PERSON 1). This is shown for dialogues between humans (PERSON 0) and either the KV Profile Memory model ("KV Profile") which conditions on its own profile, or the KV Memory model ("KV w/o Profile") which does not.  Table 14: Profile Prediction By Dialog Length. Error rates are given for predicting either the persona of speaker 0 (Profile 0) or of speaker 1 (Profile 1) given the dialogue utterances of speaker 0 (PERSON 0) or speaker 1 (PERSON 1). This is shown for dialogues between humans (PERSON 0) and the KV Profile Memory model averaged over the first N dialogue utterances from 100 conversations (where N is the "Dialogue Length"). The results show the accuracy of predicting the persona improves in all cases as dialogue length increases.
Profile Consistency Identification for Open-domain Dialogue Agents
Despite the recent advancements in assigning attribute profiles to dialogue agents (Qian et al., 2018;, maintaining a consistent profile is still challenging for an open-domain dialogue agent. Existing works mainly emphasize the incorporation of attribute information in the generated responses (Wolf et al., 2019;Song et al., 2019a;Zheng et al., 2020). Although these models have improved the response consistency by explicitly modeling the profiles, they still face the consistency issue (Welleck et al., 2019). One important reason is that they cannot identify the consistency relations between response and profile.
Profile Consistency Identification for Open-domain Dialogue Agents
As shown in Figure 1, the attribute word Beijing is incorporated in the first two responses, but only R 1 is semantically consistent with the speaker's profile. For example, R 2 "I also hope to visit Beijing one day." implies that the speaker has never Figure 1: Left: the key-value attribute profiles of the dialogue agent. Right: a dialogue query with different responses that might be related to the attribute profiles. Among these responses, R 1 entails the current location profile, while R 2 contradicts the profile. Although R 3 does not contain the attribute word Beijing, we could still understand R 3 entails the current location. been to Beijing, which contradicts the speaker's profile. On the other hand, although R 3 does not contain the attribute word Beijing, we could still infer from the words Tsinghua University that the speaker's current location entails the profile. Existing studies (Qian et al., 2018;Zheng et al., 2019) train dialogue agents to produce plausible responses that contain attribute information, but still cannot teach agents to understand the differences of consistency relations in these responses. Welleck et al. (2019) made an early step towards reducing the dialogue consistency identification to natural language inference (NLI) (Bowman et al., 2015), where they learn a mapping from two dialogue utterances to an entailment category. All utterances in Welleck et al. (2019) are natural sentences from the PersonaChat dataset (Zhang et al., 2018). However, structured attribute profiles, such as key-value pairs, are ubiquitous in real-world dialogue systems (Shum et al., 2018). Compared with natural sentences, structured profiles have fixed attribute keys from different domains and specific attribute values from limited candidates. The structure information is also essential to a better understanding of the profile. To endow agents with the ability to identify structured profile consistency, we need a new dataset with fine-grained labels between response and profile, as well as a model that can leverage the structure information in the profile.
Profile Consistency Identification for Open-domain Dialogue Agents
In this work, we introduce a human-annotated dataset, named Key-value Profile Identification (KvPI), with over 110K single-turn conversations and corresponding attribute profiles. Three representative domains, gender, location, and constellation, are involved in the human annotation. We hire an annotation team to (1) label the relation (entailed, contradicted, or irrelevant) between each conversation and structured profile, and (2) find out the detailed attribute information in each response.
Profile Consistency Identification for Open-domain Dialogue Agents
With the annotated KvPI dataset, we set up different baseline models, and propose a key-value structure information enriched BERT (KvBERT) model, which leverages dependency structures in profiles to enrich the contextual representations. Experimental results show that KvBERT obtains significant improvements over strong baselines. We further test the KvBERT model on two downstream tasks, including a reranking task (Welleck et al., 2019) and a consistency prediction task (Dziri et al., 2019). Evaluation results show that (1) the KvBERT reranking improves response consistency, and (2) the KvBERT consistency prediction has a good agreement with human annotation.
Profile Consistency Identification for Open-domain Dialogue Agents
Our contributions are summarized as below:
Profile Consistency Identification for Open-domain Dialogue Agents
• A KvPI dataset is introduced, which has over 110K fine-grained consistency annotations between responses and their key-value profiles.
Profile Consistency Identification for Open-domain Dialogue Agents
• A KvBERT model is proposed for consistency identification, which gained significant improvements over strong baselines.
Profile Consistency Identification for Open-domain Dialogue Agents
• Evaluations on downstream tasks show that the profile consistency identification model could be complementary to dialogue models.
Profile Consistency Identification for Open-domain Dialogue Agents
In this section, we describe the collection and annotation process of the KvPI dataset: (1) how we collect high-quality conversations and profiles;
Profile Consistency Identification for Open-domain Dialogue Agents
(2) how we define the consistency relations between responses and profiles; and (3) how we annotate consistency relations for the collected data.
Profile Consistency Identification for Open-domain Dialogue Agents
To study the profile consistency identification problem, we use data from Weibo 1 , a popular and plenti-1 https://en.wikipedia.org/wiki/Sina Weibo ful Chinese social media, in which people routinely respond to different posts and have publicly available profiles, such as gender and location. We follow the protocol of the previous profile-based dialogue dataset (Qian et al., 2018;Zheng et al., 2019) to collect Weibo post-response pairs, together with users' available profiles. Here we filter out overly long or short pairs and finally obtain a tuple pool that contains about 30 million tuples, which are in a {profile, post, response} format. Each profile includes three popular attributes: gender, location and constellation, and organized in a keyvalue format. For instance, {gender: female, location:Beijing, constellation: Aquarius}. This format is widely applied in real-world dialogue systems, such as Bowden et al. (2017), Shum et al. (2018), andPichl et al. (2018). Since our goal is to identify explicit consistency relations between response and profile, we filter out the tuples whose response has no profile-related information by employing a pre-trained classifier and heuristic rules. Finally, we obtain about 150K profile-related tuples after filtering.
Profile Consistency Identification for Open-domain Dialogue Agents
We define three types of consistency relation between the response and profile under the opendomain dialogue setting, which is different from the entailment categories in natural language inference (Bowman et al., 2015;Welleck et al., 2019):
Profile Consistency Identification for Open-domain Dialogue Agents
Entailed The response is exactly talking about the dialogue agent's attribute information, and the attribute is consistent with its key-value profile.
Profile Consistency Identification for Open-domain Dialogue Agents
Contradicted Although the response is talking about the dialogue agent's attribute information, it is contradicted to at least one of the given key-value pairs. For example, given the profile "{location: Beijing}", "I am in Seattle" is contradicted to the profile, while "She lives in Seattle" is not, because the latter is not talking about the dialogue agent's attribute.
Profile Consistency Identification for Open-domain Dialogue Agents
The response contains profile-related information, but the information does not reveal the dialogue agent's own attributes. As exemplified above, "She lives in Seattle" is irrelevant, rather than contradicted, to the dialogue agent's profile "{location: Beijing}". Another example is "I'm interested in the history of Beijing". Although there is the attribute word "Beijing", this response still does not reveal the dialogue agent's location.  
Profile Consistency Identification for Open-domain Dialogue Agents
The definitions in Sec 2.2 are also applied in the human annotation process. We hire an annotation team to (1) review whether the response is profilerelated, and (2) annotate the fine-grained information, including consistency labels, domains, and detailed attributes in each response. To ensure quality, each tuple is annotated by three people, and the annotation process lasts nearly four months.
Profile Consistency Identification for Open-domain Dialogue Agents
In the annotation process, about 10K tuples are filtered out due to no profile-related information in their responses, and we obtain 140K valid tuples with explicit annotations of consistency relation.
Profile Consistency Identification for Open-domain Dialogue Agents
To control the quality of the annotated dataset, we introduce different verification methods:
Profile Consistency Identification for Open-domain Dialogue Agents
First, in the annotation process, we review 200 randomly sampled tuples every 10,000 annotations. We assign a "gold" label to each tuple and then decided whether the whole annotation batch should be accepted or re-annotated according to the disagreement rate. With tolerance to the different understandings of the dialogue response, we set an empirical acceptance threshold of disagreement rate to 10%. For the majority of annotated batches, the disagreement rate varies from 3% to 7%.
Profile Consistency Identification for Open-domain Dialogue Agents
The second verification is conducted by paid annotators. Each consistency label is verified by two annotators. The tuples with a low inter-annotator agreement in their labels are directly discarded from the final dataset. Finally, we obtain 118,540 tuples in the KvPI dataset.
Profile Consistency Identification for Open-domain Dialogue Agents
From the final dataset, we randomly sampled 2,000 profile-response pairs to two new annotators. These pairs are also annotated as entailed, contradicted, and irrelevant, as in the completed annotation process. Following Bowman et al. (2015), we calculated the Fleiss' Kappa among the previous labels and two new labels and obtained a kappa of 0.857, which means almost perfect agreement (Landis and Koch, 1977). This result shows that the completed annotation is of good quality.
Profile Consistency Identification for Open-domain Dialogue Agents
We present some examples of the final KvPI dataset in Table 1. The dataset, together with trained models, will be open-sourced for public usage.
Profile Consistency Identification for Open-domain Dialogue Agents
The KvPI dataset consists of single-turn conversations and profiles, labeled as entailed, contradicted, or irrelevant. Attributes in the dataset profiles come from three domains, including gender, location, and constellation. The profile is organized in a key-value format, for example, {gender: female, location: Beijing, constellation: Leo}.
Profile Consistency Identification for Open-domain Dialogue Agents
Gender This domain includes responses that have evidence indicating they are from men or women. Both explicit gender evidence, such as  "I am a girl", and implicit gender evidence, such as "I'm hanging out with my boyfriend", are included.
Profile Consistency Identification for Open-domain Dialogue Agents
Location This domain includes responses talking about the locations. Besides the accurate matching of location, data in this domain also needs common sense reasoning, such as whether a city belongs to a province, as shown in the third example in Table 1.
Profile Consistency Identification for Open-domain Dialogue Agents
Constellation This domain includes different responses that talk about the constellation. A good number of the responses contain more than one constellation word. Both entailed and irrelevant cases in the KvPI dataset are directly obtained from the annotation results. To balance the number of cases in each relation, we collect the contradicted cases from two sources: (1) the annotated contradicted tuples, and (2) the rewritten entailed tuples. Possible reasons for the originally contradicted cases are that users may forget to update their profiles, or they are intended to present different information about themselves. Data from the first source accounts for about two-thirds of the total contradicted cases. The other part comes from entailed cases. Their profiles have been rewritten to different attributes, with a minimal edit-distance principle, so that they turn into contradicted. Cases from this source are treated as new data in the annotation process. Unqualified rewritten data is discarded. Table 2 summarizes the main statistics of the KvPI dataset. The first and third groups in Table 2 count the number of unique tuples in the dataset. Here a tuple refers to a group of data consisting of a keyvalue profile, a post, a dialogue response, as well as the corresponding domain, the annotated attribute, and the label of consistency relation. The tuple examples can be seen in Table 1. For the second group, it only calculates the average number of tokens in the dialogue responses.
Profile Consistency Identification for Open-domain Dialogue Agents
To equip dialogue agents with the ability to identify consistency, we need to build a profile consistency identification model. This model learns to identify the relation of {entailed, contradicted, irrele-vant} between a (profile, response) pair. Formally, our goal is to learn a mapping function F, and F(P, R) ∈ {e, c, i}, where P ={k 1 : v 1 , ..., k n : v n }, R = w 1 , w 2 , ..., w m . Here P denotes the keyvalue profile, and R denotes the response with m words. e, c, i denote the consistency relations.
Profile Consistency Identification for Open-domain Dialogue Agents
The main challenge of identifying profile consistency lies in how to model the key-value profiles effectively. Such structured profiles have a common dependency structure, which differs from the natural sentences. For example, from the profile { gender: female, location: Beijing, constellation: Leo}, we can clearly see three dependency relations: female → gender, Beijing → location, and Leo → constellation. Moreover, gender, location, and constellation will define the information in the kv-profile. Here we can see a hierarchical structure of the key-value profiles, as illustrated in Figure 2. More importantly, no matter how the values change, this structure will stay unchanged.
Profile Consistency Identification for Open-domain Dialogue Agents
Although large pre-trained models such as BERT implicitly capture dependency information more or less (Clark et al., 2019), we argue that such implicit syntactic information may not be enough to support a powerful contextual representation for reasoning on the highly structured key-value profiles, according to the meaningless dependency parsing results generated by BERT on the structured profiles.
Profile Consistency Identification for Open-domain Dialogue Agents
These observations motivate us to incorporate the explicit structure of profiles directly. To this end, we design the KvBERT, which integrates both language representation from BERT and structure representation from tree-LSTM (Zhu et al., 2015).    Figure 2 shows the overall framework of the KvBERT model. On the BERT side, we linearize the key-value pairs into a sequence and treating the responses as another sequence 2 . The input embedding is the sum of four embeddings, including an additional type embedding (Chen et al., 2020) to inform the model of different key-value pairs, as shown in Figure 2. Here we omit the well-known formulations of BERT (Devlin et al., 2019) for brevity. We can get a contextual representation for the linearized sequence through the BERT model.
Profile Consistency Identification for Open-domain Dialogue Agents
On the tree-LSTM side, the profiles are parsed to predefined structure, as discussed in Sec 4.2. An example of this structure can be seen in the red part of the Figure 2. In parallel, the responses are passed to a trained parser to fetch the dependency structure. Then the tree-LSTM encodes two structures to corresponding embeddings. Three operations are performed to aggregate information from two embeddings: element-wise multiplication, elementwise difference, and concatenation. The aggregated embedding is followed up by a linear layer to form the final structure representation.
Profile Consistency Identification for Open-domain Dialogue Agents
At last, the sentence representation and structure representation are concatenated to form the joint representation for the final linear output layer.
Profile Consistency Identification for Open-domain Dialogue Agents
In our model, the dependency structure for profiles is predefined, and for the response, it is obtained 2 Our data collection scheme ensures that all responses contain profile information, which frees the modeling of post. from a trained parser. To complete the structure in the profile, we add a special [KV] token on the top of the dependency structure of the profile. As a result, the [KV] token aggregates information from its child key-value nodes. In contrast, there is no universal dependency structure in the responses. To obtain the structures in the responses, we trained a parser on CDT5.0 (Chineses dependency treebank), achieving 90.72% and 88.38% unlabeled and labeled attachment score. All structure predictions are made in the data preprocessing stage.
Profile Consistency Identification for Open-domain Dialogue Agents
A tree-LSTM unit encodes multiple child units or multiple descendant units in a recursive process. Due to the length limit, we recommend readers to get the details from Zhu et al. (2015). For both the predefined structures and the parsed structures, we apply the same depth-first encoding strategy, from every leaf node to the root node, to aggregate the structure information.
Profile Consistency Identification for Open-domain Dialogue Agents
In this section, we first evaluate the performance of the proposed KvBERT model on identifying profile consistency. After that, we test the trained KvBERT model on two downstream tasks, including a reranking task and a consistency prediction task, to analyze how well the proposed approach performs under practical applications.
Profile Consistency Identification for Open-domain Dialogue Agents
In our experiments, we train the KvBERT based on the 12-layer BERT-Base-Chinese model, with an embedding and hidden dimension of 768. For the tree-LSTM, we set embedding size to 300 and  output dimension to 50. The dimension of the final representation is 818. The tree-LSTM is firstly pretrained on the KvPI dataset for 13 epochs and then jointly finetuned with BERT representations for 3 epochs. The KvBERT model is implemented in PyTorch. More setting details are in the appendix.
Profile Consistency Identification for Open-domain Dialogue Agents
We compare the performance of a variety of baseline models on identifying profile consistency:
Profile Consistency Identification for Open-domain Dialogue Agents
Feature-based classifier Our goal of setting this baseline was to better understand the difficulty of identifying profile consistency, rather than necessarily a state-of-the-art model. Here we choose SVM as the classifier, with unigram features and bigram features, i.e., SVM+uni+bi. Additionally, the overlaps between profile values and responses are extracted as another feature, which is the SVM+uni+bi+overlap.
Profile Consistency Identification for Open-domain Dialogue Agents
Rnn-based NLI model ESIM (Chen et al., 2017) is a powerful natural language inference model, which enhanced the interactions in the LSTM. This model was applied in Welleck et al. (2019) and achieved the best results. Therefore, we set ESIM as the rnn baseline for our experiments.
Profile Consistency Identification for Open-domain Dialogue Agents
Pretrained models Large pre-trained transformers have been shown effective for natural language understanding tasks. We choose the Generative Pre-trained Transformer, i.e. GPT (Radford et al., 2018), and Bidirectional Encoder Representations from Transformers, i.e. BERT (Devlin et al., 2019) as our pre-trained baselines. Chen et al. (2020) proposed a TableBERT model, which models structured table information within the BERT framework. We take this model as another pre-trained baseline. We did not explore other pre-trained models in this work, due to the expensive computational costs in preparing their Chinese models. We leave the exploration as future work.
Profile Consistency Identification for Open-domain Dialogue Agents
Considering the previous works are designed for natural sentences, for the sake of a fair and thorough comparison, we use templates to convert the key-value profiles into natural sentences. The methods experimented on the converted dataset is marked by a suffix "-template". And the comparative experiments on the original KvPI dataset are marked by "-kv", which linearizes the original key-value profiles, the same as Sec 4.3. Other models are directly evaluated on the original KvPI dataset.
Profile Consistency Identification for Open-domain Dialogue Agents
For evaluations, despite the whole dataset that includes all three domains, we are also interested in the model's performance on each individual domain 3 . We use accuracy (acc), which has been widely applied in the natural language inference tasks, to measure the overall performance on each domain. To have a better look at the model's ability on identifying different consistency relations, we also calculate the f1-score of three relations under the same domain, i.e., entail-f1, contr-f1, and irrelv-f1. The accuracy and f1-score are calculated by using toolkits from sklearn.
Profile Consistency Identification for Open-domain Dialogue Agents
We report the averaged best results of three different runs on each domain in Table 3. With the explicit modeling of profile structures, our KvBERT achieves the best performance on all metrics across all domains. More importantly, KvBERT is the only model whose all metrics are over 90% on the KvPI test set, especially compared with strong pretrained baselines. Moreover, we also obtain 3.1% absolute improvements on the overall accuracy to the latest TableBERT model (Chen et al., 2020).
Profile Consistency Identification for Open-domain Dialogue Agents
We noticed an interesting phenomenon between the BERT-kv and BERT-template: the performance of BERT-template on all three individual domains are better than the BERT-kv's. Nevertheless, on the overall test set, their performances are entirely reversed. One possible reason is that the converted profile loses the structure information. Even for the powerful BERT model, this kind of information still affects the overall performance.
Profile Consistency Identification for Open-domain Dialogue Agents
Now that the KvBERT achieves good performance on the KvPI dataset, we want to test the abilities of the proposed approach further. Similar to the evaluations of pre-trained language models, we evaluate the abilities of our trained KvBERT model on two downstream tasks, with the assistance of human annotation.
Profile Consistency Identification for Open-domain Dialogue Agents
Here we consider two types of dialogue models, i.e., retrieval model and generation model. We test the KvBERT on two tasks: (1) Reranking the top 20 responses from a retrieval model, to see whether the profile consistency is improved (Welleck et al., 2019).
Profile Consistency Identification for Open-domain Dialogue Agents
(2) Given the responses from state-of-theart generative dialogue models, to see how well the KvBERT's consistency prediction agrees with the 3 Models on each domain are trained separately.
Profile Consistency Identification for Open-domain Dialogue Agents
Entail (  human annotation (Dziri et al., 2019). To build the testbeds of different dialogue models, we use the Chinese PersonalDialog (Zheng et al., 2019) dataset, which consists of over 20 million dialogues from Weibo, together with diversified profile traits and interests tags of the user.
Profile Consistency Identification for Open-domain Dialogue Agents
Further, we manually create 100 test samples for each domain, and we abbreviate the test set in this section as Gen (gender), Loc (location), and Con (constellation). Thus there are 300 test samples in total. Each test sample consists of a (profile, post) pair, where the attribute keys are the same as in the KvPI dataset. Moreover, we confirm that these posts will lead to domain-specific responses.
Profile Consistency Identification for Open-domain Dialogue Agents
We build the retrieval model using pylucene. To retrieve responses, we index both profiles and responses in the PersonalDialog dataset, with weights 0.15 and 0.85 for the profile and response, respectively. We retrieve the top 20 candidate responses for each testing sample, and then these responses are reranked by the trained KvBERT model, according to the order Entailed >Irrelevant >Contradicted. Within the same category, the model confidence will determine the order. Among the 20 responses from one test sample, the top 5 responses, both before and after reranking, are annotated by three people into entailed (Entail), contradicted (Contr), and irrelevant (Irrelv).
Profile Consistency Identification for Open-domain Dialogue Agents
We report the statistics of annotation results in Table 4 and show some reranking examples in the appendix. Besides the entailed responses, the irrelevant ones are more acceptable than the contradicted ones. As we can see, the KvBERT reranking improves profile consistency, either by increasing the rate of entailment or by decreasing the rate of contradiction. The annotation results also concur with our intuition: selecting a proper response with the right location is difficult for the retrieval models.
Profile Consistency Identification for Open-domain Dialogue Agents
Location ent-f1 con-f1 irr-f1 κ ent-f1 con-f1 irr-f1 κ ent-f1 con-f1 irr-f1 κ  
Profile Consistency Identification for Open-domain Dialogue Agents
In this task, we want to test how well the KvBERT's consistency prediction agrees with the human annotation on generated responses. We implement two state-of-the-art profile-based dialogue generation models as the testbeds for this task, including the TransferTransfo (Wolf et al., 2019) (TT) and Atten-tionRouting (Zheng et al., 2020) (AR). Both models are based on pre-trained transformers. First, we pre-train two models on 4G Chinese news data and finetune them on the PersonalDialog dataset. Then we use the trained models to generate responses on the test data Gen, Con, Loc, respectively. The collected responses are annotated into entailed, contradicted, and irrelevant by three annotators. The annotation instructions are the same as in Sec 2.2. In parallel, the KvBERT also predicts the relations between each profile and response.
Profile Consistency Identification for Open-domain Dialogue Agents
We first report the f1-score of model prediction against the human annotation in Table 5. We also report Cohen's Kappa (Cohen, 1960) between human annotations and model prediction to measure their agreements directly. All metrics are calculated by sklearn. From the f1-scores, we can see that the model predictions are similar to the human annotations in most cases. And the κ coefficients show the good agreements more directly, where κ between 0.6 and 0.8 indicates substantial agreement, and over 0.8 indicates almost perfect agreement (Landis and Koch, 1977).
Profile Consistency Identification for Open-domain Dialogue Agents
Responses from the generative models are in a different distribution from the training data, due to the model learning process. Still, the KvBERT obtains good agreements with humans. It shows the good generalization ability of the proposed method.
Profile Consistency Identification for Open-domain Dialogue Agents
Another important question is whether the structure information is always helpful. To analyze this, we sampled 9 treeLSTM checkpoints, with accuracy on the KvPI test set from 13.4% to 83.4%. The accuracy could be an indicator of how well the structure information has been captured. Then we Figure 3: The red dashed line in the horizontal direction is the TableBERT accuracy, which has no structural information. The depicted curve is fitted by a seventhdegree polynomial.
Profile Consistency Identification for Open-domain Dialogue Agents
trained 9 different KvBERT models with initialization from the 9 treeLSTMs and get final accuracies on the KvPI test set. We depict the treeLSTM accuracy and KvBERT accuracy, as well as a seventhdegree polynomial curve fitting the 9 data points, in Figure 3. And there is a performance baseline shown by the dashed horizontal line, which has no structure information.
Profile Consistency Identification for Open-domain Dialogue Agents
As we can see, not all the structural information contributes to the final performance. When the treeLSTM is at a low accuracy, the performance of the KvBERT model is inferior to that of the baseline model. Especially when the accuracy of treeL-STM is lower than 30%, the final performance is even getting worse when the accuracy of treeLSTM grows. And only when the accuracy of treeLSTM is higher than about 80%, can the final performance be improved, as illustrated in Figure 3.
Profile Consistency Identification for Open-domain Dialogue Agents
The code, data, and trained model are available at https://github.com/songhaoyu/KvPI.
Profile Consistency Identification for Open-domain Dialogue Agents
In this work, we introduce a large-scale annotated dataset to facilitate the study of profile consistency identification in open-domain dialogues. We leverage the structure information in profiles to enrich the BERT representations and obtain significant performance improvements over strong baselines. We further test the proposed method on two downstream tasks. Evaluation results show the effectiveness of the proposed approach.
Profile Consistency Identification for Open-domain Dialogue Agents
We believe KvPI will be a useful resource for the research of open-domain dialogue consistency. Although there has been a lot of dialogue generation models in this field, most of them still can't understand the consistency relationship in the generation process. One of the major bottlenecks is the lack of data. Because the KvPI dataset has paired key-value profiles and dialogues, it can also be a high-quality resource for personalized dialogue generation tasks. Furthermore, because we have fine-grained consistency labels, this dataset also provides an opportunity to leverage natural language understanding models to assist dialogue generation models. We hope that the data will aid training dialogue agents to be more consistent.
You Impress Me: Dialogue Generation via Mutual Persona Perception
Thanks to the advance in neural models and the accessibility of massive datasets, open-domain dialogue (i.e. chit-chat) systems have made great progress towards mimicking human-like responses. Nevertheless, there still exist some serious challenges in building personalized chatbots that can deliver engaging conversations and gain user trust (Song et al., 2019). For example, current chit-chat systems tend to generate uninformative responses (Li et al., 2016b). Moreover, they are usually lack of coherent personality traits due to the fact that training dialogues actually come from a diverse set of speakers (Zhang et al., 2018b). * Work done during an internship at Microsoft Research.  Several attempts have been made to alleviate the above issues. Methods like special reward shaping to reduce generic responses (Li et al., 2016b) and representing the speakers with latent variables (Li et al., 2016a) were introduced to improve the engagingness of chit-chat systems. A more straightforward approach, which equips chit-chat systems with predefined personas, was proposed accompanied by a novel dataset, PERSONA-CHAT (Zhang et al., 2018b). Figure 1 shows a clipped dialogue from PERSONA-CHAT. Two interlocutors meet for the first time and are having a conversation in order to get to know each other. What makes PERSONA-CHAT unique is that personas of both interlocutors are explicitly described using several profile sentences, facilitating the training of chatbots with configurable and persistent personalities.
You Impress Me: Dialogue Generation via Mutual Persona Perception
PERSONA-CHAT has fueled a growing interest in developing methods for personalized dialogue  generation. Mazaré et al. (2018) incorporated additional data from Reddit to train the model. Wolf et al. (2019b) fine-tuned pretrained language model (Radford et al., 2018) to improve the dialogue generation. Although both works demonstrate promising results, they focus more on mimicking the style of human-like responses, leaving understudied the aspects of explicitly modeling understanding between interlocutors. Our work, instead, takes the perspective of understanding modeling.
You Impress Me: Dialogue Generation via Mutual Persona Perception
According to the research in cognitive science, effective communication creates similar activation maps in the brains of both interlocutors (Hasson et al., 2012), suggesting that understanding between interlocutors is an essential signal for a highquality chit-chat conversation. For instance, in the conversation shown in Figure 1, the two interlocutors foster understanding either by raising personarelated topics, "Seen any good movies lately?", or by revealing their own personas through answering questions, "I don't watch movies more of a writer.". The efforts to build understanding keep the conversation flowing.
You Impress Me: Dialogue Generation via Mutual Persona Perception
Taking into account the above, we propose Persona Perception Bot (P 2 BOT), explicitly modeling the understanding between interlocutors with a transmitter-receiver framework. Distinguished from traditional methods, P 2 BOT highlights a novel concept, mutual persona perception, which is better suited to describe the information exchange process that empowers the interlocutors to get to know each other. In order to train P 2 BOT for personalized dialogue generation, we employ supervised training and self-play fine-tuning piloted by reward signals characterizing mutual persona perception. Experiments on the PERSONA-CHAT dataset demonstrate the superiority of our approach over the baselines in both automatic met-rics and human evaluations 1 .
You Impress Me: Dialogue Generation via Mutual Persona Perception
The central idea of P 2 BOT is to explicitly model understanding between interlocutors and enhance dialogue generation via mutual persona perception. It comprises two components, Transmitter and Receiver, respectively responsible for dialogue generation and mutual persona perception. Figure 2 gives an overview of P 2 BOT: interlocutor A has a persona w A , described with L profile sentences {w A 1 , · · · , w A L }. When she first meets the other interlocutor B, they are going to know each other through a N -turn dialogue (
You Impress Me: Dialogue Generation via Mutual Persona Perception
n denotes the utterance that A says in nth turn and N denotes the number of total turns. Given the entire dialogue history up to n-th turn h A n = (x A 1 , · · · , x B n−1 ), Transmitter generates x A n according to the distribution p(x A n | w A , h A n ), and transmits it to B. The same process applies to B, keeping the conversation flowing.
You Impress Me: Dialogue Generation via Mutual Persona Perception
As the conversation goes on, impressions are gradually built via utterances. For example, when A says "I don't watch movies more of a writer.", the impression that "A is a writer." is left on B's mind. As mentioned above, a successful conversation helps interlocutors know each other, which means B's impression of A should correspond to A's persona and vice versa. Receiver aims to measure the proximity between the built impressions and the actual personas. Specifically, as demonstrated by the dashed black lines in Figure 2, Receiver first projects impressions and personas into a latent space, and then measures the relevance between them based on the impression encoding (e.g. H A , B's impression on A, projected from A's  Figure 3: The overall architecture of Transmitter. "Block" is short for "Transformer Block". Arrows bridge the current block to subsequent blocks of its following layer. Position encoding is to incorporate position information into block by assigning an embedding for each absolute position in the sequence. Here we omit the architecture inside the block, and refer the readers to Vaswani et al. (2017) for more details.
You Impress Me: Dialogue Generation via Mutual Persona Perception
[MASK] tokens are ignored in the training objective. utterances x A ), and persona encoding (e.g. W A , projected from A's persona w A ) 2 . The relevance scores serve as mutual persona perception rewards, and are further incorporated into the training of Transmitter. Details of the two components are presented in Section 3 and 4.
You Impress Me: Dialogue Generation via Mutual Persona Perception
Following previous work (Li et al., 2016b;Zhang et al., 2018b), we treat dialogue generation as a sequence generation problem. Concretely, we employ the pretraining transformer language model introduced in Radford et al. (2018) (i.e. GPT) to initialize Transmitter. The entire training procedure consists of two steps: (1) Supervised Dialogue Generation. We optimize Transmitter via maximum likelihood estimation (MLE) on the supervised dialogue generation task. (2) Self-play Model Finetuning. We simulate dialogues between two randomly paired interlocutors, encouraging Transmitter to learn a policy that maximizes reward signals via reinforcement learning (RL) (Sutton et al., 1999). The design of the reward function considers both language modeling and our proposed mutual persona perception.
You Impress Me: Dialogue Generation via Mutual Persona Perception
As illustrated in Figure 3, Transmitter follows the overall architecture of 12 stacked transformer layers to encode context and generate response. Here, the context contains the persona w A , the dialogue history h A n , and several special tokens (e.g.
You Impress Me: Dialogue Generation via Mutual Persona Perception
[PS] which indicates the start of persona). Given a training instance (w A , h A n , x A n ), the training objective of MLE is to maximize the conditional loglikelihood as:
You Impress Me: Dialogue Generation via Mutual Persona Perception
where θ is the parameter of Transmitter. x A n,t means the t-th token in x A n , and x A n,<t indicates the token sequence before t-th token. Equation 1, hereafter simplified as log p θ (x A n | w A , h A n ), applies to both A and B, and we mention A for the sake of brevity (the same as below).
You Impress Me: Dialogue Generation via Mutual Persona Perception
During inference, beam search is applied to store top-ranked response candidates {x A n }, and Transmitter subsequently chooses as prediction the one that maximizes the length-normalized score:
You Impress Me: Dialogue Generation via Mutual Persona Perception
Besides the sequence generation task, inspired by Wolf et al. (2019b), we set up an auxiliary task, Next Utterance Prediction. Apart from training Transmitter to generate responses, we also train it to discriminate whether the response is the next utterance of the given context. Concretely, we append a special token [CLS] to the tail of the generated tokens. A classifier is built on top of the token's hidden state in the last transformer layer, as indicated by the red rounded rectangle in Figure 3. In training, for each response, we randomly sample a distractor and train the classifier to give a higher score on the response than the distractor. In inference, the classifier is used to rank response candidates together with Equation 2. Denoting as y n = 1 the signal indicating the generated responsê x A n is predicted as the next utterance, Equation 2 is extended as:
You Impress Me: Dialogue Generation via Mutual Persona Perception
where α is a hyper-parameter.
You Impress Me: Dialogue Generation via Mutual Persona Perception
Although supervised dialogue generation alone can be used to mimic human-like responses, it does not inherently target at understanding. Therefore, we  Figure 4: The illustration of the self-play procedure. Arrows ⇒ represent the process of dialogue generation driven by Transmitter. Note that x A * 1 is directly taken from the dataset as it is difficult to generate high-quality utterances without any dialogue history.
You Impress Me: Dialogue Generation via Mutual Persona Perception
further fine-tune Transmitter using reinforcement learning with the goal of maximizing mutual persona perception. Analogous to Lewis et al. (2017), we apply self-play to simulate the communication between two Transmitters, both of which have been trained as described in Section 3.1.
You Impress Me: Dialogue Generation via Mutual Persona Perception
Specifically, we have the two Transmitters communicate with each other for several turns. One Transmitter serves as a user with the parameters frozen, while the other is a learnable agent. The parameter of the learnable agent, θ, is fine-tuned during the self-play. Without loss of generality, in our experiments, we let interlocutor A, who starts a conversation, be the user, and correspondingly B be the learnable agent.
You Impress Me: Dialogue Generation via Mutual Persona Perception
Here we introduce some necessary formulations for modeling our problem with reinforcement learning. A state contains the persona and the dialogue history. For example, the state for B at turn n is defined as s
You Impress Me: Dialogue Generation via Mutual Persona Perception
An action a B n is the response to be generated. The action space is infinitely large as the response can be arbitrary long. Taking s B n as input, the parameter θ defines a policy p θ (a B n |s B n ), through which the learnable agent generates its response.
You Impress Me: Dialogue Generation via Mutual Persona Perception
As illustrated in Figure 4, when it is B's turn to speak, B receives s B n and picks a B n according to the policy p θ . As for A, it receives s A n and generates the response x A * n to simulate a user. A and B alternately produce responses till the number of turns exceeds the given limit. Once a complete dialogue is generated, the reward is collected to optimize θ using policy gradient (Sutton et al., 1999). Denoting as R(a B n ) the reward B gets at turn n (more details are provided later), we can optimize it by maximizing the following objective:
You Impress Me: Dialogue Generation via Mutual Persona Perception
Applying likelihood ratio trick, θ is updated by ascending the following gradient:
You Impress Me: Dialogue Generation via Mutual Persona Perception
As aforementioned, the space of action a B n is infinite. In practice, REINFORCE algorithm (Williams, 1992) is leveraged to approximate Equation 5 by sampling a B n from policy p θ (a B n |s B n ). Furthermore, subtracting a baseline (Weaver and Tao, 2001), here the mean reward of a mini-batch, is applied on R(a B n ) to reduce variance. The agent samples tokens one by one through multinomial sampling over the output distribution of B, until the special token [EOS] is sampled or exceeding the maximum allowed decoding step (e.g. 32). Compared to beam search sampling, multinomial sampling provides more diversities.
You Impress Me: Dialogue Generation via Mutual Persona Perception
As described in Section 1, we believe that a highquality chit-chat conversation should highlight both human language modeling and mutual persona perception. Bearing this in mind, we design three rewards to address language style, discourse coherence and mutual persona perception respectively.
You Impress Me: Dialogue Generation via Mutual Persona Perception
The generated responses should conform to human language styles, which we believe can be evaluated by a pretrained language model (i.e. GPT). After length normalization, the score for a B n is given as:
You Impress Me: Dialogue Generation via Mutual Persona Perception
where a B n,t and a B n,<t have similar denotation as the previously mentioned x A n,t and x A n,<t .
You Impress Me: Dialogue Generation via Mutual Persona Perception
The language score is evaluated individually, without considering the discourse coherence. However, a reasonable response should establish links in meaning with context, which is also an important aspect of humanlike responses. To take into account the discourse coherence, we employ the well-trained Next Utterance Predictor (mentioned in Section 3.1). The reward is given by the log probability of a B n being the next utterance of s B n :
You Impress Me: Dialogue Generation via Mutual Persona Perception
R 2 (a B n ) = log p θ (y n = 1 | a B n , s B n ).
You Impress Me: Dialogue Generation via Mutual Persona Perception
RS.3 Mutual Persona Perception RS.1 and RS.2 only steer the agent training process towards human-like responding. They do not explicitly encourage understanding between interlocutors. Therefore, we meticulously design the reward to characterize mutual persona perception. Contrast from RS.1 and RS.2, mutual persona perception is a long-term goal throughout the whole dialogue, meaning that the effect of current action might only play out some time later. For instance, receiving "what are your hobbies?" from B, it is highly likely that A's response is relevant to A's hobbies. This suggests that, not only A's response but also B's initial question contributes to mutual persona perception. Denoting as γ the discount factor indicating how far ahead B looks, the reward of mutual persona perception for a B n is defined as:
You Impress Me: Dialogue Generation via Mutual Persona Perception
where r(a B n ) is the persona perception score that B obtains in n-th turn, and r(x A * k ) is defined likewise. r(a B n ) can be computed using a score function:
You Impress Me: Dialogue Generation via Mutual Persona Perception
In P 2 BOT, the score function comes from Receiver, which will be elaborated in Section 4. The final reward R(a B n ) for a B n is a weighted sum of the rewards listed above:
You Impress Me: Dialogue Generation via Mutual Persona Perception
where λ 1 , λ 2 and λ 3 are hyper-parameters.
You Impress Me: Dialogue Generation via Mutual Persona Perception
Receiver is devised to measure the proximity between the built impressions and the actual personas, implemented by negative sampling. Specifically, in training, we randomly sample a persona distractor w Z . Receiver is trained to identify the real persona w A from {w A , w Z }. In inference, for each utterance, Receiver is responsible for providing a reasonable relevance score, to model our proposed mutual persona perception. The score subsequently joins the self-play fine-tuning on Transmitter as part of the rewards, as in Equation 8. 
You Impress Me: Dialogue Generation via Mutual Persona Perception
As illustrated in Figure 5, Receiver contains two different encoders for impression and persona respectively. Initialized by BERT (Devlin et al., 2019), both encoders provide deep contextualized representations for each token. Then we average all the representations, yielding a fixed d-dimensional vector for one sentence. In this way, feeding (x A 1 , x A 2 , · · · , x A N ) into the impression encoder consecutively, we obtain the impression encoding H A ∈ R N ×d . The persona encoding W ∆ ∈ R L×d is produced likewise, where ∆ ∈ {A, Z}. The relevance score matrix U ∆ is computed via the scaled dot product (Vaswani et al., 2017):
You Impress Me: Dialogue Generation via Mutual Persona Perception
In essence, Receiver is expected to capture finegrained correlations between the persona and the dialogue. However, we do not have access to the golden fine-grained correlations. The only thing we know is that, compared with W Z , H A is more correlated to W A . Since the comparison is at a coarse granularity, we gather U ∆ into the cumulative score c ∆ through an aggregate function Agg, as shown in Figure 5. To encourage c A while at the same time depress c Z , we design a marginal loss L rec , which makes c A larger than c Z by a margin m. Moreover, considering that an utterance generally relates to zero or one profile, L 1 regularization is enforced to make U ∆ sparse. Combining all of these, the training loss for Receiver is:
You Impress Me: Dialogue Generation via Mutual Persona Perception
where β is a hyper-parameter for penalty.
You Impress Me: Dialogue Generation via Mutual Persona Perception
As for Agg, one straightforward way is to average over all positions of U ∆ . However, it maximizes every entry in U A , including all those that should not be activated (e.g. relevance scores between unrelated profile sentences and utterances), introducing unnecessary noise into the training of Transmitter. To alleviate the problem, we choose to implement Agg as a controllable weighted function, which summarizes U ∆ n,: as:
You Impress Me: Dialogue Generation via Mutual Persona Perception
where temperature τ > 0 is a tunable parameter (Hinton et al., 2015) controlling the evolution of Agg. In the beginning, Agg behaves close to average pooling. As τ anneals, Agg gradually focuses more on the highest relevance score. In this way, noise reduces as training goes on. Finally, c ∆ is given by:
You Impress Me: Dialogue Generation via Mutual Persona Perception
Agg(U ∆ n,: ).
You Impress Me: Dialogue Generation via Mutual Persona Perception
Given x A n and w A , Receiver employs the following function to obtain x A n 's persona perception score, further modeling mutual persona perception as in Equation 9:
You Impress Me: Dialogue Generation via Mutual Persona Perception
where H A n,: and W A are the impression encoding and persona encoding for x A n and w A respectively.
You Impress Me: Dialogue Generation via Mutual Persona Perception
We conducted experiments on the dataset PERSONA-CHAT, assessing P 2 BOT using both automatic metrics and human evaluations. To verify the effectiveness of our proposed mutual persona perception, we perform a thorough model analysis in Section 5.3. Finally, we probe Receiver's capability on perceiving persona in Section 5.4.
You Impress Me: Dialogue Generation via Mutual Persona Perception
PERSONA-CHAT dataset contains 8,939 / 1,000 multi-turn dialogues conditioned on 1,155 / 100 personas for train / dev. Each persona is described with at least 5 profile sentences. To make it more challenging, PERSONA-CHAT also provides revised personas by rephrasing, generalizing or specializing the original ones. For example, "I am overweight." is revised from "I weight 300 pounds.". Our implementation was based on PyTorch (Paszke et al., 2019), ParlAI (Miller et al., 2017), and HuggingFace's transformers library (Wolf et al., 2019a). We used Adam (Kingma and Ba, 2015) optimizer with a learning rate of 6.25e-5 for both Receiver and Transmitter in supervised learning. In the training of Receiver, τ reduced linearly from 10 to 0.5. In the self-play phase of Transmitter, the learning rate was set as 1e-6. The hyperparameters m, α, β, γ, λ 1 , λ 2 and λ 3 were set as 0.4, 0.1, 1e-4, 0.5, 0.4, 0.1 and 0.5 respectively. The supervised training of Transmitter lasted for 2 epochs, and the self-play fine-tuning comprised 2000 dialogues, where the number of turns was 3. The beam search size was set as 2.
You Impress Me: Dialogue Generation via Mutual Persona Perception
Our baselines fall into three categories: retrievalbased, generative-based and pretrain-finetunebased models. Among the retrieval-based baselines, KV Profile Memory (Zhang et al., 2018b) was the official baseline which employed the memory network along with profile information, and  Dually Interactive Matching Network (Gu et al., 2019) proposed a dual matching architecture to match between the responses and their corresponding contexts. Language Model, Generative Profile Memory (Zhang et al., 2018b) and SEQ2SEQ with attention mechanism (Bahdanau et al., 2015) were implemented as generative baselines for dialogue generation. The remaining methods were all pretrain-finetune-based. Transfertransfo (Wolf et al., 2019b) 3 achieved the state-of-the-art performance on automatic metrics, while Lost In Conversation 4 topped the human evaluations (Dinan et al., 2019). Analogous to our approach, they employed the pretrained language model GPT to initialize their models, and then fine-tuned it on the dataset. Table 1 shows the experimental results on automatic metrics. Following Zhang et al. (2018b), we reported the official automatic metrics to evaluate the methods: Hits@1, Perplexity (ppl) and F1. Given 20 response candidates, Hits@1 is the probability that the real response ranks the highest according to the model. Perplexity measures the negative log likelihood of the correct sequence output by the model, lower values indicating better performance. F1 is the harmonic mean of word-level precision and recall. As observed, our approach outperforms almost all baselines and achieves new state-of-the-art performance on ppl and F1, with highly competitive performance on Hits@1. In the revised mode, our approach still achieves the best performance, obtaining a relative improvement of 13.4% on F1 against the strongest baseline. It is worth noting that we also tried to employ F1 as the reward, but the result is far from satisfactory.
You Impress Me: Dialogue Generation via Mutual Persona Perception
As mentioned in Dinan et al. (2019), no automatic metric is perfect for evaluating such an opendomain task. Hence, we also performed crowdsourced human evaluations on the state-of-the-art baselines (i.e. Transfertransfo & Lost In Conversation) and our proposed P 2 BOT. Concretely, on the original dev set, we randomly sampled 200 responses generated by these methods and asked each worker to rate them. The rating ranges from 1 3 http://github.com/huggingface/transfer-learning-conv-ai 4 http://github.com/atselousov/transformer chatbot  Table 3: Variant analysis results on PERSONA-CHAT revised mode, along with relative improvements (shown inside brackets) compared with P 2 BOT-S. BLEU refers to the cumulative 4-gram BLEU score. "-Persona" means dialogue generation without personas; "-Next" ablates the auxiliary task mentioned in Section 3.1; "+ RS.1" means only using Language Style score as the reward in the self-play fine-tuning phase; " → + RS.2" means adding Discourse Coherence to the reward on the basis of RS.1; " → + RS.3" is equivalent to our proposed P 2 BOT.
You Impress Me: Dialogue Generation via Mutual Persona Perception
to 4. 1 means the response is good only in terms of grammar and sentence structure; 2 means in addition to valid grammar, the response is also coherent with the context; 3 means the coherent response is meanwhile interesting and informative, instead of just a simple response like "Yes"; And 4 means the response is consistent with the persona of the interlocutor, which is of extreme importance for the task of reflecting whether the model can effectively utilize the persona information. As shown in Table 2, the results are consistent with the automatic evaluation results, demonstrating the superiority of P 2 BOT against the baselines. We also conducted Wilcoxon signed-rank tests between our method and the baselines and the results show the improvements are significant with p < 0.05.
You Impress Me: Dialogue Generation via Mutual Persona Perception
Variant Analysis We conducted variant analysis on P 2 BOT to investigate the influence of RS.1, RS.2 and RS.3. Another metric BLEU (Papineni et al., 2002), which evaluates the quality of response, was introduced to make the analysis more comprehensive. We show the variant analysis results in Table 3   perception. Similar conclusions can be drawn from the trend of F1.
You Impress Me: Dialogue Generation via Mutual Persona Perception
Case Study For a more comprehensive comparison, we show in Table 4 some randomly sampled responses of different methods. The results suggest the responses generated by our approach are more human-like. As observed, benefiting from our proposed mutual persona perception, the responses of P 2 BOT are more consistent, engaging and informative. For instance, in the last example in Table 4, the response "I'm busy with my robot project" explicates why the speaker does not exercise, meanwhile revealing that he is working on the robot, as depicted in his persona.
You Impress Me: Dialogue Generation via Mutual Persona Perception
Error Analysis Though our approach works well in most cases, we observed that the self-play simulation might fall into repeated cycles after rounds of training, as the challenge mentioned by Li et al. (2016b). Another issue is that the bots sometimes ask redundant questions in our approach, which might be due to inappropriate hyperparameters in reward shaping.
You Impress Me: Dialogue Generation via Mutual Persona Perception
Receiver plays an important role in our approach, and we are interested in its capability on perceiving persona. Therefore, we conducted experi-  Figure 6: Visualization of the relevance scores between a sampled dialogue and its corresponding revised persona. Deeper color means higher score. We omit some context due to space limitation. ments on a synthesized dataset. We constructed the dataset by sampling 31 persona distractors for each dialogue in PERSONA-CHAT. Two widely used ranking metrics were used to evaluate the performance: Hits@1 and Mean Reciprocal Rank (MRR). Hits@1 is the same metric as the one mentioned in Section 5.2, except that the candidate size is 32. Given a dialogue and the complete set of profile sentences, MRR is the average reciprocal ranks of the dialogue-relevant profile sentences. Two simple baselines Random and IR (Sordoni et al., 2015) were chosen for comparison. Table 5 shows the experimental results of different methods on the synthesized dataset. As observed, our approach achieved excellent results on both original and revised modes. For example, compared with the IR baseline, our approach achieved an absolute improvement of 26.3% on Hits@1 in the original mode. In addition, the surprising results in the revised mode further demonstrate Receiver's capability to perceive rephrased persona.
You Impress Me: Dialogue Generation via Mutual Persona Perception
To further understand the trained Receiver, we visualize the relevance scores between a sampled dialogue and its corresponding revised persona in Figure 6. As illustrated, the relevance scores between related profile sentences and dialogue utterances are significantly higher. For example, the utterance "I volunteer at the local pool" from the interlocutor implies the profile "I love being in the water", and our Receiver successfully captures the relevance between them.
You Impress Me: Dialogue Generation via Mutual Persona Perception
We propose P 2 BOT, a transmitter-receiver framework which explicitly models understanding between interlocutors. Under this framework, mutual persona perception is incorporated as a reward signal to achieve the personalized dialogue generation. Experiments on a large public dataset PERSONA-CHAT demonstrate the effectiveness of our approach. For future work, we would like to extend Receiver to conversational recommender systems. After turns of chatting, the agent should be able to infer the user's persona, based on which personalized contents can be recommended.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
Dialogue generation is a challenging task due to the limited corpus of human conversations, complex background knowledge, and diverse relationships between utterances. Recently, pre-trained large-scale language models, such as BERT (Devlin et al., 2019) and XLNet , have achieved prominent success in natural language processing. Such models are usually constructed based on a massive scale of general text corpora, like English Wikipedia or BooksCorpus (Zhu et al., 2015), where distributed representations can be learned automatically from the raw text. With these representations being fine-tuned, breakthroughs have been continuously reported for various downstream tasks, especially those on natural language understanding, such as question answering, natural language inference, and so on. * First two authors contributed equally to this work. This pre-training and fine-tuning paradigm also sheds light on the tasks of natural language generation, like dialogue generation. However, the previous study demonstrates that there are some deficiencies in performance while directly fine-tuning BERT on small conversation datasets (Rashkin et al., 2019;. Possible reasons are three-fold: 1) the underlying linguistic patterns in human conversations can be highly different from those in general text, which suggests a potentially large gap in knowledge or data distribution; 2) the training mode of uni-directional dialogue generation is also distinct from that of bi-directional natural language understating as applied in BERT; 3) unlike most of the general NLP tasks, there exists a one-to-many relationship in dialogue generation, where the dialogue context may correspond to multiple appropriate replies.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
In this paper, we propose a new method to tackle the above challenges, aiming to obtain a highquality pre-training model for dialogue generation. First of all, to reduce the gap between data distributions, large-scale Reddit and Twitter conversations are utilized to further pre-train the generation model (upon the basis of language models pre-trained with general text). Secondly, to mitigate the difference in training mode, a flexible paradigm integrating uni-and bi-directional processing is employed in this work, which is inspired by the latest unified language modeling . Thirdly, a discrete latent variable is introduced to model the one-to-many relationship among utterances in conversations. Each value of the latent variable corresponds to the particular conversational intent of one response, which is referred as latent speech act.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
Distinct with those controllable dialogue generation based on explicit labels (including emotion, keywords, domain codes, and so on) (Huang et al., 2018;Keskar et al., 2019), our latent variable gets exempted from the restriction of human annotations and can be learned automatically from the corpus in an unsupervised way. In the pre-training of dialogue generation, response generation and latent act recognition are carried out simultaneously within a shared network. Based on the context and latent variable, the generation task tries to maximize the likelihood of the target response. Meanwhile, the recognition task aims to estimate the latent variable w.r.t. the given context and target response. Apparently, the accurate recognition of the latent variable is a crucial factor in boosting the quality of response generation.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
We conducted experiments on three different kinds of conversation tasks: chit-chat, knowledge grounded conversation, and conversational question answering. Experimental results verify the effectiveness and superiority of our pre-trained model as compared with the other state-of-the-art methods. Our pre-trained models and source code have been released at GitHub, hoping to facilitate further research progress in dialogue generation. 1 2 Dialogue Generation Pre-training Given a piece of context, there exist multiple appropriate responses, leading to diverse conversation flows. It is widely recognized that the capability of modeling one-to-many relationship is crucial for the dialogue generation system (Zhao et al., 2017;. To this end, we propose to encode discrete latent variables into transformer blocks for one-to-many relationship modeling, where two reciprocal tasks of response generation and latent act recognition are collaboratively carried out.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
In our model, there are three elements: dialogue context c, response r and latent variable z. The dialogue context c consists of several history utterances. (For knowledge grounded conversation, it is conventional to concatenate background knowledge into the context as well .) The response r is one piece of appropriate reply towards the given context. The latent variable z is one K-way categorical variable z ∈ [1, K], with each value corresponding to a particular latent speech act in the response.  The probabilistic relationships among these elements are elaborated with the graphical model in Figure 1. Given a context c, there are multiple latent speech acts which can be taken as response intents (represented by the latent variable z). Conditioned on the context and one selected latent speech act, the response is generated as p(r|c, z) (gray lines). Given a pair of context and response, the underlying latent speech act can be estimated as p(z|c, r) (dashed blue lines). As such, our pretraining of dialogue generation contains the following two tasks -response generation and latent act recognition.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
We propose a unified infrastructure for the joint learning of both tasks, shown as Figure 2. The backbone of our infrastructure is inspired by the transformer blocks in , which supports both bi-directional encoding and uni-directional decoding flexibly via specific selfattention masks. Both response generation and latent act recognition are carried out under the unified network with shared parameters. Their detailed implementations are described as follows.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
Given the context c and a specific speech act z, the response generation can be estimated as
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
where T is the length of the target response r and r <t denotes previously generated words. Since the response generation is a uni-directional decoding process, each token in the response only attends to those before it, shown as dashed orange lines in Figure 2. The latent act recognition task is included to identify the corresponding value of z for the given context and the target response in the training data. The latent act recognition shares network parameters with response generation, but has a separate self-attention mask for bi-directional encoding. As shown in Figure 2, with a special mask symbol [M] as input, it keeps collecting information from the context and target response (red lines). In this way, Figure 2: Architecture of dialogue generation with discrete latent variable. In self-attention visualization, red and blue lines denote bi-directional attention, and dashed orange lines denote uni-directional attention.  the corresponding speech act for the target response can be recognized as z ∼ p(z|c, r), where p(z|c, r) is the estimated posterior distribution over discrete latent values.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
For multi-turn conversation modeling, elaborate designs have been made on the input representation in this work. For each token, its input embedding is the sum of corresponding token, role, turn and position embeddings. One visual example is shown in Figure 3 and details are described in the following.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
• The input is the concatenation of latent variable, dialogue context and response. Following the pre-processing of BERT (Devlin et al., 2019), the input text is tokenized with WordPiece (Wu et al., 2016). A special end-of-utterance [EOU] token is appended to the end of each utterance for separation. Another begin-of-utterance [BOU] token is added at the beginning of the response, whose final hidden state (i.e., output of the last transformer block) is used to predict next token during generation. • Given that z is one K-way categorical variable, its token embedding E [z] is mapped from the latent embedding space E z ∈ R K×D . For the rest tokens in the vocabulary, they are initialized using BERT's WordPiece embeddings.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
• Role embeddings are employed to differentiate the characters evolved in the conversation. The role embedding E A is added for the response, as well as dialogue utterances generated by the same character in the context. And role embedding E B is used for the other character. (For knowledge grounded conversation, E C is used as the role embedding of background knowledge.) • In the interactive conversation, there are multiturn utterances and we employ relative order in the assignment of turn embeddings. The turn embedding for the response is set to E [0] , and the turn embedding of its last utterance is E [−1] , and etc. Our utilization of relative turn embeddings instead of absolute ones enables the model to assign turn embedding E [0] to the response consistently and makes response generation exempt from the disturbance of its round number within the dialogue. • Position embeddings are added according to the token position in each utterance. Note that for the special token of latent variable, its corresponding role, turn and position embeddings are all set to empty.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
We employ three loss functions in dialogue generation pre-training: negative log-likelihood (NLL) loss, bag-of-words (BOW) loss and response selection (RS) loss. Brief illustration is shown in the last column of Figure 2 and detailed descriptions will be provided in this section.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
In our model, the response is generated conditioned on the latent variable and the context. The widely adopted NLL loss is embraced in the pre-training:
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
where z is the latent speech act of this training pair (c, r), sampled from the probability distribution p(z|c, r). The posterior distribution over latent values is estimated through the task of latent act recognition:
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
where h [M ] ∈ R D is the final hidden state of the special mask, W 1 ∈ R K×D and b 1 ∈ R K denote the weight matrices of one fully-connected layer.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
Besides the classical NLL loss, the bag-of-words loss (Zhao et al., 2017) is also employed to facilitate the training process of latent discrete variables:
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
where V refers to the whole vocabulary. The function f tries to predict the words within the target response in a non-autoregressive way:
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
where h z is the final hidden state of the latent variable and |V | is the vocabulary size. f rt denotes the estimated probability of word r t . As compared with NLL loss, the BOW loss discards the order of words and forces the latent variable to capture the global information of the target response.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
Response selection helps distinguish whether the response is relevant with the dialogue context and consistent with the background knowledge. Meanwhile, its score can be regarded as an indicator of coherence during inference, helping to select the most coherent one from multiple candidate responses. Particularly, the training of response selection is carried out together with the bi-directional encoding of latent act recognition. The positive training samples come from the dialogue context and corresponding target response (c, r), with label l r = 1. And the negative samples are created by randomly selecting responses from the corpus (c, r − ), with label l r − = 0. The binary cross-entropy loss of response selection is defined as follows:
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
The above probability is estimated through one fully-connected layer, with the final hidden state of the special mask fed as input:
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
To sum up, the total objective of our pre-training model is to minimize the integrated loss:
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
Our pre-training model contains 12 transformer blocks, with network parameters initialized using BERT BASE . Large-scale conversation datasets -Twitter (Cho et al., 2014) and Reddit (Zhou et al., 2018; are employed for pretraining, which results in 8.3 million training samples in total. For each training sample of context and target response (c, r), it needs to pass through the network twice to accomplish the tasks of latent act recognition and response generation. And the pre-training steps are summarized as follows: 1) Latent Act Recognition -Given a pair of context and target response, estimate the posterior distribution p(z|c, r) -Randomly select r − and calculate L RS 2) Response Generation -With the sampled latent value z ∼ p(z|c, r), calculate L N LL and L BOW 3) Optimization -Sum up to obtain L, and update network parameters with back-propagation
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
The hyper-parameters used in pre-training are listed as follows. The maximum sequence length of context and response is set to 256 and 50, respectively. The number of transformer blocks in our model L is 12 and the hidden embedding dimension D is 768. The batch size is set to 64 and K is set to 20 for the discrete latent variable. Adam optimizer (Kingma and Ba, 2015) is employed for optimization with a learning rate of 5e-5. The pretraining of dialogue generation was carried out on 8 Nvidia Telsa V100 32G GPU cards for 3.5M steps, taking about two weeks to reach convergence.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
Our pre-trained model is flexible enough to support various kinds of dialogues, including chit-chat, knowledge grounded conversation, conversational question answering, etc. The fine-tuning on small conversation datasets can be carried out by following the training objectives defined in Equation (8).
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
As the fine-tuning process reaches convergence, the response towards the given context can be obtained through the following inference procedure: 1) Candidate Response Generation -Conditioned on each latent value z ∈ [1, K], generate corresponding candidate response r.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
-Calculate the probability for each response p(l r = 1|c, r) and select the one with highest coherence value as the final response. It is worth noting that the above fine-tuning and inference procedures are set up for the dialogue generation without any specific objectives. If there exists a specific objective within the conversation, such as letting both participants know more about each other (Bao et al., 2019), the fine-tuning can proceed to maximize the pre-defined rewards with reinforcement learning (RL). Under such circumstances, our latent discrete variable can be naturally treated as action within RL, and thus the response selection can be straightforwardly solved by selecting the action that results in the maximum reward.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
To evaluate the performance of our proposed method, comprehensive experiments have been carried out on three publicly available datasets.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
• Persona-Chat (Zhang et al., 2018) is a knowledge grounded conversation dataset. It provides both manually annotated conversations and corresponding persona profiles (background knowledge), where two participants chat naturally and try to get to know each other. • Daily Dialog (Li et al., 2017) is a chit-chat dataset, which contains high-quality human conversations about daily life. • DSTC7-AVSD (Alamri et al., 2019), short for Audio Visual Scene-aware Dialog of the DSTC7 challenge, is a conversational question answering dataset. In DSTC7-AVSD, the system need to generate an answer given dialogue context and background knowledge. There are two available options of knowledge utilization: 1) using singlemodal information of text only, including video's caption and summary; 2) relying on multi-modal information, including text, audio and visual features. The single-modal option is adopted by our method in the experiments. The descriptions and statistics of these datasets are summarized in Table 1.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
The following models have been compared in the experiments.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
Baseline. Sequence to sequence with attention (Seq2Seq) (Vinyals and Le, 2015) is employed as the baseline for the experiments on Persona-Chat and Daily Dialog. DSTC7-AVSD has provided a baseline system, which is built upon hierarchical recurrent encoders with multi-modal features. State of the art. Persona-Chat was also utilized in the ConvAI2 challenge (Dinan et al., 2019a), where the team of Lost in Conversation (LIC) (Golovanov et al., 2019) obtains the best performance. LIC is also one transformer based generation method and fine-tuned upon the pre-trained model of GPT (Radford et al., 2018). For the dataset of Daily Dialog, its best results are reported by the recently developed method -iVAE MI (Fang et al., 2019), which generates diverse responses with samplebased latent representation. In DSTC7-AVSD, the team of CMU (Sanabria et al., 2019) 
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
Both automatic and human evaluations are employed to assess the performance of compared methods. In automatic evaluation, the following metrics are included:
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
• BLEU (Chen and Cherry, 2014) measures the n-gram overlap between generated response and the target response. • Distinct-1/2 (Li et al., 2016) measures the generation diversity, which is defined as the number of distinct uni-or bi-grams divided by the total amount of generated words. • Knowledge Recall/Precision/F1 (Dinan et al., 2019b) measures the degree of informativeness w.r.t. background knowledge. • In DSTC7-AVSD, the MSCOCO platform (Chen et al., 2015) is employed for evaluation. It compares the generated response with six ground truth responses, using metrics of BLEU, ME-TEOR, ROUGH-L and CIDEr.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
In human evaluation, we randomly select 100 dialogue contexts and generate responses with compared methods. Three crowd-sourcing workers are asked to score the response quality on a scale of [0, 1, 2] from four aspects -fluency, coherence, informativeness and overall. The higher score, the better. Details about the criteria are given as follows.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
• Fluency measures whether the generated sentence is smooth and grammatically correct. • Coherence evaluates whether the generated response is relevant with the dialogue context and consistent with the expressed information or background knowledge. • Informativeness assesses whether the response is informative or not. • Overall represents the general evaluation, where 0 indicates a bad response, 1 refers to a normal response and 2 stands for a good response. After collecting the assessments from annotators, the response's final score is determined via majority voting. The average Fleiss's kappa (Fleiss and Cohen, 1973) on Persona-Chat and Daily Dialog is 0.515 and 0.480 respectively, indicating annotators have reached moderate agreement.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
The experimental results on Persona-Chat and Daily Dialog with automatic and human evaluations are summarized in Table 2. As suggested in the empirical study (Liu et al., 2016), the correlation between automatic metrics and human judgments is weak in open-domain dialogue generation. In the automatic evaluation, experimental results demonstrate that no method can consistently outperform the others.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
During human evaluations, our method achieves better performance consistently across all the metrics on Persona-Chat and Daily Dialog. The scores of fluency almost approach the upper bound, revealing that our generated responses are very fluent. The informativeness assessments indicate that the information in our generated responses is significantly richer, as compared with the baseline methods. Our responses are coherent with the context and favored most by crowd-sourcing workers. The ablation study with our method and our w/o latent also suggests that through the incorporation of discrete latent variables, remarkable improvements can be achieved for dialogue generation. In addition, it can be observed that the generation quality of transformed-based approaches (LIC and our method) is significantly better than that of RNNbased methods (Seq2Seq and iVAE MI ). 3 The experimental results on DSTC7-AVSD with automatic evaluation are provided in Table 3. In the experiments, our response selection is strengthened with an extra ranking step, which learns to rank the candidates according to automatic scores and selects the top one as the final answer. The results in Table 3 demonstrate that our method has brought a new breakthrough for DSTC7-AVSD. Additionally, the upper bound of our method is also reported, under the ideal scenario that the optimal candidate answer can be selected. 4 The incredible results validate the great potential of our approach.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
To further dissect the quality of our pre-trained model, several examples of generated responses are provided in Table 4. For each piece of context, our model can produce multiple responses by assigning distinct values to the latent variable and five candidate responses are selected for display in the table. It shows that our pre-trained model is able to generate diverse and appropriate responses. More examples on the conversational datasets are provided in the Appendix.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
To further analyze the effectiveness of our pretrained model, more ablation studies have been conducted on Persona-Chat. Distinct pre-trained models are included for comparison. To be fair, their transformer layers are all set to 12. There are three different sizes of training dialogues: 1k, 5k and 9k (all training data). The training configurations and experimental results measured with perplexity are summarized in Table 5. There are three groups of pre-trained models: group 1 applies direct fine-tuning of BERT or GPT-2 (Radford et al., 2019) on Persona-Chat; group 2 employs Twitter and Reddit for further training upon the basis of pretrained language models; group 3 carries out the training process with latent variable. 5 (Model 2.2 is our w/o latent one and model 3.1 is our method.)
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
These results demonstrate that our method outperforms the other pre-trained models consistently 4 Given a dialogue context and background knowledge, our model is able to generate K diverse responses. Each of them will be evaluated using MSCOCO and the one obtaining the best score will be treated as the optimal candidate answer. 5 Overall, group 1 involves two-stage training: pre-training of language model with general text and fine-tuning on small conversation datasets. Whereas, group 2 and group 3 involve three-stage training: pre-training of language model with general text, further pre-training of dialogue generation with Twitter and Reddit, and fine-tuning on small conversation datasets. with lower perplexity across different training sets. Several interesting conclusions can be also drawn from these results. Firstly, the comparison between model 1.2 and model 1.3 encourages the adoption of flexible attention mechanism to fully leverage the bi-directional context information. 6 Secondly, the superiority of group 2 over group 1 mainly comes from the employment of Twitter and Reddit, which are closer to human conversations than general text. Thirdly, the comparison between model 2.2 and model 3.1 reflects that the incorporation of discrete latent variable is able to boost the quality of response generation, whose effects have also been verified in Table 2.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
A novel pre-training model for dialogue generation is introduced in this paper, incorporated with latent discrete variables for one-to-many relationship modeling. To pre-train our model, two reciprocal tasks of response generation and latent recognition are carried out simultaneously on large-scale conversation datasets. Our pre-trained model is flexible enough to handle various down-stream tasks of dialogue generation. Extensive and intensive experiments have been carried out on three different kinds of publicly available datasets. And the results demonstrate that our model obtains significant improvements over the other state-of-the-art methods.
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
Our work can be potentially improved with more fine-grained latent variables. In the future, we will also explore to boost the latent selection policy with reinforcement learning and extend our pre-training to support dialogue generation in other languages. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019 
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
In Table 6, it provides the cases of our method and compared approaches on Persona-Chat, where two participants chat with each other according to their personas. As shown in the example, participant P2 needs to produce a response towards the given dialogue context, conditioned on his/her persona profile. The baseline Seq2Seq tends to generate common replies with low informativeness and poor coherence. LIC and our w/o latent are able to produce some coherent responses, whereas deficient in informativeness. In comparison, the response by our method is not only coherent with the context, but also expressive of the background personas. Besides, we also observe the phenomenon of diverse knowledge usage in our response generation, which suggests that the latent variable helps control the knowledge selection and utilization in an implicit way. Table 7 provides the generated responses on Daily Dialog, where two participants chat about daily life. This example shows that Seq2Seq is able to generate fluent utterances, while lacking coherence with the context. As for iVAE MI , it suffers from the difficulty to balance diversity and fluency. By contrast, our method is able to generate more coherent and high-quality responses. Table 8 provides the generated responses on DSTC7-AVSD, where two participants discuss the objects and events in a video. Participant P1 is responsible to raise questions, who only has access to the first, middle and last frames of the video. Participant P2 has watched the whole video and needs to answer the partner's questions. The generation system is developed to mimic P2 and answer the questions based on the background knowledge. The baseline approach relies on the multi-modal information, including text, audio and visual features, to produce the answer. Due to the limited network capacity, it lacks fidelity to the background knowledge and makes the generated response less accurate. (As the team of CMU has not released their codes or models, their samples are omitted here.) Our method utilizes the video caption and summary as background knowledge. It can be observed that our method generates a more appropriate answer as compared with the baseline approach. Our Method that is cool , i am looking forward to retiring in 5 years iVAE MI the cake and is very popular .
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
Our w/o Latent i'm not sure of his name , but i know it's going to be a thriller .
PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
Our Method i'm not sure , but i think he might be the director of gone with the wind .  