How did prior work leverage future utterance for dialogue generaiton?
In open-domain dialogue generation, the focus of
this paper, most models are context-aware, predicting the target response based on previous context
representations. With the development of this field,
researchers began to pay attention to the future
context: NEXUS (Shen et al., 2018) introduces
a posterior distribution for estimating the future
information as a hidden variable and decodes responses accordingly. RegDG (Feng et al., 2020)
trains a teacher model with both past and future
utterances and transfers the whole-context knowledge to a student model which has only access to
past utterances. ProphetChat (Liu et al., 2022) first
predicts two utterances forward, generating multiple responses and future utterances. With a selector,
the best response and future utterance are selected.
A final response is squeezed by both history and
future. Compared with HDLD, there is inadequate
consideration of dialogue duality in these works.
7383
They ignore the explicit interaction between the
dialogue history and future, still resulting in underutilization of future information.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Human dialogues are scenario-based and appropriate responses generally relate to the latent context knowledge entailed by the specific scenario. To enable responses that are more meaningful and context-specific, we propose to improve generative dialogue systems from the scenario perspective, where both dialogue history and future conversation are taken into account to implicitly reconstruct the scenario knowledge. More importantly, the conversation scenarios are further internalized using imitation learning framework, where the conventional dialogue model that has no access to future conversations is effectively regularized by transferring the scenario knowledge contained in hierarchical supervising signals from the scenario-based dialogue model, so that the future conversation is not required in actual inference. Extensive evaluations show that our approach significantly outperforms state-of-the-art baselines on diversity and relevance, and expresses scenario-specific knowledge.
[{"first":"Shaoxiong","last":"Feng","middle":[]},{"first":"Xuancheng","last":"Ren","middle":[]},{"first":"Hongshen","last":"Chen","middle":[]},{"first":"Bin","last":"Sun","middle":[]},{"first":"Kan","last":"Li","middle":[]},{"first":"Xu","last":"Sun","middle":[]}]
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
Typical generative dialogue models utilize the dialogue history to generate the response. However, since one dialogue utterance can often be appropriately answered by multiple distinct responses, generating a desired response solely based on the historical information is not easy. Intuitively, if the chatbot can foresee in advance what the user would talk about (i.e., the dialogue future) after receiving its response, it could possibly provide a more informative response. Accordingly, we propose a novel dialogue generation framework named ProphetChat that utilizes the simulated dialogue futures in the inference phase to enhance response generation. To enable the chatbot to foresee the dialogue future, we design a beam-search-like roll-out strategy for dialogue future simulation using a typical dialogue generation model and a dialogue selector. With the simulated futures, we then utilize the ensemble of a history-to-response generator and a future-to-response generator to jointly generate a more informative response. Experiments on two popular open-domain dialogue datasets demonstrate that ProphetChat can generate better responses over strong baselines, which validates the advantages of incorporating the simulated dialogue futures.
[{"first":"Chang","last":"Liu","middle":[]},{"first":"Xu","last":"Tan","middle":[]},{"first":"Chongyang","last":"Tao","middle":[]},{"first":"Zhenxin","last":"Fu","middle":[]},{"first":"Dongyan","last":"Zhao","middle":[]},{"first":"Tie-Yan","last":"Liu","middle":[]},{"first":"Rui","last":"Yan","middle":[]}]
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Sequence-to-Sequence (seq2seq) models have become overwhelmingly popular in building end-to-end trainable dialogue systems. Though highly efficient in learning the backbone of human-computer communications, they suffer from the problem of strongly favoring short generic responses. In this paper, we argue that a good response should smoothly connect both the preceding dialogue history and the following conversations. We strengthen this connection through mutual information maximization. To sidestep the non-differentiability of discrete natural language tokens, we introduce an auxiliary continuous code space and map such code space to a learnable prior distribution for generation purpose. Experiments on two dialogue datasets validate the effectiveness of our model, where the generated responses are closely related to the dialogue context and lead to more interactive conversations.
[{"first":"Hui","last":"Su","middle":[]},{"first":"Xiaoyu","last":"Shen","middle":[]},{"first":"Wenjie","last":"Li","middle":[]},{"first":"Dietrich","last":"Klakow","middle":[]}]
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
In this paper, we define a widely neglected property in dialogue text, duality, which is a hierarchical property that is reflected in human behaviours in daily conversations: Based on the logic in a conversation (or a sentence), people can infer follow-up utterances (or tokens) based on the previous text, and vice versa. We propose a hierarchical duality learning for dialogue (HDLD) to simulate this human cognitive ability, for generating high quality responses that connect both previous and follow-up dialogues. HDLD utilizes hierarchical dualities at token hierarchy and utterance hierarchy. HDLD maximizes the mutual information between past and future utterances. Thus, even if future text is invisible during inference, HDLD is capable of estimating future information implicitly based on dialogue history and generates both coherent and informative responses. In contrast to previous approaches that solely utilize future text as auxiliary information to encode during training, HDLD leverages duality to enable interaction between dialogue history and the future. This enhances the utilization of dialogue data, leading to the improvement in both automatic and human evaluation.
[{"first":"Ang","last":"Lv","middle":[]},{"first":"Jinpeng","last":"Li","middle":[]},{"first":"Shufang","last":"Xie","middle":[]},{"first":"Rui","last":"Yan","middle":[]}]
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Neural dialogue generation has drawn increasing attention due to its vast commercial values and practical demands. Typically, given the dialogue history, neural dialogue models, such as plain Seq2Seq model (Sutskever et al., 2014) and Transformer (Vaswani et al., 2017), learn to predict responses via maximum likelihood estimation (Vinyals and Le, 2015;Shang et al., 2015).
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Different from other sequence generation tasks, such as machine translation and paraphrase generation, the dialogue generation task can be regarded as a loose-coupling task, which has much freedom in the semantic and the linguistic aspects of the generated responses. However, it is often hard for the existing models to handle such freedom, compared to the fact that humans have no problem in  giving specific yet varied responses even for openended dialogue history Csaky et al., 2019). One important reason is that we can extend the given dialogue with many possible scenarios of enriched, imaginative background information from our experience and world knowledge, to which existing systems have no access.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
It is beneficial for the dialogue systems to build upon such scenarios to facilitate dialogue generation. However, manually annotating the scenario contexts is intractable in terms of both difficulty and quantity. In turn, we find that such scenarios are naturally contained in existing multi-turn dialogue corpora, where the entire dialogue of both dialogue history and future conversation with respect to the current utterance implicitly represents a specific dialogue scenario. An example is given in Figure 1. For Scenario 1, "for a week" in the future conversation suggests the response is related to time. For Scenario 2, "cut the price" indicates that the response contains price information.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Therefore, we reconstruct the dialogue task that only relies on dialogue history into a scenariobased response generation task. In order to enrich the conversation scenario, we employ future conversations together with dialogue histories to learn implicit conversation scenarios, which provide more semantic constraints to guide the response generation. We further propose a novel model to handle this new type of training data consisting of {implicit scenario, response} pairs.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
It should be noted that the scenario-based dialogue model relies on future conversations that are inaccessible in inference. Rather than simply searching the training corpora for possible scenarios, we propose an imitation learning framework to drive the conventional dialogue model to absorb the corresponding scenario knowledge from the scenario-based dialogue model. Specifically, the scenario-based dialogue model serves as a teacher, and the conventional dialogue model that relies solely on dialogue history serves as a student that mimics the outputs of the teacher. Under the regularization of scenario knowledge, the student is effectively guided towards a wider local minimum that represents better generalization performance (Chaudhari et al., 2017;Keskar et al., 2017). To facilitate knowledge transfer, the student mimics the teacher on every layer instead of just the top layer, which alleviates the delayed supervised signal problem using hierarchical semantic information in the teacher (Li et al., 2019a). Besides containing the information of future conversations, the distilled knowledge (Hinton et al., 2015) is also a less noisy and more "deterministic" supervised signal in comparison to real-world responses (Lee et al., 2018;Guo et al., 2019), which provides the student with smoother sequence trajectories that are easier to fit.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
We highlight our contributions as follows:
Regularizing Dialogue Generation by Imitating Implicit Scenarios
• We introduce future conversations together with dialogue histories to learn implicit conversation scenarios, which provide more semantic constraints to drive the responses to be meaningful and relevant to the real-world scenario-specific knowledge.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
• We propose an imitation learning framework that bridges the gap between training and inference in the accessibility of future conversations. We also demonstrate why imitation learning works and further how to enhance the imitation learning.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
• Our model achieves better results than stateof-the-art baselines on four datasets. Extensive analysis demonstrates the effectiveness and the scalability of the implicit conversation scenarios and the proposed imitation learning framework. 
Regularizing Dialogue Generation by Imitating Implicit Scenarios
In this section, we first introduce the scenariobased dialogue model, then describe the imitation learning framework shown in Figure 2, and finally present the training objective.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
The conventional dialogue model takes a sequence of dialogue history X = {x 1 , . . . , x T } as input, and generates a response Y = {y 1 , . . . , y T } word by word, where T and T represent the length of source side and target side respectively. The maximum likelihood estimation is usually used to train the model, which can also be expressed as minimizing the negative log-likelihood:
Regularizing Dialogue Generation by Imitating Implicit Scenarios
where |V| is the size of vocabulary, θ 1 is a set of parameters, and X h represents the input sequence that is from dialogue history. However, dialogue task allows responses to continue the dialogue topic from many aspects or even introduce a new topic depending on various conversation scenarios or semantic constraints, which dramatically increases the difficulty of prediction without any specific scenario information besides the hints from the given dialogue history. Moreover, labeling the scarce scenario information is laborconsuming and impractical. Instead, we resort to easy-to-access but underutilized future conversations that exist in all multi-turn dialogue corpora. By combining the dialogue history and its corresponding future conversation, we introduce the implicit conversation scenario into existing dialogue models to provide more semantic constraints and reduce the difficulty of prediction.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Concretely, we enforce the model to use implicit conversation scenarios to generate responses from two aspects. Different from previous dialogue models only based on the dialogue history X h to predict the response Y , the future conversation X f is also considered as part of input so that the model can look ahead and predict more purposefully. Intuitively, our training pair {(X h , X f ), Y } induces the model to imitate humans to produce the scenariospecific response.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
We also redesign the sequence generation architecture to handle the proposed training pair. The attention module in each layer calculates the weight of the contextualized token representations from the encoder based on the information that has been generated in the decoder, and then returns the context c h . In order to consider the future conversation X f , we apply another encoder to produce the contextualized token representations of X f , which will be further extracted as the context c f by the attention module. The new encoder shares the parameters with the original encoder. Meanwhile, the output of the attention module is the concatenation of the past context c h and the future context c f . Finally, the training criterion is formulated as the following negative log-likelihood:
Regularizing Dialogue Generation by Imitating Implicit Scenarios
where θ 2 is a set of parameters to minimize the NLL loss for the scenario-based dialogue model.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
In inference, future conversations are inaccessible, which means implicit conversation scenarios cannot be constructed. Thus, the performance improvement from the scenario-based dialogue model cannot facilitate the generation of high-quality responses in practice. In order to bridge this gap between training and inference, we propose an imitation learning framework, in which we regard the scenario-based dialogue model as a teacher and the conventional dialogue model as a student. Through step-by-step imitation, including fine-grained prediction imitation and intermediate representation imitation, scenario knowledge distilled from the teacher regularizes the student to reach a robust local minimum and obtain significant generalization performance in inference.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Compared with the ground-truth labels, the soft predictions (i.e., the probability distribution from the output layer) contain more fine-grained and valuable information, such as the similarity of labels and potential future conversations. Moreover, the soft predictions provide less noisy and more "deterministic" targets that are easy to mimic. To transfer knowledge from the teacher, instead of taking the one-hot representation of Y as the target, we minimize the cross-entropy of the predicted probability distribution between the teacher and the student:
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Only transferring knowledge from the output layer has a limited effect on the student to use implicit conversation scenarios. When the student network is very deep, the supervised signals from the output layer hardly conduct an effective update and regularization on the parameters of intermediate layers, which will make the imitation learning framework quickly reach saturation (Romero et al., 2015;. This problem prevents the student from scaling to deeper models to further improve the model performance.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
To tackle this problem, we extend the range of imitation learning from the soft predictions in the output layer to the output h of intermediate layers to guide the imitation process. Specifically, we penalize the discrepancy of hidden states in intermediate layers between the teacher and the student:
Regularizing Dialogue Generation by Imitating Implicit Scenarios
where |O| is the number of intermediate layers, h t il and h s il are the outputs of intermediate layers in the teacher and the student respectively, and f (·) is the measurement function.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
where φ(·) is the mean-squared-error (MSE) loss. Because we observe that directly applying the MSE loss as an additional loss hurts the stability of the imitation learning process, we set a scalar threshold α to loose this constraint.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Combining the NLL loss in Equation (1) with the IL losses in Equation (3) and Equation (4), the final objective function of the student is formulated as:
Regularizing Dialogue Generation by Imitating Implicit Scenarios
where λ 1 is a hyper-parameter that balances the importance of the NLL loss and the IL losses. Because the scenario knowledge is only transferred from the teacher by hierarchical supervised signals, our imitation framework has the following three advantages: (1) Compared with the finetuning style of knowledge transfer (Dai and Le, 2015;Howard and Ruder, 2018), the proposed imitation framework does not affect the teacher, i.e., the knowledge learned from the teacher will not be forgotten. (2) The proposed method is model agnostic. Thus, the imitation object can be extended from one teacher to multiple teachers, such as incorporating a language model besides the scenario-based dialogue model. (3) The imitation process does not change the current objective function, which means the previous work of modifying objective function can serve as a complementary to improve the model performance further.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
DailyDialog It is provided by Li et al. (2017b), which contains various dialogue topics about daily life. We randomly select 27K, 2.5K, and 1.5K pairs for training, validation, and testing.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
PersonaChat It is gathered by assigning two Amazon Turkers with their personas to chat with each other (Zhang et al., 2018a). We only use the conversation section and split it to 67K, 8.5K, and 8K pairs for training, validation, and testing.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
OpenSubtitles It is collected from movie subtitles and consists of more than 60M scripted lines (Lison and Tiedemann, 2016). We randomly extract 1500K, 50K, and 25K pairs for training, validation, and testing.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
For all datasets, every seven consecutive dialogue turns form a training example, in which the first three turns, the middle turn, and the last three turns are taken as dialogue history, response, and future conversation, respectively.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
We also conducted the experiment on a multidomain goal-oriented dataset called MultiWOZ, which is simplified by us as a general dialogue generation task. The detailed description of MultiWOZ and data pre-processing is provided in Appendix A.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
We re-implemented two classes of six baselines for comparison. The detailed settings of baselines are provided in Appendix B.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
LSTM-Based One class is based on LSTM, including Seq2Seq+Att, which contains a vanilla Seq2Seq model (Sutskever et al., 2014) with attention mechanism (Bahdanau et al., 2015), VHRED+BOW (Serban et al., 2017), which introduces a continuous latent variable attached to the response information into HRED  and applies BOW loss (Zhao et al., 2017) as a complementary with KL annealing, and NEXUS , which further uses the future conversation to incorporate more scenario information into the latent variable.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Transformer (Vaswani et al., 2017), including itself, ReCoSa (Zhang et al., 2019a), and CHMAM (Tao et al., 2018), which consists of Multi-Head Attention Mechanism (MHAM) and an attention weight regularizer. Both ReCoSa and CHMAM aim to extract more relevant and diverse scenario information from dialogue history.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Based on the performance including the loss and metrics on the validation dataset, we trained baselines and our models with the following hyperparameters. According to the scale of the dataset, the vocabulary sizes for OpenSubtitles, DailyDialog, PersonaChat, and MultiWOZ are set to 50k, 20k, 20k, and 18k, respectively. We use separate word embeddings for the encoder and the decoder, and the word embedding dimension is 256. All the parameters are initialized randomly from a normal distribution N (0, 0.0001). All models are trained using Adam (Kingma and Ba, 2015) with a learning rate of 0.001 and gradient clipping at 2.0. The batch size is 128. The hyper-parameters in our proposed appraoch are set as α = 0.01 and λ 1 = 2.0. Our models, i.e., RegDG, the imitating student conventional model, and Transformer-IF, the imitated teacher scenario-based model, are based on Transformer.    Table 1: The automatic evaluation results at the lowest point of the validation loss. The proposed approach achieves substantial improvements across all the dialogue datasets. "↑" means higher is better. "↓" means lower is better.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
We conducted both automatic and human evaluation to compare the performance of the models.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
The evaluation of opendomain dialogue generation has no well-defined automatic metrics. Thus, we employ two kinds of automatic metrics to evaluate all models. The reference-based metrics, perplexity (PPL), BLEU (%) (Papineni et al., 2002), and the embedding metrics (including embedding average (AVE), embedding greedy (GRE), embedding extrema (EXT)) (Liu et al., 2016), and coherence (COH) (Xu et al., 2018b), are widely adopted to reflect the grammaticality and semantic relevance of the responses (Serban et al., 2017;Csaky et al., 2019). The countbased metrics, distinct (Dist-{1,2,3} (%)) (Li et al., 2016) and KL divergence (Csaky et al., 2019), are used to evaluate the lexical diversity and the distribution distance of the responses (Xu et al., 2018a;Zhang et al., 2018b). We report the unigram and bigram version of KL divergence, i.e., D u kl and D b kl . Please refer to Appendix C for the detailed settings of automatic metrics.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
We conducted human evaluation to assess the quality of response. We randomly selected 200 test examples from each dataset and asked three annotators to judge which generated response in each pair (RegDG and baseline) is better (i.e., win, lose or tie) in terms of Diversity (how much the generated response contains meaningful information), Relevance (how likely the generated response is coherent to both dialogue history and future conversation), and Fluency (how likely the generated response is from human).
Regularizing Dialogue Generation by Imitating Implicit Scenarios
The results obtained at the lowest point of the validation loss are shown in Table 1. Our proposed model significantly outperforms all baselines on all datasets. The LSTMbased baselines obtain better performance than Transformer-based baselines in terms of diversity, distribution distance, and relevance, while they lose in grammaticality. It suggests that the LSTM-based model still has a certain advantage in the loose coupling dialogue task. Compared with CHMAM, Re-CoSa achieves higher scores on BLEU and embedding metrics but weaker results on Dist-{1,2,3} and KL divergence, which means that only extracting scenario information from dialogue history cannot provide sufficient semantic constraints to improve model performance across all metrics. Although NEXUS and VHRED+BOW enrich the latent variable and bring more diversity and relevance, they show a distinct decline in PPL. It verifies that our method not only effectively uses the implicit conversation scenario to boost the performance but also indeed transfers this advantage to the inference phase. The improvements of our model on all datasets are significant with p ≤ 0.01 (t-test  
Regularizing Dialogue Generation by Imitating Implicit Scenarios
The results are shown in Table 2. We only report the results of VHRED+BOW, NEXUS, ReCoSa, and CHMAM, which are more related to our work. From the results, we can observe that our model performs better than baselines in all datasets. In particular, our model obtains the most significant win-lose difference on diversity, which demonstrates that the implicit conversation scenario induces the response containing more tokens that are meaningful. We calculate the Fleiss's kappa (Fleiss, 1971) to measure the inter-annotator agreement, and the results are mainly distributed in [0.4, 0.6] (i.e., moderate agreement range) with the significance p ≤ 0.01.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
In this section, we further quantitatively analyze the effectiveness of future conversations and explore why imitation learning works and how to enhance it. For limited space, we select a set of complementary metrics, Dist-{1,2,3}, PPL, and BLEU, to report the results. The rest of the results is in Appendix F. Table 3 presents some generated responses. The responses generated by baselines are usually dull and meaningless, while the responses generated by our model show diverse and coherent semantic information that indicates distinct relations with those topics in future conversation. The improvements of our model demonstrate the effectiveness of implicit conversation scenarios and our imitation learning framework. Due to limited space, we provide more examples in Appendix E.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
We evaluate the performance of our method without fine-grained prediction imitation (FPI) or intermediate representation imitation (IRI). The ablation study results, reported in     informative includes the examples in which the {dialogue history, response} or the {response, future conversation} is a many-to-one pair. Generally, the second sequence in many-to-one pairs is dull and meaningless (Csaky et al., 2019). To determine the many-to-one pairs, we need to judge whether sentences are of the same meaning and we adopt three measures, that is, whether if the strings match (Exact Match), the words overlap more than 80% (Word Overlap), or the sentences are in the same embedding cluster (Sent. Cluster).
Regularizing Dialogue Generation by Imitating Implicit Scenarios
For the detailed settings of the above strategies, please refer to Appendix F. Table 5 shows the results of Transformer-IF on DailyDialog. We can see that the average of all metric improvements of Transformer-IF on the Uninformative set is lower than the Other set, which verifies the assumption that the informativeness of the future conversation supplementing the conversation scenario is crucial to the proposed approach.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
In order to demonstrate the impact of the information content of the implicit conversation scenario on model performance, we conducted the training and testing of both Transformer and Transformer-IF on DailyDailog (1-1-1) and DailyDailog (3-1-3), respectively. "3-1-3" represents that both dialogue history and future conversation consist of three turns, and response only contains one turn. "1-1-1" represents that all sequences in the training examples consist of one turn.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
The results are shown in Table 6. Compared with   the results on DailyDialog (1-1-1), both models on DailyDialog (3-1-3) achieve overall improvements. The absolute improvements in multi-turn conversation are higher than those in single-turn conversation, which means that Transformer-IF performs better when the implicit conversation scenario contains rich semantic information. Because the automatic metrics may still improve after the lowest point of validation loss (Csaky et al., 2019), the results of both models after 50 epochs of training are reported in Appendix F. It can be observed that Transformer-IF still substantially outperforms Transformer across all metrics under this setting.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Why does the imitation learning work? According to observations in previous work (Chaudhari et al., 2017;Keskar et al., 2017), the model generalization is related to the width of the local minimum achieved by the model. Wider local minima suggests that the model can effectively resist perturbations and obtain better performance on unseen datasets. Therefore, we inject perturbations into the student to judge whether it is guided to a wider local minimum based on the regularization of knowledge transfer. Specifically, we add Gaussian noise with varying magnitude to the parameters of the trained model and observe the perplexity drop on the test set. The results in Figure 3 show that the perplexity of all baselines rapidly increases while the perplexity of our student model grows slowly,  Table 8: Results using the hard transfer. With more hard-transferred modules, the diversity gradually improves, while the relevance gradually weakens.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Step ( indicating that the student model reaches a wider local minimum to gain better generalization. We also analyze the word distributions of the generated responses to intuitively reflect the effect of regularization from imitation learning. Concretely, we use a vector to represent all generated responses, and each element in the vector represents the frequency of a word. Only 2350 most frequent words are considered as Feng et al. (2020). Then, we calculate the distance between the word distributions from each model and the real-world data. From Table 7, it can be seen that our model significantly outperforms plain Transformer and other baselines, which indicates that knowledge transfer effectively regularizes the model so that the model avoids sticking in a relatively centralized word distribution.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Can imitation learning be accelerated? Before the student mimics the teacher, the teacher is usually well pre-trained. According to our observation, this is a redundant workflow that almost doubles the training time. It is worse if we should train a larger model on a huge dataset. In order to accelerate the training process, instead of transferring knowledge via supervised signals to train a student from scratch, we initialize the specified module of the student directly using the parameters of the teacher, and the transferred parameters are kept from updates during the training process. We call this the hard transfer. We first apply the hard transfer operation on word embedding (Word-Emb), and further extend it to the encoder. The results of Dai-lyDialog in Table 8 indicate that the performance  has been further improved on the diversity with a slight drop on the relevance. Figure 4 shows the variation curve of PPL on the validation set with the training step. The full results are provided in Appendix F. With more hard-transferred modules, the model reaches the lowest point of validation loss faster. It demonstrates that the hard transfer distinctly accelerates the convergence.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Do multiple teachers work? To take advantage of more diverse and richer prior knowledge, we consider extending the teacher from one to many. We pre-train a transformer-based language model as another teacher. The results are shown in Table 9 with full results in Appendix F. It is clear that with the help of the language model, the student further improves on all metrics, except for a weak decline in relevance, because the language model conducts unconditional sequence generation and does not consider the mapping between the dialogue history and the response. We defer the exploration of balancing multiple teachers in future work.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
In this work, we introduce the future conversation with the corresponding dialogue history to learn the implicit conversation scenario, which entails latent context knowledge and specifies how people interact in the real world. To incorporate such scenario knowledge without requiring future conversation in inference, we propose an imitation learning framework. The scenario-based teacher model first learns to generate responses with access to both the future conversation and the dialogue history and then a conventional student model is trained to imitate the teacher by hierarchical supervisory signals. As a result, the student is effectively regularized to reach a robust local minimum that represents better generalization performance. Evaluation on four datasets demonstrates the effectiveness and the scalability of our approach, compared to the state-of-the-art baselines. The proposed framework enables the generation of responses that pertain more closely to the scenario indicated by the given dialogue history. Moreover, detailed analyses illustrate how imitating implicit scenarios regularizes the student model. For future work, we will incorporate pre-trained models into our framework (e.g., BERT as a teacher and GPT as a student) to further unlock the performance improvement and explore how to balance diverse prior knowledge from multiple teachers.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
MultiWOZ  Table 10: The automatic evaluation results at the lowest point of the validation loss. "↑" means higher is better. "↓" means lower is better.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Embedding Metrics The word embedding used in embedding metrics is pre-trained on each dataset using fasttext.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Perplexity We calculate the perplexity of each individual sentence and then calculate the average perplexity, which is different from the other way that takes the whole test set (or each batch of data) as one long sentence and then calculates the perplexity (e.g., the official code of Zhao et al. (2017)). The score calculated by the former way is usually larger than the latter one. The difference will increase as the test set (or the size of each batch) gets larger.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
KL Divergence It calculates the KL divergence D kl = 1 /|Ur| w∈Ur log 2 pr(w) /pm(w) for each reference response between the reference and the generated response sets, where p r and p m are estimated based on the reference response set and the generated response set respectively. 
Regularizing Dialogue Generation by Imitating Implicit Scenarios
We provide more generated examples in Table 11.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Effect of the Informativeness of Future Conversation We use the cosine similarity as the similarity measure and set 0.8 and 0.98 as the thresholds for one-turn sequence and three-turn sequence, respectively. The sentence embedding used in singlepass algorithm is the sum of word embedding with a corresponding weight estimated on the training set. We also adopt the k-means algorithm, which achieves similar clustering proportion.
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Effect of the Capacity of Future Conversation Table 12 demonstrates that our model works better when the implicit scenario contains rich information. Table 13 indicates that our model still outperforms Transformer after a fixed epochs (50) of training. Table 14 and Table 15 report the full version of the experimental results in Section 3.6.4.  Table 12: The results on DailyDialog (1-1-1) and DailyDialog (3-1-3).
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Dist    
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
Recent years have witnessed a surge of interest in building open-domain chatbots using generative approaches (Shang et al., 2015;Zhao et al., 2017;Tao et al., 2018;. These prevailing methods typically utilize dialogue histories as the dialogue context to generate the response via maximum likelihood estimation. Different from directed text generation tasks like machine translation where the target sentences are strictly constrained by the source sentence (Holtzman et al., 2019), the dialogue history and the response in chitchat conversations are loosely coupled (Feng et al., 2020a). In other words, open-domain chatbots often have more "freedom" to decide what to respond since there often exists multiple distinct responses that can appropriately answer the given utterance. However, we argue that such excessive "freedom" also reveals that the dialogue history only may not contain enough information to generate a desired response that is informative and easy to reply to. If provided with enriched dialogue contexts that contain more useful dialogue cues, it could be easier for the model to chat with a human. So here comes the questions: what kind of dialogue cues is complementary to dialogue histories, and how to obtain and use them.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
Recent studies in representation learning have demonstrated that when representing a token in a sentence, considering the tokens on its right side in addition to its left side can bring significant improvement (Devlin et al., 2019;. Similar findings also appear in directed text generation where the future tokens on the right side can be beneficial to generate the current token (Serdyuk et al., 2017;Zhang et al., 2018c;Qi et al., 2020). Sharing the same spirit, we pursue to use the "right side" information, which is the dialogue future in our task, as the complementary dialogue cue to enhance the generation of the current response. Intuitively, if the chatbot can be told in advance what the user would probably talk about (i.e., the dialogue future) after receiving its response, it only needs to provide a response that can smoothly connect the history and the future. To verify whether the dialogue future can act as the complementary dialogue cue, we conduct empirical studies. We find that using a dialogue generation model to learn the reverse dialogue flow (i.e., using the future to generate the response) is quite effective. Furthermore, when utilizing the ensemble of the history-to-response generation model and the future-to-response generation model to generate the response conditioned on both the history and the gold future, the quality of the generated response surely improves. Though effective, the ground truth dialogue future is inaccessible in the inference phase. Therefore, all existing works in this line choose to leverage the dialogue future only in the training phase Feng et al., 2020a,b), leaving the inference phase unchanged.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
We argue that explicitly providing the possible dialogue futures in the inference phase can offer more direct help for the generation of the current response. To enable the incorporation of dialogue futures into response generation in the inference phase, we propose a response generation framework namely ProphetChat by answering two questions: how to acquire the future and how to use it. Figure 1 shows the generation process of ProphetChat. It consists of a history-to-response model (denoted as the forward model), a future-toresponse model (denoted as the backward model), an ensemble gate, and a dialogue selector. Given a dialogue history, we first utilize an effective beamsearch-like roll-out strategy to simulate possible dialogue futures. Concretely, the forward model first generates a batch of n possible responses based on the dialogue history. The dialogue selection model then comes to pick up the k-best responses. We further generate n possible futures for each of the picked responses, resulting in k · n futures. The selection model again picks up the k-best futures which are of higher quality compared with randomly sampled ones. Next, conditioned on both the history and the simulated future, we employ the forward model and the backward model to jointly generate the response by summing the per-step output probability distributions of the two models using a calculated weight. The weight is obtained by a trainable gate that learns to balance the trade-off between history and future information. Finally, we gather the k responses generated solely based on the history and the k · n responses generated based on both the history and the future, and use the selector to choose the top-ranked one as the final response. Since the ensemble generation model relies on the selector to sequentially select the response and the future, and the ensemble generation model also needs to learn how to balance the history and the future information given the selected future, we jointly train the whole model to make each module better collaborate with others to fulfill the ultimate goal: to maximize the likelihood of the gold response estimated by the ensemble generation model given the history and the selected future. We train the ensemble generation model directly using MLE objective while adopting reinforcement learning to tune the selector.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
Our contributions in this paper are three folds:
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
• We propose a novel dialogue generation framework named ProphetChat which leverages the simulated dialogue future to enhance response generation through the ensemble of the history-to-response generator and the future-to-response generator. To the best of our knowledge, we are the first to utilize the dialogue futures for response generation in the inference phase.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
• To acquire better dialogue futures in the inference phase, we propose an effective beamsearch-like roll-out strategy for dialogue future simulation with the help of a dialogue selector.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
•  Li et al. (2020) conditioned the response generation model with knowledge. Different from the above works that aimed to design specific history-to-response generation models, we propose a response generation framework where possible dialogue futures are utilized in the inference phase with the help of an effective future simulation strategy. Future Modeling. There are various scenarios where considering future information is useful. In text generation, Serdyuk et al. (2017) proposed a twin network to regularize the hidden states of the left-to-right decoder with the future-aware right-toleft decoder. Zhang et al. (2018c) used the targetside hidden states generated by the right-to-left decoder to help the right-to-left decoder during translation so that the target-side future information can help avoid under-translation. Different from these works that consider the right side tokens as the future for the current token, we define "future" as the next dialogue utterance of the current response in a dialogue session. In response generation, Feng et al. (2020a) proposed to use gold futures as the conditions of two discriminators and adopted adversarial training to encourage diversity. Feng et al. (2020b) employed gold dialogue futures to learn a future-aware teacher model and transferred the knowledge to a history-to-response student model via imitation learning. These works only use the future information in the training phase, while we utilize the simulated dialogue future in the inference phase to provide the history-to-response generation model with direct help.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
In this section, we introduce the major off-the-shelf components in our framework.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
DialoGPT ) is a GPT-based response generation model pre-trained on large-scale open-domain dialogue corpus by maximizing the likelihood of the successive dialogue utterances (i.e., the forward dialogue flow) given the initial dialogue history. While trained on the same corpus with the same architecture, DialoGPT-MMI is trained on the backward dialogue flow where the order of the utterances in a dialogue are reversed. We adopt DialoGPT as the forward generator and DialoGPT-MMI as the backward generator. GRADE ) is a graph-enhanced dialogue evaluation model that uses both utterancelevel contextualized representations and topic-level graph representations to evaluate the response. As it is one of the SOTA dialogue evaluation models, we choose it as our dialogue selector.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
Our framework consists of a forward generator G F that models the history-response-future dialogue flow, a backward generator G B that models the reversed dialogue flow, a dialogue selector S that ranks the sampled utterances conditioned on the dialogue context, and a gate g that dynamically balance the ensemble weights between G F and G B . Given a history h, we first use G F to sequentially sample the response r and the future f with the help of S. We then employ the ensemble of G F and G B using g to generate the response based on both h and f . The firstly generated responses together with the future-aware second-pass responses are finally re-ranked by S to produce the final response. Figure 2 illustrates our proposed framework.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
Given a dialogue history h, we first use G F to generate n responses {r i } n i=1 using top-k sampling . We denote these responses as the first-pass responses. Then the selector S calculates the quality scores s r ∈ R n for all historyresponse pairs. The quality scores naturally form a propability distribution p r ∈ R n over the sampled responses by using a softmax operation. We here consider the response selection procedure as sampling from such a distribution. Considering
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
" a n d g second-pass responses + · responses · Figure 2: The overall framework of ProphetChat. that the responses in open-domain dialogue are often diverse and hard to evaluate by any automatic evaluation metric, only using the response with the highest quality score or probability to further simulate the future is suboptimal. Meanwhile, generating futures conditioned on all the n sampled responses is too time-consuming. Therefore, borrowing the idea from beam search (Sutskever et al., 2014) where the k-best sentence prefixes are maintained during decoding to balance the searching performance and speed, we propose to keep the k-best responses at hand while discarding the others. For each of the selected response r i , we concatenate it with h and use G F to again sample n dialogue futures {f j r i } n j=1 , where f j r i denotes the j-th future simulated from h and r i .
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
Up to now, we obtain k · n history-responsefuture dialogue triplets for the same dialogue history h by simulation. We again resort to the selector S to calculate the quality scores of all the generated futures conditioned on h and their corresponding ancestral responses as {s f r 1 , . . . , s f r i , . . . , s f r k }. We consider all the generated futures in the same sampling space (i.e., the future space of the given history) and directly perform softmax over the k · n quality scores to get the future distribution. Considering that the responses used to generate the futures are not equal in quality, we additionally multiply each probability of the simulated future f j r i with the probability of its ancestral response p r i to get the final ranking scores based on which we select k-best dialogue futures.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
Now with the history h and k plausible dialogue futures at hand, we pursue to generate the secondpass response conditioned on both the history and the future information. Given that the simulated futures contain noise derived from error accumulation in the simulation phase, it is necessary to balance the weights between the history-conditioned G F and the future-conditioned G B when they collaboratively generate the response. Hereby we introduce a trainable gate g which takes the last hidden states from G F and G B as inputs and calculates an ensemble weighting score w using an MLP with sigmoid activation. We then generate the responsê r using the per-step weighted ensemble of G F and G B conditioned on h and f :
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
(1) where the subscript t denotes the t-th token inr and θ F , θ B and θ g denote the parameters of G F , G B and g respectively. Specifically, we sample n responses for the ensemble generation of h and each of the k futures, resulting in k · n futureaware responses. We denote these responses as the second-pass responses. To make full use of the k-best first-pass responses, we finally re-rank the k + k · n responses with S and consider the top-ranked response as our system outputs.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
Recall that there are several components G F , G B , S, and g in our framework. Although some of them can directly be used without post-training, this might be suboptimal. For one thing, post-training the models on domain-specific data with the same objective often brings better performance (Gururangan et al., 2020). For another, the original loss functions may not be thoroughly in accord with the ultimate goal in our framework. Thereby we propose a customized joint training algorithm.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
For G F and G B , we adopt a similar training objective used by . Take G F for example, we consider every consecutive three utterances in a dialogue session as a history-responsefuture triplet and fine-tune the models by minimizing the negative log likelihood of the response and the future conditioned on the history. G B is finetuned in a similar manner with the reversed inputs. After fine-tuning, G F and G B are fixed.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
For g, we can directly minimize the negative log-likelihood of the gold response r * :
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
(2) where f is simulated from either the gold response (denoted as the teacher-forcing mode) or a sampled response (denoted as the free-running mode).
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
While for S, considering that the original objective used in  is not customized for selecting better responses and futures, it is better to perform task-specific post-training. Therefore, we propose to directly optimize S to our ultimate goal which is to maximize the log-likelihood of the gold response given the history and the selected simulated future. Since the sampling operation is non-differentiable, we use REINFORCE (Williams, 1992) with a self-critic (Rennie et al., 2017) baseline to estimate the gradient. We consider the future simulation process as sequential sampling from the score distributions of the responses and the futures respectively. Given the n responses generated by G F conditioned on h, we sample a response r i from p r . Then we generate n futures conditioned on h and r i using G F and again sample a future f j r i from them. We feed this sampled future and the gold history into our ensemble generation model and calculate the log-likelihood of the gold response, which is the opposite number of Equation 2, as the reward R. To reduce the variance of gradient estimation, we introduce a self-critic baseline. Concretely, we sequentially select the response and the future with the highest scores in each sampling step and calculate the reward of using the greedy future as the baseline reward R b . The gradients are then estimated as follows:
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
(3) Intuitively, directly forcing the model to learn in a fully free-running mode may be burdensome as the futures generated from the sampled responses may contain much noise. A better choice is to allow the model to gradually learn from easy to hard. We create a curriculum schedule (Bengio et al., 2015) that gradually switches from the teacher-forcing mode to the free-running mode. Specifically, let η denote the proportion of teacher-forcing mode, we gradually decrease η from β to α with cosine annealing schedule, where 0 ≤ α < β ≤ 1.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
For the overall training, we first train g using Equation 2 and set η = β. Then we tune S using Equation 3 with the help of the above curriculum learning schedule. Finally, we jointly tune g and S with a fixed η = α.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
To verify the effectiveness of our proposed response generation framework, we experiment on two popular dialogue datasets, DailyDialog (Li et al., 2017) and PersonaChat (Zhang et al., 2018b). We follow the original train/dev/test division and reconstruct the datasets by treating each consecutive three utterances as a triplet that represents history-response-future, resulting in approximately 65k/6k/6k examples in DailyDialog and 114k/14k/13k examples in PersonaChat.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
Posterior-GAN (Feng et al., 2020a) and RegDG (Feng et al., 2020b) are two non-GPTbased response generation models that use dialogue futures in the training time through either adversarial training or knowledge distillation. DialoGPT F denotes the fine-tuned DialoGPT medium (Zhang et al., 2020) on two downstream datasets. DialoGPT F,rerank is its enhanced version which is equipped with the dialogue evaluation model (i.e., GRADE ) to select the top-ranked response.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
ProphetChat k=? denotes the model with the same model parameters but different beam sizes when simulating the futures. ProphetChat f irst and ProphetChat second are used to denote the settings where only the firstpass or the second-pass responses are used in the final re-ranking process. 
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
ProphetChat w/o history means we utilize the topranked simulated future to generate the response without the help of the history. ProphetChat w/o selector denotes the model where we sequentially sample the responses and the futures randomly without using the selector. ProphetChat w/o train means we directly utilize the fine-tuned G F , G B and the fixed S without post-training. We manually choose a fixed ensemble weight for the ensemble generation process instead of using a trainable gate. ProphetChat w/ gold future denotes the model that utilizes the history and the gold future, which is inaccessible in the inference phase, to generate the response.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
Our implementation is based on the open-source toolkit Transformers (Wolf et al., 2020). For the generator G F and G B , we initialize them with the publicly released DialoGPT medium and DialoGPT-MMI medium 1 . For the dialogue selector, we use the pre-trained GRADE 2 as initialization. We firstly use AdamW (Loshchilov and Hutter, 2017) with learning rate 3e-5 to fine-tune G F and G B . Then, we jointly train the ensemble gate g and the top non-transformer layers of the selector S with learning rate 2e-5, while keeping other parameters (i.e., G F , G B and most of the parameters of S except for its top layers) fixed. We set the curriculum hyperparameters (α, β) as (0.0, 1.0) on both datasets. We fix the sample number n of both the response and the future as 10 and vary the simulation beam size k ∈ {1, 2, 3, 5}. We choose k=5 on Daily-Dialog and k=3 on PersonaChat. We use top-k sampling  to generate the firstpass responses, the futures, and the second-pass responses with the temperature as 0.7 and k as 40. All the hyperparameters are chosen depending on   Table 3: Future simulation results on DailyDialog (the upper) and PersonaChat (the lower) datasets. TF rerank (i.e., teacher forcing) means using the history and the gold response to generate the future then re-ranking using the selector. FR k=? (i.e., free running) means using our proposed future simulation methods.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
their performance on the development set.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
Automatic Metrics. We use BLEU (Papineni et al., 2002) to measure the word overlap between the ground truth responses and the generated ones. For simplification, we use B-n to denote the ngram overlap scores. We employ Distinct 1-4 (Li et al., 2016a) to measure the diversity of the generated responses, where Distinct-n (abbreviated as D-n ) represents the ratio of distinct n-grams in responses. We adopt the embedding-based metrics (i.e., Average, Extrema, and Greedy) (Liu et al., 2016) to measure the semantic relevance between the ground truth responses and the generated ones. Human Evaluation. We ask three well-educated annotators to score 150 randomly selected responses generated by ProphetChat and other baselines. The annotators are asked to evaluate the human-likeness of the responses from three perspectives: readability, sensibleness and specificity. For readability, we ask annotators whether the re-sponse is grammatically correct and easy to read. For sensibleness and specificity, we follow Adiwardana et al. (2020) to conduct the evaluation. For all three metrics, the annotators are asked to give 0-1 labels. We provide the averaged scores and further calculate the Fleiss's kappa (Fleiss, 1971) to measure the inter-annotator agreement.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
6 Experimental Results 6.1 Overall Performance Table 1 presents the overall performance of our proposed method as well as its variants and ablations. Table 2 shows the human evaluation results. Compared with the two non-GPT baselines, GPTbased models generally achieve superior performance, especially in Distinct and human evaluation. ProphetChat outperforms all the baseline methods by a large margin on both datasets in almost all automatic metrics and all human evaluation metrics. For human evaluation results, the Fleiss's kappa scores are mainly distributed in [0.4, 0.6], which means annotators achieved moderate agreement.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
With the same model parameters, we have several model variants by using different hyperparameters or computation flow in the inference phase. Here we mainly discuss two types of model variants:
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
(1) the model with different simulation beam size k, (2) the final re-ranking among the first-pass responses or the second-pass responses.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
The simulation beam size. When simulating the dialogue future, we can choose different beam sizes to balance the computation cost and the performance. We test k ∈ {1, 2, 3, 5} on both datasets and find k = 5 is better than others on DailyDialog, while k = 3 is enough on PersonaChat. It can be seen that when k is small, increasing k can boost the performance. With the appropriate choice of the simulation beam size, ProphetChat can be deployed to various scenarios with different computation resources. We further directly test the future simulation performance by comparing the futures generated by our method and generated using the gold responses. The results are listed in Table 3. It can be observed that on DailyDialog, with the increase of k, our future simulation method gradually catches up with the teacher forcing counterpart in BLEU and Distinct, while still lagging behind in embedding-based metrics. On PersonaChat, our method even outperforms TF rerank in several metrics. These findings proves that we are able to obtain dialogue futures of good quality solely based on the history through our effectiveness future simulation algorithm.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
Re-ranking among the first pass or the second pass responses. Recall that in our main framework, we finally gather the k first-pass responses and the k · n second-pass responses together and finally re-rank them with the selector conditioned on both the history and the corresponding future. From Table 1 we can find that on both datasets, re-ranking using both groups of responses yield better performance in most of the metrics than only using one of them. When comparing their individual performance, it can be observed that on DailyDialog, ProphetChat second is superior to ProphetChat f irst in BLEU and Distinct, while ProphetChat f irst wins in embedding-based metrics. On PersonaChat, ProphetChat second wins all metrics except BLEU. There exist some cases where the simulated futures are meaningless or include irrelevant information.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
When this happens, the final re-ranking process comes as the remedy. We find that the proportions of the test cases where the final responses are picked from the second-pass responses are 40.4% on DailyDialog and 36.6% on PersonaChat, which are less than the proportions of the second-pass responses involved in re-ranking. This finding indicates that re-ranking plays a vital role to select the appropriate responses from the two groups of candidate responses of various qualities.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
We make ablation study from several perspectives including the effect of the history, the selector and the training algorithm. Table 1 shows that although only using the simulated future (i.e., ProphetChat w/o history) can generate plausible responses, the performance is largely inferior to the full model. Also, we observe that ProphetChat w/o selector underperforms the full model, demonstrating the effectiveness of the selector which helps simulate better futures. When considering the training objective, we find that ProphetChat w/o train already achieves good performance, but jointly training the whole model further makes our model perform better. Finally, when provided with the gold future, ProphetChat w/ gold future outperforms ProphetChat in terms of embedding-based metrics on both datasets, and BLEU on PersonaChat, while underperforming on other metrics. In other words, with the simulated futures, ProphetChat can achieve comparable performance with the model that "cheats" to see the gold future, which also demonstrates the effectiveness of our method.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
Figure 3 presents two cases sampled from the two datasets. For ProphetChat, in addition to its final response, we provide its corresponding firstpass response, and the simulated future of the response. From the two cases we can observe that by taking the simulated future into consideration, ProphetChat generates more informative responses than baselines. Specifically, in case 1, when a twochoice query is issued in the history, the first-pass response chooses "online" as the answer and meanwhile poses another question. ProphetChat then uses the history and this first-pass response to simulate the future where the other possible choice (i.e., the bookstore) is talked about. Given the history and the simulated future, ProphetChat finally obtains its response which not only answers the query  in the history but also incorporates the cues in the future. This response becomes more informative than the previous one. A similar phenomenon can also be found in case 2 where the final response is more comprehensive that connects the history and the future smoothly.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
We propose a novel response generation framework that utilizes the simulated dialogue futures in the inference phase to enhance response generation.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
To acquire the dialogue futures, we design an effective beam-search-like roll-out strategy using a history-to-response dialogue generation model and a dialogue selector. To make use of the simulated future, we use the dynamic ensemble of the historyto-response and the future-to-response generation model. Experiment results demonstrate the effectiveness of our proposed method on two popular datasets. In the future, we plan to enable our future simulation method to simulate multiple turns of dialogue futures.
ProphetChat: Enhancing Dialogue Generation with Simulation of Future Conversation
This paper proposes a new dialogue generation framework that utilizes the simulated dialogue futures in the inference phase to enhance the generation of the response. Generative approaches are widely used in a wide range of dialogue applications. The proposed method improves the quality of the generated responses, which could be beneficial to research and real-world applications. The research will not pose ethical issues. This paper doesn't involve any data collection and release thus there are no privacy issues. All the datasets used in this paper are publicly available and are widely adopted by researchers to test the performance of open-domain response generation models. This paper conducts human evaluation to evaluate the quality of the generated responses. Three part-time research assistants were recruited to do human evaluation with clearly demonstrated evaluation rules. They worked with the pay 100 CNY/hour during their evaluation.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
With the availability of massive online conversational data, there has been a surge of interest in building open-domain chatbots with data-driven approaches. Recently, the neural network based sequence-to-sequence (seq2seq) framework (Sutskever et al., 2014;Cho et al., 2014) has been widely adopted. In such a model, the encoder, which is typically a recurrent neural network (RNN), maps the source tokens into a fixed-sized continuous vector, based on which the decoder estimates the probabilities on the target side word by word. The whole model can be efficiently trained by maximum likelihood (MLE) and has demonstrated state-of-the-art performance in various domains. However, this architecture is not * Indicates equal contribution. X. Shen focuses on algorithm and H. Su is responsible for experiments.  suitable for modeling dialogues. Recent research has found that while the seq2seq model generates syntactically well-formed responses, they are prone to being off-context, short, and generic. (e.g., "I dont know" or "I am not sure") (Li et al., 2016a;. The reason lies in the one-to-many alignments in human conversations, where one dialogue context is open to multiple potential responses. When optimizing with the MLE objective, the model tends to have a strong bias towards safe responses as they can be literally paired with arbitrary dialogue context without semantical or grammatical contradictions. These safe responses break the dialogue flow without bringing any useful information and people will easily lose interest in continuing the conversation.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
In this paper, we propose NEXUS Network which aims at producing more on-topic responses to maintain an interactive conversation flow. Our assumption is that a good response should serve as a "nexus": connecting and being informative to both the preceding dialogue context and the follow-up conversations. For example, in Figure  1, the response from B 1 is a smooth connection, where the first half indicates the preceding context is a "Do you know" question and the second half informs that the follow-up would be an introduction about Star Wars. We establish this connection by maximizing the mutual information (MMI) of the current utterance with both the past and future contexts. In this way, generic responses can be largely discouraged as they contain no valuable information and thus have only weak correlations with the surrounding context. To enable efficient training, two challenges exist.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
The first challenge comes from the discrete nature of language tokens, hindering efficient gradient descent. One strategy is to estimate the gradient by methods like Gumbel-Softmax (Maddison et al., 2017;Jang et al., 2017) or REINFORCE algorithm (Williams, 1992, which has been applied in many NLP tasks (He et al., 2016;Shetty et al., 2017;Gu et al., 2018;Paulus et al., 2018), but the trade-off between bias and variance of the estimated gradient is hard to reconcile. The resulting model usually strongly relies on sensitive hyper-parameter tuning, careful pre-train and taskspecific tricks. Li et al. (2016a); Wang et al. (2017) avoid this non-differentiability problem by learning a separate backward model to rerank candidate responses in the testing phase while still adhering to the MLE objective for training. However, the candidate set normally suffers from low diversity and a huge sample size is needed for good performance (Li et al., 2016b).
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
The second challenge relates to the unknown future context in the testing phase. In our framework, both the history and future context need to be explicitly observed in order to compute the mutual information. When applying it to generating tasks where only the history context is given, there is no way to explicitly take into account the future information. Therefore, reranking-based models do not apply here. (Li et al., 2016c) addresses future information by policy learning, but the model suffers from high variance due to the enormous sequential search space. Serban et al. (2017); ; Shen et al. (2017) adopt the variational inference strategy to reduce the training variance by optimizing over latent continuous variables. However, they all stick to the original MLE objective and no connection with the surrounding context is considered.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
In this work, we address both challenges by introducing an auxiliary continuous code space which is learned from the whole dialogue flow. At each time step, instead of directly optimizing discrete utterances, the current, past and future utterances are all trained to maximize the mutual information with this code space. Furthermore, a learnable prior distribution is simultaneously optimized to predict the corresponding code space, enabling efficient sampling in the testing phase without getting access to the ground-truth future con-versation. Extensive experiments have been conducted to validate the superiority of our framework. The generated responses clearly demonstrate better performance with respect to both coherence and diversity.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
2 Model Structure
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Let u i be the ith utterance within a dialogue flow. The dialogue history H i−1 contains all the preceding context u 1 , u 2 , . . . , u i−1 and F i+1 denotes the future conversations u i+1 , . . . , u T . The objective of our model is to find the decoding probability p θ (u i |H i−1 , F i+1 ) that maximizes the mutual information I(H i−1 , u i ) and I(u i , F i+1 ). Formally, the objective is:
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
(1) λ 1 and λ 2 adjusts the relative weight. Mutual information is defined over p θ (u i |H i−1 , F i+1 ) and the empirical distribution p(H i−1 , F i+1 ). Now we assume the future context F i+1 is known to us when training the decoding probability, we will address the unknown future problem later.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Directly optimizing with this objective is unfortunately infeasible because the exact computation of mutual information is intractable, and backpropagating through sampled discrete sequences is notoriously difficult to train. The discontinuity prevents the direct application of the reparameterization trick (Kingma and Welling, 2014). Lowvariance relaxations like Gumbel-Softmax (Jang et al., 2017), semantic hashing (Kaiser et al., 2018) or vector quantization (van den Oord et al., 2017) lead to biased gradient estimations, which are accumulated as the sequence becomes longer. The Monte-Carlo-Simulation is unbiased but suffers from high variances. Designing a reasonable control variate for variance reduction is an extremely tricky task (Mnih and Gregor, 2014;Tucker et al., 2017). For this sake, we propose replacing u i with a continuous code space c learned from the whole dialogue flow.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
We define the continuous code space c to follow the Gaussian probability distribution with a diagonal covariance matrix conditioning on the whole dialogue:
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
The dialogue history H i−1 is encoded into vector H i−1 by a forward hierarchical GRU model E f as in . The future conversation, including the current utterance, is encoded intoF i by a backward hierarchical GRU E b .H i−1 and F i are concatenated and a multi-layer perceptron is built on top of them to estimate the Gaussian mean and covariance parameters. The code space is trained to infer the encoded historyH i−1 and futureF i+1 . The full optimizing objective is:
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
(3) whereH i−1 andF i+1 are also assumed to be Gaussian distributed given c with mean and covariance estimated from multi-layer perceptrons. We infer the encoded vectors instead of the original sequences for three reasons. Firstly, inferring dense vectors is parallelizable and computationally much cheaper than autoregressive decoding, especially when the context sequences could be unlimitedly long. Secondly, sequence vectors can capture more holistic semantic-level similarity than individual tokens. Lastly, It can also help alleviate the posterior collapsing issue (Bowman et al., 2016) when training variational inference models on text (Chen et al., 2017;Shen et al., 2018), which we will use later. It can be shown that the above objective maximizes a lower bound of λ 1 I(H i−1 , c) + λ 2 I(c, F i+1 ), given the conditional probability p φ (c|H i−1 , F i ). The proof is a direct extension of the derivation in , followed by the Data Processing Inequality (Beaudry and Renner, 2012) that the encoding function can only reduce the mutual information. As the sampling process contains only Gaussian continuous variables, the above objective can be trained through the reparameterization trick (Kingma and Welling, 2014), which is a low-variance, unbiased gradient estimator (Burda et al., 2015). After training, samples from p φ (c|H i−1 , F i ) hold high mutual information with both the history and future context. The next step is then transferring the continuous code space to reasonable discrete natural language utterances.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Our decoder transfers the code space c into the ground-truth utterance u i by defining the probability distribution p(u i |H i−1 , c), which is imple-mented as a GRU decoder going through u i word by word to estimate the output probability. The encoded historyH i−1 and code space c are concatenated as an extra input at each time step. The loss function for the decoder is then:
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
(4) which can be proved to be the lower bound of the conditional mutual information I(u i , c|H i−1 ). By maximizing the conditional mutual information, c i is trained to maintain as much information about the target sequence u i as possible.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Combining Eq. 3 and 4, our model until now can be viewed as optimizing a lower bound of the following objective:
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Compared with the original motivation in Eq. 1, we sidestep the non-differentiability problem by replacing u i with a continuous code space c, then forcing u i to contain the same information as maintained in c by additionally maximizing the mutual information between them.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Nonetheless, Eq. 5 and Eq. 1 might lead to different optimums as mutual information does not satisfy the transitive law. In the extreme case, different dimensions of c could individually maintain information about history, current and future conversations and the conversations themselves do not share any dependency relation. To avoid this issue, we restrict the dimension of c to be smaller than that of the encoded vectors. In this case, optimizing Eq. 5 will favor utterances having stronger correlations with the surrounding context to achieve a higher total mutual information.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
The last problem is the sampling mechanism of c in Eq. 2, which conditions on the ground-truth future conversation. In the testing phase, when we have no access to it, we cannot perform the decoding process as in Eq. 4. To allow for decoding with only the history context, we need to learn an appropriate prior distribution p θ (c|H i−1 ) for c. In the ideal case, we would like
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
(6) However, p φ (c|H i−1 ) is intractable as it integrates over all possible future conversations. We apply variational inference on c to maximize the variational lower bound (Jordan et al., 1999):
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
prior I|H i−1 )) (7) It can be reformulated as maximizing:
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
by minimizing the KL divergence between them. It also functions as a regularizer to prevent overfitting when learning p φ (c|H i−1 , F i ). In the testing phase, we can sample c from the learned prior distribution p θ (c|H i−1 ), then generate a response based on it.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
To sum up, the total objective function of our model is:
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Weighting can be added to individual loss functions for better performance, but we find it enough to maintain equal weights and avoid extra hyperparameters. All the parameters are simultaneously updated by gradient descent except for the encoders E f and E b , which only accept gradients from L(d) since otherwise the model can easily learn to encode no information for a lower reconstruction loss in L(c) and L(p). An overview of our training procedure is depicted in Fig. 2.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
3 Relationship to Existing Methods MMI decoding MMI decoder was proposed by (Li et al., 2016a) and further extended in (Wang et al., 2017). The basic idea is the same as our model by maximizing the mutual information with the dialogue context. However, the MMI principle is applied only at the testing phase rather than the training phase. As a result, it can only be used to evaluate the quality of a generation by estimating its mutual information with the context. To apply it in a generative task, we have to first sample some candidate responses with the seq2seq model, then rerank them by accounting for the MMI score. Our model differs from it in that we directly estimate the decoding probability thus no post-sampling rerank is needed. Moreover, we further include the future context to strengthen the connection role of the current utterances.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
The idea of learning an appropriate prior distribution in Eq. 7 is essentially a conditional variational autoencoder (Sohn et al., 2015) where the accumulated posterior distribution is trained to stay close to a prior distribution. It has also been applied in dialogue generation (Serban et al., 2017;. However, all the above methods stick to the MLE objective function and do not optimize with respect to the mutual information.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
As we will show in the experiment, they fail to learn the correlation between the utterance and its surrounding context. The generation diversity of these models comes more from the sampling randomness of the prior distribution rather than from the correct understanding of context correlation. Moreover, they suffer from the posterior collapsing problem (Bowman et al., 2016) and require special tricks like KL-annealing, BOW loss or word drop-out (Shen et al., 2018). Our model does not have such problems.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Deep Reinforcement Learning Dialogue Generation (Li et al., 2016c) first considered future success in dialogue generation and applied deep reinforcement learning to encourage more interactive conversations. However, the reward functions are intuitively hand-crafted. The relative weight for each reward needs to be carefully tuned and the training stage is unstable due to the huge search space. In contrast, our model maximizes the mutual information in the continuous space and trains the prior distribution through the reparamaterization trick. As a result, our model can be more easily trained with a lower variance. Throughout our experiment, the training process of NEXUS network is rather stable and much less data-hungry. The MMI objective of our model is theoretically more sound and no manually-defined rules need to be specified.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
We run experiments on the DailyDialog (Li et al., 2017b) and Twitter corpus (Ritter et al., 2011). DailyDialog contains 13118 daily conversations under ten different topics. This dataset is crawled from various websites for English learner to practice English in daily life, which is high-quality, less noisy but relatively smaller. In contrast, the Twitter corpus is significantly larger but contains more noise. We obtain the dataset as used in Serban et al. (2017) and filter out tweets that have already been deleted, resulting in about 750,000 multi-turn dialogues. The contents have more informal, colloquial expressions which makes the generation task harder. These two datasets are randomly separated into training/validation/test sets with the ratio of 10:1:1.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
In order to keep our model comparable with the state-of-the-art, we keep most parameter values the same as in (Serban et al., 2017). We build our vocabulary dictionary based on the most frequent 20,000 words for both corpus and map other words to a UNK token. The dimensionality of the code space c is 100. We use a learning rate of 0.001 for DailyDialog and 0.0002 for Twitter corpus. The batch size is fixed to 128. The word vector dimension is 300 and is initialized with the public Word2Vec (Mikolov et al., 2013) embeddings trained on the Google News Corpus. The probability estimators for the Gaussian distributions are implemented as 3-layer perceptrons with the hyperbolic tangent activation function. As mentioned above, when training NEXUS models, we block the gradient from L(c) and L(p) with respect to E f and E b to encourage more meaningful encodings. The UNK token is prevented from being generated in the test phase. We implemented all the models with the open-sourced Python library Pytorch (Paszke et al., 2017) and optimized using the Adam optimizer (Kingma and Ba, 2015).
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
We conduct extensive experiments to compare our model against several representative baselines.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Seq2Seq: Following the same implementation as in (Vinyals and Le, 2015), the seq2seq model serves as a baseline. We try both greedy decoding  Table 2 and beam search (Graves, 2012) with beam size set to 5 when testing. MMI: We implemented the bidirectional-MMI decoder as in Li et al. (2016a), which showed better performance over the anti-LM model. The hyperparameter λ is set to 0.5 as suggested. 200 candidates per context are sampled for re-ranking.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
VHRED: The VHRED model is essentially a conditional variational autoencoder with hierarchical encoders (Serban et al., 2017;. To alleviate the posterior collapsing problem, we apply the KL-annealing trick and early stop with the step set as 12,000 for the DailyDialog and 75,000 for the Twitter corpus.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
RL: Deep reinforcement learning chatbot as in (Li et al., 2016c). We use all the three reward functions mentioned in the paper and keep the relative weights the same as in the original paper. Policy network is initialized with the above-mentioned MMI model.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
NEXUS-H: NEXUS network maximizing mutual information only with the history (λ 2 = 0). NEXUS-F: NEXUS network maximizing mutual information only with the future (λ 1 = 0). NEXUS: NEXUS network maximizing mutual information with both the history and future.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
NEXUS-H and NEXUS-F are implemented to help us better analyze the effects of different components in our model. The hyperparameters λ 1 and λ 2 in NEXUS are set to be 0.5 and 1 respectively as we find history vector is consistently easier to be reconstructed than the future vector (A.6).
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Embedding Score We conducted three embedding-based evaluations (average, greedy and extrema) , which map responses into vector space and compute the cosine similarity (Rus and Lintean, 2012). The embedding-based metrics can to a large extent capture the semantic-level similarity between generated responses and ground truth. We represent words using Word2Vec embeddings trained on the Google News Corpus. We also measure the uncertainty of the score by assuming each data point is independently Gaussian distributed. The standard deviation yields the 95% confidence interval (Barany et al., 2007). Table 1 reports the embedding scores on both datasets. NEXUS network significantly outperforms the best baseline model in most cases. Notably, NEXUS can absorb the advantages from both NEXUS-H and NEXUS-F. The history and future information seem to help the model from different perspectives. Taking into account both of them does not create a conflict and the combination leads to an overall improvement. RL performs rather poorly on this metric, which is understandable as it does not target the ground-truth responses during training (Li et al., 2016c).
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
BLEU Score BLEU is a popular metric that measures the geometric mean of the modified ngram precision with a length penalty (Papineni et al., 2002). Table 2 reports the BLEU 1-3 scores. Compared with embedding-based metrics, the BLEU score quantifies the word-overlap between generated responses and the ground-truth. One challenge of evaluating dialogue generation by BLEU score is the difficulty of accessing multiple references for the one-to-many alignment relation. Following Sordoni et al. (2015); Zhao et al.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
DailyDialog Twitter BLEU-1 BLEU-2 BLEU-3 BLEU-1 BLEU-2 BLEU-3  (Lin and Och, 2004). p-value interval is computed base on the altered bootstrap resampling algorithm (Riezler and Maxwell, 2005) (2017); Shen et al. (2018), for each context, 10 more candidate references are acquired by using information retrieval methods (see Appendix A.4 for more details). All candidates are then passed to human annotators to filter unsuitable ones, resulting in 6.74 and 5.13 references for DailyDialog and Twitter dataset respectively. The human annotation is costly, so we evaluate it on 1000 sampled test cases for each dataset. As the BLEU score is not the simple mean of individual sentence scores, we compute the 95% significance interval by bootstrap resampling (Koehn, 2004;Riezler and Maxwell, 2005). As can be seen, NEXUS network achieves best or near-best performances with only greedy decoders. NEXUS-H generally outperforms NEXUS-F as the connection with future context is not explicitly addressed by the BLEU score metric. MMI and VHRED bring minor improvements over the seq2seq model. Even when evaluated on multiple references, RL still performs worse than most models.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Connecting the preceding We define two metrics to evaluate the model's capability of "connecting the preceding context": AdverSuc and Neg-PMI. AdverSuc measures the coherence of generated responses with the provided context by learning an adversarial discriminator (Li et al., 2017a) on the same corpus to distinguish coherent responses from randomly sampled ones. We encode the context and response separately with two different LSTM neural networks and output a binary signal indicating coherent or not 1 . The Adver-Suc value is reported as the success rate that the model fools the classifier into believing its false generations (p(generated = coherent) > 0.5).
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Neg-PMI measures the negative pointwise mutual information value − log p(c|r)/p(c) between the generated response r and the dialogue context c. p(c|r) is estimated by training a separate backward seq2seq model. As p(c) is a constant, we ignore it and only report the value of − log p(c|r). A good model should achieve a higher Adver-Suc and a lower Neg-PMI. The results are listed in Table 3. We can see there is still a big gap between ground-truth and synthesized responses. As expected, NEXUS-H leads to the most significant improvement. MMI model also performs remarkably well, but it requires post-reranking thus the sampling process is much slower. VHRED and NEXUS-F do not help much here, sometimes even slightly degrade the performance. We also tried removing the history context when computing the posterior distribution in VHRED, the resulting model has similar performance among all metrics, which suggests VHRED itself cannot actually learn the correlation pattern with the preceding context. Surprisingly, though RL explicitly set the coherence score as a reward function, its performance is far from satisfying. We assume RL requires much more data to learn the appropriate policy than other models and the training process suffers from a higher variance. The result is thus hard to be guaranteed.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Connecting the following We measure the model's capability of "connecting the following context" from two perspectives: number of the simulated turns and diversity of generated re-  sponses. We apply all models to generate multiple turns until a generic response is reached. The set of generic responses is manually examined to include all utterances providing only passive dull replies 2 . The number of generated turns can reflect the time that a model can maintain an interactive conversation. The results are reflected in the #Turns column in Table 3. As in (Li et al., 2016a), we measure the diversity by the percentage of distinct unigrams (Distinct-1) and bigrams (Distinct-2) in all generated responses. Intuitively a higher score on these three metrics implies a more interactive generation system that can better connect the future context. Again, NEXUS network dominates most fields. NEXUS-F brings more impact than NEXUS-H as it explicitly encourages more interactive turns. Most seq2seq models fail to provide an informative response in the first turn. The MMI-decoder does not change much, possibly because the sampling space is not large enough, a more diverse sampling mechanism (Vijayakumar et al., 2018) might help. NEXUS network can effectively continue the conversation for 2.8 turns for DailyDialog and 2.5 turns for Twitter, which is closest to the ground truth (4.8 and 4.0 turns respectively). It also achieves the best diversity score in both datasets. It is worth mentioning that NEXUS-H also improves over baselines, though not as significantly as NEXUS-F, so NEXUS is not a trade-off but more like an enhanced version from NEXUS-H and NEXUS-F.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
In summary, NEXUS network clearly generates higher-quality responses in both coherence and diversity, even in a rather small dataset like Daily-Dialog. NEXUS-H contributes more to the coher- 2 We use a simple rule matching method (see Appendix A.5). We manually inspect it on a validation subset and find the accuracy is more than 90%. Similar methods are adopted in (Li et al., 2016c). ence and NEXUS-F more to the diversity.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
We also employed crowdsourced judges to provide evaluations for a random sample of 500 items in the DailyDialog test dataset. Participants are asked to assign a binary score to each contextresponse pair from three perspectives: whether the response coincides with its preceding context (Pri), whether the response is interesting enough for people to continue (Post) and whether the response itself is a fluent natural sentence (Flu). Each sample gets one point if judged as yes and zero otherwise. Each pair is judged by three participants and the score supported by most people is adopted. We also evaluated the inter-annotator consistency by Fleiss'k score (Fleiss, 1971) and obtained k scores of 0.452 for Pri, 0.459 for Post (moderate agreement) and 0.621 for Flu (substantial agreement), which implies most contextresponse pairs reach a consensus on the evaluation task. We compute the average human score for each model. Unlike metric-based scores, the human evaluation is conducted only on the DailyDialog corpus as it contains less noise and can be more fairly evaluated by human judges. Table 3 shows the result in the last three columns. As can be seen, the pri and post human scores are highly correlated with the automatic evaluation metric "coherence" and "#turns", verifying the validity of these two metrics. As for fluency, there is no significant difference among most models. As we also manually examined, fluency is not a major problem and all models produce mostly well-formed sentences. Overall, NEXUS network does produce responses that are more acceptable to human judges.  VHRED, RL and NEXUS model. We see NEXUS network does generate more interactive outputs than the other three. Though reranked by the bidirectional language model, the MMI decoder still produces quite a few generic responses. VHRED's utterances are more diverse, but it only cares about answering to the immediate query and makes no efforts to bring about further topics. Moreover, it also generates more inappropriate responses than the others. RL provides diverse responses but sometimes not fluent or coherent enough. We do observe that NEXUS sometimes generate over-complex questions which are not very natural, as in the second example. But in most cases, it outperforms the others.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
In this paper, we propose "NEXUS Network" to enable more interactive human-computer conversations. The main goal of our model is to strengthen the "nexus" role of the current utterance, connecting both the preceding and the following dialogue context. We compare our model with MMI, reinforcement learning and CVAEbased models. Experiments show that NEXUS network consistently produces higher-quality re-sponses. The model is easier to train, requires no special tricks and demonstrates remarkable generalization capability even in a very small dataset.
NEXUS Network: Connecting the Preceding and the Following in Dialogue Generation
Our model can be considered as combining the objective of MMI and CVAE and is compatible with current improving techniques. For example, mutual information can be maximized under a tighter bound using Donsker-Varadhan or f-divergence representation (Donsker and Varadhan, 1983;Nowozin et al., 2016;Belghazi et al., 2018). Extending the code space distribution to more than Gaussian by importance weighted autoencoder (Burda et al., 2015), inverse autoregressive flow (Kingma et al., 2016) or VamPrior (Tomczak and Welling, 2018) should also help with the performance.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Dialogue generation achieves great success in recent years. In this task, a dialogue session lasts for multiple turns, and the goal is to predict the response to the previous context. Researchers have explored various aspects of dialogue generation models including context-aware (Tian et al., 2017), persona-based (Qian et al., 2018) and knowledgegrounded Tao et al., 2021), etc. Because dialogue generation is a context-aware process, there is a large body of literature on the context modeling among all research lines.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
In the pilot studies, contexts are modeled as the concatenation of previous utterances (Sordoni et al., Figure 1: (a) Inter-duality: the dialogue is divided into two parts. According to the first two utterances, we can infer that the follow-up part is related to the scenic spot commodities. Similarly, based on the last two utterances, we can infer that the previous conversation is about tourism. (b) Intra-duality: in an utterance, given the blue part, people know that the follow-up part is the reason. Given the pink part, people know that the previous tokens may explain the result of the argument. 2015), organized in a non-hierarchical way. A dialogue is later considered to have hierarchical characteristics in which both token semantics within an utterance and logic between utterances need consideration (Serban et al., 2016). The hierarchical context modeling among utterances can be further improved with a global hidden variable (Serban et al., 2017) indicating topic, emotion, or persona. At the same time, in addition to modeling text at multiple hierarchies, some researchers have also committed to mining the intensive interaction among the previous utterances sufficiently (Tao et al., 2019;Wu et al., 2017). In summary, most context modeling approaches focus on the previous utterances and ignore the followup utterances. In recent years, researchers (Shen et al., 2018;Feng et al., 2020;Liu et al., 2022) realize that a response of high quality should not only be related to dialogue history but also connect the follow-up parts. However, they only treat utterances after the target response (i.e., dialogue future) as auxiliary information to encode during training, which violates a human cognitive ability in daily conversations: given a dialogue segment, people can infer future contents; also, people can imagine what the interlocutors discussed before. In the whole process, there are interactions between visible dialogue parts and human-envisioned parts. We name the property in dialogue texts that humans take advantage of to infer unknown previous or follow-up parts as duality 1 . Inadequate consideration of duality ignores the interaction between dialogue history and the future, resulting in underutilized future information.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Duality in dialogues has two key characteristics. Firstly, as a property in dialogues, duality can be viewed from a hierarchical viewpoint. At token hierarchy, intra-utterance duality (abbreviated as intra-duality) is reflected in that preceding words and the subsequent words can infer each other. Similarly, at utterance hierarchy, inter-utterance duality (abbreviated as inter-duality) is reflected in that utterances before and after the target response can infer each other. Figure 1(a) and (b) illustrate dualities at two hierarchies. Secondly, due to the contextaware nature of dialogues, intra-duality does not play a positive role alone and must be tightly integrated with inter-duality.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
In this paper, from the point of view of dialogue duality, we propose the Hierarchical Duality Learning for Dialogue (HDLD) in order to fully exploit dialogue text for better generation. In HDLD, a dialogue session is divided into two parts: past and future. At the utterance hierarchy, there is a forward generation model and a backward generation model working in a dual learning framework. The forward model takes the past utterances as context and predicts utterances in the future part. Predicted outputs can be used as context for the backward model to back-predict the past utterances. The backward model performs a dual process. To integrate dualities at two hierarchies, HDLD employs a cross-hierarchy distillation mechanism: guided by both future and past contexts, we distill both context-related and intra-dual knowledge from the ground truth, then we use the distilled knowledge to train the models. Through this cross-hierarchy distillation mechanism, HDLD couples hierarchies tightly, improving generation quality at multiple granularities. We design a two-stage joint optimization of the forward and the backward model that maximizes the mutual information between the past and future. Thus, during inference even if future information is invisible, HDLD is capable of envisioning future information implicitly only based on the dialogue history and generates both coherent and informative responses.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
We conduct multi-turn dialogue generation experiments on two public datasets, DailyDialog  and OpenSubtitles (Lison and Tiedemann, 2016). Compared with previous works that also consider future utterances, exploiting dialogue duality helps better utilize dialogue data, leading to the improvement in both automatic and human evaluation. To sum up, our contributions can be summarized as follows:
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
• We first define the dialogue duality, which was widely neglected in previous works. We propose the hierarchical duality learning framework for dialogue (HDLD) to augment multi-turn dialogue generation by exploiting duality.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
• To realize HDLD, We design a cross-hierarchy distillation mechanism to couple hierarchies, and a joint optimization of two models that can be theoretically proved to augment generation.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
• We show the effectiveness of HDLD in generating coherent and informative responses.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
In open-domain dialogue generation, the focus of this paper, most models are context-aware, predicting the target response based on previous context representations. With the development of this field, researchers began to pay attention to the future context: NEXUS (Shen et al., 2018) introduces a posterior distribution for estimating the future information as a hidden variable and decodes responses accordingly. RegDG (Feng et al., 2020) trains a teacher model with both past and future utterances and transfers the whole-context knowledge to a student model which has only access to past utterances. ProphetChat (Liu et al., 2022) first predicts two utterances forward, generating multiple responses and future utterances. With a selector, the best response and future utterance are selected. A final response is squeezed by both history and future. Compared with HDLD, there is inadequate consideration of dialogue duality in these works.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
They ignore the explicit interaction between the dialogue history and future, still resulting in underutilization of future information.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Dual learning is proposed to boost neural machine translation (He et al., 2016). In the dual learning framework, one agent represents the model for the primal task, which aims to build a mapping from domain X to domain Y , and the other agent represents the model for the dual task, which aims to build a mapping from domain Y back to domain X. Based on the training feedback signals generated from each other, two models are updated iteratively until convergence. By leveraging the primal-dual structure of two tasks, the training and inference of the primal and dual tasks are improved. Many dialogue generation methods (Yang et al., 2018;Cui et al., 2019;Li et al., 2021) have mentioned 'duality' in their paper but note that there are differences in our definition of 'duality' from theirs. Duality describes the relationship between dual and primal tasks (e.g., English-to-Chinese translation and Chinese-to-English translation) in previous works. By contrast, in this paper, duality refers to a unique dialogue property that humans can take advantage of to infer dialogue future (or history) according to dialogue history (or future).
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
It is important to note the distinction between duality and "bi-directional" models such as Bi-RNN (Schuster and Paliwal, 1997) or MLM like BERT (Devlin et al., 2019). While these models extract features in two directions, they are not "dual." In these models, the backward features are typically merged with the forward features, rather than actively enhancing forward feature extraction as in dual learning approaches.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
We first present the preliminary in § 3.1. Then, we introduce a two-stage optimization that exploits dualities at two hierarchies in § 3.2, and some important details during training and inference in § 3.3.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Given a multi-turn dialogue D = {u 1 , . . . , u n } with n > 2 utterances. An utterance
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Take t-th utterance as the dividing point, we divide D into two parts: the past contexts
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Cross-hierarchy Distillation Figure 2: The concept of HDLD. Two models form a horizontal dual cycle at utterance hierarchy. Crosshierarchy distillation forms a vertical dual cycle to couple hierarchies.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
In HDLD, there are a forward generation model f xy : P → F and its dual generation model, a backward model f yx : F → P. In each model, there are one transformer encoder and two transformer decoders. At the utterance hierarchy, f xy takes a x 1:t ∈ P as context to predict utterances in y n:t+1 , and f yx takes the corresponding y n:t+1 ∈ F as context to predict utterances in x 1:t , which forms the horizontal cycle in Figure 2. Two decoders in each model process in dual directions to extract intra-duality features in utterances, and cross-hierarchy distillation integrates hierarchies: guided by both future and past contexts (top-down), we distill knowledge that is both context-related and intra-dual. Distilled knowledge is used to teach models (bottom-up), which forms the vertical cycle in Figure 2. Figure 3 shows the HDLD architecture. In the following description, we remove the subscript of x or y when there is no ambiguity.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
As a dual learning framework, our ultimate goal is to obtain the forward model f xy which generates response based on dialogue history while the f yx is auxiliary. Given any aligned x ∈ P and y ∈ F, to leverage inter-duality, we apply the two models consecutively, completing the horizontal dual cycle. This process yields four outputs:
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
, and y ′′ = f xy (x ′ ). Reconstruction errors between x ′′ and x, and between y ′′ and y reflect how well the horizontal dual cycle is completed, i.e., the utility of inter-duality. We Embedding Layer (1)
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
However, it is challenging to directly optimize the final output x ′′ and y ′′ , which is hindered by the error accumulation in practice. Thus, we find the upper bound of Eq. 1 which is more practical. Take the logarithmic probability − log p(x ′′ = x|x; f xy , f yx ) as an example, we have:
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
The inequality is derived from two facts: (1) x ′′ is only directly related to y ′ and f yx and y ′ is only directly related to x and f xy and (2) the concavity of logarithm. Therefore, we can derive the upper bound of Eq. 1:
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
(3)
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
We can optimize the upper bound by simultaneously minimizing each probability term within it. This means that in order to exploit inter-duality, it is not necessary to complete the horizontal dual cycle by consecutively applying two models and directly optimizing the final outputs. Instead, we can optimize the intermediate outputs, which divides the training into two stages. In the first stage, f xy generates y ′ and f yx generates x ′ . We fit y ′ and x ′ to their ground truth y and x, respectively. In the second stage, f xy predicts y ′′ and f yx predicts x ′′ , which fit to y and x again, respectively. Formally, we minimize the negative log-likelihoods:
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
(4)
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Another advantage of the two-stage optimization is that it allows for easier consideration of intraduality by assigning the top-down and bottom-up parts of the cross-hierarchy distillation to each stage. We delve into the detailed implementation of Eq. 4 in each stage.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Stage One. In the first stage, we apply both models once, and utilize top-down distillation to distill integrated hierarchical duality knowledge. However, reconstructing the entire dialogue history x and future response y can be challenging due to their often lengthy nature, consisting of multiple sentences. To address this challenge, we relax the constraint and allow each model to reconstruct only one subsequent utterance in the other part of the dialogue. For example, in the case of f xy , x 1:t serves as the context and u t+1 is the target, while for f yx , y n:t+1 acts as the context and u t is the target. Both models encode their respective contexts and pass the encoder output to their respective main decoders. The generated responses from the main decoders in the first stage are denoted as u ′ t+1 and u ′ t . To optimize f xy and f yx , we fit their outputs to the ground truth u t+1 and u t by minimizing the negative log-likelihoods:
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
The dual-view decoder in each model operates on the same encoder outputs as the main decoder but processes them in a reverse direction, which enables us to leverage intra-duality, as the cross attention mechanism in decoders limits tokens from attending to their subsequent tokens. To achieve this, we flip the attention mask used in the main decoder along the main diagonal to create the mask for the dual-view decoder. The generated responses from the dual-view decoders in the first stage are represented asũ ′ t+1 andũ ′ t . Figure 3(a) provides an illustration of the behaviors of both the main decoder and the dual-view decoder.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Next, we introduce the top-down part of the cross-hierarchy distillation. Denote the mixture of outputs from the main decoder and the dual-view decoder as intra-dual features I:
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
where W I ,W I ∈ R d×d , the ⊗ denotes Hadamard product, and σ denotes the sigmoid function. e · · denotes hidden state embeddings of the corresponding u · · . The representations of the entire dialogue, denoted as E D , play a crucial role in distilling both context-related and intra-dual knowledge from the intra-dual features I using the multi-head attention mechanism. In this mechanism, I serves as both the key and value, while E D acts as the query. The output of the multi-head attention, denoted as A, is then mapped to match the vocabulary size |V |, resulting in the final knowledge representation K. The detailed process is illustrated in Figure 3(c), where the linear layer weight is represented by W A of size R |V |×d :
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Stage Two. In the second stage, we use predicted responses in stage one to construct pseudo contexts and based on which, two models are applied again to complete the horizontal dual cycle. Meanwhile, the bottom-up distillation is carried out to complete the vertical dual cycle.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
As shown in Figure 3(b), we substitute e ′ t and e ′ t+1 obtained in the previous stage for e t and e t+1 in embedded x 1:t and y n:t+1 , respectively. By utilizing e ′ t and e ′ t+1 to construct pseudo contexts, we preserve the gradient flow, allowing the joint optimization of the two models. These modified context representations can be referred to as embedded pseudo contexts, denoted as x ′ 1:t and y ′ n:t+1 . Next, f xy and f yx predict each one's target again based on respective pseudo contexts. We denote the generated responses from main decoders in stage two as u ′′ t+1 and u ′′ t , respectively. Instead of using discrete ground truth sentences, we employ more informative intra-dual knowledge K x and K y to guide the decoding process of the two models. To enhance the suitability of the knowledge as labels, we further augment the target word probability in the soft distributions K x and K y , following the method proposed by (Wang et al., 2021). The guidance of the decoding process is achieved through bottom-up distillation, where we minimize the Kullback-Leibler divergence (KL) (Kullback and Leibler, 1951) between the outputs of the main decoders in both stages and the knowledge representation:
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Moreover, we optimize both the dual-view decoder outputs and knowledge to be similar to ground truth in semantics:
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
(9) For clear demonstration, we use red arrow to indicate all loss calculations in Figure. 3.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
During training, the training loss function L is defined as follows:
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
During inference, only the encoder and the main decoder of f xy remain active. Therefore, HDLD does not incur any additional inference costs than a vanilla transformer. It is feasible because to consider the mutual information I(P, F) between P and F: 
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
In Eq. (11), p(x, y) is constant and is decided by data, and HDLD maximizes p(x|y) and p(y|x) according to Eq. (4) through the joint optimization.  Thus, Eq. 11, the mutual information between dialogue history and future is maximized. Given the past context, HDLD is able to implicitly envision future information with the maximized mutual information.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Baselines We compare HDLD with baselines in two groups. The first group introduces future utterances to enhance the model: NEXUS (Shen et al., 2018) introduces a code space learned from the whole dialogue and samples from the code space during testing to estimate future context. RegDG (Feng et al., 2020) trains a teacher model that has access to the whole dialogue to teach a student model which only takes past utterances as input. ProphetChat (Liu et al., 2022) first predicts two steps forward, generating multiple responses and future utterances, which are used to squeeze the final response along with the dialogue history.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
The second group considers the relationship among tokens in an utterance comprehensively: D2GPO  augments the training with a data-dependent Gaussian prior distribution, which is generated in pre-procession based on the training set. AdaLabel (Wang et al., 2021) introduces an auxiliary decoder that uses a bidirectional attention to dynamically estimate a token distribution at each time step. Besides, we also compare to HRED (Serban et al., 2016), a pilot work for hierarchical context modeling, and Transformer (Vaswani et al., 2017), the backbone of AdaLabel, D2GPO and HDLD.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Datasets We conduct multi-turn dialogue generation experiments on two public datasets: Daily-Dialog  and OpenSubtitles (Lison and Tiedemann, 2016). DailyDialog contains high quality multi-turn dialogues collected from daily conversations while OpenSubtitles contains dialogues from movies. As a preprocessing step, we exclude dialogue sessions shorter than 4 utterances from the dataset, and we create samples where the contexts consist of no more than 100 tokens. We split a pre-processed dialogue into two parts: dialogue history and future. Baselines that utilize future utterances have access to two parts and those do not utilize future utterances only take the historical part as contexts. In this way, we ensure all experiments have access to the same amount of training samples, which is a fair setting. Due to our unique task settings, we cannot directly cite any numbers from other papers and we reproduced every baseline. Detailed datasets statistics are in Table 1.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Hyper-parameters We use grid-search to select the best hyper-parameters. The search ranges for learning rate and batch size are {0.00006, 0.00008, 0.00010, 0.00012} and {64, 128, 256, 520, 720, 1000}, respectively. The adopted learning rate for HDLD is 0.00008 and the mini-batch size is 128 on DailyDialog and 720 on OpenSubtitles. As for other important settings: the warmup steps are 4000. We use Adam optimizer with β = (0.9, 0.98). Both attention dropout and activation dropout are 0.1. We select the best parameters based on perplexity on the validation set. All experiments adopt greedy decoding. Early stop patience is 10 epochs on DailyDialog and 5 on OpenSubtitles. Some baseline-specific hyper-parameters not mentioned follow the original papers. The training is performed on two A40 GPUs.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
We consider three automatic evaluation metrics: BLEU (Papineni et al., 2002), Distinct (Dist) (Li et al., 2016), and BLEURT (Sellam et al., 2020). BLEU score measures the word overlap between generated responses and the ground truth. Dist score measures the ratio of unique n-grams in the generated responses. Because these two metrics are only sensitive to lexical variation, we evaluate BLEURT, an advanced learned semantic-sensitive evaluation metric.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
For human evaluation, we recruit five evaluators to manually judge 500 samples from all models in blind testing. All responses are re-capitalized and de-tokenized fairly. Evaluators rate samples from three aspects: readability (Read.): how readable the response is; coherency (Coh.): whether a response is coherent with the context; and informative (Info.): whether the response is informative enough. In each aspect, the evaluator can score at '0' for bad, '1' for borderline, or '2' for good.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
The salary for each evaluator is 1 dollar per 10 samples. To give a fair salary, we first evaluate 50 samples by ourselves, calculate the time and effort, and set this amount (samples evaluated by ourselves are just for evaluating the salary, which is not given to evaluators and not reported in the final results).
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Comparison with Prior Works Table 2 shows the automatic evaluation results. HDLD outperforms all baselines on all metrics. HDLD doubles scores on most metrics of baselines using no future information, i.e., Transformer and HRED, and we gain improvement by a large margin than other strong baselines. Thanks to exploited interduality, HDLD connects the past and future contexts so it matches references best (highest BLEU scores). Related to both the past and future contexts, there are few general and meaningless responses so HDLD achieves the highest diversity scores. Except above lexical metrics, we also achieve the highest semantic-sensitive BLEURT scores. From automatic evaluation, our motivation of making responses meaningful and specific by utilizing hierarchical duality is proved effective.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Human evaluation results are shown in Table 4. The highest readability shows that intra-duality in an utterance is leveraged and thus responses are more natural. The highest coherency and informative scores demonstrate that HDLD makes full use of dialogue texts and has the best context modeling capability, resulting in responses connecting and related to both past and future contexts. Overall, our model generates the highest-quality responses.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
We study three variants of HDLD by adding following approaches to a vanilla transformer: (1) +Inter.: utilizing inter-duality, i.e., let two transformer models form a horizontal dual cycle; (2) +Intra.: utilizing intra-duality, i.e., adding a dual-view decoder in a vanilla transformer to obtain knowledge for teaching the model itself;
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
(3) +Inter. + Intra.: combining previous two approaches but do not combine two hierarchies with cross-hierarchy distillation; In variant (1), there is no way to distill knowledge so we minimize the negative log-likelihood in two stages. Table 3 shows results. On two datasets, the inter-duality approach improves performance and outperforms the first group baselines except ProphetChat in Table 2. By only using intra-dual features, or by combining two duality approaches without cross-hierarchy distillation, most metrics are negatively affected. It reveals that without inter-duality, intra-duality does    not work well in dialogues, which confirms our motivation of coupling hierarchies. Finally, HDLD significantly improves the performance on almost all metrics. The study reveals that (1) coupled hierarchical duality is effective, and (2) cross-hierarchy distillation is necessary and non-trivial. Further, we provide some cases to demonstrate the impact of dualities. For inter-duality, we demonstrate a case to illustrate the effectiveness of HDLD in Table 5. Although only the past context is visible to HDLD, it still generates the response which connects the past and the future utterances best. On the contrary, responses from baselines are either unrelated to the dialogue topic or can not lead to future conversation. For intra-duality, we visual-  ize trained embedding layers of Transformer, Transformer with inter-duality approach, and HDLD in Figure 4. Comparing three sub-figures, we find that without considering intra-duality, token embeddings tend to be optimized to be indistinguishable. The result is: representations are similar whatever the inputs are, so the model prefers outputting certain specific tokens, which makes responses less diverse. Only exploiting inter-duality has little effect on token representations which are fine-grained and of low-hierarchy. By contrast, in HDLD, tokens are much more distinguishable. Because semantic information can be represented better, the context modeling ability is improved and generated responses are more diverse and meaningful. Analysis of Length Intuitively, the ratio between the length of the past context and that of the future context has an impact on the performance. We analyze this ratio. In the above experiments, we randomly divide the dialogue session into two parts to obtain as many training samples as possible. In this study, we keep the number of training samples constant by reusing randomly divided training data while the ratio constraint is satisfied by masking redundant utterances. Because dialogues in Open-Subtitles are too short to support this ablation, we only report results on DailyDialog in Figure 5. We report the variation of four metrics and for ease of comparison, we also report the performance of AdaLabel, RegDG, and D2GPO as dashed horizontal lines. It is clear that random data division performs best. Although constraining the ratio to constant hurts performance, it still outperforms all baselines. According to this study, in different dialogue sessions, responses depend to a different degree on the contexts of the past and the future. Therefore, it is better to fully utilize data without such a ratio constraint.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
This paper first augments the dialogue generation task with hierarchical duality in dialogue text which was widely ignored. We propose a hierarchical dual learning framework HDLD to utilize both interduality and intra-duality, and couple them tightly through the cross-hierarchy distillation mechanism.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Both automatic experiments and human judgment demonstrate the effectiveness of hierarchical duality in generating dialogues, which helps HDLD outperform strong baselines. In the future, more under-utilized properties except for duality in dialogues are to be explored. We will further explore the hierarchical characteristics in natural language.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Since there are two transformer-based models in HDLD, the major limitation of HDLD is the training cost. To address this limitation, a potential improvement direction is to fuse the inter-duality approach into one transformer, e.g., sharing parameters between the f xy and f yx . However, we find the significant performance drop when doing so. In the future, the trade-off between performance and cost needs more attention. Besides, like most generative models, if there is malicious information in data, there is no guarantee to avoid bad responses to users, which is a potential risk that needs to pay attention.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
A For every submission:
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
A1. Did you describe the limitations of your work?
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Following instructions, we add Limitations after Conclusion.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
A2. Did you discuss any potential risks of your work?
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
In Limitations.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
A3. Do the abstract and introduction summarize the paper's main claims?
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
The main claims in the paper are stated in the abstract and in the introduction.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
A4. Have you used AI writing assistants when working on this paper?
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
Left blank.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
We use open source code and public dataset.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
We have cited all datasets we use. We have cited open pre-trained models.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
We discuss in appendix A.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In appendix B, we report the dataset statistics.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
In Experiments.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? In terms of parameters, we report the bone architecture of most baselines: transformer with 6 encoder layers and 6 decoder layers, which has certain amount of parameters. In appendix A, we report GPU version. In Limitations, we report computational budget.
Envisioning Future from the Past: Hierarchical Duality Learning for Multi-Turn Dialogue Generation
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.