What cognition-based factors have been used as inputs for the empathetic response generation modeling task?
Rashkin et al. (2019) firstly formulate the empa-
thetic response generation task and construct the
EMPATHETICDIALOGUE dataset. Existing works
that focus on this task can be divided into two lines.
The first is to detect and utilize the user’s emotion
with diverse structures (Lin et al., 2019; Majumder
et al., 2020; Shen et al., 2021). The second is to
consider cognition-based factors other than emo-
tions (EM), such as dialogue act (DA) (Welivita
and Pu, 2020), communication mechanism (CM)
(Sharma et al., 2020), emotion cause (Jiang et al.,
2019), psychological skill (Kim et al., 2021), and
commonsense (Sabour et al., 2021; Li et al., 2022a).
Zheng et al. (2021) propose a framework CoMAE
to model the relationship among CM, DA, and EM
at the utterance level. The differences between Co-
MAE and DIFFUSEMP are: (1) Instead of predict-
ing each factor based on the context representation,
DIFFUSEMP explicitly uses control signals that are
highly related to a response as task input. (2) We
achieve the elaborate control with multi-grained
signals, i.e., tokens in response are influenced by
different signals, while CoMAE applies the same
combined factor to all decoding positions.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Sequence-to-Sequence (Seq2Seq) models have achieved encouraging performance on the dialogue response generation task. However, existing Seq2Seq-based response generation methods suffer from a low-diversity problem: they frequently generate generic responses, which make the conversation less interesting. In this paper, we address the low-diversity problem by investigating its connection with model over-confidence reflected in predicted distributions. Specifically, we first analyze the influence of the commonly used Cross-Entropy (CE) loss function, and find that the CE loss function prefers high-frequency tokens, which results in low-diversity responses. We then propose a Frequency-Aware Cross-Entropy (FACE) loss function that improves over the CE loss function by incorporating a weighting mechanism conditioned on token frequency. Extensive experiments on benchmark datasets show that the FACE loss function is able to substantially improve the diversity of existing state-of-the-art Seq2Seq response generation methods, in terms of both automatic and human evaluations.
[{"first":"Shaojie","last":"Jiang","middle":[]},{"first":"Pengjie","last":"Ren","middle":[]},{"first":"Christof","last":"Monz","middle":[]},{"first":"Maarten","last":"Rijke","middle":["de"]}]
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Empathy is a complex cognitive ability based on the reasoning of others’ affective states. In order to better understand others and express stronger empathy in dialogues, we argue that two issues must be tackled at the same time: (i) identifying which word is the cause for the other’s emotion from his or her utterance and (ii) reflecting those specific words in the response generation. However, previous approaches for recognizing emotion cause words in text require sub-utterance level annotations, which can be demanding. Taking inspiration from social cognition, we leverage a generative estimator to infer emotion cause words from utterances with no word-level label. Also, we introduce a novel method based on pragmatics to make dialogue models focus on targeted words in the input during generation. Our method is applicable to any dialogue models with no additional training on the fly. We show our approach improves multiple best-performing dialogue agents on generating more focused empathetic responses in terms of both automatic and human evaluation.
[{"first":"Hyunwoo","last":"Kim","middle":[]},{"first":"Byeongchang","last":"Kim","middle":[]},{"first":"Gunhee","last":"Kim","middle":[]}]
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Empathy is critical to successful mental health support. Empathy measurement has predominantly occurred in synchronous, face-to-face settings, and may not translate to asynchronous, text-based contexts. Because millions of people use text-based platforms for mental health support, understanding empathy in these contexts is crucial. In this work, we present a computational approach to understanding how empathy is expressed in online mental health platforms. We develop a novel unifying theoretically-grounded framework for characterizing the communication of empathy in text-based conversations. We collect and share a corpus of 10k (post, response) pairs annotated using this empathy framework with supporting evidence for annotations (rationales). We develop a multi-task RoBERTa-based bi-encoder model for identifying empathy in conversations and extracting rationales underlying its predictions. Experiments demonstrate that our approach can effectively identify empathic conversations. We further apply this model to analyze 235k mental health interactions and show that users do not self-learn empathy over time, revealing opportunities for empathy training and feedback.
[{"first":"Ashish","last":"Sharma","middle":[]},{"first":"Adam","last":"Miner","middle":["S."]},{"first":"David","last":"Atkins","middle":["C."]},{"first":"Tim","last":"Althoff","middle":[]}]
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Open-domain conversational agents or chatbots are becoming increasingly popular in the natural language processing community. One of the challenges is enabling them to converse in an empathetic manner. Current neural response generation methods rely solely on end-to-end learning from large scale conversation data to generate dialogues. This approach can produce socially unacceptable responses due to the lack of large-scale quality data used to train the neural models. However, recent work has shown the promise of combining dialogue act/intent modelling and neural response generation. This hybrid method improves the response quality of chatbots and makes them more controllable and interpretable. A key element in dialog intent modelling is the development of a taxonomy. Inspired by this idea, we have manually labeled 500 response intents using a subset of a sizeable empathetic dialogue dataset (25K dialogues). Our goal is to produce a large-scale taxonomy for empathetic response intents. Furthermore, using lexical and machine learning methods, we automatically analysed both speaker and listener utterances of the entire dataset with identified response intents and 32 emotion categories. Finally, we use information visualization methods to summarize emotional dialogue exchange patterns and their temporal progression. These results reveal novel and important empathy patterns in human-human open-domain conversations and can serve as heuristics for hybrid approaches.
[{"first":"Anuradha","last":"Welivita","middle":[]},{"first":"Pearl","last":"Pu","middle":[]}]
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
The capacity of empathy is crucial to the success of open-domain dialog systems. Due to its nature of multi-dimensionality, there are various factors that relate to empathy expression, such as communication mechanism, dialog act and emotion. However, existing methods for empathetic response generation usually either consider only one empathy factor or ignore the hierarchical relationships between different factors, leading to a weak ability of empathy modeling. In this paper, we propose a multi-factor hierarchical framework, CoMAE, for empathetic response generation, which models the above three key factors of empathy expression in a hierarchical way. We show experimentally that our CoMAE-based model can generate more empathetic responses than previous methods. We also highlight the importance of hierarchical modeling of different factors through both the empirical analysis on a real-life corpus and the extensive experiments. Our codes and used data are available at https://github.com/chujiezheng/CoMAE.
[{"first":"Chujie","last":"Zheng","middle":[]},{"first":"Yong","last":"Liu","middle":[]},{"first":"Wei","last":"Chen","middle":[]},{"first":"Yongcai","last":"Leng","middle":[]},{"first":"Minlie","last":"Huang","middle":[]}]
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Empathy is a crucial factor in open-domain conversations, which naturally shows one’s caring and understanding to others. Though several methods have been proposed to generate empathetic responses, existing works often lead to monotonous empathy that refers to generic and safe expressions. In this paper, we propose to use explicit control to guide the empathy expression and design a framework DiffusEmp based on conditional diffusion language model to unify the utilization of dialogue context and attribute-oriented control signals. Specifically, communication mechanism, intent, and semantic frame are imported as multi-grained signals that control the empathy realization from coarse to fine levels. We then design a specific masking strategy to reflect the relationship between multi-grained signals and response tokens, and integrate it into the diffusion model to influence the generative process. Experimental results on a benchmark dataset EmpatheticDialogue show that our framework outperforms competitive baselines in terms of controllability, informativeness, and diversity without the loss of context-relatedness.
[{"first":"Guanqun","last":"Bi","middle":[]},{"first":"Lei","last":"Shen","middle":[]},{"first":"Yanan","last":"Cao","middle":[]},{"first":"Meng","last":"Chen","middle":[]},{"first":"Yuqiang","last":"Xie","middle":[]},{"first":"Zheng","last":"Lin","middle":[]},{"first":"Xiaodong","last":"He","middle":[]}]
CEM: Commonsense-aware Empathetic Response Generation
A key trait of daily conversations between individuals is the ability to express empathy towards others, and exploring ways to implement empathy is a crucial step towards human-like dialogue systems. Previous approaches on this topic mainly focus on detecting and utilizing the user's emotion for generating empathetic responses. However, since empathy includes both aspects of affection and cognition, we argue that in addition to identifying the user's emotion, cognitive understanding of the user's situation should also be considered. To this end, we propose a novel approach for empathetic response generation, which leverages commonsense to draw more information about the user's situation and uses this additional information to further enhance the empathy expression in generated responses. We evaluate our approach on EmpatheticDialogues, which is a widely-used benchmark dataset for empathetic response generation. Empirical results demonstrate that our approach outperforms the baseline models in both automatic and human evaluations and can generate more informative and empathetic responses.
[{"first":"Sahand","last":"Sabour","middle":[]},{"first":"Chujie","last":"Zheng","middle":[]},{"first":"Minlie","last":"Huang","middle":[]}]
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
approaches to this task are based on Sequence-to-Sequence (Seq2Seq) frameworks [3, 11-13, 17, 19]. Current Seq2Seq frameworks suffer from a low-diversity problem. As a result, these approaches frequently generate generic responses such as "I don't know" or "I'm sorry" [3,11,12,19].
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
To address the issue of low response diversity, previous studies have produced several hypotheses about the cause of the lowdiversity problem with corresponding solutions. For example, Li et al. [3] argue that the usual max a posteriori (MAP) objective function might favor frequent responses. Instead, they propose to use a Maximum Mutual Information (MMI) objective function to encourage responses with higher mutual information regarding the user's utterance. The downside of this method is that it relies heavily on beam search and/or an inverse model that is trained by swapping inputs and outputs. Tao et al. [15] assume that the single attention layer commonly used in Seq2Seq models is only able to focus on a single semantic aspect of the input sequence, so they propose a Multi-Head Attention Mechanism (MHAM) to allow the decoder to generate more diverse responses. There are also studies that try to improve response diversity by introducing randomness [12,19].
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Recently, Jiang and de Rijke [1] have shown that there is a strong connection between the low-diversity problem and what they call model over-confidence following Pereyra et al. [10], the phenomenon that a model incorrectly assigns most probability to only a few tokens. However, the cause of this phenomenon remains unknown. In §2, we investigate and conclude that the model overconfidence problem is caused by an imbalanced training of tokens with variable frequencies, which favors frequent tokens and results in low-diversity. To correct for this, we propose a Frequency-Aware Cross-Entropy (FACE) loss function that improves the traditional Cross-Entropy (CE) loss function by taking token frequency into consideration. More specifically, we first analyze the influence of the commonly used CE loss function, and find that it prefers highfrequency tokens, which results in model over-confidence and lowdiversity responses. Then we propose a FACE loss function that improves over the CE loss function by incorporating a weighting mechanism conditioned on token frequency.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
The primary differences between FACE and previous studies in addressing the low-diversity problem are two-fold: (1) The diversity improvements brought by FACE do not rely on beam search or  [3] and MHAM [15], FACE achieves the highest diversity performance and it does so with minimum modifications to the original Seq2Seq model structure and existing hyper-parameters. 1 The main contributions we make in this paper are: • We examine the influence of token frequency on model overconfidence and response diversity. • We propose a Frequency-Aware Cross-Entropy (FACE) loss function to balance the per-token loss, which alleviates model overconfidence and, hence, improves response diversity. • We investigate two token frequency calculation methods and corresponding frequency-based weighting mechanisms for FACE.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
As illustrated by Jiang and de Rijke [1], model over-confidence [10] can cause Seq2Seq-based conversation models to have low response diversity. In this section, we show that model over-confidence and low-diversity are statistical and empirical symptoms of the same problem. Imbalanced training is the actual underlying reason. Existing Seq2Seq-based conversation models tend to be overconfident during prediction and place most probability on only a few tokens. We define a leading token in a response to be a token that appears first. From Table 1, we can see that all of the most frequent leading tokens in model responses are actually very likely to appear first in training ground-truth responses, but the percentages of the top-ranked token "i" are much higher in the validation and test sets. This suggests that the model is over-fitting for frequent leading tokens. We also find that the frequency of some subsequent tokens (e.g., 'm and not) is much higher in model responses than in the training ground truth, as illustrated in Table 2. This is also due to model over-fitting, since these tokens are relatively more frequent compared to other tokens. For example, given that the previous token is "i", the frequency of the following token "'m" is much higher than others in the training data.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
To understand this phenomenon and introduce our solution, we first look into the commonly used max a posteriori (MAP) objective function of Seq2Seq models. Given a dataset of message-response pairs (X , Y ), where X = (x 1 , x 2 , . . . , x |X | ) and Y = (y 1 , y 2 , . . . , y |Y | )  (1) are the input and output sequences, respectively, the goal of Seq2Seq training is to maximize the conditional probability P (Y |X ). Since the decoder Recurrent Neural Network (RNN) can only give one output at each time step t, and to generate grammatical responses, we are actually maximizing token-wise probability at each time step during training:
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
where y <t = (y 1 , y 2 , . . . , y t −1 ) are tokens generated in previous time steps. At test time, the response is generated with respect to:
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
In practice, we usually maximize the aforementioned conditional probability by minimizing the prediction loss at each step t, which is actually the Cross-Entropy (CE) loss:
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
where (c 1 , c 2 , . . . , c N ) is the search space of y t , δ i (y t ) = 1 if y t = c i and 0 otherwise; P (c i |·) is the predicted probability of candidate token c i and is calculated using the softmax function:
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
where f θ (·) is a non-linear scoring function with parameters θ ; f θ (·) takes the hidden state h dec t −1 , last generation y t −1 and X as inputs, and calculates a score for each possible candidate c i .
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
During training, the loss calculated using Eq. (3) is back-propagated through the whole network. The effect is that f θ will assign higher scores for c i = y t in the future, so that the loss in Eq. (3) will decrease, and meanwhile the predicted probability in Eq. (4) will increase. The ultimate effect is to maximize the probabilities of ground-truth outputs given input sequences, as illustrated in Eq. (1). The model is frequently penalized by a small number of frequent tokens, as a result of which the total loss (TL) for these tokens is higher than for less-frequent ones:
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
where t denotes the time step, with maximum training steps N t :
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Here, freq(c i ) represents the frequency that c i appears in the training ground truth. Assume that the expected value E[CE(c i )] is roughly the same for c i , ∀i ∈ {1, 2, . . . , N }, then tokens with a higher frequency will have a larger total loss during training. We refer to this phenomenon as loss imbalance. Due to loss imbalance, a Seq2Seq model favors frequent tokens and thus is over-confident about them. This is especially true for the leading token as observed in Table 1, since the decoder language model does not have a strong effect in the beginning of prediction. When a frequent token is selected as leading token, the search space for subsequent tokens is hugely restricted by the language model, and this will likely result in a frequent generic response.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Now that we have identified loss imbalance as a cause of lowdiversity problems, we propose to balance the total loss for each token by applying a weight factor to TL:
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
where w i is the weight corresponding to c i . By absorbing w i into the CE loss function, we obtain the FACE loss function:
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Our key solution to the low-diversity problem lies in the weight factors w i . Based on the analysis in §2, one straightforward way to learning w i is to take advantage of the token frequency freq(c i ), so that frequent tokens will have lower weights. Below we propose two methods to estimate freq(c i ): ground-truth or GT frequency and output frequency.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
GT frequency: Token frequency in the ground-truth responses. As illustrated in Eq. (6), the number of training steps N t equals the sum of frequencies of tokens in the training ground-truth responses, so it is intuitive to use these token frequencies to adjust the weight in Eq. (8). However, during training, the model is given data sequentially in random order. As a result, the real-time token frequency seen by the model is likely to differ from that of the entire training data. Our solution is to calculate the batch token frequency instead:
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
where b is the number of training batches seen so far and o represents the newly observed batch.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Output frequency: Token frequency in model responses. Alternatively, we can employ a train-and-refine strategy: the responses of a pre-trained Seq2Seq model can reflect which tokens the model is already overfitted for; by directly penalizing those tokens with a fine-tuning procedure, we can improve the response diversity without retraining it from scratch. The output frequency may have a more obvious effect on improving response diversity than the GT frequency, because diversity is directly exhibited by model outputs.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Given the frequency of each token, we introduce the following two methods to calculate the weight factor.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
3.1.1 Pre-weight. This method derives the weight factor w i prior to seeing new training examples, i.e., pre-weight function:
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Here we formulate it as a linear function of relative frequency:
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
is the slope, and the bias is 1 so that w i falls in [0, 1]. We then normalize {w 1 , w 2 , . . . , w N } to have a mean of 1. The pre-weight function can make sure that tokens with a higher RF value will get lower weights. In other words, the influence of high frequency tokens will be penalized by using the pre-weight function.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
This method tries to penalize the model's conservativeness: if the output token y t has a higher frequency than the ground truth c i , which indicates the model conservatively picked a "safe" token, then its loss will be scaled up by w i > 1, otherwise w i = 1 due to ReLU activations:
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Since we can only apply this weighting function after obtaining the model outputs, we refer to it as the post-weight function.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
As illustrated in [1], Confidence Penalty (CP) methods can alleviate the low-diversity problem. In the experiments of this paper, we also test the performance of the CP function [1]:
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
where p(y t |y <t , X ) is the predicted distribution at t, and H (·) is its entropy. However, during experiments we found that the parameter β needs to be carefully chosen, otherwise the loss CP(y t ) will be negative, which is counter-intuitive since losses should be greater than 0. Instead, we propose a parameter-free CP function:
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
FACE and CP can be easily combined to further improve response diversity. A trivial combination is to replace the CE loss function with the FACE loss function in Eq. (12) and Eq. (13). We also propose a CP weighting function using the entropy in Eq. (13):
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
where w (without subscript) is assumed to be independent of c i , which is different from that in Eq. (10) and Eq. (11). By using w as the weight of FACE in Eq. (8), we can penalize the model confidence by adjusting the weight of FACE.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
To prove the effectiveness of our proposed methods, we design experiments to answer the following questions: (Q1) Which combination of our proposed frequency methods (GT frequency and output frequency) and weighting functions (pre-and post-weight) performs best? (Q2) Does FACE improve the diversity of Seq2Seq conversation models? (Q3) Does CP improve the diversity of Seq2Seq conversation models? (Q4) Does the combination of FACE and CP further improve performance? (Q5) Does FACE improve the response quality besides diversity?
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
The first baseline is a vanilla Seq2Seq model with general attention [6] as implemented within the ParlAI platform [7]. We also choose the MMI models proposed in [3] and the MHAM models proposed in [15]. These models are all deterministic methods. In Table 3, we  [16] is an online-available corpus of movie subtitles. 3 Here, we use the 2011 version and set aside ∼60M lines, which constitutes ∼30M message-response pairs for training. Following Li et al. [3], we randomly select 2K pairs from the IMSDB dataset [18] for validation and test sets, respectively, and we filter out pairs whose responses or messages are shorter than 6 tokens.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Twitter is a commonly used source of data in dialogue generation research. For ease of reproducibility of the results reported in this paper, we did not follow Li et al. [3], Sordoni et al. [13] who used ∼130M context-message-response triples before preprocessing. Instead, we use the version released by [12] for training, which contains ∼4M tweet IDs in total and can be scraped in 3 days. 4 After formatting the tweets as context-message-response triples, we have 904K training examples and we concatenate context-message as input. For the training, validation and test sets, we exclude all overlapping IDs. Then we restrict the sequence length in both validation and test sets to the range [6,18], resulting in 18,162 validation and 1,897 test triples. Furthermore, since tuning the MMI models is quite time-consuming, we randomly select 2K triples from the validation set for hyper-parameter selection.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Although reported not to correlate well with human judgments [5], we still report BLEU scores [8] for fair comparisons with baselines. To evaluate response diversity, we use the d-1 and d-2 metrics proposed in [3] that are calculated as the number of distinct uni-and bigrams, divided by the total number of tokens generated.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Following Sordoni et al. [13], we recruit crowd-source workers to perform pairwise qualitative comparisons.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Since the conversation history of the OSDb dataset is only a single turn, which is hard for human annotators to judge, we choose the Twitter dataset for human evaluation, which has two-turn histories.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
We randomly select 1K test examples from the Twitter test set,  
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
For the OSDb and Twitter datasets, we keep the most frequent 25,000 tokens and replace other tokens with _UNK_. For the OSDb corpus, we use a 4-layer Long Short-Term Memory (LSTM) network for both encoder and decoder. The hidden size (HS) and word embedding size are set to 1,000. On the smaller Twitter corpus, we use a smaller network with a 2-layer LSTMs for encoder and decoder. The HS and word embedding sizes are set to 512 and 200, respectively. For both networks, we randomly initialize the model parameters from a uniform distribution U (− √ 1/HS, √ 1/HS) and the word embeddings from a normal distribution N (0, 1). We also employ dropout [14] with drop ratio p = 0.1. We use Adam [2] as our optimization method. For the hyper-parameters of the Adam optimizer, we set the learning rate α = 0.001, two momentum parameters β1 = 0.9 and β2 = 0.999, respectively, and ϵ = 10 −8 . We also clip gradients [9] to 5 during training to avoid gradient explosion. To speed up training and convergence, we use mini-batches of size 256. Since the model chosen with minimum training loss usually has very low diversity, we choose d-1 as the early stopping criterion. A scheduler is used to reduce the learning rate by a factor of 0.5 when a d-1 plateau is detected with patience = 3. β in Eq. (12) is set to 0.01.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
To answer Q1, we first identify the best performing variant of FACE to be used in later experiments. In Tables 4 and 5  On the OSDb dataset, we can see from Table 4 that FACE-OPR and FACE-GPO perform better than the other two in terms of diversity scores. Similar results are shown on the Twitter dataset in Table 5. By comparing the results in Table 4 and 5, we find that the performance of FACE-OPR is very stable. We therefore choose FACE-OPR as our primary model and refer to it as FACE in the following sections. In contrast, FACE-GPR performs worst on both datasets, which indicates that GT frequency does not work well with pre-weight. The BLEU scores in Table 4 and 5 show no obvious patterns.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
It is worth noting that all variants reported in Table 4 and 5 are using the train-and-refine strategy: first training the Seq2Seq model with CE, then fine-tuning it with FACE. We also tried training from scratch using FACE. However, the performance did not show consistent improvements, which is probably because our equal expected loss assumption (under Eq. (6)) is violated in the early training stages.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
We turn to Q2, Q3, and Q4. The automatic evaluation results are shown in Tables 6 and 7. Seq2Seq is vanilla-Seq2Seq with general attention mechanism [6]. Seq2Seq-refine is a fine-tuned version of Seq2Seq with a smaller batch size of 30. MMI-antiLM adjusts the Seq2Seq prediction probability using the decoder language model, while MMI-bidi utilizes a reverse model. Both methods use beam size 200. MHAM projects encoder hidden-states to 5 different semantic spaces, and CMHAM forces those 5 spaces to be perpendicular to each other. All of our models are fine-tuned Seq2Seq models with a smaller batch size of 30. FACE is fine-tuned using the FACE function and output token frequency. CP and CP free are fine-tuned using Eq. (12) and (13), respectively. FACE-CP is a linear combination of FACE and CP. FACE-CP free is the multiplicative combination of FACE and the CP weighting function in Eq. (14).
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
As shown in Table 6, all the methods proposed in this paper outperform the baselines in terms of diversity metrics on the OSDb dataset. FACE achieves the highest d-2 score (increase of 6.3%). FACE-CP and FACE-CP free achieve slightly higher d-1 and lower d-2 scores than FACE, which can be viewed as the trade-off between  FACE and CP's. Although the penalty strength β of CP is carefully selected, it is interesting to see that our hyper-parameter free method CP free outperforms CP by a large margin and achieves the highest d-1 score (1.9% improvement), which demonstrates the effectiveness of Eq. (13). FACE performs best in terms of BLEU scores, which suggests that FACE can generate higher-quality responses.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Although the MMI-based and MHAM-based methods all exhibit various improvements over Seq2Seq in terms of d-1 and d-2 scores, they mostly perform worse than Seq2Seq-refine. This suggests that on the OSDb dataset, carefully designed MMI-based and MHAMbased methods are not able to outperform a fine-tuned Seq2Seq baseline.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Similar to Table 6, Table 7 shows that all our methods can increase the diversity of Seq2Seq on the Twitter dataset. FACE achieves the highest d-1 score (0.4% increase) together with MMI-antiLM, but the highest d-2 score of FACE (6.5% improvement) demonstrates that our method fares better.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
In Table 7 however, Seq2Seq-refine decreases the diversity of Seq2Seq, indicating that fine-tuning does not help to address the lowdiversity problem on the Twitter dataset. This is probably because Table 8: Results of the pairwise human evaluation (%) on the Twitter dataset. "Win", "Lose" and "Gain" correspond to "FACE wins", "Baseline wins" and their difference (W in − Lose), respectively. Highest scores are highlighted in bold face, and * , * * , * * * symbols indicate significant improvements with p-value < 0.05, < 0.01, < 0.005, respectively. the diversity of Seq2Seq is already relatively high on this dataset, and there is limited space for further improvements. While MMI-bidi performs better than MMI-antiLM on the OSDb dataset, on the Twitter dataset, however, MMI-bidi performs much worse and degrades the diversity of the Seq2Seq model. The reason for this phenomenon is probably because of the reverse model of MMI-bidi: the input consisting of Twitter triples contains two turns from different speakers (context-message), thus given the response only, the prediction probability of context-message is very unreliable.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Although it achieves the highest BLEU score, MHAM slightly hurts diversity. Similarly, CMHAM degrades the diversity even more. Closer inspection reveals that most of the attention weights of both methods are on the first several tokens of the input sequence (i.e., context). It is unlikely to be able to properly learn the relation between context-message and response by only paying attention to the context.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
To illustrate the effectiveness of the weighting function, we display the frequencies and the corresponding weights of some tokens in Figure 1. From this figure, we can see that the most frequent tokens have very small weights, and less frequent tokens receive larger weights. Please note that the tokens with the largest weights are in the scope of top-200 frequent tokens.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
We now turn to Q5. Human evaluation results are reported in Table 8. We can see that FACE is significantly better than all baselines. This means that by increasing response diversity, FACE can improve relevance and interestingness of responses, without sacrificing grammatical accuracy. Specifically, by penalizing frequent tokens, FACE gives more opportunities to less frequent tokens, so the interestingness is higher than Seq2Seq. While high-frequency tokens may be appropriate responses in more situations than lowfrequency ones, it is for the same reason that higher frequency tokens convey less information, thus making the responses seem generic. Therefore, by encouraging low-frequency tokens, FACE has the potential to increase relevance of responses. By looking at the human evaluation scores in Table 8 together with the BLEU scores in Table 7, we can re-confirm that BLEU scores do not correlate well with human qualitative evaluation [5].
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Since FACE learns a language model relying entirely on training data, the grammatical accuracy of FACE is as least as good as that of Seq2Seq. The examples in Table 9 show that FACE can actually  generate more relevant and interesting responses. In contrast, although MMI-antiLM improves response diversity, it increases the risk of grammatical errors since it penalizes the language model. For example, the response of MMI-antiLM in the second example of Table 9 is ungrammatical and irrelevant to the conversation history. Similarly, MMI-bidi uses a reverse model to adjust the model outputs, in the hope that the responses can be more diverse and relevant. However, as explained earlier, the reverse model is highly unreliable, making many of the responses look weird, see, e.g., the third row of Table 9.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
Although MHAM and CMHAM can improve response diversity to some extent, it is not guaranteed that the multiple attention heads are well distributed throughout the whole history. In fact, there are many cases where the attention weights are on the Context part of the history, making the responses irrelevant to the Message, such as the fourth and fifth examples in Table 9.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
In this work, we have proposed a Frequency-Aware Cross-Entropy (FACE) function for tackling the low-diversity problem of Seq2Seqbased conversation models. Experiments on short-history conversation datasets demonstrate that the FACE loss function can effectively improve the diversity and quality of responses. FACE achieves the improvements with minimum modifications to the original Seq2Seq model, which makes it flexible to extend. We also propose a hyper-parameter free CP free , which exhibits better performance than the original parameter-dependent CP.
Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss
A limitation of FACE is that the learning procedure is not as stable as CE, which increases the difficulty of training. In future work, we would like to investigate this phenomenon in depth. We also hope to test FACE in a long-history conversation setting by applying FACE to hierarchical Seq2Seq [11]. Besides, we also plan to apply FACE to stochastic models [12,19] and examine how FACE can be used in an adversarial dialogue generation setup [4].
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Empathy is one of the hallmarks of social cognition. It is an intricate cognitive ability that requires high-level reasoning on other's affective states. The intensity of expressed empathy varies depending on the depth of reasoning. According to Sharma et al. (2020), weak empathy is accompanied by generic expressions such as "Are you OK?" or "It's just terrible, isn't it?", while stronger empathy reflects the other's specific situation: "How is your headache, any better?" or "You must be worried about the job interview". In order to respond with stronger empathy, two issues must be tackled: reasoning (i) where to focus on the interlocutor's utterance (for the reason behind the emotion) and (ii) how to generate utterances that focus on such words.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Firstly, which words should we focus on when empathizing with others? As empathy relates to other's emotional states, the reasons behind emotions (emotion cause) should be identified. Imagine you are told "I got a gift from a friend last vacation!" with a joyful face. The likely words that can be the causes of his/her happiness are "gift" and "friend". On the other hand, "vacation" has less to do with the emotion. If you respond "How was your vacation?", the interlocutor may think you are not interested; rather, it is better to say "Wow, what was the gift?" or "Your friend must really like you." by focusing on the emotion cause words.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
We humans do not rely on word-level supervision for such affective reasoning. Instead, we put ourselves in the other's shoes and simulate what it would be like. Perspective-taking is this act of considering an alternative point of view for a given situation. According to cognitive science, perspectivetaking and simulation are key components in empathetic reasoning (Davis, 1983;Batson et al., 1991;Ruby and Decety, 2004). Taking inspiration from these concepts, we propose to train a generative emotion estimator for simulating the other's situation and identifying emotion cause words.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Secondly, after reasoning which words to focus on, the problem of how to generate focused responses still remains. Safe responses that can be adopted to any situations might hurt other's feelings. Generated utterances need to convey the impression that concerns the specific situation of the interlocutor. Such communicative reasoning is studied in the field of computational pragmatics. The Rational Speech Acts (RSA) framework (Frank and Goodman, 2012) formulates communication between speaker and listener as probabilistic reasoning. It has been applied to many tasks to increase the informativeness of generated text grounded on inputs (Andreas and Klein, 2016;Fried et al., 2018;Cohn-Gordon and Goodman, 2019;Shen et al., 2019). That is, RSA allows the input to be more reflected in the generated output.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
However, controlling the RSA framework to re-flect specific parts of the input remains understudied. We introduce a novel method for the RSA framework to make models focus on targeted words in the interlocutor's utterance during generation. In summary, we recognize emotion cause words in dialogue utterances with no word-level labels and generate stronger empathetic responses focused on them without additional training. Our major contributions are as follows:
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
(1) We identify emotion cause words in dialogue utterances by leveraging a generative estimator. Our approach requires no additional emotion cause labels other than the emotion label on the whole sentence, and outperforms other baselines.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
(2) We introduce a new method of controlling the Rational Speech Acts framework (Frank and Goodman, 2012) to make dialogue models better focus on targeted words in the input context to generate more specific empathetic responses.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
(3) For evaluation, we annotate emotion cause words in emotional situations from the validation and test set of EmpatheticDialogues dataset (Rashkin et al., 2019). We publicly release our EMOCAUSE evaluation set for future research.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
(4) Our approach improves model-based empathy scores (Sharma et al., 2020) of three recent dialogue agents, MIME (Majumder et al., 2020), DodecaTransformer , and Blender (Roller et al., 2021) on EmpatheticDialogues. User studies also show that our approach improves human-rated empathy scores and is more preferred in A/B tests.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Our approach consists of two steps: (i) recognizing emotion cause words from utterances with no word-level labels ( §3), and (ii) generating empathetic responses focused on those words ( §4). In this section, we first train a generative emotion estimator to identify emotion cause words.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
We leverage a generative model by taking inspiration from perspective-taking (i.e. simulating one-self in other's shoes) to reason emotion causes; not requiring word-level labels. Our idea is to estimate the emotion cause weight of each word in the utterance while satisfying the following three desiderata.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
(1) Do not require word-level supervision for learning to identify emotion cause words in the utterances. Humans do not need word-level labels to infer the probable causes associated with the other's emotion during conversation.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
(2) Simulate the observed interlocutor's situation within the model. Simulation theory (ST) from cognitive science explains that this mental imitation helps understanding the internal mental states of others (Gallese et al., 2004). Much evidence for ST is found from neuroscience including mirror neurons (Rizzolatti and Craighero, 2004), action-perception coupling (Decety and Chaminade, 2003), and empathetic perspective-taking (Ruby and Decety, 2004).
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
(3) Reason other's internal emotional states in Bayesian fashion. Studies from cognitive science argue that human reasoning of other's affective states and minds can be described via Bayesian inference (Griffiths et al., 2008;Ong et al., 2015;Saxe and Houlihan, 2017;Ong et al., 2019).
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Interestingly, a generative emotion estimator (GEE), which models P (C, E) = P (E)P (C|E) with text sequence (e.g. context) C and emotion E, satisfies all the above conditions. First, the generative estimator computes the likelihood of C by generating C given E, which can be viewed as a simulation of C. Second, it estimates P (E|C) via Bayes' rule. Finally, the association between the emotion estimate and each word comes for free by using the likelihood of each words; without using any word-level supervision. We use BART (Lewis et al., 2020) to implement a GEE.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Dataset. To train our GEE, we leverage the Em-patheticDialogues (Rashkin et al., 2019), a multiturn English dialogue dataset where the speaker talks about an emotional situation and the listener expresses empathy. An example is shown in Table 1. The emotion and the situation sentence are only visible to the speaker. Situations are collected beforehand by asking annotators to recall related experiences for a given emotion label. The dataset includes a rich suite of 32 emotion labels that are evenly distributed.  Training. Given an emotion label E, GEE is trained to generate its corresponding emotional situation C = {w 1 , ..., w T }, where w i is a word. As a result, our GEE learns the joint probability P (C, E). The trained GEE shows perplexity of 13.6 on the test situations of EmpatheticDialogues.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Once trained, GEE can predict P (E|C = c) for a word sequence c (e.g. utterance) using Bayes' rule:
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
(1)
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
We compute the likelihood P (C = c|E) by GEE's generative ability as described in §3.1. Since emotions in EmpatheticDialogues are almost evenly distributed, we set the prior P (E) to a uniform distribution. Finally, we find the emotion with the highest likelihood of the given sequence c.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
We comparatively report the emotion classification accuracy of GEE in Appendix.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
We introduce how GEE can recognize emotion cause words solely based on emotion labels without word-level annotations. For a given word sequence c = {w 1 , w 2 , ..., w T } (e.g. utterance), GEE can reason the association P (W |E =ê) of each word w t in the sequence c to the recognized emotionê in Bayesian fashion:
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
The emotion likelihood is computed as
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
where w <t is the partial utterance up to time step t − 1. Since computing the expectation over all possible partial utterance w <t is intractable, we approximate it by a single sample. We build set E to includeê and emotions with the two lowest probability of P (E|C = c) when recognizing emotion in Eq.(1). We assume the marginal P (W ) is uniform. We choose the top-k words reasoned by GEE as emotion cause words, and focus on them during empathetic response generation.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
We introduce how to control the Bayesian Rational Speech Acts (RSA) framework (Frank and Goodman, 2012) to focus on targeted words in the context during response generation. We first preview the basics of RSA for dialogues ( §4.1). We then present how to control the RSA with word-level focus ( §4.2), where our major contribution lies. Figure 1 is the overview of our method.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Applying the RSA framework is computing the posterior of the dialogue agent's output distribution over words each time step. Hence, it is applicable to any existing pretrained dialogue agents on the fly, with no additional training. The RSA framework formulates communication as a reference game between speaker and listener. Based on recursive Bayesian formulation, the speaker (i.e. dialogue model) reasons about the listener's belief of what the speaker is referring to. We follow the approach of Kim et al. (2020) for adopting RSA to dialogues. Our goal here is to update a base speaker S 0 to a pragmatic speaker S 1 that focuses more on the emotion cause words in dialogue context c (i.e. dialogue history).
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Base Speaker S 0 . Let c and u t denote dialogue context and the output word of the model at time step t, respectively. The base speaker S 0 is a dialogue agent that outputs u t for a dialogue context and partial utterance u <t : S 0 (u t |c, u <t ). As described, one can use any dialogue models for S 0 .
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Pragmatic Listener L 0 . The pragmatic listener is a posterior distribution over which dialogue con- Figure  text the speaker is referring to. It is defined in terms of the base speaker S 0 and a prior distribution p t (C) over the context in Bayesian fashion:
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
. (4) The shared world C is a finite set comprising the given dialogue context c and other contexts (coined as distractors) different from c. Our contribution lies in how to build world C to endow the dialogue agent with controllability to better focus on targeted words, which we discuss in §4.2. We update prior p t+1 (C) with L 0 from time step t as follows: p t+1 (C) = L 0 (C|u ≤t , p t ). β is the rationality parameter which controls how much the base speaker's distribution is taken into account. We note that L 0 is simply a distribution computed in Bayesian fashion, not another separate model. Pragmatic Speaker S 1 . Integrating L 0 with S 0 , we obtain the pragmatic speaker S 1 :
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Since the pragmatic speaker S 1 is forced to consider how its utterance is perceived by the listener (via L 0 ), it favors words that have high likelihood of the given context c over other contexts in shared world C. Similar to Eq. 4, α is the rationality parameter for S 1 .
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
We aim to make dialogue models focus on targeted words from the input (i.e. dialogue context) during generation via shared world C. The shared world C consists of the given dialogue context c and other distractor contexts. It is used for computing the likelihood of the given context c in Eq. 4. Previous works of RSA in NLP manually (or randomly) select pieces of text (e.g. sentences) entirely different from the given input (Cohn-Gordon and Goodman, 2019; Shen et al., 2019;Kim et al., 2020). In our context, it means distractors will be totally different contexts from c in the dataset. For example, when given a context "I got a gift from my friend.", a distractor might be "Today, I have an exam at school.". Although such type of distractors helps improve the specificity of the model's generated outputs, it is difficult to finely control which words the models should be specific about.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Our core idea is to build distractors by replacing the emotion cause words in c with different words via sampling with GEE. It can enhance the controllability of the RSA by making models focus on targeted words (e.g. emotion cause words recognized by GEE) from the dialogue context.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
For a dialogue context c = {w 1 , ..., w T } where w i is a word, GEE outputs top-k emotion cause words regarding the recognized emotionê 1 from context c, denoted by W gee . Next, we concatenate the least likely n emotions from GEE with the context c removing the top-k emotion cause words: [ê −1 , ...,ê −n ; c − W gee ], which is input to GEE. We then sample different words (w i ,w j , . . . ,w k ) from GEE's output in place of W gee to construct a distractorc. For example, given a context c "I was sick from the flu" and "sick, flu" as the top-2 emotion cause words, a sampled distractorc can be "I was laughing from the relief ". We use these altered contexts {c 1 , ...,c i } as distractors for the shared world C in the pragmatic listener L 0 (Eq. 4). We set n and cardinality of world C to 3 (i.e. C = {c,c 1 ,c 2 }). We run experiments and find the best k (= 5) (see Appendix).
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
The only difference between the original context c and the sampled distractorc is those emotion cause words. The pragmatic speaker S 1 (Eq. 5) prefers to generate words that have a higher likelihood of the given context c (including the original emotion cause words W gee ) than the distractor contextc. As a result, the pragmatic agent can generate    Annotators are required to have a minimum of 1000 HITs, 95% HIT approval rate, and be located at one of [AU, CA, GB, NZ, US]. We pay the annotators $0.15 per description. To further ensure quality, only annotators who pass the qualification test are invited to annotate. Nevertheless, speculations for emotion causes are subjective and can vary among annotators. Therefore, we use only unanimously selected words (i.e. earning all three votes) to ensure maximum objectivity.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
We analyze the characteristics of our emotion cause words in the EMOCAUSE evaluation set. In Table 3 and Figure 2, we compare the basic statistics of our annotation set and RECCON (Poria et al., 2020), which is an English dialogue dataset annotating emotion cause spans on the DailyDialog (Li et al., 2017) and IEMOCAP (Busso et al., 2008) with a total of 8 emotions. Since our EMOCAUSE is based on emotional situations from an empathetic dialogue dataset (Rashkin et al., 2019), emotion causes play a more important role than in casual conversations from RECCON. While 74% of REC-CON's labels belong to a single emotion happy, EMOCAUSE provides a balanced range of 32 emotions labels. Therefore, our evaluation set presents a wider variety than RECCON. Table 4 shows some examples of the annotated emotion cause words. Table 5 reports the most frequent cause words for some emotions. We find "embarrassing" events happen frequently in toilets and in front of people. "Proud" and "disappointed" are closely related to children. Interestingly, phones are associated with "trusting", which may be due to smartphones containing sensitive personal information. More examples and results can be found in Appendix.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
We first evaluate our generative emotion estimator (GEE) on weakly-supervised emotion cause word recognition ( §6.2). We then show our new controlling method for the RSA framework can improve best performing dialogue agents to generate more empathetic responses by better focusing on targeted emotion cause words ( §6.3).
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
EmpatheticDialogues (ED) (Rashkin et al., 2019). This dataset is an English empathetic dialogue dataset with 32 diverse emotion types ( §3.2). The task is to generate empathetic responses (i.e. responses from the listener's side in Table 1) when only given the dialogue context (i.e. history) without emotion labels and situation descriptions. It contains 24,850 conversations partitioned into training, validation, and test set by 80%, 10%, 10%, respectively. We additionally annotate cause words for the given emotion for all situations in the validation and test set of EmpatheticDialogues ( §5).
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
EmoCause ( §5). We compare our GEE with four methods that can recognize emotion cause words with no word-level annotations: random, RAKE (Rose et al., 2010), EmpDG (Li et al., 2020), and BERT (Devlin et al., 2019). For random, we randomly choose words as emotion causes. RAKE is an automatic keyword extraction algorithm based on the word frequency and degree of co-occurrences. EmpDG leverages a rule-based method for capturing emotion cause words using EmoLex (Mohammad and Turney, 2013), a largescale lexicon of emotion-relevant words. Finally, we train BERT for emotion classification with the emotion labels in ED. For BERT, we select the words with the largest averaged weight of BERT's last attention heads for the classification token (i.e.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
[CLS]). More details can be found in Appendix.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Dialogue models for base speakers. We experiment our approach on three recent dialogue agents: MIME (Majumder et al., 2020), Dodeca-Transformer , and Blender (Roller et al., 2021). MIME is a dialogue model explicitly targeting empathetic conversation by leveraging emotion mimicry. We select MIME, since it reportedly performs better than other recent empathy-specialized models (Rashkin et al.   Automatic evaluation metrics. For weaklysupervised emotion cause word recognition, we report the Top-1, 3, 5 recall scores.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
For EmpatheticDialogues, we report coverage and two scores for specific empathy expressions (Exploration, Interpretation) measured by pretrained empathy identification models (Sharma et al., 2020). The coverage score refers to the average number of emotion cause words included in the model's generated response.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
The (i) Exploration and (ii) Interpretation are metrics for expressed empathy in text, introduced by Sharma et al. (2020). They both require responses to focus on the interlocutor's utterances and to be specific. (i) Explorations are expressions of active interest in the interlocutor's situation, such as "What happened?" or "So, did you pass the chemistry exam?". The latter is rated as a stronger empathetic response since it asks specifically about the interlocutor's situation. (ii) Interpretations are expressions of acknowledgments or understanding of the interlocutor's emotion or situation, such as "I know your feeling." or "I also had to speak in front of such audience, made me nervous." Expressions of specific understanding are considered to be more empathetic. RoBERTa models (Liu et al., 2019) that are separately pretrained for each metric rate each agent's response by returning values of 0, 1, or 2. Higher scores indicate stronger empathy.  BERT, GEE) perform better. Selecting words by BERT's attention weights does not attain better performance on capturing emotion cause words than GEE. The gap between GEE and other methods widens when the number of returned words from models is more than one (i.e. Top-3, 5).
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
We also evaluate human performance to measure the difficulty of the task. We randomly sample 100 examples from the test set and ask a human evaluator to select five best guesses for the emotion causes. As the performance gap between GEE and human is significantly large, there is much room for further improvement in weakly-supervised emotion cause recognition.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Results on Automatic Evaluation. Table 7 reports the performance of different dialogue agents on EmpatheticDialogues (Rashkin et al., 2019) with automatic evaluation metrics. Our Focused S 1 significantly outperforms the base model S 0 in terms of Interpretation and Exploration scores that measure more focused and specific empathetic expression. We also test the plain pragmatic method (Plain S 1 ) that use random distractors as in previous works (Cohn-Gordon et al., 2018;Kim et al., 2020). The Focused S 1 consistently outperforms Plain S 1 on Interpretation score with similar or better Exploration scores. The Focused S 1 models show higher coverage scores than other mod-   (Majumder et al., 2020) Focused S1 vs S0 46.7% 20.0% 33.3%
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
DodecaTransformer  Focused S1 vs S0 42.1% 28.8% 29.1%
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Blender (Roller et al., 2021) Focused S1 vs S0 44.6% 37.4% 18.0% Table 9: Comparison of our approach (Focused S 1 ) with base speakers (S 0 ) on A/B test for empathetic response generation. The win and lose rates are based on Focused S 1 .
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Blender (Roller et al., 2021) Focused S1 vs Plain S1 39.6% 32.3% 28.0% Table 10: Comparison of our approach (Focused S 1 ) with plain pragmatic approach (Plain S 1 ) on A/B test for reflecting the given emotion cause words. The win and lose rates are based on Focused S 1 . els, indicating they more reflect the context's emotion cause words in responses. As MIME is only trained on EmpatheticDialogues, its Exploration and Interpretations scores are lower than models pretrained on other larger corpus. As a result, we find our approach is effective in both large pretrained open domain dialogue models and empathyspecialized one. We also finetune DodecaTransformer and Blender with explicit emotion information (S 0 +Emotion). Following Rashkin et al. (2019), we concatenate the ground-truth emotion label to the dialogue context during training. At inference, the top predicted emotion from GEE is used. We find the Interpretation or Exploration scores of S 0 +Emotion models drop. Thus, simply adding Emotion: Angry Context: I just moved to this neighborhood and some dumb criminals shot my neighbors and ran into the woods! Emotion cause words from GEE: dumb, neighbors, shot, moved, ran S0: That's terrible. Did you get a chance to talk to them? Plain S1: Oh my god, that is horrible! Where are they? Focused S1: Oh my god, that is horrible! Did they get caught?
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Emotion: Trusting Context: I'm a big believer in listening to your gut. Sometimes I might wonder where my intuition is leading me, but it always seems to be spot on.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Emotion cause words from GEE: believer, leading, intuition, wonder, spot S0: That's great. I'm glad you have a good intuition. Plain S1: Good for you. I have a gut feeling about a lot of things. Focused S1: That's great! I'm a big believer in intuition too.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Context: We were living overseas at the times -the most impressive part was crossing the Grand Canal over a pontoon of boats to finish the marathon ;-) Emotion cause words from GEE: impressive, Canal, Grand, part, finish S0: Wow, that sounds like a lot of fun! I'm glad you had a good time! Plain S1: Wow, that sounds amazing! I bet you were so excited! Focused S1: Wow, that sounds amazing! I've never been to Europe, but I've heard of it. Table 11: Examples of recognized emotion cause words from our GEE and responses from S 0 and Focused S 1 . We underline words where our Focused S 1 reflects the emotion cause words returned by GEE. 2 emotion information is insufficient to make models focus more on the interlocutor's emotional event.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Results on Human Evaluation. We conduct user study and A/B test via Amazon Mechanical Turk. We randomly sample 100 test examples, each rated by three unique human evaluators. Following previous works (Rashkin et al., 2019;Lin et al., 2019;Majumder et al., 2020), we rate empathy, relevance, and fluency of generated responses.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Given the dialogue context and model's generated response, evaluators are asked to rate each criterion in a 4-point Likert scale, where higher scores are better. We also run human A/B test to directly compare the Focused S 1 and base S 0 . We ask three unique human evaluators to vote which response is more empathetic. They can select tie if both responses are thought to be equal. Table 8 and 9 summarizes the averaged human rating and A/B test results on MIME (Majumder et al., 2020), DodecaTransformer , and Blender (Roller et al., 2021). Our Focused S 1 agents are rated more empathetic and relevant to the dialogue context than the base agent S 0 , with better fluency. Also, users prefer responses from our Focused S 1 agent over those from the base agent S 0 . The inter-rater agreement (Krippendorff's α) for human rating and A/B test are 0.26 and 0.27, respectively; implying fair agreement.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
In addition to the coverage score in Table 7, we run A/B test on Blender (Roller et al., 2021) to compare the Focused S 1 and Plain S 1 for reflecting the given emotion cause words in the responses. We random sample 200 test examples and ask three unique human evaluators to vote which response is more focused on the given emotion cause words from the context. Table 10 is the result of A/B test for focused response generation on Blender (Roller et al., 2021). Users rate that responses from Focused S 1 more reflect the emotion cause words than those from the Plain S 1 approach. Thus, both quantitative and qualitative results show that our Focused S 1 approach helps dialogue agents to effectively generate responses focused on given target words.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Examples of the recognized emotion cause words from GEE and generated responses are in Table 11. Our Focused S 1 agent's responses reflect the context's emotion cause words returned from our GEE, implicitly or explicitly.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
We studied how to use a generative estimator for identifying emotion cause words from utterances based solely on emotion labels without word-level labels (i.e. weakly-supervised emotion cause word recognition). To evaluate our approach, we introduce EMOCAUSE evaluation set where we manually annotated emotion cause words on situations in EmpatheticDialogues (Rashkin et al., 2019). We release the evaluation set to the public for future research. We also proposed a novel method for controlling the Rational Speech Acts (RSA) framework (Frank and Goodman, 2012) to make models generate empathetic responses focused on targeted words in the dialogue context. Since the RSA framework requires no additional training, our approach is orthogonally applicable to any pretrained dialogue agents on the fly. An interesting direction for future work will be reasoning how the interlocutor would react to the model's empathetic response. Such reasoning is an essential part for expressing empathy.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Weakly-supervised emotion cause word recognition. We use rake-nltk 3 to implement RAKE (Rose et al., 2010), and the official code of Em-pDG 4 from the authors (Li et al., 2020). We respectively finetune BERT-based-uncased (Devlin et al., 2019) for BERT-Attention and BART-large (Lewis et al., 2020) for our generative emotion estimator (GEE). We set a learning rate to 3e-5 for BERT-Attention and 1e-5 for GEE. Other than the learning rate, we follow the default hyperparameters in ParlAI framework 5 (Miller et al., 2017). We select the best performing checkpoint using the Top-1 recall for emotion cause word recognition on the validation set. We run experiments 5 times with different random seeds and report averaged scores on Table 6.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Dialogue models. We use MIME (Majumder et al., 2020), DodecaTransformer , and Blender 90M (Roller et al., 2021) as dialogue models for base speakers. For MIME, we use the codes and pretrained weights of the authors' official implementation 6 as is. For DodecaTransformer and Blender, we use the ParlAI framework with the default hyperparameters and finetune them on EmpatheticDialogues (Rashkin et al., 2019). We select the best performing checkpoint via perplexity on the validation set.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
During inference, we use greedy decoding and set RSA parameter α and β to 2.0 and 0.9 for MIME, 3.0 and 0.9 for DodecaTransformer, and 4.0 and 0.9 for Blender. We select the best performing α and β from the candidates of [1.0, 2.0, 3.0, 4.0] and [0.5, 0.6, 0.7, 0.8, 0.9, 1.0] with one trial for each. Inference on the test set of EmpatheticDialogues takes 0.4 hours with Blender 90M base speaker.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Evaluation metrics. To compute Exploration and Interpretation scores (Sharma et al., 2020), we separately finetune RoBERTa-base for each score using the author's official code 7 .
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
Sensitivity to k of top-k emotion cause words. In all experiments, we use k = 5, which is found by validation with k = 1, 2, 4, 8 using Blender (Roller et al., 2021)   Experiments for emotion cause word recognition and emotion classification are run on one NVIDIA Quadro RTX 6000 GPU. Experiments for empathetic response generation are run on two GPUs.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
We report the classification performance of emotion classifiers used in empathetic response generation.    Table 15 shows Top-10 frequent cause words per emotion. Interestingly, same words can be seen in both positive and negative emotions. For example, we can find the word interview on both "Anxious" and "Confident". "Anticipating" and "Disappointed" are closely related to vacation. This result shows that understanding the context is one of key prerequisites for emotion cause word recognition.
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
We just got a new puppy . My older dog knew to let that one out first when I get home from work .
Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes
My boyfriend is going out with a bunch of people I do n't know tonight . But I trust him that he will be a good boy .   
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Approximately 20% of people worldwide are suffering from a mental health disorder (Holmes et al., 2018). Still, access to mental health care remains a global challenge with widespread shortages of workforce (Olfson, 2016). Facing limited in-person treatment options and other barriers like stigma (White and Dorman, 2001), millions of people are turning to text-based peer support platforms Dr. Atkins is a co-founder with equity stake in a technology company, Lyssn.io, focused on tools to support training, supervision, and quality assurance of psychotherapy and counseling My whole family hates me. I don't see any point in living.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Approximately 20% of people worldwide are suffering from a mental health disorder (Holmes et al., 2018). Still, access to mental health care remains a global challenge with widespread shortages of workforce (Olfson, 2016). Facing limited in-person treatment options and other barriers like stigma (White and Dorman, 2001), millions of people are turning to text-based peer support platforms Dr. Atkins is a co-founder with equity stake in a technology company, Lyssn.io, focused on tools to support training, supervision, and quality assurance of psychotherapy and counseling My whole family hates me. I don't see any point in living.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
I understand how you feel. Let me know if you want to talk. Everything will be fine.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
I wonder if this makes you feel isolated. Let me know if you want to talk.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
If that happened to me, I would feel really isolated. Let me know if you want to talk. I really hope things would improve. We differentiate between no communication, weak communication, and strong communication of these factors. Our computational approach simultaneously identifies these mechanisms and the underlying rationale phrases (highlighted portions). All examples in this paper have been anonymized using best practices in privacy and security (Matthews et al., 2017).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
If that happened to me, I would feel really isolated. Let me know if you want to talk. I really hope things would improve. We differentiate between no communication, weak communication, and strong communication of these factors. Our computational approach simultaneously identifies these mechanisms and the underlying rationale phrases (highlighted portions). All examples in this paper have been anonymized using best practices in privacy and security (Matthews et al., 2017).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
such as TalkLife (talklife.co) to express emotions, share stigmatized experiences, and receive peer support (Eysenbach et al., 2004). However, while peer supporters on these platforms are motivated and well-intentioned to help others seeking support (henceforth seeker), they are untrained and typically unaware of best-practices in therapy.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
In therapy, interacting empathically with seekers is fundamental to success (Bohart et al., 2002;Elliott et al., 2018). The lack of training or feedback to layperson peer supporters results in missed opportunities to offer empathic textual responses. NLP systems that understand conversational empathy could empower peer supporters with feedback and training. However, the current understanding of empathy is limited to traditional face-to-face, speech-based therapy (Gibson et al., 2016;Pérez-Rosas et al., 2017) due to lack of resources and methods for new asynchronous, text-based interactions (Patel et al., 2019). Also, while previous NLP research has focused predominantly on empathy as reacting with emotions of warmth and compassion (Buechel et al., 2018), a separate but key aspect of empathy is to communicate a cognitive understanding of others (Selman, 1980).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
In therapy, interacting empathically with seekers is fundamental to success (Bohart et al., 2002;Elliott et al., 2018). The lack of training or feedback to layperson peer supporters results in missed opportunities to offer empathic textual responses. NLP systems that understand conversational empathy could empower peer supporters with feedback and training. However, the current understanding of empathy is limited to traditional face-to-face, speech-based therapy (Gibson et al., 2016;Pérez-Rosas et al., 2017) due to lack of resources and methods for new asynchronous, text-based interactions (Patel et al., 2019). Also, while previous NLP research has focused predominantly on empathy as reacting with emotions of warmth and compassion (Buechel et al., 2018), a separate but key aspect of empathy is to communicate a cognitive understanding of others (Selman, 1980).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
In this work, we present a novel computational approach to understanding how empathy is expressed in text-based, asynchronous mental health conversations. We introduce EPITOME, 1 a conceptual framework for characterizing communication of empathy in conversations that synthesizes and adapts the most prominent empathy scales from speech-based, face-to-face contexts to text-based, asynchronous contexts ( §3). EPITOME consists of three communication mechanisms of empathy: Emotional Reactions, Interpretations, and Explorations (Fig. 1).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
To facilitate computational modeling of empathy in text, we create a new corpus based on EPITOME. We collect annotations on a dataset of 10k (post, response) pairs from extensively-trained crowdworkers with high inter-rater reliability ( §4). 2 We develop a RoBERTa-based bi-encoder model for identifying empathy communication mechanisms in conversations ( §5). Our multi-task model simultaneously extracts the underlying supportive evidences, rationales (DeYoung et al., 2020), for its predictions (spans of input post; e.g., highlighted portions in Fig. 1) which serve the dual role of (1) explaining the model's decisions, thus minimizing the risk of deploying harmful technologies in sensitive contexts, and (2) enabling rationale-augmented feedback for peer supporters.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We show that our computational approach can effectively identify empathic conversations with underlying rationales (∼80% acc., ∼70% macro-f1) and outperforms popular NLP baselines with a 4-point gain in macro-f1 ( §6). We apply our model to a dataset of 235k supportive conversations on TalkLife and demonstrate that empathy is associated with positive feedback from seekers and the forming of relationships. Importantly, our 1 EmPathy In Text-based, asynchrOnous MEntal health conversations 2 Our dataset can be accessed from https://bit.ly/ 2Rwy2gx. results suggest that most peer supporters do not self-learn empathy with time. This points to critical opportunities for training and feedback for peer supporters to increase the effectiveness of mental health support (Miner et al., 2019;. Specifically, NLP-based tools could give actionable, real-time feedback to improve expressed empathy, and we demonstrate this idea in a smallscale proof-of-concept ( §7).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We show that our computational approach can effectively identify empathic conversations with underlying rationales (∼80% acc., ∼70% macro-f1) and outperforms popular NLP baselines with a 4-point gain in macro-f1 ( §6). We apply our model to a dataset of 235k supportive conversations on TalkLife and demonstrate that empathy is associated with positive feedback from seekers and the forming of relationships. Importantly, our 1 EmPathy In Text-based, asynchrOnous MEntal health conversations 2 Our dataset can be accessed from https://bit.ly/ 2Rwy2gx. results suggest that most peer supporters do not self-learn empathy with time. This points to critical opportunities for training and feedback for peer supporters to increase the effectiveness of mental health support (Miner et al., 2019;. Specifically, NLP-based tools could give actionable, real-time feedback to improve expressed empathy, and we demonstrate this idea in a smallscale proof-of-concept ( §7).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
2 Background 2.1 How to measure empathy?
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Empathy is a complex multi-dimensional construct with two broad aspects related to emotion and cognition (Davis et al., 1980). The emotion aspect relates to the emotional stimulation in reaction to the experiences and feelings expressed by a user. The cognition aspect is a more deliberate process of understanding and interpreting the experiences and feelings of the user and communicating that understanding to them (Elliott et al., 2018).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Empathy is a complex multi-dimensional construct with two broad aspects related to emotion and cognition (Davis et al., 1980). The emotion aspect relates to the emotional stimulation in reaction to the experiences and feelings expressed by a user. The cognition aspect is a more deliberate process of understanding and interpreting the experiences and feelings of the user and communicating that understanding to them (Elliott et al., 2018).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Here, we study expressed empathy in text-based mental health support -empathy expressed or communicated by peer supporters in their textual interactions with seekers (cf. Barrett-Lennard (1981)). 3 Table 1 lists existing empathy scales in psychology and psychotherapy research. Truax and Carkhuff (1967) focus only on communicating cognitive understanding of others while Davis et al. (1980); also make use of expressing stimulated emotions.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Here, we study expressed empathy in text-based mental health support -empathy expressed or communicated by peer supporters in their textual interactions with seekers (cf. Barrett-Lennard (1981)). 3 Table 1 lists existing empathy scales in psychology and psychotherapy research. Truax and Carkhuff (1967) focus only on communicating cognitive understanding of others while Davis et al. (1980); also make use of expressing stimulated emotions.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
These scales, however, have been designed for in-person interactions and face-to-face therapy, often leveraging audio-visual signals like expressive voice. In contrast, in text-based support, empathy must be expressed using textual response alone. Also, they are designed to operate on long, synchronous conversations and are unsuited for the shorter, asynchronous conversations of our context.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
In this work, we adapt these scales to text-based, asynchronous support. We develop a new comprehensive framework for text-based, asynchronous conversations (Table 1; §3), use it to create a new dataset of empathic conversations ( §4), a computational approach for identifying empathy ( §5; §6), & gaining insights into mental health platforms ( §7).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
3 Note that expressed empathy may differ from the empathy perceived by seekers. However, obtaining perceived empathy ratings from seekers is challenging in sensitive contexts and involves ethical risks. Psychotherapy research indicates a strong correlation of expressed empathy with positive outcomes and frequently uses it as a credible alternative (Robert et al., 2011).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Applicable to text-based peer-support  Truax and Carkhuff (1967) Face-to-face therapy Davis et al. (1980) Daily human interactions  Face-to-face therapy Methods Buechel et al. (2018) Reaction to news stories Rashkin et al. (2019) Emotionally grounded convs. * * * Pérez- Rosas et al. (2017) Motivational interviewing EPITOME Text-based, asynchronous support Table 1: EPITOME incorporates both emotional and cognitive aspects of empathy that were previously only studied in face-to-face therapy and never computationally in text-based, asynchronous conversations. *Rashkin et al. 
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Applicable to text-based peer-support  Truax and Carkhuff (1967) Face-to-face therapy Davis et al. (1980) Daily human interactions  Face-to-face therapy Methods Buechel et al. (2018) Reaction to news stories Rashkin et al. (2019) Emotionally grounded convs. * * * Pérez- Rosas et al. (2017) Motivational interviewing EPITOME Text-based, asynchronous support Table 1: EPITOME incorporates both emotional and cognitive aspects of empathy that were previously only studied in face-to-face therapy and never computationally in text-based, asynchronous conversations. *Rashkin et al. 
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Computational research on empathy is based on speech-based settings, exploiting audio signals like pitch which are unavailable in text-based platforms (Gibson et al., 2016;Pérez-Rosas et al., 2017). Moreover, previous NLP research has predominantly focused on empathy as reacting with emotions of warmth and compassion (Buechel et al., 2018). For mental health support, however, communicating cognitive understanding of feelings and experiences of others is more valued (Selman, 1980). Recent work also suggests that grounding conversations in emotions implicitly makes them empathic (Rashkin et al., 2019). Research in therapy, however, highlights the importance of expressing empathy in interactions (Truax and Carkhuff, 1967). In this work, we present a computational approach to (1) understanding empathy expressed in textual, asynchronous conversations; (2) address both emotional and cognitive aspects of empathy.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Computational research on empathy is based on speech-based settings, exploiting audio signals like pitch which are unavailable in text-based platforms (Gibson et al., 2016;Pérez-Rosas et al., 2017). Moreover, previous NLP research has predominantly focused on empathy as reacting with emotions of warmth and compassion (Buechel et al., 2018). For mental health support, however, communicating cognitive understanding of feelings and experiences of others is more valued (Selman, 1980). Recent work also suggests that grounding conversations in emotions implicitly makes them empathic (Rashkin et al., 2019). Research in therapy, however, highlights the importance of expressing empathy in interactions (Truax and Carkhuff, 1967). In this work, we present a computational approach to (1) understanding empathy expressed in textual, asynchronous conversations; (2) address both emotional and cognitive aspects of empathy.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
To understand empathy in text-based, asynchronous, peer-to-peer support conversations, we develop EPITOME, a new conceptual framework of expressed empathy (Fig. 1). In collaboration with clinical psychologists, we adapt and synthesize existing empathy definitions and scales to textbased, asynchronous context. EPITOME consists of three communication mechanisms providing a comprehensive outlook of empathy -Emotional Reactions, Interpretations, and Explorations. For each of these mechanisms, we differentiate between -(0) peers not expressing them at all (no communication), (1) peers expressing them to some weak degree (weak communication), (2) peers expressing them strongly (strong communication).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Here, we describe our framework in detail: 4 Emotional Reactions. Expressing emotions such as warmth, compassion, and concern, experienced by peer supporter after reading seekers post. Expressing these emotions plays an important role in establishing empathic rapport and support (Robert et al., 2011). A weak communication of emotional reactions alludes to these emotions without the emotions being explicitly labeled (e.g., Everything will be fine). On the other hand, strong communication specifies the experienced emotions (e.g., I feel really sad for you).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Interpretations. Communicating an understanding of feelings and experiences inferred from the seekers post. Such a cognitive understanding in responses is helpful in increasing awareness of hidden feelings and experiences, and essential for developing alliance between the seeker and peer supporter (Watson, 2007). A weak communication of interpretations contains a mention of the understanding (e.g., I understand how you feel) while a strong communication specifies the inferred feeling or experience (e.g., This must be terrifying) or communicates understanding through descriptions of similar experiences (e.g., I also have anxiety attacks at times which makes me really terrified).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Explorations. Improving understanding of the seeker by exploring the feelings and experiences not stated in the post. Showing an active interest in what the seeker is experiencing and feeling and probing gently is another important aspect of empathy (Miller et al., 2003;Robert et al., 2011). A weak exploration is generic (e.g., What happened?) while a strong exploration is specific and labels the seeker's experiences and feelings which the peer supporter wants to explore (e.g., Are you feeling alone right now?).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Consistent with existing scales, responses that only give advice (Try talking to friends), only provide factual information (mindful meditation overcomes anxiety), or are offensive or abusive (shut the f**k up) 5 are not empathic and are characterized as no communication of empathy in our framework.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
To facilitate computational methods for empathy, we collect data based on EPITOME.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We use conversations on the following two online support platforms as our data source:
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
(1) TalkLife. TalkLife (talklife.co) is the largest global peer-to-peer mental health support network. It enables seekers to have textual interactions with peer supporters through conversational threads. The dataset contains 6.4M threads and 18M interactions (seeker post, response post pairs).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
(2) Mental Health Subreddits. Reddit (reddit. com) hosts a number of sub-communities aka subreddits (e.g., r/depression). We use threads posted on 55 mental health focused subreddits (Sharma et al. (2018)). This publicly accessible dataset contains 1.6M threads and 8M interactions.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
(2) Mental Health Subreddits. Reddit (reddit. com) hosts a number of sub-communities aka subreddits (e.g., r/depression). We use threads posted on 55 mental health focused subreddits (Sharma et al. (2018)). This publicly accessible dataset contains 1.6M threads and 8M interactions.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We use the entire dataset for in-domain pretraining ( §5) and annotate a subset of 10k interactions on empathy. We further analyze empathy on a carefully filtered dataset of 235k mental health interactions on TalkLife ( §7).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Empathy is conceptually nuanced and linguistically diverse so annotating it accurately is difficult in short-term crowdwork approaches. This is also reflected in prior work that found it challenging to annotate therapeutic constructs (Lee et al., 2019). To ensure high inter-rater reliability, we designed a novel training-based annotation process. Crowdworkers Recruiting and Training. We recruited and trained eight crowdworkers on identifying empathy mechanisms in EPITOME. We leveraged Upwork (upwork.com), a freelancing platform that allowed us to hire and work interactively with crowdworkers. Each crowdworker was trained  TalkLife  3656 2945  461  7062  Reddit  2034  899  148  3081   Interpretations  TalkLife  5533  178  1351  7062  Reddit  1645  115  1321  3081   Explorations  TalkLife  5137  767  1158  7062  Reddit  2600  104  377  3081   Table 2: Statistics of the collected empathy dataset. The crowdworkers were trained on EPITOME through a series of phone calls and manual/automated feedback on sample posts to ensure high quality annotations.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Empathy is conceptually nuanced and linguistically diverse so annotating it accurately is difficult in short-term crowdwork approaches. This is also reflected in prior work that found it challenging to annotate therapeutic constructs (Lee et al., 2019). To ensure high inter-rater reliability, we designed a novel training-based annotation process. Crowdworkers Recruiting and Training. We recruited and trained eight crowdworkers on identifying empathy mechanisms in EPITOME. We leveraged Upwork (upwork.com), a freelancing platform that allowed us to hire and work interactively with crowdworkers. Each crowdworker was trained  TalkLife  3656 2945  461  7062  Reddit  2034  899  148  3081   Interpretations  TalkLife  5533  178  1351  7062  Reddit  1645  115  1321  3081   Explorations  TalkLife  5137  767  1158  7062  Reddit  2600  104  377  3081   Table 2: Statistics of the collected empathy dataset. The crowdworkers were trained on EPITOME through a series of phone calls and manual/automated feedback on sample posts to ensure high quality annotations.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
through a series of phone calls (30 minutes to 1 hour in total) and manual/automated feedback on 50-100 posts. Refer Appendix A for more details.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Annotating Empathy. In our annotation task, crowdworkers were shown a pair of (seeker post, response post) and were asked to identify the presence of the three communication mechanisms in EPITOME (Emotional Reactions, Interpretations, and Explorations), one at a time. For each mechanism, crowdworkers annotated whether the response post contained no communication, weak communication, or strong communication of empathy in the context of the seeker post.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Highlighting Rationales. Along with the categorical annotations, crowdworkers were also asked to highlight portions of the response post that formed rationale behind their annotation. E.g, in the post That must be terrible! I'm here for you, the portion That must be terrible is the rationale for it being a strong communication of interpretations. Data Quality. Overall, our corpus has an average inter-annotator agreement of 0.6865 (average over pairwise Cohen's κ of all pairs of crowdworkers; each pair annotated >50 posts in common) which is higher than previously reported values for the annotation of empathy in face-to-face therapy (∼0.60 in Pérez-Rosas et al., 2017;Lord et al., 2015). Our ground-truth corpus contains 10,143 (seeker post, response post) pairs with annotated empathy labels from trained crowdworkers (Table 2). Privacy and Ethics. The TalkLife dataset was sourced with license and consent from the TalkLife platform. All personally identifiable information (user, platform identifiers) in both the datasets were removed. This study was approved by University of Washington's Institutional Review Board. In addition, we tried to minimize the risks of annotating mental health related content by providing crisis management resources to our annotators, follow-ing Sap et al. (2020). This work does not make any treatment recommendations or diagnostic claims.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Highlighting Rationales. Along with the categorical annotations, crowdworkers were also asked to highlight portions of the response post that formed rationale behind their annotation. E.g, in the post That must be terrible! I'm here for you, the portion That must be terrible is the rationale for it being a strong communication of interpretations. Data Quality. Overall, our corpus has an average inter-annotator agreement of 0.6865 (average over pairwise Cohen's κ of all pairs of crowdworkers; each pair annotated >50 posts in common) which is higher than previously reported values for the annotation of empathy in face-to-face therapy (∼0.60 in Pérez-Rosas et al., 2017;Lord et al., 2015). Our ground-truth corpus contains 10,143 (seeker post, response post) pairs with annotated empathy labels from trained crowdworkers (Table 2). Privacy and Ethics. The TalkLife dataset was sourced with license and consent from the TalkLife platform. All personally identifiable information (user, platform identifiers) in both the datasets were removed. This study was approved by University of Washington's Institutional Review Board. In addition, we tried to minimize the risks of annotating mental health related content by providing crisis management resources to our annotators, follow-ing Sap et al. (2020). This work does not make any treatment recommendations or diagnostic claims.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
With our collected dataset, we develop a computational approach for understanding empathy.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Let S i = s i1 , ..., s im be a seeker post and R i = r i1 , ..., r in be a corresponding response post. For the pair (S i , R i ), we want to perform two tasks:
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Task 1: Empathy Identification. Identify how empathic R i is in the context of S i . For each of the three communication mechanisms in EPIT-OME (Emotional Reactions, Interpretations, Explorations), we want to identify their level of communication (l i ) in R i -no communication (0), weak communication (1), or strong communication (2).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Task 2: Rationale Extraction. Extract rationales underlying the identified level l i ∈ {no, weak, strong} of each of the three communication mechanism in EPITOME. The extracted rationale is a subsequence of words x i in R i . We represent this subsequence as a mask m i = (m i1 , ..., m in ) over the words in R i , where m ij ∈ {0, 1} is a boolean variable: 1 -rationale word; 0 -non-rationale word.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Correspondingly, x i = m i R i .
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We propose a multi-task bi-encoder model based on RoBERTa (Liu et al., 2019) for identifying empathy and extracting rationales (Fig. 2). We multi-task over the two tasks of empathy identification and rationale extraction and train three independent but identical architectures for the three empathy communication mechanisms in EPITOME ( §3). The bi-encoder architecture (Humeau et al., 2019) facilitates a joint modeling of (S i , R i ) pairs. Moreover, the use of attention helps in providing context from the seeker post, S i . We find that such an approach is more effective than methods that concatenate S i with R i with a [SEP] token to form a single input sequence ( §6).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We propose a multi-task bi-encoder model based on RoBERTa (Liu et al., 2019) for identifying empathy and extracting rationales (Fig. 2). We multi-task over the two tasks of empathy identification and rationale extraction and train three independent but identical architectures for the three empathy communication mechanisms in EPITOME ( §3). The bi-encoder architecture (Humeau et al., 2019) facilitates a joint modeling of (S i , R i ) pairs. Moreover, the use of attention helps in providing context from the seeker post, S i . We find that such an approach is more effective than methods that concatenate S i with R i with a [SEP] token to form a single input sequence ( §6).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Two Encoders. Our model uses two independently pre-trained transformer encoders from RoBERTa BASE -S-Encoder & R-Encoder -for encoding seeker post and response post respectively. S-Encoder encodes context from the seeker post whereas R-Encoder is responsible for understanding empathy in the response post. Domain-Adaptive Pre-training. Both the S-Encoder and R-Encoder are initialized using the weights learned by RoBERTa BASE . We further perform a domain-adaptive pre-training (Gururangan et al., 2020) of the two encoders to adapt to conversational and mental health context. For this additional pre-training of the two encoders, we use the datasets of 6.4M seeker posts (182M tokens) and 18M response (279M tokens) posts respectively sourced from TalkLife ( §4). We use the masked language modeling task for pre-training (3 epochs, batch size = 8). Attention Layer. We use a single-head attention over the two encodings for generating seekercontext aware representation of the response post. Using the terminology of transformers (Vaswani et al., 2017), our query is the response post encoding e 
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Two Encoders. Our model uses two independently pre-trained transformer encoders from RoBERTa BASE -S-Encoder & R-Encoder -for encoding seeker post and response post respectively. S-Encoder encodes context from the seeker post whereas R-Encoder is responsible for understanding empathy in the response post. Domain-Adaptive Pre-training. Both the S-Encoder and R-Encoder are initialized using the weights learned by RoBERTa BASE . We further perform a domain-adaptive pre-training (Gururangan et al., 2020) of the two encoders to adapt to conversational and mental health context. For this additional pre-training of the two encoders, we use the datasets of 6.4M seeker posts (182M tokens) and 18M response (279M tokens) posts respectively sourced from TalkLife ( §4). We use the masked language modeling task for pre-training (3 epochs, batch size = 8). Attention Layer. We use a single-head attention over the two encodings for generating seekercontext aware representation of the response post. Using the terminology of transformers (Vaswani et al., 2017), our query is the response post encoding e 
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
where d = 768 (hidden size in RoBERTa BASE ). We sum the encoded response e    and pass it through a linear layer to get the predictions of the empathy levell i (0, 1, or 2) of each empathy communication mechanism. Note that we train three independent models for the three communication mechanisms in EPITOME ( §3).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Extracting Rationales. For extracting rationales y i underlying the predictions, we use final representations of the individual tokens in R i (h (R) i [r i1 , ..., r in ]) and pass them through a linear layer for making boolean predictions,m i . Loss Function. We use cross-entropy between the true and predicted labels as the loss functions of our two tasks. The overall loss of our multi-task architecture is: L = λ EI * L EI + λ RE * L RE .
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Experimental Setup. We split both the datasets into train, dev, and test sets (75:5:20). We train our model for 4 epochs using a learning rate of 2e−5, batch size of 32, λ EI = 1, and λ RE = 0.5 (Refer Appendix B for fine-tuning details).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Next, we analyze how effectively we can identify empathy with underlying rationales using our computational approach.  Table 4: Rationale extraction task results. We evaluate both at the level of tokens (T-f1) and spans (IOU-f1).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We compare the performance of our approach with a range of models popularly used in related tasks (e.g., sentiment classification, conversation analysis). We quantify how challenging identifying empathy with underlying rationales is, how well do existing models perform, and what performance is achieved by our proposed approach. Baselines. Our baselines are: 1. Log. reg. (logistic regression over tf.idf vectors); 2. RNN (twolayer recurrent neural network); 3. HRED (hierarchical recurrent encoder-decoder, often used for modeling conversations (Sordoni et al., 2015)); 4. BERT BASE (Devlin et al., 2019); 5. GPT-2 (typically used for language generation (Radford et al., 2019)); 6. DialoGPT (GPT-2 adapted to asynchronous conversations ); and 7. RoBERTa BASE (Liu et al., 2019).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We compare the performance of our approach with a range of models popularly used in related tasks (e.g., sentiment classification, conversation analysis). We quantify how challenging identifying empathy with underlying rationales is, how well do existing models perform, and what performance is achieved by our proposed approach. Baselines. Our baselines are: 1. Log. reg. (logistic regression over tf.idf vectors); 2. RNN (twolayer recurrent neural network); 3. HRED (hierarchical recurrent encoder-decoder, often used for modeling conversations (Sordoni et al., 2015)); 4. BERT BASE (Devlin et al., 2019); 5. GPT-2 (typically used for language generation (Radford et al., 2019)); 6. DialoGPT (GPT-2 adapted to asynchronous conversations ); and 7. RoBERTa BASE (Liu et al., 2019).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Empathy Identification Task. Table 3 reports the accuracy and macro-f1 scores of the three communication mechanisms (random baseline for each is 33% accurate; three levels). Log. reg., RNN, and HRED struggle to identify empathy with noticeably low macro-f1 scores indicative of failures to distinguish between the three levels of communication.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Among the baseline transformer architectures, we obtain best performance using RoBERTa but observe substantial gains over them with our approach (+1.73 acc., +4.02 macro-f1 over RoBERTa). We analyze the sources of these gains in §6.2. Rationale Extraction Task. We perform both token level and span level evaluation for this task. We use two metrics, commonly used in discrete  Table 5: Ablation results. Most of our gains are due to context provided through attention and seeker post; higher gains for the rationale extraction task. *Note that rationales cannot be predicted after removing them from training. rationale extraction tasks (DeYoung et al., 2020): 1. T-f1 (token level f1); 2. IOU-f1 (intersection over union overlap of predicted spans with ground truth spans; threshold of 0.5 on the overlap for finding true positives and the corresponding f1). We find that GPT-2 and DialoGPT perform better than BERT and RoBERTa likely due to appropriateness to the related task of generating free-text rationales (Table 4). Our approach obtains gains of +2.58 T-f1 and +6.45 IOU-f1 over DialoGPT, potentially due to the use of attention and seeker post ( §6.2).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We next analyze the components and training strategies in our approach through an ablation study.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
No Attention. Instead of using attention, we concatenate the seeker post encoding (e No Seeker Post. We train without the S-Encoder, i.e., by only encoding from the R-Encoder.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
No Rationales. We set λ RE to 0 and only train on the empathy identification task.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
No Domain-Adaptive Pre-training. We initialize by only using model weights from RoBERTa BASE .
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Results. Our most significant gains come from using attention and the seeker post (Table 5) which greatly benefits the rationale extraction task (+4.88 T-f1, +5.74 IOU-f1). Also, using rationales and pre-training only leads to small performance improvements.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We qualitatively analyze the sources of our errors. We found that the model sometimes failed to iden- tity short expressions of emotions in responses that otherwise contained a lot of instructions (e.g., Sorry to hear that! Try doing ...). Also, certain responses trying to universalize the situation (e.g., You are not alone) got incorrectly identified as strong interpretations. Furthermore, a source of error for explorations was confusions due to questions that were not an exploration of seeker's feelings or experiences (e.g., offers to talk -Do you want to talk?).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We apply our model to study how empathy impacts online peer-to-peer support dynamics. To only focus on conversations related to significant mental health challenges and filter out common social me-
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Original Response Re-written Response I cannot do anything without getting blamed today. This day is getting worse and worse.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Days end, tomorrow is a fresh start.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
I'm sorry that today sucks, but tomorrow is a fresh start.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
An hour ago i was happy an hour later i'm sad. Am i getting mad now?
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
That's something Ive struggled with too, and it really pains me to hear that youre dealing with the same thing. Have you considered trying meditation? I've found it to be very helpful. dia interactions (e.g., Merry Christmas), we carefully select 235k mental health related interactions on TalkLife using a seeker-reported indicator. 6 We investigate (1) the levels of empathy on the platform, its variation over time, and examine the relationship of empathy with (2) conversation outcomes, (3) relationship forming, and (4) gender.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
(1) Peer supporters do not self-learn empathy over time. Overall, we observe that empathy expressed by peer supporters on the platform is low (avg. total score 7 of 1.09 out of 6). In addition, we find that the emotional reactivity of users decreases over time (36% decrease over three years) and their levels of interpretations and explorations remain practically constant (Fig. 3a). This is also reflected in prior work on therapy that shows that without deliberate practice and specific feedback, even trained therapists often diminish in skills over time (Goldberg et al., 2016). We find this trend robust to potential confounding factors (new users, user dropout) and users of different groups (low vs. high activity users, moderators; Appendix C). This indicates that most users do not self-learn empathy and highlights the need of providing them feedback.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
(2) High empathy interactions are received positively by seekers. We analyze the correlation of empathic conversations with positive feedback, concretely with seeker "liking" the post. We find that strong communications of empathy are received with 45% more likes by seekers than no communication (Fig. 3b). Strong explorations get 44% less likes but receive 47% more replies than no explorations, leading to higher engagement.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
(3) Relationship forming more likely after empathic conversations. Psychology research emphasizes the importance of empathy in forming alliance and relationship with seekers (Watson, 2007). 6 We focus analyzing TalkLife alone as Reddit lacks rich publicly available signals like seeker liking the response. 7 Total empathy score is obtained by adding the level of communication across the three mechanisms.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Here, we operationalize relationship forming as seeker "following" the peer supporter after a conversation (within 24hrs). We find that seekers are 79% more likely to follow peer supporters after an empathic conversation (total score of 1+ vs. 0) than after a non-empathic one (Fig. 3c).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
(4) Females are more empathic with females than males are with males. Previous work has shown that seekers identifying as females receive more support in online communities (Wang and Jurgens, 2018). Here, we ask if empathic interactions are affected by the self-reported gender of seekers and peer supporters. We find that female peer supporters are 32% more empathic towards female seekers than males are towards male seekers (Fig. 3d). Also, females are 6% more empathic towards males than males are towards females. Implications for empathy-based feedback. These results suggest that our approach not only successfully measures empathy according to a principled framework ( §3), but that the measured empathy components are important to online supportive conversations as indicated by the positive reactions from seekers and meaningful reflections of social theories. However, peer supporters on the platform express empathy rarely and this does not improve over time. This points to critical opportunities for empathy-based feedback to peer supporters for making their interactions with seekers more effective. Here, we demonstrate the potential of feedback in a simple proof-ofconcept. When providing three participants (none are co-authors) simple feedback (Appendix D) based on EPITOME and our best-performing model, they were able to increase empathy in responses from 0.8 to 3.0 (total empathy across the three mechanisms). Table 6 shows two such examples of re-written responses that improve in communicating cognitive understanding (today sucks) and are also better with emotional reactions (I'm sorry, it pains me) and explorations (Have you considered trying mindful meditation?).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
(4) Females are more empathic with females than males are with males. Previous work has shown that seekers identifying as females receive more support in online communities (Wang and Jurgens, 2018). Here, we ask if empathic interactions are affected by the self-reported gender of seekers and peer supporters. We find that female peer supporters are 32% more empathic towards female seekers than males are towards male seekers (Fig. 3d). Also, females are 6% more empathic towards males than males are towards females. Implications for empathy-based feedback. These results suggest that our approach not only successfully measures empathy according to a principled framework ( §3), but that the measured empathy components are important to online supportive conversations as indicated by the positive reactions from seekers and meaningful reflections of social theories. However, peer supporters on the platform express empathy rarely and this does not improve over time. This points to critical opportunities for empathy-based feedback to peer supporters for making their interactions with seekers more effective. Here, we demonstrate the potential of feedback in a simple proof-ofconcept. When providing three participants (none are co-authors) simple feedback (Appendix D) based on EPITOME and our best-performing model, they were able to increase empathy in responses from 0.8 to 3.0 (total empathy across the three mechanisms). Table 6 shows two such examples of re-written responses that improve in communicating cognitive understanding (today sucks) and are also better with emotional reactions (I'm sorry, it pains me) and explorations (Have you considered trying mindful meditation?).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Previous work in NLP for mental health has focused on analysis of effective conversation strategies (Althoff et al., 2016;Pérez-Rosas et al., 2019;Zhang and Danescu-Niculescu-Mizil, 2020), identification of therapeutic actions (Lee et al., 2019), and language development of counselors (Zhang et al., 2019). Researchers have also analyzed linguistic accommodation (Sharma and De Choudhury, 2018), cognitive restructuring (Pruksachatkun et al., 2019), and selfdisclosure (Yang et al., 2019). We extend these studies and analyze empathy which is key in counseling and mental health support. Recent work has also developed proof-of-concept prototypes, such as ClientBot (Huang et al., 2020), for training users in counseling. Our approach is aimed towards developing empathy-based feedback and training systems for peer supporters (consistent with calls to action for improved treatment access and training (Miner et al., 2019;Kazdin and Rabbitt, 2013)).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Previous work in NLP for mental health has focused on analysis of effective conversation strategies (Althoff et al., 2016;Pérez-Rosas et al., 2019;Zhang and Danescu-Niculescu-Mizil, 2020), identification of therapeutic actions (Lee et al., 2019), and language development of counselors (Zhang et al., 2019). Researchers have also analyzed linguistic accommodation (Sharma and De Choudhury, 2018), cognitive restructuring (Pruksachatkun et al., 2019), and selfdisclosure (Yang et al., 2019). We extend these studies and analyze empathy which is key in counseling and mental health support. Recent work has also developed proof-of-concept prototypes, such as ClientBot (Huang et al., 2020), for training users in counseling. Our approach is aimed towards developing empathy-based feedback and training systems for peer supporters (consistent with calls to action for improved treatment access and training (Miner et al., 2019;Kazdin and Rabbitt, 2013)).
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We developed a new framework, dataset, and computational method for understanding expressed empathy in text-based, asynchronous conversations on mental health platforms. Our computational approach effectively identifies empathy with underlying rationales. Moreover, the identified components are found to be important to mental health platforms and helpful in improving peer-to-peer support through model-based feedback.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Code. Our codes are based on the huggingface library (https://huggingface.co/). We make them publicly available at https://github.com/behavioral-data/Empathy-Mental-Health. Seed Value. For all our experiments, we used the seed value of 12.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We searched through the following space of hyperparameters for fine-tuning our model:
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
• learning rate = 1e-5, 2e-5, 5e-5, 1e-4, 5e-4
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
• λ EI = 1
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
• λ RE = 0.1, 0.2, 0.5, 1
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
Domain-Adaptive Pre-training Time. We conducted domain-adaptive pre-training on four RTX 2080 Ti GPUs. Pre-training S-Encoder took around 22 hours. Pre-training R-Encoder took around 38 hours. Both are pre-trained for three epochs. Model Training Time. We trained our model on one RTX 2080 Ti GPU. The training approximately takes five minutes. Our model is trained for four epochs.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We split both the datasets into train, dev, and test sets in the ratio of 75:5:20. Table 7 contains the statistics of the train, dev, and test splits.
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
The total number of parameters of our model = 2 * number of parameters of RoBERTa BASE + parameters in the linear layers ≈ 2*125M + 2 * .5M = 251M
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
The entire Reddit dataset can be accessed through its archive on Google BigQuery at https://bigquery.  D Proof-of-Concept Details: model-based feedback for making responses empathic
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We work with three computer science students with no training in counseling and give them (seeker post, response post) pairs identified low in empathy by our approach (total empathy score ≤ 1). We show them -(1) the levels of empathy predicted by our model, (2) extracted rationales, (3) a templated feedback explaining where the response lacks and how it can be made more empathic (based on the predicted levels, extracted rationales, definitions and examples in EPITOME). A sample feedback is shown below:
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
• Seeker Post: I'm hurt so much that I don't really have feelings anymore
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
• Response Post: Yeah, I felt it once
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
• Feedback:
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
1. The response communicates an understanding of the seekers post to a weak degree in the portion I felt it once. The communication can be made stronger by talking about the seekers feelings or experiences that you interpret after reading the post. Typically, they are expressed by saying This must be terrible, I know you are in a tough situation. 2. It also lacks expressions of emotions of warmth, compassion, or concern and also does not attempt to explore the seekers emotions or feelings. Typically, they are expressed by saying I am feeling sorry for you, What makes you feel depressed?
A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support
We ask them to re-write the response post making use of the templated feedback. Overall, the participants were comfortable to re-write the responses with an average difficulty of 1.92 out of 5 (most difficult is 5) and found the feedback useful in the re-writing process with an average usefulness rating of 3.5 out of 5 (highly useful is 5).
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Inspired by the recent success of deep neural networks for natural language processing (NLP) tasks such as language modeling (Mikolov et al., 2010) and machine translation (Sutskever et al., 2014), neural response generation is currently at the forefront of research in the NLP community. Recent advances in this field have proven the efficacy of deep neural networks in modelling both task-oriented and opendomain dialogue systems (Wen et al., 2015;Sutskever et al., 2014;Vinyals and Le, 2015). Most of the existing neutral conversation models are capable of generating syntactically and contextually wellformed responses. Some of the work also focuses on enabling chatbots to generate emotionally colored and affect-rich responses (Asghar et al., 2018;. Despite the efforts in modeling affect in natural language, work that focuses specifically on modeling empathy in chatbots is relatively limited and remains an open research question (Spring et al., 2019).
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Empathy plays a vital role in human psychological processes for smooth social interaction (Decety, 2010). Empathy-related responding includes caring and sympathetic concerns for other people. Humans are born with core affect neural circuitry, and they gradually develop the ability to apprehend the emotional states of others and respond in an empathetic manner. Empathy motivates pro-social behavior and increases the sense of social bonding (Eisenberg and Eggum, 2009). Therefore, in the context of social interaction, a chatbot needs to be empathetic to maintain healthy interaction with humans and develop trust. The task of augmenting social chatbots with empathy is challenging because the generated responses have to be appropriate in terms of both content and emotion information (Spring et al., 2019).
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Several neural response generation models have attempted to address this challenge in a fully datadriven manner. For example, Rashkin et al. (2019), use the full transformer architecture (Vinyals and Le, 2015) pre-trained on 1.7 billion Reddit conversations and fine-tuned on the EmpatheticDialogues dataset (Rashkin et al., 2019) to generate empathetic responses. Lin et al. (2019) adapt the Generative Pretrained Transformer (GPT) (Radford et al., 2018) to empathetic response generation task by finetuning it on the PersonaChat  and EmpatheticDialogues datasets. Even though these models are capable of mimicking human empathetic conversation patterns in some ways, it is often unpredictable what the chatbots might generate, for example, they may generate inconsiderate remarks, redundant responses, asking the same questions repeatedly, or any combinations of them. Since it is really important to respond to humans' emotions appropriately, we believe controllability of response generation is essential.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Several other neural response generation approaches attempt to gain control over the generated response by conditioning it on a manually specified emotion label Zhou and Wang, 2018;Hu et al., 2018; or using affective loss functions based on heuristics such as minimizing or maximizing affective dissonance between prompts and responses (Asghar et al., 2018). These models claim to generate emotionally more appropriate responses than those generated from purely datadriven models. However, the primary concern of these handcrafted rules is its practicality. No prior work has shown normative associations between the speaker's emotions and the corresponding listener's emotions. As our work would reveal, listeners are much more likely to respond to sad or angry emotions with questioning than expressing similar or opposite emotions in the first turn. , however, has shown the benefit of incorporating dialogue acts as policies in designing a social chatbot. They were able to avoid the need to manually condition the next response with a label by jointly modeling dialogue act selection and response generation. Their framework first selects a dialogue act from a policy network according to the dialogue history. The generation network then generates a response based on both dialogue history and the selected dialogue act. It is thus possible to explicitly learn human-human conversational patterns in social chitchat and generate more controlled and interpretable responses. Unfortunately, they did not study empathetic response generation.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
To fill this gap, we have developed a taxonomy of empathetic listener intents by manually annotating around 500 utterances of the EmpatheticDialogues dataset (Rashkin et al., 2019), covering 32 types of emotion categories. In the following, we first describe in detail how this taxonomy was derived ( Figure 1) and how we chose the dataset to support this annotation work. To extend this subset, we employ automatic techniques to label all speaker and listener utterances, covering 25k empathetic humanhuman conversations. To be able to explain the patterns and trends of the conversation flow, we employ visualization methods to illustrate the most frequent exchanges and reveal how they temporally vary as dialogues proceed. Finally, we discuss how these results can be used to derive more informed heuristics for controlling the neural response generation process. 1
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Zillmann (2008) defines empathy as a social emotion in response to the emotions of others. Further, he states that the evoked empathetic reaction itself constitutes an emotional experience, primarily because it is associated with increased excitement and awareness. It tends to be a feeling with or feeling for the observed party. Two of the most famous theories in the psychological literature to explain the phenomenon of empathy are the "simulation theory"  and the "theory-theory" (Gopnik and Wellman, 1995). The "simulation theory" states that a person understands another or empathizes by imagining himself in the other's situation and seeing it from his perspective. The "theory-theory" states that the ability to understand what another person is feeling is based on rules for how one should think and feel.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
A recent work by Singer and Klimecki (2014) in the field of Neuroscience states that empathy refers to the general capacity of humans to resonate with others' emotional states irrespective of their valence. However, when empathizing, they suggest that one should not confuse oneself with the other; i.e., one should still know that the emotion he resonates with is the emotion of another. The failure to separate that can lead to empathetic distress. According to them, the desirable way of empathizing with others is having compassion, which is a feeling of concern for another person's suffering accompanied by the motivation to help. However, all this work does not describe in detail specific means through which humans show empathy, especially via natural language dialogues. Also, most of these studies on empathic states focus on reactions to negative rather than positive events (Buechel et al., 2018). Hence, empathy for positive events remains less understood. In our study, we explore the means through which humans empathize with others both in positive and negative scenarios. , describe an end-to-end Multi-turn Emotionally Engaging Dialog model (MEED), capable of recognizing emotions and generating emotionally appropriate and human-like responses. Their GRU based Seq2Seq dialogue model consists of a hierarchical mechanism to track the conversation history in multi-turn dialogues combined with an additional emotion RNN to process the emotional information in each history utterance. They model affect exchanges in human dialogues using a dedicated embedding layer. This emotion recognition step enables the model to produce more emotionally appropriate responses for a given context. But their approach is entirely data-driven and lacks control and interpretability over generated responses. , propose a model that can generate comments to posts in social media so that they are not only relevant in topic but also in emotion. To fully understand how emotions are expressed in conversations, they first analyse NTCIR-12 STC-1 collection (Shang et al., 2016), a social-media conversation dataset. The results show that for posts with different emotions, the distributions of comment emotions are very different from each other, and only several emotions are appropriate for responding to a given post. Inspired by the findings, they extend the basic encoder-decoder neural network architecture (Vinyals and Le, 2015) with an RNN-based response emotion estimator, which takes in a post and estimates how relevant an emotion is for responding to the post. This information is fed into the decoder when generating the response. In this, the classifier automatically determines the emotion of the response.  incorporate dialogue acts as policies in their open-domain neural response generation model by performing learning with human-human conversations tagged with a dialogue act classifier. They jointly model dialogue act selection and response generation using a GRU based neural network consisting of a policy network and a generation network. The policy network first selects a dialogue act according to the conversation history, and then the generation network generates a response based on the conversation history and the selected dialogue act. They claim that with dialogue acts, they not only achieve significant improvement over response quality for a given context but also can explain why such achievements are possible. The above work motivated us to develop explicit empathetic response intents from the dataset. We believe they can inform the development of empathetic social chatbots by providing more control to and interpretation of the responses generated and render human-machine conversations more natural and engaging.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Work has been conducted to establish dialogue act/intent taxonomies both by analysing human-human and human-machine conversation datasets. Stolcke et al. (2000) propose a taxonomy of 42 mutually exclusive dialogue acts with the intention of enabling computational dialogue act modeling for conversational speech. They follow the standard Dialog Act Markup in Several Layers (DAMSL) tag set (Core and Allen, 1997) and modify it in several ways so they can easily distinguish utterances in conversational speech. Using this taxonomy, they produce a large hand-labeled database of 1,155 conversations from the Switchboard corpus of spontaneous human-to-human telephone conversations, which is widely used to train and test dialogue act classifiers. Montenegro et al. (2019) propose a dialogue act taxonomy for a task-oriented virtual coach designed to improve the lives of the elderly. It is a multi-dimensional hierarchical taxonomy comprising of topic, intent, polarity, and entity labels at the top, in which the intent label classifies the utterance in classes related to the user's communicative intentions such as 'question', 'inform', and 'agree'. They use the taxonomy to manually annotate the user turns in 384 human-machine dialogues collected from a group of elderly. It aims to help the dialogue agent to detect goals, realities, obstacles, and ways forward of the particular topics the agent is designed to deal with.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Existing dialogue-act/intent taxonomies are either too general as they were constructed for opendomain conversations or too specific as they were constructed for specific task-oriented scenarios. These taxonomies do not necessarily model empathy in human social conversations. Also, the above approaches do not use automatic approaches to extend the manual annotations, which make their datasets comparatively smaller. In our study, we present how a smaller set of human labeled sentences can be extended using lexical methods and use it to train a classifier to automatically annotate a larger corpus.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Many open-domain conversation datasets are publicly available mainly to assist tasks such as neural dialogue generation. Out of them, some datasets are multi-modal (e.g. IEMOCAP (Busso et al., 2008), SEMAINE (McKeown et al., 2011), MELD (Poria et al., 2018)) containing visual, acoustic and textual signals. Since they contain a lot of back-channel communication through facial expressions and speech tones, the text may not fully represent the contextual expression of intent. Datasets containing dialogues extracted from social media platforms such as Twitter (e.g., the Twitter Dialog Corpus (Serban et al., 2017)) are often noisy, short, and different from real-world conversations and may contain a lot of toxic responses rather than compassionate ones. Also, datasets containing TV or movie transcripts (e.g., Emotionlines (Chen et al., 2018), OpenSubtitles (Lison et al., 2019)) and telephone recordings (e.g. Switchboard corpus (Stolcke et al., 2000)) are a translation of voice into text, which does not fully model interactions that happen only through text. Even purely text-based daily conversation datasets such as DailyDialog (Li et al., 2017) are not guaranteed to contain empathetic responses. Rashkin et al. (2019) introduced the EmpatheticDialogues dataset consisting of 24,856 open-domain, human-human conversations as a benchmark dataset to train and evaluate dialogue systems that can converse in an empathetic manner. Each conversation in this dataset is based on a situation associated with one of 32 emotions, which are selected from multiple annotation schemes, ranging from basic emotions derived from biological responses (Ekman, 1992;Plutchik, 1984) to larger sets of subtle emotions derived from contextual situations (Skerry and Saxe, 2015). The dialogues are collected using ParlAI (Miller et al., 2017), integrated with Amazon Mechanical Turk (MTurk), recruiting 810 US workers. During construction, the workers were instructed to show empathy when responding to conversations initiated by their speaker counterparts. Since almost all the dialogues in this dataset are empathetic, purely text-based, and most of which do not contain any toxic responses, we chose it to derive our taxonomy. An example conversation from this dataset is given in Table 1. Table 2 shows the basic statistics of the dataset. The average number of turns per dialogue is close to 4. The maximum number of dialogue turns in the dataset is 8. However, not many dialogues exceed 4 turns. Close to 77% of the total number of dialogues contain only up to 4 turns, and only 1.4% of the dialogues contain up to 8 turns.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Label: Afraid Situation: Speaker felt this when... "I've been hearing noises around the house at night" Conversation: Speaker: I've been hearing some strange noises around the house at night.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Listener: oh no! That's scary! What do you think it is? Speaker: I don't know, that's what's making me anxious.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Listener: I'm sorry to hear that. I wish I could help you figure it out  
A Taxonomy of Empathetic Response Intents in Human Social Conversations
To develop the taxonomy, we investigated which intents are frequently associated with listeners when responding to different emotional situations in EmpatheticDialogues. We took a subset of the dataset with situations associated with the Plutchik's 8 basic emotions (Plutchik, 1984) (joyful, anticipating, trusting, surprised, angry, afraid, sad, and disgusted), and manually analysed it to derive the listener intents associated with each type of emotions. In this process, 20 dialogues belonging to each emotion were randomly selected and each sentence in all listener utterances were manually annotated by an expert evaluator with a label that best describes their intent. This resulted in 521 sentences manually annotated with intent labels. Because an utterance can have multiple sentences, we decided to annotate each sentence in a listener's utterance with a unique intent label. For example, the two sentences comprising the utterance "Those symptoms are scary! Do you think it's Corona?" would be annotated with separate intent labels "Acknowledging" and "Questioning", respectively. After analysing their occurrences and whether some of the intents can be grouped into a common intent, we were able to come up with a taxonomy of 15 empathetic response intents. Table 3 presents this taxonomy with corresponding examples and occurrence frequencies. Words and phrases that were most helpful in annotating these examples with their corresponding intents are underlined. Manual annotation of empathetic response intents was carried out with reference to the context preceding an utterance. This way, we were able to distinguish utterances using similar words in the same order depending on their context. For example, sentences such as "I hope they find a vaccine soon." can be categorised into two different intents, "Encouraging" and "Consoling" depending on whether the sentence follows a positive or negative emotional context, respectively. 
A Taxonomy of Empathetic Response Intents in Human Social Conversations
10. Sharing or relating to own experience I had a friend who went through the same thing.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Don't take too much money with you. 2.69% 12. Expressing care or concern I hope the surgery went successfully and with no hassle.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
13. Expressing relief Phew.. That's a relief., I am glad you were okay.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
1.53%
A Taxonomy of Empathetic Response Intents in Human Social Conversations
But America is so great now! look at all the great things that are happening.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
1.15%
A Taxonomy of Empathetic Response Intents in Human Social Conversations
You are very trusting. It's nice to have a friend like you. 5 Automatic Labelling of EmpatheticDialogues Using the Taxonomy
A Taxonomy of Empathetic Response Intents in Human Social Conversations
To annotate all the speaker and listener utterances in the EmpatheticDialogues dataset with emotion labels and response intents, we trained a BERT transformer-based classifier, as suggested by Devlin et al. (2019). Prior to selecting BERT as the classifier, we trained and tested a FastText classifier on the annotation task, but its accuracy was lower compared to BERT. We proceeded with the 8 most frequent intents (questioning, acknowledging, agreeing, consoling, encouraging, sympathizing, wishing, and suggesting) in our taxonomy of empathetic listener intents and the 32 types of emotion categories given in the EmpatheticDialogues dataset. The rest of the listener intents were classified as 'neutral' since the emotion behind those intents were more on the neutral side. To expand the training data collected by manual annotation, we searched through the rest of the dataset using n-grams that are most indicative of the intent categories. For example, n-grams such as '100 %', 'absolutely', 'definitely', 'i agree', 'me neither', 'me too', and 'i completely understand' are indicative of the intent 'agreeing' and were used to collect more example utterances corresponding to that category. The most indicative n-grams used to collect more utterances for each of the intents are listed in Appendix A. During training, we initialized the representation network with weights from the pre-trained language model, RoBERTA , and fine-tuned the model on situation descriptions given in the Em-patheticDialogues dataset tagged with 32 emotions and listener utterances tagged with 8 out of 15 intents from our taxonomy of empathetic response intents. The training, validation, and test sets comprised of 25023, 3544 and 3225 sentences respectively, which spanned equally across all emotion and intent categories. We trained the model with a peak learning rate of 2e −5 and a batch size of 32 for 10 epochs and obtained the classifier giving the lowest validation loss. The top-1 accuracy of our classifier with 41 labels over the test set was 65.88%, which is significantly higher than the accuracy of FastText (Joulin et al., 2016) and DeepMoji (Felbo et al., 2017) classifiers trained on 32 emotion labels in the Empathetic-Dialogues dataset. The latter two were considered as the state-of-the-art at that time, and achieved 43% and 48% accuracy on the EmpatheticDialogues test set, respectively (Rashkin et al., 2019).
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Based on the above annotations, we analysed the most frequent response intents corresponding to different emotions expressed by speakers. In Figure 2, we visualize the emotion-intent exchanges taking place between speakers and listeners in the EmpatheticDialogues dataset. In this, each chord connects emotion-intent pairs that co-occur together. The chord leaving a particular arc represents the speaker's emotion or intent and gets connected to the arc representing the listener's emotion or intent that immediately follows. It can be seen that a significant proportion of speaker utterances contain a particular emotion in the 32 different emotion categories defined in EmpatheticDialogues, while most of the listener utterances contain a particular intent defined in our taxonomy. Instead of conveying a particular emotion, the listeners show their empathy via specific means described in our taxonomy. And the proportions of the arcs for each intent resembles the frequencies in the manually annotated subset. It serves as a validation that our taxonomy is indeed true for listener responses as it was applied to the entire dataset. Figure 2: Visualization of emotion-intent exchanges between speakers and listeners in the Empathetic-Dialogues dataset irrespective of the dialogue turn. Each chord connects co-occurring emotion-intent pairs. The chord leaving a particular arc represents the speaker's emotion or intent and gets connected to the arc representing the listener's emotion or intent that immediately follows in a conversation.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
It can be seen that 'questioning' and 'acknowledging' play a significant role in empathetic responses irrespective of the speaker's emotion-whether it is subtle or intense or has a positive or negative valence. Questioning enables the listener to sound more attentive and show interest in what the speaker describes. It prevents listeners from arriving at early conclusions, without knowing the situation in detail. It is also important to let speakers know that they have the right to feel the way they feel, even though listeners may not completely agree with their choices. Expressions of 'acknowledgment' serve this purpose. This type of emotional interaction allows the speaker to elaborate on his feelings and what he is going through and feel validated at the same time. It can also be seen that some listener intents such as 'encouraging' and 'wishing' are frequently associated with positive speaker emotions, and some intents such as 'sympathizing' and 'consoling' are frequently associated with negative speaker emotions. A list of example utterance-response pairs corresponding to some of the most frequent emotion-intent exchanges (≥ 100 times out of ≈ 50k utterance-response pairs in EmpatheticDialogues) are given in Appendix B.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Next, we analysed how emotions and response intents shift over different turns in the dialogue as the dialogues progress in time. In this analysis, we discovered the most frequent emotion-intent flows that occur between speakers and listeners from the start to the end of conversations. To visualize the shift in emotions and intents over different dialogue turns, we computed the frequency of emotion-intent flow patterns up to 4 dialogue turns and plotted the ones having a frequency ≥ 5. The reason for selecting only the first 4 turns in the dialogues is the fact that close to 77% of the dialogues in the EmpatheticDialogues dataset contain up to 4 turns and from this only 1.8% of the dialogues go up to 8 turns. Since there is comparatively much fewer data over dialogue turns from 5 to 8, we omitted these turns in our analysis. Figure 3: Visualization of the most common emotion-intent flow patterns (having a frequency ≥ 5) throughout the first four dialogue turns in the EmpatheticDialogues dataset. Figure 3 plots the most frequent emotion-intent flow patterns up to 4 turns in the dataset. Turns 1 and 3 correspond to speaker turns and turns 2 and 4 correspond to listener turns. According to the visualization, most emotions experienced by speakers are immediately followed by 'questions' as well as expressions of 'acknowledgment'. Expressions of 'sympathy' immediately follow more negatively intense emotions such as 'devastated' and 'sad'. Towards the end of dialogues, we can see more expressions of 'acknowledgment', 'agreement' and 'suggesting'. Expressions of 'encouragement' and 'wishing' can be seen in the case of positive emotional situations and 'sympathizing' and 'consoling' in the case of negative emotional situations. Another important observation is that towards the end of dialogues, listener utterances become more emotional compared to the beginning, as the speakers elaborate on their emotions. Such situations also reflect scenarios of personal distress-the phenomenon where one is unable to distinguish the emotion of their own from the emotion of another. Dialogue in Table 4 illustrates this phenomenon.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
Still, they are not as frequent as how listeners choose to empathize healthily instead of making it a distress. But this sheds light on the fact that when the speaker goes on elaborating on his situation, sometimes the listener's ability to distinguish between the speaker's emotion and the emotion of his own may decrease, leading to consequences of personal distress. In the case of intense negative emotions,  it can lead to avoidance or a deliberate change of conversation topic. This is a scenario commonly experienced by people engaged in therapeutic and health professions and is described in researches by Singer and Klimecki (2014) and Buechel et al. (2018). However, in order to verify this observation more solidly, we need a corpus with a larger number of turns per dialogue.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
The taxonomy we have developed can be incorporated into the design of social chatbots to gain more controllability and interpretability of the responses generated. It can be achieved by feeding in the conversation history, in which each utterance is tagged with an emotion or an intent label into a neural network that jointly models dialogue intent selection and response generation. Dialogue intent selection module will select the most appropriate intent based on the conversation history we feed in, and the response generation module will generate an appropriate response conditioned on the selected intent label. To help ensure more robustness, it is also possible to repeatedly sample plausible intent labels during training and feed them into the response generation module. The overall goal of modeling chatbots in this manner is to lead the conversation in a healthy and desirable direction with the controllability and interpretability provided by the taxonomy. Moreover, the taxonomy can be used as an annotation scheme to label utterances in other datasets and analyse them in terms of their empathetic quality in the same way described here. It also has implications in distinguishing between multiple forms of empathy-compassion and personal distress, as recognized in psychology and neuroscience fields.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
One limitation of this study is the analysis results are highly dependent on the EmpatheticDialogues dataset. For future studies, we intend to curate a much larger empathetic dialogue dataset using a subset of the OpenSubtitles (8 million dialogues) (Lison et al., 2019), which will help us develop a more accurate emotion classifier and establish a more general taxonomy. Another limitation is the emotion classifier trained to automatically label utterances in the EmpatheticDialogues dataset is a sentence level classifier, which is unable to accurately distinguish similar utterances whose empathetic label can differ according to the context. We intend to improve our sentence level classifier into a classifier based on dialogue history that will be able to more accurately distinguish such cases. Automated labeling of intents using lexical methods also have the possibility to injure the robustness of the model due to considering only the most indicative n-grams in individual sentences without accounting for the surrounding context.
A Taxonomy of Empathetic Response Intents in Human Social Conversations
In this paper, we introduced a taxonomy of empathetic response intents capable of supporting automatic empathetic communication in social chitchat. The strategies relying on this taxonomy are essential for a chatbot to engage in prosocial conversations, expressing empathetic concern for its users, and keeping the users engaged. Another significant contribution from our work is to provide analysis on the Em-patheticDialogues corpus after automatically annotating it based on the most frequent intents from our taxonomy and 32 types of emotion categories defined in EmpatheticDialogues. We illustrate the most frequent emotion-intent exchange patterns in the dataset and how they vary temporally over the course of interaction. These results further validate the taxonomy of empathetic listener intents we derived and shed light on the frequent empathetic conversation patterns seen among humans when engaged in social chitchat. We explained how our taxonomy can be utilized in the development of an empathetic chatbot to achieve more controllability and interpretability in the responses generation process. The method described here can also be used as an annotation scheme to label utterances from other datasets and analyse them in terms of their empathetic quality. As future work, we plan on using these findings to develop a social chatbot capable of effectively engaging in empathetic conversations.   continues on next page
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Empathy, which refers to the capacity to understand or feel what another person is experiencing (Rothschild, 2006;Read, 2019), is a critical capability to open-domain dialog systems (Zhou et al., 2018b). As shown in previous research, empathetic conversational models can improve user satisfaction and receive more positive feedback in numerous domains (Klein, 1998;Liu and Picard, 2005;Brave et al., 2005;Fitzpatrick et al., 2017;. Recently, there have also been numerous works devoted to improving the dialog models' ability to understand the feelings of interlocutors (Rashkin et al., 2019;Majumder Figure 1: Our proposed hierarchical framework: Co-MAE (right). The directed arrows denote dependencies. We also present the framework (left) of EmpTransfo (Zandie and Mahoor, 2020) for comparison. et al., 2020), which makes the dialog models more empathetic to a certain extent.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
However, empathy is a multi-dimensional construct (Davis et al., 1980) rather than merely recognizing the interlocutor's emotion  or emotional responding (Zhou et al., 2018a). It consists of two broad aspects related to cognition and affection (Omdahl, 2014;Paiva et al., 2017). The cognitive aspect requires understanding and interpreting the situation of the interlocutor (Elliott et al., 2018), which is reflected in the dialog act taken in the conversation (De Vignemont and Singer, 2006), such as questioning (e.g., What's wrong with it?), consoling (e.g., You'll get through this), etc. The affective aspect relates to properly expressing emotion in reaction to the experiences and feelings shared by the interlocutor, such as admiration (e.g., Congratulations!), sadness (e.g., I am sorry to hear that), etc. Very recently, Sharma et al. (2020) further characterizes the text-based expressed empathy based on the above two aspects as three communication mechanisms, which is a more higher-level and abstract factor that relates to empathy expression.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
In this paper, we propose a novel framework named CoMAE for empathetic response generation (Section 3), which contains the aforemen-tioned three key factors of empathy expression: Communication Mechanism (CM), dialog Act (DA) and Emotion (EM). Specifically, when model these empathy factors simultaneously, we adopt a hierarchical way instead of following previous works that treat multiple factors independently, such like EmpTransfo (Zandie and Mahoor, 2020) that considers both DA and EM (see Figure 1 for comparison). Such approaches hold the hypothesis that different factors are independent of each other, which is intuitively unreasonable. In fact, our empirical analysis (Section 4) on a Reddit corpus (Zhong et al., 2020) shows that there are obvious hierarchical relationships between different factors, which confirms the soundness and necessity of hierarchical modeling.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
We then devise a CoMAE-based model on top of the pre-trained language model GPT-2 (Radford et al., 2019) (Section 5), and compare the model performance with different combinations of empathy factors and hierarchical modeling. Automatic evaluation (Section 6.3) shows that combining all the three factors hierarchically can achieve the best model performance. Manual evaluation (Section 6.4) demonstrates that our model can generate more empathetic responses than previous methods. Extensive experiments (Section 6.5) further highlight the importance of hierarchical modeling in terms of the selection and realization of empathy factors.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
The contributions of this paper can be summarized in three folds:
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
• Based on the nature of multi-dimensionality of empathy expression, we propose a novel framework, CoMAE, for empathetic response generation. It hierarchically models three key factors of empathy expression: communication mechanism, dialog act and emotion.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
• On top of GPT-2, we devise a CoMAEbased model. Experimental results show that our model can generate more empathetic responses than previous methods.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
• We empirically analyze the necessity of hierarchical modeling, and highlight its importance especially in terms of the selection and realization of different empathy factors.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
2 Related Work
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Empathy is a complex multi-dimensional construct (Davis et al., 1980) which consists of two broad aspects related to cognition and affection (Omdahl, 2014;Paiva et al., 2017). As shown in Section 1, the two aspects are reflected in the dialog act (DA) taken and the emotion (EM) expressed in the conversation respectively. Based on the theoretical definition of empathy, Sharma et al. (2020) characterize the text-based expressed empathy as 3 communication mechanisms (CM): emotional reaction (ER) (e.g., I feel really sad for you), interpretation (IP) (e.g., This must be terrifying, I also have similar situations), and exploration (EX) (e.g., Are you still feeling alone now?). 1 These communication mechanisms are also applied in the recently proposed task of empathetic rewriting (Sharma et al., 2021).
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Besides, Zhong et al. (2020) propose that persona, which refers to the social face an individual presents to the world (Jung, 2016), has been shown to be highly correlated with personality (Leary and Allen, 2011), which in turn influences empathy expression (Richendoller and Weaver III, 1994;Costa et al., 2014). While Zhong et al. (2020) do not explain the explicit connection between persona and empathy expression, they suggest that different speakers may have different "styles" for expressing empathy.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
In the past years, empathetic response generation has attracted much research interest (Rashkin et al., 2019;Majumder et al., 2020;Zandie and Mahoor, 2020;Sun et al., 2021). Rashkin et al. (2019) suggest that dialog models can generate more empathetic responses by recognizing the interlocutor's emotion.  propose to design a dedicated decoder to respond each emotion of the interlocutor, which makes the generation process more interpretable. Majumder et al. (2020) adopt the idea of emotional mimicry (Hess and Fischer, 2014) to make the generated responses more empathetic. Inspired by the advances in generative pre-trained language models (Radford et al., 2018(Radford et al., , 2019, EmpTransfo (Zandie and Mahoor, 2020) uses GPT (Radford et al., 2018) to generate empathetic responses.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Unlike previous works that only consider the EM factor in empathy modeling, EmpTransfo takes both DA and EM into account. The fundamental difference of EmpTransfo from our work lies in two points: (1) our work further considers communication mechanism in modeling empathy, and (2) we analyze and explore in depth the importance of hierarchically modeling of these empathy factors.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Our proposed CoMAE framework is shown in Figure 1. CoMAE uses CM as a high-level factor that provides a coarse-grained guidance for empathy expression, and then takes DA and EM to achieve the fine-grained realization. Formally, given the context x, CoMAE divides the generation of the empathetic response y into four steps: (1) predict CM C y conditioned on the context, (2) predict DA A y conditioned on both the context and CoM, (3) predict EM E y based on all the conditions, and (4) generate the final response y. The whole process is formulated as Equation 1:
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Note that EM is conditioned on DA, because we intuitively think the expressed emotion is the effect rather than the cause of taking some dialog act. In the other words, one may not adopt the dialog act just for the purpose of expressing some emotion. Hence, realizing the emotion expression as expected is also important in our task, which is the motivation of that we analyze the realization of different factors in Section 6.5.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
It is also worth noting that while CoMAE only contains the three factors, such hierarchical framework can be naturally extended to more factors that relate to empathy expression. For instance, Zhong et al. (2020) suggest that persona plays an important role in empathetic conversations. Due to that persona may contain the information about the speaker's style of adopting DA or expressing EM, when integrating persona into empathetic response generation, being conditioned on DA and EM may lead to better performance.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
While no empathetic conversation corpora provide annotations of diverse empathy factors, there are abundant publicly available resources that make automatic annotation feasible. In this section, we first introduce our used corpus and the resources and tools used in automatic annotation, then we show our empirical analysis to verify the hierarchical relationships between different empathy factors. Zhong et al. (2020) propose a large-scale empathetic conversation corpus 2 crawled from Reddit. It has two different domains: Happy and Offmychest. The posts in the Happy domain mainly have positive sentiments, while those in the Offmychest domain are usually negative. We adopted their corpus for study for two major reasons: (1) the corpus is real-life, scalable and naturalistic rather than acted (Rashkin et al., 2019), and (2) the manual annotation in (Zhong et al., 2020) shows that most of the last responses are empathetic (73% and 61% for Happy and Offmychest respectively).
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Communication Mechanism (CM) 3 Sharma et al. (2020) provide two corpora annotated with CM: TalkLife (talklife.co) and Reddit (reddit.com), while only the latter is publicly accessible and we thus used the Reddit part. Note that in their original paper, each mechanism is differentiated as three classes of "no", "weak", or "strong". Due to the unbalanced distribution of three classes, we merged "weak" and "strong" into "yes". Finally, we differentiated each mechanism as two classes: "no" or "yes". Dialog Act (DA) 4 Welivita and Pu (2020) propose a taxonomy of DA (referred as "intent" in the original paper) for empathetic conversations. They first annotate 15 initial types of DA on the ED corpus (Rashkin et al., 2019), and finally obtain 8 high-frequency types of DA with other types merged as others (8+others), which are shown in Figure 2. Emotion (EM) 5 We considered the taxonomy proposed in (Demszky et al., 2020), which contains 27 emotions and a neutral one, because: (1) it has a wide coverage of emotion categories with clear definitions, and (2) the annotated corpus is large-scale and also crawled from Reddit. However, we noted that the original emotion distribution is unbalanced and the too fine-grained taxonomy may lead to the sparsity of partial emotions. Considering the task scenario of empathetic conversation, we adopted the clustering results in (Demszky et al., 2020) and modified the original taxonomy as 9 emotions and a neutral one (9+neutral), which are also shown in Figure 2. We show the mapping between our adopted emotions and the original emotions in Appendix A.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
We fine-tuned the RoBERTa 6 (Liu et al., 2019) classifiers for CM, DA and EM, whose performance is summarized in Table 1. They all achieve reasonable performance, ensuring the quality of automatic annotation. However, we noted that the source domain (Rashkin et al., 2019) of the DA classifier is different from the target domain (Reddit). To verify the quality of DA annotation, we recruited three workers from Amazon Mechanical Turk to judge whether the utterance is consistent with the annotated DA. From the utterances that are not annotated with "others", we randomly sampled 25 utterances for each DA (totally 200) to avoid the impact of unbalanced distribution. Finally, the ratio of being judged as consistent is 0.78 with Fleiss' Kappa κ = 0.621 (Fleiss, 1971), which indicates substantial agreement (0.6 < κ < 0.8) and that the automatic annotation of DA is also reliable.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Following the original data split of (Zhong et al., 2020), we first filtered those conversations where there are more than two speakers (about 15%) to ensure that the last utterance is related to the post. We used the aforementioned classifiers to automatically annotate each utterance with DA and EM, and annotate each final response additionally with CM. We found that the last responses that are not annotated with any CM are more likely to 6 https://huggingface.co/roberta-base 
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
In order to verify the hierarchical relationships between the three factors, we counted the distribution frequency of each (X, Y ) 7 pair, where (X, Y ) is one of the three factor pairs: (CM, DA), (CM, EM), (DA, EM). We approximated the statistical frequency of (X, Y ) as their joint probability distribution P(X, Y ). We then normalized P(X, Y ) along the X dimension to obtain the conditional distribution of Y given X: P(Y |X). Figure 2 shows the heat maps of the conditional distributions of the three factor pairs. The heat maps reveal obvious patterns of the occurrence of Y given X. For instance, when one adopts the DA encouraging, he usually expresses the EM caring instead of approval or joy. If one expresses empathy with the CM exploration (EX), he almost always adopts the DA questioning and expresses the EM surprise. Hence, considering the hierarchical relationships between different empathy factors is reasonable and natural, and is also necessary for better empathy modeling. Our devised CoMAE-based model uses GPT-2 as the backbone (Radford et al., 2019). The overall architecture is shown in Figure 3. Firstly, our model takes the dialog context x as input. The context x is the concatenation of history utterances: x = (u 1 , u 2 , . . . , u N ) , where N is the length of dialog history. Any two adjacent utterances are also separated by the special token [EOS]. Each history utterance u i contains a sequence of tokens:
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
where l i is the length of u i . Each utterance u i is labeled with the corresponding speaker k u i ∈ {0, 1} (only 2 speakers). We denote the annotated DA and EM of each utterance u i as A u i ∈ [0, 9) and E u i ∈ [0, 10) respectively. Suppose that the token id and the position id of u i,j are denoted as w u i,j ∈ [0, |V|) (V is the vocabulary) and p u i,j ∈ [0, 1024) (the maximum input length is 1024) respectively, the representation of each token u i,j is the summation of the following embeddings:
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
, M E ∈ R 10×d denote the embedding matrices of word, position, speaker, DA and EM respectively, and [·] denotes the indexing operation. We denote the output hidden states after feeding x into the model as H x ∈ R lx×d , where l x is the total length of context x.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Next, we use the hidden state at the last position of the context, h x = H x [−1] ∈ R d , to hierarchically predict the CM, DA and EM of the target response. We first separately predict 8 8 In the mathematical notation used in this paper, we dis-C (i) y ∈ {0, 1} for each i ∈ {ER, IP, EX}, which indicates whether to adopt the CM i:
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
where each F (i)
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
C is a non-linear layer activated with tanh, and each M (i) C ∈ R 2×d denotes the embedding matrix of the CM i ∈ {ER, IP, EX}. Based on the context x and the predicted CMs C y , we next predict DA:
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
where [·; ·] denotes vector concatenation and F A is a non-linear layer. Note that we share the parameters of DA embeddings with the classification head (Equation 6), which is consistent with the way in GPT-2 (Radford et al., 2019) where the parameters of word embeddings are shared with the LM head (Equation 10). EM is predicted similarly but conditioned additionally on the predicted DA A y :
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
where F E is also a non-linear layer. Finally, we add all the factors to obtain the fused embedding e CoMAE that controls the empathy expression of the response:
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
The embedding of each input token y t in the response is as follows:
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Suppose that the output hidden state corresponding to y t is s t , then we predict the next token y t+1 tinguish the ground truth value and the predicted value of a variable X with the symbols X * and X respectively. through the LM head: y t+1 ∼ P y t+1 y ≤t ; x, C y , A y , E y (10)
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
where the parameters of the LM head are shared with the word embedding matrix M W .
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
The optimization object contains two parts. One part is the negative log likelihood loss L NLL of the target response:
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
where l y is the length of the golden response. The other part is the prediction losses of CM L C , DA L A , and EM L E :
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
The complete optimization object is the summation of the above losses: L = L NLL + λ (L C + L A + L E ), where λ is the weight of the prediction losses. We set λ to 1.0 in our experiments.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
It is worth noting that the supervision signals of predictions (from Equation 11 to 13) combined with hierarchical modeling (from Equation 3  Hence, consider two models where one uses hierarchical modeling and the other does not (predicting each factor separately). When the two models are fed with the same empathy factors, saying the triplet (C y , A y , E y ) is designated validly, we can expect that the former model has better performance than the latter one. This conjecture will be verified in the automatic evaluation (Section 6.3).
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
We investigated the model performance with different combinations of empathy factors and hierarchical modeling:
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
(1) Vanilla: the GPT-2 model directly fine-tuned on the corpus without adding any empathy factor;
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
(2) +CM, +DA, +EM: the GPT-2 models equipped with one of the three factors;
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
(3) CM || DA, CM || EM, DA || EM, CM || DA || EM: the models equipped with two or all of the three factors, but predicting each factor separately without hierarchical modeling; (4) CM → DA, CM → EM, DA → EM, CM → DA → EM: the models that are similar to (3) but utilize the hierarchical relationships, where → denotes dependency.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Note that the baseline DA || EM is consistent with EmpTransfo 9 (Zandie and Mahoor, 2020), and CM → DA → EM is exactly our devised model described in Section 5.1.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
All the models were implemented with PyTorch 10 (Paszke et al., 2019) and the Transformers library 11 (Wolf et al., 2020). We used the pre-trained GPT-2 with the size of 117M parameters (768 hidden sizes, 12 heads, 12 layers) for all the models. The responses were decoded by Top-p sampling with p = 0.9 and the temperature τ = 0.7 (Holtzman et al., 2019). We trained all the models with Adam (Kingma and Ba, 2014) optimizer with β 1 = 0.9 and β 2 = 0.999. The learning rate was 10 −4 and was dynamically changed using the linear warmup (Popel and Bojar, 2018) with 4000 warmup steps. All the models were fine-tuned for 5 epochs with the batch size 16 on one NVIDIA RTX 2080Ti GPU. We selected the checkpoint for each model where the model obtains the lowest perplexity score on the Valid set.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
The automatic evaluation uses the golden responses as reference to evaluate the responses generated by  models. However, when the responses are generated based on the predicted CM / DA / EM, it is not appropriate to compare the generated responses with the reference ones (Liu et al., 2016). Thus, in automatic evaluation we only considered the setting where the models are fed with the ground truth empathy factors. The results where the generated responses are based on the predicted factors will be analyzed in the later experiments. The automatic metrics we adopted include perplexity (PPL), BLEU-2 (B-2) (Papineni et al., 2002), ROUGE-L (R-L) (Lin, 2004), and the BOW Embedding-based (Liu et al., 2016) Greedy matching score. The metrics except PPL were calculated with an NLG evaluation toolkit 12 (Sharma et al., 2017), where the generated responses were tokenized with NLTK 13 (Loper and Bird, 2002).
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Results are shown in Table 2. We analyze the results from the following three perspectives: General Performance Our model achieves the best performance on all the metrics on both do-12 https://github.com/Maluuba/nlg-eval 13 https://www.nltk.org/ mains, and most of the advantages over the competitors are statistically significant. Impact of Empathy Factors The model performance vary from different combinations of empathy factors. First, considering more empathy factors always leads to better performance (e.g., CM → DA → EM > CM → EM > +EM > Vanilla). Second, EM brings the most gains to the model performance among the three factors. It may be because emotion is the most explicit factor that influences empathy expression (Sharma et al., 2020). In contrast, CM brings fewer gains than DA and EM. The reason may be that CM provides a highlevel but coarse-grained guidance for empathetic response generation, lacking a fine-grained control like DA or EM. While the responses in the corpus of (Zhong et al., 2020) are not too long (≤ 30 words), we believe that CM plays an important role in generating longer empathetic responses, which may require the planning of multiple methanisms and more diverse usage of DA and EM. Impact of Hierarchical modeling We noticed that for almost all the models that adopt multiple empathy factors, hierarchical modeling always leads to better performanc (e.g., CM → DA → EM > CM || DA || EM, DA → EM > DA || EM). This phenomenon is not trivial because the models with or without hierarchical modeling are all fed with the same empathy factors as the reference responses. It confirms our conjecture in Section 5.2 that hierarchical modeling can establish the connections between the embeddings of different factors, thus leading to a better capacity of empathy modeling. However, (CM, EM) is an exception. It may be due to that the pair (CM, EM) has a weaker correlation (the lowest manual information, Section 4.5) than other pairs.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
In manual evaluation, the models generate responses based on the empathy factors sampled from the predicted probability distributions. When sampling DA or EM, we used the Top-p filtering with p = 0.9 (Holtzman et al., 2019) to ensure the validness of the sampled results.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
The manual evaluation is based on pair-wise comparison, and the metrics for manual evaluation include: Fluency (which response has better fluency and readability), Coherence (which response has better coherence and higher relevance to the context), and Empathy (which response shows bet-   Table 4: Results of the Hits@1/3 of predicting Y given that X is predicted rightly. "Prop." denotes the proportion of the cases where both models X || Y and X → Y predict X rightly. Scores that are significantly improved after using hierarchical modeling are marked with * (sign test, p-value < 0.001).
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
ter understanding of the partner's experiences and feelings, and which response expresses empathy in the way that the annotators prefer). The pair-wise comparison is conducted between three pairs of models:  Table 3. From all the three pairs, we find that the responses generated by these GPT-2-based models have similar fluency. The results of (1) indicate that further considering CM can significantly improve the empathy of generated responses, while the coherence may slightly decrease.  It may be because that the communication mechanisms like interpretation sometimes lead to the responses that are less relevant to the contexts (especially those sharing experiences). The results of (2) and (3) indicate that hierarchical modeling improves the coherence of generated responses. The more empathy factors are modeled, the larger improvement can be obtained.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
To give further insights of the superiority of hierarchical modeling, we analyzed (1) the prediction and (2) the realization of empathy factors. Prediction For each pair (X, Y ) in (CM, DA), (CM, EM), (DA, EM), we paired the models X || Y and X → Y for comparison. Our purpose is to observe whether the prediction of X improves that of Y after using hierarchical modeling. Note that when taking the ground truth as reference, it is not appropriate to directly judge the prediction accuracy by comparing Y and Y * if X = X * . We thus computed the conditional probability that Y is predicted rightly given that X is predicted rightly:
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Results are shown in Table 4. While the accuracy of predicting X of X || Y and X → Y is close, the prediction of Y is significantly enhanced by hierarchical modeling. The results demonstrate that hierarchical modeling enables the model to select more proper empathy factors. Realization Recall that in manual evaluation, the models generate a response based on the sampled empathy factors C y , A y , E y . To verify whether these factors are well realized, we used the classifiers in Section 4.3 to identify the empathy factors displayed in the generated responses. Suppose that the identification results are Z y , ∀Z ∈ {C, A, E}, we computed the ratio of Z y = Z y as the realization score of Z.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Results are shown in Table 5. The realization of all the factors is significantly improved by hierarchical modeling. It is intuitive because hierarchical modeling can avoid the cases where the sampled factors are inappropriate or even conflicting, thus reducing the noise of empathy factors in response generation.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
We show the generated responses with different empathy factors in Figure 4. The adoption of the CM emotional reaction causes our model to express the same EM admiration (i'm proud of you!) as DA → EM (good for you, man!), while the two models generate the same sentence (keep it up!) when taking the DA encouraging. However, the further adoption of the CM interpretation causes our model to further share its own experiences and feelings (i have been sober for about 10 years, and it's the best feeling ever). As a result, with the enhancement of multiple empathy factors, the response generated by our model is more engaging and empathetic while maintaining the coherence.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
Besides, we noticed another phenomenon occurring when all the three CMs are adopted. In this case, the three CMs are usually represented separately in different sentences (e.g., I am so happy for you! I also had tried to be sober but failed. How did you make it?), which is consistent with the results of empathetic rewriting (Sharma et al., 2021). Recall that we add the same CoMAE embeddings for all the tokens in the response during generation (Section 5.1). Such uniform operation seems non-optimal for the nonuniform realization of different CMs, especially when generating a longer empathetic response that contains multiple sentences with different CMs, DAs or EMs. We believe there is still much room of improvement when applying our CoMAE framework to longer response generation, like combining CoMAE's multifactor hierarchical modeling with planning-based Post you might remember me posting here when i had less than a month sober a little while back. well, yesterday i hit 100 days without alcohol and celebrated by solo hiking my state's tallest mountain! Golden ok that is an awesome pic! love it and the story thank you! 
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
In this paper, we present a multi-factor hierarchical framework CoMAE for empathetic response generation. It contains three key factors of empathy expression: communication mechanism, dialog act and emotion, and models these factors in a hierarchical way. With our devised CoMAE-based model, we empirically demonstrate the effectiveness of these empathy factors, as well as the necessity and importance of hierarchical modeling.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
As future work, the CoMAE framework can be naturally extended to more factors that relate to empathy expression, such as persona (Zhong et al., 2020), by exploring the hierarchical relationships between different factors.
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
In the original paper of (Demszky et al., 2020) 14 , the authors provide the hierarchical clustering results of the 27 emotions (Figure 2 in their paper), which reflect the nested structure of their proposed emotion taxonomy. Based on the clustering results, we merged the emotions that are highly correlated with each other, and the mapping between our adopted emotions and the original emotions is shown in Table 6.  Table 6: Mapping between our adopted emotions and the original emotions in (Demszky et al., 2020).
CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation
We computed the proportions of the last responses annotated with ER / IP / EX. In the Happy domain, the proportions are 76.0% / 10.2% / 18.7%, while in the Offmychest domain are 57.1% / 21.4% / 27.9% respectively. The statistics of DA and EM are shown in Figure 5. We can find several differences between two domains. In terms of communication mechanism, the responses in the Offmychest domain prefer interpretation and exploration, while emotional reaction occupies a larger proportion in the Happy domain. In terms of DA, the actions that provide support (such as agreeing, consoling, suggesting, and sympathizing) are more frequently adopted in the Offmychest domain. It is similar when it comes to emotion, where the emotions such as approval and caring are displayed more commonly when responding to the posts with negative sentiments. We also observed that the responses in the Offmychest domain may also display the emotions like anger and sadness, indicating that they do understand 14 https://arxiv.org/abs/2005.00547v2 the experiences and feelings of the conversation partners.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Empathetic response generation, as a conditional text generation task, aims to endow agents with the ability to understand interlocutors and accurately express empathy in their communication (Rashkin et al., 2019;Lin et al., 2019;Li et al., 2020;Shen et al., 2021). However, the generated responses tend to be generic and monotonous (Chen et al., 2022), i.e., showing shallow empathy and few connections to the context. As shown in the upper part of Figure 1, "I'm sorry to hear that." is used as a reaction to different contexts with negative feelings. To alleviate the problem, existing works mainly incorporate emotion or knowledge modules into the encoder-decoder framework and train their models I once lost my job and got mad. Context 1 I did not get a single gift during Christmas. Context 2 angry I'm sorry to hear that. sad nervous I was really nervous when starting a new job. Context CM
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Encouraging IT P ro n o u n S F Figure 1: A monotonous empathetic response (upper) and an informative empathetic response (lower). "CM", "IT", and "SF" are abbreviations for "Communication Mechanism", "Intent", and "Semantic Frame", which represent control signals at the utterance, sentence, and token level respectively. with the maximum likelihood estimation (MLE) (Rashkin et al., 2019;Lin et al., 2019;Majumder et al., 2020;Li et al., 2020;Li et al., 2022a).
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Recently, diffusion models (Ho et al., 2020;Dhariwal and Nichol, 2021) have emerged as a brand-new and promising paradigm for generative models. A few prior works that explored using diffusion models on text data are mainly designed for unconditional text generation (Austin et al., 2021;Hoogeboom et al., 2021;He et al., 2022). For text generation with extra conditions (control signals or contexts), Diffusion-LM (Li et al., 2022b) applies extra-trained classifiers to make the generated text satisfy input signals like sentiment and syntactic structure. DiffuSeq (Gong et al., 2022) is proposed as a classifier-free diffusion model that uses "partial noising" in the forward process to distinguish the input and output text.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
In this paper, we add control signals to empathetic response generation and propose a diffusion model-based framework, DIFFUSEMP, to solve the aforementioned monotonous empathy problem. First, since empathy is a multi-dimensional factor (Davis et al., 1980), i.e., several factors affect the realization of empathy, we use explicit control sig-nals at different levels to guide response generation. At the utterance level, communication mechanism (CM) (Sharma et al., 2020) divides text-based empathy into emotional reaction, interpretation, and exploration to describe the high-level functionality. Then, we use intent (IT) (Welivita and Pu, 2020) to reflect the behaviors of an agent in each sentence † , such as questioning (e.g., What happened to you?). Finally, the fine-grained signal semantic frame (SF) (Baker et al., 1998) is imposed on each token, which represents their universal categories of events, concepts, and relationships. An example of how multi-grained control signals work is illustrated in the lower part of Figure 1. To have exact guidance over responses, these signals are extracted from golden responses in the training process, while during inference, an emotion-enhanced matching method is used to obtain response candidates as the source of control signals.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
We then design a diffusion model to make the generated responses not only relevant to dialogue contexts but also express specific empathy under the multi-grained control. The dialogue context, multi-grained control, and response are considered as the model input. For the forward diffusion process, we apply the partial noising (Gong et al., 2022) strategy so that both the context and control signals are unchanged, and only the response is noised. To fulfill the reverse diffusion process, we use the transformer architecture (Vaswani et al., 2017) and introduce a masking strategy to indicate the control range of each signal on response tokens. Specifically, each CM/IT controls all tokens in an utterance/sentence, while an SF term corresponds to exactly one token. Tokens out of the control range are masked in the self-attention layer. Finally, we conduct experiments on a benchmark dataset EMPATHETICDIALOGUE to demonstrate the effectiveness of DIFFUSEMP.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
The main contribution of this paper is threefold: (1) We introduce explicit multi-grained control signals to solve the monotonous empathy problem, and convert the empathetic response generation into a controllable setting. (2) We propose DIF-FUSEMP, a novel diffusion model-based framework, to unify the utilization of dialogue context and control signals, achieve elaborate control with a specific masking strategy, and integrate an emotionenhanced matching method to produce diverse re- † An utterance, the response here, may consist of more than one sentence. sponses for a given context. (3) Experimental results show that our method outperforms competitive baselines in generating informative and empathetic responses.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
2 Related Work 2.1 Empathetic Response Generation Rashkin et al. (2019) firstly formulate the empathetic response generation task and construct the EMPATHETICDIALOGUE dataset. Existing works that focus on this task can be divided into two lines. The first is to detect and utilize the user's emotion with diverse structures (Lin et al., 2019;Majumder et al., 2020;Shen et al., 2021). The second is to consider cognition-based factors other than emotions (EM), such as dialogue act (DA) (Welivita and Pu, 2020), communication mechanism (CM) (Sharma et al., 2020), emotion cause (Jiang et al., 2019), psychological skill (Kim et al., 2021, and commonsense Li et al., 2022a).  propose a framework CoMAE to model the relationship among CM, DA, and EM at the utterance level. The differences between Co-MAE and DIFFUSEMP are: (1) Instead of predicting each factor based on the context representation, DIFFUSEMP explicitly uses control signals that are highly related to a response as task input. (2) We achieve the elaborate control with multi-grained signals, i.e., tokens in response are influenced by different signals, while CoMAE applies the same combined factor to all decoding positions.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Diffusion models are a class of generative models with promising performance and have been used in a variety of real-world applications. Most existing works of diffusion models focus on continuous data, such as vision (Nichol et al., 2021;Radford et al., 2021;Rombach et al., 2021b) and audio (Popov et al., 2021;Tae et al., 2021). Due to the discrete nature of text data, the utilization of diffusion models for NLP is challenging. Hoogeboom et al. (2021) and Austin et al. (2021) extend diffusion models to discrete state spaces for character-level text generation. Diffusion-LM (Li et al., 2022b) 
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Reverse Reverse Figure 2: The overview of DIFFUSEMP. The left part describes the training and inference stages, the middle part shows the forward process and reverse process in the diffusion model, and the right part illustrates details in a Transformer (Vaswani et al., 2017) block with control-range masking for the reverse process.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
the forward process. DiffusionBERT (He et al., 2022) combines pretrained language models with absorbing-state discrete diffusion models for text.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
To the best of our knowledge, we are the first to achieve controllable empathetic response generation using a diffusion model.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
In this paper, we perform empathetic response generation in a controllable setting. The dialogue context is an alternating sequence of utterances from a speaker and a listener, i.e. w u = {u 1 , u 2 , . . . , u n }.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Here, we aim to generate an empathetic and context-related response w y = {y 1 , y 2 , . . . , y n } conditioned on the given context w u and a set of control signals w c obtained in advance (Section 3.1). Then, the context, control signals, and response are concatenated and fed into a diffusion model with control-range masking (Section 3.2). In the training process, golden responses are used to extract control signals, while during inference, we integrate an emotion-enhanced matching method to get proper response candidates (Section 3.3). The framework of DIFFUSEMP is illustrated in Figure 2.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
To better model and express multi-dimensional empathy, we use control signals at different levels. However, the benchmark dataset EMPATHETICDI-ALOGUE does not contain such annotations. Here, we introduce three types of signals used in this paper and the way to collect them for each golden response or response candidate using pre-trained tagging models. The definition and components of empathy in psychology are complex (Davis et al., 1980;de Waal, 2008;Decety and Meyer, 2008), and we choose the control signals that intersect with computational linguistics. Note that the design of DIFFUSEMP is not limited to the following control signals, other factors of empathy can also be used. Communication Mechanism (CM). We employ the taxonomy in Sharma et al. (2020): Emotional Reaction (ER), Interpretation (IP), and Exploration (EX). ER expresses emotions such as warmth, compassion, and concern, IP represents an understanding of feelings and experiences inferred from the speaker, and EX stands for exploring the feelings and experiences not stated in previous utterances. Following Sharma et al. (2020), we use three RoBERTa-based (Liu et al., 2019) classifiers to individually identify whether a response implies a certain mechanism. Intent (IT). A previous analysis (Welivita and Pu, 2020) argues that humans demonstrate a wide range of intents when regulating empathy and proposes a dataset EMPATHETICINTENT. Besides, many works (Xie et al., 2022; insist that intents and emotions have a strong relationship. Specifically, listeners are much more likely to respond to positive or negative emotions with specific empathetic intents such as acknowledgment, consolation, and encouragement, rather than only expressing similar or opposite emotions. We train a BERT-based (Devlin et al., Table 1: The performance of tagging tools used to get control signals. Since SF is from a frame semantic parsing task, we only report the F1 score following the original task setting.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
of a token represents its universal categories of events, concepts, and relationships, and can be regarded as a high-level abstraction of meaning. For example, tokens like bird, cat, dog, horse, sheep share the same frame label Animals. Here, we utilize the open-SESAME model (Swayamdipta et al., 2017) to extract semantic frames from responses. The performance of tagging tools is listed in Table 1. Note that control signal tokens are concatenated into a flat sequence from coarse to fine.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
A diffusion model contains a forward process and a reverse process. We first concatenate a context with the control signals and corresponding response, i.e., w = w u ⊕ w c ⊕ w y . Then we use an embedding function (Li et al., 2022b) EMB(·) to map the discrete text w into a continuous representation x 0 = u 0 ⊕ c 0 ⊕ y 0 , where u 0 , c 0 , and y 0 represent parts of x 0 that belong to w u , w c , and w y , respectively. Forward Process. In forward process q, the model adds noise to the original sample x 0 step by step:
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
where x 1 , ..., x T make up a chain of Markov variants and x T ∼ N (0, I). β t ∈ (0, 1) is a noise schedule that controls the noise scale added in each step. Note that the conventional diffusion models corrupt the entire x 0 . However, empathetic response generation is a conditional text generation (Seq2Seq) task and we only concern with the generative effect on response. Therefore, we use partial noising (Gong et al., 2022) to only impose noise on the parts of x t that belong to w y , i.e., y t . Reverse process. Once the forward process is completed, the reverse process aims to gradually recover x 0 by denoising x T according to: Sounds great What is your gift ? Figure 3: An example of control signals and controlrange masking. The upper left part shows a response with labeled signals, the lower left part illustrates the control range of each signal on response tokens, and the right part is the corresponding mask matrix. "-" means the SF signal is empty.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
where µ θ (·) and σ θ (·) are predicted mean and standard variation of q(x t−1 |x t ) (derived using Bayes' rule) in forward process and can be implemented by a Transformer (Vaswani et al., 2017) model f θ . In the reverse process, we add a rounding step
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Control-Range Masking. The non-autoregressive nature of conventional diffusion models make one input token can attend to all other tokens with the full self-attention mechanism to update its representation. Instead, we need to distinguish between tokens of control signals and responses, and further model the relationship between them with a mask matrix M and integrate it into the self-attention layer in Transformer:
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
concerns by questioning after showing sympathy for a negative situation or feeling. Therefore, to model the control relationship among tokens, we design the control-range masking and utilize it in the self-attention layer of f θ . Specifically, for a mask matrix, the value on position (i, j) is 0 if token j is controlled by token i ; otherwise is negative infinity: Figure 3 gives an example of control-range masking. For the intent signal Acknowledging (index 2), it is visible to Questioning (line 3) and corresponding response tokens Sounds great! in the first sentence (line 12-14). Meanwhile, since the response token great (line 13) is controlled by Exploration (index 1), Acknowledge (index 2), Desirability (index 5), and the rest of response tokens (index 12-19), it attends to them in the mask matrix.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
With the existence of control-range masking, we can elaborately guide the generation of each response token with signals from different levels that reflect diverse factors for empathy expression.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Training. In the training process, we label control signals based on golden responses as described in 3.1. To train model f θ in the reverse process, we minimize the variational lower bound following Gong et al. (2022):
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
wheref θ (x t , t) denotes the fractions of recovered x 0 corresponding to y 0 , and R(·) is a mathematically equivalent regularization term to regularize the embedding learning. Inference. During inference, since golden responses are unavailable, we design an emotionenhanced matching method to obtain response candidates and use them to extract control signals. We treat dialogue contexts in the training set as the candidate pool and use each context in the test set as a query to perform context-context matching. Then the response corresponding to a returned context with the highest similarity is used as the candidate.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Regarding the importance of emotions in empathetic response generation, we consider two aspects to score each candidate, semantic similarity and emotional consistency, in context-context matching. Specifically, we first train a BERT model (Devlin et al., 2019) on the training set to classify emotions for contexts. Then, we use this model to get emotional distribution for contexts in both the candidate pool and queries. Finally, we compute the cosine similarity of both sentence embeddings and predicted emotional distributions for each querycontext pair. The contexts are re-ranked according to a weighted sum of two similarity scores:
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
where γ is a hyperparameter to balance the semantic and emotional similarity. There are 32 evenly-distributed emotion labels in the dataset. We apply the data provided by the original paper with the split ratio of 8:1:1 for training/validation/test set and use the script released by Lin et al. (2019) to preprocess the data.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
We compare our method with three groups of representative methods. Transformer-Based Methods. Relevance  Table 2: Automatic evaluation results. The best results of standard settings are reported in the bold format. "ACC", "D", and "sBL" are abbreviations of Accuracy, Dist, and Self-BLEU, respectively. "ACC-CM" is the average Accuracy of ER, IP, and EX, which are three mechanisms of CM. Two more results are provided as references. Under the Oracle setting, control signals are obtained from golden responses in the test set, which can be regarded as the upper bound of DIFFUSEMP. Golden responses themselves are also evaluated, which reflects human performance on the task. More details are listed in Appendix A.1.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Automatic Evaluation. We evaluate the generated responses from four aspects: (1) Relevance: BERTScore (Zhang et al., 2020a) computes a semantic similarity between generated responses and golden references. MIScore is the likelihood of generating a context with the given response, which applies the idea of Maximum Mutual Information (MMI) (Li et al., 2016;Zhang et al., 2018) and indicates whether the generated response is contextrelated.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
(2) Controllability: We calculate the success rate of empathy expression with multi-grained control signals to validate the controllability of DIF-FUSEMP. For utterance-level CM and sentencelevel IT, we report Accuracy, while for token-level SF, we report F1. (3) Informativeness: Dist-n (Li et al., 2016) calculates the number of distinct ngrams in generated responses. Self-BLEU (Zhu et al., 2018) reflects the difference of all generated responses to a large extent. We calculate the average BLEU-5 overlap between each two generated responses. (4) Response Length: AvgLen represents the average number of tokens for generated responses. Intuitively, too short text often fails to convey good content. More details about automatic metrics are shown in Appendix A.2. Human Evaluation. We evaluate the response quality based on the following aspects: (1) Empathy reflects whether a response understands the speaker's feeling or situation and responds appropriately.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
(2) Relevance considers whether a response is relevant to the topic mentioned by the speaker. (3) Informativeness evaluates whether a response provides rich and meaningful information. More details about the human evaluation guidance are given in Appendix A.3.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
DIFFUSEMP is based on the architecture of BERTbase (Devlin et al., 2019   Automatic Evaluation Results. The overall results are shown in Table 2. DIFFUSEMP substantially exceeds transformer-based and pre-trained model-based methods on almost all metrics. First, the improvement in controllability is significant. The high success rate indicates the effectiveness of control-range masking for elaborate token generation and demonstrates the ability of DIFFUSEMP to customize responses with desired factors. For informativeness, diffusion model-based methods perform the best, and DIFFUSEMP is even better than DiffuSeq. It has been proven that the diffusion model is a powerful backbone for generating diverse texts. With the integration of control signals, especially fine-grained signal SF, the meaning of each to-be-generated response token is more specific, thus the final response is more informative. When considering informativeness values along with MIScore and AvgLen, we can find that those informative responses generated by DIFFUSEMP are also context-related and long, which satisfies the demand for proper responses to speakers. The BERTScore of DIFFUSEMP is not the highest, and we think this is reasonable since BERTScore indicates the similarity of generated and golden responses, while DIFFUSEMP encourages creativity instead of similarity. Besides, the difference between BERTScore and MIScore can justify that the generated responses are both creative and coherent. Human Evaluation Results. Human evaluation results are listed in Table 3. Our method achieves the highest scores in all aspects, and the greatest improvement is achieved in informativeness, which shows that responses generated by DIFFUSEMP are preferred by annotators. Meanwhile, results of the Oracle setting show that the performance will be further improved when accurate control signals are given, which indicates that obtaining better control signals can be a feasible research topic.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Ablation on Control-Range Masking. To verify the effectiveness of control-range masking, we remove the mask matrix and conduct full selfattention on all input tokens, i.e., input tokens can control or influence the representation of each other. As shown in Table 4, the controllability of three signals decreases when the mask is removed ("w/o Mask"), which justifies that our masking strategy is useful for multi-grained control. Besides, the most significant declines appear at the sentence level, which illustrates that IT has the strongest dependency on the masking strategy. We suppose it is because sentence-level signals are not that explicit like token-level signals with word-by-word alignments or utterance-level signals with global modeling in a dialogue session. Ablation on Control Signals. Another question is whether each control signal plays the corresponding role. We keep the structure of the control-range mask untouched and remove each signal to validate. In detail, we remove the control signal from both the input text and the corresponding row(s) and column(s) in the original mask matrix. Table 4 shows that a success rate decreases when the corresponding control is removed ("w/o CM", "w/o IT", and "w/o SF"), and the finer the granularity of the control signal, the more the performance declines. We can come to the conclusion that each control signal and its control range defined in the mask matrix play an important role in response controllability.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Analysis   coarse control signals at the utterance level, we claim that a fine-grained signal is more useful for better empathy expression. To validate this claim, we remove the fine-grained labels, i.e., token-level SF, to see the performance change. Results are shown in Table 5. Without the token-level control, almost all evaluation metrics decrease in varying degrees. We conjecture that the token-level guidance gives a direct prompt on the content this token should entail, which greatly narrows the space of acceptable output generation. Analysis on Coarse-Grained Signal CM. Emotional Reaction (ER), Interpretation (IP), and Exploration (EX) are three different high-level mechanisms for empathy expression. To explore the ways in which different mechanisms express empathy, we score generated responses in these three aspects with RoBERTa-based annotators as mentioned in Section 3.1. Results are visualized in Figure 4. For each method, the average ER, IP, and EX of generated responses on the test set are represented as the coordinate value of a point. DIFFUSEMP is the closest to human responses in distance, indicating that the way our method expresses empathy is the most similar to human beings. Table 6 shows the syntactically acceptable examples generated by DIFFUSEMP and other comparable methods. Transformer-based methods tend to generate plain and safe words, lacking a deep understanding of the context. In contrast, responses generated by TransferTransfo and BART have more rich information and details. All comparable methods tend to respond in general expressions, and even the way to ask questions is also monotonous, which may be due to the large number of such samples in the dataset. DIFFUSEMP responses entail  features from both context and guidance. Feelings (disgusting, don't feel bad), questions (new relationship), and advice (study for future) fit the situation of the speaker. Our framework is also helpful for generating different responses for a given context. With the support of an emotion-enhanced matching method, multiple response candidates can be returned to further guide response generation with diverse control signals. Control A and B contain intent Suggesting and Questioning, respectively. Thus, DIFFUSEMP A aims to give advice while B focuses on asking questions. More cases are shown in Appendix C.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
We propose DIFFUSEMP, a diffusion model-based framework, for empathetic response generation. To better model multi-dimensional empathy and improve its expression, we utilize multi-grained control signals at utterance, sentence, and token levels. These control signals are directly extracted from golden responses in the training process, while response candidates obtained from an emotionenhanced matching method are used as the signal source. Then we also design a control-range masking strategy and integrate it into the diffusion language model to fulfill elaborate control on the generation of response tokens. Experimental results on a benchmark dataset EMPATHETICDIA-LOGUE show that our method outperforms compet-itive baselines in generating more context-related, informative, and empathetic responses. Our framework is scalable for more control signal types and can also be extended to other controllable conditional text generation tasks.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
In future work, we will extend DIFFUSEMP to more empathetic control signals, and improve the performance of annotators and retrieval tools. Besides, it is interesting to explore DIFFUSEMP on various controllable text generation tasks.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
We thank the reviewers for their detailed and insightful advice. This work is supported by the National Key Research and Development Program of China (NO.2022YFB3102200) and Strategic Priority Research Program of the Chinese Academy of Sciences with No. XDC02030400.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
The difficulty of obtaining accurately-labeled control signals constrains our results. As we report in Table 1, the performance of tagging tools can be further improved. However, when the original dataset lacks multi-grained annotations, relying on pre-trained tools is the most feasible solution. Considering that control signals come from response candidates in the inference stage, the performance of the context-context matching method is another constraint. Finally, the drawback of diffusion models also has an impact on our approach. Despite its high-quality generative performance, the diffusion model has a high requirement for GPU resources and still suffers from slow sampling. We discuss some attempts to address these limitations in Appendix B.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
The EMPATHETICDIALOGUE dataset (Rashkin et al., 2019) used to train and evaluate in the paper is collected by crowd-sourcing using the ParlAI platform to interact with Amazon Mechanical Tunk. Besides, we use EMPATHETICINTENT (Welivita and Pu, 2020), REDDIT (Sharma et al., 2020) and FRAMENET (Baker et al., 1998) to train tagging tools for control signals. All the above datasets are well-established and publicly available. Sensitive and personal privacy information have been removed during the dataset construction. In our human evaluation, participants were fully informed of the purpose of our study and were appropriately compensated. It is important to clarify that our work is only a study of open-domain dialogue with empathy. We claim that our system does not provide professional psychological counseling. In other words, it does not make any treatment recommendations or diagnostic claims. Joseph L Fleiss and Jacob Cohen. 1973 
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
A.1 Comparable Methods
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
The following models are chosen as comparable methods and divided into three groups according to their architecture.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Transformer-Based Methods.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
• • MIME (Majumder et al., 2020): A model utilizing emotion grouping, emotion mimicry, and stochasticity strategies to generate responses.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
• EmpDG (Li et al., 2020): An adversarial model applying two discriminators for interacting with user feedback.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
• CEM (Sahand Sabour, 2021): A model leverages commonsense as additional information to further enhance empathetic response generation.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Pre-Trained Language Model-Based Methods.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
• TransferTransfo (Radford et al., 2019;Wolf et al., 2019): A combination of a transfer learning-based training scheme and a highcapacity GPT-2 model which shows strong improvements over end-to-end conversational models.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
• BART (Lewis et al., 2020): A pre-trained encoder-decoder Transformer with great success in many seq2seq tasks.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Diffusion Model-Based Methods.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
• DiffuSeq (Gong et al., 2022): A diffusion model proposed as a conditional language model and trained end-to-end in a classifierfree manner. It is designed for sequence-tosequence text generation tasks.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Noticed that we did not use Diffusion-LM (Li et al., 2022b) as a baseline because it is incompatible with the sequence-to-sequence task setting. We provide the result of oracle setting as a reference. Under the standard setting, the attributes are not given and need to be predicted from the retrievebased methods, and we focus on evaluating the response quality. Under the oracle setting, the true attributes from the ground truth response are provided, so it can be considered as the theoretical upper limit performance of DIFFUSEMP.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
We evaluate the generated empathetic responses from the following four aspects: relevance, controllability, informativeness, and response length.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Relevance. We use BertScore and the MIScore of response to evaluate relevance.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
• BertScore (Zhang et al., 2020a): BertScore computes a similarity score using contextual embeddings for each token in the candidate sentence with each token in the reference sentence. We use deberta-large-mnli to calculate the BertScore.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
• MIScore: A good response should be informative and relevant to the context. When given the response, it should have the ability to infer its context, while a safe response is generic and can be used in any context, so it is hard to infer the context. From this perspective, we use the idea of Maximum Mutual Information (MMI) (Li et al., 2016;Zhang et al., 2018). The idea of MIScore is employing a pre-trained backward model to predict context sentences from given responses, i.e., P (Context|Response). Intuitively, MIScore encourages the model to generate responses that are more specific to the context, while generic responses are largely less preferred, since they can be used in any case. We calculate MIScore according to the following equation:
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
log P (x t |y 1 , . . . , y n , x <t ),
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
where m and n are the numbers of tokens in the context and response respectively. It is implemented with a reverse 345M Di-aloGPT (Zhang et al., 2020b), which is a finetuned GPT-2 (Radford et al., 2019) with the training objective to predict the context from the response.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Controllability. We calculate the attribute control accuracy success rate to validate the controllability of models. For session-level CM and sentence-level IT, we report accuracy. For tokenlevel SF, we report F1.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Informativeness. We use Distinct n-gram (Li et al., 2016) and self-BLEU (Zhu et al., 2018) to evaluate informativeness.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
• Distinct n-gram (Li et al., 2016): Distinct n-gram calculates the number of distinct ngrams in generated responses. The value is scaled by the total number of generated tokens to avoid favoring long sentences.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
• Self-BLEU (Zhu et al., 2018): Self-BLEU regards one sentence as a hypothesis and the others as a reference, we can calculate the BLEU score for every generated sentence, and define the average BLEU score to be the Self-BLEU of the document.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Response Length.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
• Average Length (Singh and Jin, 2016): The length of the response text is also used as a quality indicator when comparing different model generations since shorter texts usually contain less information.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
It is noteworthy that open-domain dialogue and controllable text generation contain a great deal of creativity. When a sentence is forced to remain identical to a fixed standard sentence, such evaluation metrics may unfairly penalize creative texts, notwithstanding they are capable of responding to the given context. As a result, instead of comparing the word overlap between generated responses and standard responses, we give the metric values of standard responses as a reference.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Quantitative automatic metrics are straightforward to compare, but they may be less effective at reflecting overall levels of empathy. Human judgment is necessary for an open-domain dialogue system (Liu et al., 2016).
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
We recruit three third-party graduate researchers (average age 23.3) to analyze the results of various models. We acquired permission for their participation and paid them in accordance with local hourly wages. The response quality of all models is evaluated in terms of the following three aspects: Empathy, Relevance, and Informativeness. We randomly sample 100 dialogues and corresponding generated responses for different models and then ask three professional annotators to give each response a rating score from the following aspects.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
• Empathy reflects whether the listener understands the feeling of the speaker and responds appropriately.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
• Relevance considers how the content of the reply is relevant to the topic mentioned by the speaker.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
• Informativeness evaluates grammar correctness and readability.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
The specific instruction given to them for the evaluation is shown in Figure 5. Each aspect is on a scale of 1 to 5, in which 1 is "unacceptable" and 5 is "excellent performance".
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Besides, We conduct an A/B test to directly compare our method with other baselines. Another 100 dialogues are randomly sampled from each model. Three annotators are given generated responses from either our method or baselines in random order and are asked to choose a better one. They can either choose one of the responses or select "Tie" when the quality of provided options is hard to access.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Our DIFFUSEMP calculates diffusion model parameters with a BERT-base (Devlin et al., 2019) architecture with 12 layers and 80M parameters. For diffusion settings, we set 2000 diffusion steps in both the training stage and the inference stage. We adopt the square root noise schedule. The max input length is 128, the dimensions of word embedding and time embedding are all 128, and the embedding is randomly initialized * . For training settings, we use AdamW optimizer and set the learning rate as 1e-4, dropout as 0.1. We set gradient clipping to −1.0. γ equals to 0.2. We use WordPiece tokenizer † . The batch size is 128 and the micro-batch size is 64. For all baseline models, we use their official codes to implement and keep the settings in the original paper.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
The limitations of our work have been mentioned in Section 6. Here, we propose some attempts to overcome these limitations.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Control Signals. In the acquisition of control signals, there are two main constraints for performance, including (1) the accuracy of control signals and (2) the suitability of retrieval results in the testing step.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
With regard to (1), the results of the oracle setting demonstrate that our framework has a high ceiling when ground-true control signals are given. Therefore, we have tried to enhance robustness by noising the control factors. Noising methods contain adding, removing, and replacing random control tokens. However, experimental results show that noising methods compromise the success rate of control, which is contrary to the motivation of this work. In the future, this approach can be tried to further improve language quality in scenarios where the demand for controllability is weak.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
With respect to (2), we focus on the performance of the retrieval model in the inference stage. The control signals straightforwardly come from the retrieved responses. In this paper, we have proposed a task-specific design that combines semantic and emotional similarity to retrieve but it is still simple compared to those SOTA dialogue response selection models. In future work, it is meaningful to replace our retrieval model with more powerful response selection methods.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
As an advantage of DIFFUSEMP, both the annotating taggers and the retrieval model are orthogonal to empathetic response generation. It is easy for followers to employ higher-performance response selection models and attribute annotating taggers to empower the DIFFUSEMP.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Diffusion Models. Finally, the diffusion model requires a lot of GPU computational resources and is slow when inference, which limits its application. There are many attempts to reduce the computational resources (Rombach et al., 2021a) required by the diffusion model as well as to speed up the process (Vahdat et al., 2021) and inference (Song et al., 2021;Bao et al., 2022). Theoretically, the relevant improvements would also have an enhancing effect on our framework and would be helpful for spreading the diffusion model to the NLP community.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
We give more responses generated by DIFFUSEMP in Table 7 and Table 8.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Context I feel lonely sometimes cause I live in a house with 5 girls and all of them have boyfriends and sometimes they are all hanging out with their significant others and I feel lonely.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Awww, That sucks man. Hopefully they don't say mean things to you about being lonely. TRS I am sorry to hear that. I am sorry to hear that.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Multi-TRS I am sorry to hear that. I am sorry to hear that.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
I am sorry to hear that. have you tried talking to them? MIME I am sorry to hear that. I am sorry to hear that. I hope you are able to do that.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
That is a shame. I am sorry to hear that.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
CEM I am so sorry to hear that. I am sure you will be fine.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Why were they so angry?
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
BART-FT I'm sorry to hear that. Have you tried joining any clubs?
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Why can not you too Candidate A I see. I'm sorry you're home alone. I get lonely myself when you're used to someone around.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
are you going to advertise for a new roommate? Response C Oh no, that is really amazing to hear. Why were you not able to move? 
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Yes! Traffics is the worst but other people don't pay attention to bad thing.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Candidate C Yes, the cable company is infuriating. do they eventually help you though?
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Response C Yes, the bus company was annoying. Did they already help you out? The Limitation Section on page 9.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
A2. Did you discuss any potential risks of your work?
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
The Ethics Statement section on page 9.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
A3. Do the abstract and introduction summarize the paper's main claims?
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
The Abstract section and 1. Introduction section.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
A4. Have you used AI writing assistants when working on this paper?
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
Left blank.
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
B Did you use or create scientific artifacts?
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
B1. Did you cite the creators of artifacts you used?
DiffusEmp: A Diffusion Model-Based Framework with Multi-Grained Control for Empathetic Response Generation
B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
CEM: Commonsense-aware Empathetic Response Generation
Empathy is a desirable trait of human daily conversations that enables individuals to understand, perceive, and respond appropriately to the situation and feelings of others (Keskin 2014). Previous research has demonstrated that empathetic dialogue systems can improve user experience and satisfaction in multiple domains (Fitzpatrick, Darcy, and Vierhile 2017;Liu et al. 2021;Wang et al. 2021). Hence, it is important to discover ways that allow us to equip open-domain dialogue systems with empathy. Recent work Lin et al. 2019;Majumder et al. 2020;Li et al. 2020a) has proposed various methods of generating empathetic responses that mainly rely on detecting the user's emotion.
CEM: Commonsense-aware Empathetic Response Generation
Empathy is a desirable trait of human daily conversations that enables individuals to understand, perceive, and respond appropriately to the situation and feelings of others (Keskin 2014). Previous research has demonstrated that empathetic dialogue systems can improve user experience and satisfaction in multiple domains (Fitzpatrick, Darcy, and Vierhile 2017;Liu et al. 2021;Wang et al. 2021). Hence, it is important to discover ways that allow us to equip open-domain dialogue systems with empathy. Recent work Lin et al. 2019;Majumder et al. 2020;Li et al. 2020a) has proposed various methods of generating empathetic responses that mainly rely on detecting the user's emotion.
CEM: Commonsense-aware Empathetic Response Generation
However, empathy is a broad construct that includes aspects of affection and cognition (Davis 1983). The affective aspect is concerned with the emotional simulation in reaction to the user's experiences (Cuff et al. 2016) while the cognitive aspect aims to understand the user's situation and the implied feelings (Elliott et al. 2018). Hence, though emotion is one of the important factors of empathy, it is not the Copyright © 2022, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Figure 1: Examples from the EMPATHETICDIALOGUES dataset in which commonsense is used to gain additional information about the user's emotion and situation before responding empathetically.
CEM: Commonsense-aware Empathetic Response Generation
However, empathy is a broad construct that includes aspects of affection and cognition (Davis 1983). The affective aspect is concerned with the emotional simulation in reaction to the user's experiences (Cuff et al. 2016) while the cognitive aspect aims to understand the user's situation and the implied feelings (Elliott et al. 2018). Hence, though emotion is one of the important factors of empathy, it is not the Copyright © 2022, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Figure 1: Examples from the EMPATHETICDIALOGUES dataset in which commonsense is used to gain additional information about the user's emotion and situation before responding empathetically.
CEM: Commonsense-aware Empathetic Response Generation
only determining factor. This is demonstrated in Figure 1, where both affective and cognitive empathy are used to form empathetic responses. For instance, in the first case, the user shares information about their emotion (I felt pretty good) as well as their experience (hit a new pr on the overhead press). Accordingly, we can assume that the user is Proud of their achievement and must have worked hard to reach this level. Since these assumptions are not explicitly mentioned by the user, we as humans tend to rely on our own knowledge and commonsense reasoning to draw these implications. Therefore, we believe that providing dialogue systems with this external knowledge could play a critical role in understanding the user's situation and feelings, which leads to more informative and empathetic responses.
CEM: Commonsense-aware Empathetic Response Generation
Towards this end, we propose the Commonsense-aware Empathetic Chatting Machine (CEM). CEM leverages external commonsense knowledge to obtain more information about the user's situation and feelings (i.e. user's reaction, intention, desire, etc.). Such additional information is used to improve the cognitive understanding and thus, enhance the empathy expression in the generated responses. We evaluate our approach on EMPATHETICDIALOGUES, a widelyused benchmark dataset for empathetic response generation. Both automatic and manual evaluation results demonstrate that compared to previous methods, CEM can generate more informative and empathetic responses. Our contributions are summarized as follows:
CEM: Commonsense-aware Empathetic Response Generation
• We propose to leverage commonsense to improve the understanding of interlocutors' situations and feelings, which is an important part of cognitive empathy. • We introduce CEM, a novel approach that uses various types of commonsense reasoning to enhance empathetic response generation. • Automatic and manual evaluation demonstrate that with the addition of commonsense, CEM is able to generate more informative and empathetic responses compared with the previous methods.
CEM: Commonsense-aware Empathetic Response Generation
Empathy is a fairly new term in the literature and therefore, has no specific or widely accepted definition in the fields of social psychology and psychotherapy (Macarov 1978;Elliott et al. 2011). However, empathy is commonly known as a complex multi-dimensional construct that includes broad aspects of affection and cognition (Davis 1983;Zheng et al. 2021). Affective empathy enables us to experience the emotion of others through various emotional stimuli (Cuff et al. 2016), while cognitive empathy enables us to understand the situations and implicit mental states of others, such as intentions, causes, desires, requirements, etc. (Elliott et al. 2018).
CEM: Commonsense-aware Empathetic Response Generation
Empathy is a fairly new term in the literature and therefore, has no specific or widely accepted definition in the fields of social psychology and psychotherapy (Macarov 1978;Elliott et al. 2011). However, empathy is commonly known as a complex multi-dimensional construct that includes broad aspects of affection and cognition (Davis 1983;Zheng et al. 2021). Affective empathy enables us to experience the emotion of others through various emotional stimuli (Cuff et al. 2016), while cognitive empathy enables us to understand the situations and implicit mental states of others, such as intentions, causes, desires, requirements, etc. (Elliott et al. 2018).
CEM: Commonsense-aware Empathetic Response Generation
In recent years, research on implementing empathy in dialogue systems and generating empathetic responses has gained considerable attention. Initially, Rashkin et al. (2019) demonstrated that detecting the user's emotion is an essential part of generating empathetic responses. Lin et al. (2019) designed a separate decoder for each available emotion and softly combined their outputs. Majumder et al. (2020) proposed that empathetic responses should also mimic the user's emotion to a degree. Li et al. (2020a) leveraged user feedback and proposed a multi-resolution adversarial framework for this task. Recently, (Li et al. 2020b) used commonsense knowledge from ConceptNet (Speer, Chin, and Havasi 2017) to gain a better understanding of the implied emotions within the context. However, these works usually focus on detecting the context emotion and do not pay enough attention to the cognitive aspect of empathy.
CEM: Commonsense-aware Empathetic Response Generation
In recent years, research on implementing empathy in dialogue systems and generating empathetic responses has gained considerable attention. Initially, Rashkin et al. (2019) demonstrated that detecting the user's emotion is an essential part of generating empathetic responses. Lin et al. (2019) designed a separate decoder for each available emotion and softly combined their outputs. Majumder et al. (2020) proposed that empathetic responses should also mimic the user's emotion to a degree. Li et al. (2020a) leveraged user feedback and proposed a multi-resolution adversarial framework for this task. Recently, (Li et al. 2020b) used commonsense knowledge from ConceptNet (Speer, Chin, and Havasi 2017) to gain a better understanding of the implied emotions within the context. However, these works usually focus on detecting the context emotion and do not pay enough attention to the cognitive aspect of empathy.
CEM: Commonsense-aware Empathetic Response Generation
As mentioned, a major part of cognitive empathy is understanding the situations and feelings of others. When interacting with a dialogue system, the user is not expected to explicitly share all the information about their situation and how they may feel. As humans, we use our commonsense knowledge to make connections between what is explicitly mentioned and what is implied. Hence, we hypothesize that enabling dialogue systems to leverage commonsense and drive implications from what the user has explicitly shared is highly beneficial for a better understanding of the user's situation and feelings, which leads to more effective cognitive empathy and thus, more empathetic responses.
CEM: Commonsense-aware Empathetic Response Generation
In this work, we use ATOMIC ) as our commonsense knowledge base. ATOMIC is a collection of commonsense reasoning inferences about everyday if-then events. For each event, ATOMIC infers six commonsense relations for the person involved in the event: the effect of the event on the person (xEffect), their reaction to the event (xReact), their intent before the event (xIntent), what they need in order for the event to happen (xNeed), what they would want after the event(xWant), and an inferred attribute of the person's characteristics (xAttr). Since predicting a person's attributes merely based on a given event would include judging the other person, which is not included in the empathetic process (Peloquin 1995), we neglect xAttr in our approach and use the remaining five relations.
CEM: Commonsense-aware Empathetic Response Generation
In this work, we use ATOMIC ) as our commonsense knowledge base. ATOMIC is a collection of commonsense reasoning inferences about everyday if-then events. For each event, ATOMIC infers six commonsense relations for the person involved in the event: the effect of the event on the person (xEffect), their reaction to the event (xReact), their intent before the event (xIntent), what they need in order for the event to happen (xNeed), what they would want after the event(xWant), and an inferred attribute of the person's characteristics (xAttr). Since predicting a person's attributes merely based on a given event would include judging the other person, which is not included in the empathetic process (Peloquin 1995), we neglect xAttr in our approach and use the remaining five relations.
CEM: Commonsense-aware Empathetic Response Generation
In order to generate commonsense inferences for given events, we adopt COMET (Bosselut et al. 2019), which is a pre-trained GPT-2 model (Radford et al. 2018) that is finetuned on triplets (e, r, i) from ATOMIC, where e, r, i are the event, the relation type, and the inferred knowledge respectively. More specifically, we use a modified BART-based (Lewis et al. 2020) variation of COMET, which is trained on the ATOMIC-2020 dataset (Hwang et al. 2021). This model is equipped with knowledge that is not readily available to pre-trained language models and is more suitable for inferring knowledge regarding unseen events (Hwang et al. 2021). The latter is necessary for our use-case as many of the events within an empathetic conversation may not occur on a daily basis and therefore, may not exist in the original ATOMIC dataset.
CEM: Commonsense-aware Empathetic Response Generation
In order to generate commonsense inferences for given events, we adopt COMET (Bosselut et al. 2019), which is a pre-trained GPT-2 model (Radford et al. 2018) that is finetuned on triplets (e, r, i) from ATOMIC, where e, r, i are the event, the relation type, and the inferred knowledge respectively. More specifically, we use a modified BART-based (Lewis et al. 2020) variation of COMET, which is trained on the ATOMIC-2020 dataset (Hwang et al. 2021). This model is equipped with knowledge that is not readily available to pre-trained language models and is more suitable for inferring knowledge regarding unseen events (Hwang et al. 2021). The latter is necessary for our use-case as many of the events within an empathetic conversation may not occur on a daily basis and therefore, may not exist in the original ATOMIC dataset.
CEM: Commonsense-aware Empathetic Response Generation
We conduct our experiments on the EMPATHETICDIA-LOGUES ), a large-scale multi-turn dataset containing 25k empathetic conversations between crowdsourcing workers. The dataset also provides an emotion label for each conversation from the total 32 available emotions. In this dataset, each conversation is between a speaker and a listener. The task requires a dialogue model to play the role of the listener and generate empathetic responses.
CEM: Commonsense-aware Empathetic Response Generation
is the i-th utterance that consists of M i words. Our goal is to generate the listener's next utterance u k which is coherent to the context, informative, and empathetic to the speaker's situation and feelings.
CEM: Commonsense-aware Empathetic Response Generation
Our proposed model, CEM, is built upon the standard Transformer (Vaswani et al. 2017) and its overview is illustrated in Figure 2. The process of CEM is mainly divided into five stages: context encoding, knowledge acquisition, context refinement, knowledge selection, and response generation.
CEM: Commonsense-aware Empathetic Response Generation
Our proposed model, CEM, is built upon the standard Transformer (Vaswani et al. 2017) and its overview is illustrated in Figure 2. The process of CEM is mainly divided into five stages: context encoding, knowledge acquisition, context refinement, knowledge selection, and response generation.
CEM: Commonsense-aware Empathetic Response Generation
Following previous work (Lin et al. 2019;Majumder et al. 2020), we concatenate the utterances in the dialogue history and prepend a special token [CLS] to obtain the context We acquire the embedding E C of the sequence C by summing up the word embedding, positional embedding, and dialogue state embedding. As each utterance in C could be from either the listener or the speaker, we use the dialogue state embedding to distinguish between the two parties. The sequence embedding E C is then fed to a context encoder to produce the contextual representation:
CEM: Commonsense-aware Empathetic Response Generation
Following previous work (Lin et al. 2019;Majumder et al. 2020), we concatenate the utterances in the dialogue history and prepend a special token [CLS] to obtain the context We acquire the embedding E C of the sequence C by summing up the word embedding, positional embedding, and dialogue state embedding. As each utterance in C could be from either the listener or the speaker, we use the dialogue state embedding to distinguish between the two parties. The sequence embedding E C is then fed to a context encoder to produce the contextual representation:
CEM: Commonsense-aware Empathetic Response Generation
where H CT X ∈ R L×d , L is the length of the sequence, and d is the hidden size of the context encoder.
CEM: Commonsense-aware Empathetic Response Generation
For input sequence C, we respectively append five special relation tokens ( ) to the last utterance in the dialogue history and use COMET to generate five commonsense inferences [cs r 1 , cs r 2 , ..., cs r 5 ] per relation r. For each relation, we concatenate the generated commonsense inferences to obtain its commonsense sequence CS r = cs r 1 ⊕ cs r 2 ⊕ ... ⊕ cs r 5 . Given that xReact demonstrates the knowledge regarding the affective state (i.e. user's emotion) while the other relation represents knowledge regarding the cognitive state (i.e. user's situation), we divide the relations into two groups: affective and cognitive. Accordingly, similar to the previous section, we prepend [CLS] to the cognitive sequences. As the inferences for xReact are usually emotion words (e.g. sad, happy, angry) rather than sentences, we would simply use the average of its hidden representations to represent these sequences. Based on the mentioned grouping, the resulting sequences are fed to two separate cognitive and affective encoders:
CEM: Commonsense-aware Empathetic Response Generation
where H xReact ∈ R lxReact×d , H r ∈ R lr×d , with l xReact , l r being the lengths of the commonsense inference sequences, and r ∈{xWant, xNeed, xIntent, xEffect}. Then, we use the average hidden representation for affective relations and the hidden representation of [CLS] for cognitive relations to represent these sequences respectively:
CEM: Commonsense-aware Empathetic Response Generation
where h xReact , h r ∈ R d .
CEM: Commonsense-aware Empathetic Response Generation
Similar to Majumder et al. (2020), in order to refine the context by additional information, we first respectively concatenate each of the commonsense relation representations (Equations 4 & 5) to the context representation H CT X at the token level (i.e. U r ∈ R L×2d ):
CEM: Commonsense-aware Empathetic Response Generation
Similar to Majumder et al. (2020), in order to refine the context by additional information, we first respectively concatenate each of the commonsense relation representations (Equations 4 & 5) to the context representation H CT X at the token level (i.e. U r ∈ R L×2d ):
CEM: Commonsense-aware Empathetic Response Generation
In contrast to concatenating the representations at a sequence level (i.e. adding additional information to the end of the context representation), token-level concatenation enables us to fuse the additional knowledge within each word in the sequence. Accordingly, we use two separate encoders (affectionrefined and cognition-refined), corresponding to the two groups of relations, to encode the fused representations and obtain commonsense-refined context representations for each relation respectively:
CEM: Commonsense-aware Empathetic Response Generation
where
CEM: Commonsense-aware Empathetic Response Generation
Emotion Classification In order to acquire a more accurate prediction of the user's affective state, given that we are provided with an emotion label e * for each conversation, we use the hidden representation of the [CLS] token from the affection-refined context representation (h Af f ) to perform emotion classification:
CEM: Commonsense-aware Empathetic Response Generation
where h Af f ∈ R d . Hence, we pass h Af f through a linear layer followed by a Softmax operation to produce the emotion category distribution P emo ∈ R q , where q is the number of available emotion categories:
CEM: Commonsense-aware Empathetic Response Generation
where W e ∈ R d×q is the weight vector for the linear layer. During training, we optimize these weights by minimizing the Cross-Entropy (CE) loss between the emotion category distribution P and the ground truth label e * :
CEM: Commonsense-aware Empathetic Response Generation
Merely using one of the commonsense representations to produce an empathetic response is not ideal. For instance, if we only rely on the affection-refined context, the generated responses would likely be about how the user's emotions (e.g. You must be proud.), whereas using the cognitionrefined contexts may lead to responses that focus more on the situation (e.g. You must have worked really hard.). Hence, we want to enable our model to generate responses based on the mixture of both affective and cognitive information. To this end, we first concatenate all the five relationrefined contexts at the token level:
CEM: Commonsense-aware Empathetic Response Generation
where H Ref ine ∈ R L×5d . To highlight the more important features within the refined context representation, we apply the Sigmoid function on H Ref ine to measure the importance of each relation-refined context for response generation. Then, we multiply H Ref ine by the consequent importance scores, as done by Majumder et al. (2020). Finally, the obtained representation is passed through a Multi-Layer Perceptron (MLP) with ReLU activation, which learns how to mix the commonsense knowledge of different relations into a combined contextualized representation:
CEM: Commonsense-aware Empathetic Response Generation
where H Ref ine ∈ R L×5d . To highlight the more important features within the refined context representation, we apply the Sigmoid function on H Ref ine to measure the importance of each relation-refined context for response generation. Then, we multiply H Ref ine by the consequent importance scores, as done by Majumder et al. (2020). Finally, the obtained representation is passed through a Multi-Layer Perceptron (MLP) with ReLU activation, which learns how to mix the commonsense knowledge of different relations into a combined contextualized representation:
CEM: Commonsense-aware Empathetic Response Generation
where H CT X ∈ R L×d and denotes element-wise multiplication.
CEM: Commonsense-aware Empathetic Response Generation
Lastly, the target response Y = [y 1 , . . . , y T ] with length T , which has the same meaning of u k in Section 2.3, is generated by the decoder token by token:
CEM: Commonsense-aware Empathetic Response Generation
where E y<t denotes the embeddings of the tokens that have been generated. Note that the cross attention to the encoder outputs is modified to the commonsense-refined contextual representation H CT X , which has fused the information from both the context and the commonsense inferences.
CEM: Commonsense-aware Empathetic Response Generation
We adopt the standard negative log-likelihood (NLL) loss on the target response Y :
CEM: Commonsense-aware Empathetic Response Generation
Response Diversity In our preliminary experiments, we noticed that the models trained on our studied dataset tend to generate similarly generic empathetic responses. As shown in Table 1, there are phrases that are extensively repeated within the model responses in this dataset. Hence, similar to the problem raised by Li et al. (2016) for generic response generation in Seq2Seq models, we believe that models trained on this dataset tend to assign a higher probability to responses that include these phrases and thus, generate safe empathetic responses (e.g. I am sorry to hear that, That is good to hear, and Oh no that is awful). We consider these responses safe and generic as they do not necessarily rely on nor give much information about the user's context and can be employed in many different situations.
CEM: Commonsense-aware Empathetic Response Generation
Response Diversity In our preliminary experiments, we noticed that the models trained on our studied dataset tend to generate similarly generic empathetic responses. As shown in Table 1, there are phrases that are extensively repeated within the model responses in this dataset. Hence, similar to the problem raised by Li et al. (2016) for generic response generation in Seq2Seq models, we believe that models trained on this dataset tend to assign a higher probability to responses that include these phrases and thus, generate safe empathetic responses (e.g. I am sorry to hear that, That is good to hear, and Oh no that is awful). We consider these responses safe and generic as they do not necessarily rely on nor give much information about the user's context and can be employed in many different situations.
CEM: Commonsense-aware Empathetic Response Generation
That is a / Oh no that / To hear that ≥ 67 I am so / Sorry to hear / Wow that is ≥ 50 That is really / I am sure / That is great ≥ 40 Table 1: Most common trigrams in the training set of EM-PATHETICDIALOGUES. Proportion represents the number of responses that include the trigram divided by the total number of responses (e.g. more than 50% of the responses include the trigram Sorry to hear).
CEM: Commonsense-aware Empathetic Response Generation
To tackle this issue, we implement Frequency-Aware Cross-Entropy (FACE) (Jiang et al. 2019) as an additional loss to penalize high-frequency tokens using a weighting scheme. Hence, during training and prior to receiving a new batch of samples, we first calculate the relative frequency RF i for each vocabulary token c i in the training corpus:
CEM: Commonsense-aware Empathetic Response Generation
To tackle this issue, we implement Frequency-Aware Cross-Entropy (FACE) (Jiang et al. 2019) as an additional loss to penalize high-frequency tokens using a weighting scheme. Hence, during training and prior to receiving a new batch of samples, we first calculate the relative frequency RF i for each vocabulary token c i in the training corpus:
CEM: Commonsense-aware Empathetic Response Generation
where V denotes the vocabulary size. Accordingly, we derive the frequency-based weight w i as follows:
CEM: Commonsense-aware Empathetic Response Generation
where a = −(max 1≤j≤V (RF j )) −1 is the frequency slope and 1 is added as the bias so that w i falls into [0, 1]. Since more frequent tokens would have a higher relative frequency, the obtained weights ensure that these tokens have lower weights. Lastly, we normalize w i to have a mean of 1, as done by Jiang et al. (2019). The diversity loss would then be calculated as below:
CEM: Commonsense-aware Empathetic Response Generation
where a = −(max 1≤j≤V (RF j )) −1 is the frequency slope and 1 is added as the bias so that w i falls into [0, 1]. Since more frequent tokens would have a higher relative frequency, the obtained weights ensure that these tokens have lower weights. Lastly, we normalize w i to have a mean of 1, as done by Jiang et al. (2019). The diversity loss would then be calculated as below:
CEM: Commonsense-aware Empathetic Response Generation
where c i is a candidate token in the vocabulary and δ t (c i ) is the indicator function, which equals to 1 if and only if c i = y t and 0 otherwise. All the parameters for our proposed model are trained and optimized based on the weighted sum of the three mentioned losses.
CEM: Commonsense-aware Empathetic Response Generation
where γ 1 , γ 2 , and γ 3 are hyper-parameters that we use to control the influence of the three losses. In our experiments, we set γ 1 = 1, γ 2 = 1, and γ 3 = 1.5. During our analysis, we found that setting the same coefficients for all losses did not produce sufficient penalties for the generic responses. Hence, we assigned a slightly higher value to γ 3 .
CEM: Commonsense-aware Empathetic Response Generation
We selected the following baseline models for comparison:
CEM: Commonsense-aware Empathetic Response Generation
• Transformer (Vaswani et al. 2017): The original Transformer, which is trained to optimize the NLL loss (L nll ). • Multi-Task Transformer (Multi-TRS) : A variation of the Transformer that has an additional unit for predicting the emotion. This model is trained to jointly optimize the NLL loss (L nll ) and the cross-entropy loss for emotion classification (L emo ). • MoEL (Lin et al. 2019): A Transformer-based model that uses a decoder for each possible user emotion, referred to as listener, and softly combines the representations from these decoders to generate a response. Therefore, each decoder is optimized to learn how to respond to one type of emotion while a meta decoder is optimized to combine their representations and generate a response. • MIME (Majumder et al. 2020): Another Transformerbased model that mimics the detected user emotion to a degree. In this approach, the emotions are separated into negative and positive emotions. The model initially generates mimicking and non-mimicking representations for the response based on the emotion groups and is optimized to effectively blend these representations and generate an empathetic response. • EmpDG (Li et al. 2020a): A multi-resolution adversarial framework that consists of an empathetic generator and an interactive discriminator. The generator produces empathetic responses based on the detected emotion while the discriminator ensures that the generated responses are consistent with the context and are also empathetic. To provide a fair comparison with our model and the other baselines, we only implement the empathetic generator in our experiments as the discriminator requires information from the future turns within the conversation.
CEM: Commonsense-aware Empathetic Response Generation
• Transformer (Vaswani et al. 2017): The original Transformer, which is trained to optimize the NLL loss (L nll ). • Multi-Task Transformer (Multi-TRS) : A variation of the Transformer that has an additional unit for predicting the emotion. This model is trained to jointly optimize the NLL loss (L nll ) and the cross-entropy loss for emotion classification (L emo ). • MoEL (Lin et al. 2019): A Transformer-based model that uses a decoder for each possible user emotion, referred to as listener, and softly combines the representations from these decoders to generate a response. Therefore, each decoder is optimized to learn how to respond to one type of emotion while a meta decoder is optimized to combine their representations and generate a response. • MIME (Majumder et al. 2020): Another Transformerbased model that mimics the detected user emotion to a degree. In this approach, the emotions are separated into negative and positive emotions. The model initially generates mimicking and non-mimicking representations for the response based on the emotion groups and is optimized to effectively blend these representations and generate an empathetic response. • EmpDG (Li et al. 2020a): A multi-resolution adversarial framework that consists of an empathetic generator and an interactive discriminator. The generator produces empathetic responses based on the detected emotion while the discriminator ensures that the generated responses are consistent with the context and are also empathetic. To provide a fair comparison with our model and the other baselines, we only implement the empathetic generator in our experiments as the discriminator requires information from the future turns within the conversation.
CEM: Commonsense-aware Empathetic Response Generation
We implemented all the models using PyTorch 1 and used 300-dimensional pre-trained GloVE vectors (Pennington, Socher, and Manning 2014) to initialize the word embeddings, which were shared between the encoders and the decoders. The hidden dimension for all corresponding components were set to 300. Adam (Kingma and Ba 2017) optimizer with β 1 = 0.9 and β 2 = 0.98 was used for training. The initial learning rate was set to 0.0001 and we varied this value during training according to Vaswani et al. (2017). All the models were trained on one single TITAN Xp GPU using a batch size of 16 and early stopping. We used a batch size of 1 and a maximum of 30 decoding steps during testing and inference. We used the same 8:1:1 train/valid/test split as provided by Rashkin et al. (2019).
CEM: Commonsense-aware Empathetic Response Generation
We implemented all the models using PyTorch 1 and used 300-dimensional pre-trained GloVE vectors (Pennington, Socher, and Manning 2014) to initialize the word embeddings, which were shared between the encoders and the decoders. The hidden dimension for all corresponding components were set to 300. Adam (Kingma and Ba 2017) optimizer with β 1 = 0.9 and β 2 = 0.98 was used for training. The initial learning rate was set to 0.0001 and we varied this value during training according to Vaswani et al. (2017). All the models were trained on one single TITAN Xp GPU using a batch size of 16 and early stopping. We used a batch size of 1 and a maximum of 30 decoding steps during testing and inference. We used the same 8:1:1 train/valid/test split as provided by Rashkin et al. (2019).
CEM: Commonsense-aware Empathetic Response Generation
We employed Perplexity (PPL) and Distinct-n (Dist-n) (Li et al. 2016) as our main automatic metrics. PPL represents the model's confidence in its set of candidate responses, with higher confidence resulting in a lower PPL. This can be used to evaluate the general quality of the generated responses. Dist-n measures the proportion of unique n-grams in the generated responses and is commonly used to evaluate generation diversity. In addition, since our proposed model and the baselines models (except Transformer) all perform emotion classification as a part of their training process, we also report the prediction accuracy (Acc). As Liu et al. (2016) had found that word overlap-based automatic metrics such as BLEU (Papineni et al. 2002) are not appropriate for evaluating dialogue systems, we do not report such metrics. Table 2 shows the automatic evaluation results. CEM achieves the lowest perplexity, which suggests the overall quality of our generated responses is higher than the baselines. In addition, our model also considerably outperforms the baselines in terms of Dist-n, which highlights the importance of the diversity loss. In terms of emotion classification, CEM had a much higher accuracy compared to the baselines, which suggests the addition of commonsense knowledge is also beneficial for detecting the user's emotion. 
CEM: Commonsense-aware Empathetic Response Generation
We employed Perplexity (PPL) and Distinct-n (Dist-n) (Li et al. 2016) as our main automatic metrics. PPL represents the model's confidence in its set of candidate responses, with higher confidence resulting in a lower PPL. This can be used to evaluate the general quality of the generated responses. Dist-n measures the proportion of unique n-grams in the generated responses and is commonly used to evaluate generation diversity. In addition, since our proposed model and the baselines models (except Transformer) all perform emotion classification as a part of their training process, we also report the prediction accuracy (Acc). As Liu et al. (2016) had found that word overlap-based automatic metrics such as BLEU (Papineni et al. 2002) are not appropriate for evaluating dialogue systems, we do not report such metrics. Table 2 shows the automatic evaluation results. CEM achieves the lowest perplexity, which suggests the overall quality of our generated responses is higher than the baselines. In addition, our model also considerably outperforms the baselines in terms of Dist-n, which highlights the importance of the diversity loss. In terms of emotion classification, CEM had a much higher accuracy compared to the baselines, which suggests the addition of commonsense knowledge is also beneficial for detecting the user's emotion. 
CEM: Commonsense-aware Empathetic Response Generation
In previous work, human evaluation was conducted via two tasks: first, crowdsourcing workers were asked to assign a score from 1 to 5 to the generated responses based on the aspects of fluency, relevancy, and empathy; second, they were required to choose the better response between two models within the same context. However, the criteria for giving a score from 1 to 5 is highly likely to vary between different individuals, which results in low inter-annotator agreement and is not a suitable indicator of a model's performance.
CEM: Commonsense-aware Empathetic Response Generation
In addition, asking workers to choose the better responses without any guidelines and solely relying on their own preference is not satisfactory. This is due to the fact that each person may consider aspects that are different from what is being investigated when making their choices, which is also not a reliable indicator of user preference.  To address these issues, we conducted an aspect-based pairwise preference test. That is, for a given context, we paired our model's response with a response from the baselines and asked annotators to choose the better response based on the context and the following three aspects: 1) Coherence (Coh.): which response is more coherent in con-tent and relevant to the context; 2) Empathy (Emp.): which response shows more understanding of the user's situation and presents a more appropriate emotion; 3) Informativeness (Inf.): which response conveys more information about the context. Then, we randomly sampled 100 response pairs and assigned three crowdsourcing workers to annotate each pair. Ties were allowed but the annotators were encouraged to choose one of the responses.
CEM: Commonsense-aware Empathetic Response Generation
As shown in Table 3, CEM outperforms the baselines in all of the three aspects. Particularly, with the enhancement of commonsense knowledge, our model was able to produce responses that conveyed more specific and informative content and thus were more empathetic. We also note CEM did not significantly outperform MIME in informativeness. Upon further investigation, we realized that on average, MIME tends to generate longer responses (12.8 words / response) compared to CEM (9.6 words / response). It is possibly due to some annotators considering these responses as more informative since they included more words. However, as shown by the results of the automatic evaluation (Table 2), we can observe that MIME has the second-lowest Dist-2 score, which suggests that its generated responses may follow similar patterns and have less diversity.
CEM: Commonsense-aware Empathetic Response Generation
We conducted ablation studies to verify the effectiveness of each of the components in our model. Specifically, we designed three variants of CEM: 1) w/o Aff: the affective and affection-refined encoders are removed (Equations 2 & 8), the affect representation is neglected in the commonsenserefined representation (Equation 14), and the hidden representation of the [CLS] token from the encoded context (Equation 1) is used for emotion classification; 2) w/o Cog: the cognitive and cognition-refined encoders are removed (Equations 3 & 9), the cognition representation is neglected in the commonsense-refined representation (Equation 14), and the MLP is replaced with a linear layer (Equation 15); 3) w/o Div: the diversity loss is removed from the training objectives (Equation 21).
CEM: Commonsense-aware Empathetic Response Generation
The obtained results are shown in Table 2. Both affective and cognitive information have a considerable impact on the emotion classification accuracy, which suggests that information about both the user's emotion and their situation are necessary for correctly identifying their feelings. In addition, we also observed that removing the diversity loss results in considerably lower Dist-n scores, which indicates the effectiveness of this loss in generating more diverse responses. Table 4 shows comparisons between the generated responses of CEM and the three main baselines. In the first case, the baselines fail to realize the meaning behind ready for a puppy, which implies that the user wants to buy or adopt a puppy. It can be observed that MoEL dismisses this implication while the other two baselines mistake the meaning behind the phrase for being ready for an event or exam, which may cause the user to be proud of themselves. By accessing external knowledge, CEM better acknowledges the
CEM: Commonsense-aware Empathetic Response Generation
I decided a few months I was ready for a puppy, so I put the word out.
CEM: Commonsense-aware Empathetic Response Generation
That is good, I am glad you did not get a dog.
CEM: Commonsense-aware Empathetic Response Generation
That is so sweet! I bet you are so proud! EmpDG
CEM: Commonsense-aware Empathetic Response Generation
That is awesome! I bet you are very proud of you! 
CEM: Commonsense-aware Empathetic Response Generation
Oh, it sure is, I feel the same way. They grow up so fast.
CEM: Commonsense-aware Empathetic Response Generation
Speaker: Yeah about 10 years ago I had a horrifying experience. It was 100% their fault but they hit the water barrels and survived. They had no injuries but they almost ran me off the road. Listener: Did you suffer any injuries? Speaker: No I was not hit. It turned out they were drunk. I felt guilty but realized it was his fault.
CEM: Commonsense-aware Empathetic Response Generation
That is so scary! I hope you are okay! MIME I am sorry to hear that. I hope you can find someone else to do that.
CEM: Commonsense-aware Empathetic Response Generation
Oh no! I am sorry to hear that. I hope you were able to get it fixed.
CEM: Commonsense-aware Empathetic Response Generation
Remorseful, Guilty, Sorry Cognition To be safe, No harm done, No one hit CEM I am glad you did not get hurt.
CEM: Commonsense-aware Empathetic Response Generation
Why did you feel guilty? People really shouldn't drive drunk. user's situation and implied feelings and generates an empathetic response that covers both aspects of empathy. That is, by detecting that the user might be excited and may want to get a dog, it responds with both affective (that is great) and cognitive (did you get a good dog?) statements. Similarly, in the second case, unlike the baselines, CEM successfully detects that the user is being nostalgic, happy and sad, where the latter two emotions are likely to be implied in the word bittersweet. In addition, CEM realizes that the user's intent behind looking through photos of their children was to reminisce memories, which suggests that the user enjoys having those memories.
CEM: Commonsense-aware Empathetic Response Generation
The final case demonstrates CEM's ability to express both affective and cognitive empathy in multi-turn dialogue. As shown, all the baselines dismiss the user's statement I was not hit, which implies that they are fine and no harm was done. In contrast, CEM correctly recognizes that there is no harm done to the user and regardless of detecting that the user might have remorse and guilt, it chooses to focus more on the important part of this situation, which is the user's health and safety.
CEM: Commonsense-aware Empathetic Response Generation
In this paper, we proposed the Commonsense-aware Empathetic Chatting Machine (CEM) to demonstrate how leveraging commonsense knowledge could benefit the understanding of the user's situation and feelings, which leads to more informative and empathetic responses. Our empirical automatic and manual evaluation indicated that the effectiveness of our approach in empathetic response generation.
CEM: Commonsense-aware Empathetic Response Generation
In the future, our work can inspire other approaches to leverage commonsense knowledge for empathetic response generation and similarly promising tasks (e.g. providing emotional support ).