What techniques have been proposed to measure and evaluate the effectiveness of task-oriented dialogue systems?
Task-oriented Dialogue Systems. Unlike
chitchat dialogue systems that aim at conversing
with users without specific goals, task-oriented
dialogue systems assist users to accomplish
certain tasks (Feng et al., 2021; Eric et al., 2020).
Task-oriented dialogue systems can be divided
into module-based methods (Feng et al., 2022b;
Ye et al., 2022; Su et al., 2022; Heck et al.,
2020; Chen et al., 2020a; Wu et al., 2019a; Lei
et al., 2018; Liu and Lane, 2016) and end-to-end
methods (Feng et al., 2022a; Qin et al., 2020; Yang
et al., 2020; Madotto et al., 2018; Yao et al., 2014).
To measure the effectiveness of task-oriented
dialogue systems, evaluation is a crucial part of the
development process. Several approaches have
been proposed including automatic evaluation
metrics (Rastogi et al., 2020; Mrkši ́c et al., 2017),
human evaluation (Feng et al., 2022a; Goo et al.,
2018), and user satisfaction modeling (Sun et al.,
2021; Mehrotra et al., 2019). Automatic evaluation
metrics, such as BLEU (Papineni et al., 2002),
make a strong assumption for dialogue systems,
which is that valid responses have significant word
overlap with the ground truth responses. However,
there is significant diversity in the space of valid
responses to a given context (Liu et al., 2016).
Human evaluation is considered to reflect the
overall performance of the system in a real-world
scenario, but it is intrusive, time-intensive, and
does not scale (Deriu et al., 2021). Recently, user
satisfaction modeling has been proposed as the
main evaluation metric for task-oriented dialogue
systems, which can address the issues listed above.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Detecting and understanding implicit measures of user satisfaction are essential for enhancing recommendation quality. When users interact with a recommendation system, they leave behind fine grained traces of interaction signals, which contain valuable information that could help gauging user satisfaction. User interaction with such systems is often motivated by a specific need or intent, often not explicitly specified by the user, but can nevertheless inform on how the user interacts with, and the extent to which the user is satisfied by the recommendations served. In this work, we consider a complex recommendation scenario, called Slate Recommendation, wherein a user is presented with an ordered set of collections, called slates, in a specific page layout. We focus on the context of music streaming and leverage fine-grained user interaction signals to tackle the problem of predicting user satisfaction. We hypothesize that user interactions are conditional on the specific intent users have when interacting with a recommendation system, and highlight the need for explicitly considering user intent when interpreting interaction signals. We present diverse approaches to identify user intents (interviews, surveys and a quantitative approach) and identify a set of common intents users have in a music streaming recommendation setting. Additionally, we identify the importance of shared learning across intents and propose a multi-level hierarchical model for user satisfaction prediction that leverages user intent information alongside interaction signals. Our findings from extensive experiments on a large scale real world data demonstrate (i) the utility of considering different interaction signals, (ii) the role of intents in interpreting user interactions and (iii) the interplay between interaction signals and intents in predicting user satisfaction.
[{"first":"Rishabh","last":"Mehrotra","middle":[]},{"first":"Mounia","last":"Lalmas","middle":[]},{"first":"Doug","last":"Kenney","middle":[]},{"first":"Thomas","last":"Lim-Meng","middle":[]},{"first":"Golli","last":"Hashemian","middle":[]}]
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
To alleviate the problem of structured databases' limited coverage, recent task-oriented dialogue systems incorporate external unstructured knowledge to guide the generation of system responses. However, these usually use word or sentence level similarities to detect the relevant knowledge context, which only partially capture the topical level relevance. In this paper, we examine how to better integrate topical information in knowledge grounded task-oriented dialogue and propose ``Topic-Aware Response Generation'' (TARG), an end-to-end response generation model. TARG incorporates multiple topic-aware attention mechanisms to derive the importance weighting scheme over dialogue utterances and external knowledge sources towards a better understanding of the dialogue history. Experimental results indicate that TARG achieves state-of-the-art performance in knowledge selection and response generation, outperforming previous state-of-the-art by 3.2, 3.6, and 4.2 points in EM, F1 and BLEU-4 respectively on Doc2Dial, and performing comparably with previous work on DSTC9; both being knowledge-grounded task-oriented dialogue datasets.
[{"first":"Yue","last":"Feng","middle":[]},{"first":"Gerasimos","last":"Lampouras","middle":[]},{"first":"Ignacio","last":"Iacobacci","middle":[]}]
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
Attention-based recurrent neural network models for joint intent detection and slot filling have achieved the state-of-the-art performance, while they have independent attention weights. Considering that slot and intent have the strong relationship, this paper proposes a slot gate that focuses on learning the relationship between intent and slot attention vectors in order to obtain better semantic frame results by the global optimization. The experiments show that our proposed model significantly improves sentence-level semantic frame accuracy with 4.2% and 1.9% relative improvement compared to the attentional model on benchmark ATIS and Snips datasets respectively
[{"first":"Chih-Wen","last":"Goo","middle":[]},{"first":"Guang","last":"Gao","middle":[]},{"first":"Yun-Kai","last":"Hsu","middle":[]},{"first":"Chih-Li","last":"Huo","middle":[]},{"first":"Tsung-Chieh","last":"Chen","middle":[]},{"first":"Keng-Wei","last":"Hsu","middle":[]},{"first":"Yun-Nung","last":"Chen","middle":[]}]
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Virtual assistants such as Google Assistant, Alexa and Siri provide a conversational interface to a large number of services and APIs spanning multiple domains. Such systems need to support an ever-increasing number of services with possibly overlapping functionality. Furthermore, some of these services have little to no training data available. Existing public datasets for task-oriented dialogue do not sufficiently capture these challenges since they cover few domains and assume a single static ontology per domain. In this work, we introduce the the Schema-Guided Dialogue (SGD) dataset, containing over 16k multi-domain conversations spanning 16 domains. Our dataset exceeds the existing task-oriented dialogue corpora in scale, while also highlighting the challenges associated with building large-scale virtual assistants. It provides a challenging testbed for a number of tasks including language understanding, slot filling, dialogue state tracking and response generation. Along the same lines, we present a schema-guided paradigm for task-oriented dialogue, in which predictions are made over a dynamic set of intents and slots, provided as input, using their natural language descriptions. This allows a single dialogue system to easily support a large number of services and facilitates simple integration of new services without requiring additional training data. Building upon the proposed paradigm, we release a model for dialogue state tracking capable of zero-shot generalization to new APIs, while remaining competitive in the regular setting.
[{"first":"Abhinav","last":"Rastogi","middle":[]},{"first":"Xiaoxue","last":"Zang","middle":[]},{"first":"Srinivas","last":"Sunkara","middle":[]},{"first":"Raghav","last":"Gupta","middle":[]},{"first":"Pranav","last":"Khaitan","middle":[]}]
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Human evaluations of machine translation are extensive but expensive. Human evaluations can take months to ﬁnish and involve human labor that can not be reused. We propose a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run. We present this method as an automated understudy to skilled human judges which sub-stitutes for them when there is need for quick or frequent evaluations. 1 in note.
[{"first":"Kishore","last":"Papineni","middle":[]},{"first":"Salim","last":"Roukos","middle":[]},{"first":"Todd","last":"Ward","middle":[]},{"first":"Wei-Jing","last":"Zhu","middle":[]}]
Neural Belief Tracker: Data-Driven Dialogue State Tracking
One of the core components of modern spoken dialogue systems is the belief tracker, which estimates the user’s goal at every step of the dialogue. However, most current approaches have difficulty scaling to larger, more complex dialogue domains. This is due to their dependency on either: a) Spoken Language Understanding models that require large amounts of annotated training data; or b) hand-crafted lexicons for capturing some of the linguistic variation in users’ language. We propose a novel Neural Belief Tracking (NBT) framework which overcomes these problems by building on recent advances in representation learning. NBT models reason over pre-trained word vectors, learning to compose them into distributed representations of user utterances and dialogue context. Our evaluation on two datasets shows that this approach surpasses past limitations, matching the performance of state-of-the-art models which rely on hand-crafted semantic lexicons and outperforming them when such lexicons are not provided.
[{"first":"Nikola","last":"Mrkšić","middle":[]},{"first":"Diarmuid","last":"Ó Séaghdha","middle":[]},{"first":"Tsung-Hsien","last":"Wen","middle":[]},{"first":"Blaise","last":"Thomson","middle":[]},{"first":"Steve","last":"Young","middle":[]}]
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
User Satisfaction Modeling (USM) is one of the popular choices for task-oriented dialogue systems evaluation, where user satisfaction typically depends on whether the user’s task goals were fulfilled by the system. Task-oriented dialogue systems use task schema, which is a set of task attributes, to encode the user’s task goals. Existing studies on USM neglect explicitly modeling the user’s task goals fulfillment using the task schema. In this paper, we propose SG-USM, a novel schema-guided user satisfaction modeling framework. It explicitly models the degree to which the user’s preferences regarding the task attributes are fulfilled by the system for predicting the user’s satisfaction level. SG-USM employs a pre-trained language model for encoding dialogue context and task attributes. Further, it employs a fulfillment representation layer for learning how many task attributes have been fulfilled in the dialogue, an importance predictor component for calculating the importance of task attributes. Finally, it predicts the user satisfaction based on task attribute fulfillment and task attribute importance. Experimental results on benchmark datasets (i.e. MWOZ, SGD, ReDial, and JDDC) show that SG-USM consistently outperforms competitive existing methods. Our extensive analysis demonstrates that SG-USM can improve the interpretability of user satisfaction modeling, has good scalability as it can effectively deal with unseen tasks and can also effectively work in low-resource settings by leveraging unlabeled data.Code is available at https://github.com/amzn/user-satisfaction-modeling.
[{"first":"Yue","last":"Feng","middle":[]},{"first":"Yunlong","last":"Jiao","middle":[]},{"first":"Animesh","last":"Prasad","middle":[]},{"first":"Nikolaos","last":"Aletras","middle":[]},{"first":"Emine","last":"Yilmaz","middle":[]},{"first":"Gabriella","last":"Kazai","middle":[]}]
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Evaluation is crucial in the development process of task-oriented dialogue systems. As an evaluation method, user simulation allows us to tackle issues such as scalability and cost-efficiency, making it a viable choice for large-scale automatic evaluation. To help build a human-like user simulator that can measure the quality of a dialogue, we propose the following task: simulating user satisfaction for the evaluation of task-oriented dialogue systems. The purpose of the task is to increase the evaluation power of user simulations and to make the simulation more human-like. To overcome a lack of annotated data, we propose a user satisfaction annotation dataset, USS, that includes 6,800 dialogues sampled from multiple domains, spanning real-world e-commerce dialogues, task-oriented dialogues constructed through Wizard-of-Oz experiments, and movie recommendation dialogues. All user utterances in those dialogues, as well as the dialogues themselves, have been labeled based on a 5-level satisfaction scale. We also share three baseline methods for user satisfaction prediction and action prediction tasks. Experiments conducted on the USS dataset suggest that distributed representations outperform feature-based methods. A model based on hierarchical GRUs achieves the best performance in in-domain user satisfaction prediction, while a BERT-based model has better cross-domain generalization ability.
[{"first":"Weiwei","last":"Sun","middle":[]},{"first":"Shuo","last":"Zhang","middle":[]},{"first":"Krisztian","last":"Balog","middle":[]},{"first":"Zhaochun","last":"Ren","middle":[]},{"first":"Pengjie","last":"Ren","middle":[]},{"first":"Zhumin","last":"Chen","middle":[]},{"first":"Maarten","last":"Rijke","middle":["de"]}]
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
An increasingly larger proportion of users rely on recommendation systems to pro-actively serve them recommendations based on diverse user needs and expectations. Developing a better understanding of how users interact with such recommender systems is important not only for improving user experience but also for developing satisfaction metrics for effective and efficient optimization of the recommendation algorithm. This is especially true in the case of online streaming services like Pandora, Spotify and Apple Music, wherein the system could gauge user satisfaction and adapt its recommendations to better serve user needs.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Since obtaining explicit feedback from users is prohibitively expensive and challenging to implement in real world systems, commercial systems rely on exploiting implicit feedback signals derived from user activity. When users interact with the recommendations served, they leave behind fine-grained traces of interaction patterns, which could be leveraged for predicting how satisfied was their experience, and for developing metrics of user satisfaction.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Prior work have studied implicit feedback signals (e.g., clicks, dwell time, mouse scrolling) in the case of web search, and verified their effectiveness in predicting user satisfaction, both on traditional desktop [9,11,26,27] and mobile setting [17,39]. Furthermore, past work in web search systems has also highlighted the importance of considering tasks and intents when interpreting implicit feedback signals [14,38]. On the other hand, the identification and effectiveness of corresponding implicit feedback signals as well as the role of user intents has remained understudied in the context of recommendation systems, especially in the mobile context.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
While search systems have access to explicit queries from users, based on which one could extract tasks, and interpret interaction signals, thereby differentiating between success and failure; recommender systems, on the other hand, lack such explicit indicators of user intent and clear indicators of success. Indeed, the interpretation of signals varies with goals; for example, scrolling can indicate negative experience when the goal is to quickly listen to music now, but can also indicate a positive experience when the goal is to browse the diverse collection of music the system has to offer. Furthermore, interpreting interaction signals becomes especially hard in the context of complex recommendation settings, like Slate recommendations, a scenario typical to many music streaming services, where users are recommended a set of collections (called slates), with different purposes (to explore new music, or quickly jump to recently played music, etc), and heterogeneous content (playlists, artist profiles, other audio content, etc). Thus, there is a need for a detailed, holistic view of user interactions with such recommender systems, to establish their utility in predicting satisfaction.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
In this work, we consider the use case of music streaming via slates of recommendation, and aim at understanding the relationship between interaction signals, user intents and user satisfaction. We consider the case of users interacting with a mobile based music streaming app, Spotify, and investigate the different interaction signals that can be extracted in slate recommendation setting, and how these interaction signals vary across different intents. Since the list of possible intents user might have is hitherto unknown, we adopt a mixed methods approach to understand user intents, and leverage insights from (i) face-to-face interviews, (ii) large scale in-app survey and (iii) non-parametric clustering techniques to identify the list of possible intents. We identify eight user intents and verify their validity using large scale log data.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
To predict user satisfaction, we jointly leverage insights from the extracted interaction signals and intents. While interaction signals have been directly used to predict satisfaction in search [9,24,27], we instead hypothesize that interpreting interaction signals without factoring in user intent would lead to noisy, unreliable estimates of satisfaction in our context. We demonstrate that traditional approaches of using a single, global model for satisfaction trained on aggregate data without considering intent groups does not perform well in predicting satisfaction. We show that significant performance gains are obtained in switching from a single global prediction model to separate satisfaction models for each intent.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Further, we identify issues with the global and per-intent models. While the former fails to capture intent-specific intricacies, separate per-intent models ignore information and insights from other intents. To address these, we show the importance of shared learning across intents and propose a multi-level hierarchical model that allows the estimation of intent-specific effects while at the same time learning from data across all intents, thereby compromising between the overly noisy per-intent model and the oversimplified global model that ignores intent groups. The proposed multi-level model facilitates incorporating both individual session level and intent group level effect on user satisfaction, thereby allowing for variability in user interaction behavior across intents.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Extensive experiments on a real world large scale music streaming data from Spotify highlight the benefit of explicitly considering user intent and demonstrate the effectiveness of different interaction signals in predicting user satisfaction in a slate recommendation setting. We contend that our findings provide a valuable framework for fine-grained analysis of user interaction behavior for developing metrics and gauging user satisfaction. To the best of our knowledge, the present study is among the first to identify the benefit of incorporating user intent information in understanding user interactions and satisfaction in recommendation settings, and to demonstrate the significant gains obtained over the popularly used single global intent-agnostic prediction models.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
The current research builds upon four areas: (i) slate recommendations, (ii) user interactions for satisfaction prediction, (iii) user intent modeling and (iv) mixed methods approach to analysis. Slate Recommendation. Recommendation of slates is a common problem arising in various contexts including search [33], ads and recommendations [8]. Several approaches have been proposed to generate slates, including List Conditional Variational Auto-Encoders [12] and Slate-MDPs [32]. A different approach [33] investigates the evaluation of policies that recommend slates, and then introduces a pseudo-inverse estimator for off-policy evaluation in combinatorial contextual bandits. Different from proposing a new approach to generate slates, our work is concerned with investigating user interaction and gauging user satisfaction with recommended slates.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Satisfaction & User Interaction Signals. With respect to interaction signals, implicit feedback signals (e.g., clicks, dwell time, mouse scrolling) and their sequences have been extensively studied in web search and their effectiveness in predicting satisfaction [9,11,24,26,27,29] has been verified, using techniques such as Markov models or deep neural models. A detailed overview regarding the evaluation of interactive information retrieval systems with users can be found in Kelly et al. [13].
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Scrolls and cursor movements do not exist in mobile search; however, the swipe interaction performs a similar function. Past work leveraging user interaction on mobile phones have used signals such as swipes, dwell times on landing pages and zooms for detecting search result relevance [10] as well as for gauging user attention and satisfaction [25]. User interactions with touch based devices have also been used to detect good abandonment [39] and user satisfaction with intelligent assistants [17]. Building upon such work, we leverage swipe interactions and some signals that result from swipe activity to predict satisfaction.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
In the context of recommendation systems, temporal features, including variants of dwell time have been considered for understanding satisfaction [16] and relevance [41]. Specifically focusing on evaluating recommender systems, past work has investigated preference elicitation [19], explanations [34] and user centric combinations of explicit and implicit preferences [18]. Past work has also investigated the influence of personal and situational characteristics towards explaining user experience with recommender systems [20,22].
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Our work also investigates implicit feedback signals, similarly to those works, but in a less explored context, that of music recommendation. Additionally, we aim to investigate the hypothesis on how the interpretation of such interaction signals and their role in predicting satisfaction changes across different user intents.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
User Intent Modeling. Related to our consideration of user intents, the role of query intents and search tasks have been extensively studied in the search community, with past work aimed at leveraging intents in predicting query and task satisfaction [27,37,42]. Recent work reported in [1] considers the role of intents in recommendations and propose an intent-aware recommendation model. In music recommendation, recent efforts have been made around initial exploration of music listening intents associated with common activities [35], as well as studies of why and when people of different ages and in different countries listen to music [36]. These investigations suggest that intent modeling can potentially improve recommendation quality. In web search, tasks have been considered as an analogous abstraction to intents. Prior work has emphasized understanding the role of task level differences while leveraging implicit feedback. Specifically, White et al. [38] investigated how individual and task differences impact the effectiveness of algorithms based on implicit relevance feedback. Additionally, Kelly et al. [14] considered the effects of task on display time, and quantified the potential impact of this relationship on the effectiveness of display time as implicit feedback.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Mixed methods. Finally, our mixed methods approach of leveraging interviews along with qualitative and quantitative insights follow recent works investigating user expectations, behaviors, and satisfaction in the context of music discovery [5], image [40] and product search [30].
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Our goal is to extract and leverage user interaction data to understand and predict user satisfaction in a slate recommendation setting. We consider the Homepage of a major commercial music streaming service, and base our experiments around user interaction with the resulting recommendation page.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
We first define the key concepts used throughout the paper.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
• User Session: A session is defined as a sequence of user actions with the recommendations rendered, including active browsing of content and passive listening to music. A user session ends when the user has not interacted with or streamed from the Homepage for more than 30 minutes. • Slate Recommendation: A slate is a collection of playlists grouped together under a common theme. Figure 1 shows two slates, Popular Playlists and Jump Back In, each with 3 playlists. The Slate Recommendation task is to recommend a collection of slates. • Homepage: Also referred to as the home tab, the homepage is the first landing page of the music streaming app, which surfaces a number of playlists, organized as different slates. Figure 1 gives an example of the homepage of Spotify, the music streaming app used in this study. • User Satisfaction: User satisfaction can be viewed as a subjective measure of the utility of recommendations, and we posit that user satisfaction is conditioned not only on the recommendations served, but also on user intent. We rely on implicit signals to derive satisfaction estimates and then use user reported satisfaction scores as gold standard estimate of user satisfaction.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
The slate recommendation algorithm aims to find the optimal, ordered subset of items (playlists), a.k.a. slate, given the page layout to serve users recommendations so as to maximize user satisfaction. A Homepage is a collection of such recommended slates, which the user can interact with. Since users come to the app with different goals at different points in time, it becomes important for system designers to understand user interaction to gauge user satisfaction with the slates of recommendations rendered on the app Homepage.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
In the rest of this section, we shed light on the different interaction signals we extract from user interaction with the Homepage of Spotify and briefly motivate the need for considering user intent for understanding and predicting user satisfaction in our context.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
The Homepage rendered for a user is rich enough to allow him or her to interact with it in a myriad of ways, including clicking on playlists, scrolling vertically to view more slates, scrolling horizontally to view more playlists in a specific slate, pausing to read and visually absorb content, clicking and consuming content via streaming, among others. While past work on understanding user interaction in mobile search setting have proposed few signals [17,39], we additionally propose a number of new signals resulting from the specific Slate Recommendation scenario considered in this work.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
We use back-end logs of user interactions and extract four different types of interaction signals for each user session:
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
(1) Temporal signals: these focus on the temporal aspects of user interaction, including time spent interacting with the slates, session length, dwell time, etc. (2) Surface level signals: these capture aggregate user interaction on the surface of the Homepage, including total number of clicks, total number of slates the user interacted with, maximum depth the user reached by vertical scrolling, and total number of exits the user had from the Homepage. (3) Downstream signals: these capture downstream user engagement resulting from the Homepage, i.e., user interaction with the playlists, including streaming songs, saving or downloading tracks, viewing Artist profile pages. (4) Derivative signals: these are derived using interaction features like whether the session was abandoned, or whether the session was intentional, with the user going to the Home tab from elsewhere in the app. The average value of click is derived by the considering all clicks in a session and dividing by the total number of streams observed in that session. Table 1 provides a detailed description of the different interaction signals extracted for each user session.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
In the use case of a slate recommendations on a surface like some of the big music streaming apps, users use the Homepage for different needs at different times. With differing needs and intents, users would interact with the app differently, and hence leave different traces of behavioural signals.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Indeed, a user might just want to quickly play some background music, and would not spend much time in finding music to play; on the other hand, a different user might want to carefully find music to save for future listening. In both cases, the way the user interacts with the music app differs, thereby leading to different interaction signals. Since implicit measures of user satisfaction rely heavily on interpreting interaction signals, we hypothesize that understanding and identifying user intent is important to predict user satisfaction in our context, as has been done in other scenarios, particularly in web search [3,15,21,27,31].
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
To this end, we discuss different ways of identifying user intents in a recommendation setting in Section 4, and leverage the intents identified along with the extracted interactions signals to predict user satisfaction in Section 5.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Users use recommendation platforms with different intents. The space of possible intents a user might have is hitherto unknown. In this section, we focus on identifying the different intents users might have when interacting with a music recommendation system. 
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
To discover and better understand user intents when interacting with the Homepage composed of a set of slate recommendations (playlists), we conducted in-depth 1:1 in-person interviews with users of the streaming app. We describe the interview process and discuss our findings.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Approach. We interviewed 12 users living in the city of New York who were found to have varying degrees of Homepage usage with the Spotify app in the 4 weeks prior to the study. We selected participants from both iOS and Android users with a range of levels of engagement with the app (low, medium, high) and the Homepage (rarely visit, frequently visit), with engagement defined as the number of days they actively used the app in the past month. We emailed a screener survey to a list of Spotify users with these specifications to recruit and schedule the interviews. Table 2 provides a detailed demograpic overview of the participants. With each user, we conducted 1:1 face-to-face interviews spanning 45 minutes each and provided them a $100 gift card upon completion of the interview.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Our goal was to understand and discuss their intents and satisfaction when using the Homepage. We asked them questions around (i) What intents they have when coming to the Homepage; (ii) How satisfied they are with their experience, and (iii) How they behave in relation to intent and satisfaction. The participants were encouraged to provide detailed responses to these questions which were recorded (audio and video).
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Findings. We performed a detailed post-hoc analysis of the data from the interviews by going through the video recording of the interviews and the interview transcripts. We grouped together findings that appeared in more than one interview in order to create themes. As per our initial hypothesis, different participants reported to using the Homepage of Spotify for different goals at different times. We were able to categorize user intents into two main themes, Passively Listening, and Actively Listening, and to identify the following overarching intents:
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
(1) Theme 1: Passively Listening (a) To find music to play in the background (b) To play music that matches their mood or activity (c) To quickly access their playlists or saved music (2) Theme 2: Actively Engaging (a) To discover new music to listen to now (b) To save new music or follow new playlists to listen to later (c) To explore artists or albums more deeply Additionally, some users pointed out that they use the Homepage because it is the default page that opens upon loading the music app, without having a clear intent when they go on it. We therefore added Because it is the first screen shown as an additional intent to the above, thereby making it a total of seven identified intents.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
These intents differ from those reported in [36], since they correspond to user intents in using a homepage consisting of slates of recommendations, whereas the intents identified in [36] are user intents for listening to music in general.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Importantly, users often alluded to the fact that their behavior on the Homepage varied with their intent. For example, scrolling can be an indicator of exploration, but scrolling back and forth can signal a struggle. Not scrolling could mean satisfaction with the recommendations presented, but could also indicate dissatisfaction with what was presented. This motivates the need for interpreting interaction signals differently for different intents.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Following up on the intents identified from the face-to-face interviews, we performed a large scale survey to validate and understand the prevalence of the intents identified from the interviews. The survey allowed us to collect data from users about their intent and satisfaction during a given session with the Homepage.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Methodology. We selected a random sample of 3 million iPhone users of the Spotify app residing in the US, and presented them with a brief survey at the end of their Homepage session. To not interfere with their natural interaction, and to not bias the user interaction data collected from a session, we trigger the survey only after the user ended their session and moved to another feature of the app (e.g. Search, My Library). To avoid inundating users with the survey repeatedly, the in-app survey is trigged such that each user participates in the survey at most once. Furthermore, we assume that a user has one overarching intent when they begin their Homepage session, an assumption we intend to relax as part of future work.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
To gauge user satisfaction we asked the following question: How satisfied or dissatisfied were you with your experience on the Home screen today? and presented them with 5 options ranging from Very satisfied to Very Dissatisfied, along with an additional option of "I wasn't on the Home screen today". Further, to understand their intent behind using the Homepage, we asked them the following question: Why were you on the Home screen today? and showed them the six intents identified during the interviews.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
As the listed intents may not capture all possible intents users might have, we added Other as an option. The Other intent option was accompanied by a free text block where users were asked to specify their intent; this allows us to identify any intent not identified during the interviews. Parsing the free flow natural language text and identifying common intents from it is a non-trivial task, which we address in Section 4.3.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Survey Results & Findings. We briefly discuss findings about user intents from the large scale survey, and leave the discussion of findings about user satisfaction for later (Section 6.2). The response rate was 4.5%, with over 116000 users from the 3 million users targeted responding to the survey, which is similar to the response rates we observed in our past surveys. Figure 3 presents the distribution of intents across user sessions, which indicates how prevalent the intents are. We observe a fairly even distribution of intents across passively listening and actively engaging, which the top two intents: (i) To quickly access their playlists or saved music and (ii) To discover new music to listen to now, covering 29% and 24% of sessions respectively. A sizeable portion of the sessions involve the user saving music for later consumption or exploring artists in detail, with over 20000 user sessions dedicated to these intents.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Given that the majority of the sessions could be linked to one of the intents identified during interviews, it validates the identified intents. However, since over 15% of users opted for Other, indicating that their intent is not covered in the presented list of intents, we further investigate the presence of other intents from the text entered by users. The next section describes a computational approach to analyze the responses to identify new intents.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
To ensure that the intents identified sufficiently cover the space of all possible intents users might have when interacting with the Homepage of the Spotify app, we investigate the text entered by the users when they selected the Other intent category in the survey responses. Simple clustering on the text is not much useful since the number of possible intents in not known apriori. As a result, we resort to Bayesian nonparametric clustering [7].
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
The distance dependent Chinese restaurant process (dd-CRP) [2] was recently introduced to model random partitions of data, and is a commonly used method for non-parametric clustering [2,28]. To extract intents, we consider the dd-CRP model in an embeddingspace setting and place a dd-CRP prior over the intents. While a detailed description of non-parametric clustering is beyond the scope of this work, we briefly describe the major steps below. Interested readers are referred to Blei et al. [2] who introduce the dd-CRP model, and to Mehrotra et al. [28] who apply dd-CRP model to identify query clusters.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Let z i denote the ith intent assignment, the index of the text with whom the ith text is linked. Let d i j denote the distance measurement between texts i and j, let D denote the set of all distance measurements between texts, and let f be a decay function. The distance dependent CRP independently draws the text assignments to intent cluster conditioned on the distance measurements,
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Here, d i j is an externally specified distance between texts i and j, and α determines the probability that a customer links to themselves rather than another customer. The monotonically decreasing decay function f (d ) mediates how the distance between two texts affects their probability of connecting to each other. Cosine similarity between the word2vec embedding representation of two input text describe the distance function (d i j ).
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Given a decay function f , distances between texts D, scaling parameter α, and an exchangeable Dirichlet distribution with parameter λ, N M-word queries are drawn as follows,
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
If z i R * q 1:N , set the parameter for the ith text to θ i = θ q i . Otherwise draw the parameter from the base distribution,
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
We employ a Gibbs sampler, wherein we iteratively draw from the conditional distribution of each latent variable, given the other latent variables and observations. The Gibbs sampler iteratively draws from
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
The first term is the dd-CRP prior and the second is the likelihood of observations (x) under the partition, and t (z) is the intent-cluster formed from the assignments z. We employ a Dirichlet-Multinomial conjugate distribution to model the likelihood of text terms. Input texts from users are assigned to intent clusters by considering sets of texts that are reachable from each other through the intent cluster assignments. Notice that many configurations of text assignments might lead to the same intent-cluster assignment.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Finally, text assignments can produce a cycle, e.g., text 1 linking with 2 and text 2 linking with 1. This still determines a valid intentcluster assignment; all texts linked in a cycle are assigned to the same intent cluster.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Upon running the dd-CRP model on the 15% response with user entered text, we obtained 5 different clusters of user responses. We manually investigated these clusters and identified the following intents: (i) To Find X, where X could be any specific playlist a user was trying to find; (ii) To explore or casually browse; (iii) All of the above, wherein the users wanted to select all options presented to them, (iv) I wasn't on Home tab, and (v) Miscellaneous.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Among these five newly identified intents, I wasn't on Home tab indicates that the users did not participate in a Homepage session, so we discarded these sessions. Further, Miscellaneous cluster mainly consisted of incoherent text entered by users, thereby rendering it futile. Similarly, we discard sessions tagged with All of the above cluster, since it does not present us with new intent and only had few sessions. Finally, the casually browsing intent was a very small cluster. In summary, we identified one additional intent: To Find X to be added to the pool of the seven intents identified earlier. Table 3: Final list of user intents identified when interacting with the slated recommended by the Homepage of a music streaming app. Although Intent 1 is not an actual intent, we still refer to it as Intent for ease of writing.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Homepage is the first screen shown (i.e. default screen) Intent 2
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
To quickly access my playlists or saved music Intent 3
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
To discover new music to listen to now Intent 4
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
To play music that matches my mood or activity Intent 5
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
To Find X Intent 6
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
To find music to play in the background Intent 7 To save new music or follow new playlists to listen to later Intent 8
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
To explore artists or albums more deeply. Users often use the Homepage to find something they're looking for, which could be any specific playlist or any specific shelf. The extracted additional intent caters to such search-like use-cases. Although the rigorous process of non-parametric clustering applied to user entered responses only gives us one new intent, it reaffirms the exhaustiveness and coverage of the six intents identified before.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Based on the interviews, in-app survey and non-parametric clustering of textual user responses, we end up with eight user intents, as described in Table 3. The diversity of these intents highlights the variety in what users intend to achieve when using the Homepage of the music app.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
We hypothesize that the way users interact with the recommended slates of playlists would differ across these intents. Figure 3 presents a heatmap of a subset of interaction signals (see Table  1) with the different intents. We observe that the prominence of interaction signals significantly differs across intents, with certain signals like interaction time on the Homepage significantly lower for intent 2 (to quickly access music). These differences in interaction signals highlight the fact that users indeed behave differently when having different intents. Since implicit measures of satisfaction highly depend on interaction signals, this places special emphasis of explicitly considering intents when predicting satisfaction.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Our main goal is to understand and predict user satisfaction (SAT) using interaction data. To this end, we extracted detailed user interaction signals, and identified different intents users might have.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
In this section, we leverage the extracted signals and intents and present techniques to predict user satisfaction using the signals. We present three approaches for satisfaction prediction, covering the spectrum of intent-level granularity, i.e. global model for all intents to a separate model for each intent and a shared model across intents.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
We begin by describing the most conventional approach to satisfaction prediction, referred to as Global Model. The simplest technique would be to treat all user sessions as a homogeneous collection of data, with the user intent featuring simply as a categorical variable along with user interaction signals. Almost all existing work on predicting user satisfaction from interaction signals [9,11,26,27] have employed such an approach to satisfaction (SAT) prediction.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
For the Global prediction model, we perform traditional Logistic Regression and Gradient Boosted Decision Trees classifiers. Given a dataset with n user sessions (x i ∈ X ), each of which is represented by m features, D = {(x i , y i )} (|D| = n, x i ∈ R m , y i ∈ R), the logistic regression classifier assigns a probability to each outcome as
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
where c ∈ {−1, 1} and is trained with conditional maximum likelihood estimation, wherein we choose θ that maximize the (log) probability of the labels given the observations x i . Thus, the objective function we aim at maximizing is:
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
where σ is the sigmoid function. Alternatively, tree boosting type of models (e.g. Gradient Boosted Regression Trees) optimize a tree ensemble model using additive functions to predict the output. We train both Logistic Regression and GBDT models using all the interaction features extracted for each session, along with a categorical variable describing user intent for the current session. As shown in Figure 4 (a), the same parameter is learned and shared across all intents and sessions.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
A key issue with a Global Model is that it ignores the variations in the interaction signals across different intents. Indeed, a user with an intent to quickly access her playlist to play music (Intent 2) would know what she is looking for, and not spend time in detailed exploration, unlike a user interested in exploring an artist more deeply (Intent 8). Whereas the categorical variable for the session intent would slightly help the model account for intent specific traits, a better approach would be to train a separate model for each intent, referred as Per-Intent Model illustrated in Figure 4 (b). Let η i denote the intent for session i, and K = 7 be the total number of intents, i.e. η i ∈ [1, K]. We divide the session data (D = {(x i , y i )}) into K different intent specific groups and train K different SAT prediction models, one for each intent:
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
with (θ k , b k ) representing the parameters of the k-th intent SAT model. While the Per-Intent model is able to capture intent specific intricacies of the different interaction signals, it suffers from a few drawbacks. It prohibitively collect data for each intent separately, and there may not be enough data to learn a meaningful model for each intent separately. This problem gets compounded when platform changes encourages users to use the platform for new intents. Furthermore, an intent specific model fails to benefit from shared learning of parameters and hence fails to leverage insights from the interplay between interaction signals and satisfaction data of other intents. We next present a hierarchical multi-level approach to address these shortcomings.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Both the Global and Per-Intent SAT prediction models introduced suffer from various disadvantages. The Per-Intent SAT prediction model works by using just the local intent specific information, and assumes that the data is sampled from separate models for each intent, thereby ignoring information and insights from other intents. Furthermore, it can be rendered futile for intents with small labeled data. At the other extreme, the Global SAT prediction model ignores intent-level variations in the user interaction data and inadvertently suppresses variations that can be important.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
To address these issues, we resort to Multi-level Modeling [6,23], which represents a compromise between the two extremes of a single global prediction model and a separate prediction model for each intent. Multi-level models, illustrated in Figure 4 (c), are regression and classification models in which the constituent model parameters are given probability models. This implies that the model parameters are allowed to vary by group. This approach can be used to handle clustered or grouped data (e.g. different user intents), wherein each level is (potentially) a source of unexplained variability. Multi-level modeling provides a coherent model that simultaneously incorporates both individual and group level models.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Modeling the satisfaction prediction model as a multi-level model offers numerous benefits. First, leveraging multilevel models allows us to account for the intent level grouping of the user session. Second, they facilitate incorporating both individual session level as well as intent group level effects on user satisfaction, thereby allowing for variability in user interaction behavior across different intents. Third, by assuming that the intent group level effects come from a common distribution shared across all intents, they facilitate information sharing across different intents. This can help in improving the accuracy and predictive performance for intents with relatively little data.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
We model the satisfaction prediction problem with a multi-level logistic regression model with the different user intents serving as groups for the hierarchical model. The Multi-level Model can be written as:
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
with η i denoting the intent group for the i-th user session, and x i denoting the user interaction signals extracted for the session. In our hierarchical model, parameters are viewed as a sample from a population distribution of parameters.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Thus, we view them as being neither entirely different (as was the case in the Per-Intent model) or exactly the same (as was the case in the Global Model). Figure 4 pictorially depicts the main difference between the parameter sharing across the three different satisfaction prediction models investigated in this work.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
We perform an evaluation to compare the different satisfaction prediction models, consider the role of different interaction features and present results on a large scale real world dataset.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
We work with real world user data from Spotify, a large music streaming service, and conduct a large scale in-app survey to collect judgments about intents and user satisfaction. We trigger the in-app survey to a random sample of over 3 million users, and observe a response rate of 4.5%. In total, we received responses from over 116000 users, resulting in over 200K judgments about intents and satisfaction combined. For each session, we collected back-end logs of user interactions with the slates of recommendation and extracted data for all the different interaction signals mentioned in Table 1. For each session, we use the intent selected by the user as the session intent and use the satisfaction prediction label given by the user to train and evaluate our satisfaction prediction models.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Owing to the response bias, i.e., the respondents of the survey are likely to be users with a positive bias toward the system, we would end up with a biased dataset for training, which would lead to unreliable prediction estimates. We mitogate this by oversampling the minority class by synthetic minority over-sampling technique [4]. This results in a balanced dataset with a healthy distribution of labels across both satisfaction and dissatisfaction cases.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
We begin with an analysis of the survey responses by understanding the degree to which users are satisfied overall, and across different intents. Figure 5 (left) presents the overall distribution of user satisfaction. We observe that most users are either Very satisfied or Somewhat satisfied with the slates recommended to them in their current session with over 33% user sessions being judged as very satisfying. On the other extreme, over 12000 user sessions were reported to be very or somewhat dissatisfying. A relatively larger number of user sessions were judged as neutral.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
To gauge satisfaction for each intent, we plot the satisfaction label distribution separately for each intent in Figure 5 (right). Similar to overall satisfaction, we observe that more user sessions are satisfying than they are dissatisfying. Users with intents 3, 6 and 7 were more satisfied with the slate recommendations shown to them, than users pursuing other goals, which highlights the efficacy of the system in facilitating users to discover new music for now (Intent 3), playing music in background (Intent 6) and saving music for future (Intent 7). However, the distribution of the different levels of satisfaction differs across intents. As can be seen in the violin plot, a substantial number of user sessions were judged as neutral on satisfaction scale for Intent 8, whereas the distribution is skewed more towards the top-most level of satisfaction for Intents 3 and 6.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Considering such differences in distribution of satisfaction across intents helps system designers in identifying those intents where recommendation system under-performs, an insight which instead would have been hidden when considering overall user satisfaction. 
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
We next investigate the extent to which the proposed interaction signals and the predictive models are able to predict user satisfaction. We consider the satisfaction label provided by the user during an in-app survey as ground truth label, and consider a user to be satisfied if their response was either Very Satisfied or Somewhat Satisfied. In all other cases, the user session was labeled as dissatisfied. Such binarization of user satisfaction isn't ideal, but we contend that it is a good starting point and has been extensively used in prior industrial research on understanding and predicting user satisfaction [17,27,39].
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
In line with prior work on predicting binary user satisfaction, we use four different evaluation metrics: accuracy, precision, recall and F-score. Table 4 presents detailed results comparing the Global, Per-Intent and Multi-level Model on these four metrics. We additionally divide the overall prediction performances and separately consider the performance in predicting satisfaction and dissatisfaction, since understanding them (SAT and DSAT) independently is useful in different use-case scenarios and at different stages of system development.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
We observe that the Global Model is not able to predict user satisfaction much better than random, giving an overall accuracy of 57%. The Global Model without intent as a feature (i.e. Global w/o intent) performs worse, which highlights that incorporating intent information is useful in predicting satisfaction. The Per-Intent Model, on the other hand, gives much better prediction results than the Global Model. We observe an increased performance across all eight intents, with performance gains ranging from 10% to 24% for different intents. This confirms the main hypothesis of our workconsidering intent information is crucial in accurately understanding and predicting user satisfaction. Since the interaction signals differ across different intents, the Global Model is unable to find a good universal relation between such signals and satisfaction. The Per-Intent Model creates a separate local intent-specific model and is able to capture the interplay between interaction signals and satisfaction to a much better extent.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
We observe performance improvements in not just the accuracy, but across all metrics considered for both predicting satisfaction and predicting user dissatisfaction. Furthermore, we observe that the Multi-level Model performs significantly better than the Global Model (with intent) with over 20% improvement in prediction accuracy over the Global Model. Additionally, the Multi-level Model outperforms all but one Per-Intent Model, with performance improvement ranging from 4-14% in terms of prediction accuracy across different intents, while giving comparable performance to the last intent. This shows that the Multi-level Model is able to leverage insights from other intents, which in turn help boost the performance of other intents.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Additionally, we also observe that some intents are easier to predict than others, with intent 1 being the hardest to predict satisfaction for. Indeed, intent 1 sessions are of users who do not particularly have any intent in mind, and had a homepage session because it was the first page shown to them, so they could have any intent while interacting. This makes it harder for the system to predict satisfaction. It is important to note that an intent-specific model works best for certain intents (e.g. Intents 5 & 8) which suggests that the intent specific interaction data is powerful enough in itself, and doesn't gain much from shared learnign across intents. However, the shared learning across intents via the multi-level model is more generally beneficial to most intents and gives improved performance for 7 out of 8 intents. Moreover, maintaining separate intent level models is a demanding engineering task, while the shared multi-level model provides a sweet middle-ground whcih is relatively easier to manage and support from a large scale engineering perspective.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
We next investigate the different types of interaction signals considered and analyse their importance in prediction satisfaction. Table 5 presents accuracy results for different groups of interaction signals considered, for all three models. We omit displaying results for the Global -intent Model due to space constraints. We bold the best  performing features for each prediction model. We observe that whereas temporal signals are more important than other types of signals for most intents, surface signals such as number of slates interacted with and depth are important for Intent 4 (play music matching mood). This highlights that different types of signals are important for different intents, which supports the analysis presented in Figure 3 that showed interaction signals to differ based on user intent. Additionally, we observe that the Downstream group of signals performs better for Intent 7 (save new music or follow playlist), than it does for other intents, with 6-10% better predictive performance. Indeed, downstream signals help capture user activity related to saving music and exploring artists in detail, which is what Intent 7 is focusing on. Finally, we observe a sharp increase in predictive performance when combining downstream and temporal signals, with over 10% performance jumps when compared to any individual signal group. We also observe that even in the case of different signal groups, Per-Intent and Multi-level models performs significantly better than the Global Model. Finally, combining different signals helps in improving satisfaction prediction accuracy by over 10%-15% for most intents.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Importance of signals across intents. To gain insights into which interaction signal is most useful across different intents, we present the top three signals and report their relative feature weight across all eight intents in Figure 6. As expected, we observe that the signals are important to varying degree across different intents. For the case where the user wants to quickly access saved music, signals like time to success and dwell time are most informative, while they are not informative for cases where the user wants to play music in the background. For intents where users wish to explore artists in more detail, signals involving the users building relationships (i.e. saved or downloaded tracks) are shown to be more important.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
The variation of the most informative signals across the different intents highlight the fact that different signals are indeed important to different extent for different use cases. These results clearly emphasize the need for modeling intent while interpreting user interactions for predicting user satisfaction.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Given the query-less paradigm of slate recommendation, it becomes non-trivial to understand user intents. Based on a mixedmethods approach composed of interviews, in-app survey and nonparametric clustering, we identified eight key user intents, and experimentally demonstrated the importance of explicitly considering these intents when predicting user satisfaction. Our results also indicate that different interaction signals are important to varying extent across intents. Furthermore, the significant improvement in prediction results advocate not only the need for grouping user sessions into intent groups for predicting satisfaction, but also for shared learning across all intents via the Multi-level Model. We contend that the methodology adopted in this work to identify user intents: interviews, in-app survey and quantitative modeling would provide firm starting grounds for researchers to think about user intents in other recommendation scenarios where it is prohibitively challenging to extract unspecified user intents.
Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations
Limitations & Future Work. While our findings are more generally applicable beyond just music streaming domain, a few interesting challenges remain, which serve as useful directions for future work. While we leveraged the intent labels provided by survey responders, these labels are not readily available at run time for user sessions. This motivates the need for developing an intent prediction module to predict the intent for each user session. The current study also operates under the assumption of each session having one intent, which may not always be correct, since user intents might change within a session. Development of intent boundary detection or intent switching prediction modules would help in eliminating this assumption. Additionally, the model would need to cater to intents changing over time, and across contexts. Finally, we did not incorporate user specific intricacies in our formulation, but we believe that user interaction is conditional on user groups, and differ user segments interact differently. Incorporating such user level idiosyncrasies would be a fruitful direction.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
Task-oriented (or goal-oriented) dialogue systems aim to accomplish a particular task (e.g. book a table, provide information) through natural language conversation with a user. The system's available actions are often described by a pre-defined domainspecific schema while relevant knowledge is retrieved from stuctured databases or APIs (Feng et al., 2022b;. As such, taskoriented dialogue systems are often limited on which actions can be taken and what information can be retrieved . To relax these * The work was done when the first author was an intern at Huawei Noah's Ark Lab.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
How do I apply for disability benefits?
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
You should apply for benefits as soon as you become disabled.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
Are you interested in applying now?
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
No, I don't think I am ready.  restrictions, some dialogue systems (also referred to as goal-oriented chatbots) adopt open-domain language that is by definition unconstrained by predefined actions (Feng et al., 2020), and dynamically extract any required knowledge from in-domain unstructured collections in the form of entity descriptions, FAQs, and documents. Access to external knowledge sources has also been shown to help dialogue systems generate more specific and informative responses, which helps with the "common response" problem Ren et al., 2020;Feng et al., 2021aFeng et al., , 2022aShi et al., 2022). Figure 1 shows an example of a task-oriented dialogue that exploits external unstructured knowledge sources. Given a history of previous dialogue turns, with each turn consisting of one user and system utterance, and access to in-domain unstructured knowledge sources (either a document collection or a set of candidate facts), the dialogue system needs to generate an appropriate system response for the current turn. Recent research Ren et al., 2020) tackles the task by decomposing it into two sub-tasks: to initially determine the relevant knowledge (if any) that needs to be extracted/selected from external resources, and to subsequently generate the response based on the selected knowledge and the dialogue history.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
When retrieving knowledge from unstructured sources, different sources may need to be accessed in different dialogue turns; this is to be expected in most conversation scenarios. In the example of Figure 1, the first turn is grounded on the first knowledge candidate, and subsequent turns are grounded on later candidates. If we consider that each knowledge source belongs to a different topic or domain (e.g. "how you apply", "publications", "application is denied" in our example), we can observe that as the knowledge selection shifts across sources during the course of the dialogue, a corresponding shift occurs between topics. Previous work has not actively exploited this, but we posit that attending the topic shifts in the dialogue history can provide signals that help distinguish relevant from irrelevant sources for knowledge selection, and that such topical information can help the model derive an importance weighting scheme over the dialogue history for better response generation.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
In this paper, we model topic shifts in selected knowledge sources to improve topic-aware knowledge selection and response generation in taskoriented dialogue, and propose "Topic-Aware Response Generation" (TARG), an end-to-end model for knowledge selection and response generation. Our approach incorporates multiple topic-aware attention mechanisms to derive the importance weighting scheme over previous utterances and knowledge sources, aiming for a better understanding of the dialogue history. In addition, TARG is built on top of recent breakthroughs in language representation learning by finetuning on the pretrained language model BART (Lewis et al., 2020).
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
We conduct extensive experiments with two taskoriented dialogue datasets, namely Doc2Dial (Feng et al., 2020) and DSTC9 . Our results indicate that TARG 1 is able to accurately select the appropriate knowledge source, and as a result generate more relevant and fluent responses, outperforming previous state-of-the-art by 3.2, 3.6, and 4.2 points in EM, F1 and BLEU-4 respectively on Doc2Dial, and performing comparably with previous work on DSTC9. Furthermore, we present an ablation study and a case study accompanied by analysis of the learned attention 1 The code is available at https://github.com/ huawei-noah/noah-research/tree/master/ NLP/TARG. mechanisms.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
As we mentioned in the introduction, our proposed approach (TARG) exploits topic-aware mechanisms to derive an importance weighting scheme over different utterances in the dialogue history, with the goal to better inform knowledge selection and response generation. For a brief overview of TARG, please consult Figure 2. The input in our task consists of the dialogue history of previous user and system utterances, and a set of external knowledge candidates (hereafter referred to as factoids for brevity). The goal is to generate the next system utterance in the dialogue, which may or may not be grounded in one of the factoids; some of the dialogue history utterances may also be grounded on factoids but not necessarily all of them are.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
Briefly, to generate the next turn's system utterance, TARG initially generates BART-based representations for every previous user and system utterance in the dialogue history, for every available factoid, and for both utterances' and factoids' corresponding topics. For each utterance / factoid pair, TARG extracts matching features by calculating feature interaction over their encoded representations. TARG subsequently weights the matching features by topic-aware attention mechanisms, and aggregates them in a tensor. Finally, a knowledge selection layer outputs a relevance score over factoids, and the decoder generates the system utterance based on the most relevant factoid's encoding.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
We use a BART encoder to generate representations for every utterance in the dialogue history (up to a maximum history length) and factoid in external knowledge. We similarly, but separately, generate representations for their corresponding topics. Our work assumes that the corresponding topic of factoids can be derived in some way from the available data, e.g. the topic can be interpreted as the title of the factoid's originating document or its annotated domain. While we do not explore the possibility in this paper, the topic could also potentially be inferred using topic modelling techniques. The topic of each utterances is considered the same as that of their corresponding factoids (if any). Since not all dialogue turns are necessarily grounded in external knowledge, in absence of a corresponding factoid, the topic is set to a generic "non-relevant" pseudo-topic. This process results in the semantics and topic of every utterance or factoid being represented explicitly by separate embeddings.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
Specifically, in order to generate the semantic embeddings s u and s k of every utterance and factoid respectively, the token sequence 
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
In the next step, TARG calculates feature interactions over the semantic embeddings to extract matching features, which are subsequently weighted by a number of topic-aware attention mechanisms. These attention mechanisms operate over the topic embeddings of utterances and factoids to calculate topic-aware utterance / factoid pair matching representations. The motivation is to incorporate a more flexible way to weight and aggregate matching features of different dialogue history utterances with topic-aware attention, so that the model learns to better attend over them.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
Specifically, we design three different types of topic-aware attention that are calculated between each topic embedding t i k , corresponding to the i-th factoid, and the topic embeddings of all utterances in dialogue history T u , as follows:
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
Dot Product. We concatenate the utterance topic embeddings t j u ∈ R H with the factoid topic embedding, and compute the dot product between parameter w d ∈ R 2H and the resulting vector:
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
Bilinear. We compute the bilinear interaction between t j u and t i k and then normalize the result:
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
Outer Product. We compute the outer product between t j u and t i k , then project this feature vector through a fully connected layer and a softmax:
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
where w o ∈ R H is a parameter and × is the outer product.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
In parallel, we calculate the feature interaction matrix F i ∈ R N ×H between the semantic embeddings of all utterances s j u and the factoid s i k . N is the number of dialogue utterances. Every row F i,j of F i is calculated as follows:
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
To obtain a unified utterance / factoid pair representation k i for each factoid i, we concatenate the weighted sums of all utterances / factoid interaction embeddings with the different attention mechanisms. The final topic-aware utterance / factoid pair representation across all factoids is K ∈ R 3H×M , where M is the number of factoids. The i-th column vector k i is calculated as follows:
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
For the purpose of knowledge selection, TARG treats all external knowledge as a single document, by simply concatenating all available factoids. To account for the possibility that the system response shouldn't be grounded on any external knowledge, a "non-relevant" pseudo-factoid is included. The relevant knowledge selector takes the topicaware representations of these sequential factoids as input and predicts a span over the overall document that the system response should be grounded on. Through this process, several knowledge candidates may appear in the selected span.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
The grounded span is derived by predicting the start and the end indices of the span in the document. We obtain the probability distribution of the start index and end index over the entire document by the following equations:
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
where W s , W e ∈ R 3H , b s , b e ∈ R M are trainable weight vectors.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
The system response generator decodes the response by attending on the selected knowledge span. Since the span may contain several factoids, we first use a Convolution Neural Network (CNN) to fuse the information. We apply this CNN even when only a single factoid is present in the span for consistency. The CNN receives the topic-aware utterance / factoid pair embeddings of the selected span, and outputs the fusion embedding f ∈ R H :
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
where s and e are the start and end indexes. We employ a BART decoder for the system response generator, which takes the fusion embedding f as its initial hidden state. At each decoding step t, the decoder receives the embedding of the previous item w t−1 ∈ R H , the previous hidden state h t−1 ∈ R H , and the topic-aware utterance / factoid pair embeddings of the selected span K s:e,: , and produces the current hidden state h t ∈ R H : h t = BART(w t−1 , h t−1 , K :,s:e ).
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
A linear transformation layer produces the generated word distribution p v over the vocabulary:
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
where V ∈ R L×H is the word embeddings of the vocabulary, L is the vocabulary size, and W v ∈ R H×H and b v ∈ R are transformation parameters.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
For each turn, our model selects the relevant knowledge and generates the current turn's response. We optimize the knowledge selector and response generator via their cross-entropy losses L s , L g :  where N is the number of samples, M is the number of dialogue turns, y s nm /y e nm and p s /p e respectively represent the ground truth and predicted start/end positions at m-th dialogue turn of sample n, D nm is the input dialogue context, K nm is the input knowledge, and Y nm is the ground truth system response at m-th dialogue turn of sample n. We compute the joint loss L as follows:
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
where λ ∈ [0, 1] is a balance coefficient. f
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
We evaluate our proposed approach on two benchmark data sets on task-oriented dialogue: Doc2Dial (Feng et al., 2020) and DSTC9 . Doc2Dial is a leaderboard dataset with a withheld test set used for ranking participating systems, which includes conversation dialogues between an assisting system and an end user, with an accompanying set of documents wherein distinct factoids are clearly annotated; further annotations indicate which dialogue utterances are grounded on which factoids of the associated documents. The Doc2Dial dataset includes many cases of conversations that are grounded on factoids from different documents. By considering the title of each document as a distinct topic, each of these conversations can be interpreted to involve many interconnected topics under a general inquiry, making it an ideal dataset for our approach. The DSTC9 dataset also includes conversation dialogues, but the external knowledge is in the form of FAQ documents, in essence containing question answering pairs on a specific domain; we consider each pair as a distinct factoid and their domain as the topic. In practice, these FAQs are to be used to answer follow-up user questions that are out of the coverage of a dialogue system's database. Similarly to Doc2Dial, the "topic" in the DSTC9 dataset is also varied thoughout the conversations.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
As mentioned before, we interpret the title of the factoid's originating document or its annotated domain as the topic of the factoid. However, this assumption would be reasonable only if the factoids are relatively short. Table 1 and Table 2 presents the statistics of the Doc2Dial and DSTC9 datasets, and we can observe that on average the knowledge factoids are indeed relatively short in both datasets.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
Information on the evaluation measures and implementation details can be found in the Appendix.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
In the following experiments, we compare our approach against previously published state-of-the-art approaches on the Doc2Dial and DSTC9 datasets. We have not re-implemented these approaches, but report their already published results for the datasets for which they are available.   RWTH (Daheim et al., 2021): They use a biaffine classifier to model spans, followed by an ensemble for knowledge selection, and a cascaded model that grounds the response prediction on the predicted span for response generation. Base-DSTC : The baseline provided by the DSTC9 challenge is a response generation model obtained by fine-tuning the GPT-2 (Budzianowski and Vulić, 2019) model with a standard language modeling objective. Base-DSTC-ST directly uses the topic of the previous turn as the topic of current turn. KDEAK (Chaudhary et al., 2021): A model which formulates knowledge selection as a factorized retrieval problem with three modules performing domain, entity and knowledge level analyses. The response is generated using a GPT-2 model attending on any relevant retrieved knowledge. RADGE (Tang et al., 2021): A multi-task method that exploits correlations between dialogue history and keywords extracted from the API through finetuning a sequence of ELECTRA models (Clark et al., 2020). EGR (Bae et al., 2021): An approach that uses relevance similarity to score factoids, and later reranks them with a rule-based algorithm based on entity names parsed from the dialogue. The response is generated with a BART model.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
Tables 3 and 4 show our results on Doc2Dial and DSTC9 respectively. Observe that TARG per-  forms significantly better than related work in both knowledge selection and response generation on the Doc2Dial dataset, outperforming the second best system by 3.2, 3.6, and 4.2 points in EM, F1 and BLEU-4 respectively.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
On the DSTC9 dataset, TARG outperforms the related work in most metrics, though by narrow margins. Due to the smaller differences, we consider TARG to be performing on par with stateof-the-art on DSTC9. The performance gains of TARG can be explained by the topic-aware mechanism as it provides a more flexible way to weight and aggregate different dialogue history turns. This indicates that better understanding of the dialogue history is crucial for predicting the relevant factoids and generating a reasonable response.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
The main difference between datasets is the frequency of topic shifts. The average number of topics per dialogue is 8.83 and 2.58 on Doc2Dial and DSTC9 respectively. This difference can be partially explained by how we infer each dataset's topic, e.g. since the topic in DSTC9 is the domain of each question-answer pair, and multiple pairs belong to the same domain, the topic shifts are considerably more limited than in the Doc2Dial dataset. We further examined how BLEU scores are effected if we isolate DSTC9 dialogues that have more than the average number of topics. Specifically, we evaluated TARG on DSTC9 dialogues which exclusively have 2, 3, and 4 topics, and the BLEU is 0.363, 0.372, and 0.378 respectively. This indicates that more topic shifts provide more signal for the model to exploit.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
An additional difference between the datasets is that the topic for each factoid in Doc2Dial can be considered to be fine-grained, e.g. "VA clothing allowance", "About your eligibility", and "How to get these benefits", while in the DSTC9 dataset, the topic for each factoid can be considered coarsegrained, e.g. "Restaurant", "Hotel", "Taxi", and "Train". These differences collectively show that the lower performance on DSTC9 is due to its coarse-grained topics, and the lower number of average topic shifts. This suggests that a further division of the documents on more fine-grained topics and introducing more topic shifts in DSTC9 dialogs would help TARG perform better. However, we cannot straightforwardly examine how these two improvements interact with each other, and leave such analysis for future work.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
Here we conduct an ablation study of TARG, to explore the effects of the BART model, topic-aware attention, as well as the different topic attention mechanisms. The results indicate that all these mechanisms are necessary to the performance of knowledge selection and response generation. Effect of BART: To investigate the effectiveness of using BART in the utterance / factoid encoder U3 U: I use them but wouldn't care to work on computer related things. Do you have any info for the parents to look at?
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
S3 S: Is this information for a parent that is planning ahead for a child's higher education?
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
U4 U: yes it is.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
S4 S: We have resources for parents to learn more about saving early, and finding tax breaks.  Figure 6: Visualization of learned topic-aware attention of dialogue history utterances U-X and S-X (for user and system utterance) for each topic T-X in the example in Figure 5. Lighter spots mean higher attention scores.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
and system response generator, we replace BART with a bi-directional LSTM and rerun the model for Doc2Dial and DSTC9. As shown in Figures 3 and  4, the performance of the BiLSTM-based model TARG-w/oBART decreases significantly in knowledge selection, and especially in response generation as is indicated by the drop in BLEU. As expected, this indicates that the BART model can create and utilize more accurate representations for dialogue history and unstructured knowledge.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
Effect of topic-aware attention: Next we remove the topic-aware attention mechanisms (TARGw/oAtt). Figures 3 and 4   leading to better understanding of dialogue history. Effect of topic attention mechanisms: Here we compare TARG against TARG-dot, TARG-bilinear, and TARG-outer which use exclusively doc product attention, bilinear attention, and outer product attention respectively. Table 5 shows that dot product attention underperforms compared to bilinear and outer product attention while bilinear attention's performance is comparable with outer product attention. In addition, any isolated attention mechanism performs considerably worse than their fusion, supporting its utilization. We conjecture that this is due to how different attention mechanisms focus on different topic features.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
To facilitate a better understanding of how topic shifts occur in our model, we present a case study from the Doc2Dial dataset. On the top of Figure 5 are the previous turns of dialogue history, while on the right is a subset of the available factoids.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
We can observe how the topic changes throughout the turns of dialogue history (by consulting the corresponding factoid topic), from "Exploring Your Career Options" in turns 1 and 2, to "Resources for Parents of Students" in turns 3 and 4, and finally "Preparing for College" in turn 5.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
On the bottom of Figure 5, we present responses generated by our proposed model TARG, the best of the previous work RWTH, the Doc2Dialbaseline, and the ground truth. Observing the responses and comparing with the ground truth, Doc2Dial-baseline seems to generate irrelevant response, picking the wrong topics from the candidates on the right, i.e. "Resources for Parents of Students". RWTH picks right topic, but it selects wrong factoid "Review our resources for parents" to generate response. TARG generates the more relevant and fluent response of the three, as its topic-aware attention informs knowledge selection to pick the topic and factoid that more naturally follows the dialogue history, i.e. "Reasons to Attend a College or Career School". Furthermore, TARG's BART decoder ensures the fluency of the output. Figure 6 presents a visualization of TARG's learned topic-aware attention over the dialogue utterances and topics of the case study. This includes Dot Product Attention, Bilinear Attention, and Outer Product Attention. We can see that topicaware attention captures reasonable dialogue utterance weights for each topic, with the weighing moving from topic T1 to T2 and to T3 as attentions are calculated over the dialogue history utterances. This supports our claim that modeling the topic shifts can be helpful for knowledge selection, and consequently response generation, through better understanding of the dialogue history.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
In this paper, we proposed TARG: "Topic-Aware Response Generation", a topic-aware model which incorporates multiple topic-aware attention mechanisms to derive the importance weighting scheme over both dialogue utterances and unstructured external knowledge, and through that facilitate better dialogue history understanding. Our proposed method achieves state-of-the-art results in both knowledge selection and response generation, outperforming previous state-of-the-art by 3.2, 3.6, and 4.2 points in EM, F1 and BLEU-4 respectively on Doc2Dial, and performing comparably with previous work on DSTC9. To provide further insights, we also presented an ablation study of our model that supported the importance of our method's various components, and discussed a case study accompanied by an analysis of the attention mechanisms.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
The main limitation of the proposed method is its reliance on annotated or easily inferrable topics in the external knowledge sources. Future work should explore how this method can be applied when such topics are absent, e.g. by inferring topics through Latent Dirichlet Analysis. Our analysis also shows that our method performs better when these topics are fine-grained and a large number of topic shifts are expected in the dialogue. A more technical limitation of our model is that due to the limited input context size of the pre-trained language model we used, its scalability to long dialogue context is difficult. Finally, due to data availability, we only conducted experiments on English dialogues. While little in our method should be affected by the limited morphology of the English language, our results should be confirmed to hold on more structurally complicated languages.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
We use a pre-trained BART-base model to encode utterances and factoids. The max sentence length is set to 50 and the max number of dialogue turns is set to 15. The hidden size of attentions are all set to 768. The size of the convolution and pooling kernels are set to (3, 3, 3). The joint loss λ is 0.5. The dropout probability is 0.1. The batch size is set to 8. We optimize with Adam and an initial learning rate of 3e-5.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
We make use of the following automatic evaluation metrics in our experiments. For each dataset, we calculate the metrics used by the respective challenges for consistency. Exact Match (EM): This measures what part of the predicted knowledge span matches the ground truth factoid exactly. Token-Level F1: We cast the predicted spans and ground truth factoids as bags of tokens, and compute F1 between them. MRR@5: A metric based on the rank of the first ground truth factoid in a system's top-5 ranking.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
Recall@5: This metric counts how many ground truth factoids occur in a system's top-5 ranking. BLEU-X (Papineni et al., 2002): BLEU-X estimates a generated response's via measuring its ngram precision against the ground truth. X denotes the maximum size of the considered n-grams (i.e. unigrams, bigrams, trigrams, 4-grams). ROUGE-X (Lin, 2004): ROUGE-X measures ngram recall between generated and ground truth response. ROUGE-L measures the longest common word subsequence.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
We further conduct an analysis on how the selected knowledge span differs from turn to turn as this also indicates a shift in topic. Table 6 shows the average number of knowledge span changes as observed in the grounded truth and in the predicted output of Base-D2D and TARG, on the Doc2Dial dataset. We can see that the knowledge span changes are frequent in the ground truth, and that TARG's average knowledge span changes is closer to that of the ground truth. This indicates that TARG can more accurately follow the knowledge span changes in the dataset than Base-D2D.
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
We further investigate the number of the selected factoids per turn in Doc2Dial, i.e. the average  number of factoids covered by the predicated spans. As shown in Table 6, we can again see that TARG's behavior is closer to that of the ground truth.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
Spoken language understanding (SLU) is a critical component in spoken dialogue systems. SLU is aiming to form a semantic frame that captures the semantics of user utterances or queries. It typically involves two tasks: intent detection and slot filling (Tur and De Mori, 2011). These two tasks focus on predicting speakers intent and extracting semantic concepts as constraints for the natural language. Take a movie-related utterance as an example, "find comedies by James Cameron", as shown in Figure 1. There are different slot labels for each word in the utterance, and a specific intent for the whole utterance.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
Slot filling can be treated as a sequence labeling task that maps an input word sequence x = (x 1 , · · · , x T ) to the corresponding slot label sequence y S = (y S 1 , · · · , y S T ), and intent detection can be seen as a classification problem to decide the intent label y I . Popular approaches for slot filling include conditional random fields (CRF) (Ray- 1 The code is available at: https://github.com/ MiuLab/SlotGated-SLU. W find comedies by james cameron mond and Riccardi, 2007) and recurrent neural network (RNN) (Yao et al., 2014), and different classification methods, such as support vector machine (SVM) and RNN, have been applied to intent prediction.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
Considering that pipelined approaches usually suffer from error propagation due to their independent models, the joint model for slot filling and intent detection has been proposed to improve sentence-level semantics via mutual enhancement between two tasks (Guo et al., 2014;Hakkani-Tür et al., 2016;. In addition, the attention mechanism (Bahdanau et al., 2014) was introduced and leveraged into the model in order to provide the precise focus, which allows the network to learn where to pay attention in the input sequence for each output label Lane, 2015, 2016). The attentional model proposed by Liu and Lane (2016) achieved the state-of-the-art performance for joint slot filling and intent prediction, where the parameters for slot filling and intent prediction are learned in a single model with a shared objective. However, the prior work did not "explicitly" model the relationships between the intent and slots; instead, it applied a joint loss function to "implicitly" consider both cues. Because the slots often highly depend on the intent, this work focuses on how to model the explicit relationships between slots and intent vectors by introducing a slot-gated mechanism. The contributions are three-fold: 1) the proposed slot-  gated approach achieves better performance than the attention-based models; 2) the experiments on two SLU datasets show the generalization and the effectiveness of the proposed slot gate; 3) the gating results help us analyze the slot-intent relations.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
This section first explains our attention-based RNN model and then introduces the proposed slot gate mechanism for joint slot filling and intent prediction. The model architecture is illustrated in Figure 2, where there are two different model. (a) is one with both slot attention and intent attention and (b) is another with only intent attention.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
The bidirectional long short-term memory (BLSTM) model (Mesnil et al., 2015) takes a word sequence x = (x 1 , . . . , x T ) as input, and then generates forward hidden state − → h i and backward hidden state
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
Slot Filling For slot filling, x is mapping to its corresponding slot label sequence y = (y S 1 , . . . , y S T ). For each hidden state h i , we compute the slot context vector c S i as the weighted sum of LSTM's hidden states, h 1 , ..., h T , by the learned attention weights α S i,j :
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
where the slot attention weights are computed as below.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
where σ is the activation function, and W S he is the weight matrix of a feed-forward neural network. Then the hidden state and the slot context vector are utilized for slot filling:
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
where y S i is the slot label of the i-th word in the input, and W S hy is the weight matrix. The slot attention is shown as the blue component in Figure 2(a).
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
The intent context vector c I can also be computed in the same manner as c S , but the intent detection part only takes the last hidden state of BLSTM. The intent prediction is modeled similarly:
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
This section describes the proposed slot-gated mechanism illustrated in the red part of Figure 2. The proposed slot-gated model introduces an additional gate that leverages intent context vector for modeling slot-intent relationships in order to improve slot filling performance. First, slot context vector c S i and intent context vector c I are combined (c I broadcasts in time dimension to have the tanh Figure 3: Illustration of the slot gate. same shape with c S i ) to pass through a slot gate illustrated in Figure 3:
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
where v and W are trainable vector and matrix respectively. The summation is done over elements in one time step. g can be seen as a weighted feature of the joint context vector (c S i and c I ). We use g to weight between h i and c S i to derive y S i and replace (4) as below:
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
A larger g indicates that the slot context vector and the intent context vector pay attention to the same part of the input sequence, which also infers that the correlation between the slot and the intent is stronger and the context vector is more "reliable" for contributing the prediction results.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
To compare the power of the slot gate with attention mechanism, we also propose a slot-gated model with only intent attention in which (6) and (7) are reformed as (8) and (9) respectively (shown in Figure 2
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
This version allows the slots and intent to share the attention mechanism.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
To obtain both slot filling and intent prediction jointly, the objective is formulated as 
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
where p(y S , y I | x) is the conditional probability of the understanding result (slot filling and intent prediction) given the input word sequence and is maximized for SLU.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
To evaluate the proposed model, we conduct experiments on the benchmark datasets, ATIS (Airline Travel Information System) and Snips. The statistics are shown in Table 1.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
The ATIS (Airline Travel Information Systems) dataset (Tur et al., 2010) is widely used in SLU research. The dataset contains audio recordings of people making flight reservations. The training set contains 4,478 utterances and the test set contains 893 utterances. We use another 500 utterances for development set. There are 120 slot labels and 21 intent types in the training set.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
To justify the generalization of the proposed model, we use another NLU dataset customintent-engines 2 collected by Snips for model evaluation. This dataset is collected from the Snips personal voice assistant, where the number of samples for each intent is approximately the same. The training set contains 13,084 utterances and the test set contains 700 utterances. We use another 700 utterances as the development set. There are 72 slot labels and 7 intent types.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
Compared to single-domain ATIS dataset, Snips is more complicated mainly due to the intent diver-   sity and large vocabulary. Table 2 shows the intents and associated utterance examples. Regarding the intent diversity, for example, GetWeather and BookRestaurant in Snips are from different topics, resulting larger vocabulary. In the other hand, intents in ATIS are all about flight information with similar vocabularies across them. Moreover, intents in ATIS are highly unbalanced, where atis flight accounts for about 74% of training data while atis cheapest appears only once. The comparison between two datasets can be found in Table 1.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
In all experiments, we set the size of hidden vectors to 64, the optimizer is adam, the reported numbers are averaged over 20 runs, and the maximum epoch is set to 10 and 20 on ATIS and Snips respectively with an early-stop strategy.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
We evaluate the SLU performance about slot filling using F1 score, intent prediction using accuracy, and sentence-level semantic frame parsing using whole frame accuracy. The experimental results are shown in Table 3, where the compared baselines for joint slot filling and intent prediction include the state-of-the-art sequence-based joint model using bidirectional LSTM (Hakkani-Tür et al., 2016) and attention-based model (Liu and Lane, 2016). We validate the performance improvement with statistical significance test for all experiments, where single-tailed t-test is performed to measure whether the results from the proposed model are significant better than ones from baselines. The numbers with star markers indicate that the improvement is significant with p < 0.05. Table 3 shows that the proposed slot-gated mechanism with full attention significantly outperforms the baselines for both datasets, where almost all tasks (slot filling, intent prediction, and semantic frame) obtain the improvement, demonstrating that explicitly modeling strong relationships between slots and intent can benefit SLU effectively. In ATIS dataset, the proposed slot-gated model with only intent attention achieves slightly better performance with fewer parameters (from 284K to 251K). However, it does not achieve better results in Snips dataset. Considering different complexity of these datasets, the probable reason is that a simpler SLU task, such as ATIS, does not require additional slot attention to achieve good results, and the slot gate is capable of providing enough cues for slot filling. On the other hand, Snips is more complex, so that the slot attention is needed in order to model slot filling better (as well as the semantic frame results).
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
It is obvious that our proposed model performs better especially on sentence-level semantic frame results, where the relative improvement is around 4.1% and 1.9% for ATIS and Snips respectively.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
It may credit to the proposed slot gate that learns the slot-intent relations to provide helpful information for global optimization of the joint model. In sum, for joint slot filling and intent prediction, the experiments show that leveraging explicit slotintent relations controlled by the slot-gated mechanism can effectively achieve better sentence-level semantic frame performance due to global consideration.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
This paper focuses on learning the explicit slotintent relations by introducing a slot-gated mechanism into the state-of-the-art attention model, which allows the slot filling can be conditioned on the learned intent result in order to achieve better SLU (joint slot filling and intent detection). The experiments show that the proposed approach outperforms the baselines and can be generalized to different datasets. Also, the slot-gated model is more useful for a simple understanding task, because the slot-intent relations are stronger and easily modeled, and this paper provides the guidance of model design for future SLU work.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Virtual assistants help users accomplish tasks including but not limited to finding flights, booking restaurants and, more recently, navigating user interfaces, by providing a natural language interface to services and APIs on the web. The recent popularity of conversational interfaces and the advent of frameworks like Actions on Google and Alexa Skills, which allow developers to easily add support for new services, has resulted in a major increase in the number of application domains and individual services that assistants need to support, following the pattern of smartphone applications.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Consequently, recent work has focused on scalable dialogue systems that can handle tasks across multiple application domains. Data-driven deep learning based approaches Copyright c 2020, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
for multi-domain modeling have shown promise, both for end-to-end and modular systems involving dialogue state tracking and policy learning. This line of work has been facilitated by the release of multi-domain dialogue corpora such as MultiWOZ (Budzianowski et al. 2018), M2M (Shah et al. 2018) and FRAMES (El Asri et al. 2017).
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
However, existing datasets for multi-domain task-oriented dialogue do not sufficiently capture a number of challenges that arise with scaling virtual assistants in production. These assistants need to support a large (Kim et al. 2018), constantly increasing number of services over a large number of domains. In comparison, existing public datasets cover few domains. Furthermore, they define a single static API per domain, whereas multiple services with overlapping functionality, but heterogeneous interfaces, exist in the real world.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
To highlight these challenges, we introduce the Schema-Guided Dialogue (SGD) dataset 1 , which is, to the best of our knowledge, the largest public task-oriented dialogue corpus. It exceeds existing corpora in scale, with over 16000 dialogues in the training set spanning 26 services belonging to 16 domains (more details in Table 1). Further, to adequately test the models' ability to generalize in zero-shot settings, the evaluation sets contain unseen services and domains. The dataset is designed to serve as an effective testbed for intent prediction, slot filling, state tracking and language generation, among other tasks in large-scale virtual assistants.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
We also propose the schema-guided paradigm for taskoriented dialogue, advocating building a single unified dialogue model for all services and APIs. Using a service's schema as input, the model would make predictions over this dynamic set of intents and slots present in the schema. This setting enables effective sharing of knowledge among all services, by relating semantically similar concepts across APIs, and allows the model to handle unseen services and APIs. Under the proposed paradigm, we present a novel architecture for multi-domain dialogue state tracking. By using large pre-trained models like BERT (Devlin et al. 2019), our model can generalize to unseen services and is robust to API changes, while achieving competitive results on the original and updated MultiWOZ datasets (Eric et al. 2019). 
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
An important goal of this work is to create a benchmark dataset highlighting the challenges associated with building large-scale virtual assistants. Table 1 compares our dataset with other public datasets. Our Schema-Guided Dialogue (SGD) dataset exceeds other datasets in most of the metrics at scale. The especially larger number of domains, slots, and slot values, and the presence of multiple services per domain, are representative of these scale-related challenges. Furthermore, our evaluation sets contain many services, and consequently slots, which are not present in the training set, to help evaluate model performance on unseen services.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
The 20 domains present across the train, dev and test splits are listed in Table 2. We create synthetic implementations of a total of 45 services or APIs over these domains. Our simulator framework interacts with these services to generate dialogue outlines, which are a structured representation of dialogue semantics. We then used a crowd-sourcing procedure to paraphrase these outlines to natural language utterances. Our novel crowd-sourcing procedure preserves all annotations obtained from the simulator and does not require any extra annotations after dialogue collection. In this section, we describe these steps in detail and then present analyses of the collected dataset.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
We define the schema for a service as a combination of intents and slots with additional constraints, with an example in Figure 1. We implement all services using a SQL engine. For constructing the underlying tables, we sample a set of entities from Freebase and obtain the values for slots defined in the schema from the appropriate attribute in Freebase. We decided to use Freebase to sample real-world entities instead of synthetic ones since entity attributes are often correlated (e.g, a restaurant's name is indicative of the cuisine served). Some slots like event dates/times and available ticket counts, which are not present in Freebase, are synthetically sampled.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
To reflect the constraints present in real-world services and APIs, we impose a few other restrictions. First, our dataset does not expose the set of all possible slot values  for some slots. Having such a list is impractical for slots like date or time because they have infinitely many possible values or for slots like movie or song names, for which new values are periodically added. Our dataset specifically identifies such slots as non-categorical and does not provide a set of all possible values for these. We also ensure that the evaluation sets have a considerable fraction of slot values not present in the training set to evaluate the models in the presence of new values. Some slots like gender, number of people, day of the week etc. are defined as categorical and we specify the set of all possible values taken by them. However, these values are not assumed to be consistent across services. E.g., different services may use ('male', 'female'), ('M', 'F') or ('he', 'she') as possible values for gender slot. Second, real-world services can only be invoked with a limited number of slot combinations: e.g. restaurant reservation APIs do not let the user search for restaurants by date without specifying a location. However, existing datasets simplistically allow service calls with any given combination of slot values, thus giving rise to flows unsupported by actual services or APIs. As in Figure 1, the different service calls supported by a service are listed as intents. Each intent specifies a set of required slots and the system is not allowed to call this intent without specifying values for these required slots. Each intent also lists a set of optional slots with default values, which the user can override.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
The dialogue simulator interacts with the services to generate dialogue outlines. Figure 2 shows the overall architecture of our dialogue simulator framework. It consists of two agents playing the roles of the user and the system. Both agents interact with each other using a finite set of actions specified through dialogue acts over a probabilistic automaton designed to capture varied dialogue trajectories. These dialogue acts can take a slot or a slot-value pair as argument. Figure 4b shows all dialogue acts supported by the agents.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
At the start of a conversation, the user agent is seeded with a scenario, which is a sequence of intents to be ful- filled. We identified over 200 distinct scenarios for the training set, each comprising up to 5 intents. For multi-domain dialogues, we also identify combinations of slots whose values may be transferred when switching intents e.g. the 'address' slot value in a restaurant service could be transferred to the 'destination' slot for a taxi service invoked right after.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
The user agent then generates the dialogue acts to be output in the next turn. It may retrieve arguments i.e. slot values for some of the generated acts by accessing either the service schema or the SQL backend. The acts, combined with the respective parameters yield the corresponding user actions. Next, the system agent generates the next set of actions using a similar procedure. Unlike the user agent, however, the system agent has restricted access to the services (denoted by dashed line), e.g. it can only query the services by supplying values for all required slots for some service call. This helps us ensure that all generated flows are valid.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
After an intent is fulfilled through a series of user and system actions, the user agent queries the scenario to proceed to the next intent. Alternatively, the system may suggest related intents e.g. reserving a table after searching for a restaurant. The simulator also allows for multiple intents to be active during a given turn. While we skip many implementation details for brevity, it is worth noting that we do not include any domain-specific constraints in the simulation automaton. All domain-specific constraints are encoded in the schema and scenario, allowing us to conveniently use the simulator across a wide variety of domains and services.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
The dialogue paraphrasing framework converts the outlines generated by the simulator into a natural conversation. Figure 3a shows a snippet of the dialogue outline generated by the simulator, containing a sequence of user and system actions. The slot values present in these actions are in a canonical form because they obtained directly from the service. However, users may refer to these values in various differ- Figure 2: The overall architecture of the dialogue simulation framework for generating dialogue outlines. ent ways during the conversation, e.g., "los angeles" may be referred to as "LA" or "LAX". To introduce these natural variations in the slot values, we replace different slot values with a randomly selected variation (kept consistent across user turns in a dialogue) as shown in Figure 3b.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Next we define a set of action templates for converting each action into a utterance. A few examples of such templates are shown below. These templates are used to convert each action into a natural language utterance, and the resulting utterances for the different actions in a turn are concatenated together as shown in Figure 3c. The dialogue transformed by these steps is then sent to the crowd workers. One crowd worker is tasked with paraphrasing all utterances of a dialogue to ensure naturalness and coherence.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
REQUEST(location) → Which city are you in?
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
In our paraphrasing task, the crowd workers are instructed to exactly repeat the slot values in their paraphrases. This not only helps us verify the correctness of the paraphrases, but also lets us automatically obtain slot spans in the generated utterances by string search. This automatic slot span generation greatly reduced the annotation effort required, with little impact on dialogue naturalness, thus allowing us to collect more data with the same resources. Furthermore, it is important to note that this entire procedure preserves all other annotations obtained from the simulator including the dialogue state. Hence, no further annotation is needed.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
With over 16000 dialogues in the training set, the Schema-Guided Dialogue dataset is the largest publicly available annotated task-oriented dialogue dataset. The annotations include the active intents and dialogue states for each user utterance and the system actions for every system utterance.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
We have a few other annotations like the user actions but we withhold them from the public release. These annotations enable our dataset to be used as benchmark for tasks like intent detection, dialogue state tracking, imitation learning of dialogue policy, dialogue act to text generation etc. The schemas contain semantic information about the APIs and the constituent intents and slots, in the form of natural language descriptions and other details (example in Figure 1).
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
The single-domain dialogues in our dataset contain an average of 15.3 turns, whereas the multi-domain ones contain 23 turns on an average. These numbers are also reflected in Figure 4a showing the histogram of dialogue lengths on the training set. Table 2 shows the distribution of dialogues across the different domains. We note that distribution of dialogues across the domains and services covered is largely balanced, with the exception domains which are not present in the training set. Figure 4b shows the frequency of dialogue acts contained in the dataset. Note that all dialogue acts except INFORM, REQUEST and GOODBYE are specific to either the user or the system.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Virtual assistants aim to support a large number of services available on the web. One possible approach is to define a large unified schema for the assistant, to which different service providers can integrate with. However, it is difficult to come up with a common schema covering all use cases. Having a common schema also complicates integration of tail services with limited developer support. We propose the schema-guided approach as an alternative to allow easy integration of new services and APIs.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Under our proposed approach, each service provides a schema listing the supported slots and intents along with their natural language descriptions (Figure 1 shows an example). These descriptions are used to obtain a semantic representation of these schema elements. The assistant employs a single unified model containing no domain or service specific parameters to make predictions conditioned on these schema elements. For example, Figure 5 shows how dialogue state representation for the same dialogue can vary for two different services. Here, the departure and arrival cities are captured by analogously functioning but differently named slots in both schemas. Furthermore, values for the number stops and direct only slots highlight idiosyncrasies between services interpreting the same concept. Using a single model facilitates representation and transfer of common knowledge across related services. Since the model utilizes semantic representation of schema elements  as input, it can interface with unseen services or APIs on which it has not been trained. It is also robust to changes like addition of new intents or slots to the service.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Models in the schema-guided setting can condition on the pertinent services' schemas using descriptions of intents and slots. These models, however, also need access to representations for potentially unseen inputs from new services. Recent pretrained models like ELMo (Peters et al. 2018) and BERT (Devlin et al. 2019) can help, since they are trained on very large corpora. Building upon these, we present a simple prototype model for zero-shot schema-guided DST.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
We use a single model 2 , shared among all services and domains, to make these predictions. We first encode all the intents, slots and slot values for categorical slots present in the schema into an embedded representation. Since different schemas can have differing numbers of intents or slots, predictions are made over dynamic sets of schema elements by conditioning them on the corresponding schema embeddings. This is in contrast to existing models which make predictions over a static schema and are hence unable to share knowledge across domains and services. They are also not robust to changes in schema and require the model to be retrained with new annotated data upon addition of a new intent, slot, or in some cases, a slot value to a service.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Schema Embedding This component obtains the embedded representations of intents, slots and categorical slot val-2 Our model code is available at github.com/googleresearch/google-research/tree/master/schema guided dst ues in each service schema. Table 3 shows the sequence pairs used for embedding each schema element. These sequence pairs are fed to a pretrained BERT encoder shown in Figure 6 and the output u CLS is used as the schema embedding.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Sequence 2 Intent service description intent description Slot service description slot description Value slot description value Table 3: Input sequences for the pretrained BERT model to obtain embeddings of different schema elements.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
For a given service with I intents and S slots, let {i j }, 1 ≤ j ≤ I and {s j }, 1 ≤ j ≤ S be the embeddings of all intents and slots respectively. As a special case, we let {s n j }, 1 ≤ j ≤ N ≤ S denote the embeddings for the N noncategorical slots in the service. Also, let {v k j }, 1 ≤ j ≤ V k denote the embeddings for all possible values taken by the k th categorical slot, 1 ≤ k ≤ C, with C being the number of categorical slots and N + C = S. All these embeddings are collectively called schema embeddings. (2019), we use BERT to encode the user utterance and the preceding system utterance to obtain utterance pair embedding u = u CLS and token level representations t 1 , t 2 · · · t M , M being the total number of tokens in the two utterances. The utterance and schema embeddings are used together to obtain model predictions using a set of projections (defined below).
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Projection Let x, y ∈ R d . For a task K, we define l = F K (x, y, p) as a projection transforming x and y into the vector l ∈ R p using Equations 1-3. Here, h 1 , h 2 ∈ R d , W K i and b K i for 1 ≤ i ≤ 3 are trainable parameters of suitable dimensions and A is the activation function. We use gelu (Hendrycks and Gimpel 2016) activation as in BERT.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Active Intent For a given service, the active intent denotes the intent requested by the user and currently being fulfilled by the system. It takes the value "NONE" if no intent for the service is currently being processed. Let i 0 be a trainable parameter in R d for the "NONE" intent. We define the intent network as below.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
The logits l j int are normalized using softmax to yield a distribution over all I intents and the "NONE" intent. During inference, we predict the highest probability intent as active.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Requested Slots These are the slots whose values are requested by the user in the current utterance. Projection F req predicts logit l j req for the j th slot. Obtained logits are normalized using sigmoid to get a score in [0, 1]. During inference, all slots with score > 0.5 are predicted as requested. Figure 5: The predicted dialogue state (shown with dashed edges) for the first two user turns for an example dialogue, showing the active intent and slot assignments, with two related annotation schemas. Note that the dialogue state representation is conditioned on the schema under consideration, which is provided as input, as are the user and system utterances. Figure 6: BERT encoder, taking in two sequences p and q as input and outputs an embedded sequence pair representation u CLS and token level representations {t 1 · · · t n+m }. We use BERT to obtain schema element embeddings and encode system and user utterances for dialogue state tracking.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
We define the user goal as the user constraints specified over the dialogue context till the current user utterance. Instead of predicting the entire user goal after each user utterance, we predict the difference between the user goal for the current turn and preceding user turn. During inference, the predicted user goal updates are accumulated to yield the predicted user goal. We predict the user goal updates in two stages. First, for each slot, a distribution of size 3 denoting the slot status and taking values none, dontcare and active is obtained by normalizing the logits obtained in equation 6 using softmax. If the status of a slot is predicted to be none, its assigned value is assumed to be unchanged. If the prediction is dontcare, then the special dontcare value is assigned to it. Otherwise, a slot value is predicted and assigned to it in the second stage.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
In the second stage, equation 7 is used to obtain a logit for each value taken by each categorical slot. Logits for a given categorical slot are normalized using softmax to get a distribution over all possible values. The value with the maximum mass is assigned to the slot. For each non-categorical slot, logits obtained using equations 8 and 9 are normalized using softmax to yield two distributions over all tokens. These two distributions respectively correspond to the start and end index of the span corresponding to the slot. The indices p ≤ q maximizing start[p] + end[q] are predicted to be the span boundary and the corresponding value is assigned to the slot.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
We consider the following metrics for evaluation of the dialogue state tracking task:
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
1. Active Intent Accuracy: The fraction of user turns for which the active intent has been correctly predicted.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
2. Requested Slot F1: The macro-averaged F1 score for requested slots over all eligible turns. Turns with no requested slots in ground truth and predictions are skipped.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
3. Average Goal Accuracy: For each turn, we predict a single value for each slot present in the dialogue state. The slots which have a non-empty assignment in the ground truth dialogue state are considered for accuracy. This is the average accuracy of predicting the value of a slot correctly. A fuzzy matching score is used for non-categorical slots to reward partial matches with the ground truth.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
4. Joint Goal Accuracy: This is the average accuracy of predicting all slot assignments for a turn correctly. For non-categorical slots a fuzzy matching score is used.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Performance on other datasets We evaluate our model on public datasets WOZ2.0 and MultiWOZ 2.1 (Eric et al. 2019). As results in Table 4 show, our model performs competitively on these datasets. In these experiments, we omit the use of fuzzy matching scores and use exact match while calculating the goal accuracies to keep our numbers comparable with other works. Furthermore, for the MultiWOZ 2.1 dataset, we also trained a model incorporating pointergenerator style copying for non-categorical slots, similar to Wu et al. (2019), giving us a joint goal accuracy of 0.489, exceeding the best-known result of 0.456 as reported in Eric et al. (2019). We omit the details of this model since it is not the main focus of this work.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Performance on SGD The model performs well for Active Intent Accuracy and Requested Slots F1 across both seen and unseen services, shown in Table 4. For joint goal and average goal accuracy, the model performs better on seen services compared to unseen ones (Figure 7). The main reason for this performance difference is a significantly higher OOV rate for slot values of unseen services.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Performance on different domains (SGD) The model performance also varies across various domains. The performance for the different domains is shown in Table 5. We observe that one of the major factors affecting the performance across domains is still the presence of the service in the training data (seen services). In most cases, the performance can be observed to degrade for domains with more unseen services. Among the unseen services, those in the 'Rental-Cars' and 'Buses' domain, have a very high OOV rate for slot values leading to worse performance. A low joint goal accuracy and high average goal accuracy for these two domains indicates a possible skew between the performance of different slots. Among seen services, 'RideSharing' domain also exhibits poor performance, since it possesses the largest number of the possible slot values across the dataset. We also notice that for categorical slots, with similar slot values (e.g. "Psychologist" and "Psychiatrist"), there is a very weak signal for the model to distinguish between the different classes, resulting in inferior performance.   Table 5: Model performance per domain (GA: goal accuracy). Domains marked with '*' are those for which the service in the test set is not present in the training set. Domains like Hotels marked with '**' has one unseen and one seen service. For other domains, the service in the test set was also seen in the training set. We see that the model generally performs better for domains containing services seen during training.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
It is often argued that simulation-based data collection does not yield natural dialogues or sufficient coverage, when compared to other approaches such as Wizard-of-Oz. We argue that simulation-based collection is a better alternative for collecting datasets like this owing to the factors below.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
• Fewer Annotation Errors: All annotations are automatically generated, so these errors are rare. In contrast, Eric et al. (2019) reported annotation errors in 40% of turns in MultiWOZ 2.0 which utilized a Wizard-of-Oz setup. • Simpler Task: The crowd worker task of paraphrasing a readable utterance for each turn is simple. The error-prone annotation task requiring skilled workers is not needed. Furthermore, Wizard-of-Oz style collection requires domain specific task definitions and instructions, making the collection of a diverse dataset like ours time consuming. • Low Cost: The simplicity of the crowd worker task and lack of an annotation task greatly cut data collection costs. • Better Coverage: A wide variety of dialogue flows can be collected and specific usecases can be targeted.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
To ensure naturalness of the generated conversations, we used the conversational flows present in other public datasets like MultiWOZ 2.0 and WOZ2.0 as a guideline while developing the dialogue simulator. It was difficult for us to conduct a side-by-side comparison with existing datasets since this is the first dataset to cover many new domains at such scale, but we plan to explore it in the future.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
We presented the Schema-Guided Dialogue dataset to encourage scalable modeling approaches for virtual assistants. We also introduced the schema-guided paradigm for taskoriented dialogue that simplifies the integration of new services and APIs with large scale virtual assistants. Building upon this paradigm, we present a simplistic model for zeroshot dialogue state tracking achieving competitive results.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
My buddy is getting married and I need to find a bus ticket. System I can help you with that. Can you give me a little more information like where you are traveling from, where you are traveling to, and when you would like to leave. User I need to go from Las Vegas to Sacramento on the 7th of this month. System Great, I was able to find 4 buss's matching that criteria. How does a bus with 1 transfer, leaving at 7:20 am, for $88 sound? User
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
That sounds like just what i'm looking for. I will also be needing a rental car until the 11th of this month. System Will this car be from Sacramento from March 7th. User I would like to pick up the car on the 8th around two in the afternoon. System The first of 5 cars I have found for you is a standard Accord, located at SMF International Airport, available for pickup on March 8th.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
That one sounds perfect. System Would you like me to proceed with making that reservation for you? User
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Yes, let's go ahead with that car. System Before I reserve this I would like to make sure all the details I have are correct. You would like to pickup a standard vehicle from SMF International Airport on March 8th at 2 pm, and will be returning the car on March 11th.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Yes, that is all correct but can you give me an estimated price too please? System I was successfully able to reserve that with a total cost of $105 for you.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Great, could you also help me find an online movie to watch? System What genre of movie are you looking for? User I am really into detective movies right now. System The 2 movies I found that you may like are Body Double and Citizen Kane. User I would really like to see Body Double now if possible, I heard it's pretty good. System Okay, so you would like me to begin playing the movie Body Double without subtitles now. Is that correct? User
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Yes, that would be great. System I've went ahead and started playing that movie for you.
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
Thanks so much! System Is there anything else I can help you with for now? User
Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
No, that's all for now. System Have a good day and enjoy your movie! Figure A.4: An example multi-domain dialogue from our dataset covering "Buses", "RentalCars" and "Media" domains. Figure A.5: Interface of the dialogue paraphrasing task where the crowd workers are asked to rephrase the dialogue outlines to a more natural expression. The actual interface shows the entire conversation, but only a few utterances have been shown in this figure. All non-categorical slot values are highlighted in blue. The task cannot be submitted unless all highlighted values in the outline are also present in the conversational dialogue.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Human evaluations of machine translation (MT) weigh many aspects of translation, including adequacy, fidelity , and fluency of the translation (Hovy, 1999;White and O'Connell, 1994). A comprehensive catalog of MT evaluation techniques and their rich literature is given by Reeder (2001). For the most part, these various human evaluation approaches are quite expensive (Hovy, 1999). Moreover, they can take weeks or months to finish. This is a big problem because developers of machine translation systems need to monitor the effect of daily changes to their systems in order to weed out bad ideas from good ideas. We believe that MT progress stems from evaluation and that there is a logjam of fruitful research ideas waiting to be released from the evaluation bottleneck. Developers would benefit from an inexpensive automatic evaluation that is quick, language-independent, and correlates highly with human evaluation. We propose such an evaluation method in this paper.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
How does one measure translation performance?
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
The closer a machine translation is to a professional human translation, the better it is. This is the central idea behind our proposal. To judge the quality of a machine translation, one measures its closeness to one or more reference human translations according to a numerical metric. Thus, our MT evaluation system requires two ingredients:
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
1. a numerical "translation closeness" metric 2. a corpus of good quality human reference translations
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
We fashion our closeness metric after the highly successful word error rate metric used by the speech recognition community, appropriately modified for multiple reference translations and allowing for legitimate differences in word choice and word order. The main idea is to use a weighted average of variable length phrase matches against the reference translations. This view gives rise to a family of metrics using various weighting schemes. We have selected a promising baseline metric from this family.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
In Section 2, we describe the baseline metric in detail. In Section 3, we evaluate the performance of BLEU. In Section 4, we describe a human evaluation experiment. In Section 5, we compare our baseline metric performance with human evaluations.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Typically, there are many "perfect" translations of a given source sentence. These translations may vary in word choice or in word order even when they use the same words. And yet humans can clearly distinguish a good translation from a bad one. For example, consider these two candidate translations of a Chinese source sentence: Example 1.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Candidate 1: It is a guide to action which ensures that the military always obeys the commands of the party. Candidate 2:
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
It is to insure the troops forever hearing the activity guidebook that party direct.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Although they appear to be on the same subject, they differ markedly in quality. For comparison, we provide three reference human translations of the same sentence below.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Reference 1: It is a guide to action that ensures that the military will forever heed Party commands.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
It is the guiding principle which guarantees the military forces always being under the command of the Party. Reference 3: It is the practical guide for the army always to heed the directions of the party.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
It is clear that the good translation, Candidate 1, shares many words and phrases with these three reference translations, while Candidate 2 does not. We will shortly quantify this notion of sharing in Section 2.1. But first observe that Candidate 1 shares "It is a guide to action" with Reference 1, "which" with Reference 2, "ensures that the military" with Reference 1, "always" with References 2 and 3, "commands" with Reference 1, and finally "of the party" with Reference 2 (all ignoring capitalization). In contrast, Candidate 2 exhibits far fewer matches, and their extent is less.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
It is clear that a program can rank Candidate 1 higher than Candidate 2 simply by comparing ngram matches between each candidate translation and the reference translations. Experiments over large collections of translations presented in Section 5 show that this ranking ability is a general phenomenon, and not an artifact of a few toy examples.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
The primary programming task for a BLEU implementor is to compare n-grams of the candidate with the n-grams of the reference translation and count the number of matches. These matches are positionindependent. The more the matches, the better the candidate translation is. For simplicity, we first focus on computing unigram matches.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
The cornerstone of our metric is the familiar precision measure. To compute precision, one simply counts up the number of candidate translation words (unigrams) which occur in any reference translation and then divides by the total number of words in the candidate translation. Unfortunately, MT systems can overgenerate "reasonable" words, resulting in improbable, but high-precision, translations like that of example 2 below. Intuitively the problem is clear: a reference word should be considered exhausted after a matching candidate word is identified. We formalize this intuition as the modified unigram precision. To compute this, one first counts the maximum number of times a word occurs in any single reference translation. Next, one clips the total count of each candidate word by its maximum reference count, 2 adds these clipped counts up, and divides by the total (unclipped) number of candidate words.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Candidate: the the the the the the the.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Reference 1: The cat is on the mat.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Reference 2: There is a cat on the mat.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
In Example 1, Candidate 1 achieves a modified unigram precision of 17/18; whereas Candidate 2 achieves a modified unigram precision of 8/14. Similarly, the modified unigram precision in Example 2 is 2/7, even though its standard unigram precision is 7/7.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Modified n-gram precision is computed similarly for any n: all candidate n-gram counts and their corresponding maximum reference counts are collected. The candidate counts are clipped by their corresponding reference maximum value, summed, and divided by the total number of candidate ngrams. In Example 1, Candidate 1 achieves a modified bigram precision of 10/17, whereas the lower quality Candidate 2 achieves a modified bigram precision of 1/13. In Example 2, the (implausible) candidate achieves a modified bigram precision of 0. This sort of modified n-gram precision scoring captures two aspects of translation: adequacy and fluency. A translation using the same words (1-grams) as in the references tends to satisfy adequacy. The longer n-gram matches account for fluency. 4
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
How do we compute modified n-gram precision on a multi-sentence test set? Although one typically evaluates MT systems on a corpus of entire documents, our basic unit of evaluation is the sentence. A source sentence may translate to many target sentences, in which case we abuse terminology and refer to the corresponding target sentences as a "sentence." We first compute the n-gram matches sentence by sentence. Next, we add the clipped n-gram counts for all the candidate sentences and divide by the number of candidate n-grams in the test corpus to compute a modified precision score, p n , for the entire test corpus.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
To verify that modified n-gram precision distinguishes between very good translations and bad translations, we computed the modified precision numbers on the output of a (good) human translator and a standard (poor) machine translation system using 4 reference translations for each of 127 source sentences. The average precision results are shown in Figure 1. The strong signal differentiating human (high precision) from machine (low precision) is striking. The difference becomes stronger as we go from unigram precision to 4-gram precision. It appears that any single n-gram precision score can distinguish between a good translation and a bad translation. To be useful, however, the metric must also reliably distinguish between translations that do not differ so greatly in quality. Furthermore, it must distinguish between two human translations of differing quality. This latter requirement ensures the continued validity of the metric as MT approaches human translation quality.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
To this end, we obtained a human translation by someone lacking native proficiency in both the source (Chinese) and the target language (English). For comparison, we acquired human translations of the same documents by a native English speaker. We also obtained machine translations by three commercial systems. These five "systems" -two humans and three machines -are scored against two reference professional human translations. The average modified n-gram precision results are shown in Figure 2.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Each of these n-gram statistics implies the same ranking: H2 (Human-2) is better than H1 (Human-1), and there is a big drop in quality between H1 and S3 (Machine/System-3). S3 appears better than S2 which in turn appears better than S1. Remarkably, this is the same rank order assigned to these "systems" by human judges, as we discuss later. While there seems to be ample signal in any single n-gram precision, it is more robust to combine all these signals into a single number metric.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
How should we combine the modified precisions for the various n-gram sizes? A weighted linear average of the modified precisions resulted in encouraging results for the 5 systems. However, as can be seen in Figure 2, the modified n-gram precision decays roughly exponentially with n: the modified unigram precision is much larger than the modified bigram precision which in turn is much bigger than the modified trigram precision. A reasonable averaging scheme must take this exponential decay into account; a weighted average of the logarithm of modified precisions satisifies this requirement.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
BLEU uses the average logarithm with uniform weights, which is equivalent to using the geometric mean of the modified n-gram precisions. 5 ,6 Experimentally, we obtain the best correlation with mono-5 The geometric average is harsh if any of the modified precisions vanish, but this should be an extremely rare event in test corpora of reasonable size (for N max ≤ 4). 6 Using the geometric average also yields slightly stronger correlation with human judgments than our best results using an arithmetic average. lingual human judgments using a maximum n-gram order of 4, although 3-grams and 5-grams give comparable results.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
A candidate translation should be neither too long nor too short, and an evaluation metric should enforce this. To some extent, the n-gram precision already accomplishes this. N-gram precision penalizes spurious words in the candidate that do not appear in any of the reference translations. Additionally, modified precision is penalized if a word occurs more frequently in a candidate translation than its maximum reference count. This rewards using a word as many times as warranted and penalizes using a word more times than it occurs in any of the references. However, modified n-gram precision alone fails to enforce the proper translation length, as is illustrated in the short, absurd example below.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Candidate: of the Reference 1: It is a guide to action that ensures that the military will forever heed Party commands.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
It is the guiding principle which guarantees the military forces always being under the command of the Party.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Reference 3: It is the practical guide for the army always to heed the directions of the party.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Because this candidate is so short compared to the proper length, one expects to find inflated precisions: the modified unigram precision is 2/2, and the modified bigram precision is 1/1.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Traditionally, precision has been paired with recall to overcome such length-related problems. However, BLEU considers multiple reference translations, each of which may use a different word choice to translate the same source word. Furthermore, a good candidate translation will only use (recall) one of these possible choices, but not all. Indeed, recalling all choices leads to a bad translation.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Here is an example.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Candidate 1: I always invariably perpetually do. Candidate 2: I always do. Reference 1: I always do. Reference 2: I invariably do. Reference 3: I perpetually do.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
The first candidate recalls more words from the references, but is obviously a poorer translation than the second candidate. Thus, naïve recall computed over the set of all reference words is not a good measure. Admittedly, one could align the reference translations to discover synonymous words and compute recall on concepts rather than words. But, given that reference translations vary in length and differ in word order and syntax, such a computation is complicated.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Candidate translations longer than their references are already penalized by the modified n-gram precision measure: there is no need to penalize them again. Consequently, we introduce a multiplicative brevity penalty factor. With this brevity penalty in place, a high-scoring candidate translation must now match the reference translations in length, in word choice, and in word order. Note that neither this brevity penalty nor the modified n-gram precision length effect directly considers the source length; instead, they consider the range of reference translation lengths in the target language.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
We wish to make the brevity penalty 1.0 when the candidate's length is the same as any reference translation's length. For example, if there are three references with lengths 12, 15, and 17 words and the candidate translation is a terse 12 words, we want the brevity penalty to be 1. We call the closest reference sentence length the "best match length."
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
One consideration remains: if we computed the brevity penalty sentence by sentence and averaged the penalties, then length deviations on short sentences would be punished harshly. Instead, we compute the brevity penalty over the entire corpus to allow some freedom at the sentence level. We first compute the test corpus' effective reference length, r, by summing the best match lengths for each candidate sentence in the corpus. We choose the brevity penalty to be a decaying exponential in r/c, where c is the total length of the candidate translation corpus.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
We take the geometric mean of the test corpus' modified precision scores and then multiply the result by an exponential brevity penalty factor. Currently, case folding is the only text normalization performed before computing the precision.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
We first compute the geometric average of the modified n-gram precisions, p n , using n-grams up to length N and positive weights w n summing to one.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Next, let c be the length of the candidate translation and r be the effective reference corpus length. We compute the brevity penalty BP,
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Then,
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
The ranking behavior is more immediately apparent in the log domain,
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
In our baseline, we use N = 4 and uniform weights w n = 1/N.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
The BLEU metric ranges from 0 to 1. Few translations will attain a score of 1 unless they are identical to a reference translation. For this reason, even a human translator will not necessarily score 1. It is important to note that the more reference translations per sentence there are, the higher the score is. Thus, one must be cautious making even "rough" comparisons on evaluations with different numbers of reference translations: on a test corpus of about 500 sentences (40 general news stories), a human translator scored 0.3468 against four references and scored 0.2571 against two references. Table 1 shows the BLEU scores of the 5 systems against two references on this test corpus. The MT systems S2 and S3 are very close in this metric. Hence, several questions arise:  • Is the difference in BLEU metric reliable?
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
• What is the variance of the BLEU score?
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
• If we were to pick another random set of 500 sentences, would we still judge S3 to be better than S2?
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
To answer these questions, we divided the test corpus into 20 blocks of 25 sentences each, and computed the BLEU metric on these blocks individually. We thus have 20 samples of the BLEU metric for each system. We computed the means, variances, and paired t-statistics which are displayed in Table  2. The t-statistic compares each system with its left neighbor in the table. For example, t = 6 for the pair S1 and S2. Note that the numbers in Table 1 are the BLEU metric on an aggregate of 500 sentences, but the means in Table 2 are averages of the BLEU metric on aggregates of 25 sentences. As expected, these two sets of results are close for each system and differ only by small finite block size effects. Since a paired t-statistic of 1.7 or above is 95% significant, the differences between the systems' scores are statistically very significant. The reported variance on 25-sentence blocks serves as an upper bound to the variance of sizeable test sets like the 500 sentence corpus.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
How many reference translations do we need? We simulated a single-reference test corpus by randomly selecting one of the 4 reference translations as the single reference for each of the 40 stories. In this way, we ensured a degree of stylistic variation. The systems maintain the same rank order as with multiple references. This outcome suggests that we may use a big test corpus with a single reference translation, provided that the translations are not all from the same translator.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
We had two groups of human judges. The first group, called the monolingual group, consisted of 10 native speakers of English. The second group, called the bilingual group, consisted of 10 native speakers of Chinese who had lived in the United States for the past several years. None of the human judges was a professional translator. The humans judged our 5 standard systems on a Chinese sentence subset extracted at random from our 500 sentence test corpus. We paired each source sentence with each of its 5 translations, for a total of 250 pairs of Chinese source and English translations. We prepared a web page with these translation pairs randomly ordered to disperse the five translations of each source sentence. All judges used this same webpage and saw the sentence pairs in the same order. They rated each translation from 1 (very bad) to 5 (very good). The monolingual group made their judgments based only on the translations' readability and fluency.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
As must be expected, some judges were more liberal than others. And some sentences were easier to translate than others. To account for the intrinsic difference between judges and the sentences, we compared each judge's rating for a sentence across systems. We performed four pairwise t-test comparisons between adjacent systems as ordered by their aggregate average score. Figure 3 shows the mean difference between the scores of two consecutive systems and the 95% confidence interval about the mean. We see that S2 is quite a bit better than S1 (by a mean opinion score difference of 0.326 on the 5-point scale), while S3 is judged a little better (by 0.114). Both differences are significant at the 95% level. 7 The human H1 is much better than the best system, though a bit worse than human H2. This is not surprising given that H1 is not a native speaker of either Chinese or English, whereas H2 is a native English speaker. Again, the difference between the human translators is significant beyond the 95% level.  Figure 4 shows the same results for the bilingual group. They also find that S3 is slightly better than S2 (at 95% confidence) though they judge that the human translations are much closer (indistinguishable at 95% confidence), suggesting that the bilinguals tended to focus more on adequacy than on fluency. Figure 4: Bilingual Judgments -pairwise differential comparison Figure 5 shows a linear regression of the monolingual group scores as a function of the BLEU score over two reference translations for the 5 systems. The high correlation coefficient of 0.99 indicates that BLEU tracks human judgment well. Particularly interesting is how well BLEU distinguishes between S2 and S3 which are quite close. Figure 6 shows the comparable regression results for the bilingual group. The correlation coefficient is 0.96. We now take the worst system as a reference point and compare the BLEU scores with the human judg-ment scores of the remaining systems relative to the worst system. We took the BLEU, monolingual group, and bilingual group scores for the 5 systems and linearly normalized them by their corresponding range (the maximum and minimum score across the 5 systems). The normalized scores are shown in Figure 7. This figure illustrates the high correlation between the BLEU score and the monolingual group. Of particular interest is the accuracy of BLEU's estimate of the small difference between S2 and S3 and the larger difference between S3 and H1. The figure also highlights the relatively large gap between MT systems and human translators. 8 In addition, we surmise that the bilingual group was very forgiving in judging H1 relative to H2 because the monolingual group found a rather large difference in the fluency of their translations. 
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
We believe that BLEU will accelerate the MT R&D cycle by allowing researchers to rapidly home in on effective modeling ideas. Our belief is reinforced by a recent statistical analysis of BLEU's correlation with human judgment for translation into English from four quite different languages (Arabic, Chinese, French, Spanish) representing 3 different language families (Papineni et al., 2002)! BLEU's strength is that it correlates highly with human judg-8 Crossing this chasm for Chinese-English translation appears to be a significant challenge for the current state-of-the-art systems. ments by averaging out individual sentence judgment errors over a test corpus rather than attempting to divine the exact human judgment for every sentence: quantity leads to quality.
IBM Research Report Bleu: a Method for Automatic Evaluation of Machine Translation
Finally, since MT and summarization can both be viewed as natural language generation from a textual context, we believe BLEU could be adapted to evaluating summarization or similar NLG tasks.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
Spoken dialogue systems (SDS) allow users to interact with computer applications through conversation. Task-oriented dialogue systems help users achieve goals such as finding restaurants or booking flights. The dialogue state tracking (DST) component of the SDS serves to interpret user input and update the belief state, which is the system's internal representation of the state of the conversation. This is a probability distribution over dialogue states used by the dialogue manager component to decide which action the system should perform next. The Dialogue State Tracking Challenge (DSTC) series of shared tasks has provided a common evaluation framework accompanied by labelled datasets (Williams et al., 2016). In this framework, systems must track the search constraints expressed by users (goals) and questions the users ask about search results (requests), taking into account the user's utterance (passed through a speech recogniser) and the dialogue context (e.g., what the system just said). The following example shows the true state after each user utterance in a three-turn conversation:
Neural Belief Tracker: Data-Driven Dialogue State Tracking
User: I'm looking for a cheap restaurant inform(price=cheap) System: How about Thai food? User: Yes please inform(price=cheap, food=Thai) System: The House serves cheap Thai food User: Where is it? inform(price=cheap, food=Thai); request(address) System: The House is at 106 Regent Street A task-based dialogue system is backed by an ontology which describes the range of user intents that the system can process and provides the building blocks of the belief state (i.e. price=cheap, address, etc.). DST systems depend on identifying mentions of ontology items in user utterances, which becomes a non-trivial task when confronted with lexical variation, noisy speech recognition and the effect of context. Some approaches assume that a separate Spoken Language Understanding (SLU) module will solve this problem for them. However, coupling SLU and DST in a single model has been shown to improve tracking performance (Henderson et al., 2014d). On the other hand, these coupled models often rely on manually constructed dictionaries to identify inexact mentions of ontology items; this is clearly not scalable to larger, more complex domains.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
In this paper, we present two new models, collectively called the Neural Belief Tracker (NBT) family. The NBT models couple SLU and DST, efficiently learning to handle lexical variation and input noise without requiring any hand-crafted resources. In evaluations on two datasets, we show that NBT models match the performance of state-of-the-art models that have greater resource requirements and outperform models with comparable task-specific requirements. Consequently, we believe this work proposes a framework better-suited to scaling belief tracking models for deployment in real-world dialogue systems operating over sophisticated application domains.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
Models for probabilistic dialogue state tracking, or belief tracking, were introduced as components of spoken dialogue systems in order to better handle noisy speech recognition output and other sources of uncertainty in understanding a user's goals (Bohus and Rudnicky, 2006;Williams and Young, 2007;Young et al., 2010). Modern dialogue management policies can learn to use a tracker's distribution over intentions to decide whether to execute an action or request clarification from the user. As mentioned above, the DSTC shared tasks have spurred research on this problem and institutionalised a standard evaluation paradigm (Williams et al., 2013;Henderson et al., 2014b;Henderson et al., 2014a). In this setting, the task is defined by an ontology that enumerates the goals a user can inform and the attributes of entities that the user can request information about. Many different belief tracking models have been proposed in the literature, from generative (Thomson and Young, 2010) and discriminative (Henderson et al., 2014d) statistical models to rule-based systems (Wang and Lemon, 2013). In order to motivate the work presented here, we can categorise prior research according to their reliance (or otherwise) on a separate SLU module for interpreting the user's utterances: 1 1 The best-performing models in DSTC2 all used both raw ASR output and the output of SLU decoders (Williams, 2014; Separate SLU: Traditional SDS pipelines use Spoken Language Understanding (SLU) decoders to detect slot-value pairs expressed in the Automated Speech Recognition (ASR) output. The downstream DST model then combines this information with the past dialogue context to update the belief state (Thomson and Young, 2010;Wang and Lemon, 2013;Lee and Kim, 2016;Perez, 2016;Sun et al., 2016). In the DSTC challenges, some systems used the output of template-based matching systems such as Phoenix (Wang, 1994). However, more robust and accurate statistical SLU systems are available. Many discriminative approaches to spoken dialogue SLU train independent binary models that decide whether each slot-value pair was expressed in the user utterance. Given enough data, these models can learn which lexical features are good indicators for a given value and can capture elements of paraphrasing (Mairesse et al., 2009). This line of work later shifted focus to robust handling of rich ASR output (Henderson et al., 2012;Tur et al., 2013). SLU has also been treated as a sequence labelling problem, where each word in an utterance is labelled according to its role in the user's intent; standard labelling models such as CRFs or Recurrent Neural Networks can then be used (Raymond and Ricardi, 2007;Yao et al., 2014;Celikyilmaz and Hakkani-Tur, 2015;Mesnil et al., 2015). Other approaches adopt a more complex modelling structure inspired by semantic parsing (Saleh et al., 2014;Vlachos and Clark, 2014). One drawback shared by these methods is their resource requirements, either because they need to learn independent parameters for each slot and value or because they need finegrained manual annotation at the word level. This hinders scaling to larger, more realistic domains.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
Joint SLU/DST: Research on belief tracking has found it advantageous to reason about SLU and DST jointly, taking ASR predictions as input and generating belief states as output (Henderson et al., 2014d;Sun et al., 2014;Zilka and Jurcicek, 2015). In DSTC2, systems with no external SLU component outperformed all systems that only used external SLU features. Joint models often rely on a strategy known as delexicalisation whereby slots and values mentioned in the text are replaced with generic labels.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
Once the dataset is transformed in this manner, one can extract a collection of template-like n-gram features such as [want tagged-value food]. To perform belief tracking, the shared model iterates over all slot-value pairs, extracting delexicalised feature vectors and making a separate binary decision regarding each pair. Delexicalisation introduces a hidden dependency that is rarely discussed: how do we identify slot/value mentions in text? For toy domains, one can manually construct semantic dictionaries which list the potential rephrasings for all slot values. As shown by , the use of such dictionaries is essential for the performance of current delexicalisation-based models. Again though, this will not scale to the rich variety of user language or to general domains.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
The primary motivation for the work presented in this paper is to address the limitations that affect previous belief tracking models: we learn efficiently from the available data by maximising the number of parameters that are shared across the ontology, while still giving the system the flexibility to learn paraphrasings and other kinds of variation that make it infeasible to rely on exact matching and delexicalisation as a robust strategy.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
The Neural Belief Tracker (NBT) is a model designed to detect the slot-value pairs expressed in any given user utterance during the flow of dialogue. Its input consists of the system dialogue acts preceding the user input, the user utterance itself, and a single slotvalue pair it needs to make a decision about. To do belief tracking, we iterate over all candidate pairs (defined by the domain ontology) and use the NBT to decide which ones have been expressed by the user. Figure 1 presents the flow of information in the model. The first layer in the NBT hierarchy performs representation learning for the three model inputs, producing vector representations for the user utterance (r), the current candidate slot-value pair (c) and the system dialogue acts (t q , t s , t v ). Subsequently, the learned vector representations interact through the context modelling and semantic decoding submodules to obtain the intermediate interaction summary vectors d r , d c and d. These are used as input to the final decision-making module which decides whether the user expressed the intent represented by the current candidate slot-value pair.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
For any given user utterance, system act(s) and candidate slot-value pair, the representation learning submodules produce vector representations which act as input for the downstream components of the model. All representation learning subcomponents make use of pre-trained collections of word vectors. As shown by , specialising word vectors to express semantic similarity rather than relatedness-by-association is essential for improving belief tracking performance. For this reason, we use the semantically-specialised Paragram-SL999 word vectors (Wieting et al., 2015) throughout this work. The NBT training procedure keeps these vectors fixed: that way, at test time, unseen words semantically related to familiar slot values (i.e. inexpensive to cheap) will be recognised purely by their position in the original vector space. This also means that candidate slot-value pairs implicitly parametrise the NBT model, allowing its parameters to be shared across all values of a slot, or even across all slots.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
We consider a user utterance u consisting of k u words u 1 , u 2 , . . . , u ku . Each word has an associated word vector u 1 , . . . , u ku . We propose two different models for producing a representation of u: NBT-DNN and NBT-CNN. Both act over the constituent ngrams of the utterance. Let v n i be the concatenation of the n word vectors starting at index i, so that:
Neural Belief Tracker: Data-Driven Dialogue State Tracking
where ⊕ denotes vector concatenation. The simpler version of the two models, which we term NBT-DNN, is shown in Figure 2. This model computes cumulative n-gram representation vectors r 1 , r 2 and r 3 , which are the n-gram 'summaries' of the unigrams, bigrams and trigrams in the user utterance:
Neural Belief Tracker: Data-Driven Dialogue State Tracking
Each of these vectors is then non-linearly mapped to intermediate representations of the same size:   where the weight matrices and bias terms map the cumulative n-grams to vectors of the same size and σ denotes the sigmoid activation function. Here we maintain a separate set of parameters for each slot (indicated by superscript s); we later investigate tying parameters across all slots (Section 6.2). The three vectors are summed to obtain a single representation for the user utterance:
Neural Belief Tracker: Data-Driven Dialogue State Tracking
The cumulative n-gram representations used by this model are just an unweighted sum of all word vectors in the utterance. Ideally, the model should learn to recognise which parts of the utterance are more relevant for the subsequent classification task. For instance, it could learn to ignore verbs or stop words and pay more attention to adjectives and nouns which are more likely to express slot values.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
Our second model, which we term NBT-CNN, draws inspiration from successful applications of Convolutional Neural Networks (CNNs) for language understanding (Collobert et al., 2011;Kalchbrenner et al., 2014;Kim, 2014). These models typically apply a number of convolutional filters to n-grams in the input sentence, followed by non-linear activation functions and max-pooling. Following this approach, the NBT-CNN model applies L = 300 different filters for n-gram lengths of 1, 2 and 3 (Figure 3). Let F s n ∈ R L×nD denote the collection of filters for each value of n, where D = 300 is the word vector dimensionality. If v n i denotes the concatenation of n word vectors starting at index i, let m n = [v n 1 ; v n 2 ; . . . ; v n ku−n+1 ] be the list of n-grams that convolutional filters of length n run over. The three intermediate representations are then given by:
Neural Belief Tracker: Data-Driven Dialogue State Tracking
Each column of the intermediate matrices R n is produced by a single convolutional filter of length n. We obtain summary n-gram representations by pushing these representations through a rectified linear unit (ReLu) activation function and max-pooling over time (i.e. columns) to get a single feature for each of the L filters applied to the utterance:
Neural Belief Tracker: Data-Driven Dialogue State Tracking
where b s n is a bias term broadcast across all filters. Finally, the three summary n-gram representations are summed to obtain the final utterance representation vector r (as in Equation 4). The NBT-CNN model is (by design) better suited to longer utterances, as its convolutional filters interact with subsequences of the utterance, and not just the noisy summaries given by the NBT-DNN's cumulative n-grams.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
The NBT diagram in Figure 1 shows that the utterance representation r and the candidate slot-value pair representation c directly interact through the semantic decoding module. This component decides whether the user explicitly expressed an intent matching the current candidate pair (i.e. without taking the dialogue context into account). Examples of such matches would be 'I want Thai food' with food=Thai or more demanding ones such as 'a pricey restaurant' with price=expensive. This is where the use of high-quality pre-trained word vectors comes into play: a delexicalisation-based model could deal with the former example but would be helpless in the latter case, unless a human expert had provided a semantic dictionary listing all potential rephrasings for each value in the domain ontology.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
Let the vector space representations of a candidate pair's slot name and value be given by c s and c v (with word vectors of multi-word slot names or values summed together). The NBT model learns to map this vector tuple into a single vector c of the same dimensionality as the utterance representation r. These two representations are then forced to interact in order to learn a similarity metric which discriminates between interactions of utterances with slot-value pairs that they either do or do not express:
Neural Belief Tracker: Data-Driven Dialogue State Tracking
where ⊗ denotes element-wise vector multiplication. The dot product, which may seem like the more intuitive similarity metric, would reduce the rich set of features in d to a single scalar. The element-wise multiplication allows the downstream network to make better use of its parameters by learning non-linear interactions between sets of features in r and c. This network (Binary Decision Maker in Figure 1) uses one intermediate hidden layer of size 100 to make the final decision about the current candidate pair.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
This 'decoder' does not yet suffice to extract intents from utterances in human-machine dialogue. To understand some queries, the belief tracker must be aware of context, i.e. the flow of dialogue leading up to the latest user utterance. While all previous system and user utterances are important, the most relevant one is the last system utterance, in which the dialogue system could have performed (among others) one of the following two system acts:
Neural Belief Tracker: Data-Driven Dialogue State Tracking
1. System Request: The system asks the user about the value of a specific slot T q . If the system utterance is: 'what price range would you like?' and the user answers with any, the model must infer the reference to price range, and not to other slots such as area or food type.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
The system asks the user to confirm whether a specific slot-value pair (T s , T v ) is part of their desired constraints. For example, if the user responds to 'how about Turkish food?' with 'yes', the model must be aware of the system act in order to correctly update the belief state.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
If we make the Markovian decision to only consider the last set of system acts, we can incorporate context modelling into the NBT model. Let t q and (t s , t v ) be the word vector representations of the arguments for the system request and confirm acts (zero vectors if none). The model computes the following measures of similarity between the system acts, candidate pair (c s , c v ) and utterance representation r:
Neural Belief Tracker: Data-Driven Dialogue State Tracking
where · denotes dot product. The computed similarity terms act as gating mechanisms which only pass the utterance representation through if the system asked about the current candidate slot or slot-value pair. This type of interaction is particularly useful for the confirm system act: if the system asks the user to confirm, the user is likely not to mention any slot values, but to just respond affirmatively or negatively. This means that the model must consider the three-way interaction between the utterance, candidate slot-value pair and the slot value pair offered by the system. If (and only if) the latter two are the same should the model consider the affirmative or negative polarity of the user utterance when making the subsequent binary decision. Finally, these two context modelling summary representations are passed to the decision making module, which combines them with the semantic decoding output to make the final decision.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
In spoken dialogue systems, belief tracking models operate over the output of automated speech recognition (ASR). Despite ever-improving speech recognition, the need to make the most out of imperfect ASR will persist as dialogue systems are used in increasingly noisy environments. In this work, we use a simple rule-based belief state update mechanism which can be applied to ASR N -best lists. Let u denote a list of N ASR hypotheses h i with posterior probabilities p i which follow system output sys. For any hypothesis h, slot s and slot value v ∈ V s , the NBT model estimates P(s, v | h, sys), i.e. the probability that (s, v) was expressed in h. The predictions for N such hypotheses are combined as:
Neural Belief Tracker: Data-Driven Dialogue State Tracking
For slot s, the set of its detected values is given by:
Neural Belief Tracker: Data-Driven Dialogue State Tracking
For informable slots, the value in V * s with the highest probability is chosen (if V * s = ∅). This goal value persists in subsequent turns until a new value is detected for this slot. For requests, all slots in V * req are deemed to have been requested. As requestable slots serve to model users' single-turn queries, they require no belief tracking across turns.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
Two datasets were used for training and evaluation. Both consist of user conversations with task-oriented dialogue systems designed to help users find suitable restaurants around Cambridge, UK. The two corpora share the same domain ontology, which contains three informable (i.e. goal) slots: food type, area and price. The users can specify values for these slots in order to find restaurants which best meet their criteria. Once the system suggests a restaurant, the users can ask about the values of up to eight requestable slots (phone number, address, etc.). The two datasets are:
Neural Belief Tracker: Data-Driven Dialogue State Tracking
1. DSTC2: we use the transcriptions, ASR hypotheses and turn-level semantic labels provided for the Dialogue State Tracking Challenge 2 (Henderson et al., 2014a). The official transcriptions contain various spelling errors which we corrected manually; the cleaned version of the dataset will be made available on the first author's website. The training data contains 2207 dialogues (15611 turns) and the test set consists of 1117 dialogues. We train the NBT models on transcriptions and report their belief tracking performance on test set ASR hypotheses.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
Oz style experiment in which Amazon Mechanical Turk users assumed the role of the system or the user of a task-oriented dialogue system based on the DSTC2 ontology. Users typed instead of using speech, which means performance in the WOZ experiments is more indicative of the model's capacity for semantic understanding than its robustness to ASR errors. The experimental design gave users more freedom to use sophisticated language, making understanding much more difficult. The WOZ dataset contains 536 training dialogues (2209 turns) and 140 test dialogues.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
The two corpora are used to synthesise training data for the NBT models. We iterate over all utterances, generating one example for each of the slot-value pairs in the DSTC2 ontology. Examples consist of a transcription, its context (i.e. list of system acts) and a candidate slot-value pair. The binary label for each example indicates whether or not its utterance and context express the example's candidate pair. To train the model, we used the Adam optimizer (Kingma and Ba, 2015) with a cross-entropy loss function, backpropagating through all the NBT subcomponents (but keeping the word vectors fixed). Both models were implemented using the Tensor-Flow framework (Abadi et al., 2015).   Table 1 shows the belief tracking performance of models trained and evaluated on DSTC2 data. We compare to a delexicalisation-based RNN model introduced by Henderson et al. (2014d;2014c). The baseline model uses no semantic dictionary, while the improved baseline uses a hand-crafted semantic dictionary designed for the DSTC2 ontology by Henderson et al. The NBT models clearly outperform the first baseline in terms of average goal accuracy but fare marginally worse on requests. The NBT models' goal tracking performance is comparable to the improved baseline. This shows that these models can handle semantic relations which otherwise had to be explicitly encoded in semantic dictionaries. As NBT's rule-based belief state update is less sophisticated than Henderson et al.'s RNN, we believe that further gains are possible.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
The WOZ dataset provides a more focused benchmark for assessing the semantic understanding capacity of belief tracking models. The baseline performance is a delexicalisation-based model incorpo- rating an RNN for belief state updates and a CNN for turn-level feature extraction . Again, the improved baseline uses a hand-crafted semantic dictionary. Table 2 shows the performance of NBT models trained on either DSTC2 or WOZ transcriptions. The NBT-CNN model outperforms the baselines across the board, even when it is trained using the (simpler) DSTC2 transcriptions. This shows that the CNN-based model is capable of handling long utterances and previously unseen vocabulary. Conversely, the NBT-DNN model falls short of both goal tracking baselines, as its cumulative n-gram representations cannot cope with some of the very long and complex WOZ utterances. Contrary to the results in Table 1, both NBT models trained on in-domain WOZ data match or exceed the baselines' performance over requestable slots.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
To train a single model which can be applied to all slots in the ontology (as opposed to using a separate model for each slot), we coalesce all training data and use it to train an NBT model. As shown by , such parameter tying can facilitate transfer learning and lead to improved belief tracking performance.  requestable slots. Figure 4 shows the t-SNE visualisation (van der Maaten and Hinton, 2008) of the user utterance representations produced by the NBT-DNN model with tied parameters. The figure shows that the model learns meaningful representations which cluster utterances with related intents together. Moreover, it reveals that utterances such as 'yes' and 'right' or 'I don't care' and 'It doesn't matter' are mapped close together, proving that the NBT utterance representations capture a lot of semantic information from the underlying word vectors and the (relatively few) annotated transcriptions that they are exposed to.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
The NBT models use the semantic relations embedded in the pre-trained word vectors to handle semantic variation and produce high-quality intermediate representations. Table 4 shows the effect of using different word vectors to train the two NBT models. Using GloVe vectors (Pennington et al., 2014) in place of Paragram-SL999 (Wieting et al., 2015) drastically reduced the models' goal tracking capabilities. Interestingly, it had much less of an effect on requests, suggesting that the training set is large enough for the NBT models to learn specific phrases which trigger these ('How much?', 'Where is it?'), but not to ensure coverage for the numerous rephrasings of categorical slot values.
Neural Belief Tracker: Data-Driven Dialogue State Tracking
In this paper, we have proposed a novel belief tracking framework designed to overcome current obstacles to deploying dialogue systems in larger dialogue domains. The Neural Belief Tracker can use a single set of parameters to decide whether the given user utterance (and its context) express each of the intents defined by the domain ontology. This is achieved by learning similarity metrics which operate over intermediate representations composed from semantically specialised word vector representations. The NBT framework offers the known advantages of coupling Spoken Language Understanding and Dialogue State Tracking, without requiring the use of (hand-crafted) semantic lexicons to boost SLU recall. Our evaluation demonstrates these benefits, with NBT matching the performance of models that rely on such lexicons. Finally, we showed that sharing NBT models' parameters across slots can boost overall belief tracking performance, supporting our argument that these models promise to scale to complex domains with diverse and unequally distributed slots.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Task-oriented dialogue systems have emerged for helping users to solve specific tasks efficiently (Hosseini-Asl et al., 2020). Evaluation is Figure 1: Task-oriented dialogue system has a predefined schema for each task, which is composed of a set of task attributes. In a dialogue, the user's task goal is encoded by the task attribute and value pairs. The user is satisfied with the service when the provided solution fulfills the user's preferences for the task attributes. a crucial part of the development process of such systems. Many of the standard automatic evaluation metrics, e.g. BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), have been shown to be ineffective in task-oriented dialogue evaluation (Deriu et al., 2021;. As a consequence, User Satisfaction Modeling (USM) (Sun et al., 2021;Kachuee et al., 2021;Bodigutla et al., 2020;Song et al., 2019;Rebensburg et al., 2023) has gained momentum as the core evaluation metric for task-oriented dialogue systems. USM estimates the overall satisfaction of a user interaction with the system. In task-oriented dialogue systems, whether a user is satisfied largely depends on how well the user's task goals were fulfilled. Each task would typically have an associated task schema, which is a set of task attributes (e.g. location, date for check-in and check-out, etc. for a hotel booking task), and for the user to be satisfied, the system is expected to fulfill the user's preferences about these task attributes. Figure 1 shows an example of USM for task-oriented dialogues.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Effective USM models should have the following abilities: (1) Interpretability by giving insights on what aspect of the task the system performs well. For instance, this can help the system to recover from an error and optimize it toward an individual aspect to avoid dissatisfaction. (2) Scalability in dealing with unseen tasks, e.g. the model does not need to retrain when integrating new tasks. (3) Cost-efficiency for performing well in low-resource settings where it is often hard to collect and expensive to annotate task-specific data.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Previous work in USM follows two main lines of research. First, several methods use user behavior or system actions to model user satisfaction. In this setting, it is assumed that user satisfaction can be reflected by user behaviors or system actions in task-oriented dialogue systems, such as click, pause, request, inform Guo et al., 2020). A second approach is to analyze semantic information in user natural language feedback to estimate user satisfaction, such as sentiment analysis (Sun et al., 2021;Song et al., 2019) or response quality assessment (Bodigutla et al., 2020;Zeng et al., 2020). However, both of these two lines of work do not take into account the abilities of interpretability, scalability, and cost-efficiency.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
In this paper, we propose a novel approach to USM, referred to as Schema-Guided User Satisfaction Modeling (SG-USM). We hypothesize that user satisfaction should be predicted by the fulfillment degree of the user's task goals that are typically represented by a set of task attribute and value pairs. Therefore, we explicitly formalize this by predicting how many task attributes fulfill the user's preferences and how important these attributes are. When more important attributes are fulfilled, taskoriented dialogue systems should achieve better user satisfaction.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Specifically, SG-USM comprises a pre-trained text encoder to represent dialogue context and task attributes, a task attribute fulfillment representation layer to represent the fulfillment based on the relation between the dialogue context and task attributions, a task attribute importance predictor to cal-culate the importance based on the task attributes popularity in labeled and unlabeled dialogue corpus, and a user satisfaction predictor which uses task attributes fulfillment and task attributes importance to predict user satisfaction. SG-USM uses task attributes fulfillment and task attributes importance to explicitly model the fulfillment degree of the user's task goals (interpetability). It uses an task-agnostic text encoder to create representations of task attributes by description, no matter whether the task are seen or not (scalability). Finally, it uses unlabeled dialogues in low-resource settings (cost-efficiency).
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Experimental results on popular task-oriented benchmark datasets show that SG-SUM substantially and consistently outperforms existing methods on user satisfaction modeling. Extensive analysis also reveals the significance of explicitly modeling the fulfillment degree of the user's task goals, the ability to deal with unseen tasks, and the effectiveness of utilizing unlabeled dialogues.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Our SG-USM approach formalizes user satisfaction modeling by representing the user's task goals as a set of task attributes, as shown in Figure 1.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
The goal is to explicitly model the degree to which task attributes are fulfilled, taking into account the importance of the attributes. As shown in Figure 2, SG-USM consists of a text encoder, a task attribute fulfillment representation layer, a task attribute importance predictor, and a user satisfaction predictor. Specifically, the text encoder transforms dialogue context and task attributes into dialogue embeddings and task attribute embeddings using BERT (Devlin et al., 2019). The task attribute fulfillment representation layer models relations between the dialogue embeddings and the task attribute embeddings by attention mechanism to create task attribute fulfillment representations. Further, the task attribute importance predictor models the task attribute popularity in labeled and unla-beled dialogues by the ranking model to obtain task attribute importance weights. Finally, the user satisfaction predictor predicts user satisfaction score on the basis of the task attribute fulfillment representations and task attribute importance weights using a multilayer perceptron.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
The text encoder takes the dialogue context (user and system utterances) and the descriptions of task attributes as input and uses BERT to obtain dialogue and task attribute embeddings, respectively. Considering the limitation of the maximum input sequence length of BERT, we encode dialogue context by each dialogue turn. Specifically, the BERT encoder takes as input a sequence of tokens with length L, denoted as X = (x 1 , ..., x L ). The first token x 1 is [CLS], followed by the tokens of the user utterance and the tokens of the system utterance in one dialogue turn, separated by [SEP]. The representation of [CLS] is used as the embedding of the dialogue turn. Given a dialogue with N dialogue turns, the output dialogue embeddings is the concatenation of all dialogue turn embeddings
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
To obtain task attribute embeddings, the input is a sequence of tokens with length K, denoted as Y = {y 1 , ..., y K }. The sequence starts with [CLS], followed by the tokens of the task attribute description. The representation of [CLS] is used as the embedding of the task attribute. The set of task attribute embeddings are denoted as T = {t 1 , t 2 , ..., t M }, where M is the number of task attributes.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
The task attribute fulfillment representation layer takes the dialogue and task attribute embeddings as input and calculates dialogue-attended task attribute fulfillment representations. This way, whether each task attribute can be fulfilled in the dialogue context is represented. Specifically, the task attribute fulfillment representation layer constructs an attention vector by a bilinear interaction, indicating the relevance between dialogue and task attribute embeddings. Given the dialogue embeddings D and i-th task attribute embedding t i , it calculates the relevance as follows, where W a is the bilinear interaction matrix to be learned. A i represents the attention weights of dialogue turns with respect to the i-th task attribute. Then the dialogue-attended i-th task attribute fulfillment representations are calculated as follows,
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
The dialogue-attended task attribute fulfillment representations for all task attributes are denoted as:
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
where M is the number of the task attributes.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
The task attribute importance predictor also takes the dialogue and task attribute embeddings as input and calculates attribute importance scores. The importance scores are obtained by considering both the task attribute presence frequency and task attribute presence position in the dialogue. First, we use the Maximal Marginal Relevance (MMR) (Carbonell and Goldstein, 1998) to select the top relevant task attributes for the dialogue context. The selected task attributes are then used to calculate the task attribute presence frequency in the dialogue. The MMR takes the j-th dialogue turn embeddings d j and task attribute embeddings T as input, and picks the top K relevant task attributes for the j-th dialogue turn:
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
where U is the subset of attributes already selected as top relevant task attributes, cos() is the cosine similarity between the embeddings. λ trades off between the similarity of the selected task attributes to the dialogue turn and also controls the diversity among the selected task attributes. The task attribute presence frequency vector for the j-th dialogue turn is computed as follows,
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
where M is the number of the task attributes. However, the task attribute presence frequency vector does not reward task attributes that appear in the beginning of the dialogue. The premise of task attribute importance score is that task attributes appearing near the end of the dialogue should be penalized as the graded importance value is reduced logarithmically proportional to the position of the dialogue turn. A common effective discounting method is to divide by the natural log of the position:F
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
The task attribute importance predictor then computes the importance score on the basis of the sum of the discounted task attribute presence frequency of all dialogues. Given the dialogue corpus (including both labeled and unlabeled dialogues) with Z dialogues C = {D 1 , D 2 , ..., D Z }, the task attribute importance scores are calculated as follow:
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
where Num() is the number of the dialogue turn in dialogue D l , andF l j is the discounted task attribute presence frequency of j-th dialogue turn in dialogue D l .
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Given the dialogue-attended task attribute fulfillment representations T a and the task attribute importance scores S, the user satisfaction labels are obtained by aggregating task attribute fulfillment representations based on task attribute importance scores. This way, the user satisfaction is explicitly modeled by the fulfillment of the task attributes and their individual importance.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Specifically, an aggregation layer integrates the dialogue-attended task attribute fulfillment representations by the task attribute importance scores as follows:
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Then the Multilayer Perceptron (MLP) (Hastie et al., 2009) with softmax normalization is employed to calculate the probability distribution of user satisfaction classes:
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
We train SG-USM in an end-to-end fashion by minimizing the cross-entropy loss between the predicted user satisfaction probabilities and the ground-truth satisfaction:
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
where y is the ground-truth user satisfaction. Pretrained BERT encoders are used for encoding representations of utterances and schema descriptions respectively. The encoders are fine-tuned during the training process.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
We conduct experiments using four benchmark datasets containing task-oriented dialogue on different domains and languages (English and Chinese), including MultiWOZ2.1 (MWOZ) (Eric et al., 2020), Schema Guided Dialogue (SGD) (Rastogi et al., 2020), ReDial , and JDDC (Chen et al., 2020b). MWOZ and SGD are English multi-domain taskoriented dialogue datasets, which include hotel, restaurant, flight, etc. These datasets contain domain-slot pairs, where the slot information could correspond to the task attributes.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
ReDial is an English conversational recommendation dataset for movie recommendation. The task attributes are obtained from the Movie 2 type on Schema.org. JDDC is a Chinese customer service dialogue dataset in E-Commerce. The task attributes are obtained from the Product 3 type on Schema.org.cn, which provides schemas in Chinese. Specifically, we use the subsets of these datasets with the user satisfaction annotation for evaluation, which is provided by Sun et al (Sun et al., 2021). We also use the subsets of these datasets without the user satisfaction annotation to investigate the semi-supervised learning abilities of SG-USM.  
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
We compare our SG-USM approach with competitive baselines as well as state-of-the-art methods in user satisfaction modeling.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
HiGRU (Jiao et al., 2019) proposes a hierarchical structure to encode each turn in the dialogue using a word-level gated recurrent unit (GRU) (Dey and Salem, 2017) and a sentence-level GRU. It uses the last hidden states of the sentence-level GRU as inputs of a multilayer perceptron (MLP) (Hastie et al., 2009) Table 2: Performance of SG-USM and baselines on various evaluation benchmarks. Numbers in bold denote the best model performance for a given metric. Numbers with * indicate that SG-USM model is better than the best-performing baseline method (underlined scores) with statistical significance (t-test, p < 0.05).
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
HiGRU to represent dialogues. An MLP takes the dialogue representation as inputs to predict the user satisfaction level. Transformer (Vaswani et al., 2017) is a simple baseline that takes the dialogue context as input and uses the standard Transformer encoder to obtain the dialogue representations. An MLP is used on the encoder to predict the user satisfaction level. BERT (Devlin et al., 2019) concatenates the last 512 tokens of the dialogue context into a long sequence with a [SEP] token for separating dialogue turns. It uses the [CLS] token of a pre-trained BERT models to represent dialogues. An MLP is used on the BERT to predict the user satisfaction level. USDA ) employs a hierarchical BERT encoder to encode the whole dialogue context at the turn-level and the dialogue-level. It also incorporates the sequential dynamics of dialogue acts with the dialogue context in a multi-task framework for user satisfaction modeling.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
We also report the performance of two simpler SG-USM variants: SG-USM(L) only uses the dialogues with groundtruth user satisfaction labels to train the model. SG-USM(L&U) uses both labeled and unlabeled dialogues in the training process. It takes the dialogues without user satisfaction annotation as the inputs of task attribute importance predictor module to obtain more general and accurate task attribute importance scores.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
For a fair comparison with previous work and without loss of generality, we adopt BERT as the backbone encoder for all methods that use pretrained language models.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Following previous work Cai and Chen, 2020;Choi et al., 2019;Song et al., 2019), we consider a three-class classification task for user satisfaction modeling by treating the rating "</=/> 3" as "dissatisfied/neutral/satisfied". Accuracy (Acc), Precision (P), Recall (R), and F1 are used as the evaluation metrics.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
We use BERT-Base uncased, which has 12 hidden layers of 768 units and 12 self-attention heads to encode the utterances and schema descriptions. We apply a two-layer MLP with the hidden size as 768 on top of the text encoders. ReLU is used as the activation function. The dropout probability is 0.1. Adam (Kingma and Ba, 2014) is used for optimization with an initial learning rate of 1e-4. We train up to 20 epochs with a batch size of 16, and select the best checkpoints based on the F1 score on the validation set. Table 2 shows the results of SG-USM on MWOZ, SGD, ReDial, and JDDC datasets. Overall, we observe that SG-USM substantially and consistently outperforms all other methods across four datasets with a noticeable margin. Specifically, SG-USM(L) improves the performance of user satisfaction modeling via explicitly modeling the degree to which the task attributes are fulfilled. SG-USM(L&U) further aids the user satisfaction modeling via predicting task attribute importance based on both labeled dialogues and unlabeled dialogues. It appears that the success of SG-USM is due to its architecture design which consists of the task attribute fulfillment representation layer and the task attribute importance predictor. In addition, SG-USM can also effectively leverage unlabeled dialogues to alleviate the cost of user satisfaction score annotation. 
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Doctor: § Type: speciality of the doctor. § City: city where the doctor is located.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Importance:
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
(a) Example 1
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
U: Would you show me attractions to visit in Philadelphia? I prefer a museum, and someplace without an entry fee. S: Barnes Foundation is an art museum that you may like. U: Okay. Is it free? S: No. The ticket for an adult is $25. U: Sorry, I want a museum without an entry fee.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Travel: § Category: category to which the attraction belongs. § FreeEntry: whether entrance to attraction is free.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
(b) Example 2 Figure 4: Case study on SG-USM and USDA on SGD dataset. The yellow ★ represents the importance of task attributes. The texts in green are the users' preferences for the task attributes. The texts in red are the attributes of the provided solutions.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
We also conduct an ablation study on SG-USM to study the contribution of its two main components: task attribute importance and task attribute fulfillment.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
To investigate the effectiveness of task attribute importance in user satisfaction modeling, we eliminate the task attribute importance predictor and run the model on MWOZ, SGD, ReDial, and JDDC. As shown in Figure 3, the performance of SG-USMw/oImp decreases substantially compared with SG-USM. This indicates that the task attribute importance is essential for user satisfaction modeling. We conjecture that it is due to the user satisfaction relates to the importance of the fulfilled task attributes.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
To investigate the effectiveness of task attribute fulfillment in user satisfaction modeling, we compare SG-USM with SG-USM-w/oFul which eliminates the task attribute fulfillment representation. Figure 3 shows the results on MWOZ, SGD, Re-Dial, and JDDC in terms of F1. From the results, we can observe that without task attribute fulfillment representation the performances deteriorate considerably. Thus, utilization of task attribute fulfillment representation is necessary for user satisfaction modeling.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
We also perform a qualitative analysis on the results of SG-USM and the best baseline USDA on the SGD dataset to delve deeper into the differences of the two models. We first find that SG-USM can make accurate inferences about user satisfaction by explicitly modeling the fulfillment degree of task attributes. For example, in the first case in Figure 4, the user wants to find a gynecologist in New York. SG-USM can correctly predict the dissatisfied label by inferring that the first important task attribute "Type" is not fulfilled. In the second case, the user wants to find a museum without an entry fee. SG-USM can yield  Table 3: Performance of SG-USM and the best baseline USDA on zero-shot learning ability. All the models are trained on SGD and tested on MWOZ and ReDial. Numbers in bold denote best results in that metric. Numbers with * indicate that the model is better than the performance of baseline with statistical significance (t-test, p < 0.05). the correct neural label by inferring that the second important task attribute "FreeEntry" is not fulfilled. From our analysis, we think that SG-USM achieves better accuracy due to its ability to explicitly model how many task attributes are fulfilled and how important the fulfilled task attributes are. In contrast, the USDA does not model the fulfillment degree of task attributes, thus it cannot properly infer the overall user satisfaction.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
We furhter analyze the zero-shot capabilities of SG-USM and the best baseline of USDA. The SGD, MWOZ, and ReDial datasets are English dialogue datasets that contain different task attributes. Therefore, we train models on SGD, and test models on MWOZ and ReDial to evaluate the zero-shot learning ability. Table 3 presents the Accuracy, Precision, Recall, and F1 of SG-USM and USDA on MWOZ and ReDial. From the results, we can observe that SG-USM performs significantly better than the baseline USDA on both datasets. This indicates that the agnostic task attribute encoder of SG-USM is effective. We argue that it can learn shared knowledge between task attributes and create more accurate semantic representations for unseen task attributes to improve performance in zeroshot learning settings.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
To analyze the effect of the unlabeled dialogues for SG-USM, we test different numbers of unlabeled dialogues during the training process of SG-USM. Figure 5 shows the Accuracy and F1 of SG-USM when using 1 to 4 thousand unlabeled dialogues for training on MWOZ, SGD, ReDial, and JDDC. From the results, we can see that SG-USM can achieve higher performance with more unlabeled dialogues. This indicates that SG-USM can effectively utilize unlabeled dialogues to improve the performance of user satisfaction modeling. We reason that with a larger corpus, the model can more accurately estimate the importance of task attributes.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
User satisfaction modeling is an important yet challenging problem for task-oriented dialogue systems evaluation. For this purpose, we proposed to explicitly model the degree to which the user's task goals are fulfilled. Our novel method, namely SG-USM, models user satisfaction as a function of the degree to which the attributes of the user's task goals are fulfilled, taking into account the importance of the attributes. Extensive experiments show that SG-USM significantly outperforms the state-of-the-art methods in user satisfaction modeling on various benchmark datasets, i.e. MWOZ, SGD, ReDial, and JDDC. Our extensive analysis also validates the benefit of explicitly modeling the fulfillment degree of a user's task goal based on the fulfillment of its constituent task attributes. In future work, it is worth exploring the reasons of user dissatisfaction to better evaluate and improve task-oriented dialogue systems.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Our approach builds on a task schema that characterizes a task-oriented dialogue system's domain. For example, the schema captures various attributes of the task. For some domains, when a schema is not pre-defined, it first needs to be extracted, e.g., from a corpus of dialogues. In this paper, we used BERT as our LM to be comparable with related work, but more advanced models could further improve the performance. A limitation of our task attribute importance scoring method is that it currently produces a static set of weights, reflecting the domain. In the future, the importance weights may be personalized to the current user's needs instead.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
A For every submission:
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
In the last section A2. Did you discuss any potential risks of your work?
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
A3. Do the abstract and introduction summarize the paper's main claims? B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Section 5 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5 D Did you use human annotators (e.g., crowdworkers) or research with human participants?
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
Left blank.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? No response.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
Schema-Guided User Satisfaction Modeling for Task-Oriented Dialogues
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Task-oriented systems are developed to help users solve a specific task as efficiently as possible [56]. Evaluation is a crucial part of the development process of task-oriented dialogue systems. For evaluating the performance of each module of a dialogue system, human evaluation, user satisfaction modeling, corpus-based approaches, and user simulation have all been leveraged [13]. Human evaluation through in-field experiments [6,27] or crowd-sourcing [25] is considered to reflect the overall performance of the system in a real-world scenario, but it is intrusive, time-intensive, and does not scale [13]. User satisfaction modeling can be an alternative; it aims to automatically estimate user satisfaction based on human-machine interaction log data, but still requires human involvement. To evaluate a dialogue system fully automatically, offline evaluation based on test sets is commonly used. However, this method is limited to a single turn and does not inform us about the overall usefulness of the system or about users' satisfaction with the flow of the dialogue [57]. Therefore, evaluation results of offline methods have limited consistency with the results of human evaluation. Simulation-based evaluation methods address the issues listed above; they are a viable choice for large-scale automatic evaluation [13]. User simulations can be used to evaluate functionalities of dialogue systems and they can serve as an environment to train reinforcement learning-based systems [13], leveraging agendabased [42] or model-based simulation [3]. Building human-like user simulation is still an open challenge [22].
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
To bridge the gap between human evaluation and user simulation, we attempt to combine user simulation with user satisfaction (cf. Figure 1). To this end, we first look into existing task-oriented dialogues and carry out a user study to investigate the characteristics of user satisfaction. We arrive at two main observations: (1) User dissatisfaction is mainly caused by the system's failure in meeting the user's needs. Specifically, 36% of the conversations are labeled as very dissatisfied because the system does not understand the user's needs, and 43% are because the system understands the  Figure 1: (a) Previous work on user simulation; (b) previous work on user satisfaction prediction; (c) our proposed task: simulating user satisfaction for evaluating task-oriented dialogues systems. We leave utterance generation (dotted line) as future work.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
user's problems but cannot provide proper solutions. Figure 2 illustrates the scenario. (2) Different degrees of satisfaction result in different sequences of user actions. For example, the right-side user in Figure 2 may switch to customer service or explain further when encountering the same failed system reply in the context of different emotions. We convert this intuition to a hypothesis that we verify by checking the records in the corpus. When faced with a dialogue system's failure in understanding user needs, about 17.1% of all users will switch to manual customer service, and about 64.3% and 9.7% will continue by providing additional information, or quit the conversation, respectively. This observation suggests that user simulation should work differently in different user satisfaction scenarios. Informed by the observations just listed, we propose a novel task: to simulate user satisfaction for the evaluation of task-oriented dialogue systems. Figure 1 illustrates the main difference between our task and previous work. We extend the evaluation capability of user simulations and make the simulation more human-like by incorporating user satisfaction prediction and user action prediction.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
To facilitate research on user satisfaction simulation, we develop a user satisfaction annotation dataset, User Satisfaction Simulation (USS). We invite 40 annotators to label both the dialogue level and exchange level user satisfaction of 5 commonly used taskoriented dialogue datasets in different domains. This results in a dataset of 6,800 dialogues, where each individual user utterance, as well as each complete dialogue, is labeled on a 5-point satisfaction scale. Each dialogue is labeled by 3 annotators; the expert ratings are highly correlated, with a Fleiss Kappa score of 0.574. The USS dataset shares some characteristics with existing datasets for user satisfaction, but also differs in important ways (see Table 1): (1) Our user satisfaction labeling occurs before the user utterance, and is based on the dialogue context between user and system instead of the satisfaction expressed in the user's utterance. (2) The USS dataset includes multiple domains, such as e-commerce, reservations, recommendations, etc. (3) The USS dataset exceeds existing user satisfaction data in scale.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
We share three baseline approaches to perform satisfaction prediction and user action prediction based on the newly collected data in USS: a feature-based method, a hierarchical GRU-based method, and a BERT-based method. Experimental results suggest that distributed representations outperform feature-based methods. The hierarchical GRU-based method achieves the best performance Do you want to return the product. 
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Hello, my order is repeated.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Dear, do you need to cancel it.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Yes, cancel my order.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
OK, I see. I'll contact you to handle the refund.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
I've been canceling orders since yesterday, and I haven't stopped them until now.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Yes, it has been cancelled. The financial audit is passed and the refund is in progress.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
OK, thank you.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Is there anything else I can do for you?
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Wouldn't it be arranged?
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
No, because it has been cancelled successfully. Now the orders are in the process of refund.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Dialogue-level Exchange-level Exchange-level Figure 2: Two examples of dialogues in the JDDC dataset [11], with different degrees of user satisfaction. The right-side system fails to understand the user's needs, and causes the user to be dissatisfied emotions and have a poor user experience. The left-side dialogue demonstrates an opposite case.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
in in-domain user satisfaction prediction, while the BERT-based method has a better cross-domain generalization ability thanks to the pre-training. We also show that the BERT-based method achieves state-of-the-art performance on the action prediction task. In summary, this paper makes the following contributions: (1) We propose the novel task of simulating user satisfaction for the evaluation of task-oriented dialogue systems. (2) We collect and share a dataset, USS, that includes 6,800 annotated dialogues in multiple domains. (3) We introduce three baseline methods for the tasks of satisfaction prediction and action prediction using the USS dataset.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Unlike chitchat systems, which focus on conversing with human on open domains, task-oriented dialogue systems aim to complete specific tasks for user [29,51]. Task-oriented dialogue systems can be divided into module-based and end-to-end-based methods [22]. The former decomposes the dialogue system into four stages: language understanding, dialogue state tracking, dialogue policy learning, and response generation. Recently, each stage in the module-based task-oriented dialogue systems has received increased attention [20,37,38,51,53,55]. End-to-end task-oriented dialogue systems rely on neural dialogue generation, which has received a lot of attention in recent years [2,4,56]. Among all these approaches, sequence-to-sequence structure neural generation models [10,24,28,31,46,49] have been proved to be capable in multiple dialogue systems with promising performance.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Evaluation is a crucial part of the development process of taskoriented dialogue systems. Corpus-based approaches, user simulation, and user satisfaction modeling have all been leveraged [57] for evaluating the performance of a task-oriented dialogue system. Offline evaluation based on test sets is commonly used, but it is limited in scope to a single turn and does not inform us about the overall usefulness of the system or about users' satisfaction with the flow of the dialogue [57]. Employing simulation-based evaluation can tackle the above issues and become one viable choice for large-scale automatic evaluation [13]. User simulators are tools that are designed to simulate the user's behavior, which can be used to train the dialogue manager in an offline environment [13] or to evaluate the dialogue policy [42]. Eckert et al. [15] propose the first statistical user simulator. Scheffler and Young [43] propose a graph-based model. Georgila et al. [18] use a Markov Model, and a hidden Markov model has been proposed by Cuayáhuitl et al. [12]. Schatzmann et al. [42] propose an agenda-based user simulator, which represents the user state elegantly as a stack of necessary user actions, called the agenda. Zhang and Balog [57] evaluate conversational recommender systems via an agenda-based user simulator. Recent work employs neural approaches, esp. sequenceto-sequence models [3,26]. As far as we know, no previous study explicitly models the user satisfaction in user simulations. Unlike previous work, we are the first to incorporate user satisfaction into user simulation to make the simulation more human-like. Next to user simulations, user satisfaction modeling is the other evaluation method that is based on the idea that the usability of a system can be approximated by the satisfaction of its users [13]. Ultes et al. [48] note the impracticability of having a user rate a live dialogue. Thus, automatic prediction can be an alternative. Walker et al. [50] propose the PARADISE framework, which estimates user ratings on the dialogue level. Evaluation methods that estimate user satisfaction at the exchange level have also been proposed [16,19,21]. They yield more fine-grained predictions and are especially useful for online dialogue breakdown detection. Schmitt and Ultes [44] propose Interaction Quality (IQ) to assign user ratings by experts instead of real users. Bodigutla et al. [7] introduce the Response Quality (RQ) scheme to improve generalizability to multiple-domain conversations.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Unlike previous work on user satisfaction modeling, we simulate the user satisfaction changes without human involvement.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
To formulate the task of simulating user satisfaction, we first carry out a user study to explore the characteristics of user satisfaction in task-oriented dialogues. Specifically, we invite 12 experts and let each expert annotate 20 dialogues sampled from the JDDC dataset; we used the JDDC dataset since it is more realistic than data constructed by the Wizard-of-Oz approach. We ask each expert to score the user satisfaction for each dialogue turn and the entire conversation. In addition, a rational explanation is requested. We ask the experts to judge the user action changes after a change in satisfaction. Based on this study, we answer the following questions:
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
(1) What causes the user's dissatisfaction? We collect the results and find that, although annotators are satisfied with the system overall, about 12% of the dialogue turns are labeled as unsatisfying. This indicates that there are fluctuations in user satisfaction when interacting with the system. We analyze the annotators' explanations and find that the main reason for dissatisfaction relates to the system's failure to understand the user's needs or handling the user's requests. Specifically, 36% of all conversations labeled as very dissatisfied are because the system does not understand the user's needs, whereas 43% are because the user does not approve the system's response. In 64% of the data, users had a bad user experience because the system was not professional enough or did not respond in time. Figure 2 illustrates the scenario where the system does not understand the user's needs and causes low user satisfaction.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
(2) How does user satisfaction influence the user's behavior? Different degrees of satisfaction result in different sequences of user actions. Specifically, when encountering a failure in the a dialogue system's understanding of user needs, about 17.1% of all users switch to manual customer service, and about 64.3% and 9.7% continue by providing additional information, or quit the conversation, respectively. Figure 2 shows an example, where the right-side user switches to customer service or explains further when encountering the same failed system reply in light of different degrees of satisfaction. Apart from user actions, we also observe changes such as attitude and information-seeking goal.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
The above observations indicate that predicting the fluctuations of user satisfaction during interaction is non-trivial. Thus, we formulate our research task, i.e., to simulate user satisfaction for the evaluation of task-oriented dialogue systems.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
This simulation task focuses on the prediction of the next user action as well as user satisfaction. Suppose that we have a dataset D = {( , , )} =1 , where for all ∈ [1, ], is the dialogue context, is the next-turn user action, and denotes user satisfaction. The task objective is to learn a classification model ( , | ) from D, and thus given a dialogue context , it predicts the next-turn user action and user satisfaction based on ( , | ). The purpose of the task to increase the evaluation power of user simulations and to make the simulation more human-like by incorporating the user's potential changes in satisfaction in a simulator.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
We propose a user satisfaction annotation dataset, User Satisfaction Simulation (USS). Below, we detail the creation of the dataset. We divide this section into 3 phases: data preparation, user satisfaction assessment, and measures and disclaimers.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
The USS dataset is based on five benchmark task-oriented dialogue datasets: JDDC [11], Schema Guided Dialogue (SGD) [41], MultiWOZ 2.1 [17], Recommendation Dialogues (ReDial) [33], and Coached Conversational Preference Elicitation (CCPE) [40]. We first distinguish the user's emotion in the conversation by a classifier trained on annotated reddit data (weibo for Chinese), and then filter out all conversations that do not show negative emotions (i.e., anger, disgust, fear, sadness).
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
(1) JDDC is a large-scale, real-world Chinese e-commerce conversation corpus with over 1 million multi-turn dialogues. We first classify the conversation into 11 types according to the type of transaction, e.g., delivery, return, invoice, etc. Then, we sample 300 dialogue sessions from each type, for a total of 3,300 conversations. The JDDC data set provides the action of each user utterance, including 234 categories. We compress them into 12 categories based on a manually defined classification method. (2) SGD is a dataset consisting of over 20K annotated task-oriented conversations between a human and a virtual assistant spanning 16 domains. MultiWOZ 2.1 is a multi-domain dialogue dataset spanning 7 distinct domains and containing over 10K dialogues. We sample 1,000 conversations from the two datasets. We directly use the action annotation from the original datasets. The SGD has 12 actions, and MultiWOZ has 21 actions. (3) ReDial is an annotated dataset consisting of over 10K conversations, where users recommend movies to each other. We sample 1,000 dialogues. Since the original dataset does not provide actions, we use the action annotation provided by IARD [9]. (4) CCPE is a dataset consisting of 502 dialogues with 12K annotated utterances between a user and an assistant discussing movie preferences. We sample 300 dialogues from the CCPE dataset and used the actions provided by the original dataset.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
We hired 40 annotators to annotate exchange-level and dialoguelevel user satisfaction levels of each conversation with five levels (1)(2)(3)(4)(5). We first show a dialogue between user and system in text form to the annotators and ask the annotators to label the user satisfaction of each user sentence at the exchange-level. We require annotators to rate user satisfaction based on past conversations, so the satisfaction is assessed before the user's sentence, not after writing the sentence. In this regard, we differ from previous annotation work [7,44,50]. The scale we asked annotators to follow was: (1) Very dissatisfied (the system fails to understand and fulfill user's request); (2) Dissatisfied (the system understands the request but fails to satisfy it in any way); (3) Normal (the system understands users request and either partially satisfies the request or provides information on how the request can be fulfilled); (4) Satisfied (the system understands and satisfies the user request, but provides more information than what the user requested or takes extra turns before meeting the request); and (5) Very satisfied (the system understands and satisfies the user request completely and efficiently).
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Using a 5 point scale over a binary scale provides an option for the annotators to factor in their subjective interpretation of the extent of success or failure of a system's response to satisfy a user's request. In addition, we ask the annotators to rate the dialogue-level satisfaction to capture the overall satisfaction of a user's interaction with the system. We divide the data into two groups based on language, JDDC (Chinese) and Others (English). In each group, we randomly assign data to annotators to ensure that the different types of conversations in the group are evaluated according to a consistent standard. For the JDDC group, we also ask annotators to give a textual explanation for the rating. 
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
To guarantee annotation quality, we ask at least three annotators to repeatedly label the data. If there is a discrepancy among the three annotators (i.e., three annotators give three different ratings), we ask a fourth annotator to recheck it. We removed the results of annotators that were inconsistent with others. Finally, expert ratings are highly correlated with a Fleiss Kappa score of 0.574. See Table 2 for descriptive statistics of the USS dataset.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
In all the provided instruction materials, we described the purpose of this data construction effort and pointed out that the data will only be used for research. We did not record any information about the annotators and warned the annotators not to divulge any of their private information.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Inspired by previous work [5,23,54], we consider three types of approach: Feature-based, RNN-based, and BERT. 
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Given the dialogue context = { } =1 , we first encode it to get the context representation h , and then predict the user satisfaction by ( | ) = softmax(MLP(h )). Inspired by previous work, we compare three methods for context representation encoding: (1) GRU, which first concatenates the dialogue history into a long sentence, and then feeds the sentence into a Bidirectional GRU (BiGRU) model. Then the context representation is defined as the average pooled outputs of the BiGRU model. (2) HiGRU, which explores the hierarchical structure. First, it encodes each utterance in the dialogue using a word-level Bi-GRU to get the utterance representations h . Then it feeds the utterance representations into a sentence-level GRU, and define the context representation as the last hidden state of the sentence-level GRU [23]. (3) HiGRU+ATTN, which applies a two-level attention mechanism in HiGRU [54]. ]. Then we encode it into a latent representation via BERT [14], and convert it into the condensed representation h through an average pooling operation. User satisfaction is predicted as ( | ) = softmax(MLP(h )).
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
To integrate the user satisfaction prediction and action prediction, we train two independent models for two tasks, in which action prediction takes the predicted output of satisfaction prediction model as the input. We use ground truth satisfaction in training and the model predicted satisfaction in testing. The Feature-based models are implemented using the scikit-learn toolkit. For the BERTbased model, we use BERT-Base (110M) pretrained weights 1 (hidden size is 768). We use the BERT vocabulary (size: 30,522) for all models (the Chinese BERT vocabulary for the JDDC domain), set the batch size = 64, the learning rate to 2e-5 for BERT and 1e-4 for others, use the AdamW optimizer ( 1 = 0.9, 2 = 0.999, and = 10 −8 ) to optimize parameters, use gradient clipping with a maximum gradient norm of 0.2, train up to 50 epochs on one NVIDIA TITAN RTX GPU, and select the best checkpoints based on performance on the validation set. Due to the serious imbalance of the satisfaction label, we up-sample the non-3 rating data during training. We take dialogue-level satisfaction as the last user utterance and use "overall" as the identification. As in previous work [9], we use 10-fold crossvalidation to evaluate the outcome.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
For the user satisfaction prediction task, following [44], we use the Unweighted Average Recall (UAR), the arithmetic average of all class-wise recalls, a linearly weighted version of Cohen's Kappa, and Spearman's Rho as evaluation metrics. We also use the F1-score for the dissatisfactory (rating < 3) class as the binary classification metric, as most turns and dialogues belong to the satisfactory (rating ≥ 3) class. For the user action prediction task, we use Accuracy (Acc, the proportion of predicted correct labels over the total number of predicted and actual labels for every utterance), Precision (Prec, the proportion of the predicted correct labels over the number of predicted labels), Recall (the proportion of the predicted correct labels over the number of actual labels), and the F1-score (the harmonic mean of precision and recall) as evaluation measures. Table 3 shows the results for the user satisfaction prediction task. The best results in terms of the corresponding metric are shown in bold. If there are multiple similar best results, we show them all underlined. In general, HiGRU achieves the best overall performance (e.g., an absolute improvement of +3 for UAR, +2 for Kappa, and +4 for F1 over BERT in SGD data). BERT and HiGRU+ATTN can achieve performance comparable to HiGRU, followed by GRU. Among the 3 feature-based methods, SVM performs best, followed by LR. XGBoost is significantly weaker than other methods in all metrics, except Rho. Table 3 further shows that all deep learning methods perform better than feature-based metrics. Table 4 shows the results for the user action prediction task. In general, the BERT-based model performs best among all methods, followed by HiGRU. BERT outperforms HiGRU on all performance measures except for the ReDial data, possibly due to the lack of sufficient training data. Among the 3 feature-based methods, XGBoost achieves the best performance, obtaining an absolute improvement of about +6 for Acc, +7 for Prec, +3 for Recall, and +4 for F1 compared to LR. XGBoost also outperforms GRU in many metrics.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
Since we have multiple domains in the dataset, we further analyze the cross-domain generalization capabilities of the user satisfaction prediction model. Table 5 shows the results. The rows and columns in Table 5 indicate training data and test data, respectively (e.g., 0.233 in the first column of the sixth row indicates that a BERT model trained on MultiWOZ can get a UAR score of 0.233 on SGD data). In terms of datasets, the models trained on SGD and Multi-WOZ get the best performance on each other's data respectively, and the models trained on ReDial get the best performance on CCPE, possibly due to the similarity between domains. The model trained on CCPE has relatively poor generalization ability, possibly due to limited training data size. In terms of methods, BERT achieves better generalization performance than SVM and HiGRU, possibly due to the improvement of pre-training on the large-scale corpus.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
We have developed resources that are meant to help answer the question of what is a good dialogue. Our annotations and prediction task offer a better characterization of what is a good dialogue than existing datasets. Exchange-level user satisfaction and action prediction can reflect what kind of system behavior will bring positive user satisfaction and what behavior will harm the user experience, which makes our method applicable to many related fields.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
In most prior work, user simulations mechanically give the slots, and thus measure very limited aspects of a dialogue. Building a human-like user simulation remains an open challenge. In this study, we propose the task of user satisfaction simulation and release a dataset for the task. Inspired by previous work on similar tasks [5,23,54], we provide a series of baselines. However, due to the challenging nature of the task, there is plenty of room to improve user satisfaction prediction, and to explore how user satisfaction prediction can be combined with action prediction. Response generation based on user satisfaction (i.e., reflect user satisfaction in a generated utterance) is still an open problem. Previous work on open-domain dialogue may serve as a reference [58]. In addition to user satisfaction, how to ground a user simulator by introducing external knowledge [35,36,47,52] and persona [32] to establish a more human-like user simulator has not yet been studied.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
The USS dataset can be used not only for user simulation but also for other conversational information access tasks. As a user satisfaction annotation dataset that exceeds existing ones in scale, our data can      facilitate research on user satisfaction modeling [39] and POMDPbased dialogue systems [30,56]. Moreover, the USS dataset can also facilitate research into dialogue breakdown detection, and humanmachine hand-off prediction [34]. In the JDDC domain, we provide annotators' explanations on user satisfaction annotations, which includes a total of 9,900 explanation texts. This information can be applied to user studies of user satisfaction, and interpretability studies of evaluations.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
We have proposed the task of simulating user satisfaction for evaluating task-oriented dialogue systems, so as to enhance the evaluation of dialogue systems. We have collected and released a new benchmark dataset, namely USS, for the proposed task. Our dataset contains a total of 6,800 dialogues spanning multiple domains. We have introduced three baselines for our task: feature-based, RNNbased, and BERT-based methods. Experiments conducted on the newly collected dataset suggest that distributed representations do outperform feature-based methods. Besides, HiGRU achieves the best performance in in-domain user satisfaction prediction, while a BERT-based method has better cross-domain generalization ability. As to our future work, we would like to continue to investigate the combination of the user satisfaction prediction and action prediction task, and response generation based on user satisfaction.
Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems
We share the USS dataset at https://github.com/sunnweiwei/usersatisfaction-simulation.