15265
Experimental Evaluation of Bi-directional Multimodal Interaction with Conversational Agents
In the field of intuitive HCI, Embodied Conversational Agents (ECAs) are being developed mostly with speech input. In this paper, we study whether another input modality leads to a more effective and pleasant “bi-directional” multimodal communication. In a Wizard-of-Oz experiment, adults and children were videotaped while interacting with 2D animated agents within a game application. Each subject carried out a multimodal scenario (speech and/or pen input) and a speech-only scenario. The results confirm the usefulness of multimodal input, which yielded shorter scenarios, higher and more homogeneous ratings of easiness. Additional results underlined the importance of gesture interaction for children, and showed a modality specialization for certain actions. Finally, multidimensional analyses revealed links between behavioral and subjective data, such as an association of pen use and pleasantness for children. These results can be used for both developing the functional prototype and in the general framework of ECA-systems evaluation and specification.
14471368
A Unified Framework for Constructing Multimodal Experiments and Applications
Inspired by a Wizard of Oz (WOZ) simulation experiment, we developed a working prototype of a system that enables users to interact with a map display through synergistic combinations of pen and voice. To address many of the issues raised by multimodal fusion, our implementation employed a distributed multi-agent framework to coordinate parallel competition and cooperation among processing components. Since then, the agent-based infrastructure has been enhanced with a collaboration technology, creating a framework in which multiple humans and automated agents can naturally interact within the same graphical workspace.Our current endeavour is the leveraging of this architecture to create a unified implementation framework for simultaneously developing both WOZ simulated systems and their fully-automated counterparts. Bootstrapping effects made possible by such an approach are illustrated by an experiment currently under way in our laboratory: as a naive subject draws, writes, and speaks requests to a (simulated) interactive map, a hidden Wizard responds as efficiently as possible using our best fullyautomated system, through either standard graphical interface devices or multimodal combinations of pen and voice. The input choices made by both subject and Wizard are invaluable, and the data collected from each can be applied directly to evaluating and improving the automated part of the system.
4484500
Perception of Blended Emotions: From Video Corpus to Expressive Agent
Real life emotions are often blended and involve several simultaneous superposed or masked emotions. This paper reports on a study on the perception of multimodal emotional behaviors in Embodied Conversational Agents. This experimental study aims at evaluating if people detect properly the signs of emotions in different modalities (speech, facial expressions, gestures) when they appear to be superposed or masked. We compared the perception of emotional behaviors annotated in a corpus of TV interviews and replayed by an expressive agent at different levels of abstraction. The results provide insights on the use of such protocols for studying the effect of various models and modalities on the perception of complex emotions.
12923227
Levels of Representation in the Annotation of Emotion for the Specification of Expressivity in ECAs
In this paper we present a two-steps approach towards the creation of affective Embodied Conversational Agents (ECAs): annotation of a real-life nonacted emotional corpus and animation by copy-synthesis. The basis of our approach is to study how coders perceive and annotate at several levels the emotions observed in a corpus of emotionally rich TV video interviews. We use their annotations to specify the expressive behavior of an agent at several levels. We explain how such an approach can be useful for providing knowledge as input for the specification of non-basic patterns of emotional behaviors to be displayed by the ECA (e.g. which perceptual cues and levels of annotation are required for enabling the proper recognition of the emotions).
18135407
Guidelines for multimodal user interface design
JMUI (Journal on Multimodal User Interfaces), Special issue “Best of affective computing and intelligent Guidelines for multimodal user interface design. support, human multi-modal information processing. characteristics to the design of a user-oriented and guidelines of multimodal interface design. Artifact lifecycle management, Consumer and user, Interfaces in Automated.Aug 2 Aug 7Los Angeles, CA, USAThursday, 6 August 2015 / HCI International 20152015.hci.international/thursday​CachedDefining and Optimizing User Interfaces Information Complexity for AI Design and Development of Multimodal Applications: A Vision on Key Issues and Traditional Heuristics and Industry Guidelines to Evaluate Multimodal Digital Artifacts
618620
Corpus-based static branch prediction
Correctly predicting the direction that branches will take is increasingly important in today's wide-issue computer architectures. The name program-based branch prediction is given to static branch prediction techniques that base their prediction on a program's structure. In this paper, we investigate a new approach to program-based branch prediction that uses a body of existing programs to predict the branch behavior in a new program. We call this approach to program-based branch prediction, evidence-based static prediction, or ESP. The main idea of ESP is that the behavior of a corpus of programs can be used to infer the behavior of new programs. In this paper, we use a neural network to map static features associated with each branch to the probability that the branch will be taken. ESP shows significant advantages over other prediction mechanisms. Specifically, it is a program-based technique, it is effective across a range of programming languages and programming styles, and it does not rely on the use of expert-defined heuristics. 
In this paper, we describe the application of ESP to the problem of branch prediction and compare our results to existing program-based branch predictors. We also investigate the applicability of ESP across computer architectures, programming languages, compilers, and run-time systems. Averaging over a body of 43 C and Fortran programs, ESP branch prediction results in a miss rate of 20%, as compared with the 25% miss rate obtained using the best existing program-based heuristics.
18962931
Analyzing xfig using the Rigi tool suite
An experiment is conducted on how well expert users of program comprehension tools are able to perform specific program understanding and maintenance tasks on the xfig drawing program using these tools. The paper reports on the experiences of the users of the Rigi reverse engineering tool suite. Rigi is an interactive, visual tool designed to help developers better understand and redocument their software. Rigi includes parsers to read the source code of the subject software and produce a graph of extracted artifacts such as procedures, variables, calls, and data accesses. To manage the complexity of the graph, an editor allows the software engineer to automatically or manually collapse related artifacts into subsystems. These subsystems typically represent concepts such as abstract data types or personnel assignments. The created hierarchy can be navigated, analyzed, and presented using various automatic or user-guided graphical layouts.
206630030
Statistical Gesture Recognition Through Modelling of Parameter Trajectories
The recognition of human gestures is a challenging problem that can contribute to a natural man-machine interface. In this paper, we present a new technique for gesture recognition. Gestures are modelled as temporal trajectories of parameters. Local sub-sequences of these trajectories are extracted and used to define an orthogonal space using principal component analysis. In this space the probabilistic density function of the training trajectories is represented by a multidimensional histogram, which builds the basis for the recognition. Experiments on three different recognition problems show the general utility of the approach.
15685117
MetaBank: A Knowledge-Base of Metaphoric Language Conventions
The frequent and conventional use of nonliteral language has been a major stumbling block for natural language processing systems since the early machine translation efforts. Metaphor, metonymy, and indirect speech acts are among the most troublesome phenomena. Recent computational efforts addressing these problems have taken an approach that emphasizes the use of systematic knowledge about nonliteral language conventions. We are currently engaged in an effort to supply this knowledge in the case of conventional metaphor. We are constructing MetaBank: an empirically derived and theoretically motivated knowledge-base of English metaphorical conventions. This article describes our three-part approach to the construction of MetaBank: the collection of on-line textual resources and databases of linguistic generalizations, the development of a methodology for analyzing these resources, and the construction of a knowledge-base based on the preceding analyses.
33984452
Using System Networks to Build Rhetorical Structures
We are currently engaged in a project to study the generation of instructional texts for common consumer devices. Our initial efforts have focused on an exhaustive corpus-based analysis of instruction booklets for cordless telephones. In this paper, we present our analysis of the way the various processes used in this domain are expressed. Our emphasis here is on the relationship between the surface grammatical coding of these processes and the underlying rhetorical structure of the text. This analysis has been formalized using a systemic-functional framework, with the resulting system networks forming the basis for the Imagene text generation system.