6908603
Exploring the limits of single-iteration clarification dialogs
Single-iteration clarification dialogs, as implemented in the TREC HARD track, represent an attempt to introduce interaction into ad hoc retrieval, while preserving the many benefits of large-scale evaluations. Although previous experiments have not conclusively demonstrated performance gains resulting from such interactions, it is unclear whether these findings speak to the nature of clarification dialogs, or simply the limitations of current systems. To probe the limits of such interactions, we employed a human intermediary to formulate clarification questions and exploit user responses. In addition to establishing a plausible upper bound on performance, we were also able to induce an "ontology of clarifications" to characterize human behavior. This ontology, in turn, serves as the input to a regression model that attempts to determine which types of clarification questions are most helpful. Our work can serve to inform the design of interactive systems that initiate user dialogs.
12592432
User simulations for evaluating answers to question series
Recently, question series have become one focus of research in question answering. These series are comprised of individual factoid, list, and ''other'' questions organized around a central topic, and represent abstractions of user-system dialogs. Existing evaluation methodologies have yet to catch up with this richer task model, as they fail to take into account contextual dependencies and different user behaviors. This paper presents a novel simulation-based methodology for evaluating answers to question series that addresses some of these shortcomings. Using this methodology, we examine two different behavior models: a ''QA-styled'' user and an ''IR-styled'' user. Results suggest that an off-the-shelf document retrieval system is competitive with state-of-the-art QA systems in this task. Advantages and limitations of evaluations based on user simulations are also discussed.
22235434
The role of context in question answering systems
Despite recent advances in natural language question an-swering technology, the problem of designing effective user interfaces has been largely unexplored. We conducted a user study to investigate the problem and discovered that overall, users prefer a paragraph-sized chunk of text over just an exact phrase as the answer to their questions. Fur-thermore, users generally prefer answers embedded in con-text, regardless of the perceived reliability of the source documents. When users research a topic, increasing the amount of text returned to users significantly decreases the number of queries that they pose to the system, suggesting that users utilize supporting text to answer related ques-tions. We believe that these results can serve to guide future developments in question answering user interfaces.
9391992
Web question answering: is more always better?
This paper describes a question answering system that is designed to capitalize on the tremendous amount of data that is now available online. Most question answering systems use a wide variety of linguistic resources. We focus instead on the redundancy available in large corpora as an important resource. We use this redundancy to simplify the query rewrites that we need to use, and to support answer mining from returned snippets. Our system performs quite well given the simplicity of the techniques being utilized. Experimental results show that question answering accuracy can be greatly improved by analyzing more and more matching passages. Simple passage ranking and n-gram extraction techniques work well in our system making it efficient to use with many backend retrieval engines.
6918069
Methods for automatically evaluating answers to complex questions
Evaluation is a major driving force in advancing the state of the art in language technologies. In particular, methods for automatically assessing the quality of machine output is the preferred method for measuring progress, provided that these metrics have been validated against human judgments. Following recent developments in the automatic evaluation of machine translation and document summarization, we present a similar approach, implemented in a measure called POURPRE, an automatic technique for evaluating answers to complex questions based on n-gram co-occurrences between machine output and a human-generated answer key. Until now, the only way to assess the correctness of answers to such questions involves manual determination of whether an information “nugget” appears in a system's response. The lack of automatic methods for scoring system output is an impediment to progress in the field, which we address with this work. Experiments with the TREC 2003, TREC 2004, and TREC 2005 QA tracks indicate that rankings produced by our metric correlate highly with official rankings, and that POURPRE outperforms direct application of existing metrics.
16198992
Evaluation of resources for question answering evaluation
Controlled and reproducible laboratory experiments, enabled by reusable test collections, represent a well-established methodology in modern information retrieval research. In order to confidently draw conclusions about the performance of different retrieval methods using test collections, their reliability and trustworthiness must first be established. Although such studies have been performed for ad hoc test collections, currently available resources for evaluating question answering systems have not been similarly analyzed. This study evaluates the quality of answer patterns and lists of relevant documents currently employed in automatic question answering evaluation, and concludes that they are not suitable for post-hoc experimentation. These resources, created from runs submitted by TREC QA track participants, do not produce fair and reliable assessments of systems that did not participate in the original evaluations. Potential solutions for addressing this evaluation gap and their shortcomings are discussed.
16556041
Segmented fuzzy checkpointing for main memory databases
'Toscane' can be used as a decorative pot plant that flowers early and abundantly. The plant can be sold in different sizes from approximately 40 cm to 80 cm in height. The plant produces abundant flowers that enlarge in size as the plant becomes bigger. The decorative leaves are glossy, dark green, coriaceous and extremely durable. 'Toscane' produces bright, long-lived and large flowers that are held above the foliage. The heart-shaped flowers maintain their original red color as they mature.
14212803
Merging Databases Under Constraints
The problem of integrating information from conflicting sources comes up in many current applications, such as cooperative information systems, heterogeneous databases, and multiagent systems. We model this by the operation of merging first-order theories. We propose a formal semantics for this operation and show that it has desirable properties, including abiding by majority rule in case of conflict and syntax independence. We apply our semantics to the special case when the theories to be merged represent relational databases under integrity constraints. We then present a way of merging databases that have different or conflicting schemas caused by problems such as synonyms, homonyms or type conflicts mentioned in the schema integration literature.
15165046
A Classification Tree Based on Discriminant Functions
The classification problem is an important topic in knowledge discovery and machine learning. Traditional classification tree methods and their improvements have been discussed widely. This work proposes a new approach to construct decision trees based on discriminani functions which are learned using genetic programming. A discriminant function is a mathematical function for classifying data into a specific class. To learn discriminant functions effectively and efficiently, a distance-based fitness function for genetic programming is designed. After the set of discriminant functions for all classes is generated, a classifier is created as a binary decision tree with the Z-value measure to resolve the problem of ambiguity among discriminant functions. Several popular datasets from the UCI Repository were selected to illustrate the effectiveness of the proposed classifiers by comparing with previous methods. The results show that the proposed classification tree demonstrates high accuracy on the selected datasets.
1721157
Modeling the constraints of human hand motion
Hand motion capture is one of the most important parts of gesture interfaces. Many current approaches to this task generally involve a formidable nonlinear optimization problem in a large search space. Motion capture can be achieved more cost-efficiently when considering the motion constraints of a hand. Although some constraints can be represented as equalities or inequalities, there exist many constraints which cannot be explicitly represented. In this paper, we propose a learning approach to model the hand configuration space directly. The redundancy of the configuration space can be eliminated by finding a lower-dimensional subspace of the original space. Finger motion is modeled in this subspace based on the linear behavior observed in the real motion data collected by a CyberGlove. Employing the constrained motion model, we are able to efficiently capture finger motion from video inputs. Several experiments show that our proposed model is helpful for capturing articulated motion.
6272838
A visual language for sketching large and complex interactive designs
Informal, sketch-based design tools closely match the work practices of user interface designers. Current tools, however, are limited in the size and complexity of interaction that can be specified. We have created an advanced sketch-based visual language that allows for easy prototyping of large, complex interactive designs. In its current embodiment in the denim web design tool, the visual language allows designers to sketch reusable components for recurring page elements, such as navigation bars, as well as conditionals to illustrate and test transitions that depend on a user's input. Designers can also specify sites that accept richer user input than simple clicking. Our informal evaluation shows that these features allow designers with little programming experience to quickly create prototypes of large, complex web sites while still working inside an informal, sketch-based environment