426181
Omnibase: Uniform Access to Heterogeneous Data for Question Answering
Although the World Wide Web contains a tremendous amount of information, the lack of uniform structure makes finding the right knowledge difficult. A solution is to turn the Web into a "virtual database" and to access it through natural language. We built Omnibase, a system that integrates heterogeneous data sources using an object-property-value model. With the help of Omnibase, our START natural language system can now access numerous heterogeneous data sources on the Web in a uniform manner, and answers millions of user questions with high precision.
9391992
Web question answering: is more always better?
This paper describes a question answering system that is designed to capitalize on the tremendous amount of data that is now available online. Most question answering systems use a wide variety of linguistic resources. We focus instead on the redundancy available in large corpora as an important resource. We use this redundancy to simplify the query rewrites that we need to use, and to support answer mining from returned snippets. Our system performs quite well given the simplicity of the techniques being utilized. Experimental results show that question answering accuracy can be greatly improved by analyzing more and more matching passages. Simple passage ranking and n-gram extraction techniques work well in our system making it efficient to use with many backend retrieval engines.
12592432
User simulations for evaluating answers to question series
Recently, question series have become one focus of research in question answering. These series are comprised of individual factoid, list, and ''other'' questions organized around a central topic, and represent abstractions of user-system dialogs. Existing evaluation methodologies have yet to catch up with this richer task model, as they fail to take into account contextual dependencies and different user behaviors. This paper presents a novel simulation-based methodology for evaluating answers to question series that addresses some of these shortcomings. Using this methodology, we examine two different behavior models: a ''QA-styled'' user and an ''IR-styled'' user. Results suggest that an off-the-shelf document retrieval system is competitive with state-of-the-art QA systems in this task. Advantages and limitations of evaluations based on user simulations are also discussed.
6918069
Methods for automatically evaluating answers to complex questions
Evaluation is a major driving force in advancing the state of the art in language technologies. In particular, methods for automatically assessing the quality of machine output is the preferred method for measuring progress, provided that these metrics have been validated against human judgments. Following recent developments in the automatic evaluation of machine translation and document summarization, we present a similar approach, implemented in a measure called POURPRE, an automatic technique for evaluating answers to complex questions based on n-gram co-occurrences between machine output and a human-generated answer key. Until now, the only way to assess the correctness of answers to such questions involves manual determination of whether an information “nugget” appears in a system's response. The lack of automatic methods for scoring system output is an impediment to progress in the field, which we address with this work. Experiments with the TREC 2003, TREC 2004, and TREC 2005 QA tracks indicate that rankings produced by our metric correlate highly with official rankings, and that POURPRE outperforms direct application of existing metrics.
22235434
The role of context in question answering systems
Despite recent advances in natural language question an-swering technology, the problem of designing effective user interfaces has been largely unexplored. We conducted a user study to investigate the problem and discovered that overall, users prefer a paragraph-sized chunk of text over just an exact phrase as the answer to their questions. Fur-thermore, users generally prefer answers embedded in con-text, regardless of the perceived reliability of the source documents. When users research a topic, increasing the amount of text returned to users significantly decreases the number of queries that they pose to the system, suggesting that users utilize supporting text to answer related ques-tions. We believe that these results can serve to guide future developments in question answering user interfaces.
16198992
Evaluation of resources for question answering evaluation
Controlled and reproducible laboratory experiments, enabled by reusable test collections, represent a well-established methodology in modern information retrieval research. In order to confidently draw conclusions about the performance of different retrieval methods using test collections, their reliability and trustworthiness must first be established. Although such studies have been performed for ad hoc test collections, currently available resources for evaluating question answering systems have not been similarly analyzed. This study evaluates the quality of answer patterns and lists of relevant documents currently employed in automatic question answering evaluation, and concludes that they are not suitable for post-hoc experimentation. These resources, created from runs submitted by TREC QA track participants, do not produce fair and reliable assessments of systems that did not participate in the original evaluations. Potential solutions for addressing this evaluation gap and their shortcomings are discussed.
8007626
DENIM: finding a tighter fit between tools and practice for Web site design
Through a study of web site design practice, we observed that web site designers design sites at different levels of refinement—site map, storyboard, and individual page—and that designers sketch at all levels during the early stages of design. However, existing web design tools do not support these tasks very well. Informed by these observations, we created DENIM, a system that helps web site designers in the early stages of design. DENIM supports sketching input, allows design at different refinement levels, and unifies the levels through zooming. We performed an informal evaluation with seven professional designers and found that they reacted positively to the concept and were interested in using such a system in their work.
6272838
A visual language for sketching large and complex interactive designs
Informal, sketch-based design tools closely match the work practices of user interface designers. Current tools, however, are limited in the size and complexity of interaction that can be specified. We have created an advanced sketch-based visual language that allows for easy prototyping of large, complex interactive designs. In its current embodiment in the denim web design tool, the visual language allows designers to sketch reusable components for recurring page elements, such as navigation bars, as well as conditionals to illustrate and test transitions that depend on a user's input. Designers can also specify sites that accept richer user input than simple clicking. Our informal evaluation shows that these features allow designers with little programming experience to quickly create prototypes of large, complex web sites while still working inside an informal, sketch-based environment
7688101
(Not) Finding Rules in Time Series: A Surprising Result with Implications for Previous and Future Research
Time series data is perhaps the most frequently encountered type of data examined by the data mining community. Clustering is perhaps the most frequently used data mining algorithm, being useful in it’s own right as an exploratory technique, and also as a subroutine in more complex data mining algorithms such as rule discovery, indexing, summarization, anomaly detection, and classification. Given these two facts, it is hardly surprising that time series clustering has attracted much attention. The data to be clustered can be in one of two formats: many individual time series, or a single time series, from which individual time series are extracted with a sliding window. Given the recent explosion of interest in streaming data and online algorithms, the latter case has received much attention. In this work we make a surprising claim. Clustering of streaming time series is completely meaningless. More concretely, clusters extracted from streaming time series are forced to obey a certain constraint that is pathologically unlikely to be satisfied by any dataset, and because of this, the clusters extracted by any clustering algorithm are essentially random. While this constraint can be intuitively demonstrated with a simple illustration and is simple to prove, it has never appeared in the literature. An important implication of our findings is that widely used, time series rule discovery algorithms are producing random results!
8944311
Recent shortcuts: using recent interactions to support shared activities
We present an empirical study of teams that revealed the amount of extraneous individual work needed to enable collaboration: finding references to other people, finding files to attach to email, managing incoming email attachments, managing the variety of files used in shared activities, and tracking what work is owed to others. Much of this work involves finding recently accessed objects that are needed again in the user's current task focus. These observations led to the design of Recent Shortcuts, a tool to help support coordination by making recently used objects easily accessible. Recent Shortcuts enables quick access to people (including groups of people), received attachments, files, and file folders that the user interacted with recently for re-use in the user's current context. Recent Shortcuts makes it easy to use these objects across applications with no additional user input and minimal changes to the user's applications or work practice. Early user experiences with a working prototype led to an extension that integrates recently accessed objects across multiple devices.
11896424
Finding Time Series Discords Based on Haar Transform
The problem of finding anomaly has received much attention recently. However, most of the anomaly detection algorithms depend on an explicit definition of anomaly, which may be impossible to elicit from a domain expert. Using discords as anomaly detectors is useful since less parameter setting is required. Keogh et al proposed an efficient method for solving this problem. However, their algorithm requires users to choose the word size for the compression of subsequences. In this paper, we propose an algorithm which can dynamically determine the word size for compression. Our method is based on some properties of the Haar wavelet transformation. Our experiments show that this method is highly effective.