6918069
Methods for automatically evaluating answers to complex questions
Evaluation is a major driving force in advancing the state of the art in language technologies. In particular, methods for automatically assessing the quality of machine output is the preferred method for measuring progress, provided that these metrics have been validated against human judgments. Following recent developments in the automatic evaluation of machine translation and document summarization, we present a similar approach, implemented in a measure called POURPRE, an automatic technique for evaluating answers to complex questions based on n-gram co-occurrences between machine output and a human-generated answer key. Until now, the only way to assess the correctness of answers to such questions involves manual determination of whether an information “nugget” appears in a system's response. The lack of automatic methods for scoring system output is an impediment to progress in the field, which we address with this work. Experiments with the TREC 2003, TREC 2004, and TREC 2005 QA tracks indicate that rankings produced by our metric correlate highly with official rankings, and that POURPRE outperforms direct application of existing metrics.
5024817
Answer Extraction, Semantic Clustering, and Extractive Summarization for Clinical Question Answering
This paper presents a hybrid approach to question answering in the clinical domain that combines techniques from summarization and information retrieval. We tackle a frequently-occurring class of questions that takes the form "What is the best drug treatment for X?" Starting from an initial set of MEDLINE citations, our system first identifies the drugs under study. Abstracts are then clustered using semantic classes from the UMLS ontology. Finally, a short extractive summary is generated for each abstract to populate the clusters. Two evaluations---a manual one focused on short answers and an automatic one focused on the supporting abstract---demonstrate that our system compares favorably to PubMed, the search system most widely used by physicians today.
16344930
The role of knowledge in conceptual retrieval: a study in the domain of clinical medicine
Despite its intuitive appeal, the hypothesis that retrieval at the level of "concepts" should outperform purely term-based approaches remains unverified empirically. In addition, the use of "knowledge" has not consistently resulted in performance gains. After identifying possible reasons for previous negative results, we present a novel framework for "conceptual retrieval" that articulates the types of knowledge that are important for information seeking. We instantiate this general framework in the domain of clinical medicine based on the principles of evidence-based medicine (EBM). Experiments show that an EBM-based scoring algorithm dramatically outperforms a state-of-the-art baseline that employs only term statistics. Ablation studies further yield a better understanding of the performance contributions of different components. Finally, we discuss how other domains can benefit from knowledge-based approaches.
13185193
Question answering from the web using knowledge annotation and knowledge mining techniques
We present a strategy for answering fact-based natural language questions that is guided by a characterization of real-world user queries. Our approach, implemented in a system called Aranea, extracts answers from the Web using two different techniques: knowledge annotation and knowledge mining. Knowledge annotation is an approach to answering large classes of frequently occurring questions by utilizing semi\-structured and structured Web sources. Knowledge mining is a statistical approach that leverages massive amounts of Web data to overcome many natural language processing challenges. We have integrated these two different paradigms into a question answering system capable of providing users with concise answers that directly address their information needs.
1116766
Leveraging Reusability: Cost-Effective Lexical Acquisition for Large-Scale Ontology Translation
Thesauri and ontologies provide important value in facilitating access to digital archives by representing underlying principles of organization. Translation of such resources into multiple languages is an important component for providing multilingual access. However, the specificity of vocabulary terms in most ontologies precludes fully-automated machine translation using general-domain lexical resources. In this paper, we present an efficient process for leveraging human translations when constructing domain-specific lexical resources. We evaluate the effectiveness of this process by producing a probabilistic phrase dictionary and translating a thesaurus of 56,000 concepts used to catalogue a large archive of oral histories. Our experiments demonstrate a cost-effective technique for accurate machine translation of large ontologies.
8985170
Consistent Belief Reasoning in the Presence of Inconsistency
Since everything is a consequence of an inconsistency, classical logics are not useful in modeling the reasoning of an agent who has inconsistent beliefs. In this paper, we differentiate consistent beliefs from inconsistent beliefs. We propose two belief operators Bc and B, standing for consistent belief and belief, respectively. Bc has the modus ponens property, by which the agent is able to reason with consistent beliefs as normal and draw consistent conclusions. B tolerates inconsistency, and by B the agent can reason about his inconsistent beliefs as well. The concept of consistent belief and our logical formalism for it are new, in that reasoning consistently about the information in an inconsistent knowledge base is possible. We also present a complete axiomatization for the logic and discuss the application of Bc and B in reasoning about implicit knowledge in a group of agents and eliminating inconsistency from a knowledge base.
14394566
Integration, verification and layout of a complex multimedia SOC
We present our experience of designing a single-chip controller for an advanced digital still camera from specification all the way to mass production. The process involves collaboration with camera system designer, IP vendors, EDA vendors, silicon wafer foundry, package and testing houses, and camera maker. We also co-work with academic research groups to develop a JPEG codec IP and memory BIST and SOC testing methodology. We cover the problems encountered, our solutions, and lessons learned.
12964519
Development and evaluation of emerging design patterns for ubiquitous computing
Design patterns are a format for capturing and sharing design knowledge. In this paper, we look at a new domain for design patterns, namely ubiquitous computing. The overall goal of this work is to aid practice by speeding up the diffusion of new interaction techniques and evaluation results from researchers, presenting the information in a form more usable to practicing designers. Towards this end, we have developed an initial and emerging pattern language for ubiquitous computing, consisting of 45 pre-patterns describing application genres, physical-virtual spaces, interaction and systems techniques for managing privacy, and techniques for fluid interactions. We evaluated the effectiveness of our pre-patterns with 16 pairs of designers in helping them design location-enhanced applications. We observed that our pre-patterns helped new and experienced designers unfamiliar with ubiquitous computing in generating and communicating ideas, and in avoiding design problems early in the design process.
3682795
Multibody grouping via orthogonal subspace decomposition
Multibody structure from motion could be solved by the factorization approach. However, the noise measurements would make the segmentation difficult when analyzing the shape interaction matrix. This paper presents an orthogonal subspace decomposition and grouping technique to approach such a problem. We decompose the object shape spaces into signal subspaces and noise subspaces. We show that the signal subspaces of the object shape spaces are orthogonal to each other. Instead of using the shape interaction matrix contaminated by noise, we introduce the shape signal subspace distance matrix for shape space grouping. Outliers could be easily identified by this approach. The robustness of the proposed approach lies in the fact that the shape space decomposition alleviates the influence of noise, and has been verified with extensive experiments.
207155257
Visually mining and monitoring massive time series
Moments before the launch of every space vehicle, engineering discipline specialists must make a critical go/no-go decision. The cost of a false positive, allowing a launch in spite of a fault, or a false negative, stopping a potentially successful launch, can be measured in the tens of millions of dollars, not including the cost in morale and other more intangible detriments. The Aerospace Corporation is responsible for providing engineering assessments critical to the go/no-go decision for every Department of Defense space vehicle. These assessments are made by constantly monitoring streaming telemetry data in the hours before launch. We will introduce VizTree, a novel time-series visualization tool to aid the Aerospace analysts who must make these engineering assessments. VizTree was developed at the University of California, Riverside and is unique in that the same tool is used for mining archival data and monitoring incoming live telemetry. The use of a single tool for both aspects of the task allows a natural and intuitive transfer of mined knowledge to the monitoring task. Our visualization approach works by transforming the time series into a symbolic representation, and encoding the data in a modified suffix tree in which the frequency and other properties of patterns are mapped onto colors and other visual properties. We demonstrate the utility of our system by comparing it with state-of-the-art batch algorithms on several real and synthetic datasets.