13621180
Natural Language Annotations for the Semantic Web
Because the ultimate purpose of the Semantic Web is to help users locate, organize, and process information, we strongly believe that it should be grounded in the information access method humans are most comfortable with--natural language. However, the Resource Description Framework (RDF), the foundation of the Semantic Web, was designed to be easily processed by computers, not humans. To render RDF friendlier to humans, we propose to augment it with natural language annotations, or metadata written in everyday language. We argue that natural language annotations are not only intuitive and effective, but can also accelerate the pace with which the Semantic Web is being adopted. We demonstrate the use of natural language annotations from within Haystack, an end user Semantic Web platform that also serves as a testbed for our ideas. In addition to a prototype Semantic Web question answering system, we describe other opportunities for marrying natural language and Semantic Web technology.
12592432
User simulations for evaluating answers to question series
Recently, question series have become one focus of research in question answering. These series are comprised of individual factoid, list, and ''other'' questions organized around a central topic, and represent abstractions of user-system dialogs. Existing evaluation methodologies have yet to catch up with this richer task model, as they fail to take into account contextual dependencies and different user behaviors. This paper presents a novel simulation-based methodology for evaluating answers to question series that addresses some of these shortcomings. Using this methodology, we examine two different behavior models: a ''QA-styled'' user and an ''IR-styled'' user. Results suggest that an off-the-shelf document retrieval system is competitive with state-of-the-art QA systems in this task. Advantages and limitations of evaluations based on user simulations are also discussed.
16198992
Evaluation of resources for question answering evaluation
Controlled and reproducible laboratory experiments, enabled by reusable test collections, represent a well-established methodology in modern information retrieval research. In order to confidently draw conclusions about the performance of different retrieval methods using test collections, their reliability and trustworthiness must first be established. Although such studies have been performed for ad hoc test collections, currently available resources for evaluating question answering systems have not been similarly analyzed. This study evaluates the quality of answer patterns and lists of relevant documents currently employed in automatic question answering evaluation, and concludes that they are not suitable for post-hoc experimentation. These resources, created from runs submitted by TREC QA track participants, do not produce fair and reliable assessments of systems that did not participate in the original evaluations. Potential solutions for addressing this evaluation gap and their shortcomings are discussed.
9391992
Web question answering: is more always better?
This paper describes a question answering system that is designed to capitalize on the tremendous amount of data that is now available online. Most question answering systems use a wide variety of linguistic resources. We focus instead on the redundancy available in large corpora as an important resource. We use this redundancy to simplify the query rewrites that we need to use, and to support answer mining from returned snippets. Our system performs quite well given the simplicity of the techniques being utilized. Experimental results show that question answering accuracy can be greatly improved by analyzing more and more matching passages. Simple passage ranking and n-gram extraction techniques work well in our system making it efficient to use with many backend retrieval engines.
6918069
Methods for automatically evaluating answers to complex questions
Evaluation is a major driving force in advancing the state of the art in language technologies. In particular, methods for automatically assessing the quality of machine output is the preferred method for measuring progress, provided that these metrics have been validated against human judgments. Following recent developments in the automatic evaluation of machine translation and document summarization, we present a similar approach, implemented in a measure called POURPRE, an automatic technique for evaluating answers to complex questions based on n-gram co-occurrences between machine output and a human-generated answer key. Until now, the only way to assess the correctness of answers to such questions involves manual determination of whether an information “nugget” appears in a system's response. The lack of automatic methods for scoring system output is an impediment to progress in the field, which we address with this work. Experiments with the TREC 2003, TREC 2004, and TREC 2005 QA tracks indicate that rankings produced by our metric correlate highly with official rankings, and that POURPRE outperforms direct application of existing metrics.
22235434
The role of context in question answering systems
Despite recent advances in natural language question an-swering technology, the problem of designing effective user interfaces has been largely unexplored. We conducted a user study to investigate the problem and discovered that overall, users prefer a paragraph-sized chunk of text over just an exact phrase as the answer to their questions. Fur-thermore, users generally prefer answers embedded in con-text, regardless of the perceived reliability of the source documents. When users research a topic, increasing the amount of text returned to users significantly decreases the number of queries that they pose to the system, suggesting that users utilize supporting text to answer related ques-tions. We believe that these results can serve to guide future developments in question answering user interfaces.
6272838
A visual language for sketching large and complex interactive designs
Informal, sketch-based design tools closely match the work practices of user interface designers. Current tools, however, are limited in the size and complexity of interaction that can be specified. We have created an advanced sketch-based visual language that allows for easy prototyping of large, complex interactive designs. In its current embodiment in the denim web design tool, the visual language allows designers to sketch reusable components for recurring page elements, such as navigation bars, as well as conditionals to illustrate and test transitions that depend on a user's input. Designers can also specify sites that accept richer user input than simple clicking. Our informal evaluation shows that these features allow designers with little programming experience to quickly create prototypes of large, complex web sites while still working inside an informal, sketch-based environment
2759655
Cha-Cha: A System for Organizing Intranet Search Results
Although search over World Wide Web pages has recently received much academic and commercial attention, surprisingly little research has been done on how to search the web pages within large, diverse intranets. Intranets contain the information associated with the internal workings of an organization. 
 
A standard search engine retrieves web pages that fall within a widely diverse range of information contexts, but presents these results uniformly, in a ranked list. As an alternative, the Cha-Cha system organizes web search results in such a way as to reflect the underlying structure of the intranet. In our approach, an "outline" or "table of contents" is created by first recording the shortest paths in hyperlinks from root pages to every page within the web intranet. After the user issues a query, these shortest paths are dynamically combined to form a hierarchical outline of the context in which the search results occur. The system is designed to be helpful for users with a wide range of computer skills. Preliminary user study and survey results suggest that some users find the resulting structure more helpful than the standard retrieval results display for intranet search.
14394566
Integration, verification and layout of a complex multimedia SOC
We present our experience of designing a single-chip controller for an advanced digital still camera from specification all the way to mass production. The process involves collaboration with camera system designer, IP vendors, EDA vendors, silicon wafer foundry, package and testing houses, and camera maker. We also co-work with academic research groups to develop a JPEG codec IP and memory BIST and SOC testing methodology. We cover the problems encountered, our solutions, and lessons learned.
6644016
Multistar Decomposition of Complete Multigraphs
546802
A Low-Cost Checkpointing Technique for Distributed Databases
For distributed databases, checkpointing is used to ensure an efficient way to perform global reconstruction. However, the need for global reconstruction is infrequent. Most current checkpointing approaches for distributed databases are too expensive during run time. Some of them allow the checkpointing process to run in parallel with normal transactions at the cost of more data and resource contention, which in turn causes longer response time for normal transactions. Thus, an efficient way to checkpoint distributed databases is needed to avoid degrading the system performance. This paper presents a low-cost solution, called Loosely Synchronized Local Fuzzy Checkpointing (LSLFC), to these problems. LSLFC supports global reconstruction, and our performance study shows that LSLFC has little overhead during run time.