- Eric Nyberg, T. Mitamura. 2002. Evaluating QA Systems on Multiple Dimensions. Abstract: Question-answering systems are expanding beyond information retrieval and information extraction, to become fullfledged, complex NLP applications. In this paper we discuss the evaluation of question-answering systems as complex NLP systems, and suggest three different dimensions for evaluation: objective or information-based evaluation; subjective evaluation; and architectural evaluation. We also discuss the role of ambiguity resolution in QA systems, and how ambiguity resolution might be evaluated.
