7723669
Better home shopping or new democracy?: evaluating community network outcomes
This is a perspective paper on community networks - socio-technical infrastructures supporting villages, towns, and neighborhoods. Community networking is well-established, world wide, and addresses critical societal issues, such as the “crisis of community” and the sociality of the Internet. However, community network projects have not emphasized evaluation. Relatively little is known about the economic, social, and psychological consequences of community networks for the individuals, groups, and communities served. Evaluating community networks is a momentous mutual opportunity for the development of CHI evaluation methodologies and for bringing technical CHI expertise to bear on societal issues.
1566611
Managing evaluation goals for training
goals and are characteristically underresourced. In this article, we describe a framework for managing training evaluation in such contexts. We illustrate by example the feasibility of taking a broad approach toward training evaluation. Scriven [21] introduced the terms formative and summative in his classic work on the objectives and processes of instructional evaluation. Formative evaluation seeks to identify aspects of a design that can be improved. A typical implementation would be having users think aloud as they work through a tutorial to produce a real-time commentary on the nature of the learning experience. Each comment is tightly coupled to a specific design feature, facilitating the identification of problems and reasoning about design solutions. Summative evaluation seeks to gauge a design product. A typical implementation would be measuring time-on-task and error rate to guide decision-making about whether a training system (or the software it supports) should be produced and marketed, or purchased. Formative evaluation work succeeds to the extent that it identifies priorities for redesign and refinement, and all the more to the extent that it provides concrete guidance as to how that redesign and refinement should be executed. Summative evaluation work succeeds to the extent that it allows a design product to be located on various measurement scales with respect to other design products; the stronger the scale (i.e., ratio scale versus interval scale versus ordinal scale versus nominal scale), the better the summative evaluation. Scriven [21] also distinguished two basic types of evaluation method, intrinsic and pay-off :
1921535
Interface design issues for advice-giving expert systems
Advice giving could become the first successful domain for intelligent interfaces.
52799724
Comparative usability evaluation: critical incidents and critical threads
148806746
Human-computer interaction: psychology as a science of design.
14643603
Making use is more than a matter of task analysis
I am pleased and honored that such an impressive group of commentators considered my book Making Use (Carroll, 2000) worthy of discussion and analysis. I would not expect, nor even prefer really, to have everyone agree with me. What would bother me, what would bother any author, is to not be read at all. Dan Diaper’s critical review and the commentaries by David Benyon/Catriona Macaulay, Tom Carey, and Fabio Paterno, will help to make that less likely! Many important themes are raised in Diaper (2002) review and in the commentaries. In this response I want to concentrate on refining a few of the main issues and distinctions. First, I briefly consider contrasts in values. I think that preferences for scenario-based design versus traditional task analytic methods reflect differences in the underlying world views and values of designers and researchers. To some extent, therefore, the issues in this discussion are not just technical, but paradigmatic and moral. Second, I consider the state of design practices in HCI and its implications for where onus lies in contemporary methodological argumentation and for what sorts of data are pertinent and required to address outstanding methodological issues. Finally, I consider the question of how task analysis, in the sense of structured task decomposition, might be reconstructed in the emerging paradigm of scenario-based design.
11544923
Unparsing RDF/XML
It is difficult to serialize an RDF graph as a humanly readable RDF/XML document. This paper describes the approach taken in Jena 1.2, in which a design pattern of guarded procedures invoked using top down recursive descent is used. Each procedure corresponds to a grammar rule; the guard makes the choice about the applicability of the production. This approach is seen to correspond closely to the design of an LL(k) parser, and a theoretical justification of this correspondence is found in universal algebra.
1557137
Encoding Frequency Information in Lexicalized Grammars
We address the issue of how to associate frequency information with lexicalized grammar formalisms, using Lexicalized Tree Adjoining Grammar as a representative framework. We consider systematically a number of alternative probabilistic frameworks, evaluating their adequacy from both a theoretical and empirical perspective using data from existing large treebanks. We also propose three orthogonal approaches for backing off probability estimates to cope with the large number of parameters involved.
1954
Corpus Annotation for Parser Evaluation
We describe a recently developed corpus annotation scheme for evaluating parsers that avoids shortcomings of current methods. The scheme encodes grammatical relations between heads and dependents, and has been used to mark up a new publicdomain corpus of naturally occurring English text. We show how the corpus can be used to evaluate the accuracy of a robust parser, and relate the corpus to extant resources.
12738746
Software support for practical grammar development
Even though progress in theoretical linguistics does not necessarily rely on the construction of working programs, a large proportion of current research in syntactic theory is facilitated by suitable computational tools. However, when natural language processing applications seek to draw on the results from new developments in theories of grammar, not only the nature of the tools has to change, but they face the challenge of reconciling the seemingly contradictory requirements of notational perspicuity and efficiency of performance. In this paper, we present a comparison and an evaluation of a number of software systems for grammar development, and argue that they are inadequate as practical tools for building wide-coverage grammars. We discuss a number of factors characteristic of this task, demonstrate how they influence the design of a suitable software environment, and describe the implementation of a system which has supported efficient development of a large computational grammar of English.
9629533
The Automatic Acquisition of Verb Subcategorisations and Their Impact on the Performance of an HPSG Parser
We describe the automatic acquisition of a lexicon of verb subcategorisations from a domain-specific corpus, and an evaluation of the impact this lexicon has on the performance of a “deep”, HPSG parser of English. We conducted two experiments to determine whether the empirically extracted verb stems would enhance the lexical coverage of the grammar and to see whether the automatically extracted verb subcategorisations would result in enhanced parser coverage. In our experiments, the empirically extracted verbs enhance lexical coverage by 8.5%. The automatically extracted verb subcategorisations enhance the parse success rate by 15% in theoretical terms and by 4.5% in practice. This is a promising approach for improving the robustness of deep parsing.