1566611
Managing evaluation goals for training
goals and are characteristically underresourced. In this article, we describe a framework for managing training evaluation in such contexts. We illustrate by example the feasibility of taking a broad approach toward training evaluation. Scriven [21] introduced the terms formative and summative in his classic work on the objectives and processes of instructional evaluation. Formative evaluation seeks to identify aspects of a design that can be improved. A typical implementation would be having users think aloud as they work through a tutorial to produce a real-time commentary on the nature of the learning experience. Each comment is tightly coupled to a specific design feature, facilitating the identification of problems and reasoning about design solutions. Summative evaluation seeks to gauge a design product. A typical implementation would be measuring time-on-task and error rate to guide decision-making about whether a training system (or the software it supports) should be produced and marketed, or purchased. Formative evaluation work succeeds to the extent that it identifies priorities for redesign and refinement, and all the more to the extent that it provides concrete guidance as to how that redesign and refinement should be executed. Summative evaluation work succeeds to the extent that it allows a design product to be located on various measurement scales with respect to other design products; the stronger the scale (i.e., ratio scale versus interval scale versus ordinal scale versus nominal scale), the better the summative evaluation. Scriven [21] also distinguished two basic types of evaluation method, intrinsic and pay-off :
15595014
Notification and awareness: synchronizing task-oriented collaborative activity
People working collaboratively must establish and maintain awareness of one another's intentions, actions and results. Notification systems typically support awareness of the presence, tasks and actions of collaborators, but they do not adequately support awareness of persistent and complex activities. We analysed awareness breakdowns in use of our Virtual School system--stemming from problems related to the collaborative situation, group, task and tool support--to motivate the concept of activity awareness. Activity awareness builds on prior conceptions of social and action awareness, but emphasizes the importance of activity context factors like planning and coordination. This work suggests design strategies for notification systems to better support collaborative activity.
148806746
Human-computer interaction: psychology as a science of design.
2269827
Training wheels in a user interface
New users of high-function application systems can become frustrated and confused by the errors they make in the early stages of learning. A training interface for a commercial word processor was designed to make typical and troublesome error states “unreachable,” thus eliminating the sources of some new-user learning problems. Creating a training environment from the basic function of the system itself afforded substantially faster learning coupled with better learning achievement and better performance on a comprehension post-test. A control group spent almost a quarter of their time recovering from the error states that the training interface blocked off. We speculate on how this training strategy might be refined, and more generally, on how function should be organized in a user interface.
9978523
Community collective efficacy: structure and consequences of perceived capacities in the Blacksburg Electronic Village
Bandura's social cognitive construct "perceived self-efficacy" has been used widely to understand individual behavior as a function of domain-specific beliefs about personal capacities. Collective efficacy is the extension of the self-efficacy construct to organizations and groups; it refers to beliefs about collective capacities in specific domains. Our research is investigating the use of collective efficacy in understanding attitudes and behaviors of members of proximal residential communities with respect to issues like attachment, engagement, and sociality, specifically as modulated by use of the Internet and community networks. This paper describes our analysis of the structure and external validity of the collective efficacy construct.
14643603
Making use is more than a matter of task analysis
I am pleased and honored that such an impressive group of commentators considered my book Making Use (Carroll, 2000) worthy of discussion and analysis. I would not expect, nor even prefer really, to have everyone agree with me. What would bother me, what would bother any author, is to not be read at all. Dan Diaper’s critical review and the commentaries by David Benyon/Catriona Macaulay, Tom Carey, and Fabio Paterno, will help to make that less likely! Many important themes are raised in Diaper (2002) review and in the commentaries. In this response I want to concentrate on refining a few of the main issues and distinctions. First, I briefly consider contrasts in values. I think that preferences for scenario-based design versus traditional task analytic methods reflect differences in the underlying world views and values of designers and researchers. To some extent, therefore, the issues in this discussion are not just technical, but paradigmatic and moral. Second, I consider the state of design practices in HCI and its implications for where onus lies in contemporary methodological argumentation and for what sorts of data are pertinent and required to address outstanding methodological issues. Finally, I consider the question of how task analysis, in the sense of structured task decomposition, might be reconstructed in the emerging paradigm of scenario-based design.
5587138
Automatic Extraction of Subcategorization from Corpora
We describe a novel technique and implemented system for constructing a subcategorization dictionary from textual corpora. Each dictionary entry encodes the relative frequency of occurrence of a comprehensive set of subcategorization classes for English. An initial experiment, on a sample of 14 verbs which exhibit multiple complementation patterns, demonstrates that the technique achieves accuracy comparable to previous approaches, which are all limited to a highly restricted set of subcategorization classes. We also demonstrate that a subcategorization dictionary built with the system improves the accuracy of a parser by an appreciable amount1.
2252135
Disambiguating Nouns, Verbs, and Adjectives Using Automatically Acquired Selectional Preferences
Selectional preferences have been used by word sense disambiguation (WSD) systems as one source of disambiguating information. We evaluate WSD using selectional preferences acquired for English adjectivenoun, subject, and direct object grammatical relationships with respect to a standard test corpus. The selectional preferences are specific to verb or adjective classes, rather than individual word forms, so they can be used to disambiguate the co-occurring adjectives and verbs, rather than just the nominal argument heads. We also investigate use of the one-senseper-discourse heuristic to propagate a sense tag for a word to other occurrences of the same word within the current document in order to increase coverage. Although the preferences perform well in comparison with other unsupervised WSD systems on the same corpus, the results show that for many applications, further knowledge sources would be required to achieve an adequate level of accuracy and coverage. In addition to quantifying performance, we analyze the results to investigate the situations in which the selectional preferences achieve the best precision and in which the one-sense-per-discourse heuristic increases performance.
15057511
Completing design in use: closing the appropriation cycle
Users appropriate a technology innovation as they adapt and adapt to its capabilities. This paper argues that the appropriation of technology innovations – information and communication technologies such as devices and systems – is actually part of the design process. The design of a technology innovation is completed by users as they appropriate it. The contributions of the paper are to draw attention to the crucial role played by users’ actions in completing the design process and to examine the implications for the design and implementation of technology innovations. The challenge for designers is to design malleable technologies that can be adapted to users’ organisational, social and personal practices, and then to harvest users’ needs from the appropriated innovation in order to improve its design. For managers, trainers and IS staff involved in the implementation of innovations, the challenge is to encourage and support users’ appropriation activities.
14983459
Fixed vs. Dynamic Sub-Transfer in Reinforcement Learning
We survey various task transfer methods in Qlearning and present a variation on fixed sub-transfer which we call dynamic sub-transfer. We discuss the benefits and drawbacks of dynamic sub-transfer as compared with the other transfer methods, and we describe qualitatively the situations where this method would be preferred over the fixed version of sub-transfer. We test this method against several other transfer methods in a simple three room grid world where portions of the source’s policy are relevant to the target task and other portions are not. In this situation we found that dynamic sub-transfer converged to the optimal solution, avoiding the suboptimality inherent in fixed sub-transfer, while also avoiding some of the convergence problems often experienced by fixed sub-transfer.
14287805
A Formalism and Environment for the Development of a Large Grammar of English
Natural language grammars with large coverage are typically the result of many person-years of effort, working with clumsy formalisms and sub-optimal software support for grammar development. This paper describes our approach to the task of writing a substantial grammar, as part of a collaboration to produce a general purpose morphological and syntactic analyser for English. The grammatical formalism we have developed for the task is a metagrammatical notation which is a more expressive and computationally tractable variant of Generalized Phrase Structure Grammar. We have also implemented a software system which provides a highly integrated and very powerful set of tools for developing and managing a large grammar based on this notation. The system provides a grammarian with an environment which we have found to be essential for rapid but successful production of a substantial grammar.