17975975
The Inefficiency of Batch Training for Large Training Sets
Multilayer perceptrons are often trained using error backpropagation (BP). BP training can be done in either a batch or continuous manner. Claims have frequently been made that batch training is faster and/or more "correct" than continuous training because it uses a better approximation of the true gradient for its weight updates. These claims are often supported by empirical evidence on very small data sets. These claims are untrue, however, for large training sets. This paper explains why batch training is much slower than continuous training for large training sets. Various levels of semi-batch training used on a 20,000-instance speech recognition task show a roughly linear increase in training time required with an increase in batch size.
652801
The general inefficiency of batch training for gradient descent learning
Gradient descent training of neural networks can be done in either a batch or on-line manner. A widely held myth in the neural network community is that batch training is as fast or faster and/or more 'correct' than on-line training because it supposedly uses a better approximation of the true gradient for its weight updates. This paper explains why batch training is almost always slower than on-line training-often orders of magnitude slower-especially on large training sets. The main reason is due to the ability of on-line training to follow curves in the error surface throughout each epoch, which allows it to safely use a larger learning rate and thus converge with less iterations through the training data. Empirical results on a large (20,000-instance) speech recognition task and on 26 other learning tasks demonstrate that convergence can be reached significantly faster using on-line training than batch training, with no apparent difference in accuracy.
5565732
Instance Pruning Techniques
The nearest neighbor algorithm and its derivatives are often quite successful at learning a concept from a training set and providing good generalization on subsequent input vectors. However, these techniques often retain the entire training set in memory, resulting in large memory requirements and slow execution speed, as well as a sensitivity to noise. This paper provides a discussion of issues related to reducing the number of instances retained in memory while maintaining (and sometimes improving) generalization accuracy, and mentions algorithms other researchers have used to address this problem. It presents three intuitive noise-tolerant algorithms that can be used to prune instances from the training set. In experiments on 29 applications, the algorithm that achieves the highest reduction in storage also results in the highest generalization accuracy of the three methods.
5584468
Improved Heterogeneous Distance Functions
Instance-based learning techniques typically handle continuous and linear input values well, but often do not handle nominal input attributes appropriately. The Value Difference Metric (VDM) was designed to find reasonable distance values between nominal attribute values, but it largely ignores continuous attributes, requiring discretization to map continuous values into nominal values. This paper proposes three new heterogeneous distance functions, called the Heterogeneous Value Difference Metric (HVDM), the Interpolated Value Difference Metric (IVDM), and the Windowed Value Difference Metric (WVDM). These new distance functions are designed to handle applications with nominal attributes, continuous attributes, or both. In experiments on 48 applications the new distance metrics achieve higher classification accuracy on average than three previous distance functions on those datasets that have both nominal and continuous attributes.
18150918
A context-aware recognition survey for data collection using ubiquitous sensors in the home
Identifying what people do in the home can both inform ubiquitous computing application design decisions and provide training data to the machine learning algorithms used in their implementation. This paper describes an unsupervised technique in which contextual information gathered by ubiquitous sensors is used to help users label a multitude of anonymous activity episodes. This context-aware recognition survey is a game-like computer program in which users attempt to correctly guess which activity is happening after seeing a series of symbolic images that represent sensor values generated during the activity. We report a user study of the system, focusing on how well subjects were able to recognize their own activities, the activities of others, and counterfeits that did not correspond to any activity.
26356097
Adaptive Presentation and Navigation for Geospatial Imagery Tasks
Advances in technology for digital image capture have given rise to information overload problems in the geosciences and fields that rely on geospatial image retrieval. To help address such imagery information overload, our research is developing methods to extract and apply contextual knowledge relating user task goals to the images being used. As users analyze imagery retrieved to support specific tasks, multiple relevant images and salient image content can be captured together with task annotations to provide a basis for contextual knowledge management support. This paper describes how our environment for image interaction leverages captured task-based knowledge to support adaptive presentation and navigation in the space of available imagery.
921515
Combining CBR with Interactive Knowledge Acquisition, Manipulation and Reuse
Because of the complexity of aerospace design, intelligent systems to support and amplify the abilities of aerospace designers have the potential for profound impact on the speed and reliability of design generation. This article describes a framework for supporting the interactive capture of design cases and their application to new problems, illustrating the approach with a discussion of its use in a support system for aircraft design. The project integrates case-based reasoning with interactive tools for capturing expert design knowledge through "concept mapping." Concept mapping tools provide crucial functions for interactively generating and examining design cases and navigating their hierarchical structure, while CBR techniques provide capabilities to facilitate retrieval and to aid interactive adaptation of designs. The project aims simultaneously to develop a useful design aid and more generally to develop practical interactive approaches to fundamental issues of case acquisition and representation, context-sensitive retrieval, and case adaptation.
6770135
Empirical comparison of analog and digital auditory preprocessing for automatic speech recognition
Results from digital and analog filter bank preprocessors are compared in order to establish the validity of analog processing for automatic speech recognition (ASR) systems. Three systems are evaluated using speaker and context independent phoneme recognition tasks. The three ASR systems are identical except for the preprocessing techniques used to derive three signal representations: extraction of (1) the digital mel-frequency spectrum, (2) the mel-frequency spectrum from commercial discrete bandpass filters and (3) the exponential spectrum from analog VLSI bandpass filter bank. The discrete analog system exhibits a 38% increase in recognition accuracy over the digital preprocessing technique. The digital and analog VLSI-based techniques perform comparably (within 3% of each other).
15992473
Categorizing Case-Base Maintenance: Dimensions and Directions
Experience with the growing number of large-scale CBR systems has led to increasing recognition of the importance of case-base maintenance. Multiple researchers have addressed pieces of the case-base maintenance problem, considering such issues as maintaining consistency and controlling case-base growth. However, despite the existence of these cases of case-base maintenance, there is no general framework of dimensions for describing case-base maintenance systems. Such a framework would be useful both to understand the state of the art in case-base maintenance and to suggest new avenues of exploration by identifying points along the dimensions that have not yet been studied. This paper presents a first attempt at identifying the dimensions of case-base maintenance. It shows that characterizations along such dimensions can suggest avenues for future case-base maintenance research and presents initial steps exploring one of those avenues: identifying patterns of problems that require generalized revisions and addressing them with lazy updating.