39682007
TreeWrapper: Automatic Data Extraction Based on Tree Representation
This paper introduces a new algorithm that learns to extract data from Web pages with relatively regular data structures. Current existing systems require training on either manually labelled pages or at least two similar unlabelled pages, and they often have difficulties on handling Web pages with complex formats such as nested tables or lists. Our previous system AutoWrapper does not need any training and can automatically extract data from any single page. This paper improves AutoWrapper by handling nested structures and finding multiple regular data areas. The main contributions include a tree-based representation for Web pages, an expressive language for representing information extraction patterns, and a learning algorithm that automatically detects regular data areas by finding similar sub-trees.
1036423
Investigation of Brood Size in GP with Brood Recombination Crossover for Object Recognition
This paper describes an approach to the investigation of brood size in the brood recombination crossover method in genetic programming for object recognition problems. The approach is examined and compared with the standard crossover operator on three object classification problems of increasing difficulty. The results suggest that the brood recombination method outperforms the standard crossover operator for all the problems in terms of the classification accuracy. As the brood size increases, the system effective performance can be improved. When it exceeds a certain point, however, the effective performance will not be improved and the system will become less efficient.
43193547
Knowledge-Based Information Agents
This paper explains our approach to building knowledge-based information agents. Instead of building an agent from scratch, we advocate building an agent by adding knowledge to a framework. Knowledge is classified into three categories - general knowledge, domain specific knowledge and site specific knowledge - which enables knowledge reuse and sharing. The paper details the agent architecture, the components of each category of knowledge and the main functions of the framework.
1168192
Approximately Repetitive Structure Detection for Wrapper Induction
In recent years, much work has been invested into automatically learning wrappers for information extraction from HTML tables and lists. Our research has focused on a system that can learn a wrapper from a single unlabelled page. An essential step is to locate the tabular data within the page. This is not trivial when the structures of data tuples are similar but not identical. In this paper we describe an algorithm that can automatically detect approximate repetitive structures within one sequence. The algorithm does not rely on any domain knowledge or HTML heuristics and it can be used in detecting repetitive patterns and hence to learn wrappers from a single unlabeled tabular page.
26423041
GP for Object Classification: Brood Size in Brood Recombination Crossover
The brood size plays an important role in the brood recombination crossover method in genetic programming. However, there has not been any thorough investigation on the brood size and the methods for setting this size have not been effectively examined. This paper investigates a number of new developments of brood size in the brood recombination crossover method in GP. We first investigate the effect of different fixed brood sizes, then construct three dynamic models for setting the brood size. These developments are examined and compared with the standard crossover operator on three object classification problems of increasing difficulty. The results suggest that the brood recombination methods with all the new developments outperforms the standard crossover operator for all the problems. As the brood size increases, the system effective performance can be improved. When it exceeds a certain point, however, the effective performance will not be improved and the system will become less efficient. Investigation of three dynamic models for the brood size reveals that a good variable brood size which is dynamically set with the number of generations can further improve the system performance over the fixed brood sizes.
40362881
A Knowledge Learning Approach to Information Extraction from Multiple Text Based Web Sites
This paper describes a learning/adaptive approach to automatically building knowledge bases for information extraction from text based web pages. A frame based representation is introduced to represent domain knowledge as knowledge unit frames. A frame learning algorithm is developed to automatically learn knowledge unit frames from training examples. Some training examples can be obtained by automatically parsing a number of tabular web pages in the same domain, which greatly reduced the amount of time consuming manual work. This approach was investigated on ten web sites of real estate advertisements and car advertisements and nearly all the information was successfully extracted with very few false alarms. These results suggest that both the knowledge unit frame representation and the frame learning algorithm work well, domain specific knowledge bases can be learned from training examples, and the domain specific knowledge base can be used for information extraction from flexible text-based semi-structured Web pages on multiple Web sites. The investigation of the knowledge representation on five other domains suggests that this approach can be easily applied to other domains by simply changing the training examples.
46154955
A Novel Typical-Sample-Weighted Clustering Algorithm for Large Data Sets
In the field of cluster analysis, most of existing algorithms are developed for small data sets, which cannot effectively process the large data sets encountered in data mining. Moreover, most clustering algorithms consider the contribution of each sample for classification uniformly. In fact, different samples should be of different contribution for clustering result. For this purpose, a novel typical-sample-weighted clustering algorithm is proposed for large data sets. By the atom clustering, the new algorithm extracts the typical samples to reduce the data amount. Then the extracted samples are weighted by their corresponding typicality and then clustered by the classical fuzzy c-means (FCM) algorithm. Finally, the Mahalanobis distance is employed to classify each original sample into obtained clusters. It is obvious that the novel algorithm can improve the speed and robustness of the traditional FCM algorithm. The experimental results with various test data sets illustrate the effectiveness of the proposed clustering algorithm.
206603960
FCM-Based Clustering Algorithm Ensemble for Large Data Sets
In the field of cluster analysis, most of the available algorithms were designed for small data sets, which cannot efficiently deal with large scale data set encountered in data mining. However, some sampling-based clustering algorithms for large scale data set cannot achieve ideal result. For this purpose, a FCM-based clustering ensemble algorithm is proposed. Firstly, it performs the atom clustering algorithm on the large data set. Then, randomly select a sample from each atom as representative to reduce the data amount. And the ensemble learning technique is used to improve the clustering performance. For the complex large data sets, the new algorithm has high classification speed and robustness. The experimental results illustrate the effectiveness of the proposed clustering algorithm.
30258061
A Speculative Strategy
In this contribution we propose a strategy which focuses on the game as well as on the opponent. Preference is given to the thoughts of the opponent, so that the strategy might be speculative. We describe a generalization of OM search, called (D, d)-OM search, where D stands for the depth of search by the player and d for the opponent's depth of search. A difference in search depth can be exploited by deliberately chosing a suboptimal move in order to gain a larger advantage than when playing the optimal best move. The idea is that the opponent does not see the variant in sufficiently deep detail. Simulations using a game-tree model including an opponent model as well as experiments in the domain of Othello confirm the effectiveness of the proposed strategy.
11815466
A Wavelet Based Image Watermarking Scheme
A novel robust watermarking scheme based on wavelet transform is presented in this paper. The watermark is embedded into significant triplet of wavelet coefficients in the three lowest detail subband images. The embedding algorithm only needs to modify a small amount of the wavelet coefficients, which leads to very small distortion and guarantees the visual quality of the watermarked image. The performance of the proposed algorithm is tested on a series of normal attacks in image processing, such as JPEG compression, Gaussian noise addition, median filter, etc. The corresponding experimental results show that the proposed watermarking scheme has better performance.
33275321
A Shot Boundary Detection Method for News Video Based on Rough Sets and Fuzzy Clustering
With the rapid growing amount of multimedia, content-based infomation retrieval has become more and more important. As a crucial step in content-based news video indexing and retrieval system, shot boundary detection attracts much more research interests in recent years. To partition news video into shots, many metrics were constructed to measure the similarity among video frames based on all the available video features. However, too many features will reduce the efficiency of the shot boundary detection. Therefore, it is necessary to perform feature reduction for every decision of shot boundary. For this purpose, the classification method based on rough sets and fuzzy c-means clustering for feature reduction and rule generation is proposed. According to the particularity of news scenes, shot transition can be divided into three types: cut transition, gradual transition and no transition. The efficacy of the proposed method is extensively tested on more than 2 h of news programs and 98.0% recall with 96.6% precision have been achieved.