15575280
SPOT: A System for Detecting Projected Outliers From High-dimensional Data Streams
In this paper, we present a new technique, called stream projected ouliter detector (SPOT), to deal with outlier detection problem in high-dimensional data streams. SPOT is unique in a number of aspects. First, SPOT employs a novel window-based time model and decaying cell summaries to capture statistics from the data stream. Second, sparse subspace template (SST), a set of top sparse subspaces obtained by unsupervised and/or supervised learning processes, is constructed in SPOT to detect projected outliers effectively. Multi-Objective genetic algorithm (MOGA) is employed as an effective search method in unsupervised learning for finding outlying subspaces from training data. Finally, SST is able to carry out online self- evolution to cope with dynamics of data streams. This paper provides details on the motivation and technical challenges of detecting outliers from high-dimensional data streams, present an overview of SPOT, and give the plans for system demonstration of SPOT.
15427261
Anomaly detection in high-dimensional network data streams: A case study
In this paper, we study the problem of anomaly detection in high-dimensional network streams. We have developed a new technique, called Stream Projected Outlier deTector (SPOT), to deal with the problem of anomaly detection from high-dimensional data streams. We conduct a case study of SPOT in this paper by deploying it on 1999 KDD Intrusion Detection application. Innovative approaches for training data generation, anomaly classification and false positive reduction are proposed in this paper as well. Experimental results demonstrate that SPOT is effective in detecting anomalies from network data streams and outperforms existing anomaly detection methods.
16517186
A Color Indexing Scheme Using Two-level Clustering Processing for Effective and Efficient Image Retrieval
[Abstract]: In this paper, we present a clustering-based color indexing scheme for effective amd efficient image retrieval, which is essentially an exploration on the application of clustering technique to image retrieval. In our approach, the color features are clustered automatically using a color clustering algorithm twice (called two-level clustering processing) and two color feature summarizations are obtained, e.g. Local Color Centriods (LCCs) and Global Color Centroids (GCCs). Based upon LCCs and GCCs, a three-level R-tree is bulit for indexing database images and performing effecitve and efficient image retrieval. The experiments show that this indexing scheme is effective and efficient in performing image retrieval.
4732400
XAR-miner: efficient association rules mining for XML data
In this paper, we propose a framework, called XAR-Miner, for mining ARs from XML documents efficiently. In XAR-Miner, raw data in the XML document are first preprocessed to transform to either an Indexed Content Tree (IX-tree) or Multi-relational databases (Multi-DB), depending on the size of XML document and memory constraint of the system, for efficient data selection and AR mining. Task-relevant concepts are generalized to produce generalized meta-patterns, based on which the large ARs that meet the support and confidence levels are generated.
11328445
Gene Specific Co-regulation Discovery: An Improved Approach
Discovering gene co-regulatory relationships is a new but important research problem in DNA microarray data analysis. The problem of gene specific co-regulation discovery is to, for a particular gene of interest, called the target gene , identify its strongly co-regulated genes and the condition subsets where such strong gene co-regulations are observed. The study on this problem can contribute to a better understanding and characterization of the target gene. The existing method, using the genetic algorithm (GA), is slow due to its expensive fitness evaluation and long individual representation. In this paper, we propose an improved method for finding gene specific co-regulations. Compared with the current method, our method features a notably improved efficiency. We employ k NN Search Table to substantially speed up fitness evaluation in the GA. We also propose a more compact representation scheme for encoding individuals in the GA, which contributes to faster crossover and mutation operations. Experimental results with a real-life gene microarray data set demonstrate the improved efficiency of our technique compared with the current method.
37278978
PC-Filter: A Robust Filtering Technique for Duplicate Record Detection in Large Databases
In this paper, we will propose PC-Filter (PC stands for Partitio n Comparison), a robust data filter for approximately duplicate record detection in large databases. PC-Filter distinguishes itself from all of existing methods by using the notion of partition in duplicate detection. It first sorts the whole database and splits the sorted database into a number of record partitions. The Partition Comparison Graph (PCG) is then constructed by performing fast partition pruning. Finally, duplicate records are effectively detected by using internal and external partition comparison based on PCG. Four properties, used as heuristics, have been devised to achieve a remarkable efficiency of the filter based on triangle inequity of record similarity. PC-Filter is insensitive to the key used to sort the database, and can achieve a very good recall level that is comparable to that of the pair-wise record comparison method but only with a complexity of O(N 4/3). Equipping existing detection methods with PC-Filter, we are able to well solve the ”Key Selection” problem, the ”Scope Specification” problem and the ”Low Recall” problem that the existing methods suffer from.
3723162
Topic-Level Random Walk through Probabilistic Model
In this paper, we study the problem of topic-level random walk, which concerns the random walk at the topic level. Previously, several related works such as topic sensitive page rank have been conducted. However, topics in these methods were predefined, which makes the methods inapplicable to different domains. In this paper, we propose a four-step approach for topic-level random walk. We employ a probabilistic topic model to automatically extract topics from documents. Then we perform the random walk at the topic level. We also propose an approach to model topics of the query and then combine the random walk ranking score with the relevance score based on the modeling results. Experimental results on a real-world data set show that our proposed approach can significantly outperform the baseline methods of using language model and that of using traditional PageRank.
3408795
Arnetminer: expertise oriented search using social networks
Expertise Oriented Search (EOS) aims at providing comprehensive expertise analysis on data from distributed sources. It is useful in many application domains, for example, finding experts on a given topic, detecting the confliction of interest between researchers, and assigning reviewers to proposals. In this paper, we present the design and implementation of our expertise oriented search system, Arnetminer (http://www.arnetminer.net). Arnetminer has gathered and integrated information about a half-million computer science researchers from the Web, including their profiles and publications. Moreover, Arnetminer constructs a social network among these researchers through their co-authorship, and utilizes this network information as well as the individual profiles to facilitate expertise oriented search tasks. In particular, the co-authorship information is used both in ranking the expertise of individual researchers for a given topic and in searching for associations between researchers. We have conducted initial experiments on Arnetminer. Our results demonstrate that the proposed relevancy propagation expert finding method outperforms the method that only uses person local information, and the proposed two-stage association search on a large-scale social network is order of magnitude faster than the baseline method.
17362173
A New Heuristic Reduct Algorithm Base on Rough Sets Theory
Real world data sets usually have many features, which increases the complexity of data mining task. Feature selection, as a preprocessing step to the data mining, has been shown very effective in reducing dimensionality, removing irrelevant data, increasing learning accuracy, and improving comprehensibility. To find the optimal feature subsets is the aim of feature selection. Rough sets theory provides a mathematical approach to find optimal feature subset, but this approach is time consuming. In this paper, we propose a novel heuristic algorithm based on rough sets theory to find out the feature subset. This algorithm employs appearing frequency of attribute as heuristic information. Experiment results show in most times our algorithm can find out optimal feature subset quickly and efficiently.
2813744
Landmine Feature Extraction and Classification of GPR Data Based on SVM Method
In this paper, the problem of detecting buried landmine is tackled in the feature extraction and classification. Determining the likelihood set of an unknown pattern (feature vector), extracted from ground penetrating radar data by using SVM method. The advantage of SVM method in feature extraction and classification of image processing is: A classifier works well both on the training samples and on previously unseen samples; In addition, the SVM provides, enable a classification performance improvement based on from high feature dimensions to two or three feature dimensions. Finally, SVM method has a standard theory and a good implementation algorithm.
6648098
Reliability Modeling and Analysis of Service-Oriented Robot Management System
It is an obvious trend that computer network and robot technology will more and more closely combine with each other. And the robot, especially the service robot such as house robot, entertainment robot, etc, will provide different kinds of services to the mankind. This paper introduces a service-oriented network robotic management system (SONRMS) to coordinate and manage the behaviors between robots and their users so as to provide high reliability and safety measures to users. Then SONRMS' architecture is put forward. SONRMS reliability model is described by use of stochastic reward net, reliability block datagram and hierarchical model. The influence among parameters of the system upon SONRMS reliability is analyzed in the end