17964688
A Novel Method for Detecting Outlying Subspaces in High-dimensional Databases Using Genetic Algorithm
Detecting outlying subspaces is a relatively new research problem in outlier-ness analysis for high-dimensional data. An outlying subspace for a given data point p is the sub- space in which p is an outlier. Outlying subspace detection can facilitate a better characterization process for the detected outliers. It can also enable outlier mining for high- dimensional data to be performed more accurately and efficiently. In this paper, we proposed a new method using genetic algorithm paradigm for searching outlying subspaces efficiently. We developed a technique for efficiently computing the lower and upper bounds of the distance between a given point and its kth nearest neighbor in each possible subspace. These bounds are used to speed up the fitness evaluation of the designed genetic algorithm for outlying subspace detection. We also proposed a random sampling technique to further reduce the computation of the genetic algorithm. The optimal number of sampling data is specified to ensure the accuracy of the result. We show that the proposed method is efficient and effective in handling outlying subspace detection problem by a set of experiments conducted on both synthetic and real-life datasets.
4732400
XAR-miner: efficient association rules mining for XML data
In this paper, we propose a framework, called XAR-Miner, for mining ARs from XML documents efficiently. In XAR-Miner, raw data in the XML document are first preprocessed to transform to either an Indexed Content Tree (IX-tree) or Multi-relational databases (Multi-DB), depending on the size of XML document and memory constraint of the system, for efficient data selection and AR mining. Task-relevant concepts are generalized to produce generalized meta-patterns, based on which the large ARs that meet the support and confidence levels are generated.
15427261
Anomaly detection in high-dimensional network data streams: A case study
In this paper, we study the problem of anomaly detection in high-dimensional network streams. We have developed a new technique, called Stream Projected Outlier deTector (SPOT), to deal with the problem of anomaly detection from high-dimensional data streams. We conduct a case study of SPOT in this paper by deploying it on 1999 KDD Intrusion Detection application. Innovative approaches for training data generation, anomaly classification and false positive reduction are proposed in this paper as well. Experimental results demonstrate that SPOT is effective in detecting anomalies from network data streams and outperforms existing anomaly detection methods.
16517186
A Color Indexing Scheme Using Two-level Clustering Processing for Effective and Efficient Image Retrieval
[Abstract]: In this paper, we present a clustering-based color indexing scheme for effective amd efficient image retrieval, which is essentially an exploration on the application of clustering technique to image retrieval. In our approach, the color features are clustered automatically using a color clustering algorithm twice (called two-level clustering processing) and two color feature summarizations are obtained, e.g. Local Color Centriods (LCCs) and Global Color Centroids (GCCs). Based upon LCCs and GCCs, a three-level R-tree is bulit for indexing database images and performing effecitve and efficient image retrieval. The experiments show that this indexing scheme is effective and efficient in performing image retrieval.
37278978
PC-Filter: A Robust Filtering Technique for Duplicate Record Detection in Large Databases
In this paper, we will propose PC-Filter (PC stands for Partitio n Comparison), a robust data filter for approximately duplicate record detection in large databases. PC-Filter distinguishes itself from all of existing methods by using the notion of partition in duplicate detection. It first sorts the whole database and splits the sorted database into a number of record partitions. The Partition Comparison Graph (PCG) is then constructed by performing fast partition pruning. Finally, duplicate records are effectively detected by using internal and external partition comparison based on PCG. Four properties, used as heuristics, have been devised to achieve a remarkable efficiency of the filter based on triangle inequity of record similarity. PC-Filter is insensitive to the key used to sort the database, and can achieve a very good recall level that is comparable to that of the pair-wise record comparison method but only with a complexity of O(N 4/3). Equipping existing detection methods with PC-Filter, we are able to well solve the ”Key Selection” problem, the ”Scope Specification” problem and the ”Low Recall” problem that the existing methods suffer from.
11328445
Gene Specific Co-regulation Discovery: An Improved Approach
Discovering gene co-regulatory relationships is a new but important research problem in DNA microarray data analysis. The problem of gene specific co-regulation discovery is to, for a particular gene of interest, called the target gene , identify its strongly co-regulated genes and the condition subsets where such strong gene co-regulations are observed. The study on this problem can contribute to a better understanding and characterization of the target gene. The existing method, using the genetic algorithm (GA), is slow due to its expensive fitness evaluation and long individual representation. In this paper, we propose an improved method for finding gene specific co-regulations. Compared with the current method, our method features a notably improved efficiency. We employ k NN Search Table to substantially speed up fitness evaluation in the GA. We also propose a more compact representation scheme for encoding individuals in the GA, which contributes to faster crossover and mutation operations. Experimental results with a real-life gene microarray data set demonstrate the improved efficiency of our technique compared with the current method.
45190241
Factor analysis based anomaly detection
We propose a novel anomaly detection algorithm based on factor analysis and Mahalanobis distance. Factor analysis is used to uncover the latent structure (dimensions) of a set of variables. It reduces attribute space from a larger number of variables to a smaller number of factors. The Mahalanobis distance is used to determine the "similarity" of a set of values from an "unknown" sample to a set of values measured from a collection of "known" samples. Combined with factor analysis, Mahalanobis distance is extended to examine whether a given vector is an outlier from a model identified by "factors" based on factor analysis. We present a factor analysis-based network anomaly detection algorithm and apply it to DARPA intrusion detection evaluation data. The experimental results show that the proposed algorithm is able to detect network intrusions with relatively low false alarms.
6081651
A Novel Lossless Compression for Hyperspectral Images by Adaptive Classified Arithmetic Coding in Wavelet Domain
In this paper, we propose a lossless compression algorithm for hyperspectral images; it is based on the adaptive classified arithmetic coding in wavelet domain and the adaptive spectral band reordering algorithm. The adaptive classified scheme divides each of the residual images after wavelet transform into different classes, and then the adaptive arithmetic coding is performed for each of the classes. This classified coding scheme saves a lot of coding bits. The adaptive spectral band reordering algorithm finds out the nearly best reference band for each of the bands, so the spectral correlation is better used. Combining these two algorithms makes full use of the characteristics of hyperspectral images. Experiments show that our method is capable of providing a high compression performance.
14748047
A Square Root Extended Set Membership Algorithm with Applications to Nonlinear System Estimation
In this paper, a square root extended set membership (SRESM) state estimation algorithm is proposed to further improve the numerical accuracy and stability of the nonlinear system estimation. In comparison with the existing extended set membership (ESM), the proposed square root ESM using the efficient and stable updating recursions provides much more accurate estimation results. It has improved numerical stability and real time applicability. Simulation results in estimation of mobile robot localization are given to show the effectiveness and robustness of the proposed algorithm.
18935532
Equivalent Inertia Moment in Linear Stability Calculation of Stepped Beam
In the process of simplifying the stability calculation for light main boom of crawler crane, the structure can be considered as non-symmetrical one with variable cross-section beam. The equivalence of the simply supported beam with asymmetrical cross-section has not been solved. This paper attains the equivalent inertia moment of the stepped beam with the small deformation by using the differential and energy methods. In addition, comparing with the software ANSYS, the results of differential method is better than energy method. Then the results of differential errors are amended and reduced. Then, the paper presents the equivalent inertia moment coefficients curves, which greatly simplifies the calculation process of critical load and provides the reference for boom design.
5348752
Toward Protein Structure Analysis with Self-Organizing Maps
Establishing structure-function relationships on the proteomic scale is a unique challenge faced by bioinformatics and molecular biosciences. Large protein families represent natural libraries of analogues of a given catalytic or protein function, thus making them ideal targets for the investigation of structure-function relationships in proteins. To this end, we have developed a new technique for analyzing large amounts of detailed molecular structure information focusing on the functional centers of homologous proteins. Our approach uses unsupervised machine learning, in particular, self-organizing maps. The information captured by a self-organizing map and stored in its reference models highlights the essential structure of the proteins under investigation and can be effectively used to study detailed structural differences and similarities among homologous proteins. Our preliminary results obtained with a prototype based on these techniques demonstrate that we can classify proteins and identify common and unique structures within a family and, more importantly, identify common and unique structural features of different conformations of the same protein. The approach developed here outperforms many of today’s structure analysis tools. These tools are usually either limited by the number of proteins they can process at the same time or they are limited by the structural resolution they can accommodate, that is, many of the structural analysis tools that can handle multiple proteins at the same time limit themselves to secondary structure analysis and therefore miss fine structural nuances within proteins.