500117
A probabilistic relational model and algebra
Although the relational model for databases provides a great range of advantages over other data models, it lacks a comprehensive way to handle incomplete and uncertain data. Uncertainty in data values, however, is pervasive in all real-world environments and has received much attention in the literature. Several methods have been proposed for incorporating uncertain data into relational databases. However, the current approaches have many shortcomings and have not established an acceptable extension of the relational model. In this paper, we propose a consistent extension of the relational model. We present a revised relational structure and extend the relational algebra. The extended algebra is shown to be closed, a consistent extension of the conventional relational algebra, and reducible to the latter.
29720896
Assessing data quality for information products
Data quality is a critical issue in the design and implementation of databases. Often, the information provided to decision-makers is derived from heterogeneous databases. To determine the quality characteristics of information provided to decision-makers (derived data), it is necessary to determine how quality characteristics associated with diverse data sources affect the quality of the derived data. We discuss a methodology to determine two data quality characteristics, accuracy and incompleteness, that are of critical importance to decision-makers. We examine how the quality metrics of source data affect metrics for data derived using relational algebra operations.
476243
Reconciling Attribute Values from Multiple Data Sources
Because of the heterogeneous nature of multiple data sources, data integration is often one of the most challenging tasks of today’s information systems. While the existing literature has focused on problems such as schema integration and entity identification, our current study attempts to answer a basic question: When an attribute value for a real-world entity is recorded differently in two databases, how should the “best” value be chosen from the set of possible values? We first show how probabilities for attribute values can be derived, and then propose a framework for deciding the cost-minimizing value based on the total cost of type I, type II, and misrepresentation errors.
14335537
A Data Perturbation Approach to Privacy Protection in Data Mining
Advances in data mining techniques have raised growing concerns about privacy of personal information. Organizations that use their customers’ records in data mining activities are forced to take actions to protect the privacy of the individuals involved. A common practice for many organizations today is to remove the identity-reated attributes from customer records before releasing them to data miners or analysts. In this study, we investigate the effect of this practice and demonstrate that a majority of the records in a dataset can be uniquely identified even after identity related attributes are removed. We propose a data perturbation method that can be used by organizations to prevent such unique identification of individual records, while providing the data to analysts for data mining. The proposed method attempts to preserve the statistical properties of the data based on privacy protection parameters specified by the organization. We show that the problem can be solved in two phases, with a linear programming formulation in phase one (to preserve the marginal distribution), followed by a simple Bayes-based swapping procedure in phase two (to preserve the joint distribution). The proposed method is compared with a random perturbation method in classification performance on two real-world datasets. The results of the experiments indicate that it significantly outperforms the random method.
33218100
A Distance-Based Approach to Entity Reconciliation in Heterogeneous Databases
In modern organizations, decision makers must often be able to quickly access information from diverse sources in order to make timely decisions. A critical problem facing many such organizations is the inability to easily reconcile the information contained in heterogeneous data sources. To overcome this limitation, an organization must resolve several types of heterogeneity problems that may exist across different sources. We examine one such problem called the entity heterogeneity problem, which arises when the same real-world entity type is represented using different identifiers in different applications. A decision-theoretic model to resolve the problem is proposed. Our model uses a distance measure to express the similarity between two entity instances. We have implemented the model and tested it on real-world data. The results indicate that the model performs quite well in terms of its ability to predict whether two entity instances should be matched or not. The model is shown to be computationally efficient. It also scales well to large relations from the perspective of the accuracy of prediction. Overall, the test results imply that this is certainly a viable approach in practical situations.
9695284
A Hybrid Model for Part-of-Speech Tagging and its Application to Bengali
This paper describes our work on Bengali Part of Speech (POS) tagging using a corpus-based approach. There are several approaches for part of speech tagging. This paper deals with a model that uses a combination of supervised and unsupervised learning using a Hidden Markov Model (HMM). We make use of small tagged corpus and a large untagged corpus. We also make use of Morphological Analyzer. Bengali is a highly ambiguous and relatively free word order language. We have obtained an overall accuracy of 95%.
4723885
Learning while solving problems in best first search
We investigate the role of learning in search-based systems for solving optimization problems. We use a learning model, where the values of a set of features can be used to induce a clustering of the problem state space. The feasible set of h* values corresponding to each cluster is called h*set. If we relax the optimality guarantee, and tolerate a risk factor, the distribution of h*set can be used to expedite search and produce results within a given risk of suboptimality. The off-line learning method consists of solving a batch of problems by using A* to learn the distribution of the h*set in the learning phase. This distribution can be used to solve the rest of the problems effectively. We show how the knowledge acquisition phase can be integrated with the problem solving phase. We present a continuous online learning scheme that uses an "anytime" algorithm to learn continuously while solving problems.
34366805
An efficient dynamic program slicing technique
Abstract An important application of the dynamic program slicing technique is program debugging. In applications such as interactive debugging, the dynamic slicing algorithm needs to be efficient. In this context, we propose a new dynamic program slicing technique that is more efficient than the related algorithms reported in the literature. We use the program dependence graph as an intermediate program representation, and modify it by introducing the concepts of stable and unstable edges . Our algorithm is based on marking and unmarking the unstable edges as and when the dependences arise and cease during run-time. We call this algorithm edge-marking algorithm . After an execution of a node x , an unstable edge ( x , y ) is marked if the node x uses the value of the variable defined at node y . A marked unstable edge ( x , y ) is unmarked after an execution of a node z if the nodes y and z define the same variable var , and the value of var computed at the node y does not affect the present value of var defined at the node z . We show that our algorithm is more time and space efficient than the existing ones. The worst case space complexity of our algorithm is O ( n 2 ), where n is the number of statements in the program. We also briefly discuss an implementation of our algorithm.
16820690
Introducing Local Optimization for Effective Initialization and Crossover of Genetic Decision Trees
We introduce a new genetic operator, Reduction, that recties decision trees not correct syntactically and at the same time removes the redundant sections within, while preserving its accuracy during operation. A novel approach to crossover is presented that uses the reduction operator to systematically extract building blocks spread out over the entire second parent to create a subtree that is valid and particularly useful in the context it replaces the subtree in the rst parent. The crossover introduced also removes unexplored code from the ospring and hence prevents redundancy and bloating. Overall, reduction can be viewed as a local optimization step that directs the population, generated initially or over generations through crossovers, to potentially good regions in the search space so that reproduction is performed in a highly correlated landscape with a global structure. Lexical convergence is also ensured implying identical individuals always produce the same ospring.
1183276
Effective Use of Memory in Iterative Deepening Search
Abstract The Iterative Deepening A∗ (IDA∗) (R.E. Korf, Artificial Intelligence 27 (1985)) algorithm often reexpands too many nodes while solving certain combinatorial problems. Algorithm IDA∗_CR (U.K. Sarkar, Artificial Intelligence 50 (1991)) attempted to remedy this drawback. These algorithms require very little memory although much more is available in practice. This paper introduces an algorithm IDA∗_CRM which shows how the available memory can be advantageously utilized in IDAast;_CR in order to reduce the number of expanded nodes. IDA∗_CRAM, an approximation scheme derived from IDA∗_CRM, is also presented. Computational results are shown for the Flow-Shop Scheduling Problem.