knowledge discovery
49304841
Advances in Knowledge Discovery and Data Mining
Evaluating a trained system is an important component of machine learning. Labeling test data for large scale evaluation of a trained model can be extremely time consuming and expensive. In this paper we propose strategies for estimating performance of a classifier using as little labeling resource as possible. Specifically, we assume a labeling budget is given and the goal is to get a good estimate of the classifier performance using the provided labeling budget. We propose strategies to get a precise estimate of classifier accuracy under this restricted labeling budget scenario. We show that these strategies can reduce the variance in estimation of classifier accuracy by a significant amount compared to simple random sampling (over 65% in several cases) In terms of labeling resource, the reduction in number of samples required (compared to random sampling) to estimate the classifier accuracy with only 1% error is high as 60% in some cases.
Lecture Notes in Computer Science
Dinh Q Phung Vincent S Tseng Geoffrey I Webb Bao Ho Mohadeseh Ganji Lida Rashidi
61287995
From Data Mining to Knowledge Discovery in Databases
Data mining and knowledge discovery in databases have been attracting a significant amount of research, industry, and media attention of late. What is all the excitement about? This article provides an overview of this emerging field, clarifying how data mining and knowledge discovery in databases are related both to each other and to related fields, such as machine learning, statistics, and databases. The article mentions particular real world applications, specific data mining techniques, challenges involved in real world applications of knowledge discovery, and current and future research directions in the field.
AI Mag.
Usama M Fayyad Gregory Piatetsky-Shapiro Padhraic Smyth
67748106
Knowledge Discovery
Encyclopedia of Social Network Analysis and Mining. 2nd Ed.
42846121
Semantic Web in data mining and knowledge discovery: A comprehensive survey
Data Mining and Knowledge Discovery in Databases (KDD) is a research field concerned with deriving higher level insights from data. The tasks performed in that field are knowledge intensive and can often benefit from using additional knowledge from various sources. Therefore, many approaches have been proposed in this area that combine Semantic Web data with the data mining and knowledge discovery process. This survey article gives a comprehensive overview of those approaches in different stages of the knowledge discovery process. As an example, we show how Linked Open Data can be used at various stages for building content based recommender systems. The survey shows that, while there are numerous interesting research works performed, the full potential of the Semantic Web and Linked Open Data for data mining and KDD is still to be unlocked.
J. Web Semant.
Petar Ristoski Heiko Paulheim
32737414
Advances in Knowledge Discovery and Data Mining
We present a denoising technique in the domain of time series data that presumes a model for the uncorrupted underlying signal rather than a model for noise. Specifically, we show how the nonlinear reconstruction of the underlying dynamical system by way of time delay embedding yields a new solution for denoising where the underlying dynamics is assumed to be highly non linear yet low dimensional. The model for the underlying data is recovered using a non parametric Bayesian approach and is therefore very flexible. The proposed technique first clusters the reconstructed phase space through a Dirichlet Process Mixture of Exponential density, an infinite mixture model. Phase Space Reconstruction is accomplished by time delay embedding in the framework of Taken's Embedding Theorem with the underlying dimension being determined by the False Neighborhood method. Next, an Infinite Mixtures of Linear Regression via Dirichlet Process is used to nonlinearly map the phase space data points to their respective temporally subsequent points in the phase space. Finally, a convex optimization based approach is used to restructure the dynamics by perturbing the phase space points to create the new denoised time series. We find that this method yields significantly better performance in noise reduction, power spectrum analysis and prediction accuracy of the phase space.
Lecture Notes in Computer Science
James Bailey L Khan Takashi Washio G Dobbie Joshua Zhexue Huang Ruili Wang
167706570
Prior Knowledge and the Discovery of Entrepreneurial Opportunities
Before technological change leads to new processes, products, markets, or ways of organizing, entrepreneurs must discover opportunities in which to exploit the new technology. To date, research has not explained adequately why entrepreneurs discover these opportunities, which creates several conceptual problems in the entrepreneurship literature. Drawing on Austrian economics, I argue that opportunity discovery is a function of the distribution of information in society (Hayek 1945) Through in depth case studies of eight sets of entrepreneurs who exploit a single MIT invention, I show that entrepreneurs discover opportunities related to the information that they already possess. I use these findings to draw several implications that differ from those prevailing in the entrepreneurship literature, including: (1) entrepreneurs do not always select between alternative market opportunities for new technologies; (2) the source of entrepreneurship lies in differences in information about opportunities; (3) the results of prior studies of entrepreneurial exploitation may suffer from bias; and (4) individual differences influence the opportunities that people discover, how their entrepreneurial efforts are organized, and how the government can influence this process.
Scott A Shane
108578540
A framework for knowledge discovery in massive building automation data and its application in building diagnostics
Abstract Building Automation System (BAS) plays an important role in building operation nowadays. A huge amount of building operational data is stored in BAS; however, the data can seldom be effectively utilized due to the lack of powerful tools for analyzing the large data. Data mining (DM) is a promising technology for discovering knowledge hidden in large data. This paper presents a generic framework for knowledge discovery in massive BAS data using DM techniques. The framework is specifically designed considering the low quality and complexity of BAS data, the diversity of advanced DM techniques, as well as the integration of knowledge discovered by DM techniques and domain knowledge in the building field. The framework mainly consists of four phases, i.e. data exploration, data partitioning, knowledge discovery, and post mining. The framework is applied to analyze the BAS data of the tallest building in Hong Kong. The analysis of variance (ANOVA) method is adopted to identify the most significant time variables to the aggregated power consumption. Then the clustering analysis is used to identify the typical operation patterns in terms of power consumption. Eight operation patterns have been identified and therefore the entire BAS data are partitioned into eight subsets. The quantitative association rule mining (QARM) method is adopted for knowledge discovery in each subset considering most of BAS data are numeric type. To enhance the efficiency of the post mining phase, two indices are proposed for fast and conveniently identifying and utilizing potentially interesting rules discovered by QARM. The knowledge discovered is successfully used for understanding the building operating behaviors, identifying non typical operating conditions and detecting faulty conditions.
Cheng Fan Fu Xiao Chengchu Yan
16068842
Knowledge Discovery from Biomedical Ontologies in Cross Domains
In recent years, there is an increasing demand for sharing and integration of medical data in biomedical research. In order to improve a health care system, it is required to support the integration of data by facilitating semantic interoperability systems and practices. Semantic interoperability is difficult to achieve in these systems as the conceptual models underlying datasets are not fully exploited. In this paper, we propose a semantic framework, called Medical Knowledge Discovery and Data Mining (MedKDD) that aims to build a topic hierarchy and serve the semantic interoperability between different ontologies. For the purpose, we fully focus on the discovery of semantic patterns about the association of relations in the heterogeneous information network representing different types of objects and relationships in multiple biological ontologies and the creation of a topic hierarchy through the analysis of the discovered patterns. These patterns are used to cluster heterogeneous information networks into a set of smaller topic graphs in a hierarchical manner and then to conduct cross domain knowledge discovery from the multiple biological ontologies. Thus, patterns made a greater contribution in the knowledge discovery across multiple ontologies. We have demonstrated the cross domain knowledge discovery in the MedKDD framework using a case study with 9 primary biological ontologies from Bio2RDF and compared it with the cross domain query processing approach, namely SLAP. We have confirmed the effectiveness of the MedKDD framework in knowledge discovery from multiple medical ontologies.
PloS one
Feichen Shen Yugyung Lee
109477115
Temporal knowledge discovery in big BAS data for building energy management
Abstract With the advances of information technologies, today's building automation systems (BASs) are capable of managing building operational performance in an efficient and convenient way. Meanwhile, the amount of real time monitoring and control data in BASs grows continually in the building lifecycle, which stimulates an intense demand for powerful big data analysis tools in BASs. Existing big data analytics adopted in the building automation industry focus on mining cross sectional relationships, whereas the temporal relationships, i.e. the relationships over time, are usually overlooked. However, building operations are typically dynamic and BAS data are essentially multivariate time series data. This paper presents a time series data mining methodology for temporal knowledge discovery in big BAS data. A number of time series data mining techniques are explored and carefully assembled, including the Symbolic Aggregate approXimation (SAX) motif discovery, and temporal association rule mining. This study also develops two methods for the efficient post processing of knowledge discovered. The methodology has been applied to analyze the BAS data retrieved from a real building. The temporal knowledge discovered is valuable to identify dynamics, patterns and anomalies in building operations, derive temporal association rules within and between subsystems, assess building system performance and spot opportunities in energy conservation.
Cheng Fan Fu Xiao Henrik Madsen Dian Wang
16466942
A Cognitive Adopted Framework for IoT Big Data Management and Knowledge Discovery Prospective
In future IoT big data management and knowledge discovery for large scale industrial automation application, the importance of industrial internet is increasing day by day. Several diversified technologies such as IoT (Internet of Things) computational intelligence, machine type communication, big data, and sensor technology can be incorporated together to improve the data management and knowledge discovery efficiency of large scale automation applications. So in this work, we need to propose a Cognitive Oriented IoT Big data Framework (COIB framework) along with implementation architecture, IoT big data layering architecture, and data organization and knowledge exploration subsystem for effective data management and knowledge discovery that is well suited with the large scale industrial automation applications. The discussion and analysis show that the proposed framework and architectures create a reasonable solution in implementing IoT big data based smart industrial applications.
Int. J. Distributed Sens. Networks
Nilamadhab Mishra Chung-Chih Lin Hsien-Tsung Chang