Data imputation 
211482719
Attention based Learning for Missing Data Imputation in HoloClean
We study the problem of missing data imputation, a data validation task that machine learning researchers and practitioners confront regularly. We focus on mixed (discrete and continuous) data and AimNet, an attention based learning network for missing data imputation in HoloClean, a state of the art ML based data cleaning framework. AimNet utilizes a variation of the dot product attention mechanism to learn structural properties of the mixed data distribution and relies on the learned structure to perform imputation. We perform an extensive experimental study over 14 real world data sets to understand the role of attention and structure on data imputation. We find that the simple attention based architecture of AimNet outperforms state of the art baselines, such as ensemble tree models and deep learning architectures (e.g. generative adversarial networks) by up to 43% in accuracy on discrete values and up to 26.7% in normalized RMS error on continuous values. A key finding of our study is that, by learning the structure of the underlying distribution, the attention mechanism can generalize better on systematically missing data where imputation requires reasoning about functional relationships between attributes.
MLSys
Richard Wu Aoqian Zhang Ihab F Ilyas Theodoros Rekatsinas
218718729
Lung Segmentation from Chest X rays using Variational Data Imputation
Pulmonary opacification is the inflammation in the lungs caused by many respiratory ailments, including the novel corona virus disease 2019 (COVID 19) Chest X rays (CXRs) with such opacifications render regions of lungs imperceptible, making it difficult to perform automated image analysis on them. In this work, we focus on segmenting lungs from such abnormal CXRs as part of a pipeline aimed at automated risk scoring of COVID 19 from CXRs. We treat the high opacity regions as missing data and present a modified CNN based image segmentation network that utilizes a deep generative model for data imputation. We train this model on normal CXRs with extensive data augmentation and demonstrate the usefulness of this model to extend to cases with extreme abnormalities.
ArXiv
Raghavendra Selvan Erik Bjornager Dam Sofus Rischel Kaining Sheng Mads Nielsen Akshay Pai
47008092
GAIN: Missing Data Imputation using Generative Adversarial Nets
We propose a novel method for imputing missing data by adapting the well known Generative Adversarial Nets (GAN) framework. Accordingly, we call our method Generative Adversarial Imputation Nets (GAIN) The generator (G) observes some components of a real data vector, imputes the missing components conditioned on what is actually observed, and outputs a completed vector. The discriminator (D) then takes a completed vector and attempts to determine which components were actually observed and which were imputed. To ensure that D forces G to learn the desired distribution, we provide D with some additional information in the form of a hint vector. The hint reveals to D partial information about the missingness of the original sample, which is used by D to focus its attention on the imputation quality of particular components. This hint ensures that G does in fact learn to generate according to the true data distribution. We tested our method on various datasets and found that GAIN significantly outperforms state of the art imputation methods.
ICML
Jinsung Yoon James Jordon Mihaela van der Schaar
57394512
Enriching Data Imputation under Similarity Rule Constraints
Incomplete information often occurs along with many database applications, e.g. in data integration, data cleaning, or data exchange. The idea of data imputation is often to fill the missing data with the values of its neighbors who share the same/similar information. Such neighbors could either be identified certainly by editing rules or extensively by similarity relationships. Owing to data sparsity, the number of neighbors identified by editing rules w.r.t. value equality is rather limited, especially in the presence of data values with variances. To enrich the imputation candidates, a natural idea is to extensively consider the neighbors with similarity relationship. However, the candidates suggested by these (heterogenous) similarity neighbors may conflict with each other. In this paper, we propose to utilize the similarity rules with tolerance to small variations (instead of the aforesaid editing rules with strict equality constraints) to rule out the invalid candidates provided by similarity neighbors. To enrich the data imputation, i.e. imputing the missing values more, we study the problem of maximizing the missing data imputation. Our major contributions include (1) the np hardness analysis on solving as well as approximating the problem, (2) exact algorithms for tackling the problem, and (3) efficient approximation with performance guarantees. Experiments on real and synthetic data sets demonstrate the superiority of our proposal in filling accuracy. We also demonstrate that the record matching application is indeed improved, after applying the proposed imputation.
IEEE Transactions on Knowledge and Data Engineering
Shaoxu Song Yu Sun Aoqian Zhang Lei Chen Jianmin Wang
196193501
Similarity learning information fusion schemes for missing data imputation
Abstract Missing data imputation is a very important data cleaning task for machine learning and data mining with incomplete data. This paper proposes two novel methods for missing data imputation, named kEMI and kEMI that are based on the k Nearest Neighbours algorithm for pre imputation and the Expectation Maximization algorithm for posterior imputation. The former is a local search mechanism that aims to automatically find the best value for k and the latter makes use of the best k nearest neighbours to estimate missing scores by learning global similarities. kEMI makes use of a novel information fusion mechanism. It fuses top estimations through the Dempster Shafer fusion module to obtain the final estimation. They handle both numerical and categorical features. The performance of the proposed imputation techniques are evaluated by applying them on twenty one publicly available datasets with different missingness and ratios, and, then, compared with other state of the art missing data imputation techniques in terms of standard evaluation measures such as the normalized root mean square difference and the absolute error. The attained results indicate the effectiveness of the proposed novel missing data imputation techniques.
Knowl. Based Syst.
Roozbeh Razavi-Far Boyuan Cheng Mehrdad Saif Majid Ahmadi
117311340
A Bayesian tensor decomposition approach for spatiotemporal traffic data imputation
Abstract The missing data problem is inevitable when collecting traffic data from intelligent transportation systems. Previous studies have shown the advantages of tensor completion based approaches in solving multi dimensional data imputation problems. In this paper, we extend the Bayesian probabilistic matrix factorization model by Salakhutdinov and Mnih (2008) to higher order tensors and apply it for spatiotemporal traffic data imputation tasks. In doing so, we care about not only the model configuration but also the representation of data (i.e. matrix, third order tensor and fourth order tensor) Using a nine week spatiotemporal traffic speed data set (road segment x day x time of day) collected in Guangzhou, China, we evaluate the performance of this fully Bayesian model and explore how different data representations affect imputation performance through extensive experiments. The results show the proposed model can produce accurate imputations even under temporally correlated data corruption. Our experiments also show that data representation is a crucial factor for model performance, and a third order tensor structure outperforms the matrix and fourth order tensor representations in preserving information in our data set. We hope this work could give insights to practitioners when performing spatiotemporal data imputation tasks.
Xinyu Chen Zhaocheng He Lijun Sun
59316466
CollaGAN: Collaborative GAN for Missing Image Data Imputation
In many applications requiring multiple inputs to obtain a desired output, if any of the input data is missing, it often introduces large amounts of bias. Although many techniques have been developed for imputing missing data, the image imputation is still difficult due to complicated nature of natural images. To address this problem, here we proposed a novel framework for missing image data imputation, called Collaborative Generative Adversarial Network (CollaGAN) CollaGAN convert the image imputation problem to a multi domain images to image translation task so that a single generator and discriminator network can successfully estimate the missing data using the remaining clean data set. We demonstrate that CollaGAN produces the images with a higher visual quality compared to the existing competing approaches in various image imputation tasks.
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Dongwook Lee Junyoung Kim Won-Jin Moon J C Ye
198351111
Comparison of Performance of Data Imputation Methods for Numeric Dataset
ABSTRACT Missing data is common problem faced by researchers and data scientists. Therefore, it is required to handle them appropriately in order to get better and accurate results of data analysis. Objective of this research paper is to provide better understanding of data missingness mechanism, data imputation methods, and to assess performance of the widely used data imputation methods for numeric dataset. It will help practitioners and data scientists to select appropriate method of data imputation for numeric dataset while performing data mining task. In this paper, we comprehensively compare seven data imputation methods namely mean imputation, median imputation, kNN imputation, predictive mean matching, Bayesian Linear Regression (norm) Linear Regression, non Bayesian (norm.nob) and random sample. We have used five different numeric datasets obtained from UCI machine learning repository for analyzing and comparing performance of the data imputation methods. Performance of the data imputation methods is assessed using Normalized Root Mean Square Error (RMSE) method. The results of analysis show that kNN imputation method outperforms the other methods. It has also been found that performance of the data imputation method is independent of the dataset and percentage of missing values in the dataset.
Appl. Artif. Intell.
Anil S Jadhav Dhanya Pramod Krishnan Ramanathan
109937380
Analysis and Impact Evaluation of Missing Data Imputation in Day ahead PV Generation Forecasting
Over the past decade, PV power plants have increasingly contributed to power generation. However, PV power generation widely varies due to environmental factors; thus, the accurate forecasting of PV generation becomes essential. Meanwhile, weather data for environmental factors include many missing values; for example, when we estimated the missing values in the precipitation data of the Korea Meteorological Agency, they amounted to ~16% from 2015 2016, and further, 19% of the weather data were missing for 2017. Such missing values deteriorate the PV power generation prediction performance, and they need to be eliminated by filling in other values. Here, we explore the impact of missing data imputation methods that can be used to replace these missing values. We apply four missing data imputation methods to the training data and test data of the prediction model based on support vector regression. When the k nearest neighbors method is applied to the test data, the prediction performance yields results closest to those for the original data with no missing values, and the prediction model's performance is stable even when the missing data rate increases. Therefore, we conclude that the most appropriate missing data imputation for application to PV forecasting is the KNN method.
Taeyoung Kim Woong Ko Jinho Kim
187697940
Missing Data Imputation Through SGTM Neural Like Structure for Environmental Monitoring Tasks
The article describes a new missing data imputation method. It is based on the use of a high speed neural like structure of the Successive Geometric Transformations Model. The importance of the research is based on the analysis of disadvantages of the known methods for missing data processing. Various simple and complex algorithms are analyzed, among which are the arithmetic mean algorithm, regression modeling, etc. It is shown that the above mentioned imputation methods in data monitoring of air pollution do not allow to obtain reliable results due to the low prediction accuracy. An effective method for processing data imputation through SGTM neural like structure is proposed. An example of filling data by forecasting CO, NO and NO2 missed parameters in data monitoring of air pollution is given. A comparison of the proposed method with the arithmetic mean algorithm is carried out. Accuracies of the data imputation by developed method and by arithmetic mean algorithm are based on calculated evaluation criteria: Root mean squared errors. Experimentally established that the data imputation method through SGTM neural like structure has a three times higher accuracy of the data imputation than the arithmetic mean algorithm. The proposed approach can be used in various areas such as medicine, materials science, economics, science services, etc.
Advances in Computer Science for Engineering and Education II
Oleksandra Mishchuk Roman Tkachenko Ivan Izonin