Next Article in Journal
Prediction of the Number of Cumulative Pulses Based on the Photon Statistical Entropy Evaluation in Photon-Counting LiDAR
Next Article in Special Issue
A Hybrid Genetic Algorithm Based on Imitation Learning for the Airport Gate Assignment Problem
Previous Article in Journal
RNNCon: Contribution Coverage Testing for Stacked Recurrent Neural Networks
Previous Article in Special Issue
A New Reliability Coefficient Using Betting Commitment Evidence Distance in Dempster–Shafer Evidence Theory for Uncertain Information Fusion
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Entropy
Volume 25
Issue 3
10.3390/e25030521
entropy-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Analyzing the Effect of Imputation on Classification Performance under MCAR and MAR Missing Mechanisms

by
Philip Buczak
1,*,
Jian-Jia Chen
2 and
Markus Pauly
1,3
1
Department of Statistics, TU Dortmund University, 44227 Dortmund, Germany
2
Department of Computer Science, TU Dortmund University, 44227 Dortmund, Germany
3
UA Ruhr, Research Center Trustworthy Data Science and Security, 44227 Dortmund, Germany
*
Author to whom correspondence should be addressed.
Entropy 2023, 25(3), 521; https://doi.org/10.3390/e25030521
Submission received: 20 January 2023 / Revised: 10 March 2023 / Accepted: 14 March 2023 / Published: 17 March 2023
(This article belongs to the Special Issue Advances in Information Sciences and Applications)
Browse Figure
Versions Notes

Abstract

:
Many datasets in statistical analyses contain missing values. As omitting observations containing missing entries may lead to information loss or greatly reduce the sample size, imputation is usually preferable. However, imputation can also introduce bias and impact the quality and validity of subsequent analysis. Focusing on binary classification problems, we analyzed how missing value imputation under MCAR as well as MAR missingness with different missing patterns affects the predictive performance of subsequent classification. To this end, we compared imputation methods such as several MICE variants, missForest, Hot Deck as well as mean imputation with regard to the classification performance achieved with commonly used classifiers such as Random Forest, Extreme Gradient Boosting, Support Vector Machine and regularized logistic regression. Our simulation results showed that Random Forest based imputation (i.e., MICE Random Forest and missForest) performed particularly well in most scenarios studied. In addition to these two methods, simple mean imputation also proved to be useful, especially when many features (covariates) contained missing values.

1. Introduction

Missing data is a reoccurring challenge in statistical analyses in the life sciences and many other domains from the information sciences. For example, patients may refuse to share sensitive details about their health. In repeated measurement designs, patients may either miss single measurements or drop out completely. Survey data may also be missing by design where not all respondents receive the same set of questions. Generally speaking, there are three different mechanisms to distinguish when handling missing data [1]. The missing completely at random (MCAR) mechanism assumes that missingness does not depend on the data (neither the observed nor the unobserved part). The other two mechanisms allow for missingness to depend on the data. For missing at random (MAR), the missingness depends on the observed components of an observation (i.e., the dependance relation is encoded in the observed data), while for missing not at random (MNAR), the missingness depends on the unobserved compontents of an observation (i.e., the dependance relation is not encoded in the observed data).
Thus, depending on the (usually unknown) missing mechanism that governs the occurrence of given missing values, the common practice of listwise deletion (i.e., deletion of observations that contain at least one missing value) may lead to reduced sample size and thus information loss. As a remedy, missing data is often imputed through different techniques such as simple mean or mode imputation, or advanced imputation methods such as MICE [2,3] or techniques stemming from machine learning (ML) [4,5,6,7]. However, data imputation may also affect the quality and validity of prediction or inference from resulting models [3,8,9,10,11]. It is therefore crucial to analyze the extent to which imputation methods influence subsequent regression or classification models obtained from imputed data. In the context of classification, a few studies have compared the performance of different imputation methods [12,13,14,15]. For our purposes most notably due to the variety of imputation and classification methods, Farfanghar et al. [12] compared the predictive performance of six different imputation methods w.r.t. the predictive performance of five classifiers. As imputation methods, the authors used Hot Deck imputation, Naive Bayes imputation, mean imputation as well as a polytomous regression-based imputation method. Additionally, the former two methods, were also embedded within a custom imputation framework meant to improve the performance of their standalone counterpart. Although they found that imputation generally improves performance, no imputation method was found to regularly outperform its competitors. Since their 2008 study, new imputation methods were suggested, for example, the Refs. [4,6]. In particular, tree-based ML approaches have shown some enhancements with respect to accuracy for regression problems [10,11,16]. For example, Ramosaj et al. [10] have recently analyzed how different imputation methods influence the subsequent predictive performance of linear regression and tree-based ML approaches. In their work, they found a certain preference to use the Random Forest based missForest [4] imputation method or a MICE [2] model based on Bayesian linear regression.
In light of their findings for the regression context, we investigated whether similar conclusions regarding the performance of the different imputation methods can be drawn for classification problems. Compared to the previous studies in the classification context, we used a more modern suite of imputation algorithms, that is, missForest and MICE, and further considered MCAR as well as MAR missing mechanisms with varying missing patterns. In the next section, we describe our simulation set-up in more detail. We report our results in Section 3 and follow up with a discussion in Section 4.

2. Materials and Methods

For our analysis, we used six binary classification problems from the life sciences. Table 1 provides an overview over the datasets including the number of observation and features as well as a short description of the target variable and features (covariates) used for prediction. The datasets Phoneme and Pima Indians were obtained from the Open Machine Learning Project [17], while the datasets Haberman, Skin [18], SPECT and Wilt [19] were obtained from the UCI Machine Learning Repository [20]. Except for the Skin dataset, we used the original datasets as they came. The Skin dataset had an original size of 245,057 observations. In our analysis, we used a random sample of 5% where the original class balance was preserved through stratified sampling. None of the original datasets included any missing values. To limit the scope of our investigation, we have decided to only focus on datasets with numerical features at this time.
The general flow of our analysis is depicted in Figure 1. Starting from an original complete dataset, we generated ten train/test partitions as in a 10-fold cross-validation. In each fold, we generated missing values in the feature data of the respective training set. To this end, we used the ampute function from the MICE R package [2] which implements the multivariate amputation procedure proposed in [21]. We varied the proportion of missing values between 10%, 30% and 50%, that is, we set 10%, 30% and 50% of the original feature data as missing, respectively. The missingness was generated via a MCAR as well as different MAR mechanisms. One key component of the amputation procedure is the ability to specify a missingness pattern that governs which set of features may contain missing values and which set of features is kept complete. This allows for creating flexible missingness scenarios. To study different MAR mechanisms, we specified three missingness patterns that vary in the amount of features that may contain missing values and as such cover diverse scenarios. First, a pattern where for any given observation only one feature value at a time could be set missing. Second, a pattern where missing values could only occur in the middle third of the features. Third, a pattern where missing values could only occur in the first and last third of the features. We will refer to these three MAR patterns as the One at a Time, One Third and Two Thirds pattern, respectively.
Having introduced missingness into the training set, we then imputed the missing values using three MICE [3] algorithms as well as missForest [4], Hot Deck imputation [22] and a naive mean imputation. The three MICE algorithms we used were Bayesian linear regression (denoted as MICE Norm), Predictive Mean Matching (denoted as MICE PMM) and Random Forest (denoted as MICE RF). A common approach of the MICE algorithms and missForest is the concept of treating the imputation for a feature containing missing values as a prediction problem where the respective feature acts as the target variable that is predicted using the remaining features. Usually, some model (e.g., linear model or decision tree) is learned on the data subset for which the respective feature was observed. This idea is fleshed out in varying ways between the different imputation methods.
MICE Norm is based on Rubin’s [23] imputation method under the assumption of normality. The parameters of the linear model are sampled from their respective posterior distribution which is estimated using the observed data [3]. MICE PMM extends upon this by sampling a set of candidate donors (five per default) from the observed data whose values are closest to the predictions for missing data points as obtained from the Bayesian linear model. From this set of candidates, one donor is then chosen at random. Thus, PMM only imputes values that were actually observed and consequently does not suffer from the issue of out-of-range imputation [3]. For more details on the matching procedure see the Ref. [24].
MICE RF and missForest fall into the category of tree-based imputation methods. MICE RF is based on the algorithm proposed in Doove et al. [25] in which k individual tree models are fit on bootstrap samples from the observed data. The data point requiring imputation is then passed through each tree and falls into a terminal node, respectively. For each of these terminal nodes, one donor is sampled at random from all observations belonging to the node, thus resulting in a set of k donors overall. Out of this set, one donor is chosen at random for the imputation. One commonality of all MICE methods is the concept of multiple imputation. To account for the variability of the imputation process due to the probabilistic nature of the methods, multiple imputed datasets are created. Typically, subsequent analysis is performed on each dataset and the respective model results are pooled. Because we did not analyze uncertainty or perform inference, and in order to limit the computational complexity and to keep our simulation setup consistent, we aggregated the imputed datasets into a combined dataset. This was performed by averaging numeric features and selecting the mode for categorical features, respectively.
In contrast to MICE RF, missForest uses Random Forests to iteratively improve upon an initial imputation guess. The algorithm repeatedly cycles through all originally non-complete features and updates its imputations by fitting Random Forest models and obtaining new predictions for the missing entries. Since the features used in a respective step for prediction potentially contain imputed values themselves that were improved upon in previous steps, the procedure gradually refines its imputations over time. Another difference between MICE and missForest is that the latter does not use multiple imputation.
Last, Hot Deck imputation obtains imputations by sampling from the set of observed values where observations that are similar to the observation requiring imputation have a higher chance of being selected as the donor through proximity-based weighting. For imputing with Hot Deck imputation, MICE and missForest in R (version 4.0.0; [26]), we used the hot.deck package [27], the mice package [2] and the missRanger package [6], respectively. The latter allows for additional PMM and is a faster implementation of missForest since it uses the computationally efficient ranger package [28]. For MICE, we used the Bayesian linear regression (MICE Norm), Predictive Mean Matching (MICE PMM) and Random Forest (MICE RF) variants. For missForest, we included both a non-PMM and a PMM variant with three candidate non-missing values from which the imputed value was sampled. We used default values for MICE and missForest settings except for the number of trees and the maximum chaining iterations of missForest which we set to 100 and 3, respectively. The number of multiple imputations for MICE was five as per default. Having imputed the missing values, we continued with the task of classification. As classifiers, we used Elastic Net regularized logistic regression (denoted EN-LR in the following), Random Forest (RF), Support Vector Machine (SVM) and Extreme Gradient Boosting (XGBoost). All ML experiments were performed with the mlr package that provides a unified interface for ML-based analysis in R. For our classifiers, we used the glmnet package [29] for EN-LR, the ranger package [28] for RF, the e1071 package [30] for (radial basis) SVM and the xgboost package [31] for XGBoost. All of these learners have individual sets of hyperparameters that must be specified in advance. Their optimal choice is problem-dependent and approximated via hyperparameter tuning. Incorporating tuning into a benchmark experiment of different ML algorithms requires a nested resampling approach, where tuning is performed in the inner, and validation in the outer resampling loop. Otherwise, tuning and validating on overlapping data samples may lead to optimistic error estimates due to overfitting [32]. Thus, we perform an additional 3-fold cross-validation for hyperparameter tuning on the respective imputed training sets. Table A1 shows the respective hyperparameters and search spaces considered for tuning via a random search with 30 iterations. For hyperparameters that were not tuned, we used the default values.
After tuning, the classification models were learned on the entire imputed training set using the optimal hyperparameter settings, and validated on the test set. For each fold, this yielded a classification performance as measured by the Mean Misclassification Error (MMCE), that is, the proportion of wrongly classified instances in relation to all instances. Averaging over the fold-specific performances resulted in an overall cross-validated classification performance on which further comparisons are based. For each combination of dataset, imputation method and learner, we performed 100 replications.

3. Results

Table 2 shows the mean ranks based on the MMCE achieved by the respective classifiers for the 100 replications under a MCAR mechanism. For each row in the table, the best (i.e., lowest) mean rank is signified by bold font and a grey-colored cell. We have prepared similar tables for the standard deviation of the MMCE values for all scenarios in the Appendix (Table A2, Table A3, Table A4 and Table A5). As the observed variability is low and homogeneous between the imputation methods, we will only focus on the MMCE ranks from now on. It can be seen that the optimal imputation method varied for the different classifiers.
For RF, SVM and XGBoost, MICE RF performed best overall by leading to the lowest mean MMCE ranks in more scenarios than the other imputation methods. For SVM, MICE PMM and missForest PMM were close in performance to MICE RF. In contrast to the other classifiers, MICE RF did not perform as well in the case of EN-LR. Instead, MICE PMM and mean imputation performed slightly better than the other imputation methods. Except for XGBoost, where MICE RF slightly suffered from the increased proportion of missing values while mean imputation benefitted from it, the proportion of missing values did not noticeably affect the results for RF, SVM and EN-LR. Overall, Hot Deck, MICE Norm and missForest imputation were less competitive in the MCAR case.
When looking at the results for the One at a Time MAR pattern, Table 3 shows that some of the results from the MCAR case carried over. MICE RF performed well again for RF, SVM and XGBoost winning about a third to a half of the scenarios. For XGBoost, however, MICE PMM and missForest performed similary well. For EN-LR, the results were less clear-cut as well with MICE PMM, MICE Norm and mean imputation similarly competing for the best performance. Overall, Hot Deck and missForest PMM imputation were not as competitive for this missing pattern. MICE Norm was only competitive for classification with EN-LR and fell behind for the other classifiers.
When using the One Third pattern for MAR missingness, Table 4 shows that missForest clearly outperformed the other imputation methods for classification with RF, SVM and XGBoost. For these three classifiers, missForest was consistently optimal under almost all missingness proportions for the Phoneme, Skin and Wilt datasets. The results were more mixed for EN-LR where MICE Norm and MICE RF performed slightly better than the other imputation methods. In contrast to the MCAR and default MAR mechanism where Hot Deck imputation fell behind in almost all scenarios, it regularly achieved the lowest mean MMCE ranks on the Haberman dataset.
Finally, Table 5 displays the results for the MAR missing mechanism using the Two Thirds pattern. It can be seen that in most scenarios either missForest or mean imputation led to the lowest mean MMCE rank. For EN-LR, missForest and mean imputation performed similarly. When using RF, mean imputation was optimal for nearly all combinations of dataset and missingness proportion. The results for SVM and XGBoost were tied, with missForest and mean imputation winning about a third of all scenarios each. The results for EN-LR and XGBoost were sensitive to the missingness proportion. For both classifiers, mean imputation benefitted similarly from an increased proportion of missing values. Apart from missForest and mean imputation as well as MICE Norm for EN-LR, the remaining imputation methods were seldomly competetive.

4. Discussion

In this work, we studied the effect of imputation on the classification error under different missing mechanisms and missing proportions. To this end, we compared seven imputation methods, namely Hot Deck imputation, MICE Norm, MICE RF, MICE PMM, missForest, missForest PMM and mean imputation. As classifiers, we used EN-LR, RF, SVM and XGBoost. In our simulation study, we found that the optimal imputation method depended on the classifier, missing mechanism, as well as missingness pattern.
For a MCAR mechanism, we found that imputation via MICE RF worked best for RF, SVM and XGBoost. For EN-LR, the results were more mixed. Between the three MAR missing patterns (One at a Time, One Third and Two Thirds) we studied, the results for the One at a Time missing pattern resembled the MCAR results the most. Since for this pattern, only one feature value at a time could be missing for any given observation, the range of possible dependency structures that can arise are limited. Compared to the other two patterns, this situation is most similar to the MCAR mechanism where no dependency structure is present. Further, since the One at a Time pattern was allowed to vary w.r.t. to the features selected for containing the missing value, whereas the One Third and Two Thirds had a fixed set of features (i.e., the middle third, or the first and last third, respectively) where missing values could occur, the former pattern leads to more uncertainty. As such, the results for MCAR and One at a Time MAR are plausible, because MICE is designed to handle imputation uncertainty through multiple imputation. The missForest method, on the other hand, does not use multiple imputation and accordingly performed better in scenarios that included less uncertainty, that is, when using the One Third and Two Thirds patterns where it was optimal for many combinations of dataset, classifier and missing proportion.
Concerning practical insights, our study showed that RF-based imputation worked well under all MCAR and MAR missing mechanisms considered here. However, the optimality of MICE RF and missForest varied depending on the missing mechanism and pattern. Thus, this needs to be considered when using either. Even though the missing mechanism is generally unknown in practice, it is often feasible to form some assumptions based on the data-generating process. In most scenarios one will find that the underlying missingness seldomly follows a true MCAR mechanism. Potential patterns of missingness can also be gauged from exploratory data analysis by analyzing the presence and frequency of missing values. Alternatively, we found mean imputation to be a viable option when many features contained missing values. However, there might be a caveat to this finding. We did not specifically simulate the data and its distribution, so we did not explicitly examine cases in which the assumptions of mean imputation are violated or challenged. Our findings in this regard might have benefitted from studying classification as opposed to regression tasks. Future research should examine the impact of missing and mean imputation for heavily skewed features, for instance.
As for the MICE results, it should be noted that our approach of aggregating the imputed datasets did not make use of MICE‘s inherent advantage of controlling for the between-imputation variability by performing model analysis on the individual imputed datasets and subsequently averaging the resulting models. When performing inference or studying uncertainty, this step is detrimental as otherwise resulting standard errors are overconfident or Type I errors inflated, respectively. As we were only interested in studying classification errors, we have decided for the data aggregation to keep the simulation setup more consistent for all imputation methods and to limit the computational complexity (as each imputed dataset would have required a costly individual hyperparameter tuning step). However, as some of the MICE methods were not as competitive in our simulation study, future work should (if the computational resources permit) study whether the fitting and averaging of classification models on the individual imputed datasets leads to different results regarding the classification error.
Future work could also include listwise deletion as a benchmark method. We have refrained from using it here since the nested resampling approach resulted in small data subsets in the inner cross-validation folds and reducing the sample sizes even more through listwise deletion led to numerical issues with the logistic regression classifier on the smaller datasets from our simulation. Thus, we have decided to exclude this benchmark method for reasons of consistency.
Overall, our results indicated the importance of not only considering the missing mechanism when imputing, but also the pattern of missingness. The fact that the imputation methods were quite sensitive to the pattern choice, warrants further research in the future to investigate the effect of missingness patterns on imputation quality in more detail. This also includes studying more realistic missingness patterns. In our simulation study, the distinction between features that could contain missing values and those that could not was imposed by their order in the dataset (e.g., missings could only occur in the middle third of the features). In theory, this may occur in survey scenarios where a part of the questions is blocked from certain respondents (e.g., through filter questions). However, while helping to standardize the simulation process for all the different datasets, this design choice did not realistically reflect the common occurrence of relationships between features where the value of one feature regulates the probability of missingness for another feature. For example, in an online survey context, older people may have higher probabilities of missing answers or not completing their survey since they may be more challenged by technical aspects of the survey than younger participants. In another example, in-person measurements could be affected by the place of residence where respondents living far away might be more inclined to miss measurements due to the long travel time or due to insufficient public transportation options. For future work, one could design missingness patterns to better reflect such phenomena and thus make them more realistic. Instead of randomly selecting the features that may contain missing values, one could also study how imputation is affected when missingness is induced in “important” features (as measured by variable importance measures for example). Furthermore, to limit the scope of our analysis we only considered datasets with numerical features. It would be interesting to study whether imputation for categorical features yields different results. This may also impact the performance of Hot Deck imputation which is more suitable to categorical features. To conclude, our work showed that (i) using modern RF imputation methods such as MICE RF or missForest may be favorable in terms of subsequent classification accuracy and that (ii) basing the choice of imputation method on the context in which they are to be used, may lead to improved classification performance.

Author Contributions

Conceptualization, P.B. and M.P.; methodology, P.B. and M.P.; software, P.B.; validation, P.B and M.P.; formal analysis, P.B. and M.P.; investigation, P.B.; writing—original draft preparation, P.B.; writing—review and editing, P.B., M.P. and J.-J.C.; visualization, P.B.; supervision, M.P.; project administration, M.P. All authors have read and agreed to the published version of the manuscript.

Funding

The work of Markus Pauly was supported within the DFG project PA 2409/3-2.

Data Availability Statement

The Phoneme and Pima Indians datasets were obtained from the Open Machine Learning Project [17]. The Haberman, Skin, SPECT and Wilt datasets were obtained from the UCI Machine Learning Repository [20]. The R script for our simulation study is available at our OSF Repository https://osf.io/3z9sb/ (accessed on 10 March 2023).

Acknowledgments

The authors gratefully acknowledge the computing time provided on the Linux HPC cluster at Technical University Dortmund (LiDO3), partially funded in the course of the Large-Scale Equipment Initiative by the German Research Foundation (DFG) as project 271512359.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table
Table A1. Hyperparameters and search spaces for tuning.
Table A1. Hyperparameters and search spaces for tuning.
ClassifierHyperparameterSearch SpaceTransformation
EN-LRalpha [ 0 , 1 ] –
lambda [ − 15 , 15 ] 2 x
RFmtry { 2 , … , # features } –
min.node.size { 1 , … , 10 } –
splitrule{gini, extratrees}–
SVMcost [ − 5 , 5 ] 2 x
gamma [ − 5 , 5 ] 2 x
XGBoostnrounds { 10 , … , 200 } –
max_depth { 1 , … , 20 } –
eta [ 0.1 , 0.5 ] –
lambda [ − 1 , 0 ] 10 x
Table
Table A2. Standard deviations of MMCE values for imputation methods under a MCAR mechanism.
Table A2. Standard deviations of MMCE values for imputation methods under a MCAR mechanism.
Classifier% Miss.DatasetHot DeckMICE NormMICE PMMMICE RFmissForestmissForest PMMMean
EN-LR10%Haberman0.0070.0080.0080.0080.0070.0080.007
Phoneme0.0010.0010.0010.0010.0010.0010.001
Pima Indians0.0040.0040.0040.0050.0050.0050.004
Skin0.0000.0000.0000.0000.0010.0000.000
SPECT0.0130.0110.0120.0110.0110.0110.011
Wilt0.0340.0360.0300.0360.0310.0400.028
30%Haberman0.0060.0080.0080.0070.0070.0070.007
Phoneme0.0010.0010.0010.0010.0010.0010.001
Pima Indians0.0040.0040.0050.0050.0050.0040.005
Skin0.0000.0000.0000.0000.0020.0010.000
SPECT0.0120.0120.0100.0110.0100.0120.011
Wilt0.0000.0190.0140.0190.0070.0000.000
50%Haberman0.0070.0080.0070.0080.0070.0070.008
Phoneme0.0010.0010.0010.0010.0010.0010.001
Pima Indians0.0050.0050.0040.0040.0050.0050.004
Skin0.0010.0000.0000.0000.0020.0020.000
SPECT0.0140.0110.0110.0110.0110.0110.012
Wilt0.0000.0010.0010.0090.0000.0000.000
RF10%Haberman0.0110.0110.0110.0120.0100.0120.011
Phoneme0.0020.0020.0020.0020.0020.0020.002
Pima Indians0.0070.0070.0060.0070.0070.0070.006
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0110.0110.0100.0120.0130.0120.011
Wilt0.0010.0010.0010.0010.0010.0010.001
30%Haberman0.0160.0110.0120.0140.0120.0130.012
Phoneme0.0020.0020.0020.0030.0020.0020.002
Pima Indians0.0060.0060.0070.0060.0070.0070.007
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0110.0110.0110.0120.0120.0120.011
Wilt0.0010.0010.0010.0010.0010.0010.001
50%Haberman0.0150.0140.0150.0140.0140.0130.013
Phoneme0.0030.0030.0020.0020.0020.0030.002
Pima Indians0.0080.0070.0060.0060.0070.0080.007
Skin0.0000.0000.0000.0000.0010.0010.000
SPECT0.0120.0110.0110.0120.0110.0120.011
Wilt0.0010.0010.0010.0010.0010.0010.001
SVM10%Haberman0.0100.0090.0080.0090.0090.0090.009
Phoneme0.0030.0030.0030.0030.0030.0030.002
Pima Indians0.0060.0060.0060.0060.0060.0060.007
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0120.0150.0120.0110.0120.0140.013
Wilt0.0010.0010.0010.0010.0010.0010.001
30%Haberman0.0100.0100.0090.0080.0110.0080.009
Phoneme0.0030.0030.0030.0030.0030.0030.004
Pima Indians0.0060.0060.0070.0060.0070.0060.006
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0130.0130.0130.0120.0130.0130.013
Wilt0.0010.0010.0010.0010.0010.0010.001
50%Haberman0.0080.0090.0080.0100.0090.0080.009
Phoneme0.0030.0030.0030.0030.0030.0030.003
Pima Indians0.0070.0070.0060.0060.0060.0070.007
Skin0.0000.0000.0000.0000.0010.0010.000
SPECT0.0140.0130.0130.0130.0140.0140.013
Wilt0.0010.0010.0010.0010.0020.0010.001
XGB10%Haberman0.0150.0140.0130.0150.0150.0130.015
Phoneme0.0020.0030.0020.0030.0020.0020.002
Pima Indians0.0090.0100.0080.0080.0090.0090.009
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0120.0120.0120.0120.0120.0120.012
Wilt0.0010.0010.0010.0010.0010.0010.001
30%Haberman0.0150.0150.0150.0160.0150.0150.014
Phoneme0.0030.0030.0030.0030.0030.0030.003
Pima Indians0.0090.0090.0090.0080.0080.0080.008
Skin0.0010.0000.0000.0000.0010.0010.000
SPECT0.0120.0110.0120.0130.0120.0110.012
Wilt0.0010.0010.0010.0010.0010.0010.001
50%Haberman0.0160.0150.0150.0150.0150.0160.016
Phoneme0.0030.0030.0030.0030.0030.0030.003
Pima Indians0.0080.0090.0090.0090.0100.0090.008
Skin0.0010.0010.0010.0000.0010.0010.000
SPECT0.0120.0120.0130.0130.0110.0110.012
Wilt0.0010.0010.0010.0010.0010.0020.001
Table
Table A3. Standard deviations of MMCE values for imputation methods under a MAR mechanism with One at a Time pattern.
Table A3. Standard deviations of MMCE values for imputation methods under a MAR mechanism with One at a Time pattern.
Classifier% Miss.DatasetHot DeckMICE NormMICE PMMMICE RFmissForestmissForest PMMMean
EN-LR10%Haberman0.0070.0070.0070.0080.0080.0080.007
Phoneme0.0010.0010.0010.0010.0010.0010.001
Pima Indians0.0050.0040.0040.0040.0040.0050.004
Skin0.0000.0000.0000.0000.0010.0010.000
SPECT0.0120.0110.0120.0100.0110.0110.011
Wilt0.0140.0410.0360.0330.0160.0210.000
30%Haberman0.0070.0080.0080.0080.0080.0070.007
Phoneme0.0010.0010.0010.0010.0010.0010.001
Pima Indians0.0050.0040.0050.0040.0040.0050.005
Skin0.0000.0000.0000.0000.0020.0020.000
SPECT0.0120.0100.0120.0120.0110.0120.012
Wilt0.0000.0150.0370.0260.0160.0010.000
50%Haberman0.0070.0080.0080.0070.0080.0060.007
Phoneme0.0010.0010.0010.0010.0010.0010.001
Pima Indians0.0050.0040.0040.0050.0050.0040.005
Skin0.0010.0000.0000.0000.0030.0020.001
SPECT0.0130.0100.0120.0110.0110.0110.011
Wilt0.0000.0060.0100.0160.0060.0000.000
RF10%Haberman0.0100.0120.0120.0120.0120.0110.013
Phoneme0.0020.0020.0020.0020.0020.0020.002
Pima Indians0.0060.0060.0060.0070.0060.0060.006
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0120.0110.0120.0120.0110.0110.011
Wilt0.0010.0010.0010.0010.0010.0010.001
30%Haberman0.0130.0130.0110.0130.0140.0140.014
Phoneme0.0020.0020.0020.0020.0020.0030.003
Pima Indians0.0060.0070.0070.0070.0070.0070.006
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0110.0110.0100.0120.0110.0110.011
Wilt0.0010.0010.0010.0010.0010.0010.001
50%Haberman0.0150.0140.0150.0150.0130.0140.014
Phoneme0.0030.0030.0030.0020.0030.0020.002
Pima Indians0.0070.0070.0070.0070.0070.0070.007
Skin0.0010.0000.0000.0000.0010.0010.000
SPECT0.0120.0110.0110.0110.0110.0110.012
Wilt0.0010.0010.0010.0010.0010.0010.001
SVM10%Haberman0.0100.0100.0100.0100.0110.0090.009
Phoneme0.0030.0030.0030.0020.0030.0030.003
Pima Indians0.0060.0050.0070.0070.0060.0060.006
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0140.0130.0130.0120.0130.0140.014
Wilt0.0010.0010.0010.0010.0010.0010.001
30%Haberman0.0090.0110.0090.0090.0100.0110.009
Phoneme0.0030.0040.0040.0030.0030.0030.004
Pima Indians0.0060.0070.0060.0060.0070.0070.006
Skin0.0000.0000.0000.0000.0010.0010.000
SPECT0.0130.0120.0130.0130.0130.0130.012
Wilt0.0010.0010.0010.0010.0010.0010.001
50%Haberman0.0090.0100.0080.0110.0080.0090.008
Phoneme0.0030.0030.0030.0030.0040.0030.003
Pima Indians0.0070.0070.0070.0070.0070.0070.007
Skin0.0010.0000.0000.0000.0010.0010.000
SPECT0.0160.0130.0140.0130.0120.0130.014
Wilt0.0010.0010.0010.0010.0020.0010.001
XGB10%Haberman0.0130.0130.0140.0150.0140.0160.013
Phoneme0.0020.0020.0020.0020.0030.0020.003
Pima Indians0.0080.0090.0080.0080.0090.0100.009
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0130.0130.0120.0120.0120.0130.012
Wilt0.0010.0010.0010.0010.0010.0010.001
30%Haberman0.0140.0130.0140.0160.0150.0140.015
Phoneme0.0030.0030.0030.0030.0030.0030.003
Pima Indians0.0100.0090.0090.0080.0080.0100.008
Skin0.0000.0000.0000.0000.0010.0010.000
SPECT0.0130.0130.0130.0140.0130.0130.012
Wilt0.0010.0010.0010.0010.0010.0010.001
50%Haberman0.0160.0150.0150.0160.0160.0130.014
Phoneme0.0030.0030.0030.0030.0030.0030.002
Pima Indians0.0090.0100.0100.0090.0090.0090.008
Skin0.0010.0000.0000.0000.0010.0010.000
SPECT0.0130.0120.0120.0130.0130.0120.012
Wilt0.0010.0010.0010.0010.0010.0010.001
Table
Table A4. Standard deviations of MMCE values for imputation methods under a MAR mechanism with One Third pattern.
Table A4. Standard deviations of MMCE values for imputation methods under a MAR mechanism with One Third pattern.
Classifier% Miss.DatasetHot DeckMICE NormMICE PMMMICE RFmissForestmissForest PMMMean
EN-LR10%Haberman0.0070.0070.0070.0070.0070.0070.008
Phoneme0.0010.0010.0010.0010.0010.0010.001
Pima Indians0.0040.0050.0050.0050.0050.0050.004
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0110.0110.0120.0130.0120.0120.012
Wilt0.0010.0290.0330.0190.0370.0370.001
30%Haberman0.0080.0070.0080.0080.0080.0070.007
Phoneme0.0010.0010.0010.0010.0010.0010.001
Pima Indians0.0050.0050.0050.0040.0060.0050.005
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0110.0110.0130.0110.0130.0130.012
Wilt0.0000.0010.0020.0220.0020.0020.001
50%Haberman0.0070.0070.0080.0080.0080.0070.007
Phoneme0.0010.0010.0010.0010.0010.0010.001
Pima Indians0.0050.0050.0050.0050.0060.0050.005
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0110.0110.0130.0120.0130.0140.012
Wilt0.0000.0010.0010.0010.0020.0010.000
RF10%Haberman0.0110.0120.0120.0110.0110.0110.013
Phoneme0.0020.0020.0020.0020.0020.0020.002
Pima Indians0.0060.0070.0070.0070.0060.0060.007
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0110.0110.0120.0130.0120.0120.011
Wilt0.0010.0010.0010.0010.0010.0010.001
30%Haberman0.0120.0130.0130.0120.0140.0140.013
Phoneme0.0020.0020.0020.0020.0020.0020.002
Pima Indians0.0070.0070.0080.0070.0070.0070.007
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0130.0110.0110.0110.0110.0110.012
Wilt0.0010.0010.0010.0010.0010.0010.001
50%Haberman0.0130.0150.0150.0140.0120.0140.013
Phoneme0.0020.0020.0020.0020.0020.0020.002
Pima Indians0.0080.0070.0070.0070.0070.0080.007
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0130.0120.0120.0110.0130.0120.013
Wilt0.0020.0010.0010.0010.0010.0010.001
SVM10%Haberman0.0100.0110.0100.0100.0110.0090.010
Phoneme0.0030.0030.0020.0030.0030.0030.002
Pima Indians0.0060.0060.0060.0070.0060.0060.006
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0130.0140.0130.0140.0130.0140.015
Wilt0.0010.0010.0010.0010.0010.0010.001
30%Haberman0.0100.0110.0100.0120.0110.0120.010
Phoneme0.0030.0030.0030.0030.0030.0030.003
Pima Indians0.0070.0070.0070.0070.0070.0070.007
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0130.0130.0140.0140.0150.0130.017
Wilt0.0010.0010.0010.0010.0010.0010.001
50%Haberman0.0110.0110.0100.0100.0120.0110.010
Phoneme0.0030.0030.0030.0030.0030.0030.003
Pima Indians0.0080.0080.0060.0090.0090.0060.007
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0160.0150.0150.0140.0150.0140.018
Wilt0.0010.0010.0010.0010.0010.0010.001
XGB10%Haberman0.0150.0150.0140.0140.0140.0130.013
Phoneme0.0020.0020.0020.0020.0020.0020.002
Pima Indians0.0080.0090.0080.0090.0080.0100.010
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0120.0140.0120.0130.0140.0120.012
Wilt0.0010.0010.0010.0010.0010.0010.001
30%Haberman0.0140.0150.0140.0140.0150.0150.015
Phoneme0.0030.0030.0020.0020.0020.0020.002
Pima Indians0.0080.0090.0100.0080.0090.0080.009
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0140.0130.0120.0130.0130.0120.012
Wilt0.0010.0010.0010.0010.0010.0010.001
50%Haberman0.0160.0140.0150.0130.0150.0130.015
Phoneme0.0030.0020.0020.0030.0020.0030.003
Pima Indians0.0080.0080.0090.0090.0100.0090.008
Skin0.0010.0000.0000.0000.0000.0000.000
SPECT0.0130.0110.0140.0120.0130.0140.014
Wilt0.0010.0010.0010.0010.0010.0010.001
Table
Table A5. Standard deviations of MMCE values for imputation methods under a MAR mechanism with Two Thirds pattern.
Table A5. Standard deviations of MMCE values for imputation methods under a MAR mechanism with Two Thirds pattern.
Classifier% Miss.DatasetHot DeckMICE NormMICE PMMMICE RFmissForestmissForest PMMMean
EN-LR10%Haberman0.0080.0080.0090.0080.0090.0080.007
Phoneme0.0010.0010.0010.0010.0010.0010.001
Pima Indians0.0050.0050.0050.0050.0050.0050.005
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0130.0130.0130.0110.0120.0140.012
Wilt0.0010.0270.0150.0090.0130.0030.000
30%Haberman0.0070.0080.0070.0090.0090.0070.009
Phoneme0.0050.0050.0060.0020.0010.0050.002
Pima Indians0.0060.0050.0050.0060.0050.0060.007
Skin0.0020.0000.0010.0010.0010.0030.000
SPECT0.0150.0150.0150.0140.0140.0140.015
Wilt0.0000.0010.0160.0090.0010.0020.000
50%Haberman0.0050.0090.0100.0090.0090.0050.010
Phoneme0.0010.0000.0020.0060.0050.0020.007
Pima Indians0.0090.0070.0070.0060.0060.0080.006
Skin0.0000.0020.0030.0020.0030.0120.001
SPECT0.0150.0200.0170.0150.0150.0140.016
Wilt0.0000.0020.0000.0000.0090.0000.000
RF10%Haberman0.0120.0110.0130.0120.0130.0130.012
Phoneme0.0020.0020.0020.0020.0020.0020.002
Pima Indians0.0080.0070.0060.0070.0070.0080.006
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0120.0120.0120.0120.0110.0140.012
Wilt0.0010.0010.0010.0010.0010.0010.001
30%Haberman0.0120.0120.0130.0120.0130.0140.012
Phoneme0.0030.0030.0030.0030.0030.0030.003
Pima Indians0.0080.0080.0070.0070.0080.0090.007
Skin0.0000.0000.0000.0000.0000.0010.000
SPECT0.0150.0160.0120.0140.0120.0140.014
Wilt0.0020.0010.0010.0010.0010.0010.001
50%Haberman0.0140.0120.0150.0140.0140.0150.014
Phoneme0.0040.0030.0030.0040.0030.0030.003
Pima Indians0.0100.0080.0100.0090.0080.0090.009
Skin0.0010.0010.0000.0000.0000.0010.000
SPECT0.0160.0160.0140.0140.0150.0150.014
Wilt0.0010.0010.0010.0010.0010.0020.001
SVM10%Haberman0.0090.0110.0090.0090.0100.0100.009
Phoneme0.0030.0030.0030.0030.0030.0030.003
Pima Indians0.0070.0060.0060.0070.0060.0070.006
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0120.0130.0140.0140.0130.0140.014
Wilt0.0010.0010.0010.0010.0010.0010.001
30%Haberman0.0070.0090.0080.0090.0100.0080.010
Phoneme0.0040.0040.0030.0040.0030.0040.003
Pima Indians0.0090.0070.0070.0080.0080.0080.008
Skin0.0000.0000.0000.0000.0010.0010.000
SPECT0.0150.0170.0160.0150.0170.0160.015
Wilt0.0010.0010.0010.0010.0010.0010.001
50%Haberman0.0070.0080.0080.0090.0100.0060.012
Phoneme0.0050.0030.0040.0040.0040.0040.004
Pima Indians0.0120.0090.0100.0100.0110.0130.010
Skin0.0010.0000.0000.0000.0010.0030.000
SPECT0.0150.0160.0180.0150.0140.0160.012
Wilt0.0020.0010.0010.0010.0010.0020.001
XGB10%Haberman0.0120.0150.0130.0130.0130.0140.014
Phoneme0.0030.0030.0030.0030.0030.0030.003
Pima Indians0.0100.0090.0090.0100.0080.0110.009
Skin0.0000.0000.0000.0000.0000.0000.000
SPECT0.0150.0140.0120.0140.0130.0140.012
Wilt0.0010.0010.0010.0010.0010.0010.001
30%Haberman0.0150.0130.0140.0140.0150.0150.016
Phoneme0.0040.0030.0030.0030.0030.0040.004
Pima Indians0.0100.0100.0110.0100.0080.0120.009
Skin0.0010.0010.0000.0000.0000.0010.000
SPECT0.0140.0150.0130.0140.0160.0170.015
Wilt0.0010.0010.0010.0010.0010.0010.001
50%Haberman0.0150.0130.0140.0140.0160.0160.013
Phoneme0.0050.0040.0040.0050.0040.0060.004
Pima Indians0.0130.0100.0120.0130.0110.0120.010
Skin0.0020.0010.0010.0010.0010.0020.001
SPECT0.0160.0160.0140.0150.0140.0160.014
Wilt0.0020.0010.0010.0010.0020.0020.001

References

  1. Rubin, D.B. Inference and missing data. Biometrika 1976, 63, 581–592. [Google Scholar] [CrossRef]
  2. Van Buuren, S.; Groothuis-Oudshoorn, K. mice: Multivariate Imputation by Chained Equations in R. J. Stat. Softw. 2011, 45, 1–67. [Google Scholar] [CrossRef] [Green Version]
  3. Van Buuren, S. Flexible Imputation of Missing Data; CRC Press: Boca Raton, FL, USA, 2018. [Google Scholar]
  4. Stekhoven, D.J.; Bühlmann, P. MissForest—Non-parametric missing value imputation for mixed-type data. Bioinformatics 2011, 28, 112–118. [Google Scholar] [CrossRef] [Green Version]
  5. Liao, S.G.; Lin, Y.; Kang, D.D.; Chandra, D.; Bon, J.; Kaminski, N.; Sciurba, F.C.; Tseng, G.C. Missing value imputation in high-dimensional phenomic data: Imputable or not, and how? BMC Bioinform. 2014, 15, 1–12. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  6. Mayer, M. missRanger: Fast Imputation of Missing Values, 2019. R package version 2.1.0. Available online: https://CRAN.R-project.org/package=missRanger (accessed on 20 December 2022).
  7. Ramosaj, B.; Pauly, M. Predicting missing values: A comparative study on non-parametric approaches for imputation. Comput. Stat. 2019, 34, 1741–1764. [Google Scholar] [CrossRef]
  8. Little, R.J.; Rubin, D.B. Statistical Analysis with Missing Data; John Wiley & Sons: Hoboken, NJ, USA, 2019; Volume 793. [Google Scholar]
  9. Ramosaj, B.; Amro, L.; Pauly, M. A cautionary tale on using imputation methods for inference in matched-pairs design. Bioinformatics 2020, 36, 3099–3106. [Google Scholar] [CrossRef] [PubMed]
  10. Ramosaj, B.; Tulowietzki, J.; Pauly, M. On the Relation between Prediction and Imputation Accuracy under Missing Covariates. Entropy 2022, 24, 386. [Google Scholar] [CrossRef]
  11. Thurow, M.; Dumpert, F.; Ramosaj, B.; Pauly, M. Imputing missings in official statistics for general tasks–our vote for distributional accuracy. Stat. J. IAOS 2021, 37, 1379–1390. [Google Scholar] [CrossRef]
  12. Farhangfar, A.; Kurgan, L.; Dy, J. Impact of imputation of missing values on classification error for discrete data. Pattern Recognit. 2008, 41, 3692–3705. [Google Scholar] [CrossRef]
  13. Twala, B. An Empirical Comparison of Techniques for Handling Incomplete Data Using Decision Trees. Appl. Artif. Intell. 2009, 23, 373–405. [Google Scholar] [CrossRef]
  14. Ding, Y.; Simonoff, J.S. An Investigation of Missing Data Methods for Classification Trees Applied to Binary Response Data. J. Mach. Learn. Res. 2010, 11, 131–170. [Google Scholar]
  15. García-Laencina, P.J.; Sancho-Gómez, J.L.; Figueiras-Vidal, A.R. Pattern Classification with Missing Data: A Review. Neural Comput. Appl. 2010, 19, 263–282. [Google Scholar] [CrossRef]
  16. Thurow, M.; Dumpert, F.; Ramosaj, B.; Pauly, M. Goodness (of fit) of imputation accuracy: The GoodImpact analysis. arXiv 2021, arXiv:2101.07532. [Google Scholar]
  17. Vanschoren, J.N.; van Rijn, J.; Bischl, B.; Torgo, L. OpenML: Networked Science in Machine Learning. SIGKDD Explor. 2013, 15, 49–60. [Google Scholar] [CrossRef] [Green Version]
  18. Bhatt, R.B.; Sharma, G.; Dhall, A.; Chaudhury, S. Efficient Skin Region Segmentation Using Low Complexity Fuzzy Decision Tree Model. In Proceedings of the 2009 Annual IEEE India Conference, Ahmedabad, India, 18–20 December 2009; pp. 1–4. [Google Scholar]
  19. Johnson, B.A.; Tateishi, R.; Hoan, N.T. A hybrid pansharpening approach and multiscale object-based image analysis for mapping diseased pine and oak trees. Int. J. Remote Sens. 2013, 34, 6969–6982. [Google Scholar] [CrossRef]
  20. Dua, D.; Graff, C. UCI Machine Learning Repository; School of Information and Computer Sciences, University of California: Irvine, CA, USA, 2022. [Google Scholar]
  21. Schouten, R.M.; Lugtig, P.; Vink, G. Generating missing values for simulation purposes: A multivariate amputation procedure. J. Stat. Comput. Simul. 2018, 88, 2909–2930. [Google Scholar] [CrossRef] [Green Version]
  22. Cranmer, S.J.; Gill, J. We Have to Be Discrete About This: A Non-Parametric Imputation Technique for Missing Categorical Data. Br. J. Political Sci. 2013, 43, 425–449. [Google Scholar] [CrossRef] [Green Version]
  23. Rubin, D.B. Multiple Imputation for Nonresponse in Surveys; Wiley: New York, NY, USA, 1987. [Google Scholar]
  24. Little, R.J.A. Missing-Data Adjustments in Large Surveys. J. Bus. Econ. Stat. 1988, 6, 287–296. [Google Scholar]
  25. Doove, L.; Van Buuren, S.; Dusseldorp, E. Recursive partitioning for missing data imputation in the presence of interaction effects. Comput. Stat. Data Anal. 2014, 72, 92–104. [Google Scholar] [CrossRef]
  26. R Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria, 2022. [Google Scholar]
  27. Gill, J.; Cranmer, S.; Jackson, N.; Murr, A.; Armstrong, D.; Heuberger, S. hot.deck: Multiple Hot Deck Imputation, 2021. R package version 1.2. Available online: https://CRAN.R-project.org/package=hot.deck (accessed on 20 December 2022).
  28. Wright, M.N.; Ziegler, A. ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R. J. Stat. Softw. 2017, 77, 1–17. [Google Scholar] [CrossRef] [Green Version]
  29. Friedman, J.; Hastie, T.; Tibshirani, R. Regularization Paths for Generalized Linear Models via Coordinate Descent. J. Stat. Softw. 2010, 33, 1–22. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  30. Meyer, D.; Dimitriadou, E.; Hornik, K.; Weingessel, A.; Leisch, F. e1071: Misc Functions of the Department of Statistics, Probability Theory Group (Formerly: E1071), TU Wien, 2022. R package version 1.7-11. Available online: https://CRAN.R-project.org/package=e1071 (accessed on 20 December 2022).
  31. Chen, T.; He, T.; Benesty, M.; Khotilovich, V.; Tang, Y.; Cho, H.; Chen, K.; Mitchell, R.; Cano, I.; Zhou, T.; et al. xgboost: Extreme Gradient Boosting, 2020. R package version 1.0.0.2. Available online: https://CRAN.R-project.org/package=xgboost (accessed on 20 December 2022).
  32. Cawley, G.C.; Talbot, N.L. On Over-fitting in Model Selection and Subsequent Selection bias in performance evaluation. J. Mach. Learn. Res. 2010, 11, 2079–2107. [Google Scholar]
Entropy 25 00521 g001 550
Figure 1. Workflow used in simulation study.
Figure 1. Workflow used in simulation study.
Entropy 25 00521 g001
Table
Table 1. Descriptions, class distributions and feature counts of datasets used in simulation study.
Table 1. Descriptions, class distributions and feature counts of datasets used in simulation study.
DatasetClass 1Class 0FeaturesDescription
Haberman225813Survival status of breast cancer patients using age at operation,
year of operation and number of positive axillary lymph nodes
Phoneme381815865Classification of oral and nasal sounds using five frequency-
related characteristics of the sound sample
Pima Indians5002688Diabetes status in indigenous population using features such as
BMI, blood pressure, insulin level, etc.
Skin254297093Segmentation of skin texture based on random samples of
RGB color values from face images
SPECT9525444Diagnosis of computed tomography using information about
22 regions of interest in stress and rest mode
Wilt45782615Detection of diseased trees in segments of pansharpened
images using spectral and texture information
Table
Table 2. Mean MMCE ranks (lower = better) for imputation methods under a MCAR mechanism. Best value per row printed in bold and colored grey.
Table 2. Mean MMCE ranks (lower = better) for imputation methods under a MCAR mechanism. Best value per row printed in bold and colored grey.
Classifier% Miss.DatasetHot DeckMICE NormMICE PMMMICE RFmissForestmissForest PMMMean
EN-LR10%Haberman4.164.024.083.833.764.323.85
Phoneme5.664.614.573.493.484.621.57
Pima Indians4.403.674.383.384.344.033.81
Skin4.191.591.482.975.015.996.77
SPECT4.443.923.873.673.964.064.08
Wilt4.473.433.203.394.084.664.76
30%Haberman4.324.033.634.293.844.343.56
Phoneme6.604.464.763.013.254.861.07
Pima Indians4.203.743.744.124.104.183.91
Skin4.631.761.243.004.655.896.82
SPECT5.004.193.883.603.403.924.02
Wilt5.732.021.973.104.105.345.74
50%Haberman4.543.803.843.933.754.413.74
Phoneme6.924.264.622.793.075.331.01
Pima Indians4.223.833.613.553.634.534.62
Skin5.051.701.442.893.995.937.00
SPECT5.243.803.903.713.883.623.87
Wilt5.501.641.443.015.435.505.50
RF10%Haberman4.143.814.393.943.943.903.90
Phoneme4.434.313.922.903.754.004.69
Pima Indians3.983.944.153.884.114.033.92
Skin4.663.963.171.364.644.216.00
SPECT3.884.083.714.394.123.784.04
Wilt6.232.852.712.624.244.804.54
30%Haberman4.343.933.964.413.923.723.71
Phoneme4.885.383.892.103.183.894.70
Pima Indians4.024.064.014.444.243.753.49
Skin5.464.503.121.026.185.212.50
SPECT4.153.824.144.094.103.793.92
Wilt6.851.982.102.675.205.803.41
50%Haberman4.163.624.244.023.784.084.12
Phoneme5.575.484.061.622.714.364.20
Pima Indians4.244.164.193.983.913.843.68
Skin6.083.943.181.045.835.941.99
SPECT4.283.863.994.023.983.764.12
Wilt6.862.161.613.045.215.933.18
SVM10%Haberman3.684.004.264.033.644.284.11
Phoneme4.044.703.793.213.903.904.45
Pima Indians3.713.713.924.324.223.894.22
Skin5.343.522.511.324.574.276.48
SPECT5.244.103.313.814.113.763.68
Wilt5.962.011.742.344.224.806.93
30%Haberman4.074.284.124.164.063.503.79
Phoneme3.885.844.961.912.903.984.53
Pima Indians3.983.943.983.963.984.283.88
Skin5.663.302.221.025.335.115.36
SPECT6.563.613.343.443.613.813.64
Wilt6.012.241.292.464.204.826.98
50%Haberman3.803.834.384.183.803.804.22
Phoneme4.315.945.261.422.864.413.80
Pima Indians3.944.364.043.804.073.644.16
Skin6.423.082.141.005.345.644.39
SPECT6.953.463.463.523.603.413.61
Wilt6.032.321.092.594.064.956.96
XGB10%Haberman4.283.794.224.043.793.864.03
Phoneme4.264.344.373.383.514.064.08
Pima Indians3.954.253.983.874.044.043.88
Skin5.844.262.971.475.125.642.70
SPECT3.773.853.784.114.224.124.16
Wilt6.112.502.262.484.995.394.27
30%Haberman4.283.903.934.074.104.173.56
Phoneme5.224.823.902.413.444.064.14
Pima Indians4.123.644.323.783.884.214.06
Skin6.104.023.091.495.795.791.73
SPECT3.883.964.084.223.944.273.65
Wilt6.722.082.112.405.195.893.61
50%Haberman4.304.044.264.053.504.003.87
Phoneme5.764.784.002.013.224.483.76
Pima Indians4.113.873.844.403.884.243.66
Skin6.203.853.191.615.396.331.43
SPECT4.143.734.113.814.084.253.88
Wilt6.682.291.922.925.365.942.88
Table
Table 3. Mean MMCE ranks (lower = better) for imputation methods under a MAR mechanism with One at a Time pattern. Best value per row printed in bold and colored grey.
Table 3. Mean MMCE ranks (lower = better) for imputation methods under a MAR mechanism with One at a Time pattern. Best value per row printed in bold and colored grey.
Classifier% Miss.DatasetHot DeckMICE NormMICE PMMMICE RFmissForestmissForest PMMMean
EN-LR10%Haberman4.523.624.163.653.944.193.92
Phoneme5.414.374.673.563.694.601.70
Pima Indians4.543.944.014.003.413.964.14
Skin4.611.651.472.905.115.486.79
SPECT4.433.823.833.654.124.144.01
Wilt4.303.643.573.733.683.745.34
30%Haberman4.514.093.643.764.114.113.78
Phoneme6.554.154.483.293.465.011.07
Pima Indians4.093.854.044.023.943.974.09
Skin4.601.671.333.004.986.196.23
SPECT5.064.063.603.763.693.674.17
Wilt5.541.743.033.463.814.875.54
50%Haberman4.783.703.694.283.784.203.57
Phoneme6.873.704.643.123.405.261.01
Pima Indians4.383.483.603.773.924.324.53
Skin5.041.661.512.834.375.616.98
SPECT5.543.853.603.873.693.444.00
Wilt5.421.291.923.385.155.425.42
RF10%Haberman3.483.974.163.963.824.004.62
Phoneme4.364.374.093.303.524.104.26
Pima Indians4.214.244.213.593.963.794.00
Skin5.263.662.841.424.334.615.88
SPECT4.503.963.833.883.604.024.22
Wilt6.342.562.782.464.105.014.74
30%Haberman4.543.674.064.023.624.094.00
Phoneme5.144.934.061.903.274.544.16
Pima Indians4.413.764.643.673.723.883.92
Skin6.173.522.821.045.735.992.73
SPECT3.964.023.904.253.634.034.22
Wilt6.892.141.932.554.955.943.60
50%Haberman4.383.563.833.763.884.264.34
Phoneme5.605.153.781.373.044.954.12
Pima Indians4.323.854.083.903.933.704.20
Skin6.283.992.881.015.576.102.17
SPECT4.203.624.224.183.604.014.18
Wilt6.812.061.842.635.305.893.46
SVM10%Haberman4.004.133.834.283.903.764.11
Phoneme3.884.384.203.293.303.875.10
Pima Indians4.423.904.283.743.863.993.83
Skin5.363.062.281.644.354.846.46
SPECT5.614.043.833.473.533.763.77
Wilt6.122.101.932.244.124.866.62
30%Haberman3.723.904.204.434.083.943.71
Phoneme4.195.404.861.682.844.444.59
Pima Indians4.143.924.223.983.833.884.03
Skin6.122.842.191.105.225.704.82
SPECT6.753.383.313.823.863.443.46
Wilt6.162.141.452.414.124.946.78
50%Haberman3.834.123.904.324.123.903.82
Phoneme5.005.584.531.192.764.993.95
Pima Indians3.903.924.144.004.043.924.09
Skin6.622.942.181.015.285.724.24
SPECT6.973.383.313.803.663.713.17
Wilt6.132.201.252.544.114.906.86
XGB10%Haberman3.964.453.933.714.103.854.00
Phoneme4.243.854.183.483.464.054.74
Pima Indians4.043.804.113.834.013.984.24
Skin5.873.292.741.605.035.863.60
SPECT4.234.163.833.903.773.984.14
Wilt6.182.422.262.754.595.284.54
30%Haberman4.433.504.243.873.784.273.90
Phoneme5.594.343.882.353.594.533.73
Pima Indians3.964.103.883.903.863.834.47
Skin6.423.592.771.405.456.092.29
SPECT3.744.164.164.273.694.003.97
Wilt6.742.102.002.374.995.843.96
50%Haberman4.553.803.673.574.044.793.58
Phoneme5.794.874.021.853.394.743.33
Pima Indians4.143.953.524.084.143.934.24
Skin6.513.852.931.445.396.091.79
SPECT3.983.894.113.823.804.354.06
Wilt6.611.991.902.615.336.023.56
Table
Table 4. Mean MMCE ranks (lower = better) for imputation methods under a MAR mechanism with One Third pattern. Best value per row printed in bold and colored grey.
Table 4. Mean MMCE ranks (lower = better) for imputation methods under a MAR mechanism with One Third pattern. Best value per row printed in bold and colored grey.
Classifier% Miss.DatasetHot DeckMICE NormMICE PMMMICE RFmissForestmissForest PMMMean
EN-LR10%Haberman3.784.433.854.103.734.243.86
Phoneme4.942.403.904.784.945.101.95
Pima Indians4.433.794.004.363.883.963.58
Skin5.971.313.574.363.222.587.00
SPECT4.784.433.943.703.133.924.10
Wilt5.423.403.272.763.855.403.90
30%Haberman3.644.094.173.983.964.084.08
Phoneme6.573.996.172.683.853.151.59
Pima Indians3.993.924.143.814.223.933.98
Skin6.001.294.124.642.762.197.00
SPECT5.684.244.023.602.863.464.14
Wilt6.701.651.784.192.644.846.20
50%Haberman3.723.944.014.303.963.734.34
Phoneme7.004.525.401.903.851.983.36
Pima Indians4.353.674.043.814.044.094.01
Skin6.002.004.184.662.181.977.00
SPECT4.583.923.733.713.843.784.44
Wilt6.341.461.614.032.924.996.64
RF10%Haberman3.863.944.264.264.103.364.22
Phoneme5.534.324.643.602.863.783.27
Pima Indians3.883.583.684.204.123.954.58
Skin7.003.894.112.571.583.465.39
SPECT4.173.874.024.243.794.203.71
Wilt6.953.342.723.082.214.155.55
30%Haberman2.854.204.164.284.234.004.30
Phoneme6.465.275.042.891.993.273.07
Pima Indians3.984.013.564.164.243.834.22
Skin7.004.924.282.271.054.773.71
SPECT5.244.113.393.604.584.142.94
Wilt7.002.692.154.061.855.984.26
50%Haberman3.124.394.144.224.023.424.70
Phoneme6.545.185.363.061.913.772.19
Pima Indians3.833.693.774.194.343.504.68
Skin7.004.764.422.941.095.622.16
SPECT4.863.513.803.815.304.841.89
Wilt7.002.662.654.881.456.003.36
SVM10%Haberman3.974.184.183.693.813.794.38
Phoneme5.004.384.943.542.873.513.76
Pima Indians3.724.284.163.823.964.054.01
Skin6.993.883.903.291.663.734.55
SPECT4.853.963.643.833.703.834.18
Wilt6.222.942.442.872.064.696.78
30%Haberman3.524.294.283.773.973.774.40
Phoneme5.615.635.852.652.012.833.42
Pima Indians3.564.244.034.583.983.633.98
Skin7.004.384.752.901.164.842.98
SPECT5.643.513.163.474.223.374.62
Wilt7.003.102.183.621.115.006.00
50%Haberman3.294.604.503.664.203.614.14
Phoneme5.745.625.882.541.612.643.97
Pima Indians3.294.123.973.924.922.914.86
Skin7.004.624.703.121.285.361.92
SPECT4.973.183.363.544.862.006.10
Wilt7.002.802.383.821.015.055.95
XGB10%Haberman3.664.163.944.184.133.884.05
Phoneme4.704.204.723.623.103.793.87
Pima Indians4.123.804.303.943.684.223.92
Skin6.863.273.272.642.074.285.62
SPECT3.674.084.364.494.213.593.59
Wilt6.972.732.473.632.734.944.52
30%Haberman3.824.144.303.823.963.954.03
Phoneme5.475.415.343.202.223.253.10
Pima Indians3.504.183.944.204.263.973.96
Skin6.983.974.272.441.465.513.38
SPECT3.414.304.044.535.344.062.33
Wilt7.002.562.544.002.475.973.44
50%Haberman3.654.304.404.273.663.604.12
Phoneme5.825.535.513.311.913.532.39
Pima Indians3.744.044.023.794.273.644.50
Skin6.994.294.533.081.295.861.97
SPECT3.044.124.334.556.163.602.20
Wilt7.002.732.714.861.736.002.98
Table
Table 5. Mean MMCE ranks (lower = better) for imputation methods under a MAR mechanism with Two Thirds pattern. Best value per row printed in bold and colored grey.
Table 5. Mean MMCE ranks (lower = better) for imputation methods under a MAR mechanism with Two Thirds pattern. Best value per row printed in bold and colored grey.
Classifier% Miss.DatasetHot DeckMICE NormMICE PMMMICE RFmissForestmissForest PMMMean
EN-LR10%Haberman4.483.923.853.813.724.413.81
Phoneme6.815.364.322.991.775.491.26
Pima Indians4.463.613.874.573.193.724.57
Skin7.002.461.764.021.996.004.78
SPECT3.834.183.944.143.924.033.96
Wilt4.322.134.344.334.144.434.32
30%Haberman4.904.143.823.413.374.963.41
Phoneme5.626.425.862.991.934.101.08
Pima Indians5.773.943.583.623.543.813.74
Skin6.991.003.664.563.466.012.32
SPECT5.384.824.222.963.153.194.28
Wilt4.381.004.574.624.574.474.38
50%Haberman4.724.024.114.073.424.722.94
Phoneme5.825.965.382.971.144.821.92
Pima Indians6.743.463.702.873.005.502.72
Skin7.002.063.244.893.816.001.00
SPECT5.695.114.322.851.973.164.90
Wilt4.471.004.474.474.624.504.47
RF10%Haberman4.303.403.813.944.214.234.12
Phoneme5.433.843.444.023.404.643.23
Pima Indians4.014.013.763.983.984.873.40
Skin3.903.685.534.683.634.801.78
SPECT4.234.344.143.713.693.564.32
Wilt7.003.362.543.392.194.365.16
30%Haberman4.702.983.664.024.264.403.97
Phoneme6.483.943.663.592.536.181.62
Pima Indians4.883.833.953.593.365.522.88
Skin6.273.604.763.641.816.681.24
SPECT4.544.243.644.533.814.342.90
Wilt7.002.481.894.072.165.984.42
50%Haberman4.723.103.903.983.724.743.84
Phoneme6.323.993.843.892.176.661.13
Pima Indians5.853.713.673.182.566.502.52
Skin6.934.533.683.791.886.071.12
SPECT4.784.464.034.004.165.241.32
Wilt7.002.081.714.822.446.003.94
SVM10%Haberman3.533.904.344.043.854.174.18
Phoneme4.784.264.284.243.475.261.72
Pima Indians4.283.873.634.183.943.924.18
Skin3.853.835.195.005.313.441.39
SPECT4.764.233.923.982.953.814.36
Wilt7.002.652.292.772.504.865.93
30%Haberman4.084.304.143.813.763.854.08
Phoneme5.784.393.973.942.256.631.04
Pima Indians4.404.043.783.653.584.643.90
Skin4.922.453.553.036.386.621.04
SPECT3.794.524.084.792.693.954.18
Wilt7.002.891.863.381.895.705.28
50%Haberman3.864.064.144.143.544.154.11
Phoneme6.153.713.744.352.206.851.00
Pima Indians5.993.693.562.782.446.552.99
Skin5.072.483.083.425.937.001.03
SPECT2.224.334.444.743.773.624.88
Wilt7.002.711.604.181.696.004.82
XGB10%Haberman4.423.713.754.143.944.313.72
Phoneme5.553.963.503.592.884.893.63
Pima Indians3.833.883.763.853.554.574.57
Skin6.504.084.543.931.725.182.06
SPECT3.904.244.303.864.283.863.58
Wilt6.993.023.232.852.884.784.24
30%Haberman4.473.523.763.544.014.983.72
Phoneme6.273.803.493.551.926.682.29
Pima Indians4.404.284.133.602.945.243.39
Skin6.263.664.533.691.496.741.62
SPECT3.674.874.504.584.404.181.80
Wilt7.002.562.453.692.665.963.68
50%Haberman4.673.274.003.873.784.593.82
Phoneme6.093.883.564.202.136.911.22
Pima Indians5.404.033.863.292.566.082.80
Skin6.594.393.913.661.586.411.46
SPECT3.675.244.764.444.724.121.04
Wilt7.002.432.294.812.936.002.54
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Buczak, P.; Chen, J.-J.; Pauly, M. Analyzing the Effect of Imputation on Classification Performance under MCAR and MAR Missing Mechanisms. Entropy 2023, 25, 521. https://doi.org/10.3390/e25030521

AMA Style

Buczak P, Chen J-J, Pauly M. Analyzing the Effect of Imputation on Classification Performance under MCAR and MAR Missing Mechanisms. Entropy. 2023; 25(3):521. https://doi.org/10.3390/e25030521

Chicago/Turabian Style

Buczak, Philip, Jian-Jia Chen, and Markus Pauly. 2023. "Analyzing the Effect of Imputation on Classification Performance under MCAR and MAR Missing Mechanisms" Entropy 25, no. 3: 521. https://doi.org/10.3390/e25030521

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop