Experimental Study and Comparison of Imbalance Ensemble Classifiers with Dynamic Selection Strategy

Imbalance ensemble classification is one of the most essential and practical strategies for improving decision performance in data analysis. There is a growing body of literature about ensemble techniques for imbalance learning in recent years, the various extensions of imbalanced classification methods were established from different points of view. The present study is initiated in an attempt to review the state-of-the-art ensemble classification algorithms for dealing with imbalanced datasets, offering a comprehensive analysis for incorporating the dynamic selection of base classifiers in classification. By conducting 14 existing ensemble algorithms incorporating a dynamic selection on 56 datasets, the experimental results reveal that the classical algorithm with a dynamic selection strategy deliver a practical way to improve the classification performance for both a binary class and multi-class imbalanced datasets. In addition, by combining patch learning with a dynamic selection ensemble classification, a patch-ensemble classification method is designed, which utilizes the misclassified samples to train patch classifiers for increasing the diversity of base classifiers. The experiments’ results indicate that the designed method has a certain potential for the performance of multi-class imbalanced classification.


Introduction
Data imbalance is ubiquitous and encountered in the field of classification problems.It occurs when the number of instances for different classes are significantly out of proportion.The minority classes with fewer instances usually contain the essential information, which has been observed in broad application areas, such as medical diagnosis [1][2][3][4][5][6], sentiment or image classification [7,8], fault identification [9,10], etc.Many typical classifiers may generate unsatisfactory results due to a concentration on global accuracy while ignoring the identification performance for minority samples.
It is a significant task to identify minority samples among majority samples to accurately attain essential information.With the rising prominence of machine learning and artificial intelligence, as well as the continuous emergence of new problems and technologies involving imbalanced data, imbalanced classification methods are widely concerned and developed because of its development prospects [11][12][13][14][15][16][17][18].Improving classification algorithms, introducing cost sensitive strategies, and using balance algorithms are common methods in imbalanced learning.
One of the most widely-used strategies is ensemble learning [19][20][21], which combines classification algorithms with data processing techniques or cost-sensitive solutions.The superiority of ensemble learning in dealing with imbalanced data is that it implements a combined strategy for the classification results based on multiple base classifiers, so that the classifier group can identify a skewed distribution between data categories as much as possible.It has seen great success in improving the identification of minority class samples for imbalanced classification problems.During the training process, the construction way of ensemble classifiers is diverse and flexible.A single classification algorithm can be used to construct homogeneous ensemble classifiers, whereas different classification algorithms can be used to obtain heterogeneous classification systems [22].
The existing body of research on the application of the ensemble model suggests ensemble classification algorithms for imbalanced data have great potential effectiveness in practice [23][24][25][26].For further improving the classification efficiency, multiple effective ways [27,28] have been developed from several perspectives.In more recent studies, one of the most promising strategies is dynamic selection [29], in which the most competent or an ensemble classifier is selected by estimating each classifier's competence level in the classification pool.The benefit of this approach is to identify different unknown samples by choosing different optimum classifiers.Therefore, each base classifier can be regarded as an expert for a special sample in the classification space.Dynamic selection classifiers exhibit a higher accuracy over traditional combined approaches in solving several real-world problems, such as face recognition [30] and text verification [31].
Due to a dynamic selection strategy emerging as an interesting technique for extending ensemble algorithms, we are motivated to provide a comprehensive review of the development of the latest ensemble classification algorithms for imbalanced datasets, and offer a detailed experimental comparison of the performance of the state-of-the-art existing ensemble algorithms incorporating dynamic selection.This study is arranged as follows.We collect several sources of imbalanced datasets and discuss several pertinent evaluation indexes in Section 2. In Section 3, we categorically summarize the latest progress in imbalance ensemble algorithms.In an attempt to improve the classification performance for both binary class and multi-class imbalanced datasets, an experimental scheme is designed by adopting a dynamic selection strategy in Section 4. A series of experimental comparisons are conducted to support the role of imbalance ensemble classification with the dynamic selection strategy in Section 5. Section 6 draws conclusions.

Imbalanced Datasets and Evaluation Measures
In this section, the public access of data sources and comparison criteria used in the related literature are summarized.

Sources of Imbalanced Datasets
Although imbalanced datasets exist widely, most researchers generally select experimental datasets from public access databases to compare the performance of classification models.Table 1 provides a summary of several databases used in the literature.The UCI database covers 376 datasets used for machine learning classification tasks, including binary class and multi-class datasets.OpenML is an online machine learning platform for sharing or organizing data, machine learning algorithms, and experiments, with a total of 3121 datasets.KEEL includes a module named imbalanced data for imbalance learning containing multi-class imbalanced datasets.In the Defect Prediction database, a series of typical imbalanced datasets concerning network fault detection are available.

Evaluation of Imbalanced Classification
Owing to the skewed distribution of imbalanced data, overall accuracy and precision cannot be enough to measure the recognition performance of the model for the minority classes, especially when the imbalance is extremely high.The G-mean [32] and F-measure [33] are typical evaluations of imbalance learning, which can describe categorization situations from different perspectives.During the experimental process, a comprehensive evaluation relying on multiple indicators should be recruited to examine the effectiveness of a classification model.Suppose n is the number of classes, the following performance measures of imbalanced classification are suitable for both binary class and multi-class classification.
(I) MAvA: where Acc i is the accuracy of the i-th class; (II) G-mean: (III) Precision: where TP i denotes the number of correctly classified samples in the i-th class, and FP i is the number of instances misclassified into the i-th class; (IV) F-measure: where Recall i = TP i TP i +FN i , FN i denotes the number of samples in the i-th class which are misclassified into the other class label.

Ensemble Approaches for Imbalanced Classification
The challenges of imbalanced classification and its prevalence have resulted in much research in this area.An effective solution is to design classifiers based on ensemble approaches [34][35][36][37][38][39][40][41][42].Ensemble schemes for imbalanced classification have been developed from the perspectives of data decomposition, cost-sensitive schemes, sampling methods, and dynamic selection.These methods can achieve desirable results for binary imbalanced classification problems.However, multi-class classification involving multiple minority classes has to face the complexity of the internal structure of multi-class imbalanced datasets, and the differences of a decision boundary of two classes bring more difficulties to the classification task and require additional effort.Therefore, multi-class imbalanced classification [43][44][45][46][47][48] has always attracted attention in the machine learning field.This part summarizes the state-of-the-art imbalance ensemble classification algorithms for binary class and multi-class imbalanced datasets.Some of the pertinent methods are shown in Figure 1 and later elucidated in our comparative study.

Imbalanced Learning Based on Ensemble Classification and a Decomposition Strategy
The most notable characteristic of multi-class classification is the diversity of categories compared with binary classification.A common method is to convert a multi-class dataset into several binary datasets via decomposition, for example, OVA [49], OVO [50], AO [51], and OAHO [52].
OVA is a relatively straightforward decomposition strategy and is designed for disassembling data by marking each category as a positive class and all the other classes as a negative part, on which a classifier is trained.If the original dataset has n classes, n binary classifiers can be obtained.OVO trains one binary classifier for each pair of classes, a total of n(n − 1)/2 classifiers are required to be trained.Based on OVA and OVO, Garcia-Pedrajas et al. [51] proposed the AO strategy, which employs OVA to obtain two predicted results with the label (l i , l j ), and other predicted results from OVO classifiers related to l i and l j are chosen to make the final prediction.OAHO sorts classes according to the number of instances in descending order {l 1 , l 2 , • • • , l n }, where l 1 is the class which has the largest number of samples.The training process starts from l 1 until l n and sequentially regards the current class as the positive class and all the lower ranking classes as negative classes.Binary classifiers will be trained on sub-datasets.
Data decomposition is an easily applicable conversion method for multi-class problems and has been combined with ensemble classification in succession.For instance, to address the multi-class imbalance ensemble classification issue, Ghanem et al. [53] combined AO and the PRMS-IM algorithm [54] to design the MultiIMAO classifier.It effectively demonstrated that the data decomposition strategy could enhance classification performance to a certain extent.Besides, Bi et al. [55] incorporated the OVA, OVO, and OAHO methods in PRMs-IM classification, named MultiIMOVA, MultiIMOVO, and MultiIMOAHO, respectively, to further investigate their cooperative performance.Dietterich and Bakiri proposed the ECOC decomposition method [48] to classify multi-class datasets using error correction output coding.On the basis of ECOC, the IMECOC method developed in [56] was an improved ECOC method that simultaneously premeditated the between-class and the within-class imbalance in classifying imbalanced datasets.Further-more, different weights were assigned to different binary classifiers in the IMECOC [56].According to different encoding methods, IMECOC was further extended to ImECOCOVA, ImECOCsparse, ImECOCdense [34], and so on.
The above classification approaches provide a basic starting point for discovering the potential synergy between ensembles for imbalanced data and data decomposition strategies.

Imbalanced Learning Based on Ensemble Classification and Cost-Sensitive Scheme
By considering the cost of classifications, numerous effective cost-sensitive-based ensemble algorithms have been developed [57], which favor the minority class by assigning different weights for different samples.Ensemble classification incorporating cost-sensitive schemes can promote more robust performances than a single classification by merely combining multiple classifiers.The representative algorithm for binary classification is AdaBoost (Adaptive Boosting) [46], proposed by Freund and Schapire, in which weak classifiers were integrated to build a stronger classifier by updating weights.Given a training dataset {(x 1 , y 1 ), (x 2 , y 2 ), • • • , (x N , y N )}, the weight updating rule in AdaBoost was defined as: where the initial weight takes D 1 (i) = 1/N, α t is the weight parameter of the t-th weak classifier, h t is a weak classifier in the t-th iteration, and Z t is a normalization factor.The output of AdaBoost was defined as: To tackle multi-class classification problems, a series of AdaBoost extensions were presented.Both AdaBoost.M1 and SAMME [58] extended AdaBoost in terms of updating weights and the combination strategy of classifiers.AdaC2.M1 [59] inherited the general learning framework of AdaBoost.M1 except that it introduced misclassification costs into the weight update formula.The optimal cost setting in AdaC2.M1 was determined by employing a genetic algorithm.AdaBoost.NC [60] was an algorithm emphasizing ensemble diversity in the training process, in which a weight update rule with a penalty term was introduced.The PIBoost classifier [61] based on the ensemble method and costsensitivity scheme dealt with multi-class imbalanced data via a series of binary weaklearners and a margin-based exponential loss function.In addition, cost-sensitive schemes and data balancing algorithms have synergistic effects for handling imbalanced datasets in ensemble learning.

Imbalanced Learning Based on Ensemble Classification and Sampling Methods
For relieving the impact of the imbalanced training data on the classification model, a large number of ensemble algorithms have been improved by incorporating data-balancing algorithms.The training set is re-sampled before constructing the classification model so that the imbalance rate among various categories is close to an equilibrium.In general, ensemble methods with balancing algorithms mainly combine classifiers with undersampling, over-sampling, or mixed sampling methods.
Under-sampling is a common method for minimizing the proportion of majority class samples in imbalanced data and improving predictive performance.Seiffert et al. [62] proposed the RUSBoost algorithm by combining the under-sampling and boosting method, which randomly removes the instances from majority classes until the desired proportion is achieved.Galar [63] presented a new EUSBoost algorithm based on RUSBoost and the random under-sampling algorithm.The diversity of base classifiers were promoted because more subclassifiers were embedded in the ensemble process.In [64], Luo et al. presented an innovative XGBoost classification method based on bagging to handle classification problems involving imbalanced data.The bagging procedure was designed with random under-sampling.XGBoost synthesizes new samples in a sufficiently small neighborhood of minority samples which averts increasing noisy samples near the classification boundary.
In terms of over-sampling, Ramentol et al. [65] construct a new synthetic minority over-sampling technique, based on the rough set theory and the lower approximation of a minority sample subset.Similarly, Douzas et al. [66] proposed an effective ensemble algorithm based on K-means clustering and SMOTE, stipulating the generation of new data in crucial areas of the space produced by clustering.The imbalance ratio, as well as the average distance among minority samples, was used as the assessment criteria to determine whether new instances should be generated.
When the imbalance rate closes to 1 after the sampling process, the dataset will achieve equilibrium.In addition, the classification accuracy is an alternative for measuring whether data strikes a balance between different classes.This was adopted by Lu et al. for designing HSBagging [67], in which a pre-processing step was conducted by using both random under-sampling and SMOTE at each bagging iteration.The classifier employed the predictive accuracy on out-of-bag instances as an optimal sampling rate for SMOTE and random under-sampling.Among classical imbalanced learning based on ensemble classification and sampling methods, UnderBagging and SMOTEBagging are used to achieve excellent performances [19].However, HSBagging was demonstrated to show a better classification performance compared to single UnderBagging or SMOTEBagging in [67].
Some other related works based on ensemble classification and sampling methods also contribute to resolving imbalanced classification.For example, Ahmed [68] applied hybrid sampling in the RSYNBagging classifier, which considered the diversification of imbalanced data.Additionally, the ADASYNBagging algorithm [69] was coined by incorporating an algorithm and over-sampling.Although most of the above work was aimed at binary class datasets, it provided a solid foundation for the classification of multi-class imbalanced datasets.Wei et al. [70] put forward a SMOTE-decision-tree classifier that modified a binary classification algorithm for handling multi-class imbalance problems effectively.

Imbalanced Learning Based on Ensemble Classification and Dynamic Selection
With the extensive application of ensemble approaches, it has become an important issue for designing a more efficient ensemble classification algorithm.Compared with static ensemble algorithms, dynamic selection ensemble algorithms [71][72][73][74][75][76][77][78][79][80][81][82][83] have been shown to effectively improve the F-measure and G-mean values.A dynamic selection ensemble algorithm predicts the label of the test sample by evaluating the capability level of each classifier and selects the set of the most capable or competitive classifiers.In the process of dynamic ensemble classification (Figure 2), each test sample or each subset can select the optimal classification model.Generally, the selection of classifiers is realized by estimating the classification ability in the local region of the test samples or calculating the prediction accuracy of the classifiers.
A function to evaluate the classification capability can be considered as a tool to assist in the selection of classifiers.For example, Garcia et al. [84]   In addition to the selection manner of classifiers, base classifiers' generalization ability also has an extremely important impact on dynamic selection results.Cruz et al. [88] developed an improved dynamic selection approach.In the first stage, prototype selection techniques were applied to the training data to reduce the overlap between classes.In the next generalization process, a local adaptive K-nearest neighbor algorithm was adopted to minimize the influence of noisy samples on the competency area.Meanwhile, they demonstrated that the distribution of imbalanced datasets would directly affect selecting the optimal classifier during the dynamic selection process, and datasets with a complex structure would result in poor classification.Focusing on the complex structure of imbalanced data, Brun et al. [89] selected the classifier trained on the data subset whose complexity is similar to the neighborhood of the test instances.They also conducted an in-depth consideration and analysis of the data structure in the field of dynamic ensemble learning.Cruz et al. developed a novel dynamic selection framework [90] which extracted meta-features from the training data for training meta-classifiers to judge whether the base classifier had the sufficient ability to classify test instances.
Although each of the strategies mentioned above has its own merits and improves the performance in the design of classifiers, there is still room for improvement in terms of performance optimization by designing dynamic selection in multi-class imbalanced classification.Inspired by the related literature, this study focuses on the investigation of the classification performance of classic multi-classification algorithms combining dynamic selection approaches.

Experimental Comparison of Multi-Class Imbalanced Classifiers by Incorporating Dynamic Selection
Due to the significant advantages of ensemble classification algorithms in dealing with class imbalance, this study merges dynamic selection with popular ensemble classification algorithms for multi-class datasets, aimed towards verifying the effectiveness of dynamic selection.

Experimental Procedure
We employ a homogeneous classifier to generate the candidate classifier pool, and 14 multi-class imbalance ensemble classification algorithms (Figure 1) are employed as base classifiers, respectively.By combining the above-mentioned base classifiers and the dynamic selection process proposed in [84], dynamic ensemble classifiers are designed for both binary class and multi-class imbalanced datasets.The overall process of dynamic selection is shown in Figure 2. The function for capability evaluation referred to in [84] is defined as follows.
Given a test sample x i and a classifier h, we calculate the classification capability of classifier h for x i : where k is the number of nearest neighbor instances of x i in the training data; x it is the t-th neighbor instance of x i ; y t is the true label of instance x it ; w it is the weight of nearest neighbor sample x it , w it = 1 1+exp(λ * m) , λ is the scaling coefficient, and m is the number of samples with the same class as x it .Obviously, the more samples with the same class as x it , the lower the weight, which indirectly increases the weight of a minority sample.Meanwhile, if the predicted labels of the K-nearest neighbor samples are identical to the true label, I returns 1; otherwise, I returns 0. The classification capability of classifier h for x i is reflected by the classification performance of its neighbor samples.Taking AdaBoostNC as the base classifier, the procedure of the dynamic AdaBoostNC model is described in Algorithm 1.The other 13 dynamic ensemble algorithms are constructed similarly.Calculate and standardize the weight [84] for each neighbor sample.

6
for each h j , j = 1, 2, ..., p do 7 Calculate the classification capability of h j to x i , respectively.The datasets for the experiments come from the four repositories mentioned in Table 1, including 56 datasets, 32 of which are suitable for binary classification, and the other 24 datasets are available for multi-class classification.These data are closely related to the fields of life sciences, medical research, bank credit evaluation, network fault detection, etc.More information concerning these datasets is offered in Table 2 and 3.As can be seen from these tables, the experimental datasets are diverse in the number of attributes (4∼40), classes (2∼10), continuous or discrete attributes, class distribution, and imbalance ratio (1.003∼853).In this study, 14 groups of comparative experiments between dynamic ensemble algorithms and state-of-the-art methods are carried out on both binary and multi-class imbalanced datasets.In the experimentation, the nearest neighbor number k is set to 9. The dynamic selection parameter N takes 9 (the settings for both parameters are based on reference [84]), the number of base classifiers p is chosen to be 16.Five-fold cross-validation is carried out for all methods for each dataset.The average values of 5 runs of experimental results are calculated for obtaining the predictive performance of the designed methods.

Experiments Results for Binary Class Datasets
This study first examines the performance of the designed dynamic model on 32 binary class datasets.Aiming to illustrate the performance of the classifier more distinctly, we compare the MAvA, G-mean, precision, and F-measure results with classic algorithms before and after adding the dynamic selection process (Figure 3), where the dashed line at "Number of datasets = 16" means half of the number of binary datasets, and the points represent numbers of datasets on which prediction results are improved after dynamic selection.If the points are above the dashed line, it indicates that the dynamic selection effect on the corresponding index is satisfactory, i.e., the performance of the dynamic selection ensemble classifiers are better than original algorithms and vice versa.We observe from Figure 3 that by incorporating with the dynamic selection strategy, the results of 12 classical imbalanced classification algorithms are improved under the MAvA, precision, and F-measure indicators, which mean higher precision is obtained for more than half of the binary datasets.The effect of dynamic selection is not obvious for MHDDTECOC and HDDTOVA.The points are below the dashed line under MAvA and F-measure indicators, which indicate the higher results only shown on individual datasets.However, other results for 12 classical imbalanced classification algorithms are over the dashed line except the above two classification algorithms, which indicate incorporating dynamic selection can promote predictive performance (MAvA, precision, and F-measure) for binary class datasets.

Experiments Results for Multi-Class Imbalanced Datasets
In this study, the above-mentioned classifiers are tested on 24 multi-class imbalanced datasets, where the dashed line at "Number of datasets = 12" means half of the number of multi-class datasets.The multi-class results are shown in Figure 4, and the main observations are enumerated as follows: (I) The effect of dynamic selection for MCHDDT and HDDTOVA are not satisfactory (the points of 3 indicators are below the dashed line), and the performance of the other 12 dynamic selection ensemble classifiers are better than the original algorithms.The MAvA, precision, and F-measure indicators of the improved classifiers have been distinctly improved after dynamic selection.However, due to the extremely low representation of the minority samples, the recognition rate for the minority category may drop sharply, resulting in a lower G-mean value.In fact, 11 of the classifiers demonstrate favorable G-mean results on more than half of the multi-class datasets after syncretizing with dynamic selection.In this regard, it is obvious that dynamic selection models for multi-class imbalanced datasets showed superior characteristics compared to those for binary data.(II) By combining dynamic selection algorithms, MultiImOVA, MultiImOVO, Multi-ImOAHO, and MultiImAO have exceptional performance, on which predictive performances are effectively improved compared to using a single classification algorithm, on the whole.Therefore, we further validate the conclusion that data decomposition techniques and dynamic selection have a positive synergy during classification [86].In particular, the classification MAvA values of the above four algorithms on all 56 datasets are shown in Figure 5-8.The results indicate that dynamic selection provides potential strategies for dealing with imbalanced datasets covering binary class and multi-class imbalanced datasets.Moreover, for the same dataset, we have counted the number of classification algorithms with improved classification results after dynamic selection.As shown in Tables 4-7, regardless of the structure of the data (both binary class and multi-class imbalanced data), the classification algorithms, for the most part, can better classify imbalanced data after employing dynamic selection.The results reveal that incorporating dynamic selection can relieve the impact of imbalanced training data on the classifier performance.Therefore, a dynamic selection ensemble algorithm can be a potential solution for the imbalanced classification problem.

Patch-Ensemble Classification for Imbalanced Data
Dynamic selection strategies and ensemble classification algorithms have a synergistic effect in classification.Training a dynamic selection ensemble classification scheme with excellent performance is usually an uncertain task, which depends on previous experience and a trial-and-error experiment process.Patch Learning (PL), proposed by Wu et al. [91], is a new machine learning strategy for solving the fitting problem in classification.As shown in Figure 9, patch learning is a combination of parallel and serial models, which focuses on the misclassified samples during the training procedure, and enhances the classification diversity with the construction of multiple local classifiers [91].In this study, a patch-ensemble classification method is designed for classifying imbalanced data, which connects patch learning with a dynamic selection ensemble classification scheme.For a new testing sample, PL firstly determines if the sample belongs to a patch so that the corresponding local patch model is selected for a classification task.Otherwise, the global model is employed.

Patch-Ensemble Classifier
In classification, a classifier tries to distinguish boundaries between binary or multiclass.If the training samples belong to the same class, we can directly detect whether a new testing sample belongs to this class during the predicted phase.One-Class SVM can better solve the above problem [92].Considering that in patch learning, when the testing sample selects the global classifier or several patch classifiers, it is necessary to detect the similarity between the samples and several patches.The ensemble classification method with patch designed in this paper uses the existing imbalanced ensemble classification algorithm as the global classifier and One-Class SVM as the patch classifier for experimental design.The specific process is shown in Algorithm 2.
In the training process, the number of patch classifiers is determined according to the number of classes misclassified by the global classifier.To ensure that the global classifier maintains its best classification effect for multi-class imbalanced datasets, we weaken the boundary between global and patch training data allowing the training samples of the global classifier to partly overlap with that of the patch classifier.During the testing process, the distances between the new sample and various center points of the training sample are calculated, used as the selection condition to dynamically choose a patch classifier or a global classifier for dealing with the testing data.

Experiments and Analysis
To explore the effectiveness of patch ensemble classifiers for imbalanced classification, in this study, the patch ensemble classifier is compared with the classical imbalance ensemble algorithms.We choose the AdaC2M1 algorithm as the global classifier and One-Class SVM as the patch classifier, and the relevant experimental data are detailed in Tables 2 and 3.The performance of the designed patch ensemble classifiers are evaluated by five-fold cross validation on multiple data sets.The results of the designed patch ensemble classifier in this paper are compared with the top five of 14 classical imbalance ensemble algorithms.To show the results more clearly, the classification results of each method are sorted as a whole.The times of dominant classification results of each algorithm in 56 datasets are counted respectively under MAVA, Precision, G-Mean, and F-Measure indexes.As shown in Table 8-11, the proposed patch ensemble classification method achieves better MAvA, Precision, and F-measure values in more than half of the datasets.If only the top two results of the classification are considered, the same conclusion can be drawn.Under the G-mean indicator, the proposed method obtains the optimal and suboptimal classification results on 12 imbalanced datasets, which has a certain degree classification potential compared with other methods.The bold number indicates that the classification result is equal to or better than that of the Patch Ensemble classifier.The bold number indicates that the classification result is equal to or better than that of the Patch Ensemble classifier.The bold number indicates that the classification result is equal to or better than that of the Patch Ensemble classifier.

Conclusions
In this study, we reviewed state-of-the-art ensemble classification algorithms for imbalanced data and compared the performance of 14 existing multi-class imbalanced classification algorithms by incorporating a dynamic selection strategy.By applying these dynamic imbalance ensemble classifiers to 56 public datasets, the experimental results demonstrate that the dynamic ensemble classification methods obtain significantly better MAvA, precision, G-mean, and F-measure performances than the original 14 algorithms.In particular, dynamic multi-class ensemble classifiers have the potential to achieve an ideal identification performance.We also designed a patch ensemble classification method, which uses misclassified samples to train patch classifiers for increasing classification diversity.Experiments showed that this method has a certain classification potential for multi-class imbalanced classification.In future work, we will consider the imbalance ratio to further improve the classification performance and tackle practical problems under its guidance.
constructed a capability function by calculating the classification score of each base classifier and selected out the top nine classifiers with the highest capability values.Specifically, the selection structure for base classifiers was a key component in their dynamic selection model.The classification accuracy can be regarded as another measure for the selection of classifiers.The approach proposed by Lin et al.[85] used a selective ensemble model to deal with multi-class classification by choosing the classifier with the highest classification accuracy in the local region of the test sample.Mendialdua et al.[86] established a more intensive approach to select classifiers for each pair of classes.The model attempted to extract the best classifier in every sub-dataset of OVO.They demonstrated that OVO and dynamic selection have a positive synergy during classification, which enabled the extension of decomposition algorithms to dynamic ensemble selection strategies.Woloszynski et al.[87] used a probability model to evaluate the classification ability of the base classifiers and introduced a random reference classifier in the process of ensemble classification.The probability of the correct classification of the random reference classifier was employed as the measure of the competence of the actual classifier, which combined the dynamic selection of the classifiers with the probabilistic method.

Figure 2 .
Figure 2. The process of dynamic ensemble classification.

Algorithm 1 : 4 Find
Dynamic AdaBoostNC classifier.Input: D = {d i } M i=1 be a collection of training data, X = {x i } T i=1 be a collection of testing data and the number of nearest neighbor is k; N denotes dynamic selection parameter; the number of base classifiers is p.Output: Predicted labels PL. 1 Initialize an empty matrix V . 2 Generate p base classifiers of AdaBoostNC type {h 1 , ..., h p } on training dataset D. 3 for each x i in D, i = 1, 2, ..., T do its k nearest neighbors from D .

8 end 9
Sort the capability values, and select the top N classifiers to predict x i .10Add the current predict results to V .11 end 12 Vote using V , and gain predicted labels PL. 13 return PL.

Figure 3 .
Figure 3.The total number of binary class imbalanced datasets on which prediction results are improved after adding the dynamic selection.

Figure 4 .Figure 5 .
Figure 4.The total number of multi-class imbalanced datasets on which prediction results are improved after adding the dynamic selection.

Figure 9 .
Figure 9. Patch classifier.5.1.Patch Learning Patch learning consists mainly of the following three steps [91]: (I) Train an initial global model using all the training data; (II) Select the incorrectly classified training data to construct several local patch models; (III) The correctly classified training samples are utilized to update the global model.

Algorithm 2 : 5 7
Patch ensemble classifier.Input: The training data, the testing data, the initial global classifier C 0 and the patch classifier is C p .Output: Patch ensemble classifier. 1 Train the initial global classifier C 0 by using training samples D and predict the training labels. 2 Update the global classifier by selecting correctly classified samples in the training set. 3 Utilize the misclassified samples in the training set and calculate the number of classes N. 4 for i = 1 : N do Select the corresponding class in the training set to train the patch classifier i. 6 end Calculate the distances from the test data to the centers of the training data set and to the centers of each class.8 Choose the nearest classifier to make predictions and save classification results.9 return Patch ensemble classifier.

Table 2 .
Description of binary datasets.

Table 3 .
Description of multi-class imbalanced datasets.

The MAvA results of MultiImOVO Before_MultiImOVO After_MultiImOVO Figure 6.
The MAvA results of MultiImOVO for 56 datasets.

The MAvA results of MultiImOVA Before_MultiImOVA After_MultiImOVA Figure 7.
The MAvA results of MultiImOVA for 56 datasets.

The MAvA results of MultiImOAHO Before_MultiImOAHO After_MultiImOAHO Figure 8.
The MAvA results of MultiImOAHO for 56 datasets.

Table 4 .
Numbers of improved classification algorithms after dynamic selection under the MAvA index.

Table 5 .
Numbers of improved classification algorithms after dynamic selection under the G-mean index.

Table 6 .
Numbers of improved classification algorithms after dynamic selection under the Precision index.

Table 7 .
Numbers of improved classification algorithms after dynamic selection under the F-measure index.

Table 8 .
MAvA results.The 14 classical imbalanced classification algorithms are AdaBoostNC (A), AdaC2M1 (B), FuzzyImbECOC (C), MHDDTECOC (D), HDDTOVA (E), ImECOCdense (F), ImECOCOVA (G), ImECOCsparse (H), MCHDDT (I), MultiIMAO (J), MultiIMOAHO (K), MultiIMOVA (L), MultiIMOVO (M), and Piboost (N). 2. "Times of the optimal MAvA" in the table represents the times of best classification result of each algorithm in 56 datasets."Times of the second MAvA" in the table represents the times of suboptimum classification result of each algorithm, and so on.3. The bold number indicates that the classification result is equal to or better than that of the Patch Ensemble classifier.