Customized Instance Random Undersampling to Increase Knowledge Management for Multiclass Imbalanced Data Classification

Abstract

Imbalanced data constitutes a challenge for knowledge management. This problem is even more complex in the presence of hybrid (numeric and categorical data) having missing values and multiple decision classes. Unfortunately, health-related information is often multiclass, hybrid, and imbalanced. This paper introduces a novel undersampling procedure that deals with multiclass hybrid data. We explore its impact on the performance of the recently proposed customized naïve associative classifier (CNAC). The experiments made, and the statistical analysis, show that the proposed method surpasses existing classifiers, with the advantage of being able to deal with multiclass, hybrid, and incomplete data with a low computational cost. In addition, our experiments showed that the CNAC benefits from data sampling; therefore, we recommend using the proposed undersampling procedure to balance data for CNAC.

1. Introduction

Assessing the behavior of clients and employees has been vital for business profitability [1,2,3]. Obtaining insights into the expected behavior of subjects helps to increase the client’s satisfaction [4], predict customers’ purchases [5], enhance human resources responsibilities regarding employees [6,7], and acquire new clients [8]. Several researchers have addressed the problem of increasing business profitability by using individuals’ information [9].

In the healthcare industry, recent research has focused on the patient’s experiences and satisfaction, as well as their behavior, by considering the patients as critical layers in this industry [10]. Patient-centric health care has gained attention in the scientific community, with several investigations on artificial intelligence and its applications [11,12,13] and technological issues [14,15].

A vital issue in knowledge management in healthcare is the ability to deal with the available data. Most of the information from real-world problems is described for numerical and categorical features (hybrid data) and includes incomplete (missing) attribute values. In addition, such information is often organized in several decision classes (for instance, several diseases), with no guarantee of equal representation of the instances in the decision classes. Therefore, most business-related data is multiclass, hybrid, incomplete, and imbalanced. This last characteristic arises when one or more decision classes are overrepresented or underrepresented, which is extremely common in medical data [16,17].

Several supervised classifiers exist in the literature that have been applied to decision-making with data related to people’s behavior and satisfaction. Among them, we are interested in the recently proposed customized naïve associative classifier (CNAC) [4], which was found to be suitable for customers’ preferences assessment.

It is well-known that data imbalance is a challenge for supervised classifiers [18] and that many classifiers benefit from data sampling procedures [19]. In addition, several of the existing sampling procedures in the literature do not deal with multiclass or hybrid and incomplete data. The objective of the study was to assess the impact of undersampling methods for the CNAC in data related to individuals’ behavior and to introduce a novel undersampling algorithm, as well as to determine its performance concerning existing algorithms.

Despite several sampling procedures existing in the literature, most are designed for binary-class problems. In addition, the handling of hybrid and missing data is often missing. Another known drawback is the computational complexity associated with sampling algorithms. This is why there is a need to propose a novel under-sampling procedure. The significance of the paper lies in its assessment of the impact of sampling procedures in the recently proposed CNAC classifier, as well as in the introduction of a novel undersampling algorithm, with tractable computational complexity, able to deal simultaneously with hybrid, multiclass and missing data. The research contributions are the following:

• An experimental study to assess the impact of data sampling for the customized naïve associative classifier;
• A new undersampling algorithm for dealing with multiclass, hybrid and missing data, with tractable computational complexity bounded by $O\left(n+n^2\right)$.

2. Materials and Methods

To deal with class imbalance, researchers have focused on three approaches: data sampling; algorithm adaptation; and classifier ensembles and cost-sensitive learning [20]. Data sampling s devoted to modifying the imbalanced training data to obtain a balanced dataset using undersampling the majority classes, oversampling the minority classes, or both (hybrid sampling) [19]. On the other hand, the algorithm adaptation paradigm focused on transforming existing supervised classifiers, making them robust in the presence of imbalanced data. Lastly, the classifier ensemble approach constructs a committee of classifiers to handle the imbalanced data and minimize the error committed by individual (base) classifiers [20]. As stated before, this paper focuses on the data sampling approach and its impact on the customized naïve associative classifier.

Data sampling for supervised classification has been addressed in the literature since the 1960s [21], and its focus on data balancing has been part of the scientific community since the 1990s [22]. Several studies show that data sampling enhances the performance of supervised classifiers, such as artificial neural networks [23], C4.5 [24], nearest neighbor, support vector machines, and naïve Bayes [25]. Sampling methods are often divided into oversampling, undersampling, and hybrid [18]. We briefly reviewed some of the existing data sampling methods in the following.

2.1. Oversampling Methods

Oversampling methods obtain a balanced data set by creating new synthetic instances from the minority class. Some of the most used oversampling algorithms are briefly described in the following.

ADAptive SYNthetic Sampling (ADASYN) [26] adaptively generates minority class instances. Its main disadvantage is its inability to deal with hybrid, incomplete or multiclass data.

Adjusting the direction of the synthetic minority clasS examples (ADOMS) [27] uses principal components axis (PCA) to generate synthetic minority class instances. Like ADASYN, its main disadvantage is dealing with hybrid, incomplete or multiclass data.

Agglomerative hierarchical clustering [28] (AHC) creates synthetic instances by clustering. Due to its agglomerative approach, the AHC algorithm has high computational complexity. In addition, it does not deal with hybrid, incomplete or multiclass data.

Random over-sampling [29] (ROS) randomly creates new synthetic instances. This method is fast, and is easily extended to multiclass data.

The synthetic minority over-sampling technique [30] (SMOTE) uses the k nearest neighbors to create synthetic minority instances. SMOTE does not deal with categorical, incomplete, or multiclass data. There are several variants of SMOTE [31], such as Borderline-SMOTE [32], Safe level SMOTE [33], SMOTE-RSB* [34], Geometric SMOTE [35] and SMOTE-WENN [36].

Other recent works in oversampling are: studies by Jiang et al. [37] which used the concept of classification contribution degree; and by [38] et al. which introduced a variant of SMOTE called a synthetic minority oversampling technique with natural neighbors (NaNSMOTE); the SCOTE algorithm [39] which transforms multiclass problems into binary ones; and [40] which employed generative adversarial networks for oversampling purposes.

Another interesting approach is to generate virtual samples. Virtual sample generation methods (VSG) aim at predicting the trends in small-size data, and then generate new data from different distributions [41,42,43].

For medical data and other data related to individuals, creating synthetic, unrealistic instances can be considered controversial and unethical. In addition, most oversampling procedures use mathematical tools to create new data, such as means, standard deviations, and others. For mixed and incomplete data, such tools are not applicable. Another drawback of oversampling procedures is the consequent increase in the computational execution time of the decision-making algorithms due to the increased number of instances in the dataset.

2.2. Undersampling Methods

From a different point of view, undersampling methods aim to balance the data by selecting representative instances from the majority classes to reduce the data while preserving the quality of the original data. Some of the most used undersampling algorithms are briefly reviewed in the following:

Condensed nearest neighbor + Tomek’s modification of condensed nearest neighbor [29] (CNNTL) merges the CNN [21] and TL [44] algorithms. Although this combination of strategies is not per se a data balancing method, CNNTL has been used widely for dealing with data imbalance [45,46].

Class purity maximization [47] (CPM) selects two instances (one majority and one minority) as centers. By forming clusters, a classifier committee decides which instances are removed.

Neighborhood cleaning rule [48] (NCL) applies the ENN [49] to delete majority class instances. Although NCL does not deal with multiclass data, it can be extended for this task.

One-sided selection [22] (OSS) uses Tomek’s links to select the instances to delete. Majority class instances in Tomek’s links are deleted. The OSS method does not deal with multiclass data.

Random under-sampling [29] (RUS) randomly selects instances of the majority class. RUS, the same as ROS, can be easily extended to deal with multiclass data.

Simulated annealing-based undersampling [46] (SAUS) uses the simulated annealing metaheuristic to obtain a balanced dataset, combining sensitivity and specificity measures as one optimization function. It deals with incomplete data but not with multiclass data.

Tomek’s modification of condensed nearest neighbor [44] (TL) also uses the information of Tomek’s links to guide the selection. Although this is not a data balancing method, like CNNTL, it has been widely used for data balancing.

Unlike oversampling, undersampling does not create synthetic data, and most undersampling algorithms can deal with mixed and incomplete data. In addition, by obtaining small datasets, undersampling algorithms can diminish the computational execution time of the decision-making algorithms. The main disadvantage of undersampling methods is that often the majority classes end up underrepresented after applying the sampling procedures, and therefore, the data quality is lost.

2.3. Hybrid Sampling Methods

Combining over and undersampling algorithms, hybrid sampling techniques obtained a balanced dataset by deleting instances of the majority classes and creating synthetic instances of the minority classes.

3. Results

Given the limitations of some of the existing sampling algorithms, we aimed to design an undersampling method suitable for multiclass data and able to deal with hybrid (numerical and categorical) incomplete data. Let us have a set of features $A=\left\{A_1,\mathrm{...},A_m\right\}$ than can be numeric or categorical. Each instance $x$ from a set of instances $X$ will be described in terms of the features in $A$ in a way such that $x\left[i\right]$ represents the value of the feature $A_i$ in the instance $x$. If the value is missing, we will consider that $x\left[i\right]=?$.

In addition, we have a decision feature (label) $D$, which can have multiple categorical values in the instances $D\left(x\right)=d_i \in \left\{d_1,\mathrm{...},d_c\right\}$. To know the class value of an instance, we will use $D\left(x\right)$.

The number of instances in a class $d_i$ will be given as $\left|\left\{x\in X: D\left(x\right)=d_i\right\}\right|$. The minority class will be $d_{min}=\underset{j=1\dots c}{\mathrm{argmin}}\left|\left\{x\in X: D\left(x\right)=d_j\right\}\right|$. All other classes will be considered as majority classes.

3.1. Proposed Undersampling Method

For our undersampling proposal, we wanted to deal with multiclass hybrid and incomplete data, and we wanted our proposal to be computationally tractable. Therefore, we chose to hybridize the concepts of structuralizations of the logical combinatorial approach to pattern recognition (LCPR) [50] with the ideas of random undersampling (RUS) [29].

Our proposal is named customized instance random undersampling (CIRUS) and works as follows. It computes the compact sets for each majority class, obtaining a structuralization of the class. Then, it randomly selects one of the compact sets and an instance of the chosen compact set. The selected instance will be added to the resulting set. The process will continue until as many instances as minority class count are selected. Algorithm 1 shows the pseudocode of the proposed CIRUS.

Effortlessly, a compact set is a connected component of a maximum similarity graph; a maximum similarity graph is obtained by forming an arc between each instance and its more similar instances. We computed the similarity between all pairs of instances to obtain such a graph and create the desired arcs.

Algorithm 1: CIRUS
	Inputs: Dissimilarity function diss, Imbalanced set of instances X
1	$B\leftarrow \varnothing$
2	Compute the minority class count as $min\_count=\underset{j=\mathrm{1..}c}{\min }\left|\left\{x\in X: D\left(x\right)=d_j\right\}\right|$.
3	for each subset $X_i\in S$
4	if $\left|X_i\right|=min\_count$, then $B\leftarrow B\cup X_i$ //minority classes are preserved as is
5	else
6	$\mathrm{Compute} \mathrm{the} \mathrm{compact} \mathrm{sets} \mathrm{of} X_i$$, \mathrm{obtaining} \mathrm{a} \mathrm{multi}-\mathrm{list} cs$ with the compact sets. Each item of the multi-list is also a list, containing the instances belonging to the current compact
7	$count=0$
8	repeat
9	Select a compact set randomly, as $idx\_cs=Random\left(0,cs.Count\right)$
10	$\mathrm{Select} \mathrm{an} \mathrm{instance} \mathrm{of} \mathrm{the} \mathrm{compact} \mathrm{set} \mathrm{randomly}, \mathrm{as} idx=Random\left(0,cs\left[idx\_cs\right].Count\right)$
11	$\mathrm{Add} \mathrm{the} \mathrm{selected} \mathrm{instance}, \mathrm{as} B\leftarrow B\cup \left\{cs\left[idx\_cs\right]\left[idx\right]\right\}$
12	$count=count+1$
13	until $count=min\_count$
14	end if
15	end for
16	return $B$

Formally, having a similarity threshold ${\beta }_0$, a subset $N\ne \varnothing$ of $X$ is a compact set if and only if [50]:

• $\forall x\in X\left[y\in N\wedge \left(\begin{array}{c}\underset{\begin{array}{c}y\in X\\ y\ne x\end{array}}{\max }\left\{\mathrm{sim}\left(\mathrm{y},\mathrm{x}\right)\right\}=\mathrm{sim}\left(\mathrm{y},\mathrm{x}\right) \ge {\beta }_0\\ \vee \underset{\begin{array}{c}y\in X\\ y\ne x\end{array}}{\max }\left\{\mathrm{sim}\left(\mathrm{x},\mathrm{y}\right)\right\}=\mathrm{sim}\left(\mathrm{x},\mathrm{y}\right)\ge {\beta }_0 \end{array}\right)\right]x\in N$
• $\forall y,x\in N,\exists y_1,\cdots ,y_q\in N\left[\begin{array}{c}y=y_1\wedge x=x_q\wedge \forall p \left\{1,\cdots ,\mathrm{q}-1\right\}\\ \left[\begin{array}{c}\underset{\begin{array}{c}z\in X\\ z\ne y_p\end{array}}{\max }\left\{\mathrm{sim}\left(y_p,\mathrm{z}\right)\right\}=\mathrm{sim}\left(y_p,y_{p+1}\right)\ge {\beta }_0\\ \vee \underset{\begin{array}{c}z\in X\\ z\ne x_p\end{array}}{\max }\left\{\mathrm{sim}\left(y_{p+1},\mathrm{z}\right)\right\}=\mathrm{sim}\left(y_{p+1},y_p\right)\ge {\beta }_0 \end{array}\right]\end{array}\right]$
• Every isolated instance is a compact set, degenerated.

Compact sets help find the inner structure of classes [51], with the benefit of dealing with hybrid and incomplete data.

For CIRUS, we neglect the ${\beta }_0$ parameter, using ${\beta }_0=0$, to guarantee that each instance is connected to their most similar instances, disregarding the similarity value. CIRUS needs a similarity function to compute the compact sets. Its ability to handle hybrid and incomplete data will depend on the similarity function used. Several dissimilarities have been proposed to handle such data, perhaps the most known is the HEOM dissimilarity [52]. Generally, a dissimilarity function $\mathrm{diss}$ is transformed into a similarity function $\mathrm{sim}$ as $sim=1/diss$. The capability of CIRUS to use user-defined similarity is beneficial because it does not predefine a function to compare the instance and gives the user the ability to select the desired function in a customized way.

The computational complexity of the proposed CIRUS is bounded by $O\left(n+n^2\right)$, where $\mathrm{n}$ is the number of instances.

3.2. Experimental Setup

To test the impact of data sampling procedures in the customized naïve associative classifier (CNAC), we used 20 datasets associated with the business environment. Such datasets relate to the behavior of patients, clients, users, employees, and others. The description of the used datasets is given in Table 1. We use five-fold cross-validation and average results.

To assess the degree of imbalance of a dataset, the Imbalance Ratio (IR) measure is used. A dataset is considered imbalanced if $IR > 1.5$.

$$IR=\frac{\underset{j=\mathrm{1..}c}{\max }\left|\left\{x\in X: D\left(x\right)=d_j\right\}\right|}{\underset{j=\mathrm{1..}c}{\min }\left|\left\{x\in X: D\left(x\right)=d_j\right\}\right|}$$

(1)

We tested the performance of the CNAC classifier before and after data sampling. To assess the performance of the CNAC, we used the non-error rate measure (NER) [53], which is robust for multiclass imbalanced data.

Table 1. Description of the used datasets.

Dataset	Instances	Attributes		Missing Values	Classes	IR	Reference
		Numeric	Categorical
alpha_bank	30,477	1	6	No	2	6.90	[54]
attribute_dataset	500	1	11	Yes	2	1.38	[55]
aug	19,158	2	10	No	2	3.01	[56]
churn_modelling	10,000	6	4	No	2	3.91	[57]
customer_behaviour	400	2	1	No	2	1.80	[58]
customer_segmentation	8068	3	6	Yes	4	1.22	[59]
customer_targeting	6620	66	4	No	3	1.85	[60]
deposit2020	40,000	5	8	No	2	12.81	[61]
df_clean	358	3	16	No	2	9.23	[62]
employee_satisfaction	500	2	9	No	2	1.11	[63]
in-vehicle-coupon	12,684	1	24	Yes	2	1.32	[64]
marketing_campaign	2240	14	13	Yes	2	5.71	[65]
marketing_series	6499	3	16	Yes	2	2.79	[66]
non-verbal-tourist	73	4	18	Yes	6	9.00	[4]
online_shoppers	12,330	10	7	No	2	5.46	[67]
promoted	24,016	4	2	Yes	5	5.44	[68]
telecom_churn	3333	10	0	No	2	5.90	[69]
telecom_churnV2	3333	15	4	No	2	5.90	[70]
telecust	1000	4	7	No	4	1.29	[71]
term_deposit	31,647	7	9	No	2	7.52	[72]

Table 1. Description of the used datasets.

Dataset	Instances	Attributes		Missing Values	Classes	IR	Reference
		Numeric	Categorical
alpha_bank	30,477	1	6	No	2	6.90	[54]
attribute_dataset	500	1	11	Yes	2	1.38	[55]
aug	19,158	2	10	No	2	3.01	[56]
churn_modelling	10,000	6	4	No	2	3.91	[57]
customer_behaviour	400	2	1	No	2	1.80	[58]
customer_segmentation	8068	3	6	Yes	4	1.22	[59]
customer_targeting	6620	66	4	No	3	1.85	[60]
deposit2020	40,000	5	8	No	2	12.81	[61]
df_clean	358	3	16	No	2	9.23	[62]
employee_satisfaction	500	2	9	No	2	1.11	[63]
in-vehicle-coupon	12,684	1	24	Yes	2	1.32	[64]
marketing_campaign	2240	14	13	Yes	2	5.71	[65]
marketing_series	6499	3	16	Yes	2	2.79	[66]
non-verbal-tourist	73	4	18	Yes	6	9.00	[4]
online_shoppers	12,330	10	7	No	2	5.46	[67]
promoted	24,016	4	2	Yes	5	5.44	[68]
telecom_churn	3333	10	0	No	2	5.90	[69]
telecom_churnV2	3333	15	4	No	2	5.90	[70]
telecust	1000	4	7	No	4	1.29	[71]
term_deposit	31,647	7	9	No	2	7.52	[72]

Considering the values in a confusion matrix (Figure 1), the NER is calculated as:

$$NER=\frac{{\sum }_{g=1}^{G}S{n}_g}{G}$$

(2)

where

$$S{n}_g=\frac{c_{gg}}{n_g}$$

(3)

We applied six algorithms of the undersampling approach (CIRUS, CNNTL, NCL, OSS, RUS, and TL). The CIRUS algorithm is available in the EPIC software [73,74], and the remaining sampling algorithms are in the KEEL software [75].

It is essential to mention that several sampling algorithms do not deal with incomplete data; therefore, for those algorithms, we imputed missing values in the datasets using mean and mode (most common missing values), also integrated into the KEEL software. In addition, the KEEL software internally stores all feature values as numeric ones; due to such functionality, all sampling methods tested work, disregarding having hybrid feature data.

In

Table 2. Parameters of the algorithms.

Algorithm	Parameters
CNNTL	Number of neighbors: 5
NCL	Number of neighbors: 5
OSS	Number of neighbors: 5
RUS	None
TL	None
CIRUS	Dissimilarity: HEOM [52], except for the non-verbal-tourist data, in which we used the function suggested in [4]
CNAC	Dissimilarity: HEOM [52], except for the non-verbal-tourist data, in which we used the function suggested in [4]; Attribute weighting: None

, we show the sampling algorithms’ parameters and the ones for the CNAC classifier. For CNNTL, NCL and OSS, we used the default parameter values offered in the KEEL software.

We used nonparametric statistical tests to determine the existence or not of significant differences in the performance of CNAC and to compare the proposed sampling algorithms with respect to the state-of-the-art undersampling procedures. We selected the Friedman test [76], the Holm post hoc test [77], and the Wilcoxon test [78] for these purposes. In the next section, we show the experimental results and discuss them.

4. Discussion

In this section, we analyze two topics: the impact of sampling methods on the performance of CNAC; and the behavior of the proposed CIRUS with respect to the state-of-the-art sampling algorithms. Section 4.1 is devoted to the former topic, while Section 4.2 addresses the latter.

4.1. Impact of Undersampling Methods on the Performance of CNAC

First, we want to determine if the undersampling techniques impact the performance of the CNAC.

Table 3. NER results of CNAC, before and after applying undersampling methods. The best results are in bold, and results of CIRUS with the same or better classifier performance are in italics.

Dataset	CNAC	CNAC after
		NCL	OSS	RUS	TL	CNNTL	CIRUS
alpha_bank	0.84	0.55	0.53	0.54	0.54	0.53	0.53
attribute_dataset	0.48	0.51	0.47	0.50	0.50	0.49	0.49
aug	0.66	0.53	0.61	0.52	0.63	0.59	0.60
churn_modelling	0.56	0.56	0.54	0.56	0.56	0.54	0.56
customer_behaviour	0.72	0.77	0.73	0.77	0.78	0.67	0.72
customer_segmentation	0.42	-	-	-	-	-	0.34
customer_targeting	0.45	-	-	-	-	-	0.46
deposit2020	0.84	0.78	0.77	0.78	0.78	0.78	0.78
df_clean	0.55	0.63	0.60	0.60	0.57	0.58	0.68
employee_satisfaction	0.53	0.51	0.53	0.52	0.55	0.53	0.52
in-vehicle-coupon	0.54	0.53	0.53	0.53	0.53	0.53	0.53
marketing_campaign	0.66	0.63	0.61	0.60	0.64	0.61	0.67
marketing_series	0.72	0.69	0.64	0.68	0.69	0.65	0.71
non-verbal-tourist	0.73	-	-	-	-	-	0.61
online_shoppers	0.64	0.65	0.66	0.61	0.65	0.66	0.65
promoted	0.99	-	-	-	-	-	0.98
telecom_churn	0.65	0.65	0.65	0.65	0.65	0.65	0.65
telecom_churnV2	0.67	-	-	-	-	-	0.67
telecust	0.36	-	-	-	-	-	0.34
term_deposit	0.76	-	-	-	-	-	0.69

shows the non-error rate (NER) results for CNAC after undersampling.

Some undersampling algorithms could not execute in datasets with more than two decision classes. In some cases, this is due to the algorithm’s nature, which considers only two classes. For other algorithms, such as RUS, the KEEL software does not offer an implementation able to handle multiple classes.

Figure 2 presents the maximum, minimum, and average results for CNAC before and after undersampling algorithms.

As shown in Table 3, the CNAC obtained the best results using the unprocessed data for 13 datasets, and its performance was improved or maintained in six datasets (after the CIRUS algorithm). These results support the idea that the proposed undersampling method does not degrade the performance of CNAC. However, three datasets suffer huge performance drops: the datasets alpha_bank, deposit2020, and term_deposit.

We used the Friedman test to determine whether or not the performance differences were significant. We define a significance of $\alpha =0.05$ for a 95% of confidence. The results of the Friedman ranking are shown in

Table 4. Friedman ranking of the results of CNAC before and after undersampling.

Algorithm	Ranking
CNAC	2.150
CIRUS	3.400
TL	3.875
NCL	4.150
RUS	4.750
CNNTL	4.800
OSS	4.875

. Friedman test returns a probability value of $p=0.00037$, and we reject the null hypothesis.

We define the null and alternative hypotheses as:

Hypothesis 0 (H0).

There are no differences in the performance of CNAC before and after undersampling the datasets.

Hypothesis 1 (H1).

There are differences in the performance of CNAC before and after undersampling the datasets.

Due to the tests rejecting the null hypothesis, we applied the Holm post hoc test to determine the significant differences between which algorithms. Such results are given in

Table 5. Holm’s post hoc test, comparing CNAC before and after undersampling algorithms.

i	Algorithm	z	p	$\mathbf{Holm} (\mathit{\alpha }/\mathit{i})$
6	OSS	3.988992	0.000066	0.008333
5	CNNTL	3.879203	0.000105	0.010000
4	RUS	3.80601	0.000141	0.012500
3	NCL	2.9277	0.003415	0.016667
2	TL	2.525141	0.011565	0.025000
1	CSRUS	1.829813	0.067278	0.050000

.

The Holm test finds no significant differences between the performance of CNAC before and after applying the proposed CIRUS algorithm and finds the performance of CNAC before all other undersampling algorithms significantly worse than the performance of CNAC before their application. These results show that the proposed CIRUS does not significantly degrade the performance of CNAC.

However, further research is needed to determine under which specific conditions the datasets can be fully beneficiated from the proposed CIRUS to avoid high drops in performance for the alpha_bank, deposit2020, and term_deposit.

According to the time used to classify instances, the reduction achieved by the proposed algorithm was outstanding (

Table 6. Testing time results (in seconds) of CNAC before and after CIRUS.

Datasets	Before CIRUS	After CIRUS	Gain
alpha_bank	73.40	9.57	7.67
attribute_dataset	0.06	0.02	2.98
aug	148.74	12.72	11.69
churn_modelling	13.45	5.23	2.57
customer_behaviour	0.02	0.00	4.00
customer_segmentation	7.58	4.63	1.64
customer_targeting	82.59	30.02	2.75
deposit2020	575.30	23.70	24.27
df_clean	0.03	0.00	12.10
employee_satisfaction	0.07	0.04	1.79
in-vehicle-coupon	123.69	14.51	8.52
marketing_campaign	3.06	0.45	6.81
marketing_series	7.75	2.77	2.79
non-verbal-tourist	0.00	0.00	0.00
online_shoppers	169.36	9.39	18.03
promoted	40.72	13.56	3.00
telecom_churn	4.07	0.62	6.57
telecom_churnV2	6.03	0.96	6.31
telecust	0.20	0.09	2.20
term_deposit	536.08	28.43	18.85

shows the results according to time), with a maximum gain of 24.3 times faster for the biggest dataset (deposit2020). Figure 3 shows the relation between the time expended by CNAC without undersampling and the time spent after CIRUS.

The results confirm that the proposed CIRUS is helpful for the CNAC classifier because it maintains classifier performance using less time, for instance classifying. We used the Wilcoxon test [78] to determine if the differences in time are significant or not.

We set as null and alternative hypotheses the following:

Hypothesis 0 (H0).

There are no differences in the time expended by CNAC before and after undersampling the datasets.

Hypothesis 1 (H1).

There are differences in the time expended by CNAC before and after undersampling the datasets.

We define a significance of $\alpha =0.05$ for a 95% of confidence. The test returns a probability value of $p=0.000121$, so we reject the null hypothesis. The results allow us to state that CIRUS significantly decreases the time expended by CNAC without scarifying correctness.

4.2. Performance of the Proposed Undersampling Method with Respect to Others

To assess the performance of sampling methods, we use two measures: the NER of the classifier and the obtained IR of the datasets. Sampling aims to return a balanced dataset suitable for classification. Therefore, the best sampling methods will return datasets with IR = 1. Because we already presented the NER of the CNAC classifier after undersampling (Table 3), we will only provide in this section the analysis of the imbalance ratio obtained by the compared undersampling methods (

Table 7. Results of imbalance ratio for the undersampling algorithms.

Dataset	NCL	OSS	RUS	TL	CNNTL	CIRUS
alpha_bank	1.82	4.89	2.21	1.00	6.20	1.00
attribute_dataset	6.60	3.79	4.07	1.00	1.62	1.00
aug	2.40	1.44	1.37	1.00	2.23	1.00
churn_modelling	1.98	2.08	1.13	1.00	3.18	1.00
customer_behaviour	5.90	1.37	4.48	1.00	1.58	1.00
customer_segmentation	-	-	-	-	-	1.00
customer_targeting	1.22	1.22	1.22	1.22	1.22	1.00
deposit2020	1.16	9.93	1.47	1.00	11.95	1.00
df_clean	1.12	6.21	1.98	1.00	8.16	1.00
employee_satisfaction	7.41	5.60	6.39	1.00	2.08	1.00
in-vehicle-coupon	5.34	2.78	3.40	1.00	1.27	1.00
marketing_campaign	1.63	3.82	1.07	1.00	4.98	1.00
marketing_series	2.81	1.54	1.69	1.00	2.10	1.00
non-verbal-tourist	-	-	-	-	-	1.00
online_shoppers	1.58	3.34	1.16	1.00	4.59	1.00
promoted	-	-	-	-	-	1.00
telecom_churn	1.91	4.36	1.12	1.00	5.32	1.00
telecom_churnV2	-	-	-	-	-	1.00
telecust	-	-	-	-	-	1.00
term_deposit	3.46	6.08	4.67	1.00	5.52	1.00

,

Table 8. Friedman ranking of the undersampling algorithms, according to IR.

Algorithm	Ranking
CIRUS	1.325
RUS	2.375
OSS	3.950
NCL	4.250
CNNTL	4.450
TL	4.650

and

Table 9. Results of Holm’s post hoc test, comparing undersampling algorithms according to IR.

i	Algorithm	z	p	$\mathbf{Holm} (\mathit{\alpha }/\mathit{i})$
5	TL	5.620276	0.000000	0.010000
4	CNNTL	5.282214	0.000000	0.012500
3	NCL	4.944152	0.000001	0.016667
2	OSS	4.437060	0.000009	0.025000
1	RUS	1.774824	0.075927	0.050000

) and the statistical comparison of the NER of all undersampling algorithms (

Table 10. Friedman ranking of NER values obtained by CNAC after applying undersampling algorithms.

Algorithm	Ranking
CIRUS	2.425
TL	3.150
NCL	3.375
RUS	3.900
OSS	4.075
CNNTL	4.075

). The hypotheses are:

Hypothesis 0 (H0).

There is no difference in the imbalance ratio obtained by the compared undersampling methods.

Hypothesis 1 (H1).

There are differences in the imbalance ratio obtained by the compared sampling methods.

We set a significance value of 0.05 for 95% of confidence. For the datasets in which the algorithms could not execute, we consider the original IR of the dataset (without sampling) as the corresponding IR value.

Both RUS and the proposed CIRUS obtained a perfectly balanced dataset. The remaining methods fail in several datasets, bringing IRs as high as 11.95 for TL in the deposit2020 dataset. The Friedman test returns a p-value of zero, and the null hypothesis is rejected. The corresponding ranking is shown in Table 8.

The Holm test found no significant differences in IR for CIRUS and RUS; and found CIRUS to obtain significantly more balanced datasets than OSS, NCL, CNNTL, and TL methods.

We used the Friedman test to determine whether or not the differences in performance according to NER values (disregarding the baseline results without instance sampling) were significant. We define the null and alternative hypotheses as:

Hypothesis 0 (H0).

There are no differences in the performance of CNAC after undersampling the datasets.

Hypothesis 1 (H1).

There are differences in the performance of CNAC after undersampling the datasets.

We define a significance of $\alpha =0.05$ for a 95% of confidence. The results of the Friedman ranking are shown in Table 10. Friedman test returns a probability value of $p=0.033632$, and therefore we reject the null hypothesis.

Due to the rejected null hypothesis, we applied the Holm post hoc test to determine the significant differences between which algorithms t. Such results are given in

Table 11. Holm’s post hoc test comparison of undersampling algorithms according to the NER.

i	Algorithm	z	p	$\mathbf{Holm} (\mathit{\alpha }/\mathit{i})$
5	OSS	2.789009	0.005287	0.010000
4	CNNTL	2.789009	0.005287	0.012500
3	RUS	2.493205	0.012660	0.016667
2	NCL	1.605793	0.108319	0.025000
1	TL	1.225474	0.220397	0.050000

.

The experimental results show that the proposed CIRUS significantly exceeds RUS, CNNTL, and OSS according to NER values. Considering both IR and non-error rate measures, the proposal has excellent results, being significantly better in at least one measure (

Table 12. Friedman ranking of NER values obtained by CNAC after applying undersampling algorithms.

CIRUS vs.	IR	NER	Overall
OSS	Better	Better	Better
CNNTL	Better	Better	Better
RUS	Equal	Better	Better
NCL	Better	Equal	Better
TL	Better	Equal	Better

).

When we want to consider several performance measures simultaneously, the Pareto front (also called the Pareto frontier or Pareto curve) can be handy. The Pareto front is the set of all Pareto efficient solutions [79].

Given a certain state, a Pareto improvement is a new state where some measures are improved, and no measure is downgraded. A state is called Pareto-dominated if there exists a possible Pareto improvement. Two states are called Pareto-optimal or Pareto-efficient if one state is better in at least one measure and worst in at least another measure.

As shown in Figure 4, CIRUS dominates all state-of-the-art algorithms while considering IR and NER measures; for this scenario, only CIRUS will be in the Pareto front.

5. Conclusions

We were able to fulfill the objectives of the study, by introducing a novel undersampling algorithm (CIRUS). We also evaluate its performance with respect to state-of-the-art sampling methods and find that CIRUS obtained datasets significantly more balanced than the ones obtained by the compared undersampling methods, with significant differences in performance. It is essential to mention that the proposed CIRUS deals with multiclass hybrid (numeric and categorical) and incomplete (missing) data.

In addition, we assessed the impact of undersampling algorithms on the performance of the Customized Naïve Associative Classifier. From the experiments and the corresponding statistical analysis, we can conclude that CNAC is beneficiated from using the proposed sampling algorithm for data balancing. Accordingly, we do recommend using CIRUS as data preprocessing for CNAC. However, the statistical analysis made finds the performance of CNAC before all other undersampling algorithms were significantly worse than the performance of CNAC before their application. Therefore, we do not recommend using other undersampling techniques but CIRUS for CNAC. We consider that these results are due to our proposal using a customized similarity function and compact sets to guarantee a good representation of the majority classes.

The main advantages of CIRUS are the following: it does not need parameters but the similarity function; therefore, it does not predefine a function to compare the instance and gives the user the ability to select the desired function in a customized way. It handles hybrid and incomplete data as well as multiclass data.

Its main disadvantage is its computational complexity, bounded by $O\left(n+n^2\right)$, where $n$ is the number of instances. Although this complexity is tractable, it may be prohibitive for big-data scenarios.

In future work, we want to explore the impact of the proposed sampling methods for other lazy classifiers, such as the nearest neighbor [80] and the extended gamma [81] classifiers. In addition, we want to work on diminishing the complexity of the algorithm.