Nonlinear Random Forest Classification, a Copula-Based Approach

Abstract

In this work, we use a copula-based approach to select the most important features for a random forest classification. Based on associated copulas between these features, we carry out this feature selection. We then embed the selected features to a random forest algorithm to classify a label-valued outcome. Our algorithm enables us to select the most relevant features when the features are not necessarily connected by a linear function; also, we can stop the classification when we reach the desired level of accuracy. We apply this method on a simulation study as well as a real dataset of COVID-19 and for a diabetes dataset.

1. Introduction

Dimension reduction is a major area of interest within the field of data mining and knowledge discovery, especially in high-dimensional analysis. Recently, the issue of machine learning has received considerable attention; hence, a number of researchers have sought to perform more accurate dimension reductions in this issue [1,2]. While dimension reduction tries to reduce the dimension of data by selecting some functions of the original dataset, feature selection is one of its special cases, which selects the most important features among all of them. There are many areas of statistics and machine learning that benefit from feature selection techniques. From the statistics point of view, Han and Liu et al. (2013) [3] and Basabi (2008) [4] have applied feature selection for multivariate time series. Debashis et al. (2008) [5] have investigated feature selection and regression in high-dimensional problems.

It is known that selecting the most important and relevant features is the main aim in decision tree/random forest algorithms. Although there are many classification approaches proposed in the literature, they rarely deal with the possible existence of nonlinear relations between attributes. On the other hand, note that mutual information-based filter methods have gained popularity due to their ability to capture the non-linear association between dependent and independent variables in a machine learning setting. Mutual information based on a copula function will be a good choice to carry out a feature selection in which the results are stable against noises and outliers [6,7]. So, one of the major aims of this work is using feature selection in a classification context based on a copula function, especially in random forest classification.

Random forests are commonly used machine learning algorithm, which are a combination of various independent decision trees that are trained independently on a random subset of data and use averaging to improve the predictive accuracy and control over-/under-fitting [8,9,10,11]. In this work, in order to extract the most important features in random forest, we use associated copula of features. In this regard, the connection copula between the exploratory variables as well as the associated copula of exploratory attributes and the class labeled attribute are considered. The rest of the paper is organized as follows: we review preliminaries and introduce our method in the next section; we illustrate our algorithm considering simulated data as well as two real datasets in Section 3; finally, Section 4 is devoted to some concluding remarks.

2. Preliminaries and Related Works

The application of feature selection in machine learning and data mining techniques has been extensively considered in the literature. Kabir et al. (2020) [12] used a neural network to carry out a feature selection, while Zheng et al. (2020) [11] used a feature selection approach in a deep neural network. Li et al. (2017) [13] reviewed the feature selection techniques in data mining; see also the book of Lin and Motoda (2012) [14]. For more information, we refer to Hastie et al. (2009) [3], Chao et al. (2019) [15] and Sheikhpour et al. (2017) [16].

Peng et al. (2019) [17] and Yao et al. (2020) [18] have discussed random forest-based feature selection. It is known that the dependence structure between features plays an important role in dimension reduction. Huag et al. (2009) [19] carried out a dimension reduction based on extreme dependence between attributes. Paul et al. (2017) [5] used feature selection for outcome prediction in medical sciences. Zhang and Zhou (2010) [20] investigated multi-label dimensionality reduction features by maximizing the dependence between the original feature description and the associated class labels; see also Zhong et al. (2018) [21]. Shin and Park (2011) [22] analyzed a correlation-based dimension reduction.

In this work, we use dependence structures between variables to find the best feature selection and construct an agglomerative information gain of random forest. We apply our algorithm to classify influenza and COVID-19 patients. Iwendi et al. (2020) [23] carried out a COVID-19 patient health prediction using a random forest algorithm. Li et al. (2020) [13] applied machine learning methods to generate a computational classification model for discriminating between COVID-19 patients and influenza patients only based on clinical variables. See also Wu et al. (2020) [24], Ceylan (2020) [25] and Li et al. (2020) [13] and references therein for more information. Azar et al. (2014) [26] applied a random forest classifier for lymph diseases. See also Subasi et al. (2017) [27] for chronic kidney disease diagnosis using random forest; Açıcı et al. (2017) [28] for a random forest method to detect Parkinson disease; Jabbar et al. (2016) [29] for a prediction of heart disease using random forest. Additionally, a review work of Remeseiro et al. (2019) [30] may be helpful regarding this subject.

Sun et al. (2020) [31] have implemented a mutual information-based feature selection.

Assume that $F_{X_1,X_2,\dots ,X_d}$ is the joint multivariate distribution function of the random vector $\mathit{X}=\left(X_1,X_2,\dots ,X_d\right)$ and $F_{X_i}$, $i=1,2,\dots ,d$, are the related marginal distribution functions. A grounded $d$-increasing uniformly marginal function $C:{\left[0,1\right]}^d\to \left[0,1\right]$ is called a copula of $\mathit{X}$ whenever it couples the multivariate distribution function $F_{X_1,X_2,\dots ,X_d}$ to its marginals $F_{X_i},i=1,2,\dots ,d$, i.e.,

$$F_{X_1,X_2,\dots ,X_d}\left(x_1,x_2,\dots ,x_d\right)=C_X\left(F_{X_1}\left(x_1\right),F_{X_2}\left(x_2\right),\dots ,F_{X_d}\left(x_d\right)\right).$$

(1)

Note that if $\mathit{X}$ is a continuous random vector, then the copula $C_{\mathit{X}}$ is unique. For more details concerning copulas, their families and association measures, we recommend Nelsen (2006) [32] and Durante and Sempi (2016) [33]. Merits of copulas and dependence measures in dimension reduction have been discussed in the literature. See, for instance, Snehalika et al. (2020) [34] and Chang et al. (2016) [35] for copula-based feature selection; Ozdemir et al. (2017) [36] and Salinas-Gutiérrez et al. (2010) [37] for classification algorithms using copulas; Marta et al. (2017) [38] and Lascio et al. (2018) [39] for copula-based clustering approaches; Houari et al. (2106) [40] and Kluppelberg and Kuhn (2009) [41] for copula functions used in dimension reduction.

A well-known measure of uncertainty in a probability distribution is its average Hartley information measure called (Shannon) entropy. For a discrete random variable $X$ with values $x_1,x_2,\dots ,x_n$ and mass density function $p\left(.\right)$, its entropy is defined as:

$$H\left(X\right)=-\underset{i=1}{\sum ^n}p\left(x_i\right)\log p\left(x_i\right),$$

(2)

and for a continuous random variable $X$, its (differential) entropy is given by:

$$H\left(X\right)=-{\int }_{\mathcal{X}}p\left(x\right)\log p\left(x\right)dx,$$

(3)

where $\mathcal{X}$ is the support of $X$. Similarly, for a (continuous) multivariate random vector $\mathit{X}$ of dimension $k$ with the multivariate density $p\left(\mathit{X}\right)$, the entropy is defined as:

$$H\left(\mathit{X}\right)=-{\oint }_{\mathcal{X}}p\left(\mathit{X}\right)\log p\left(\mathit{X}\right)d\mathit{X},$$

(4)

where ∮ is an $k$-integral on $\mathcal{X}$. For two random variables $X$ and $Y$ with joint distribution $p\left(x,y\right)$, the conventional information gain (IG) or mutual information (MI) is defined as:

$$I\left(X,Y\right)={\int }_{\mathcal{X}}{\int }_{\mathcal{Y}}p\left(x,y\right)\log \frac{p\left(x,y\right)}{p\left(x\right)p\left(y\right)}dxdy,$$

(5)

which is used to measure the amount of information shared by $X$ and $Y$ together, with convention $\frac{0}{0}=1$. Moreover, one may generalize this concept to a continuous random vector $\mathit{X}=\left(X_1,X_1,\dots ,X_k\right)$ as:

$$I\left(\mathit{X}\right)={\oint }_{\mathcal{X}}p\left(\mathit{X}\right)\log \frac{p\left(\mathit{X}\right)}{{\prod }_{i=1}^{k}p\left(x_i\right)}d\mathit{X},$$

(6)

Ma and Sun (2011) [42] defined the concept of “copula entropy”. Based on their definition, for a multivariate random vector $\mathit{X}$, which is associated with copula density $c\left(\mathit{u}\right)$, its copula entropy is:

$$h_c\left(\mathit{X}\right)=-{\oint }_{u}c\left(\mathit{u}\right)\log c\left(\mathit{X}\right)d\mathit{u}.$$

Additionally, they have pointed out that the mutual information is a copula entropy. Indeed we have the following lemmas

Lemma 1.

Ref. [32] For a multivariate random vector $\mathit{X}$with the multivariate density $p\left(\mathit{X}\right)$and copula density $c_X\left(\mathit{u}\right)$,

$$I\left(\mathit{X}\right)=-h_c\left(\mathit{X}\right).$$

Finally, the conditional mutual information is useful to express the mutual information of two random vectors conditioned by a third random vector. If we have a $k$-dimensional random vector $\mathit{X}$, $m$-dimensional random vector $\mathit{Y}$ and $n$-dimensional random vector $\mathit{Z}$, such that $\mathit{X}\sim p_X\left(\mathit{x}\right), \mathit{Y}\sim p_Y\left(\mathit{y}\right), \mathit{Z}\sim p_Z\left(\mathit{z}\right)$, $\left(\mathit{X},\mathit{Z}\right)\sim p_{X,Z}\left(\mathit{x},\mathit{z}\right), \left(\mathit{Y},\mathit{Z}\right)\sim p_{Y,Z}\left(\mathit{y},\mathit{z}\right)$ and $\left(\mathit{X},\mathit{Y},\mathit{Z}\right)\sim p_{X,Y,Z}\left(\mathit{x},\mathit{y},\mathit{z}\right)$, then mutual information of $\mathit{X}$,$\mathit{Y}$ given $\mathit{Z}$ which is referred to as “conditional information gain” or “conditional mutual information” of variables $\mathit{X}$ and $\mathit{Y}$ given $\mathit{Z}$ is obtained as:

$$I\left(\mathit{X},\mathit{Y}|\mathit{Z}\right)={\oint }_{\mathcal{Z}}{\oint }_{\mathcal{Y}}{\oint }_{\mathcal{X}}{p}_{X,Y,Z}\left(\mathit{x},\mathit{y},\mathit{z}\right)\log \frac{p_Z\left(\mathit{z}\right)p_{X,Y,Z}\left(\mathit{x},\mathit{y},\mathit{z}\right)}{p_{X,Z}\left(\mathit{x},\mathit{z}\right)p_{\mathit{Y},\mathit{Z}}\left(,y,z\right)}d\mathit{x}d\mathit{y}d\mathit{z}.$$

(7)

3. Copula-Based Random Forest

The connection between mutual information and copula function has been investigated in the literature. We can also represent the conditional mutual information via the copula function through the following proposition:

Proposition 1.

If the random vector  $\left(\mathit{X},\mathit{Y},\mathit{Z}\right)$ is associated with copula  $C_{X,Y,Z}\left(\mathit{u},\mathit{v},\mathit{w}\right)$ , then Equation (7) is

$$I(\mathit{X},\mathit{Y}|\mathit{Z})=h_c\left(\mathit{X},\mathit{Z}\right)+h_c\left(\mathit{Y},\mathit{Z}\right)-h_c\left(\mathit{X},\mathit{Y},\mathit{Z}\right)-h_c\left(\mathit{Z}\right)$$

(8)

Proof of Proposition 1.

By an appropriate equivalent modification of the argument of the log function in the integrand of (7), we readily obtain:

$$I(\mathit{X},\mathit{Y}|\mathit{Z})={\oint }_{\mathcal{Z}}{\oint }_{\mathcal{Y}}{\oint }_{\mathcal{X}}{p}_{X,Y,Z}\left(\mathit{x},\mathit{y},\mathit{z}\right)\log \frac{p_Z\left(\mathit{z}\right)p_{X,Y,Z}\left(\mathit{x},\mathit{y},\mathit{z}\right)}{p_{X,Z}\left(\mathit{x},\mathit{z}\right)p_{Y,Z}\left(,\mathit{y},\mathit{z}\right)}d\mathit{x}d\mathit{y}d\mathit{z}$$

$$=-h\left(\mathit{X},\mathit{Z}\right)-h\left(\mathit{Y},\mathit{Z}\right)+h\left(\mathit{X},\mathit{Y},\mathit{Z}\right)+h\left(\mathit{Z}\right)$$

$$=h_c\left(\mathit{X},\mathit{Z}\right)+h_c\left(\mathit{Y},\mathit{Z}\right)-h_c\left(\mathit{X},\mathit{Y},\mathit{Z}\right)-h_c\left(\mathit{Z}\right).$$

The last equality comes from one of the results of Ma and Sun (2011), which proves that for $\mathit{X}=\left(X_1,X_2,\dots ,X_n\right),$

$$h\left(\mathit{X}\right)=\sum _{i=1}^{n}h\left({\mathit{X}}_i\right)+h_c\left(\mathit{X}\right).$$

In order to use the mutual information in the decision trees, assume we have a dataset $\mathit{D}=\left\{\left({\mathit{x}}_1,y_1\right),\left({\mathit{x}}_2,y_2\right),\dots ,\left({\mathit{x}}_n,y_n\right)\right\}$, where ${\mathit{x}}_i\in \mathcal{X}$ is the $i$-th input or observation and $y_i\in \mathcal{Y}$ is the corresponding outcome variable. In a machine learning approach, the major goal is constructing (or finding) a classification map $f:\mathcal{X}\to \mathcal{Y}$ which takes the features $\mathit{x}\in \mathcal{X}$ of a data point as its input and outputs a predicted label. The special case of the outcome variable is a class label $y_i\in \left\{-1,1\right\}$; i.e., it has two possible values, such as: negative/positive, pathogenic/benign, patient/normal, etc. The general objective function that must be maximized is:

$$J_{CMI}\left({\mathit{X}}_k\right)=I\left({\mathit{X}}_k,Y\right)-\beta \underset{i=1}{\sum ^n}I\left({\mathit{X}}_i,{\mathit{X}}_k\right)-\gamma \underset{i=1}{\sum ^n}I({\mathit{X}}_i,{\mathit{X}}_j|\mathit{Y})$$

(9)

where $I\left({\mathit{X}}_k,Y\right)$ measures the relation between term ${\mathit{X}}_k$ and target variable $Y$, $I\left({\mathit{X}}_i,{\mathit{X}}_k\right)$ quantifies the redundancy between ${\mathit{X}}_i$ and ${\mathit{X}}_k$; while $I({\mathit{X}}_i,{\mathit{X}}_j|Y)$ measures the complementarity between terms ${\mathit{X}}_i$ and $\mathit{X}$.□

Similar to Proposition 1, one may state the Equation (9) based on copula as:

$$J_{CMI}\left({\mathit{X}}_k\right)=h_c\left({\mathit{X}}_k,Y\right)+\beta \sum _{i=1}^n{h}_c\left({\mathit{X}}_i,{\mathit{X}}_k\right)$$

$$-\gamma \sum _{i=1}^n\left[h_c\left({\mathit{X}}_i,Y\right)+h_c\left({\mathit{X}}_k,Y\right)-h_c\left({\mathit{X}}_i,{\mathit{X}}_k,Y\right)-h_c\left(Y\right)\right]$$

$$=\left(1-\gamma \left(n+1\right)\right)h_c\left({\mathit{X}}_k,Y\right)+n\gamma h\left(Y\right)$$

$$+\beta \sum _{i=1}^n{h}_c\left({\mathit{X}}_i,{\mathit{X}}_k\right)-\gamma \sum _{k\ne i=1}^n{h}_c\left({\mathit{X}}_i,Y\right)+\gamma \sum _{i=1}^n{h}_c\left({\mathit{X}}_i,{\mathit{X}}_k,Y\right),$$

where the first term of the last part of equality refers to the relevancy of the new feature ${\mathit{X}}_k$. Peng et al. (2019) [17] introduced the “Minimum Redundancy Maximum Relevance (mRMR)” criterion to set the value of $\beta$ to be the reverse of the number of selected features and $\gamma =0$. We generalize their results by simplifying $J_{CMI}\left({\mathit{X}}_k\right)$. In particular, we have the following criterion, which we have to maximize:

$$J_{MRMR}\left({\mathit{X}}_k\right)=I\left({\mathit{X}}_k,Y\right)-\frac{1}{\left|\mathcal{S}\right|}\underset{i=1}{\sum ^n}I\left({\mathit{X}}_i,{\mathit{X}}_k\right)=-h_c\left({\mathit{X}}_k,Y\right)+\frac{1}{\left|\mathcal{S}\right|}\underset{i=1}{\sum ^n}{h}_c\left({\mathit{X}}_i,{\mathit{X}}_k\right)$$

(10)

From this formula, it can be seen that mRMR tries to select features that have high correlation to the target variable while they are mutually far away from each other, and in this case, the couple of functions plays an important role in the connection between the input and class level variables.

Since decision trees are prone to overfitting and do not globally find an optimal solution, their generalization, random forests, are suggested to overcome these disadvantages. Our algorithm considers the dependence between attributes to provide the best feature selection set and embeds these selected features to a random forest procedure. In this approach, we first use the dependence between attributes to choose the max-dependent as well as the max-relevant features to the class label and eliminate the max-redundant features. From the point of view of these three criteria, our approach is equivalent to the method presented by Peng et al. (2019) [17].

The confusion matrix is a metric that is often used to measure the performance of a classification algorithm. It is also called a contingency table and in a binary classification it is a 2 × 2 table, as shown in Figure 1.

$$Sensitivity=\frac{TP}{TP+FN}.$$

(11)

$$Specificity=\frac{TN}{TN+FP}.$$

(12)

$$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}.$$

(13)

We use our copula-based random forest to find the most relevant features and carry out a classification task. For this, using the copula function which connects input variables with each other as well as with the class variable, we find the most important variables by maximizing of these three criterions and then, based on their priorities, we embed them to a random forest approach to classify the class label feature. We continue our selection to find the most important feature until we reach the desired level of criteria. Traditional criteria can define some values of accuracy/sensitivity/specificity. Inspired by Snehalika et al. (2020) [34], Algorithm 1 presents a pseudo code of this method. Without loss of generality, we consider that the criterion is the accuracy. The algorithm for the sensitivity and specificity is the same.

Algorithm 1. Algorithm of copula-based random forest classification.

Result Data: data set $\mathcal{D}=\left(\mathit{X},Y\right)$, thereshold value $\delta$.

Result: Selected feature set $\mathcal{S}$, Classification results.

1 Initialization: $\mathcal{S}$ = 0, accuracy = 0, $\mathit{F}$ = all features,

2 while accuracy $\le \delta$ do

3 ${\mathit{x}}_R=\underset{{\mathit{X}}_k\in \mathit{F}\backslash \mathcal{S}}{\underbrace{argmax}}\left[-h_c\left({\mathit{X}}_k,Y\right)+\frac{1}{\left|\mathcal{S}\right|}{\sum }_{i=1}^n{h}_c\left({\mathit{X}}_i,{\mathit{X}}_k\right)\right]$;

4 $\mathcal{S}=\mathcal{S}\bigcup {\mathit{x}}_{\mathit{R}}$;

5 $\mathit{F}=\mathit{F}\backslash \mathcal{S};$

6 Perform a random Forrest classification;

7 the accuracy of random forest classification using 13;

8 accuracy = Accnew + accuracy;

9 end

4. Numerical Results

A simulated dataset as well as real data analysis is presented to illustrate our method.

4.1. Simulation Study

In order to carry out a simulation study, we generated data from normal distribution with copula dependence. Our considered copulas were Gaussian, t and Gumbel copulas; see, e.g., Nelsen (2006) [32]. Using the copula library, we first generated n = 10,000 random samples x₁,…, x₁₀ from a 10-variate Gaussian copula where all off-diagonal elements of their correlation matrix equal to ρ = 0.85 and their marginals follow the standard normal distribution. In a similar fashion, again, we generated another 10 variates x₁₁,…, x₂₀ independent from the first 10 variables. Then, for simulating from t-copula, we generated 10-variates x₂₁,…, x₃₀ from t-copula with all correlation values equal to ρ = 0.85, df = 19 [43] and their marginals follow the standard normal distribution. Finally, a bivariate Gumbel copula with θ = 5 [44] and normal marginals were generated and inserted into x₃₁, x₃₂. A schematic heatmap plot of these 32 features is shown in Figure 2. Using a linear combination, we added values of these feathers and made the outcome variable. In order to obtain a class-valued variable, we recoded the negative values of the outcome variable to “0” and other values to “1”.

Using Algorithm 1, we started with n = 2 features. The most important features to classify y were x₂₆ and x₃₂ with sensitivity = 0.869, specificity = 0.867 and accuracy = 0.875. Continuing the selection of the most relevant features has led us to x₂₆, x₃₂ and x₃₁ as the first three relevant features. In order to obtain unbiased results, we performed a 10-fold cross validation, and in each fold, we left out 1000 cases as a test group and the remainder for the train set. Averages of sensitivity, specificity and accuracy were calculated to assess the algorithm.

Table 1. Selected feature with their measures of assessment in simulate data.

n	Relevant Attributes	Sensitivity	Specificity	Accuracy	Running Time
2	x26, x32	0.869	0.867	0.875	4.51
3	x26, x32, x31	0.879	0.869	0.879	4.91
4	x26 x32 x31 x16	0.880	0.873	0.880	5.12
5	x26, x32, x31, x16, x13	0.881	0.878	0.881	5.55
6	x26, x32, x31, x16, x13, x12	0.888	0.880	0.886	5.83
7	x26, x32, x31, x16, x13, x12, x20	0.891	0.888	0.891	6.33
8	x26, x32, x31, x16, x13, x12, x20, x18	0.893	0.892	0.892	6.55
10	x26, x32, x31, x16, x13, x20, x12, x18, x17, x14	0.898	0.895	0.893	6.92
15	x26, x32, x31, x16, x13, x12, x20, x18, x14, x17 x15, x11, x19, x3, x7	0.908	0.896	0.901	8.34
20	x26, x32, x31, x16, x13, x12, x20, x18, x14, x17 x11, x15, x19, x3, x7, x9, x8, x5, x6, x1	0.918	0.909	0.917	11.30
25	x26, x32, x31, x16, x13, x12, x20, x18, x17, x14, x11, x15, x19, x3, x7, x8, x9, x6, x5, x1 x4, x2, x10, x27, x29	0.929	0.939	0.934	13.61
32	All attributes: x1, x2,…,x32	0.982	0.979	0.981	16.75
32	Traditional random forest	0.982	0.979	0.981	16.75

shows the most relevant and least redundant features with their evaluation scales sensitivity, specificity and accuracy. This table helps us to assess our algorithm by monitoring its running time as well as its comparison with other algorithms. Since, after selecting the features, we use the traditional random forest approach, it is reasonable that we compare our results with the results of the traditional random forest approach. Comparing the last two rows of Table 1, we deduce that the results of our algorithm and the traditional random forest algorithm are the same.

Additionally, from the running time point of view, as seen from the last column of the table, for a small number of attributes, the running time (based on seconds) is negligible, and by increasing the number of attributes, the running time increases significantly. From the pros and cons point of view of the proposed approach, as understood from this table, there is a design-of-experiment approach that physicians may encounter. They can regulate the number of desired attributes to carry out a reasonable random forest classification based on the percentage of accuracy, specificity and sensitivity. Evidently, as seen from the last column of Table 1, after selecting attributes using copula, such a classification algorithm will run fast for a small number of attributes; one may think this is an operation research problem. Specifically, the sample size, the number of attributes and the complexity of relationship between attributes play important roles in such a classification procedure. So, from the point of view of the practical implications, these results enable researchers to specify the number of attributes based on the desired levels of sensitivity, specificity and accuracy, and if the relationship between attributes is not complicated, one can choose a greater number of attributes and achieve more accuracy, and vice versa.

4.2. COVID-19 Dataset

Li et al. (2020) [13], in a meta-analysis, merged 151 datasets of COVID-19 including patient symptoms and routine test results. Nineteen clinical variables were included as explanatory inputs. The variables included age, sex, serum levels of neutrophil (continuous and ordinal), serum levels of leukocytes (continuous and ordinal), serum levels of lymphocytes (continuous and ordinal), results of CT scans, results of chest X-rays, reported symptoms (diarrhea, fever, coughing, sore throat, nausea and fatigue), body temperature, and underlying risk factors (renal diseases and diabetes) [13]. By applying machine learning methods, they reanalyzed these data and investigated correlation between explanatory variables and generated a computational classification model for discriminating between COVID-19 patients and influenza patients based on clinical variables alone.

In a COVID-19 patient case, an agglomerative approach test may help diagnosis of illness. We used our copula-based feature selection to identify the most effective attributes to make a discrimination between COVID-19 patients and influenza patients. We started with two attributes. The most relevant attributes were “age” and “fatigue”. We then applied these two attributes to separate the COVID-19 and influenza patients and obtained evaluation values sensitivity, specificity and accuracy, respectively as 0.755, 0.864 and 0.836. Seeking the three most important classification attributes lead us to “age”, “fatigue” and “nausea/vomiting” with sensitivity equaling 0.840, specificity equaling 0.886 and accuracy equaling 0.873.

Table 2. Selected features with their measures of assessment in COVID-19 dataset.

n	Names of Attributes	Sensitivity	Specificity	Accuracy
2	Age, Fatigue	0.755	0.864	0.836
3	Age, Fatigue, Nausea/Vomiting	0.840	0.886	0.873
4	Age, Fatigue, Nausea/Vomiting, Diarrhea	0.826	0.875	0.860
5	Age, Fatigue, Nausea/Vomiting, Diarrhea, Sore Throat	0.783	0.891	0.860
10	Age, Fatigue, Nausea/Vomiting, Diarrhea, Sore Throat, X-ray Results, Shortness of Breath, Neutrophil, Serum Levels of White Blood Cell, Risk Factors	0.735	0.922	0.865
15	Age, Fatigue, Nausea/Vomiting, Diarrhea, Sore Throat, X-ray Results, Shortness of Breath, Neutrophil, Serum Levels of White Blood Cell, Risk Factors, Temperature, Coughing, Lymphocytes, Neutrophil Categorical, Sex	0.873	0.929	0.914

summarizes the 10 most important features with their classification evaluation’s scores. As understood from this table, there is a design-of-experiment approach that a physician may encounter. In fact, the required percentage of information determines the number and types of tests of patients. For example, if there is a required 85% accuracy of classification only, then it is enough to know “age”, “fatigue” and “nausea/vomiting” of patients, while for 91.4% accuracy, we need to test the 15 most important attributes.

4.3. Diabetes 130-US Hospitals Dataset

In this subsection, we assess our approach in a big data analysis. We apply our algorithm to classify the Diabetes 130-US hospitals dataset [45]. This dataset represents 10 years (1999–2008) of clinical care at 130 US hospitals and integrated delivery networks. It is comprised of 101,721 observations of 50 features representing patient and hospital outcomes. The data contains such attributes as “race”, “gender”, “age”, “admission type”, “time in hospital” and another 45 attributes. Detailed descriptions of all the attributes are provided in Strack et al. (2014).

We used the “diabetesMed” variable (0 and 1) as our response/target class variable and applied the other attributes to classify patients into two groups: no medical prescription needed and medical prescription needed. Similar to the previous subsection, the results are summarized in

Table 3. Selected features with their measures of assessment in Diabetes 130-Us hospital dataset.

n	Names of Attributes	Sensitivity	Specificity	Accuracy
2	num_medications, num_procedures	0.742	0.756	0.708
3	num_medications, num_procedures, A1Cresult	0.766	0.771	0.780
5	num_medications, num_procedures, A1Cresult, epaglinide, max_glu_serum	0.837	0.806	0.801
10	num_medications, number_diagnoses, age, A1Cresult, repaglinide, max_glu_serum, weight, glimepiride, rosiglitazone, pioglitazone	0.921	0.948	0.871
20	num_medications, number_diagnoses, age, A1Cresult, repaglinide, max_glu_serum, weight, glimepiride, rosiglitazone, pioglitazone, glyburide, number_emergency, glipizide, number_outpatient, race, metformin, diag_2, readmitted, repaglinide, diag_3	0.981	0.977	0.972
50	All attributes	0.986	0.981	0.978

.

5. Conclusions

A copula-based algorithm has been employed in a random forest classification. In this regard, the most important features were extracted based on their associated copulas. The simulation study as well as real data analysis have shown that the proposed couple-based algorithm may be helpful when the explanatory variables are connected nonlinearly and when we are going to extract the most important features instead of all features.

The idea of this paper may be extended in some manners. One may use this idea in a multi-class random forest classification. Additionally, a random forest regression considering the connecting copula of features will be useful. Moreover, the associated copula of features in order classification tasks such as the support vector machine, discriminant analysis and naive Bayes classification will be of interest. Many extensions of random forest have been investigated by several authors, for example, boosted random forest, deep dynamic random forest, ensemble learning methods random forest, etc. Each extension of the random forest classification may be combined with our approach to obtain better results. We are going to extend these results in a longitudinal dataset in which the outcome variables are connected using some copulas.