Feature Selection Methods for Extreme Learning Machines

Abstract

Extreme learning machines (ELMs) have gained acceptance owing to their high efficiency and outstanding generalization ability. As a key component of data preprocessing, feature selection methods can decrease the noise or irrelevant data for ELMs. However, ELMs still do not have their own practical feature selection method for their special mechanism. In this study, we proposed a feature selection method for the ELM, named FELM. The proposed algorithm achieves highly efficient dimensionality reduction due to the feature ranking strategy. The FELM can simultaneously complete the feature selection and classification processes. In addition, by incorporating a memorization–generalization kernel into the FELM, the nonlinear case of it is issued (called FKELM). The FKELM can achieve high classification accuracy and extensive generalization by applying the property of memorization of training data. According to the experimental results on different artificial and benchmark datasets, the proposed algorithms achieve significantly better classification accuracy and faster training than the other methods.

1. Introduction

Feature selection plays an essential role by removing the irrelevant features and identifying the relevant ones. In 1997, Dash [1] reported on the four parts of a typical feature selection method: generation procedures, evaluation functions, stopping criteria, and verification processes. The feature selection progresses as summarized in the flowchart in Figure 1. First, a subset is extracted from the complete set of features. Then, the feature subset is evaluated by applying an evaluation function. Next, the evaluation result is compared with the stopping criteria. Last, if the evaluation result meets the stopping criteria, the feature selection process stops; otherwise, a new iteration starts by extracting another feature subset. The selected subset of features is generally validated.

Depending on whether they are independent of a learning algorithm, feature selection methods can be divided into four categories: filter [2], hybrid [3], embedded [4], and wrapper methods [5]. Filter methods, such as the fisher [6] and distance [7] scores, are general methods that do not employ any classifier. They can be applied to datasets with a large number of features and require high computing speed. Embedded methods, such as decision tree classification [8], take considerable time interacting with other learning algorithms to complete the feature selection process. Hybrid methods studied in recent years combine the performance of filter and wrapper methods. Wrapper methods are greedy algorithms that depend on classifiers to select or exclude several features based on objective functions [9]. Different classification models can be constructed for different feature subsets, and the merit of these to be considered the best subset is measured by taking the classification performance of selected features as the criterion [10]. Wrapper methods are extensively used in different areas leveraging their high identification accuracy [11].

In 1995, Vapnik proposed an algorithm named support vector machine (SVM), a two-class classifier with maximum interval on the feature space, which can provide the solution to convex quadratic programming problems with good classification performance [12]. A twin support vector machine (TWSVM) comprises two nonparallel hyperplanes four times faster than an SVM [13]. Extreme learning machines (ELMs), first proposed by Huang [14], are learning algorithms for single-hidden-layer feedforward networks (SLFNs) that have attracted significant attention because of their high efficiency and robustness [15]. ELMs randomly generate input bias and weights and obtain outputs by computing the Moore–Penrose generalized inverse matrix H. Nevertheless, according to the results in [16], ELMs have greater generalization ability and higher learning speed than SVMs and TWSVMs.

Various feature selection methods that can accelerate computation and reduce dimensionality have been investigated in many fields. Numerous feature selection methods are based on SVMs and ELMs. These methods have attracted some attention because they can improve the performance of the original classifiers [17]. In 2002, Guyon [18] proposed a wrapper feature selection method based on SVMs, called support vector machine recursive feature elimination, which can reduce the search space and provide a priori knowledge to improve the recognition efficiency and accuracy. Chang [19] achieved good performance by combining linear SVMs with various feature ranking methods. However, these methods are highly complex and can only solve linear problems. Thus, finding an algorithm that is both highly efficient and capable of solving nonlinear problems is crucial. In 2004, Mangasarian presented a new algorithm called the reduced feature support vector machine (RFSVM), which combined a diagonal matrix E with nonlinear SVM classifiers [20]. Similarly, Bai [21] proposed a wrapper feature selection method that added a diagonal matrix E to a TWSVM (FTSVM), effectively identifying the relevant features and improving the performance of the TWSVM. Both used wrapper methods, simultaneously accomplishing the feature selection and classification processes. Man [22] proposed a novel ELM ensemble model that predicts reservoir properties utilizing the nonlinear capability of functional networks to select the optimal input features. In 2014, Adesina formulated an evolutionary wrapper feature selection method using an ELM as the base classifier of a genetic algorithm that explored the space of feature combinations [23]. The feature selection method and the ELM processed the datasets independently. Unlike SVMs, ELMs offer highly accurate and stable classification results at a high learning speed. Therefore, wrapper feature selection methods must be investigated based on ELMs and kernel extreme learning machines (KELMs). Vapnik et al. [24] incorporated a memorization–generalization kernel into an SVM applicable in nonlinear cases. The resulting model exhibited an extensive generalization ability, but it only performed well on datasets with a moderate number of low-dimensional training samples.

This paper makes the following three contributions:

• A wrapper feature selection method is proposed for the ELM, called FELM. In the FELM, the corresponding objective function and hyperplane are introduced by adding a feature selection matrix, a diagonal matrix with element either 1 or 0 to the objective functions of the ELM. The FELM can effectively reduce the dimensionality of the input space and remove redundant features.
• The FELM is extended to the nonlinear case (called FKELM) by incorporating a kernel function that combines generalization and memorization radial basis function (RBF) kernels into the FELM. The FKELM can obtain high classification accuracy and extensive generalization by fully applying the property of memorization of training data.
• A feature ranking strategy is proposed, which can evaluate features based on their contributions to the objective functions of the FELM and FKELM. After obtaining the best matrix E, the resulting methods significantly improve the classification accuracy, generalization performance, and learning speed.

The rest of this study is organized as follows. Section 2 introduces related works, including the ELM and memorization–generalization kernel. In Section 3, the FELM and FKELM are presented. The effectiveness of the proposed algorithms is assessed, and the experimental results, including dimensionality reduction and classification, are dis-cussed in Section 4. Section 5 presents the conclusions of the study and suggests future research directions.

2. Related Works

2.1. ELM

The use of ELMs as SLFNs has gained huge importance in academia and industry [12]. The development of SLFNs has enhanced the generalization performance and speed of ELMs for classification and regression. In ELMs, as long as the hidden layer node of the feedforward nerve is nonlinear phase continuous, the neural network can arbitrarily approach the continuous objective function or classification target without adjusting the hidden layer node. Figure 2 shows the basic structure of ELMs.

In an ELM, for Q random distinct samples $\left(x_i,t_i\right)$, $x_i={\left[x_{i1},x_{i2},\dots ,x_{im}\right]}^T\in R^m$ and $t_i={\left[t_{i1},t_{i2},\dots ,t_{in}\right]}^T\in R^n$, and $x_i$ and $t_i$ represent the i-th input or actual output vectors, respectively. The relationship between input $x_i$ and the expected output $f\left(x_i\right)$is given by:

$$f\left(x_i\right)={\displaystyle \sum _{j=1}^P{\beta }_{j}G\left({\varpi }_j,b_j,x_i\right)=}{\displaystyle \sum _{j=1}^P{\beta }_{j}G\left({\varpi }_j\cdot x_i+b_j\right)}=h\left(x\right)\beta , i=1,2,\dots ,Q$$

(1)

where ${\varpi }_j={\left[{\varpi }_{j1},{\varpi }_{j2},\dots ,{\varpi }_{jn}\right]}^T$ and $b_j={\left[b_{j1},b_{j2},\dots ,b_{jn}\right]}^T$ are the randomly generated learning parameters of the hidden nodes; ${\beta }_j={\left[{\beta }_{j1},{\beta }_{j2},\dots ,{\beta }_{jn}\right]}^T$ is the vector of weights connecting the hidden layer with the output nodes; $h\left(x\right)={\left[h_1\left(x\right),\dots ,h_P\left(x\right)\right]}^T$ is the hidden node output for input $x\in R^m$; $G\left(\cdot \right)$ represents the activation function; and P is the number of hidden nodes. The output function of the ELM can be expressed as:

$$H\beta =y$$

(2)

where $\beta ={\left[{\beta }_1,{\beta }_2,\dots ,{\beta }_P\right]}^T$ is the matrix of output weights and $y={\left[y_1,y_2,\dots ,y_Q\right]}^T$ is the matrix of targets. The hidden layer output matrix is:

$$H=\left[\begin{array}{ccc}G\left({\varpi }_1,b_1,x_1\right)& \dots & G\left({\varpi }_P,b_P,x_1\right)\\ ⋮& \ddots & ⋮\\ G\left({\varpi }_1,b_1,x_Q\right)& \cdots & G\left({\varpi }_P,b_P,x_Q\right)\end{array}\right]$$

(3)

When the training error reaches zero, ${\displaystyle \sum _{i=1}^N‖\beta \cdot h\left(x_i\right)-y_i‖}=0$. The value of the output weights $\beta$ can be determined by calculating the linear system given by Equation (2):

$$\beta {=\mathrm{H}}^{+}y$$

(4)

where ${\mathrm{H}}^{+}$ is the Moore–Penrose generalized inverse matrix [25].

To reduce the possibility of over-fitting the training error should not be equal to zero. When ${\displaystyle \sum _{i=1}^N‖\beta \cdot h\left(x_i\right)-y_i‖}\ne 0$, the objective function of the ELM can be written as:

$$\begin{array}{l}\min \frac{1}{2}{‖\beta ‖}^2+C{\displaystyle \sum _{i=1}^N{\xi }_i}\\ s.t. y_i\beta \cdot h\left(x_i\right)\ge 1-{\xi }_i , {\xi }_i\ge 0, i=1,\dots ,N \end{array}$$

(5)

where ${\xi }_i$ is the training error of the ith input pattern.

2.2. Memorization–Generalization Kernel

Vapnik et al. [24] proposed a memorization–generalization kernel for SVMs and demonstrated that SVMs using such a kernel could be accomplished with no training errors and display superior performance.

Specifically, the kernel contains both a generalization RBF kernel:

$$K_g\left(x,x^{\text{'}}\right)=\exp \left\{-{\sigma }^2{\left(x-x^{\text{'}}\right)}^2\right\}$$

(6)

and a memorization RBF kernel:

$$K_m\left(x,x^{\text{'}}\right)=\exp \left\{-{\sigma }_{\ast }{}^2{\left(x-x^{\text{'}}\right)}^2\right\}$$

(7)

Figure 3a,b show plots of ${\sigma }_{\ast }\gg \sigma >0$, respectively. The memorization–generalization algorithm uses the weighted combination of both kernels, as follows:

$$K_{m\mathrm{g}}\left(x,x\text{'}\right)=\left(1-\tau \right)\exp \left\{-{\sigma }^2{\left(x-x\text{'}\right)}^2\right\}+\tau \exp \left\{-{\sigma }_{\ast }{}^2{\left(x-x\text{'}\right)}^2\right\}$$

(8)

where $\tau$ (0 ≤ $\tau$ ≤ 1) is the weight of the memorization kernel. The combined kernel is shown in Figure 3c. One of these kernels (denoted ${\sigma }_{\ast }$) is responsible for memorization, while the other one (denoted $\sigma$) is responsible for generalization. When $\tau =1$ and ${\sigma }_{\ast }\gg \sigma >0$, the kernel only memorizes classification in small areas around the training data point [24].

In this study, we embedded the combined kernel into a feature selection method based on an ELM, significantly improving its classification performance. The resulting FKELM method is described in Section 3 and the experimental results are discussed in Section 4.

3. Proposed Methods

In this section, we describe two novel feature selection algorithms for solving classification problems, namely FELM and FKELM. In addition, the memorization–generalization kernel is first explored within the FELM framework. The weighted kernel combines the benefits of both the generalization and memorization RBF kernels, thereby significantly decreasing the training error of the classifiers and improving the generalization accuracy of the resulting model. The proposed schemes are obtained by solving quadratic programming problems (QPPs). Furthermore, we use a feature ranking method to assess the contributions of the features to the objective functions of ELM and KELM, respectively, achieving a stable performance.

3.1. Mathematical Model

To select features based on an ELM, we introduce a feature selection binary diagonal matrix E and seek one hyperplane:

$$g\left(x\right)=h\left(x\right)\cdot E\cdot \beta =0$$

(9)

The zeros in the diagonal of E correspond to the suppressed input space features; in contrast, the ones corresponding to the important features effectively utilized in Equation (9), i.e., the feature selection matrix E, defines a subspace spanned by the selected features. The resulting FELM objective function is:

$$\begin{array}{l}\min \frac{1}{2}{‖\beta ‖}_2^2+C{e}^T\xi +\psi e^{T}Ee\\ s.t. h\left(x\right)\cdot E\cdot \beta \ge e-\xi , \xi \ge 0, E=diag\left(0 or 1\right)\end{array}$$

(10)

where e is a vector of ones of appropriate dimension, and $C>0$ and $\psi \ge 0$ are parameters used to control the loss $\xi ={\displaystyle \sum _{i=1}^N‖h\left(x_i\right)\cdot E\cdot \beta -y_i‖}$ and the number of selected features (since $e^{T}Ee=trace\left(E\right)$), respectively. As $\psi$ increases, the penalty on the number of features begins to dominate the objective function.

Before solving Equation (10), we extend it to a nonlinear case, namely FKELM, by seeking a kernel-generated surface corresponding to those in [20] with a feature selection matrix E, as follows:

$$g\left(x\right)=K_{mg}\left(xE,Ex\text{'}\right)\cdot \beta =0$$

(11)

The modified FKELM objective function is:

$$\begin{array}{l}\min \frac{1}{2}{‖K_{mg}\left(xE,Ex\text{'}\right)\beta ‖}_2^2+C{e}^T\xi +\psi e^{T}Ee\\ s.t. K_{mg}\left(xE,Ex\text{'}\right)\cdot \beta \ge e-\xi , \xi \ge 0, E=diag\left(0 or 1\right)\end{array}$$

(12)

where $C>0$ and $\psi \ge 0$ are parameters to control the loss $\xi ={\displaystyle \sum _{i=1}^N‖K_{mg}\left(xE,Ex\text{'}\right)\cdot \beta -y_i‖}$ and number of selected features (since $e^{T}Ee=trace\left(E\right)$), respectively. As $\psi$ increases, the penalty on the number of features begins to dominate the objective.

3.2. Solutions of FELM and FKELM

Using the feature selection matrix E, the relevant features can be easily determined by solving QPPs. A local solution to Equations (9) and (11) can be obtained by first fixing E and finding the corresponding output weights $\beta$, and then fixing the output weights $\beta$ and going through the components of E, successively updating only those whose modification decreases the objective function.

The Lagrangian functions are as follows:

$$L\left(\beta ,\xi ,\psi ,\alpha ,\lambda \right)=\frac{1}{2}{‖\beta ‖}_2^2+C{e}^T\xi +\psi e^{T}Ee-{\alpha }^T\left(h\left(x\right)\cdot E\cdot \beta +\xi -e\right)-{\lambda }^T\xi$$

(13)

$$L\left(\beta ,\xi ,\psi ,\alpha ,\lambda \right)=\frac{1}{2}{‖K_{mg}\left(xE,Ex\text{'}\right)\cdot \beta ‖}_2^2+C{e}^T\xi +\psi e^{T}Ee-{\alpha }^T\left(K_{mg}\left(xE,Ex\text{'}\right)\cdot \beta +\xi -e\right)-{\lambda }^T\xi$$

(14)

where $\alpha ={\left({\alpha }_1,{\alpha }_2,\dots ,{\alpha }_Q\right)}^T$ and $\lambda ={\left({\lambda }_1,{\lambda }_2,\dots ,{\lambda }_Q\right)}^T$ are the vectors of Lagrange multipliers.

The Karush–Kuhn–Tucker (KKT) conditions for $\beta ,\xi ,\alpha$ and $\lambda$ in FELM are given by:

$${\nabla }_{\beta }L=\beta -h\left(x\right)\cdot E\cdot \alpha =0$$

(15)

$${\nabla }_{\xi }L=Ce-\alpha -\lambda =0$$

(16)

$${\alpha }^T\left(h\left(x\right)\cdot E\cdot \beta +\xi -e\right)=0$$

(17)

$${\lambda }^T\xi =0$$

(18)

$$\alpha \ge 0,\lambda \ge 0$$

(19)

According to Equation (16), $0\le \lambda \le C$ is obtained. Therefore, the output weight of FELM is $\beta =h\left(x\right)\cdot E\cdot \alpha$.

Similarly, the KKT conditions for $\beta ,\xi ,\alpha$ and $\lambda$ in FKELM are given by:

$${\nabla }_{\beta }L=K_{mg}{\left(xE,Ex\text{'}\right)}^T\cdot K_{mg}\left(xE,Ex\text{'}\right)\cdot \beta -K_{mg}\left(xE,Ex\text{'}\right)\cdot \alpha =0$$

(20)

$${\nabla }_{\xi }L=Ce-\alpha -\lambda =0$$

(21)

$${\alpha }^T\left(K_{mg}\left(xE,Ex\text{'}\right)\cdot \beta +\xi -e\right)=0$$

(22)

$${\lambda }^T\xi =0$$

(23)

$$\alpha \ge 0,\lambda \ge 0$$

(24)

According to Equation (21), $0\le \lambda \le C$ is obtained. Thus, the output weight of FKELM is $\beta ={\left(K_{mg}{\left(xE,Ex\text{'}\right)}^T\cdot K_{mg}\left(xE,Ex\text{'}\right)\right)}^{-1}\cdot K_{mg}\left(xE,Ex\text{'}\right)\cdot \alpha$.

Before solving the above equations, the feature selection matrix E should be initialized. The strategy of actively initializing E, instead of just assigning random values as in RFSVM and FTSVM, makes the algorithm significantly more stable. Thus, the value of each feature is computed after solving the FELM and FKELM, as follows:

$$Valu{e}_{FELM}\left(i\right)= \frac{1}{2} {‖\beta ‖}_2^2+C{e}^T{\left(e-h\left(x\right)\cdot \beta \right)}_{+}$$

(25)

$$Valu{e}_{FKELM}\left(i\right)= \frac{1}{2} {‖K_{mg}\left(x\left(i\right),x\text{'}\left(i\right)\right)\beta ‖}_2^2+C{e}^T{\left(e-K_{mg}\left(x\left(i\right),x\text{'}\left(i\right)\right)\cdot \beta \right)}_{+}$$

(26)

where ${\left(\cdot \right)}_{+}$ replaces the negative elements with zeros, and $x\left(i\right)$ refers to the ith column of x. Note that for the selected features, $\psi e^{T}Ee$ in Equation (10) and (12) is a constant, and this part is ignored in Equations (25) and (26). The scores of the ith feature that show its importance are computed as follows:

$$Scor{e}_{FELM}\left(i\right)=\frac{Valu{e}_{FELM}\left(i\right)}{{\displaystyle \sum Valu{e}_{FELM}\left(i\right)}}$$

(27)

$$Scor{e}_{FKELM}\left(i\right)=\frac{Valu{e}_{FKELM}\left(i\right)}{{\displaystyle \sum Valu{e}_{FKELM}\left(i\right)}}$$

(28)

Consequently, the algorithm progresses as follows. First, an initial E matrix is generated based on the resulting scores, by assigning a value of $E_{ii}=0$ if $Score\left(i\right)<\frac{1}{n}$, otherwise $E_{ii}=1$. Next, the matrix E is updated by switching the value of each element in the diagonal (from 1 to 0, or from 0 to 1) if the objective function (Equations (10) or (12)) decreases more than the tolerance. Otherwise, the first n elements in the diagonal are changed, until no diagonal element in E can be changed to make the objective function decrease more than the tolerance. Lastly, after updating E, $\beta$ is recomputed. The algorithm terminates if the objective function decreases less than the tolerance. By repeating this process, input space features can be suppressed. We summarize the proposed FELM and FKELM approaches below.

3.3. Computational Complexity Analysis

The time complexity of the two proposed approaches when solving Equations (10) and (12), including the computation of $f\left(E\right)$ and the repeated execution of step 6, are presented in Algorithms 1 and 2. According to [24], the newly developed methods incorporating a memorization–generalization kernel into the FELM have the same computational complexity as the FELM. Thus, we only discuss the computational complexity of the latter. It is clear that $f\left(E\right)$ is computed no more than k times and $\beta$ is easy to obtain by calculating the generalized inverse. For matrix $H\in R^{Q\times P}$, where P is the number of hidden nodes, n is the number of outputs, and Q is the number of datasets, the computational complexity for obtaining the generalized inverse is $O\left(P^3+P^{2}Q+PQn\right)$[25]. Compared with RFSVM and FTSVM, FELM and FKELM do not need compute bias b, dramatically reducing the computational time.

Algorithm 1 Feature selection method for ELM

Input: Samples $A\in R^{m\times n}$;       Appropriate parameters C,$\psi$;       A fixed and large integer k, which is the number of sweeps through E;       Stopping tolerance tol = 1e-6; Output: Output weight $\beta$ ;     $E=diag\left(1 or 0\right)$   1. Set $E=I$, solve the Equation (10) with the fixed E;   2. Compute each feature score by Equation (27), for $i=1,\dots ,n$, if $Scor{e}_{FELM}\left(i\right)<\frac{1}{n}$, then   set $E_{ii}=0$;   3. Solve Equation (10) with the fixed E, and obtain $\beta$;   4. Repeat   5. For $l=1,2,\dots ,kn$ and $j=1+\left(l-1\right) \mathrm{mod} n$:     (1) Replace $E_{jj}$ from 1 to 0 (or 0 to 1).     (2) Compute $f\left(E\right)=\frac{1}{2}{‖\beta ‖}_2^2+C{e}^T{\left(e-h\left(x\right)\cdot \beta \right)}_{+}+\psi e^{T}Ee$ before and after changing    $E_{jj}$.     (3) Keep the new $E_{jj}$ only if $f\left(E\right)$ decreases more than the tolerance. Else undo the    change of $E_{jj}$, and go to (1) if $j<n$.     (4) Go to step 6 if the total decrease in $f\left(E\right)$ is less than or equal to tol in the last n    steps.   6. Solve the Equation (10) with the new E, and compute $\beta$.   7. Until the objective function decrease of Equation (10) is less than tol if $\beta$ is changed.

Algorithm 2 Feature selection method for KELM

Input: Samples $A\in R^{m\times n}$;       Appropriate parameters C,$\psi$;       Appropriate parameters in kernel $\tau ,\sigma ,{\sigma }_{\ast }$;       A fixed and large integer k, which is the number of sweeps through E;       Stopping tolerance tol=1e-6; Output: Output weight $\beta$ ;       $E=diag\left(1 or 0\right)$.   1. Set $E=I$, solve the Equation (12) with the fixed E;   2. Compute each feature score by Equation (28), for $i=1,\dots ,n$, if $Scor{e}_{FKELM}\left(i\right)<\frac{1}{n}$, then   set $E_{ii}=0$;   3. Solve Equation (12) with the fixed E, and obtain $\beta$;   4. Repeat   5. For $l=1,2,\dots ,kn$ and $j=1+\left(l-1\right) \mathrm{mod} n$:     (1) Replace $E_{jj}$ from 1 to 0 (or 0 to 1).     (2) Compute $f\left(E\right)=\frac{1}{2}{‖K_{mg}\left(xE,Ex\text{'}\right)\beta ‖}_2^2+C{e}^T{\left(e-K_{mg}\left(xE,Ex\text{'}\right)\cdot \beta \right)}_{+}+\psi e^{T}Ee$ be-    fore and after changing $E_{jj}$.     (3) Keep the new $E_{jj}$ only if $f\left(E\right)$ decreases more than the tolerance. Else undo the    change of $E_{jj}$, and go to (1) if $j<n$.     (4) Go to step 6 if the total decrease in $f\left(E\right)$ is less than or equal to tol in the last n    steps.   6. Solve the Equation (12) with the new E, and compute $\beta$.   7. Until the Equation (12) is less than tol if $\beta$ is changed.

4. Experimental Results

We conducted experiments to evaluate the efficiency of the proposed algorithms in dimensionality reduction and classification. Section 4.1 describes classification performance, while Section 4.2 shows qualitative assessment of the different algorithms. For comparison, we also evaluated related algorithms, including the RFSVM and FTSVM, on two artificial datasets and eight benchmark datasets obtained from the UCI machine learning repository [26] and gene datasets [27,28]. All experiments were performed using MATLAB R2016a on a desktop computer with an Intel Core i7–1160G7 CPU at 2.11–GHz, 16 GB of memory, and Windows 10.

4.1. Classification Performance

This section discusses several experiments conducted to verify the classification capacity of the proposed FELM and FKELM methods. We compared their performance with the RFSVM and FTSVM using two artificial and eight real-world datasets collected from the UCI machine learning repository. In the experiments, the FELM and FKELM used the sigmoid function $F(a,b,x)=\frac{1}{1+\exp \left(-a_i^{T}x+b_i\right)}$ as the activation function. The grid search with a cross-validation technique was used to select the optimum parameters [29] C for each dataset, which were, in turn, tuned from $\left\{2^{-10},\dots ,2^0,\cdots ,2^{10}\right\}$. For each dataset, we evaluated the accuracy and time at $\psi =\left\{0,2^0,\dots ,2^4\right\}$. Specifically, the memorization–generalization kernel used in FEKLM is given as $K_{m\mathrm{g}}\left(x,x\text{'}\right)=\left(1-\tau \right)\exp \left\{-{\sigma }^2{\left(x-x\text{'}\right)}^2\right\}+\tau \exp \left\{-{\sigma }_{\ast }{}^2{\left(x-x\text{'}\right)}^2\right\}$ with kernel parameters $\tau , \sigma , {\sigma }_{\ast }$. According to ${\sigma }_{\ast }\gg \sigma >0$, in this study we set ${\sigma }_{\ast }=2^{-10}\sigma$. The parameters $\tau ,\sigma$ are selected from $\left\{0,\dots ,0.5,\dots ,1\right\}$ and $\left\{2^{-10},\dots ,2^0,\cdots ,2^{10}\right\}$, respectively. For the RFSVM and FTSVM, the popular Gaussian kernel $K\left(x,x_i\right)=\exp \left\{-\frac{{‖x-x_i‖}^2}{2{\sigma }^2}\right\}$ with parameter $\sigma$ selected from $\left\{2^{-5},\dots ,2^0,\cdots ,2^5\right\}$ is used for the non-linear cases. We select the parameters $C_1=c_{11}=c_{21}\in \left\{2^{-8},\dots ,2^7\right\}$ and $C_2=c_{12}=c_{22}\in \left\{2^{-8},\dots ,2^7\right\}$ for the FTSVM [21]. Taking the “two-moon dataset” as an example, we analysed the sensitivity of the FELM and FKELM to the number of hidden layer nodes P. As shown in Figure 4 and Figure 5, as the number of hidden layer nodes increases, the training accuracies of the FKELM and FELM do not exhibit a significant variation. Thus, the optimum number of hidden layer neurons P is set as 300 for all cases. All the datasets are binary class datasets. Therefore, to test the performance of the methods listed above, we defined the accuracy as follows, $accuracy=\left(TP+TN\right)/\left(TP+FP+TN+FN\right)\times 100\%$, with TP, TN, FP, and FN representing the number of true positive, true negative, false positive, and false negative samples, respectively. The training and test accuracies were taken as the average of 10 experiments.

4.1.1. Performance on the Artificial Datasets

Two different artificial datasets were created to graphically depict the classifiers and to verify their accuracy and feature selection ability. All the positive and negative samples on the “two-moon dataset” were completely divided. On the contrary, several positive and negative samples were mixed up in the self-defining function dataset. The details of both datasets used are listed in

Table 1. Functions used for generating the dataset.

Dataset	Function Definition
Self-defining function	$\mathrm{covariance}\left(\begin{array}{l}1 \\ 0 \end{array}\right.\left.\begin{array}{l}0\\ 1\end{array}\right)$ mean value (0,0) and (0.8,0.8)
Two-moon	$r\sim U\left(r-\frac{w}{2},r+\frac{w}{2}\right)$ ${\theta }_1\sim U(0,\pi ){\theta }_2\sim U(-\pi ,0)$

and

Table 2. Details of the artificial dataset.

Dataset	Number of Test Samples	Number of Training Samples	Range of Independent Variables
Self-defining function	50	350	$x\in \left(0,2\right)$
Two-moon	100	501	$x\in \left(0,4.5\right)$

. In particular, in the self-defining function dataset, the positive and negative samples had the same covariance but a different centre.

We divided the two-dimensional datasets into categories 1 or -1. Hence, we used the zero-valued contour line to simulate the classification hyperplanes, as plotted in Figure 6 and Figure 7 for the RFSVM and FTSVM as well as the proposed approaches, on the two datasets. Furthermore, the classification accuracies and time of all the methods are presented in

Table 3. Experimental results from the artificial dataset.

Dataset	Algorithm	Training Accuracy	Test Accuracy	Time (s)	Parameters $(\mathit{C},{\mathit{C}}_1,{\mathit{C}}_2,\mathit{\psi },\mathit{\sigma },\mathit{\tau })$
Two-moon	RFSVM	0.9948	0.9600	0.8632	2–8, –, –, 22, 2–3, –
	FTSVM	1.0000	0.9819	0.6929	–, 2–3, 22, 21, 2–3, –
	FELM	1.0000	1.0000	0.3208	2–6, –, –, 22, 2–1, –
	FKELM	1.0000	1.0000	0.3524	2–6, –, –, 22, 2–1, 0.3
Self-defining function	RFSVM	0.8977	0.9012	0.5007	2–7, –, –, 20, 2–3, –
	FTSVM	0.9293	0.9110	0.2410	–, 20, 23, 21, 2–4, –
	FELM	0.9922	0.9879	0.1375	2–5, –, –, 22, 2–2, –
	FKELM	0.9972	0.9913	0.1398	2–5, –, –, 22, 2–2, 0.4

.

The results show that the FKELM and FELM outperform the other algorithms, achieving superior classification performance at a high learning speed. Additionally, it can be seen in Table 3 that the FELM and FKELM have significantly faster and better classification effects than the RFSVM and FTSVM. Because samples are definitely divided, almost all the classifiers obtain high accuracy on the “two-moon dataset”. It is also clear in Figure 7 that the classification hyperplanes obtained from the RFSVM and FTSVM misclassify some positive and negative samples, whereas the proposed FELM and FKELM can correctly divide them into different datasets.

4.1.2. Performance on the Benchmark Datasets

In this section, we compare the classification and feature selection performance of the four considered algorithms when analysing the six UCI and two gene benchmark datasets detailed in

Table 4. Details of the benchmark datasets analysed.

Dataset	Number of Test Samples	Number of Training Samples	Number of Features
Australian	207	483	14
Heart	81	189	13
Ionosphere	105	246	33
WDBC	171	398	30
WPBC	59	139	34
Sonar	62	146	60
Colon	18	44	2000
Leukaemia	21	51	7129

. The datasets include cases with low, medium, and high dimensions, and are divided into two parts: 70% is used as the training set, and the remaining 30% is used as the testing set.

To effectively evaluate the classification ability of the four algorithms under consideration, we calculated their classification accuracy on the training and test datasets. The feature scores were computed for each algorithm and dataset, and the features were suppressed one by one according to their scores.

Table 5. Experimental results on the benchmark datasets.

Dataset	Algorithm	Training accuracy	Test accuracy	Time (s)	Parameter (C,C1,C2,Ψ,σ,τ)
Australian	RFSVM	0.8617	0.8310	3.4092	24, –, –, 22, 2–7, –
	FTSVM	0.8638	0.8573	2.2620	–, 23, 25, 22, 2–8, –
	FELM	0.8732	0.8662	0.5612	2–5, –, –, 21, 2–1, –
	FKELM	0.8707	0.8595	0.6425	2–5, –, –, 21, 2–1, 0.6
Heart	RFSVM	0.8496	0.8485	3.8659	21, –, –, 21, 2–7, –
	FTSVM	0.8370	0.8321	2.2051	–, 22, 22, 21, 2–9, –
	FELM	0.8759	0.8679	0.1528	2–2, –, –, 20, 2–2, –
	FKELM	0.8796	0.8785	0.1607	2–2, –, –, 20, 2–2, 0.4
Ionosphere	RFSVM	0.8957	0.8912	5.9061	2–1, –, –, 20, 2–3, –
	FTSVM	0.9007	0.8976	4.5770	–, 20, 23, 22, 2–2, –
	FELM	0.8729	0.8728	0.6012	2–5, –, –, 22, 2–2, –
	FKELM	0.9157	0.9072	0.6314	2–5, –, –, 22, 2–2, 0.4
WDBC	RFSVM	0.9597	0.9537	0.3372	2–2, –, –, 21, 2–3, –
	FTSVM	0.9573	0.9586	0.1798	–, 20, 20, 21, 2–4, –
	FELM	0.9592	0.9518	0.0146	23, –, –, 20, 2–5, –
	FKELM	0.9823	0.9762	0.0177	25, –, –, 20, 2–5, 0.5
WPBC	RFSVM	0.7992	0.8034	2.5007	27, –, –, 21, 2–5, –
	FTSVM	0.8141	0.8202	1.5451	–, 25, 26, 21, 2–5, –
	FELM	0.8422	0.8397	0.1279	23, –, –, 21, 2–8, –
	FKELM	0.8409	0.8513	0.1398	23, –, –, 21, 2–8, 0.3
Sonar	RFSVM	0.7972	0.7911	2.9832	22, –, –, 20, 2–3, –
	FTSVM	0.8933	0.8849	1.7132	–, 21, 25, 21, 2–2, –
	FELM	0.8995	0.8879	0.1375	23, –, –, 22, 2–4, –
	FKELM	0.9172	0.9013	0.1452	24, –, –, 22, 2–4, 0.5
Colon	RFSVM	0.8583	0.8512	0.5160	22, –, –, 20, 21, –
	FTSVM	0.7208	0.7010	0.3197	–, 20, 23, 21, 20, –
	FELM	0.7882	0.7871	0.0133	2–1, –, –, 22, 22, –
	FKELM	0.7971	0.7922	0.0139	2–1, –, –, 22, 20, 0.6
Leukaemia	RFSVM	0.8235	0.8017	0.9209	27, –, –, 23, 2–5, –
	FTSVM	0.6293	0.6110	0.5244	–, 25, 26, 23, 2–5, –
	FELM	0.6471	0.6572	0.0399	24, –, –, 24, 2–6, –
	FKELM	0.6882	0.6938	0.0451	24, –, –, 24, 2–5, 0.6

presents the results obtained for both the proposed and control models. Figure 8 and Figure 9 compare the classification accuracy of the different algorithms, while Figure 10 compares their computational cost. According to these results, it is obvious that the FELM and FKELM show higher classification accuracy than the RFSVM and FTSVM in most scenarios. Particularly, the FKELM outperforms the other three algorithms, owing to its kernel’s excellent generalization ability and the ELM’s high learning speed. For the high-dimensional datasets, namely the Colon and Leukaemia cases, the classification accuracy can vary significantly, with the RFSVM performing slightly better than the other three methods. Hence, the proposed FELM and FKELM approaches can be considered suitable for processing datasets with low and medium dimensions and obtaining high classification accuracy at significantly high speed on all types of datasets. Furthermore, as they need not estimate the bias, the computational time required by the FELM and FKELM dramatically reduces, allowing them to find the optimal classification much faster than the RFSVM and FTSVM. Consequently, the proposed wrapper feature selection methods achieve superior classification performance with extremely high efficiency.

To analyse the classification accuracy more clearly,

Table 6. Average accuracy ranks.

Dataset		RFSVM	FTSVM	FELM	FKELM
Australian	Training	4	3	1	2
	Test	4	3	1	2
Heart	Training	3	4	2	1
	Test	3	4	2	1
Ionosphere	Training	3	2	4	1
	Test	3	2	4	1
WDBC	Training	2	4	3	1
	Test	3	2	4	1
WPBC	Training Test	4 4	3 3	1 1	2 2
Sonar	Training	4	3	2	1
	Test	4	3	2	1
Colon	Training	1	4	3	2
	Test	1	4	3	2
Leukaemia	Training	1	4	3	2
	Test	1	4	3	2
Average rank		2.8125	3.25	2.5	1.4375

presents the average ranks based on the results listed in Table 5. As can be observed, the FKELM is ranked first, followed by the FELM, RFSVM, and FTSVM. The experimental results also show the expected achievement of each algorithm. Thanks to the memorization–generalization kernel and feature ranking strategy, the FKELM attains the highest classification accuracy in most cases. Similarly, due to the superior generalization ability of the ELM and its high efficiency in feature selection, the FELM achieves the second-best classification results. The RFSVM only behaves slightly better than the other algorithms on high-dimensional datasets.

To further contrast the performance of the different algorithms, we used the Friedman statistical method to assess their classification performance fairly. Let N be the number of samples, m denote the counts of the algorithms, and $R_i$ represent the average ranks, as presented in Table 6. The Friedman statistic [30] follows the distribution of ${\chi }_{{}_F}^2$ with $m-1$ degrees of freedom, which can be defined as:

$${\chi }_{{}_F}^2=\frac{12N}{m\left(m+1\right)}\left[{\displaystyle \sum _i{R}_i^2-\frac{m{\left(m+1\right)}^2}{4}}\right]$$

(29)

Based on Equation (29), Iman et al. [31] proposed an improved statistic that follows the F-distribution with $m-1$ and $\left(m-1\right)\left(N-1\right)$ degrees of freedom:

$$F_F=\frac{\left(N-1\right){\chi }_F^2}{N\left(m-1\right)-{\chi }_F^2}$$

(30)

When $N=8$ and $m=4$, ${\chi }_{{}_F}^2=8.5875$ and $F_F\approx 3.900$. By referring to the F-distribution critical value table, $F_{0.05}\left(3,21\right)$ equals 3.739. Hence, it is clear that $F_F=3.900>F_{0.05}\left(3,21\right)=3.739$, indicating that the null hypothesis should be rejected and that the performance of the algorithms is significantly different.

The four algorithms were further compared using the Nemenyi test [32], which is defined as:

$$CD=q_{\alpha }\sqrt{\frac{m\left(m+1\right)}{6N}}$$

(31)

where $q_{\alpha }$ is the critical value of the Tukey distribution. When $\alpha =0.05$ and $m=4$, $q_{\alpha }=2.569$ according to the Nemenyi inspection table. The null hypothesis that two algorithms have the same performance is rejected if the corresponding average ranks differ by at least a critical difference of $CD\approx 1.6442$. Because the average rank difference between the FKELM and FTSVM, which is 3.25−1.4375 = 1.8125, is more significant than the critical difference, the performance of the FKELM can be accepted as substantially better than that of the FTSVM. Furthermore, because 2.5−1.4375 = 1.0625 < 1.6442 and 2.8125 − 1.4375 = 1.375 < 1.6442, the Nemenyi test does not indicate any significant difference among the FKELM, FELM and RFSVM. The above-mentioned comparison is depicted in Figure 11 using the Friedman test chart, where the dots represent the average ranks, and the length of the horizontal line segments centred on the dots corresponds to the CD. If there are overlaps between the horizontal line segments of the two algorithms, then there is no remarkable difference between them. Among the four algorithms, the FKELM has the best accuracy while the FTSVM has the worst. Hence, it can be clearly seen that the FKELM has significantly better performance than the FTSVM. Further, the FELM has slightly better performance than the RFSVM and significantly better performance than the FTSVM.

4.2. Discussion of Results

Table 5 and Table 6 and Figure 8, Figure 9, Figure 10 and Figure 11 report quantitative comparisons between all the algorithms considered, namely the RFSVM, FTSVM, FELM, and FKELM.

Table 7. Qualitative assessment of the different algorithms.

Algorithm	Size of Dataset	Time	Classification Performance
RFSVM	Small and medium samples	Much	Poor
	Large samples	Much	Sightly good
FTSVM	Small and medium samples	Medium	Poor
	Large samples	Medium	Poor
FELM	Small and medium samples	Little	Significantly good
	Large samples	Little	Good
FKELM	Small and medium samples	Little	Significantly good
	Large samples	Little	Good

presents a qualitative assessment of their performance. According to the Friedman test, the FKELM is ranked first, followed by the FELM, RFSVM and FTSVM.

5. Conclusions

In general, the ELM has better generalization capacity and higher efficiency than the traditional SVM. In order to improve efficiency and classification accuracy, in this study, two new algorithms FELM and FKELM were proposed, where they used a feature ranking strategy and a memorization–generalization kernel, respectively. Both algorithms can be applied to datasets with small or medium sample sizes. The FELM and FKELM can simultaneously complete the feature selection and classification processes, improving their training efficiency.

Experiments on artificial and benchmark datasets demonstrated that the proposed approaches have higher classification accuracy and higher learning speed than the RFSVM and FTSVM. According to the Friedman statistical method and the Nemenyi classification performance test, the FKELM is ranked first, followed by the FELM, RFSVM, and FTSVM. Based on these ranks, the FKELM exhibits significantly better performance than the FTSVM, while the FELM has slightly better performance than the RFSVM and significantly better performance than the FTSVM. In most cases, the FKELM is the most accurate in the classification performance, whereas the FTSVM is the worst.

By comparing the classification accuracy on ultra-high-dimension datasets, for example, Colon and Leukaemia datasets, it was concluded that the FELM and FKELM have slightly lower performance than the RFSVM. As seen in Table 5, it can be obtained that the FELM and FKELM can complete the classification process at a significantly high learning speed. Therefore, the focuses of our future research are improving the accuracy and robustness of the proposed algorithms on such ultra-high-dimension datasets and using the feature ranking strategy on other improved ELM models. Furthermore, it should be noted that in this study, we only verified the classification accuracy. In future studies, we will also attempt to verify the regression capacity of the proposed algorithms.