RCBLS: An Outlier-Robust Broad Learning Framework with Compact Structure

Abstract

Recently, the broad learning system (BLS) has been widely developed in virtue of its excellent performance and high-computational efficiency. However, two deficiencies still exist in BLS and preclude its deployment in real applications. First, the standard BLS performs poorly in outlier environments because the least squares loss function it uses to train the network is sensitive to outliers. Second, the model structure of BLS is likely to be redundant since the hidden nodes in it are randomly generated. To address the above two issues, a new robust and compact BLS (RCBLS), based on M-estimator and sparsity regularization, is proposed in this paper. The RCBLS develops from the BLS model and maintains its excellent characteristics, but replaces the conventional least squares learning criterion with an M-estimator-based loss function that is less sensitive to outliers, in order to suppress the incorrect feedback of the model to outlier samples, and hence enhance its robustness in the presence of outliers. Meanwhile, the RCBLS imposes the sparsity-promoting ${\mathcal{l}}_{2,1}$ -norm regularization instead of the common ${\mathcal{l}}_2$-norm regularization for model reduction. With the help of the row sparsity of ${\mathcal{l}}_{2,1}$-norm regularization, the unnecessary hidden nodes in RCBLS can be effectively picked out and removed from the network, thereby resulting in a more compact network. The theoretical analyses on outlier robustness, structural compactness and computational complexity of the proposed RCBLS model are provided. Finally, the validity of the RCBLS is verified by regression, time series prediction and image classification tasks. The experimental results demonstrate that the proposed RCBLS has stronger anti-outlier ability and more compact network structure than BLS and other representative algorithms.

1. Introduction

Over the past few years, deep learning has attracted considerable research interest in the field of artificial intelligence for its outstanding performance. However, deep learning methods also have some shortcomings, such as complicated network structures, time-consuming learning processes, poor interpretabilities, and so on. To alleviate this situation, Chen and Liu [1] recently proposed a novel neural network learning framework called the Broad Learning System (BLS). Unlike deep networks, BLS is built to construct neural networks via a broader application. In BLS, the original input data are first mapped into a group of feature spaces through feature learning. Then, the mapped features are further enhanced by nonlinear transformation. Finally, all the mapped and enhanced features are broadly concatenated and fed to the output layer, and the desired output weights are usually obtained by optimizing a regularized least squares (LS) problem. Compared with traditional deep learning methods, BLS has two distinct advantages [2]. First, drawing on the random learning theory of random vector functional link neural network [3], all the learning parameters used for feature mapping and feature enhancement in BLS are generated randomly and remain unchanged, then the output weights are the only model parameters that can be determined analytically, without iterative adjustments, thus making the learning process of BLS very efficient. Second, the model structure of BLS is very flexible, and it can be easily expanded to the incremental learning mode when there are new neural nodes or training samples. Plenty of experimental results on representative benchmarks and practical applications have demonstrated that BLS can learn thousands of times faster than traditional deep learning models, while obtaining a comparable or even better learning accuracy [1,4].

1.1. Related Work

Due to its strong generalization capability and superior learning efficiency, BLS has attracted extensive attentions from both academia and industry, and various improvements and expansions have been made to it [4]. As a novel neural network model, BLS has much room for network structure expansion, and many BLS-based structural variants have been constantly proposed, such as cascade BLS [5], deep cascade BLS [6], intergroup cascade BLS [7], stacked BLS [8], recurrent BLS and gated BLS [9,10], cascade BLS with dropout or dense connection [11], incremental multilayer BLS [12], etc. By introducing Takagi–Sugeno fuzzy systems into BLS, a new fuzzy BLS was established [13] and studied deeply [14,15,16,17,18]. Aiming at the imbalance learning problem, a kernel-based BLS [19] with a double kernel-mapping mechanism and an adaptive weighted BLS [20], with a density-based weight generation scheme, were put forward, respectively. To cope with datasets with a few labeled samples, the BLS was extended based on the manifold regularized sparse autoencoder method to develop a semi-supervised BLS [21]. To expand the capability of BLS in big data scenarios, a new hybrid recursive implementation of BLS was presented in [22]. In order to explain why BLS works well, the literature [23] has provided some theoretical interpretations of BLS from the perspective of the frequency domain. Recently, reinforcement learning has been widely introduced into BLS for performance improvement [24,25,26]. In addition, BLS and its variants have been widely applied to many practical applications, such as industrial intrusion detection [27], driver fatigue detection [28], disease detection [29,30,31], tropical cyclogenesis detection [32], intelligent fault diagnosis [33,34,35], license plate recognition [36], indoor fingerprint localization [37], air quality forecasting [38], photovoltaic power forecasting [39], stock price prediction [40], traffic flow prediction [41], machinery remaining useful life prediction [42], static voltage stability index prediction [43], time series prediction [44,45,46,47] and many others.

Although BLS has made great strides in many fields, it still suffers from the following two drawbacks. First, it lacks robustness to outliers. In standard BLS, the LS is taken as the default optimization criterion to train the model. The LS criterion can provide good performance in clean or Gaussian data environments, but it is very sensitive to outliers, which will severely degrade the performance of BLS when the training data are corrupted by outliers. To improve this situation, several robust versions of BLS have been proposed. In [48], three robust broad learning systems (RBLS) with different regularizers were developed to handle the data modeling with outliers contaminated. The core idea of RBLS is to use a ${\mathcal{l}}_1$-norm-based loss function as the optimization criterion for training, in order to improve the robustness of the models. In [49], several weighted penalty factor methods were applied to BLS to deal with outliers, and then the weighted BLS (WBLS) was put forward. By using the weighted penalty factor to constrain each sample’s contribution to modeling, the adverse effects of abnormal samples can be greatly reduced. Inspired by the robustness characteristic of correntropy, the maximum correntropy criterion (MCC) was adopted to train the BLS, generating a correntropy-based BLS (CBLS) [50]. Because of the intrinsic advantages of MCC, CBLS can achieve good performance in both outlier and noise-free environments. Additionally, a graph regularized robust broad learning system (GRBLS) [51] and a progressive ensemble kernel-based broad learning system (PEKB) [52] were recently proposed for noisy data classification. Though the above robust BLS variants are potential candidates when the training data contain outliers, they still have certain limitations. For instance, both RBLS and WBLS are primarily developed to handle single-output regression problems, but cannot cope with multi-output regression and multi-classification problems. Similarly, both GRBLS and PEKB are stinted on classification problems other than regression problems. As for CBLS, its robustness depends heavily on the kernel size, but how to determine this parameter is difficult under a lack of guidance. Therefore, the development of a more general robust BLS model remains to be further studied.

Another drawback of BLS is that its network structure may be redundant. As a kind of random neural network, BLS generates feature nodes and enhancement nodes by random projection. This randomness makes BLS very efficient in training. However, due to the instability and uncertainty of random projection, it is easy to produce redundant and irrelevant hidden nodes. These unnecessary hidden nodes not only increase the complexity of the BLS model and bring more computational overhead, but also degenerate its performance in turn due to overfitting. For this problem, targeted research work is insufficient. In [53], a new feature reduction layer based on sparse preservation projection was constructed and integrated into BLS to remove redundant or noisy information embedded in high-dimensional data, thus greatly improving the training efficiency and accuracy of large-scale multi-label classification. In [54], a joint regularization method was utilized to constrain the output weights of BLS for feature selection, in order to avoid the negative effects of feature redundancy and overfitting. Similarly, Miao et al. [55] applied a combined sparse regularization method to BLS to remove unnecessary feature information and thus boost classification performance. Nevertheless, these efforts are mainly focused on improving the performance of BLS for specific tasks through feature reduction [53] or feature selection [54,55], while research on the model structure simplification of BLS has not attracted enough attention. Therefore, how to pick out and eliminate the useless hidden nodes in BLS, in order to build a lightweight BLS model, is still an open problem to be solved.

1.2. Our Work

To address the above two concerns simultaneously, in this paper, we propose a novel, robust and compact broad learning system (RCBLS) for noisy data modeling. Different from the common BLS models that use LS as the Learning Criterion, the proposed RCBLS adopts a robust M-estimator-based loss function to train the model, in order to attenuate the adverse impacts of outliers and improve its anti-interference ability to outliers. Meanwhile, the RCBLS employs a new ${\mathcal{l}}_{2,1}$-norm regularization to replace the traditional ${\mathcal{l}}_2$-norm regularization for model reduction (the ${\mathcal{l}}_2$-norm and ${\mathcal{l}}_{2,1}$-norm are defined in Equations (1) and (2), respectively, in Section 2.1). As a sparsity promotion regularization method, the ${\mathcal{l}}_{2,1}$-norm regularization encourages row sparsity and can automatically compress the output weights corresponding to redundant hidden nodes to zero, which helps eliminate unnecessary hidden nodes and results in a more compact network. Extensive experiments on benchmark datasets are conducted to verify the effectiveness of the proposed method.

The key contributions of our work are as follows:

• A unified and general robust broad learning framework, called RCBLS, is proposed for noisy data regression and classification. By exploiting the robustness property of the M-estimator technique, different M-estimator functions such as Huber, Bisquare and Cauchy can be incorporated into RCBLS as optimization criterion to realize robust learning.
• Within this framework, the sparsity-promoting regularization with ${\mathcal{l}}_{2,1}$-norm is employed for model structure simplification. To the best of our knowledge, this is the first work that unifies the M-estimator and the sparse regularization into one single framework to build a robust and compact BLS model.
• An efficient optimization algorithm is devised to solve the RCBLS model. Additionally, the robustness, compactness and computational complexity of the proposed method are analyzed theoretically and verified empirically.

The rest of the paper is organized as follows. Section 2 provides the preliminaries of this paper. Section 3 presents the details of the proposed RCBLS. In Section 4, the robustness, compactness and computational complexity of RCBLS are analyzed theoretically. In Section 5, the performance of RCBLS is verified by multiple problems including regression, time series prediction and image classification. Finally, the conclusions and aims of future work are provided in Section 6.

2. Preliminaries

2.1. Notations

To ensure the readability and completeness of this work, we first summarize the notations used in the paper. Scalars are written in lowercase italics, vectors are written in lowercase boldface, and matrices are written in uppercase boldface. For matrix $\mathbf{X}\in {ℝ}^{m\times n}$, its i-th row, j-th column are denoted by ${\mathbf{x}}_i\in {ℝ}^{1\times n}$, ${\mathbf{x}}^j\in {ℝ}^{m\times 1}$ respectively, and the (i, j) element of $\mathbf{X}$ is declared as $x_{ij}$. The inverse matrix and transposed matrix of $\mathbf{X}$ are denoted as ${\mathbf{X}}^{-1}$ and ${\mathbf{X}}^T$, respectively. The Frobenius norm (${\mathcal{l}}_2$-norm) of $\mathbf{X}$ is defined as

$${‖\mathbf{X}‖}_F=\sqrt{{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^n{x}_{ij}^2}}}=\sqrt{{\displaystyle \sum _{i=1}^m{‖{\mathbf{x}}_i‖}_2^2}}.$$

(1)

The ${\mathcal{l}}_{2,1}$-norm of $\mathbf{X}$ is defined as

$${‖\mathbf{X}‖}_{2,1}={\displaystyle \sum _{i=1}^m\sqrt{{\displaystyle \sum _{j=1}^n{x}_{ij}^2}}}={\displaystyle \sum _{i=1}^m{‖{\mathbf{x}}_i‖}_2}.$$

(2)

Here ${‖{\mathbf{x}}_i‖}_2$ represents the ${\mathcal{l}}_2$-norm of the row vector ${\mathbf{x}}_i$, which is defined as ${‖{\mathbf{x}}_i‖}_2=\sqrt{{\displaystyle {\sum }_{j=1}^n{x}_{ij}^2}}$.

2.2. BLS

BLS is an emerging neural network learning framework that has been proposed recently, and its typical structure is depicted in Figure 1. As the name implies, BLS is built in a broad sense rather than a deep manner, and the detailed establishment process is described below.

For a given training dataset $\mathbf{X}={[{\mathbf{x}}_1^T,{\mathbf{x}}_2^T,\dots ,{\mathbf{x}}_N^T]}^T\in {ℝ}^{N\times d},\mathbf{Y}={[{\mathbf{y}}_1^T,{\mathbf{y}}_2^T,\dots ,{\mathbf{y}}_N^T]}^T\in {ℝ}^{N\times c}$, where N is the number of training samples, d is the dimension of the input ${\mathbf{x}}_i$, and c is the dimension of the output ${\mathbf{y}}_i$. First, the input data $\mathbf{X}$ are mapped into n groups of random feature spaces in “feature nodes”:

$${\mathbf{Z}}_i={\varphi }_i({\mathbf{X}\mathbf{W}}_{fi}+{\mathit{\beta }}_{fi})\in {ℝ}^{N\times p},i=1,2,\dots ,n$$

(3)

where ${\varphi }_i(\cdot )$ is usually a linear activation function for feature mapping, ${\mathbf{W}}_{fi}\in {ℝ}^{d\times p}$ and ${\mathit{\beta }}_{fi}\in {ℝ}^{N\times p}$ are, respectively, the randomly generated weights and biases of the i-th feature mapping group, and p is the number of feature nodes in each group. Denote the concatenation of all the n groups of feature nodes as ${\mathbf{Z}}^n=[{\mathbf{Z}}_1,{\mathbf{Z}}_2,\dots ,{\mathbf{Z}}_n]\in {ℝ}^{N\times np}$, and which are further transformed to generate m groups of enhancement features in the “enhancement nodes”:

$${\mathbf{H}}_j={\xi }_j({\mathbf{Z}}^n {\mathbf{W}}_{ej}+{\mathit{\beta }}_{ej})\in {ℝ}^{N\times q},j=1,2,\dots ,m$$

(4)

where ${\xi }_{ j}(\cdot )$ is usually a nonlinear activation function for feature enhancement,${\mathbf{W}}_{ej}\in {ℝ}^{np\times q}$ and ${\mathit{\beta }}_{ej}\in {ℝ}^{N\times q}$ are, respectively, the randomly generated weights and biases of the j-th enhancement group, and q is the number of enhancement nodes in each group. Similarly, ${\mathbf{H}}^m=[{\mathbf{H}}_1,{\mathbf{H}}_2,\dots ,{\mathbf{H}}_m]\in {ℝ}^{N\times mq}$ denotes the concatenation of all the m groups of enhancement nodes.

Finally, all the feature and enhancement nodes are concatenated together and linked to the output layer; therefore, the BLS model can be expressed as:

$$\mathbf{Y}=\mathbf{U}\mathbf{W}$$

(5)

where $\mathbf{U}=[{\mathbf{Z}}^n|{\mathbf{H}}^m]\in {ℝ}^{N\times L}$ is a transformed feature representation of the original input X by feature mapping and feature enhancement, $\mathbf{W}\in {ℝ}^{L\times c}$ are the output weights between the hidden layer and output layer, and $L=np+mq$ denotes the total number of feature and enhancement nodes. As mentioned above, all the feature nodes in ${\mathbf{Z}}^n$ and all the enhancement nodes in ${\mathbf{H}}^m$ are randomly generated and stay the same, then the broad feature matrix $\mathbf{U}$ is a constant matrix. Therefore, training a BLS is now reduced to finding a LS solution $\mathbf{W}$ of the linear system $\mathbf{Y}=\mathbf{U}\mathbf{W}$. In standard BLS, the following objective function with ${\mathcal{l}}_2$-norm regularization is usually used to optimize $\mathbf{W}$:

$$\underset{\mathbf{W}}{\arg \hspace{0.17em}\min }{‖\mathbf{Y}-\mathbf{U}\mathbf{W}‖}_F^2+\lambda {‖\mathbf{W}‖}_F^2$$

(6)

where ${‖\mathbf{Y}-\mathbf{U}\mathbf{W}‖}_F^2$ is the sum of squared training errors, which represents the loss, ${‖\mathbf{W}‖}_F^2$ is the squared ${\mathcal{l}}_2$-norm of the network output weights, which acts as a regularizer, and λ is a regularization parameter to balance the two items and enhance the generalization ability of the model.

Then, setting the derivative of Equation (6), with respect to $\mathbf{W}$ equal to zero, we finally obtain the optimal output weights as

$$\mathbf{W}={\left({\mathbf{U}}^T\mathbf{U}+\lambda \mathbf{I}\right)}^{-1}{\mathbf{U}}^T\mathbf{Y}$$

(7)

where $\mathbf{I}\in {ℝ}^{L\times L}$ is an identity matrix.

3. The Proposed RCBLS

In this section, a novel, robust and compact broad learning system (RCBLS), combining M-estimator technique and sparse regularization method, is proposed. First, we present a unified learning framework of RCBLS. Next, an iterative optimization method is derived to solve the model, and then a complete RCBLS algorithm is summarized.

3.1. The RCBLS Model

As mentioned in Section 2.2, the standard BLS uses LS as the loss function, which aims to minimize the sum of squared training errors. That is, Equation (6) can be equivalently rewritten as

$$\begin{array}{l}\underset{\mathbf{W}}{\arg \hspace{0.17em}\min }{\displaystyle \sum _{i=1}^N{‖{\mathbf{e}}_i‖}_2^2}+\lambda {‖\mathbf{W}‖}_F^2\\ \mathrm{s}.\mathrm{t}. {\mathbf{e}}_i={\mathbf{y}}_i-{\mathbf{u}}_i\mathbf{W}, i=1,2,\dots ,N\end{array}$$

(8)

where ${\mathbf{e}}_i$ is the residual between the actual observation and the network output corresponding to the i-th training sample. Although LS loss is a good choice in a clean data environment, it is very sensitive to outliers and prone to generate overfitting models, resulting in serious performance degradation of BLS in the presence of outliers. Intuitively, contaminated training samples, such as outliers, will produce large estimation errors, while the square operation used in LS may amplify them further, making the objective function highly biased and thus causing the learned model to be inaccurate. Therefore, employing robust metrics instead of the traditional LS criteria is more likely to overcome outliers and obtain robust BLS models.

In addition, due to the random generation rules of feature nodes and enhancement nodes, the obtained broad feature matrix $\mathbf{U}\in {ℝ}^{N\times L}$ in BLS may contain redundant and irrelevant components, especially when the number of feature nodes and enhancement nodes L is large. As shown in Equation (8), the general BLS model usually uses the ${\mathcal{l}}_2$-norm-based regularizer ${‖\mathbf{W}‖}_F^2$ in the objective function for optimization. As a common choice, the ${\mathcal{l}}_2$-norm regularization is capable of avoiding overfitting and improving the generalization capability of the model, but it is powerless to eliminate redundancy. More specifically, by constraining $\mathbf{W}$ with ${\mathcal{l}}_2$-norm, Equation (8) usually produces a dense solution for the output weights, in which all the values are discretely distributed in a non-zero form. This means that, for all the feature and enhancement nodes that connect the output weights, they play a certain role within the network and should be retained in the final BLS model. That is to say, the default ${\mathcal{l}}_2$-norm regularization is prone to train a large and redundant BLS model due to the non-zero output weights of the network. In contrast, sparse regularization methods tend to produce sparse solutions with partial components of zeros, which is conducive to eliminating unnecessary hidden nodes and improving the compactness of the BLS model.

Based on the above analysis, we introduce the M-estimator robust learning framework and ${\mathcal{l}}_{2,1}$-norm-based sparse regularization method into BLS, and propose a new RCBLS model, whose objective function can be formulated as

$$\begin{array}{l}\underset{\mathbf{W}}{\arg \hspace{0.17em}\min } {\displaystyle \sum _{i=1}^N\rho \left( {‖{\mathbf{e}}_i‖}_2\right)}+\lambda {‖ \mathbf{W}‖}_{2,1}\\ \mathrm{s}.\mathrm{t}. {\mathbf{e}}_i={\mathbf{y}}_i-{\mathbf{u}}_i\mathbf{W}, i=1,2,\dots ,N\end{array}$$

(9)

where $\rho \left( \cdot \right)$ is an M-estimator function, and ${‖ \cdot ‖}_{2,1}$ denotes the ${\mathcal{l}}_{2,1}$-norm of a matrix, which is defined as Equation (2). Compared with the original BLS, the proposed RCBLS mainly has the following two improvements.

First, RCBLS adopts a robust M-estimator function to replace the traditional LS as a loss function. The M-estimator is one of the most popular approaches to combat outliers, and its basic idea is to use some mild piecewise functions as the fitness measure [56]. When the training residual is significantly large, its growth trend is more gentle than that of the LS estimator or even remains unchanged, in order to alleviate the adverse impacts of outlying residual errors. As a unified robust learning framework, a variety of M-estimators can be applied as $\rho \left( \cdot \right)$, such as Huber, Bisquare, Cauchy, and their definitions and robustness analyses will be provided in Section 4.1.

Second, instead of using the common ${\mathcal{l}}_2$-norm regularization, a new ${\mathcal{l}}_{2,1}$-norm-based sparse regularizer $\lambda {‖ \mathbf{W}‖}_{2,1}$ is imposed in our RCBLS for generalization improvement, as well as structure simplification. As a sparsity promotion regularization method, the ${\mathcal{l}}_{2,1}$-norm regularization penalizes each row of $\mathbf{W}$ as a whole and enforces sparsity among rows. Then, the derived solution $\mathbf{W}$ is row sparse, in which many of the weight vectors ${\mathbf{w}}_i$ are zero vectors. This means that the corresponding hidden nodes connecting zero weights are useless and can be deleted from the network, thus achieving a compact and lightweight network. A more detailed compactness analysis of RCBLS will be presented in Section 4.2.

3.2. Optimization of RCBLS

The objective function in Equation (9) is convex but non-smooth, so it cannot be solved analytically. Here, an efficient iterative optimization method based on augmented Lagrangian multiplier is devised to solve this problem. According to the KKT theorem, optimizing Equation (9) is equivalent to solving the following dual optimization problem:

$$\mathcal{L}(\mathbf{W},\mathbf{E},\mathbf{A})= {\displaystyle \sum _{i=1}^N\rho \left( {‖{\mathbf{e}}_i‖}_2\right)}+\lambda {‖ \mathbf{W}‖}_{2,1}+{\displaystyle \sum _{i=1}^N\left( {\mathbf{y}}_i-{\mathbf{u}}_i\mathbf{W}-{\mathbf{e}}_i\right)}{\mathbf{a}}_i^T$$

(10)

where $\mathbf{W}={\left[{\mathbf{w}}_1^T,{\mathbf{w}}_2^T,\dots ,{\mathbf{w}}_L^T\right]}^T\in {ℝ}^{L\times c}$ are the output weights to be solved, $\mathbf{E}={[{\mathbf{e}}_1^T,{\mathbf{e}}_2^T,\dots ,{\mathbf{e}}_N^T]}^T\in {ℝ}^{N\times c}$, in which ${\mathbf{e}}_i=\left[e_{i1},e_{i2},\dots ,e_{ic}\right], i=1,2,\dots ,N$ are the residual errors corresponding to the i-th sample, $\mathbf{A}={\left[{\mathbf{a}}_1^T,{\mathbf{a}}_2^T,\dots ,{\mathbf{a}}_N^T\right]}^T\in {ℝ}^{N\times c}$, in which ${\mathbf{a}}_i=\left[a_{i1},a_{i2},\dots ,a_{ic}\right], i=1,2,\dots ,N$ are the Lagrangian multipliers for the i-th equality constraint in Equation (9).

Taking the derivatives of $\mathcal{L}(\mathbf{W}, \mathbf{E},\mathbf{A})$ in Equation (10), with respect to $\mathbf{W}$, ${\mathbf{e}}_i$, ${\mathbf{a}}_i$, respectively, and setting the derivatives to zero, we arrive at:

$$\left\{\begin{array}{}\mathrm{(11a)}& \frac{\partial \mathcal{L}(\mathbf{W},\mathbf{E},\mathbf{A})}{\partial \mathbf{W}} = 0\Rightarrow \lambda \mathbf{D}\mathbf{W} = {\displaystyle \sum _{i=1}^N}{\mathbf{u}}_i^T{\mathbf{a}}_i\hfill \mathrm{(11b)}& \frac{\partial \mathcal{L}(\mathbf{W}, \mathbf{E},\mathbf{A})}{\partial {\mathbf{e}}_i} = 0 \Rightarrow {\mathbf{a}}_i=\frac{\partial \rho \left( {‖{\mathbf{e}}_i‖}_2\right)}{\partial {‖{\mathbf{e}}_i‖}_2}\frac{\partial {‖{\mathbf{e}}_i‖}_2}{\partial {\mathbf{e}}_i}=\frac{\partial \rho \left( {‖ {\mathbf{e}}_i‖}_2\right)}{\partial {‖ {\mathbf{e}}_i‖}_2}\frac{{\mathbf{e}}_i}{ {‖{\mathbf{e}}_i‖}_2}, i=1,2,\dots ,N\hfill \mathrm{(11c)}& \frac{\partial \mathcal{L}(\mathbf{W}, \mathbf{E},\mathbf{A})}{\partial {\mathbf{a}}_i} = 0\Rightarrow {\mathbf{y}}_i-{\mathbf{u}}_i\mathbf{W}-{\mathbf{e}}_i=0, i=1,2,\dots ,N\hfill \end{array}\right.$$

where $\mathbf{D} \in {ℝ}^{L\times L}$ is a diagonal matrix with the j-th diagonal element $d_{jj}=\frac{1}{{‖{\mathbf{W}}_j‖}_2}$, that is

$$\mathbf{D} =diag\left(\frac{1}{{‖{\mathbf{W}}_1‖}_2},\frac{1}{{‖{\mathbf{W}}_2‖}_2},\dots ,\frac{1}{{‖{\mathbf{W}}_L‖}_2}\right) .$$

(12)

Since $\mathbf{D}$ is a coefficient matrix corresponding to the derivation of the ${\mathcal{l}}_{2,1}$-norm regularization term, we call it the regularization matrix.

Defining

$$\theta \left({‖{\mathbf{e}}_i‖}_2\right)=\frac{\partial \rho \left( {‖ {\mathbf{e}}_i‖}_2\right)}{\partial {‖ {\mathbf{e}}_i‖}_2}/ {‖ {\mathbf{e}}_i‖}_2$$

(13)

as the weight function, with respect to M-estimator $\rho \left( \cdot \right)$, and letting ${\theta }_i=\theta \left({‖ {\mathbf{e}}_i‖}_2\right)$, Equation (11) can be rewritten into the following matrix form:

$$\left\{\begin{array}{cc}\lambda \mathbf{D}\mathbf{W} ={\mathbf{U}}^{T}A\hfill & \left(14\mathrm{a}\right)\\ \mathbf{A}=\mathsf{\theta } \mathbf{E}\hfill & \left(14\mathrm{b}\right)\\ \mathbf{U}\mathbf{W}-\mathbf{Y}+\mathbf{E}=0\hfill & \left(14\mathrm{c}\right)\end{array}\right.$$

where $\mathsf{\theta }=diag\left({\theta }_1,{\theta }_2,\dots ,{\theta }_N\right)$ is the weight matrix.

Then, multiplying both sides of (14c) by ${\mathbf{U}}^T\mathsf{\theta }$, we obtain

$${\mathbf{U}}^T\mathsf{\theta }\mathbf{U}\mathbf{W}-{\mathbf{U}}^T\mathsf{\theta }\mathbf{Y}+{\mathbf{U}}^T\mathsf{\theta }\mathbf{E}=0.$$

(15)

Substitute Equation (14b) into Equation (14a), we have

$$\lambda \mathbf{D} \mathbf{W}={\mathbf{U}}^T\mathsf{\theta }\mathbf{E}.$$

(16)

Substitute Equation (16) into Equation (15), we have

$${\mathbf{U}}^T\mathsf{\theta }\mathbf{U}\mathbf{W}-{\mathbf{U}}^T\mathsf{\theta }\mathbf{Y}+\lambda \mathbf{D}\mathbf{W}=0.$$

(17)

Consequently, we obtain the analytic solution of output weights as

$$\mathbf{W}={\left({\mathbf{U}}^T\mathsf{\theta }\mathbf{U}+\lambda \mathbf{D}\right)}^{-1}{\mathbf{U}}^T\mathsf{\theta }\mathbf{Y}.$$

(18)

By comparing Equations (18) and (7), it can be seen that the main difference between RCBLS and BLS is that two matrices, the weight matrix $\mathsf{\theta }$ and the regularization matrix $\mathbf{D}$, are added to the analytic solution of RCBLS. The essence of the weight matrix $\mathsf{\theta }$ is to reversely assign different weights to each sample according to their training errors, in order to reduce the disturbances of outlier samples to the model. The regularization matrix $\mathbf{D}$ is introduced as an auxiliary variable to solve the ${\mathcal{l}}_{2,1}$-norm optimization problem, in order to achieve a sparse solution.

In addition, we can observe from Equation (18) that the computing of $\mathbf{W}$ relies on $\mathsf{\theta }$ and $\mathbf{D}$, while $\mathsf{\theta }$ and $\mathbf{D}$ also depend on $\mathbf{W}$. So, it is impractical to calculate $\mathbf{W}$ directly. In this work, we design an iterative method to solve this problem by computing $\mathsf{\theta }$, $\mathbf{D}$ and $\mathbf{W}$ alternately. More specifically, in each iteration, the values of $\mathsf{\theta }$ and $\mathbf{D}$ are first calculated with the current $\mathbf{W}$, then $\mathbf{W}$ is updated based on the newly calculated $\mathsf{\theta }$ and $\mathbf{D}$. The iteration process is repeated until reaching the maximum number of iterations. By incorporating the above iterative optimization process, Algorithm 1 describes the complete procedure of our RCBLS.

It is worth noting that in step 15 of the RCBLS algorithm, when ${‖{\mathbf{w}}_j‖}_2=0$, then the corresponding $d_{jj}=\frac{1}{{‖{\mathbf{W}}_j‖}_2}$ will tend to infinity. In this case, the proposed algorithm cannot be guaranteed to converge. So, in the implementation of our RCBLS, we regularize $d_{jj}$ as $d_{jj}=\frac{1}{{‖{\mathbf{W}}_j‖}_2+\varsigma }$, where $\varsigma ={10}^{-6}$ is a very small constant to avoid the denominator being zero. It is easy to see that $\frac{1}{{‖{\mathbf{W}}_j‖}_2+\varsigma }$ approximates $\frac{1}{{‖{\mathbf{W}}_j‖}_2}$ when $\varsigma \to 0$.

Algorithm 1: Robust compact broad learning system (RCBLS)

4. Discussions

In this section, we theoretically analyze the outlier robustness, structural compactness and computational complexity of the proposed RCBLS.

4.1. Robustness Analysis

As stated in Section 3, the adoption of M-estimator and its corresponding weighting strategy is the key to ensure the strong robustness of RCBLS. The commonly used M-estimators mainly include Huber, Bisquare, Cauchy, etc. [56], and their cost functions, weight functions and default parameters are listed in

Table 1. List of cost functions, weight functions and default parameters for three typical M-estimators.

Name	$\mathbf{Cost} \mathbf{Function}\mathit{\rho }\left(\mathit{e}\right)$	$\mathbf{Weight} \mathbf{Function}\mathit{\theta }\left(\mathit{e}\right)$	Default Tuning
Huber	$\left\{\begin{array}{c}\frac{1}{2}{e}^2, \left|e\right|\le k\hfill \\ k\left|e\right|-\frac{1}{2}{k}^2, \left|e\right|>k\end{array}\right.$	$\left\{\begin{array}{c}1 , \left|e\right|\le k\hfill \\ \frac{k}{\left|e\right|}, \left|e\right|>k\end{array}\right.$	1.345
Bisquare	$\left\{\begin{array}{c}\frac{k^2}{6}\left(1-{\left(1-{\left(\frac{e}{k}\right)}^2\right)}^3\right) , \left|e\right|\le k\hfill \\ \frac{k^2}{6}, \left|e\right|>k\hfill \end{array}\right.$	$\left\{\begin{array}{c}{\left(1-{\left(\frac{e}{k}\right)}^2\right)}^2 , \left|e\right|\le k\hfill \\ 0, \left|e\right|>k\hfill \end{array}\right.$	4.685
Cauchy	$\frac{k^2}{2}\log \left(1+{\left(\frac{e}{k}\right)}^2\right)$	$\frac{1}{1+{\left(\frac{e}{k}\right)}^2}$	2.385

. Next, we take Huber as an example for robustness analysis. For ease of interpretation, the typical curves of the cost function and weight function of Huber are depicted in Figure 2. From Table 1 and Figure 2, it can be analyzed that when the error e is less than or equal to the threshold k, it means that the current observation is a normal sample, then the cost function of Huber is equivalent to the ordinary LS function, and the corresponding weight of the sample is 1. On the other hand, when the error e is greater than k, it indicates that the current observation is likely to be an outlier. Then, a weight less than 1 is assigned to the sample, and the weight decreases monotonically with the increase of the error, that is, the larger the error, the smaller the weight. Through this reverse weighting strategy, large perturbations stemming from outliers can be effectively suppressed and the sensitivity of the model to outliers can be significantly reduced. The robustness analyses of other M-estimators are similar to that of the Huber, except for a slight difference in the weighting skills. Due to space limitation, we will not reiterate them here.

From the perspective of robustness learning, the threshold k can be regarded as the tolerance degree of M-estimator function to outlier residuals, which has an important impact on the robustness and generalization performance of the RCBLS. A smaller value of k can produce stronger resistance to outliers, but it increases the probability of normal samples being misjudged as outliers, which makes the learning process insufficient. Conversely, a larger value of k increases the risk of outliers being misjudged as normal samples, resulting in inaccurate models. According to the robust statistics theory [57], the threshold k can be determined as

$$k=tuning\times \sigma$$

(19)

where tuning is an adjustment parameter, and its default values for three typical M-estimators are shown in Table 1; $\sigma$ is the estimated standard deviation of the training errors, and a commonly used robust estimator of $\sigma$ in outlier environment is provided as

$$\sigma =\mathrm{MAD}\left(\mathbf{e}\right)/0.6745=\mathrm{med}\left(\left|e_i-\mathrm{med}\left(\mathbf{e}\right)\right|\right)/0.6745,i=1,2,\dots ,N$$

(20)

where med(·) represents the median operation, and the constant value 0.6745 is introduced to make $\sigma$ an unbiased estimator for Gaussian errors [57].

4.2. Compactness Analysis

The original BLS uses the random projection method to generate feature and enhancement nodes, which avoids the tedious training process and greatly reduces the computational cost, but at the same time makes the generated neural network model redundant. More specifically, because of the instability and uncertainty of random projection, the obtained broad feature matrix $\mathbf{U}$ inevitably contains a certain number of redundant hidden nodes. However, since the ${\mathcal{l}}_2$-norm regularization used by default in BLS does not have sparsity, the resulting solution of the output weights $\mathbf{W}$ is usually dense. That is to say, all the hidden nodes in $\mathbf{U}$, including redundant ones, are endowed with certain connection weights and should be retained in the network, which makes the structure of BLS very bloated.

As an improvement, the proposed RCBLS imposes a new sparse regularizer with ${\mathcal{l}}_{2,1}$-norm in the objective function to optimize the output weights. Since the introduced ${\mathcal{l}}_{2,1}$-norm regularization encourages row-level sparsity, the derived output weights matrix $\mathbf{W}$ contains many zero rows. The row sparsity of the obtained $\mathbf{W}$ is of great significance for further model reduction. First, from the perspective of network structure analysis, the row vector ${\mathbf{w}}_i$ of $\mathbf{W}$ represents the output weights between the i-th hidden node and all output nodes. Therefore, a ${\mathbf{w}}_i$ of the zero vector means that all the connection weights from the current hidden node to each output node are zeros. Obviously, this hidden node and its out-going connections are unnecessary and can be removed from the network (highlighted in red in Figure 3a). In this way, a lightweight network with only necessary hidden nodes can be obtained. Moreover, the sparsity analysis and model reduction of RCBLS can also be explained from the perspective of model optimization. As discussed in Section 2.2, training a BLS model is equivalent to solving the linear system $\mathbf{Y}=\mathbf{U}\mathbf{W}$, where $\mathbf{U}=[{\mathbf{Z}}_1, {\mathbf{Z}}_2,\dots ,{\mathbf{Z}}_n | {\mathbf{H}}_1, {\mathbf{H}}_2,\dots ,{\mathbf{H}}_m]\in {ℝ}^{N\times L}$, $\mathbf{W}={\left[{\mathbf{w}}_1^T,{\mathbf{w}}_2^T,\dots ,{\mathbf{w}}_L^T\right]}^T\in {ℝ}^{L\times c}$. According to the rules of matrix multiplication, the elements in the i-th column of $\mathbf{U}$ are multiplied by the elements in the i-th row of $\mathbf{W}$, respectively. That is, the i-th column of $\mathbf{U}$ is corresponding to the i-th row of $\mathbf{W}$. Hence, for these zero rows, ${\mathbf{w}}_i$ in $\mathbf{W}$, their corresponding columns ${\mathbf{u}}^i$ (also known as the i-th hidden node) in $\mathbf{U}$ are considered useless and can be deleted, and only the other columns in $\mathbf{U}$, corresponding to the non-zero rows of $\mathbf{W}$, are useful and should be preserved in the network. As an example, Figure 3b shows the model reduction process by setting N = 6, L = 5, c = 3. In this example, the first, third, and fifth rows in $\mathbf{W}$ are zero rows. Therefore, we are able to remove the corresponding hidden nodes in the first, third and fifth columns in $\mathbf{U}$, in order to obtain a much simpler model. In summary, with the help of the row sparsity of the ${\mathcal{l}}_{2,1}$-norm regularization, our RCBLS is prone to generate a compact network with fewer hidden nodes.

4.3. Computational Complexity Analysis

Finally, we analyze the computational complexity of the proposed RCBLS. Using the same notations as in Algorithm 1, we simply set $\mathbf{Y}\in {ℝ}^{N\times c}$, $\mathbf{U}\in {ℝ}^{N\times L}$, $\mathsf{\theta } \in {ℝ}^{N\times N}$, $\mathbf{D} \in {ℝ}^{L\times L}$, where N denotes the number of training samples, L denotes the total number of feature and enhancement nodes, and c denotes the dimension of training outputs. From Algorithm 1, we can see that the main computational overheads of RCBLS come from the calculations of the output weights $\mathbf{W}$ in steps 11 and 16, while the computational costs of other steps are insignificant and negligible. In step 11, the complexity in calculating the initial $\mathbf{W}$ is $O\left(L^3+N{L}^2+N{L}^2+LNc\right)$, where $O\left(L^3\right)$ is the computational cost of the matrix inversion operation, and $O\left(N{L}^2+N{L}^2+LNc\right)$ are the computational costs of three separate matrix multiplication operations. In general, $N>L>c$, so the computational complexity in step 11 could be simplified as $O\left(N{L}^2\right)$. In step 16, the complexity in calculating $\mathbf{W}$ in each iteration is $O\left(L^3+L{N}^2+N{L}^2+N{L}^2+L{N}^2+LNc\right)$, where $O\left(L^3\right)$ is the computational cost of matrix inversion operation, and $O\left(L{N}^2+N{L}^2+N{L}^2+L{N}^2+LNc\right)$ are the computational costs of five separate matrix multiplication operations. For the same reason above, the computational complexity in step 16 could be simplified as $O\left(L{N}^2\right)$. Hence, the overall computational complexity of RCBLS is $O\left(N{L}^2+tL{N}^2\right)$, where t is the number of iterations. Empirical results in Section 5.5 will show that the convergence of RCBLS is fast and can converge to a stable value within only a few iterations, usually less than 10. Therefore, the computational efficiency of the proposed algorithm can be guaranteed in theory.

Compared to the original BLS with a computational complexity of $O\left(N{L}^2\right)$, the added computational complexity of our RCBLS is $O\left(tL{N}^2\right)$, which is produced by iterative optimization calculations. Though the complexity of RCBLS is somewhat higher than that of the original BLS, it is acceptable to trade a little more computation time for the desired robustness improvement and structure simplification.

5. Experiments

In this section, extensive experiments, including regression, time series prediction and image classification, are carried out for performance evaluation. Since the proposed RCBLS is a unified robust learning framework based on M-estimator, three typical M-estimators, Huber, Bisquare and Cauchy, are selected for empirical research, and the corresponding algorithms are denoted as Huber-RCBLS, Bisquare-RCBLS and Cauchy-RCBLS, respectively. To demonstrate the effectiveness and superiority of the proposed algorithms, the experimental results of our RCBLS are compared with that of the original BLS [1], FBLS [13] and WBLS [49]. For regression and time series prediction problems, the root mean squared error (RMSE) is used to measure the performances of the algorithms. For image classification problems, the testing accuracy is adopted for performance evaluation. To make the experimental results more convincing, ten independent experimental runs are performed for each experiment, and the obtained experimental results are statistically analyzed and reported. Our experiments are conducted in the MATLAB R2016a software platform running on a laptop with a 2.9 GHZ CPU and 16 GB RAM, except for special explanations.

5.1. Robustness Evaluation on Regression Problems

We first evaluate the performances of RCBLS on six widely used regression datasets, which are detailed in

Table 2. Specification of six regression datasets.

Datasets	Training Samples	Testing Samples	Attributes
Abalone	2784	1393	8
Bodyfat	168	84	14
Housing	337	169	13
Mortgage	699	350	15
Strike	416	209	6
Weather Izmir	974	487	9

. To assess the robustness of the methods, for each dataset, a certain proportion of training data are randomly selected and contaminated by outliers, while the testing data remain clean. Specifically, for the selected training sample (x_i, y_i), a random value between [y_min, y_max] is added to replace the corresponding output y_i. In our study, the percentage of contaminated training samples to the total training samples is called the outlier level and denoted as R. In this experiment, four outlier levels are considered, i.e., R = 0%, 10%, 20% and 30%.

Six BLS-related algorithms, namely BLS, FBLS, WBLS, Huber-RCBLS, Bisquare-RCBLS and Cauchy-RCBLS, are used to model the contaminated datasets, and the experimental settings are as follows. For BLS, Huber-RCBLS, Bisquare-RCBLS and Cauchy-RCBLS, the common parameters to be set mainly include: the number of feature nodes per group p, feature nodes groups n, the number of enhancement nodes per group q, enhancement nodes groups m, and the regularization parameter λ. Referring to [1], we first set m = 1, and the other four parameters (p, n, q, λ) are determined by the grid search method, whose search spaces are {2, 4, …, 18, 20}, {2, 4, …, 18, 20}, {10, 20, …, 190, 200}, {10⁻¹⁰, 10⁻⁹, …, 10⁴, 10⁵}, respectively. Additionally, the maximum number of iterations t_max is uniformly set to 20 in our three RCBLS algorithms. For WBLS, since the feature mapping parameters p and n are not used, the key model parameters of this algorithm mainly include (q, λ), which are selected by grid search with the same search ranges as above. For FBLS, referring to [13], the key model parameters (N_r, N_t, N_e, λ) are decided by a similar grid search method, whose search spaces are {1, 2, …, 19, 20}, {1, 2, …, 9, 10}, {1, 2, …, 99, 100}, {10⁻¹⁰, 10⁻⁹, …, 10⁴, 10⁵}, respectively.

Table 3. Performance comparison of different algorithms on Abalone.

Algorithms	R = 0%			R = 10%			R = 20%			R = 30%
	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)
BLS	(12,2,200,−2)	0.05	7.58 ± 0.02	(12,2,140,−1)	0.02	8.83 ± 0.03	(2,8,40,4)	0.02	10.16 ± 0.15	(2,20,180,5)	0.06	10.34 ± 0.08
FBLS	(2,10,78,−10)	0.17	7.55 ± 0.02	(19,5,3,−4)	0.22	8.80 ± 0.02	(20,3,9,4)	0.13	9.52 ± 0.06	(19,1,37,4)	0.05	9.47 ± 0.15
WBLS	(200,−3)	0.17	7.61 ± 0.02	(130,−3)	0.10	7.58 ± 0.01	(200,−2)	0.17	7.85 ± 0.01	(190,0)	0.13	8.44 ± 0.01
Huber-RCBLS	(12,2,90,−3)	0.09	7.66 ± 0.04	(12,2,200,−2)	0.20	7.60 ± 0.03	(16,8,200,−1)	0.38	7.83 ± 0.04	(16,4,200,−1)	0.26	8.46 ± 0.03
Bisquare-RCBLS	(12,2,90,−3)	0.08	7.84 ± 0.04	(12,2,90,−3)	0.08	7.75 ± 0.03	(10,4,190,−2)	0.21	7.64 ± 0.02	(18,2,150,−2)	0.15	7.61 ± 0.03
Cauchy-RCBLS	(12,2,90,−3)	0.08	7.68 ± 0.04	(12,2,200,−2)	0.20	7.62 ± 0.02	(12,2,140,−2)	0.13	7.66 ± 0.03	(12,2,110,−2)	0.10	8.02 ± 0.03

,

Table 4. Performance comparison of different algorithms on Bodyfat.

Algorithms	R = 0%			R = 10%			R = 20%			R = 30%
	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)
BLS	(4,6,10,−10)	0.01	0.47 ± 0.04	(20,4,10,0)	0.01	4.93 ± 0.28	(6,20,20,1)	0.03	8.75 ± 0.39	(18,18,10,1)	0.03	13.47 ± 0.15
FBLS	(20,8,79,−4)	0.07	0.46 ± 0.05	(7,10,86,−1)	0.06	4.93 ± 0.19	(2,10,56,−1)	0.04	8.84 ± 0.65	(1,10,76,−2)	0.03	12.99 ± 0.40
WBLS	(190,−3)	0.02	0.80 ± 0.14	(90,−2)	0.01	1.53 ± 0.19	(120,−2)	0.01	1.57 ± 0.26	(100,−1)	0.01	3.09 ± 0.32
Huber-RCBLS	(4,4,50,−10)	0.01	0.19 ± 0.04	(4,6,10,−7)	0.01	0.31 ± 0.05	(4,6,10,−5)	0.01	0.38 ± 0.06	(2,20,10,−7)	0.03	0.98 ± 0.37
Bisquare-RCBLS	(4,4,50,−10)	0.01	0.19 ± 0.04	(4,8,40,−5)	0.02	0.23 ± 0.05	(4,4,30,−10)	0.01	0.22 ± 0.06	(10,16,10,−4)	0.04	0.40 ± 0.06
Cauchy-RCBLS	(4,4,50,−10)	0.01	0.17 ± 0.01	(4,8,40,−6)	0.02	0.21 ± 0.04	(4,4,30,−10)	0.01	0.22 ± 0.05	(10,16,10,−4)	0.04	0.39 ± 0.05

,

Table 5. Performance comparison of different algorithms on Housing.

Algorithms	R = 0%			R = 10%			R = 20%			R = 30%
	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)
BLS	(6,12,190,−1)	0.02	8.42 ± 0.24	(20,2,200,0)	0.01	10.21 ± 0.39	(20,2,140,1)	0.01	14.21 ± 0.30	(16,10,190,2)	0.02	16.59 ± 0.31
FBLS	(19,9,56,−3)	0.09	8.02 ± 0.33	(1,10,67,−7)	0.04	10.38 ± 0.61	(2,9,62,−4)	0.05	13.97 ± 0.55	(3,9,31,−3)	0.06	16.46 ± 0.51
WBLS	(180,−2)	0.03	9.02 ± 0.54	(180,−1)	0.03	9.49 ± 0.43	(200,1)	0.02	11.45 ± 0.19	(120,0)	0.02	11.55 ± 0.43
Huber-RCBLS	(18,8,180,−2)	0.07	9.06 ± 0.85	(8,8,150,−2)	0.04	9.36 ± 0.63	(16,14,200,0)	0.12	10.40 ± 0.34	(16,10,150,0)	0.07	11.02 ± 0.42
Bisquare-RCBLS	(14,2,180,−3)	0.03	10.16 ± 1.13	(12,2,200,−2)	0.04	10.30 ± 0.76	(20,2,140,−1)	0.03	11.02 ± 0.70	(12,2,200,−2)	0.03	9.83 ± 0.70
Cauchy-RCBLS	(14,2,180,−3)	0.03	9.49 ± 0.80	(12,6,190,−2)	0.05	9.70 ± 0.50	(16,14,200,0)	0.12	10.51 ± 0.41	(8,2,200,−2)	0.03	10.62 ± 1.02

,

Table 6. Performance comparison of different algorithms on Mortgage.

Algorithms	R = 0%			R = 10%			R = 20%			R = 30%
	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)
BLS	(16,6,200,−3)	0.02	0.50 ± 0.02	(6,10,10,1)	0.02	5.22 ± 0.06	(20,18,10,3)	0.05	8.27 ± 0.06	(20,4,10,3)	0.01	12.05 ± 0.16
FBLS	(19,8,62,−6)	0.14	0.52 ± 0.02	(2,10,17,−2)	0.07	5.26 ± 0.09	(1,7,100,−2)	0.04	8.43 ± 0.08	(2,2,1,2)	0.02	10.33 ± 1.29
WBLS	(200,−4)	0.06	0.51 ± 0.03	(200,−3)	0.06	0.53 ± 0.02	(190,−3)	0.05	0.55 ± 0.02	(160,−2)	0.04	0.84 ± 0.02
Huber-RCBLS	(18,6,170,−4)	0.10	0.49 ± 0.03	(20,2,160,−4)	0.05	0.56 ± 0.03	(4,8,70,−7)	0.03	0.60 ± 0.03	(8,6,80,−3)	0.03	0.88 ± 0.03
Bisquare-RCBLS	(18,6,170,−4)	0.10	0.53 ± 0.05	(20,2,180,−5)	0.06	0.56 ± 0.05	(18,6,170,−4)	0.10	0.54 ± 0.03	(6,6,100,−6)	0.03	0.61 ± 0.04
Cauchy-RCBLS	(18,6,170,−4)	0.10	0.50 ± 0.04	(20,4,200,−4)	0.09	0.50 ± 0.03	(4,16,100,−6)	0.05	0.51 ± 0.03	(6,8,90,−6)	0.04	0.57 ± 0.03

,

Table 7. Performance comparison of different algorithms on Strike.

Algorithms	R = 0%			R = 10%			R = 20%			R = 30%
	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)
BLS	(4,12,200,0)	0.02	10.13 ± 0.06	(4,18,30,4)	0.03	10.62 ± 0.01	(2,4,50,4)	0.01	10.60 ± 0.03	(4,12,200,5)	0.02	10.61 ± 0.02
FBLS	(5,7,79,−7)	0.05	10.16 ± 0.09	(8,1,14,3)	0.01	10.83 ± 0.04	(14,2,94,4)	0.02	10.83 ± 0.01	(4,1,4,3)	0.01	10.83 ± 0.05
WBLS	(200,−4)	0.04	10.07 ± 0.03	(130,−3)	0.02	10.11 ± 0.03	(150,−4)	0.02	10.06 ± 0.06	(140,0)	0.02	10.24 ± 0.02
Huber-RCBLS	(4,14,200,−3)	0.07	10.01 ± 0.05	(4,10,120,−3)	0.04	10.01 ± 0.08	(4,10,120,−3)	0.04	9.96 ± 0.09	(6,16,190,−1)	0.08	10.13 ± 0.07
Bisquare-RCBLS	(4,18,190,−4)	0.07	10.08 ± 0.15	(6,18,190,−3)	0.09	10.13 ± 0.08	(6,20,180,−3)	0.11	10.19 ± 0.17	(6,20,180,−2)	0.09	10.06 ± 0.09
Cauchy-RCBLS	(4,14,180,−4)	0.06	10.05 ± 0.09	(6,14,200,−3)	0.07	10.02 ± 0.07	(6,14,200,−3)	0.10	10.02 ± 0.08	(4,14,200,−2)	0.07	9.99 ± 0.06

and

Table 8. Performance comparison of different algorithms on Weather Izmir.

Algorithms	R = 0%			R = 10%			R = 20%			R = 30%
	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)	Parameters	Time (s)	RMSE Mean ± SD (×10−2)
BLS	(6,2,160,−3)	0.01	1.93 ± 0.02	(8,6,180,2)	0.02	5.51 ± 0.07	(6,18,200,3)	0.04	9.81 ± 0.07	(4,14,190,4)	0.03	12.35 ± 0.63
FBLS	(20,9,87,−4)	0.17	1.93 ± 0.01	(3,10,2,−1)	0.10	5.49 ± 0.08	(8,7,3,3)	0.10	9.28 ± 0.31	(3,8,1,3)	0.08	9.59 ± 0.45
WBLS	(130,−3)	0.03	1.91 ± 0.03	(140,−2)	0.04	1.94 ± 0.01	(130,−2)	0.04	2.14 ± 0.02	(80,−2)	0.02	2.35 ± 0.04
Huber-RCBLS	(8,4,150,−3)	0.06	1.92 ± 0.03	(4,8,200,−3)	0.09	1.94 ± 0.02	(6,2,130,−3)	0.04	2.10 ± 0.02	(8,4,140,−2)	0.06	2.33 ± 0.05
Bisquare-RCBLS	(6,2,100,−4)	0.03	1.93 ± 0.02	(8,4,150,−3)	0.06	1.94 ± 0.03	(6,4,200,−3)	0.08	1.94 ± 0.03	(8,2,140,−3)	0.05	1.94 ± 0.03
Cauchy-RCBLS	(6,4,200,−3)	0.08	1.93 ± 0.03	(8,4,150,−3)	0.06	1.92 ± 0.03	(8,2,160,−3)	0.06	1.96 ± 0.02	(8,2,160,−3)	0.06	1.96 ± 0.03

report the experimental results of all comparison algorithms on six regression datasets with different outlier levels, including the optimal parameters, mean of training time, mean and standard deviation (SD) of testing RMSE. For the Parameters column, it corresponds to (p, n, q, λ) for BLS, Huber-RCBLS, Bisquare-RCBLS and Cauchy-RCBLS, (q, λ) for WBLS, (N_r, N_t, N_e, λ) for FBLS, in which the last value in parentheses denotes the power of the regularization parameter of the corresponding algorithm. That is, the real regularization parameter is ${10}^{\lambda }$ (the same below). For the RMSE column, its best mean results for each dataset are shown in bold. The following results can be obtained from Table 3, Table 4, Table 5, Table 6, Table 7 and Table 8:

(1)

When the training set does not contain outliers, i.e., R = 0%, the performances of all the six algorithms are basically equivalent. Especially on the Bodyfat, Mortgage and Strike, our RCBLS even achieves the smallest testing RMSE. This indicates that the proposed robust learning model is also suitable for clean data modeling in non-outlier environments.

(2)

When the training set contains outliers, the testing RMSE of BLS and FBLS increase rapidly with the increment of the outlier level, which indicates that these two algorithms are very sensitive to outliers. By contrast, as the outlier level increases, the testing RMSE of WBLS and the three RCBLS algorithms proposed in this paper only increase slightly, or even remain basically unchanged in some cases, showing their stronger resistance to outliers. But due to the abandonment of the feature mapping process, the performance of WBLS is not stable. For example, on the Bodyfat dataset, the testing RMSE of WBLS is much worse than our RCBLS. In addition, for all datasets at different outlier levels, except for the Abalone dataset under R = 10%, the optimal testing RMSE are all obtained by our RCBLS. From the above results, it can be concluded that our RCBLS performs best among all comparison algorithms in outlier environments.

(3)

By comparing the three proposed RCBLS algorithms, Huber-RCBLS, Bisquare-RCBLS and Cauchy-RCBLS, which incorporate different M-estimator functions, we can see that their learning performances are basically the same in most cases, and they can obtain the optimal results in different testing cases, respectively.

(4)

In terms of training time, due to the iterative optimization calculations in the learning process, our RCBLS algorithms take a little more training time than the original BLS, but their training times are usually only a few tenths of a second and still maintain a high learning efficiency.

To sum up, compared with other algorithms, our RCBLS can provide comparable performance in non-outlier environments and much better results in outlier environments without sacrificing much learning efficiency, showing good effectiveness and practicability for regression problems.

5.2. Robustness Evaluation on Time Series Prediction

In this section, four classical chaotic time series, Henon, Logistic, Lorenz and Rossler, are utilized to further verify the robustness of BLS, FBLS, WBLS and our RCBLS. The length of each time series is 2000, of which 1500 samples are used for training and 500 samples are used for prediction. For robustness evaluation, some outliers are added to the training set of each time series to replace the normal samples in order to construct the contaminated time series, while the testing set remains outlier-free. The manually added outliers are randomly generated between the minimum and maximum values of the corresponding training set.

Phase space reconstruction is a basic step in time series modeling and prediction, which is mainly used to reconstruct the input space and output space from the original time series. In our experiments, the phase space reconstruction parameters of each time series, namely, the embedding dimension and time delay, are empirically set as follows: four and one for Henon and Logistic, nine and one for Lorenz, one and five for Rossler, respectively. It should be noted that, different from the regression datasets in Section 5.1 where only the target output of the training samples contains outliers, in the reconstructed time series here, both the input and output of the training samples contain outliers. Apart from that, the experimental settings of each algorithm in this part are exactly the same as those in Section 5.1.

Table 9. Performance comparison of different algorithms on contaminated time series.

Time Series	Algorithms	Parameters	Time (s)	RMSE Mean ± SD
Henon	BLS	(8,2,160,−5)	0.01	1.78 ± 0.06 (×10−2)
	FBLS	(20,10,18,−3)	0.24	1.83 ± 0.04 (×10−2)
	WBLS	(200,−10)	0.10	1.67 ± 0.48 (×10−4)
	Huber-RCBLS	(4,8,200,−9)	0.13	1.16 ± 0.43 (×10−4)
	Bisquare-RCBLS	(10,4,200,−9)	0.13	2.95 ± 0.87 (×10−4)
	Cauchy-RCBLS	(18,2,200,−9)	0.13	1.59 ± 0.49 (×10−4)
Logistic	BLS	(4,2,130,−7)	0.01	8.35 ± 0.39 (×10−3)
	FBLS	(16,10,4,−6)	0.19	8.16 ± 0.15 (×10−3)
	WBLS	(200,−10)	0.10	2.86 ± 0.80 (×10−5)
	Huber-RCBLS	(18,2,200,−10)	0.12	2.88 ± 0.52 (×10−6)
	Bisquare-RCBLS	(6,10,190,−10)	0.15	1.24 ± 0.67 (×10−5)
	Cauchy-RCBLS	(6,10,200,−10)	0.15	4.05 ± 1.42 (×10−6)
Lorenz	BLS	(6,2,200,−10)	0.01	1.86 ± 0.11 (×10−1)
	FBLS	(3,10,100,−8)	0.16	1.94 ± 0.19 (×10−1)
	WBLS	(150,0)	0.07	9.03 ± 1.14 (×10−2)
	Huber-RCBLS	(8,2,200,−7)	0.11	1.47 ± 0.41 (×10−2)
	Bisquare-RCBLS	(2,12,60,−8)	0.05	3.82 ± 2.17 (×10−3)
	Cauchy-RCBLS	(8,2,200,−7)	0.11	9.86 ± 3.31 (×10−3)
Rossler	BLS	(10,18,10,0)	0.04	6.46 ± 1.56 (×10−3)
	FBLS	(1,9,98,−6)	0.04	5.23 ± 0.36 (×10−3)
	WBLS	(10,−6)	0.01	2.65 ± 0.01 (×10−3)
	Huber-RCBLS	(12,20,10,−3)	0.13	2.66 ± 0.02 (×10−3)
	Bisquare-RCBLS	(12,20,10,−3)	0.14	2.66 ± 0.03 (×10−3)
	Cauchy-RCBLS	(12,20,10,−3)	0.14	2.67 ± 0.03 (×10−3)

shows the parameter settings and comparison results of different algorithms on the polluted time series. As can be seen from Table 9, for all four chaotic time series contaminated by outliers, the proposed RCBLS algorithms can always obtain the best or approximate best prediction results, in comparison with other competitive algorithms, showing stronger anti-outlier ability and robustness. In addition, the experimental results demonstrate that our RCBLS algorithms are also applicable and effective when both the input and output of the training samples contain outliers.

5.3. Robustness Evaluation on Image Classification

In addition to regression and time series prediction, we further evaluate the effectiveness of our method for image classification problem in this section. The used benchmark here is the popular NORB dataset, which consists of 48,600 images of 50 different 3D toy objects belonging to five distinct categories: cars, trucks, airplanes, humans, and animals. Among them, half of the images are selected for training and testing, respectively. To test the robustness of RCBLS, the learning with corrupted labels is considered in this example. Following [58], we manually corrupt the original clean dataset by a noise transition matrix Q, where $Q_i{}_j=Pr\left(\tilde{y}=j|i=i\right)$ means that noisy label $\tilde{y}$ is flipped from clean label y. Specifically, a typical asymmetric structure called “pair flipping” is applied to construct Q:

$$Q=\left[\begin{array}{ccccc}1-R& R& 0& 0& 0\\ 0& 1-R& R& 0& 0\\ 0& 0& 1-R& R& 0\\ 0& 0& 0& 1-R& R\\ R& 0& 0& 0& 1-R\end{array}\right]$$

(21)

where R is the outlier level to control the ratio of noisy labels. Here, R is selected from {30%, 40%, 45%} to test the performance of our RCBLS in extremely noisy environments.

Since WBLS is stinted on single-output regression tasks and cannot handle multi-classification problems, only the original BLS and FBLS are used here as comparison algorithms. Referring to [1], the common model parameters for BLS and RCBLS are set the same as (p, n, q) = (100, 10, 5000), and the regularization parameter λ is selected by the grid search method. For FBLS, in order to be fair, we first set the number of enhancement nodes to the same 5000, and the other parameters of the algorithm are determined by the grid search method. Moreover, since the original machine configuration is not capable of performing such a large-scale image classification task, all the algorithms are implemented on a workstation equipped with a 3.7 GHz CPU and 64 GB RAM.

Table 10. Performance comparison of BLS, FBLS and RCBLS on Norb with corrupted labels.

Algorithms	R = 30%			R = 40%			R = 45%
	Parameters	Time (s)	Accuracy (%) Mean ± SD	Parameters	Time (s)	Accuracy (%) Mean ± SD	Parameters	Time (s)	Accuracy (%) Mean ± SD
BLS	(100,10,5000,−2)	11.09	86.00 ± 0.30	(100,10,5000,−1)	11.16	76.36 ± 0.67	(100,10,5000,−1)	11.18	67.97 ± 0.65
FBLS	(500,20,5000,2)	2401.71	83.23 ± 0.17	(500,15,5000,2)	1797.01	77.55 ± 0.18	(500,15,5000,3)	1794.59	68.38 ± 0.31
Huber-RCBLS	(100,10,5000,−2)	205.86	89.07 ± 0.37	(100,10,5000,−1)	202.14	84.76 ± 0.77	(100,10,5000,−1)	201.21	77.33 ± 0.77
Bisquare-RCBLS	(100,10,5000,−3)	201.28	89.19 ± 0.31	(100,10,5000,−1)	198.93	84.41 ± 0.68	(100,10,5000,−1)	201.87	78.45 ± 0.63
Cauchy-RCBLS	(100,10,5000,−3)	203.85	89.60 ± 0.42	(100,10,5000,−1)	203.10	83.98 ± 1.09	(100,10,5000,−1)	201.51	77.06 ± 1.44

reports the comparison results of BLS, FBLS and RCBLS on Norb dataset at different outlier levels. As can be seen from Table 10, though our RCBLS algorithms require more training time in comparison with BLS, they are still more efficient than FBLS. In addition, under different outlier levels, our three RCBLS algorithms can always obtain significantly higher recognition accuracy than BLS and FBLS, and the higher the outlier level, the more obvious the advantage. For example, when R = 45%, the testing accuracy of Bisquare-RCBLS is 10.48% higher than the standard BLS and 10.07% higher than the FBLS. This experiment proves that the proposed RCBLS algorithms are also effective for classification problems with corrupted labels.

In fact, for the datasets used in the above three experiments, except for the time series datasets which are artificially generated, the regression datasets and the classification dataset are all datasets from the real world, and our method has achieved the expected results in all experiments. Therefore, it is reasonable to conclude that our method can function well and effectively in real world solutions.

5.4. Compactness Comparison

In this section we take Norb as an example for compactness comparison of our model structure. To better demonstrate the benefits of using ${\mathcal{l}}_{2,1}$-norm regularization in our RCBLS, the original BLS, with ${\mathcal{l}}_2$-norm regularization, is chosen as a typical representative for comparison. In this experiment, the parameter settings of BLS and RCBLS are exactly the same as in Table 10. That is, there are 1000 feature nodes and 5000 enhancement nodes in the networks. Accordingly, the size of the output weights $\mathbf{W}$ to be solved in both BLS and RCBLS is 6000 × 5, here 5 denotes the number of classes in Norb. After training, we obtain the $\mathbf{W}$, and all weights under 10⁻³ in absolute value are set to 0.

Table 11. Compactness comparison of BLS and RCBLS on Norb.

Algorithms	R = 30%		R = 40%		R = 45%
	$\mathit{\lambda }$	Sparsity	$\mathit{\lambda }$	Sparsity	$\mathit{\lambda }$	Sparsity
BLS	−2	0%	−1	0%	−1	0%
Huber-RCBLS	−2	28.87%	−1	75.08%	−1	74.52%
Bisquare-RCBLS	−3	11.67%	−1	72.53%	−1	73.08%
Cauchy-RCBLS	−3	7.17%	−1	72.12%	−1	75.75%

shows the model parameters and the corresponding compactness results of BLS and our three RCBLS on the Norb dataset. Since the network structure parameters (p, n, q) = (100, 10, 5000) are the same for all models, only the regularization parameters λ are given for brevity. In addition, here, we use Sparsity as a metric to measure the structural compactness of each model, which is measured as the proportion of all-zero rows in $\mathbf{W}$, or the percentage of the hidden nodes whose output weights are all zero with respect to the total number of hidden nodes in the network. From Table 11, we can draw the following two key conclusions. Firstly, our RCBLS model is more likely to generate sparse and compact networks than the original BLS. Since BLS adopts a non-sparse ${\mathcal{l}}_2$-norm regularization in the objective function for optimization, the obtained output weights are very dense, with a sparsity of 0% at different outlier levels. In contrast, by employing the ${\mathcal{l}}_{2,1}$-norm regularization, our RCBLS tends to produce row-sparse output weights, in which many row vectors are all zero. For example, when λ = 10⁻¹, more than 70% of the output weight vectors in RCBLS are zero, which implies that most of the hidden nodes and their output connections are useless and can be removed from the network, thus greatly improving the compactness of the network structure. Secondly, the regularization parameter λ has great influence on the sparsity (compactness) of RCBLS. Specifically, the larger the λ, the higher the sparsity and the more compact the model.

Further, we visualize the learned output weights matrices $\mathbf{W}$ of BLS and RCBLS on the Norb dataset in Figure 4 for a more intuitive comparison of their sparsity and compactness. Since our three RCBLS all have similar sparsity behavior, here we just take Huber-RCBLS as a representative for demonstration. As previously mentioned, the size of $\mathbf{W}$ in both BLS and Huber-RCBLS is 6000 × 5, which is too large to display clearly. So, for the sake of clarity, only a portion of $\mathbf{W}$ from row 5901 to row 5950 is plotted in the figure, where the horizontal axis represents the row index and the vertical axis represents the corresponding weight vector (or row vector). As shown in Figure 4b, except for the three weight vectors at indexes 5909, 5919 and 5931, all other weight vectors of Huber-RCBLS are all zero, which is the so-called “row-sparsity”. This means that the corresponding hidden nodes with zero weights can be eliminated directly and then a more compact network can be achieved. As a contrast, Figure 4a shows that the weight vector of BLS is not sparse, and its values are scattered far away from zero, which is not applicable for model reduction.

5.5. Convergence Demonstration

As described in Algorithm 1, several iterative calculations are required in RCBLS for optimizing the output weights. In order to visually illustrate the convergence behavior of RCBLS in the iterative learning process, we take the six regression datasets with 20% outlier level as examples and provide the corresponding convergence graphs, shown in Figure 5, where the horizontal axis represents the number of iterations and the vertical axis represents the corresponding testing RMSE. We can observe from the figures that on all the six datasets, the testing RMSE of the three RCBLS algorithms decrease rapidly during the first few iterations and then converge to a stable value after about 10 iterations. This proves the fast convergence speed and stable convergence result of the proposed method, and guarantees its efficiency and effectiveness in practical implementation.

Compared with the original BLS, only one extra parameter is added in RCBLS, namely the maximum number of iterations $t_{\max }$. According to the above convergence results, the learning performance of RCBLS is insensitive to $t_{\max }$. In other words, $t_{\max }$ can be chosen in a relatively loose range without affecting the learning performance of RCBLS. For example, for all the experimental datasets in this paper, $t_{\max }$ in RCBLS is uniformly set to 20, and strong testing results are obtained. In fact, if we want to further reduce the training time of RCBLS, it is also feasible to set $t_{\max }$ to a smaller value. In summary, the newly added parameter $t_{\max }$ does not impair the usability of the proposed method.

6. Conclusions and Future Work

In this paper, a novel RCBLS model is proposed to improve the robustness and compactness of BLS. Compared with the original BLS model, the proposed RCBLS has two major innovations in terms of robust modeling and structure simplification. Firstly, our RCBLS uses the M-estimator-based robust learning framework instead of the traditional LS criterion as the loss function, in order to alleviate the negative impacts of outlier samples on the model and thus improve its robustness in the presence of outliers. Secondly, our RCBLS adopts the sparsity-promoted ${\mathcal{l}}_{2,1}$-norm regularization to replace the common ${\mathcal{l}}_2$-norm regularization for model reduction. With the assistance of the row sparsity of the ${\mathcal{l}}_{2,1}$-norm regularization, redundant hidden nodes in the network can be effectively distinguished and eliminated, thus generating a more compact network. Due to the introduction of M-estimator loss and ${\mathcal{l}}_{2,1}$-norm regularization in the objective function, the proposed model cannot be solved analytically, then an efficient iterative method based on augmented Lagrangian multiplier is derived to optimize it. In addition, solid theoretical analyses and diverse comparative experiments have confirmed the effectiveness and superiority of the proposed RCBLS model.

In contrast to existing robust BLS models which are designed only for a single type of learning task such as regression [48,49] or classification [51,52], and (or) require careful tuning of additional control parameters to achieve satisfactory performance [49,50], our RCBLS is a more general robust learning framework that can be widely applied to different situations, such as single/multiple output regression and binary/multi-class classification problems, without complicated parameter tuning. In addition, our RCBLS is equipped with a sparse and compact network structure, which is not available in other models. These distinct advantages guarantee that RCBLS has a good application prospect.

In future work, we plan to expand and apply the RCBLS to other practical fields, such as business performance evaluation and risk analysis [59,60], to further explore its application potential in real-world applications. In addition, we attempt to develop incremental versions of RCBLS to meet the needs of online or incremental learning.