A Novel Meta-ELM with Orthogonal Constraints for Regression Problems

Abstract

ELM is an innovative learning algorithm that minimizes output error by only finding optimal output weights. Meta-learning is composed of base ELMs and exhibits good generalization. To improve its performance further by introducing orthogonal constraints into the base ELMs and “top” ELM, we propose a novel Meta-ELM with orthogonal constraints (Meta-QEC-ELM). Because of the particularity of the Meta-ELM, its orthogonal constraint problem is the quadratic equality constraint problem—that is, a one-column Procrustes problem—and it can preserve much more information from feature space to output subspace. The experimental results show that the Meta-QEC-ELM is both effective and feasible.

1. Introduction

ELM (Extreme Learning Machine) [1,2] is an efficient learning algorithm for single hidden-layer feed-forward neural networks. Its main contribution is that its internal parameters are assigned randomly without the need for tuning and updating, thus alleviating the burden of parameter tuning associated with widely used traditional optimization algorithms [3] and saving training time.

In recent years, a substantial amount of work around ELMs has been carried out in order to meet certain requirements [3,4,5,6]. Some research mainly focuses on optimizing and improving the ELM itself. An incremental ELM [7,8,9,10,11] controls the number of hidden-layer neurons by adding them one by one, while a regularized ELM [12,13,14] can prune the model structure and maintain stability by controlling the trade-off between the penalty term and training error. A kernel ELM [15,16,17] introduces the kernel function to address the situation in which the feature mapping functions of hidden-layer neurons are unknown to users [18]. Another important development in ELM research is the orthogonality-based ELM. This method imposes orthogonal constraints on the transformation matrix, and generates orthogonal basis functions to preserve local structure information and avoid trivial problems [19,20,21]. To further improve performance, an orthogonal constraint is introduced into the I-ELM. By incorporating Gram–Schmidt orthogonalization, it efficiently reduces redundant hidden-layer neurons, making the neural network more compact; moreover, the OI-ELM has a much faster convergence rate and better generalization [22,23].

Some studies have effectively combined multiple independent models and, based on statistical results, found that the variance of the average values of multiple independent models is less than that of individual models, thus demonstrating that combination models outperform single ones [24,25]. Heeswijk et al. proposed a parallel combination model [26] that trains each ELM independently and determines output weights based on leave-one-out cross-validation. Liao and Feng [27] proposed a new model incorporating ELMs, called a Meta-ELM, which is considered a “top” ELM whose hidden nodes are base ELMs. After theoretical analysis and experimental verification, it was found to exhibit efficiently decreased computational cost and improved performance. Zou et al. [28] introduced an error feedback I-ELM with meta-learning as the base-learner, proposing an improved Meta-ELM, which includes two main stages: it obtains compact neural networks and efficiently avoids the overfitting problem. Xu et al. [29] used the basic MAML framework and integrated the characteristics of ELMs and MAML to create an MLELM method. During learning and training, the MLELM learned information from multiple tasks and determined the initial parameters for the ELM, enhancing the capability to handle limited samples. Song et al. [30] considered the time characteristics in time series data and proposed the weighted Meta-ELM, or WMeta-ELM. Different weights were assigned to each base ELM to guide the model to learn different information, with the results showing that it is efficient and workable.

Motivated by these works and to further explore the spatial characteristics of data and alleviate the overfitting problem, in this study, a novel meta-learning model is proposed incorporating a QEC-ELM as the base-learner, called a Meta-QEC-ELM. Different from an OELM, the output weight in the model is a column vector; thus, the traditional optimal problem with orthogonal constraints is transformed to one with a quadratic equality constraint, i.e., a one-column Procrustes problem. To project the vector itself onto the one-dimensional subspace, it efficiently avoids locally quadratic convergence [31]. The remainder of this paper is organized as follows. Section 2 briefly reviews the basic ELM and Meta-ELM. Section 3 describes the Meta-QEC-ELM in detail. Section 4 briefly analyzes the convergence and complexity of the proposed model. In Section 5, experiments are conducted to analyze the performance of the Meta-QEC-ELM.

The main contributions of the study include the following: (1) reducing computational complexity by transforming the optimal problem to a quadratic equality constraint problem; and (2) preserving more data information by introducing orthogonal constraints.

2. Related Theory

2.1. ELM

Given $N$ distinct samples ${\left(x_i,y_i\right)}_{i=1}^N$, $x_i\in R^n$, $y_i\in R$, $x_i$ is the input and $y_i$ is the corresponding expected output. SLFN with $L$ hidden layer neurons and activation function $\mathrm{g}\left(\cdot \right)$ is modeled by ELM as

$$f\left(x_j\right)=\sum _{i=1}^L{\beta }_i\mathrm{g}\left({\omega }_i,x_j,b_i\right)=y_j, j\in \left[1,N\right].$$

(1)

where ${\omega }_i$ is input weight, which connects the input layer and the ith hidden-layer neuron; $b_i$ is the basis of the ith hidden-layer neuron; and ${\beta }_i$ is the output weight, which connects $i\mathrm{t}\mathrm{h}$ hidden-layer neuron and output layer.

Equation (1) can be rewritten as follows:

$$H\beta =\mathrm{Y},$$

(2)

in which $H$ is the output matrix of the hidden-layer neurons, $\beta ={\left[{\beta }_1,\dots ,{\beta }_L\right]}^T$, and $Y={\left[y_1{y}_2{\dots y}_N\right]}^T$.

$$H={\left[\begin{array}{ccc}\mathrm{g}\left({\omega }_1,x_1,b_1\right)& \dots & \mathrm{g}\left({\omega }_L,x_1,b_L\right)\\ ⋮& \ddots & ⋮\\ \mathrm{g}\left({\omega }_1,x_N,b_1\right)& \dots & \mathrm{g}\left({\omega }_L,x_N,b_L\right)\end{array}\right]}_{N\times L}.$$

(3)

For ELM [1,2], if $L=N$, the ELM can approximate the training sample with zero error; however, this can lead to overfitting. In most applications, $L$ is much less than $N$ so as to avoid overfitting and improve performance. According to the base theory of ELM, the optimal solution is

$$\beta =H^{†}\mathrm{Y},$$

(4)

in which $H^{†}$ is the Moore–Penrose inverse of $H$, and $H^{†}={\left(H^{T}H\right)}^{-1}{H}^T$.

2.2. Meta-ELM

The Meta-ELM is a hybrid model with a hierarchical structure. By configurating it with multiple base ELMs and the meta-learner, it can learn information generated by multiple base ELMs [24,27,32]. Each of the base ELMs generates one approximator, and the meta-learner generates the meta-approximator. Finally, its goal is to achieve overall accuracy [32], not to pick the “best” base ELM.

The optimal problem of the Meta-ELM is minimizing the cost function in the following form:

$$J\left(\beta \right)=\sum _{i=1}^N{\left(\sum _{j=1}^L{\beta }_j\left(x_i\right){ELM}_j\left(x_i\right)-y_i\right),}^2$$

(5)

in which $N$ is the size of the training dataset; $L$ is the number of base ELMs; and ${ELM}_j\left(x_i\right)$ and ${\beta }_j\left(x_i\right)$ are the output and output weight of the jth base ELM, with $x_i$ as input.

The optimal problem of the Meta-ELM is to minimize the cost function $J\left(\beta \right)$ by solving the least-square solution ${\beta }^{\ast }$; that is, $‖H{\beta }^{\ast }-Y‖$ = $min‖H\beta -Y‖$, where $H$ is the output matrix of multiple base ELMs:

$$H=\left[\begin{array}{c}h\left(x_1\right)\\ ⋮\\ h\left(x_N\right)\end{array}\right]={\left[\begin{array}{ccc}{ELM}_1\left(x_1\right)& \dots & {ELM}_L\left(x_1\right)\\ ⋮& \ddots & ⋮\\ {ELM}_1\left(x_N\right)& \dots & {ELM}_L\left(x_N\right)\end{array}\right]}_{N\times L},$$

(6)

and $\mathrm{h}\left(x\right)=\left[\begin{array}{ccc}{ELM}_1\left(x\right)& \dots & {ELM}_L\left(x\right)\end{array}\right]$; thus, the complete output of the Meta-ELM is $f\left(x\right)=\mathrm{h}\left(x\right)\beta$, $\beta =H^{†}Y$.

3. Meta-QEC-ELM

Referring to the idea of the OELM and Meta-ELM, we propose a novel model in this section.

3.1. QEC-ELM

For the Meta-ELM, ${\beta }_L\in R^L$ is a column vector; thus, the optimal problem (5) with orthogonal constraints is not rational. For processing the same performance with an OELM (Orthogonal ELM), ${‖{\beta }_L‖}^2=1$, ${\beta }_L$ is the unit. Thus, a simple ELM with a quadratic equality constraint (abbreviated to QEC-ELM) is proposed:

$$J\left({\beta }_L\right)=\underset{{‖{\beta }_L‖}^2=1}{\mathit{min}}{‖H_L{\beta }_L-Y‖}^2.$$

(7)

The optimal problem (7) is a one-column Procrustes problem that is equivalent to solving the normal form with constraint:

$$\begin{array}{c}\left(H_L^T{H}_L+\lambda I\right){\beta }_L=H_L^{T}Y\\ s.t. {‖{\beta }_L‖}^2=1,\end{array}$$

(8)

with vector ${\beta }_L$ and multiplier $\lambda$. Thus, the optimal problems of (7) or (8) involve solving the following secular equation:

$$J\left(\lambda \right)={‖{\left(H_L^T{H}_L+\lambda I\right)}^{-1}{H}_L^{T}Y‖}^2=1.$$

(9)

It is proven that when ${\lambda }^{\ast }\ge -{\sigma }_{min}^2$ [31,33], the secular equation holds, ${\lambda }^{\ast }$ is the optimal solution, and ${\sigma }_{min}$ is the minimum singular value of $H_L$. When ${\lambda }^{\ast }>-{\sigma }_{min}^2$, the optimal solution ${\beta }_L^{\ast }$ of (7) is unique, but it is difficult to determine ${\lambda }^{\ast }$. If ${\lambda }^{\ast }=-{\sigma }_{min}^2$, there are multiple solutions. However, if $-{\sigma }_{min}^2$ is close to the optimal ${\lambda }^{\ast }$, it leads to local quadratic convergence. Thus, the projection method [31] is used to solve the optimal problem with a quadratic equality constraint. Here, the vector $\beta \left(\lambda \right)={\left(H_L^T{H}_L+{\lambda }^{\ast }I\right)}^{-1}{H}_L^{T}Y$ is projected onto the one-dimensional subspace spanned by $\omega \left(\lambda \right)$. The projection method can be expressed as

$${\lambda }_{k+1}={\lambda }_k-{\displaystyle \frac{{\beta }_K^T{\rho }_k}{{‖{\rho }_k‖}^2+{‖{\beta }_k‖}^{\alpha -2}K}},$$

(10)

where ${\beta }_k={\phi \left({\lambda }_k\right)H}_L^{T}Y$, ${\rho }_k=\phi \left({\lambda }_k\right){\beta }_k$, ${\phi \left({\lambda }_k\right)=\left(H_L^T{H}_L+{\lambda }_{k}I\right)}^{-1}$, $K={‖{\beta }_k‖}^2{‖{\rho }_k‖}^2-{\left({\beta }_K^T{\rho }_k\right)}^2$, and $\alpha$ is the acceleration factor When $\alpha =0.5$, the iterations of the projection method are fewer and the convergence is stronger.

As is known, $k$ exists, which makes ${‖{\beta }_k‖}^2\ge 1$ hold; this implies that ${∆}_k\ge {\left({\beta }_k^T{\rho }_k\right)}^2$, ${\lambda }_{k+1}-{\lambda }_k=\left(\sqrt{{∆}_k}-{\beta }_k^T{\rho }_k\right)/{‖{\rho }_k‖}^2$, and then ${\lambda }_{k+1}-{\lambda }_k\ge 0$, so ${\lambda }_{k+1}\ge {\lambda }_k$. If ${∆}_k\ge 0$, $J\left({\lambda }_{K+1}\right)\ge {\mathrm{\varnothing }}_k\left({\lambda }_{K+1}\right)=1$,

$${\varnothing }_k\left(\lambda \right)={\displaystyle \frac{{‖{\beta }_k‖}^4}{{‖{\beta }_k‖}^2+2{\beta }_k^T{\rho }_k\left(\lambda -{\lambda }_K\right)+{‖{\rho }_k‖}^2{\left(\lambda -{\lambda }_K\right)}^2}}$$

Thus obtaining the inequality ${\lambda }_{k+1}\le {\lambda }^{\ast }$, ${\lambda }_k\le {\lambda }_{k+1}\le {\lambda }^{\ast }$. So, if ${\lambda }_{k+1}\to {\lambda }^{\ast }$, ${\beta }_k$ is closer to the optimal ${\beta }_K^{\ast }$.

3.2. Meta-QEC-ELM Model

Different from the Meta-ELM, the QEC-ELM is introduced to replace the base ELM as the hidden nodes in the proposed model, and the other component, the meta-learner, is also achieved through the base configuration, which consists of the base QEC-ELM. As depicted in Figure 1, it can learn knowledge from the output of the base QEC-ELMs.

To balance the structural and empirical risk based on statistical learning theory, the output weight is introduced into the cost function. Then, the Meta-QEC-ELM can optimize the following function with orthogonal constraints:

$$\begin{array}{l} J\left(\beta \right)={\displaystyle \sum _{i=1}^N}{\left({\displaystyle \sum _{j=1}^L}{\beta }_j\left(x_i\right){QEC\text{-}ELM}_j\left(x_i\right)-y_i\right)}^2+{\displaystyle \sum _{i=1}^N}{\displaystyle \sum _{j=1}^L}{\beta }_j^2\left(x_i\right)\\ s.t.\\ {\beta }^T\beta =1\end{array}$$

(11)

where $N$ is the size of the training dataset, $L$ is the number of base QEC-ELMs in the Meta-QEC-ELM, ${QEC\text{-}ELM}_{\mathrm{j}}\left({\mathrm{x}}_{\mathrm{i}}\right)$ is the output, and ${\beta }_j\left(x_i\right)$ is the weight of the jth base QEC-ELM, which takes $x_i$ as the input, as depicted in Figure 2.

To resolve the minimizing cost function $J\left(\beta \right)$, Equation (1) is rewritten in its general form:

$$J\left(\beta \right)=\underset{{\beta }^T\beta =1}{\mathit{min}}\left({‖QEC\text{-}ELM\left(x\right)\beta \left(x\right)-Y‖}^2+{‖\beta \left(x\right)‖}^2\right)$$

(12)

$$\begin{array}{c}QEC\text{-}ELM\left(x\right)={\left[{QEC\text{-}ELM}_1\left(x\right) {QEC\text{-}ELM}_2\left(x\right) \dots {QEC\text{-}ELM}_L\left(x\right)\right]}^T\\ =\left[\begin{array}{ccc}{QEC\text{-}ELM}_1\left(x_1\right)& \dots & {QEC\text{-}ELM}_L\left(x_1\right)\\ ⋮& \ddots & ⋮\\ {QEC\text{-}ELM}_1\left(x_N\right)& \dots & {QEC\text{-}ELM}_L\left(x_N\right)\end{array}\right]\end{array}$$

(13)

Because ${\beta }^T\beta =1$, $\beta$ is unit vector, so ${‖\beta \left(x\right)‖}^2=\beta ·\beta ={\beta }^T\beta =1$, and Equation (12) is transformed into the following form:

$$J\left(\beta \right)=\underset{{‖\beta \left(x\right)‖}^2=1}{\mathit{min}}\left({‖QEC\text{-}ELM\left(x\right)\beta \left(x\right)-Y‖}^2+1\right)$$

(14)

Finally, resolving the minimizing cost function $J\left(\beta \right)$ is equivalent to

$$J\left(\beta \right)=\underset{{‖\beta \left(x\right)‖}^2=1}{\mathit{min}}{‖QEC\text{-}ELM\left(x\right)\beta \left(x\right)-Y‖}^2$$

(15)

The meta-learner and meta-approximator are basically the same as the base QEC-ELM and QEC-ELM-approximator, only the number of hidden-layer neurons is different. Since resolving $J\left(\beta \right)$ is the same with the QEC-ELM, it will not be detailed here.

Based on the analysis, the proposed method is a two-stage learning algorithm, showing in the Algorithm 1. The first step is to train the base QEC-ELMs with the same training dataset. This is a parallel process used to save more time when training and predicting. The second step is to train the “top” QEC-ELM, whose hidden nodes are the base QEC-ELM.

Algorithm 1. Meta-QEC-ELM
Input: training samples $X=\left\{{\left(x_i,y_i\right)}_{i=1}^N|x_i\in R^n,y_i\in R\right\}$.
set threshold $\epsilon$ and ${\lambda }_0$
Output: // the output weight in the section stage
First Stage:
	S1	Partition the training dataset into $L$ subset $S_i$, the size of $S_i$ is $⌊N/L⌋$. $S=\left\{S_i|S_i\subseteq X,\bigcup S_i=X,S_i\bigcap S_j=\varphi ,i\ne j,i,j=\mathrm{1,2},\dots ,L\right\}$
	S2	For $S_i\in S$
	S3	Each QEC-ELM learn ${\beta }_i$ on $S_i$
			SS1	${\lambda }_k={\lambda }_0$
			SS2	Calculate the ${\beta }_i^k={\left(H_{L^{\prime }}^T{H}_{L^{\prime }}+{\lambda }_{k}I\right)}^{-1}{H}_{L^{\prime }}^{T}Y$ ($H_{L^{\prime }}$ is the output of hidden layer of one based QEC-ELM with $L^{\prime }$ hidden nodes)
			SS3	If $\left|‖{\beta }_i^k‖-1\right|>\epsilon$
				${\rho }_i^k={\left(H_{L^{\prime }}^T{H}_{L^{\prime }}+{\lambda }_{k}I\right)}^{-1}{\beta }_i^k$
				Calculate ${\lambda }_{k+1}$ based on the Equation (11)
				${{\lambda }_k=\lambda }_{k+1}$
				Go to step SS2
				Else
				Go to step S3
				End else
		End For
Second Stage:
	S4	Calculate the output matrix ${QEC\text{-}ELM}_i\left(x\right)$ of each base QEC-ELM
		$H_L={\left[{QEC\text{-}ELM}_1\left(x\right){QEC\text{-}ELM}_2\left(x\right)\dots {QEC\text{-}ELM}_L\left(x\right)\right]}^T$
	S5	${\lambda }_k={\lambda }_0$
	S6	Calculate the ${\beta }_L^k={\left(H_L^T{H}_L+{\lambda }_{k}I\right)}^{-1}{H}_L^{T}Y$ ($L$ is the number of base QEC-ELM)
	S7	If $\left|‖{\beta }_L^k‖-1\right|>\epsilon$
		${\rho }_L^k={\left(H_L^T{H}_L+{\lambda }_{k}I\right)}^{-1}{\beta }_L^k$
		Calculate ${\lambda }_{k+1}$ based on the Equation (11)
		${{\lambda }_k=\lambda }_{k+1}$
		Go to step S6
		End If

4. Performance Analysis

Considering the convergence of the proposed model, $\left\{{\lambda }_k\right\}$ and $\left\{{\beta }_L^k\right\}$ are the sequences of ${\lambda }^{\ast }$ and ${\beta }_L^{\ast }$, respectively, generated during iterating, where $k$ is the number of iterations. Because of the perturbations $H_L$ and $Y$, $\widehat{\lambda }$, and ${\widehat{\beta }}_L$ are the optimal solutions closest to ${\lambda }^{\ast }$ and ${\beta }_L^{\ast }$. Set $k\to \mathrm{\infty }$, ${\lambda }_k\to \widehat{\lambda }$.

Using Equation (8), $\left(H_L^T{H}_L+\widehat{\lambda }I\right){\widehat{\beta }}_L=H_L^{T}Y$, $\left(H_L^T{H}_L+{\lambda }_{k}I\right){\beta }_L^k=H_L^{T}Y$, yields the following:

$$\left(\widehat{\lambda }-{\lambda }_k\right){\beta }_L^k=\left(H_L^T{H}_L+\widehat{\lambda }I\right)\left({\widehat{\beta }}_L-{\beta }_L^k\right).$$

(16)

By multiplying ${\left(H_L^T{H}_L+\widehat{\lambda }I\right)}^{-1}$ by the right and left side of Equation (16), the following is obtained for all $k$:

$$\left({\widehat{\beta }}_L-{\beta }_L^k\right)={\left(\widehat{\lambda }-{\lambda }_k\right)\left(H_L^T{H}_L+\widehat{\lambda }I\right)}^{-1}{\beta }_L^k.$$

(17)

Therefore, $k\to \mathrm{\infty }$, ${\beta }_L^k\to {\widehat{\beta }}_L$, meaning that sequence $\left\{{\beta }_L^k\right\}$ convergences quadratically.

As we know, the role of the ELM in training is to obtain the output weight ${\beta }_L$, that is, to solve the inverse of $H^{T}H+C$. Thus, its complexity is $O\left(L^3\right)$. In most cases, $L\ll N$, so its computational complexity is significantly reduced in comparison with an LS-SVM and PSVM, as they must solve the inverse of the $N\times N$ matrix. Regarding the complexity of the Meta-QEC-ELM, the main complexity is that in order to obtain the optimal solution ${\beta }_L$, in each iteration, it needs to solve the intermediate variables ${\beta }_L^k$ and ${\rho }_L^k$, respectively, which also need to solve the inverse of ${\left(H_L^T{H}_L+{\lambda }_{k}I\right)}^{-1}$. Thus, the complexity of the “top” ELM with the QEC-ELM is $O\left({kL}^3\right)$. In addition, the proposed model runs in parallel to train each of the base QEC-ELMs, which also need to obtain the output weight ${\beta }_i^k$ and solve the inverse of the matrix. Thus, the complexity of each base QEC-ELM is $O\left(k^{\prime }{L^{\prime }}^3\right)$. So, the complexity of the Meta-QEC-ELM is $O\left({kL}^3+k^{\prime }{L^{\prime }}^3\right)$, where $k$ and $k^{\prime }$ are the number of updates to ${\beta }_L^k$ and ${\beta }_i^k$, respectively. In real applications, $k$ is much less than $L$, and $k^{\prime }$ is much less than $L^{\prime }$.

Since $H_L{\beta }_L=Y$, then ${\left(H_L{\beta }_L\right)}^T=Y^T$, and ${\beta }_L^k{H}_L^T\left(x_t\right)=y_t$. The spatial distance between any two points $dis\left(y_t,y_s\right)$, $dis\left(y_t,y_s\right)=‖y_t-y_s‖$ (Euclidean distance). Due to the introduction of the quadratic equality constraint, ${‖\beta \left(x\right)‖}^2=\beta ·\beta ={\beta }^T\beta =1$, we obtain $‖y_t-y_s‖=‖H_L\left(x_t\right)-H_L\left(x_s\right)‖$. As is known, $H_L\left(x_t\right)$ is the point in the feature space of ELM, therefore $‖H_L\left(x_t\right)-H_L\left(x_s\right)‖$ is the distance in the feature space. This means that the proposed Meta-QEC-ELM is superior in preserving the data metric structure from feature space to reduced subspace.

5. Experimental Verification

To verify the performance of the Meta-QEC-ELM proposed in this paper, some tests were carried out to compare it with other algorithms using several benchmark datasets under the Matlab 2023a environment. These algorithms included an ELM, NOELM [19], Ensemble model [34], Meta-ELM [27], Meta-LSTM [35], Improved Meta-ELM [28], and Meta-QEC-ELM. Their activation function is the sigmoid function and the number of hidden-layer neurons was set to three times the input dimension. Some other key parameters (the biases, input weights, etc.) were generated randomly from [−1, 1].

5.1. Function Approximation with Noise Data

In the experiments, the ELM, NOELM, and Meta-QEC-ELM approximate the “SinC” function:

$$y=\left\{\begin{array}{c}{\displaystyle \frac{sinx}{x}}, x\ne 0,\\ 1, x=0. \end{array}\right.$$

(18)

First, 5000 data are generated as a training and a testing dataset $(x,y)$, respectively, in which $x$ is randomly and uniformly distributed at $\left(-10, 10\right)$. Then, to make the experiments more “real”, noise data are generated and added to the training dataset, which are uniformly distributed at $\left(-2, 2\right)$. The training time of the Meta-QEC-ELM consists of two parts: one is the maximum time to train the base QEC-ELMs, and the other is the time to train the “top” ELM. For greater accuracy, 30 trials were conducted.

Table 1. Comparative analysis for approximation function “SinC”.

Method	Training/s	Test/RMSE	Base Nodes	Hidden Nodes
ELM	0.3125	0.00425	1	20
QEC-ELM	0.3251	0.00417	1	20
Meta-QEC-ELM	0.3183	0.00382	16	14

presents the performance comparison for the approximation function “SinC”. The RMSE of the ELM is 0.00425, and its training time is 0.3125 s. The QEC-ELM algorithm obtains a performance close to that of the ELM. The Meta-QEC-ELM is a more complex algorithm than the ELM and obtains better testing accuracy, but does not take much more time to train.

Figure 3, Figure 4 and Figure 5 show the approximation results for the algorithms ELM, QEC-ELM, and Meta-QEC-ELM, respectively. The green curve is the expected result and the red curve is the actual result, which are approximately symmetrical. All of the algorithms can approximate function “SinC” accurately, but the performance of the Meta-QEC-ELM is better.

5.2. Benchmark Regression

To verify the performance of the Meta-QEC-ELM on the regression problems in comparison with the NOELM, Ensemble model, Meta-ELM, Meta-LSTM, and Improved Meta-ELM, seven benchmark datasets were gathered from the UCI repository [36], as shown in

Table 2. Regression problem dataset.

Datasets	Train	Test	Attributes
Auto price	80	79	14
Breast cancer	100	94	32
Buston housing	250	256	13
Auto-MPG	320	78	8
Abalone	2000	1177	8
California housing	8000	12,640	8
Census house (8L)	10,000	12,784	8

. Due to potential problems such as noise and missing data in these datasets, it must be preprocessed, including identifying abnormal data, deleting extreme data, and filling in missing data. For each dataset, 30 trials were conducted for all of the algorithms.

For the Meta-ELM and Meta-QEC-ELM, both the base ELM and the “top” ELM need to be trained, meaning that their computing complexity is relatively higher. Theoretically, the training time is significantly longer, especially for smaller datasets. The shortest training time is 0.0228 for the Meta-QEC-ELM on the dataset “Auto price”. Due to the small dataset and relatively small node number, its comprehensive complexity is less than others. However, as the size of the dataset increases, the neural network structure becomes more complex with an increasing number of hidden nodes, and the number of base ELMs does increase significantly. This leads to a shorter training time than for the NOELM, Ensemble model, and Meta-LSTM. Its longest training time is about 0.62, but this has been shortened by more than 5%, as shown in

Table 3. Network complexity and training time of different algorithms.

Datasets	Algorithm	Base ELM	Hidden Nodes	Training Time
Auto price	NOELM	1	42	0.0241
	Ensemble model	/	/	0.0317
	Meta-LSTM	/	/	0.0283
	Meta-ELM	3	29	0.0254
	Meta-QEC-ELM	3	29	0.0228
Breast cancer	NOELM	1	96	0.7568
	Ensemble model	/	/	0.8152
	Meta-LSTM	/	/	0.7723
	Meta-ELM	5	57	0.7073
	Meta-QEC-ELM	5	57	0.7159
Buston housing	NOELM	1	39	0.0336
	Ensemble model	/	/	0.0453
	Meta-LSTM	/	/	0.0398
	Meta-ELM	2	25	0.0354
	Meta-QEC-ELM	2	25	0.0318
Auto-MPG	NOELM	1	24	0.6544
	Ensemble model	/	/	0.7529
	Meta-LSTM	/	/	0.6479
	Meta-ELM	7	16	0.6089
	Meta-QEC-ELM	7	16	0.6191
Abalone	NOELM	1	56	0.3582
	Ensemble model	/	/	0.4538
	Meta-LSTM	/	/	0.3937
	Meta-ELM	6	33	0.3275
	Meta-QEC-ELM	6	33	0.3389
California housing	NOELM	1	330	1.2427
	Ensemble model	/	/	1.5825
	Meta-LSTM	/	/	1.4853
	Meta-ELM	11	139	1.3829
	Meta-QEC-ELM	11	139	1.1942
Census house (8L)	NOELM	1	194	0.5679
	Ensemble model	/	/	0.7183
	Meta-LSTM	/	/	0.6749
	Meta-ELM	9	69	0.5986
	Meta-QEC-ELM	9	69	0.5372

. Because the quadratic equality constraint has been introduced, the complexity of the Meta-QEC-ELM is a little larger than the Meta-ELM, but its training time does not increase significantly. For some datasets, the training time is still less than for the Meta-ELM: the maximum difference is only 0.1887 on the dataset “California housing” because of its more complex network structure, with 11 base ELMs and up to 139 nodes in each base ELM. As observed from

Table 4. Comparative analysis of different methods.

Algorithm	NOELM		Ensemble Model		Meta-LSTM		Meta-ELM		Meta-QEC-ELM
	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test
Auto price	0.1056	0.1161	0.0978	0.1040	0.0967	0.1024	0.0989	0.1056	0.0945	0.0991
Breast cancer	0.107	0.1245	0.0994	0.1104	0.0974	0.1089	0.1014	0.1119	0.0933	0.1058
Buston housing	0.1243	0.1379	0.1139	0.1201	0.1119	0.1190	0.1159	0.1212	0.1078	0.1168
Auto-MPG	0.1159	0.1279	0.1042	0.1096	0.1058	0.1109	0.1026	0.1084	0.1089	0.1133
Abalone	0.1077	0.1082	0.1000	0.0959	0.0983	0.0945	0.1018	0.0973	0.0947	0.0916
California housing	0.1992	0.2003	0.1840	0.1776	0.1810	0.1750	0.1869	0.1802	0.1751	0.1698
Census house (8L)	0.1173	0.1063	0.1078	0.0971	0.1063	0.0971	0.1096	0.0974	0.1029	0.0968

, the RMSE of the Meta-QEC-ELM is similar to or less than that of the Meta-ELM in all datasets, except on “Auto-MPG”, where its RMSE is a little larger, but only by less than 3%. The maximum testing difference is about 0.006 on the dataset “Auto price”, an improvement of more than 6%, and it preserves more data information. The Meta-LSTM needs to optimize the LSTM, so its training time is relatively long, and the longest time has increased by about 25%. Moreover, the Ensemble model involves more factors and needs more iterative training, so its training time is longer. The average training time has increased by 30%, as shown in Table 3.

In summary, compared with the NOELM, Ensemble Model, Meta-LSTM, and Meta-ELM, our proposed Meta-QEC-ELM model achieves better results, although the network structure is more complex.

6. Conclusions

In this paper, learning from the basic idea of the NOELM, we introduced orthogonal constraints into the Meta-ELM and proposed the novel Meta-QEC-ELM. Different from the original and improved Meta-ELM, orthogonal constraints are introduced by the base ELM and “top” ELM in the Meta-QEC-ELM, respectively. Because of the particularity of the Meta-ELM, the Meta-QEC-ELM transforms the orthogonal Procrustes problem into a quadratic equality constraint problem based on single vectors—that is, the one-column Procrustes problem—and also preserves more data information. Compared with the ELM, NOELM, Meta-LSTM, and Meta-ELM, the Meta-QEC-ELM can achieve much better results. It is slightly weaker than Meta-ELM in some points, but the weakness is limited and still completely acceptable. Because of the good characteristics of the Meta-QEC-ELM, it could also be applied to regression problems such as traffic flow prediction. Thus, the node number of the Meta-QEC-ELM should be optimized further to improve its ability to solve regression problems and enhance scalability across diverse problem domains.