Training of an Extreme Learning Machine Autoencoder Based on an Iterative Shrinkage-Thresholding Optimization Algorithm

Abstract

Orthogonal transformations, proper decomposition, and the Moore–Penrose inverse are traditional methods of obtaining the output layer weights for an extreme learning machine autoencoder. However, an increase in the number of hidden neurons causes higher convergence times and computational complexity, whereas the generalization capability is low when the number of neurons is small. One way to address this issue is to use the fast iterative shrinkage-thresholding algorithm (FISTA) to minimize the output weights of the extreme learning machine. In this work, we aim to improve the convergence speed of FISTA by using two fast algorithms of the shrinkage-thresholding class, called greedy FISTA (G-FISTA) and linearly convergent FISTA (LC-FISTA). Our method is an exciting proposal for decision-making involving the resolution of many application problems, especially those requiring longer computational times. In our experiments, we adopt six public datasets that are frequently used in machine learning: MNIST, NORB, CIFAR10, UMist, Caltech256, and Stanford Cars. We apply several metrics to evaluate the performance of our method, and the object of comparison is the FISTA algorithm due to its popularity for neural network training. The experimental results show that G-FISTA and LC-FISTA achieve higher convergence speeds in the autoencoder training process; for example, in the Stanford Cars dataset, G-FISTA and LC-FISTA are faster than FISTA by 48.42% and 47.32%, respectively. Overall, all three algorithms maintain good values of the performance metrics on all databases.

1. Introduction

In pattern recognition systems, efficient methods of feature selection and extraction can reduce the dimensionality problem, thus reducing both the computation time and the memory requirements of the training algorithms [1]. An autoencoder is a feed-forward neural network that builds a compact representation of the input data, and is mainly used for unsupervised learning [2]. It is composed of an encoder and a decoder: the encoder reads the input data and maps them to a lower-dimensionality space, while the decoder reads the compact representation and reconstructs the neural network input. In the same way as for all supervised learning neural networks [3,4], the core aspect of the training of an autoencoder is the backpropagation algorithm. This algorithm iteratively tunes the weights and biases of the neural network by applying the gradient descent method. Autoencoder networks are in great demand in multiple applications in modern society, for example, for dimensionality reduction, image retrieval, denoising, and data augmentation [2].

Several works in the literature have developed feature extraction methods using autoencoders and backpropagation. A comparative study between the performance of a traditional autoencoder and a denoising sparse autoencoder is presented in [5]. Another widely used architecture is the stacked autoencoder, which includes the stacked denoising sparse autoencoder [6] and the symmetric stacked autoencoder with shared weights [7]. Semi-supervised learning algorithms have also been used as autoencoder training algorithms [8,9]. However, the backpropagation training technique limits the generalization ability of the autoencoder, which results in low computational efficiency and the presence of local optima.

An extreme learning machine (ELM) is an emerging training paradigm for neural networks which includes supervised [10], semi-supervised [11], and unsupervised training [12]. The fundamental principle of this architecture is the random assignment of weights and biases in the hidden layer and the analytical determination of the weights of the output layer, in which the least squares method is applied to a system of linear equations [13,14]. The simplicity of the model, the lower numbers of training parameters, the low convergence speed, and its high generalization capacity mean that ELM neural networks have advantages for classification, regression, and clustering problems. Variants such as the kernel-ELM (K-ELM) [15], due to the significant reduction in the number of parameters and the increase in the training speed, and the online sequential ELM [16], due to its capacity to process batches of data, can even deal efficiently with problems involving large volumes of data. In the field of rolling element fault diagnosis, ELM can also solve the problem of weak signals caused by long transmission paths [17]; for example, the novel method presented in [18] can help in frame feature selection for ELM. In computer vision, the researchers of [19] proposed a visual object tracking scheme with promising results.

On the other hand, to handle the problems that local minima can cause, some studies have used heuristic optimization techniques to estimate the network parameters, such as differential evolution, particle swarming (PSO), and genetic algorithms [20]. These heuristic algorithms always search for the global minimum of the objective function and are efficient in several research fields, as is the case for the PSO algorithm that adjusts the parameters of an underactuated surface search model [21]. An improved scheme of this method integrates the PSO with global optimization capability, a 3-Opt algorithm with local search capability, and a fuzzy system with fuzzy reasoning ability [22]. The probabilistic ant colony algorithm is another optimization technique integrated into the architecture of an ELM network [23]. The K-ELM method improves the hyperspectral image classification capacity by coupling the principal component analysis, local binary pattern, and gray wolf optimization algorithm with global search capability [24].

Due to the advantages of ELM over backpropagation, the authors of [25] proposed to train autoencoder networks using an extreme learning machine (ELM-AE). A sparse ELM-AE was presented in [26] by adding the $L_1$-norm in the quadratic term to give a more significant representation of data. Another approach for unlabeled feature extraction consisted of a generalized ELM-AE in which a manifold regularization factor was added to the objective function of the ELM-AE [27]. An unsupervised learning ELM-AE was proposed in [28], inspired by the use of embedding graphs to capture the structure of the input patterns, and was constructed using local Fisher discrimination analysis. A dense connection autoencoder was introduced into multi-layer learning, in which the output features of the previous layers were used by the following layers [29]. A double random hidden layer ELM autoencoder can achieve efficient extraction of features with dimension reduction capability for deep learning [30]. The $L_{21}$-norm regularization has been used for the ELM-AE optimization problem, creating a new unsupervised learning framework that included a disperse representation despite the influence of atypical values and noisy data [31]. In addition, the correntropy-based ELM autoencoder presented in [32] was developed to extract features from noisy input patterns. To lower the number of parameters to be tuned and for the efficient calculation of the pseudo-inverse matrix, an ELM-AE with a kernel function was proposed in [33,34]. From a different perspective, the need for unsupervised batch learning means that an online sequential ELM-AE [35,36] is an exciting approach to solving large-scale applications.

From the works described above, we see that the backpropagation method and the Moore–Penrose inverse are two classical methods of estimating the weights of an autoencoder network. However, using these methods may cause overfitting when the number of unknown variables is larger than the number of training data. In addition, the accuracy is low when the number of hidden neurons is small, and the training time increases significantly when solving higher-dimensional applications. To overcome these drawbacks, researchers have proposed the use of sparse regularization techniques [37], with FISTA being the most widely used algorithm [38]. Although its origin dates back to 2009, this algorithm is still under study, thanks to its convergence speed, accuracy, and straightforward implementation. Several state-of-the-art studies have demonstrated the successful use of this optimization tool in machine learning. For example, in multilayer network design, the minimization problem defined in the autoencoder uses shrinkage-thresholding optimization techniques for dimension reduction and sparse feature representation [26,32,39]. For noisy image and video processing, a mixed scheme was presented in [40], in which sparse coding was adopted with deep learning to solve supervised and unsupervised tasks. A visual dictionary classification scheme was addressed in [41] using a simple (deep) convolutional autoencoder, for which the source of inspiration was sparse optimization and the iterative shrinkage-thresholding algorithm [38]. More recent studies [42,43] have demonstrated this algorithm’s development, continuity, and importance for solving convex minimization problems. In particular, they estimated the output weights of ELM and validated their results on classification problems.

Due to the importance of sparse regularization in the mathematical modeling of many real situations and the ability of shrinkage-thresholding algorithms to minimize these optimization problems, our work aims to study these optimization techniques in the architecture of an ELM-AE since there are few studies in the literature. Specifically, this paper proposes the use of fast algorithms from the shrinkage-thresholding family to improve the convergence speed achieved by FISTA during the training of an ELM-AE. In particular, two representative models of this class are considered: G-FISTA [44] and LC-FISTA [45,46]. To evaluate the efficiency of our scheme, six datasets that are widely used in the validation of ELM algorithms are adopted: MNIST, NORB, CIFAR10, UMist, Caltech256, and Stanford Cars. The main contributions of our article are as follows:

• We present a novel method for computing the output layer weights of the ELM-AE network that avoids the need for the Moore–Penrose inverse. Instead of solving the linear system analytically, we use first-order iterative algorithms from the class of shrinkage-thresholding algorithms to minimize the output weights of the ELM-AE.
• We demonstrate experimentally that the proposed method can effectively improve the computational time of the FISTA algorithm while maintaining its generalization capability. According to the theory presented in [47], the ELM-AE experiences a better performance when it achieves the lowest training error.
• Compared to FISTA, the G-FISTA, and LC-FISTA algorithms show a considerable improvement in the training time of the ELM-AE, for all of the databases, and a very competitive reconstruction capability. Thus, applying this mathematical tool in other contexts would constitute a significant scientific contribution.

The rest of this work is structured as follows: in Section 2, we briefly explain the ELM-AE, FISTA, G-FISTA, and LC-FISTA algorithms. Section 3 describes the method proposed to train the ELM-AEs. The datasets and the results are presented in Section 4. The discussion of results is presented in Section 5. Finally, Section 6 gives the conclusions of this work.

2. ELM-AE and Shrinkage-Thresholding Algorithms

2.1. Extreme Learning Machine Autoencoder

The ELM-AE [25] is trained with an ELM algorithm [10] in an unsupervised manner. It corresponds to a single hidden layer feedforward neural network, in which the outputs match the inputs. The ELM-AE has both a high training speed and a strong generalization ability due to the pseudo-random creation of neurons in the hidden layer and the application of a pseudoinverse matrix to calculate the weights of the output layer. To improve the performance of this method, the authors of [25] assigned random orthogonal weights and biases to the hidden layer. This approach was shown to have good generalization, and to minimize ${\parallel \mathit{H}\mathit{\beta }-\mathit{X}\parallel }_2^2$ and the square norm of the coefficients ${\parallel \mathit{\beta }\parallel }_2^2$. Given a set of N training samples $\left\{{\mathit{x}}_i\right|i\le N\}$, where ${\mathit{x}}_i\in {\mathbb{R}}^m$ are both the input and output data, the output with L hidden neurons can be expressed as follows:

$$f_L\left({\mathit{x}}_j\right)=\sum _{i=1}^L{\mathit{\beta }}_{i}g({\mathit{w}}_i{\mathit{x}}_j+b_i),j=1,\dots N$$

(1)

where ${\mathit{w}}_i$ and $b_i$ represent the pseudorandom weights and biases of the hidden layer, g is an activation function, and ${\mathit{\beta }}_i$ represents the weights of the output layer. The previous overdetermined linear system can be compactly described as follows:

$$\mathit{H}\mathit{\beta }=\mathit{X},$$

(2)

where $\mathit{H}\in {\mathbb{R}}^{N\times L}$ is the hidden layer output matrix, $\mathit{\beta }\in {\mathbb{R}}^{L\times m}$ is the output layer weight matrix, and $\mathit{X}\in {\mathbb{R}}^{N\times m}$ is the input data matrix.

Indeed, $\mathit{\beta }={\mathit{H}}^{†}\mathit{X}$ is the standard solution to the overdetermined problem in (2), where ${\mathit{H}}^{†}={\left({\mathit{H}}^T\mathit{H}\right)}^{-1}{\mathit{H}}^T$ is the Moore–Penrose inverse of matrix $\mathit{H}$. To improve the performance of an ELM-AE, the authors of [25] determined the output weights $\mathit{\beta }$ by means of the equation $\mathit{\beta }={\left({\displaystyle \frac{\mathit{I}}{\lambda }}+{\mathit{H}}^T\mathit{H}\right)}^{-1}{\mathit{H}}^T\mathit{X}$, which is the result of solving the following regularized optimization problem:

$$\mathit{\beta }={\mathrm{argmin}\parallel \mathit{H}\mathit{\beta }-\mathit{X}\parallel }_2^2+\lambda {\parallel \mathit{\beta }\parallel }_2^2,$$

(3)

where $\lambda$ is a tuning regularization parameter.

2.2. The Shrinkage-Thresholding Class of Optimization Algorithms

A large number of applications can be formulated as a convex optimization problem [48], as shown by the following expression:

$$min:h\left(\mathit{\tau }\right)=f\left(\mathit{\tau }\right)+\varphi \left(\mathit{\tau }\right),$$

(4)

where $f,\varphi :\mathit{\tau }\in {\mathbb{R}}^n\to \mathbb{R}$ are two convex functions,$\nabla f$ is $L_f$-Lipschitz continuous and $\varphi$ is a non-differentiable function that cannot be easily minimized in several machine learning applications. The solution to $\varphi$ can be evaluated using the proximal operator [48], as shown in the following equation:

$${\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{x}}_{\rho \varphi }\left(\mathit{z}\right)=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\mathit{\tau }}{min}\frac{1}{2}{\parallel \mathit{\tau }-\mathit{z}\parallel }_2^2+\rho \varphi \left(\mathit{\tau }\right),$$

(5)

where $\rho$ is the stepsize and $\mathit{z}$ is the initial estimate of the solution. As a consequence, the optimization of the function h uses a two-phase method: (i) performance of a forward gradient descent step on the smooth function f; and (ii) performance of the proximal operator or a backward gradient descent step on the non-smooth function $\varphi$.

In the following, a brief description of the FISTA, G-FISTA, and LC-FISTA algorithms is presented. These algorithms can be used to train ELM-AEs.

2.2.1. Fast Iterative Shrinkage Thresholding Algorithm

FISTA [38] is characterized by its convergence speed and high level of efficiency. When minimizing the convex problem in (4), the algorithm has a convergence rate on the order of $\mathcal{O}(1/k^2)$. The sequence of steps of the FISTA algorithm is as follows:

(1)

Update: ${\mathit{\tau }}_k={\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{x}}_{\rho \varphi }({\mathit{z}}_k-\rho \nabla f\left({\mathit{z}}_k\right))$.

(2)

Execute the intermediate step: $t_{k+1}=\frac{1+{(1+4t_k^2)}^{1/2}}{2}$.

(3)

Update: ${\mathit{z}}_{k+1}={\mathit{\tau }}_k+{\gamma }_k({\mathit{\tau }}_k-{\mathit{\tau }}_{k-1})$.

The term $\rho \in (0,\frac{1}{L_f}]$ represents the step size, and ${\gamma }_k=\frac{t_k-1}{t_{k+1}}$ represents the impulse factor. FISTA’s convergence rate depends mainly on its proximal operator and the use of a weighted step ${\mathit{z}}_k$, which is expressed as a combination of both ${\mathit{\tau }}_{k-1}$ and ${\mathit{\tau }}_k$. In addition, the configuration of the Lipschitz constant $L_f$ of the gradient f plays an essential part in the convergence of the algorithm.

2.2.2. Greedy FISTA

In general terms, G-FISTA [44] is a variant of FISTA that is characterized by a higher speed. Given the conditions imposed by G-FISTA, its architecture can restart the FISTA algorithm with the fixed impulse factor ${\gamma }_k=1$. This algorithm updates the solution to the optimization problem in (4), setting $\rho \in [\frac{1}{L_f},\frac{2}{L_f}]$, $\xi <1$ and $S>1$ until the maximum number of iterations is reached. The updating scheme is as follows:

(1)

Update: ${\mathit{\tau }}_k={\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{x}}_{\rho \varphi }({\mathit{z}}_k-\rho \nabla f\left({\mathit{z}}_k\right))$.

(2)

Update: ${\mathit{z}}_{k+1}={\mathit{\tau }}_k+({\mathit{\tau }}_k-{\mathit{\tau }}_{k-1})$.

(3)

Restart: if ${({\mathit{z}}_k-{\mathit{\tau }}_k)}^T({\mathit{\tau }}_k-{\mathit{\tau }}_{k-1})\ge 0$, then ${\mathit{z}}_{k+1}={\mathit{\tau }}_k$.

(4)

Safeguard: if $\parallel {\mathit{\tau }}_k-{\mathit{\tau }}_{k-1}\parallel \ge S\parallel {\mathit{\tau }}_1-{\mathit{\tau }}_0\parallel$, then $\rho =max\{\xi \rho ,\frac{1}{L_f}\}$.

The G-FISTA algorithm improves the convergence rate and the reconstruction capability of FISTA, which uses a self-adaptive restart and adjustment scheme to obtain an additional acceleration and alleviate the oscillation problem in the reconstruction process. Based on the results discussed in [44,49], G-FISTA has efficient performance when $\rho \in [\frac{1}{L_f},\frac{1.3}{L_f}]$. For this reason, all experiments performed in this work were configured with $\rho =1.3/L_f$, $\xi =0.96$, and $S=1.1$.

2.2.3. Linearly Convergent FISTA

LC-FISTA is an accelerated version of FISTA derived from the research in [45,46]. Under specific error conditions, LC-FISTA works with global linear convergence for a large group of real applications, which minimizes the formulation of the convex problem of the equation in (4). The iterative steps of the algorithm are as follows:

(1)

Set both ${\mathit{\tau }}_0={\mathit{z}}_0=\mathbf{0}$, and $\alpha$, $\theta$, $\mu$ y $L_f$.

(2)

Update: ${\mathit{y}}_k=\frac{1}{1+\theta }{\mathit{\tau }}_k+\frac{\theta }{1+\theta }{\mathit{z}}_k$.

(3)

Update: ${\mathit{\tau }}_{k+1}={\mathrm{prox}}_{\rho \varphi }({\mathit{y}}_k-\rho \nabla f\left({\mathit{y}}_k\right))$.

(4)

Update: ${\mathit{z}}_{k+1}=(1-\theta ){\mathit{z}}_k+\theta {\mathit{y}}_k+\alpha ({\mathit{\tau }}_{k+1}-{\mathit{y}}_k)$.

With a mathematical analysis analogous to that presented in [45,46], the sequence ${\left\{{\mathit{\tau }}_k\right\}}_{k\in \mathbb{N}}$ generated by the LC-FISTA scheme exhibits a linear convergence rate, for fixed values of $\alpha ={\left(\frac{L_f}{\mu }\right)}^{1/2}$ and $\theta ={\left(\frac{\mu }{L_f}\right)}^{1/2}$. The practical use of LC-FISTA requires tuning parameters such as the strong convexity parameter $\mu$ for the differentiable function f. In addition, we need to set ${\mathit{\tau }}_0={\mathit{z}}_0$ to obtain convergence rates similar to the accelerated variant of FISTA presented in [50]. It is common to set ${\mathit{x}}_0=0$, since the location of the global minimum of the objective function h is not known.

Remark 1.

The proximal operator ${prox}_{\rho \varphi }\left({\mathit{u}}_k\right)$ is the step that is common to FISTA, G-FISTA, and LCFISTA. This value ${\mathit{\tau }}_{k+1}={prox}_{\rho \varphi }\left({\mathit{u}}_k\right)$ is closer to the global minimum of h compared with the maximum descent step ${\mathit{u}}_k:={\mathit{z}}_k-\rho \nabla f\left({\mathit{z}}_k\right)$. Since $t_k$ causes oscillations in the FISTA scheme when it has a value of one, G-FISTA shortens the interval between two restarts by setting $t_k=1$, which is an essential condition for G-FISTA to converge with fewer iterations. In LC-FISTA, the θ and α momentum terms incorporated in the two additional equations (see (1) and (3) for LC-FISTA) improve both the accuracy of the gradient and the speed of convergence of LC-FISTA via the proximal operator.

3. Proposed Method of Training ELM-AEs Based on G-FISTA and LC-FISTA

ELM-AE has an extremely fast training speed, and good pattern reconstruction capability that is better than backpropagation-based learning. The formulation of this model can be expressed as a convex optimization problem which involves minimizing the weights of the $\mathit{\beta }$ output layer, as shown in the following equation:

$$\underset{\mathit{\beta }}{min}{\parallel \mathit{H}\mathit{\beta }-\mathit{X}\parallel }_u^{{\kappa }_1}+\lambda {\parallel \mathit{\beta }\parallel }_v^{{\kappa }_2},$$

(6)

where ${\kappa }_1,{\kappa }_2>0$, $u,v=0,\frac{1}{2},1,2,\dots$ and $\lambda$ is a control parameter between the training error and the generalization ability.

Several methods exist for computing the solution to (6), such as the Moore–Penrose inverse, orthogonal projection, and proper decomposition [51]. However, in a real situation, the number of independent variables L in the linear system $\mathit{H}\mathit{\beta }=\mathit{X}$ is much larger than the number of data points N, which may cause overfitting. On the other hand, the reconstruction capacity is low whenever the number of hidden neurons L is small. As the training dataset increases, the computational cost of the inverse matrix ${\mathit{H}}^{†}$ increases significantly. Several regularization methods have been introduced in the literature to address the above problems. The two most commonly used classical techniques are ridge regression [52] and sparse regularization [37]. The sparse regularization plays an important role in feature selection, which has shown successful results in machine learning. This study presents an unsupervised feature selection framework, which integrates sparse optimization and signal reconstruction for pattern recognition models. The minimization problem takes the following form:

$$\underset{\mathit{\beta }}{min}{\parallel \mathit{H}\mathit{\beta }-\mathit{X}\parallel }_2^2+\lambda {\left|\mathit{\beta }\right|}_1,$$

(7)

which is a special case of (4), in which we set $f\left(\mathit{\beta }\right)={\parallel \mathit{H}\mathit{\beta }-\mathit{X}\parallel }_2^2$ and $\varphi \left(\mathit{\beta }\right)={\lambda \left|\mathit{\beta }\right|}_1$.

From the description given above, we see that the estimation of $\mathit{\beta }$ and the time required are the main challenges for training the ELM-AE. Our study proposes two novel iterative schemes for autoencoder training, G-FISTA, and LC-FISTA, which reduce the computational time of FISTA maintaining its accuracy, where FISTA is the classical algorithm for training neural networks. This optimization approach is also interesting because it can be extended to other regularization terms, which control the variables selection by calculating a derivative in the weakest sense through convex envelopes. In addition, the closed form of the proximal operator ${\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{x}}_{\rho \varphi }(·)$ of $\varphi$ is computed by the shrinkage operator $\mathrm{s}\mathrm{h}\mathrm{r}\mathrm{i}\mathrm{n}\mathrm{k}(\mathit{z},\lambda \rho )$ [38], whose i-th element is calculated as follows:

$$\mathrm{s}\mathrm{h}\mathrm{r}\mathrm{i}\mathrm{n}\mathrm{k}{(\mathit{z},\lambda \rho )}_i=\mathrm{s}\mathrm{h}\mathrm{r}\mathrm{i}\mathrm{n}\mathrm{k}({\mathit{z}}_i,\lambda \rho )=\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{n}\left(z_i\right)max\left\{\right|z_i|-\lambda \rho ,0\},$$

(8)

where $\mathit{z}$ is the initial estimation of the weight of the $\mathit{\beta }$ output layer.

The speed that can be achieved by the use of G-FISTA and LC-FISTA in the training of the ELM-AE is significantly higher than that obtained using FISTA. In addition, for poorly conditioned matrices, the two algorithms have the same ability as FISTA to reconstruct the input signals. This is realized by the omission of several steps of the gradient descent, and correction of the solution by means of the proximal gradient. Although the iterative scheme used by G-FISTA and LC-FISTA effectively has additional terms compared to FISTA, the computational cost can be similar, since they converge with fewer iterations. The following algorithm presents the steps that are followed in the training of the ELM-AE, where the parameters to be configured are the number of iterations I, the number of neurons L, the stopping criterion P and the regularization parameter $\lambda$. The variables represent features associated with the pixels of the images, ordered according to the columns of the matrix $\mathit{X}$. In the reconstruction and variable selection process, the input data, expected target, network weights, and biases are matrix arrays that facilitate the computation of many vector operations.

4. Experimental Evaluation

This section presents the databases and evaluation metrics selected to evaluate the performance of the proposed method.

4.1. Hardware, Software and Databases

For this work, a server from the cluster of the Laboratory of Technological Research in Pattern Recognition (LITRP) of the Universidad Católica del Maule was used. The server was equipped with two CPUs, an Intel Xeon Gold 6238 252 CPU @ 2.20–4.00 GHz (56 physical cores), 126 GB RAM and a GPU NVIDIA Titan RTX. The algorithms were implemented in the MATLAB R2020a programming language.

To examine the performance of Algorithm 1, we used the CIFAR10, UMist, MNIST, NORB, Stanford Cars and Caltech256 databases, which were obtained from different free online repositories. These datasets are briefly described below.

(1)

MNIST. This dataset contains 70,000 images of handwritten digits from zero to nine. These are grayscale images of size $28\times 28$ pixels which generate a vector of 784 components.

(2)

NORB. This consists of 48,600 grayscale stereo images of 50 toys from five generic classes: cars, four-legged animals, human figures, airplanes, and trucks. The size of the images is $96\times 96$ pixels, and the output is a vector of $96\times 96\times 2$ = 18,432 components.

(3)

CIFAR10. This dataset consists of 60,000 color images distributed into 10 classes: airplanes, birds, automobiles, cats, dogs, frogs, deer, ships, horses, and trucks. The number of images per class is 6000, with 5000 used for training and the remainder for testing. In this experiment, the color images were converted to grayscale and had a size of $32\times 32$ pixels.

(4)

UMist. This contains 575 grayscale images of 20 people, showing individuals in different positions. The images are rectangular, with a size of $92\times 112$ pixels.

(5)

Caltech256. This dataset contains 30,607 color images grouped into 257 categories, each of which contain at least 80 images. For the purposes of this study, the images were converted to grayscale and had a size of $100\times 100$ pixels.

(6)

Stanford Cars (SCars). The Stanford Cars dataset contains 16,185 images of 196 categories of automobiles. For training, the color images were converted to grayscale images of size $96\times 97$ pixels.

Algorithm 1 Training of the ELM-AE with FISTA, G-FISTA, and LC-FISTA.

Require: training set ${\left\{{\mathit{x}}_i\right\}}_{i=1}^N$, maximum number of iterations I, stopping criterion P,       number of neurons L, regularization parameter $\lambda$.   1: Random and orthogonal assignment of weights $\mathit{W}$ and biases $\mathit{b}$ in the hidden layer.   2: Calculation of the matrix $\mathit{H}$ evaluating g in the terms $({\mathit{x}}_i,\mathit{W},\mathit{b})$.   3: while P does not occur do   4: Calculate the optimal value of (4) using the G-FISTA or LC-FISTA algorithms for       $f\left(\mathit{\beta }\right)={\parallel \mathit{H}\mathit{\beta }-\mathit{X}\parallel }_2^2$ and $\varphi \left(\mathit{\beta }\right)={\lambda \left|\mathit{\beta }\right|}_1$.   5: end while

Table 1. Information from the databases to train the ELM-AE.

Dataset	# Training	# Testing	# Features	# Category
UMist	400	175	10,304	20
CIFAR10	50,000	10,000	1024	10
MNIST	60,000	10,000	784	10
NORB	24,300	24,300	18,432	5
SCars	8144	8041	9312	196
Caltech256	21,314	9293	10,000	257

shows the numbers of training and test data, the numbers of categories, and the sizes of the input vectors for CIFAR10, UMist, MNIST, NORB, Stanford Cars, and Caltech256.

4.2. Selection of Evaluation Metrics

To analyze the performance of the proposed algorithm, we used the root mean square error (RMSE), mean absolute error (MAE), ${\mathrm{R}}^2$ score, index of agreement (AI), Theil inequality coefficient (TIC), minimum error (MIN), and maximum error (MAX), as defined in

Table 2. Seven evaluation criteria to evaluate the performance of the proposed method.

Metric	Definition	Mathematical Expression
RMSE	Root mean square error	$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\sqrt{\frac{{\sum }_{i=1}^n{(y_i^a-y_i^e)}^2}{n}}$
MAE	Mean absolute error	$\mathrm{M}\mathrm{A}\mathrm{E}=\frac{{\sum }_{i=1}^n|y_i^a-y_i^e|}{n}$
${\mathrm{R}}^2$	${\mathrm{R}}^2$ score	${\mathrm{R}}^2=1-\frac{{\sum }_{i=1}^n{(y_i^a-y_i^e)}^2}{{\sum }_{i=1}^n{(y_i^a-{\overline{y}}^a)}^2}$
IA	Willmott’s index of agreement	$\mathrm{I}\mathrm{A}=1-\frac{\sqrt{\frac{1}{n}{\sum }_{i=1}^n{(y_i^a-y_i^e)}^2}}{{\sum }_{i=1}^n\left(\right|y_i^e-{\overline{y}}^a|+|y_i^a-{\overline{y}}^a\left|\right)}$
TIC	Theil inequality coefficient	$\mathrm{T}\mathrm{I}\mathrm{C}=\frac{\sqrt{\frac{1}{n}{\sum }_{i=1}^n{(y_i^a-y_i^e)}^2}}{\sqrt{\frac{1}{n}{\sum }_{i=1}^n{\left(y_i^a\right)}^2}+\sqrt{\frac{1}{n}{\sum }_{i=1}^n{\left(y_i^e\right)}^2}}$
MIN	Minimum error	$\mathrm{M}\mathrm{I}\mathrm{N}=min|y_i^a-y_i^e|,i=1,\dots ,n$
MAX	Maximum error	$\mathrm{M}\mathrm{A}\mathrm{X}=max|y_i^a-y_i^e|,i=1,\dots ,n$

Note: y^a denote the vectors of actual and estimated values, respectively $\stackrel{-}{y}$^a is the average of the actual values, and n is the number of pairs of $y_i^a$ and $y_i^e$

[53]. We used the RMSE and MAE metrics to represent the average error between the estimated and actual values. The lower the values of these measures, the higher the efficiency of the autoencoder.

The ${\mathrm{R}}^2$ score is a statistical measure that is defined on the domain $[0,1]$, where a value of one indicates perfect prediction and zero an inefficient model [54]. In contrast, the TIC metric indicates the accuracy capability of the proposed system, with a value between zero and one. The closer the value is to zero, the higher the predictive efficiency of the ELM-AE method. Likewise, the IA prediction index can provide external statistical information on the predictive ability of the model. To verify the efficiency of the ELM-AE, each experiment was repeated 10 times, and the average values of the training times and the evaluation criteria represented in Table 2 were recorded.

4.3. Experimental Setup

In each experiment, the training and test data were standardized to the range $[0,1]$ using the following equation:

$$y_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}^a=\frac{y_i^a-y_{\mathrm{m}\mathrm{i}\mathrm{n}}^a}{y_{\mathrm{m}\mathrm{a}\mathrm{x}}^a-y_{\mathrm{m}\mathrm{i}\mathrm{n}}^a}$$

(9)

where $y_i^a$ is the actual value to be standardized, $y_{\mathrm{m}\mathrm{i}\mathrm{n}}^a$ and $y_{\mathrm{m}\mathrm{a}\mathrm{x}}^a$ are the minimum and maximum of the actual values. In the hidden layer, the weights and biases are random values generated by means of a uniform distribution in the ranges $[-1,1]$ and $[0,1]$, and the sigmoid $g\left(x\right)=\frac{1}{1+e^{-x}}$ was used as the activation function. We chose this activation function due to its universal approximation capability in ELM networks.

The algorithm presented in Section 3 has the following parameters that need to be configured: the number of neurons L, the stopping criterion P, the maximum number of iterations I, and the regularization term $\lambda$. In the first scenario of the proposed method, the regularization parameter set to $\lambda =\frac{1}{2}\sqrt{\frac{2log\left(L\right)}{N}}$, the stopping criterion P is expressed as $\frac{\parallel \mathit{H}{\mathit{\beta }}_k-\mathit{X}\parallel }{\parallel \mathit{X}\parallel }<\epsilon$ and the maximum number of iterations is $I=50$. The threshold $\epsilon$ is a user-specified value that is applied to interrupt the iteration of the algorithm. For the UMist, CIFAR10, MNIST, NORB, SCars, and Caltech256 databases, we set $\epsilon =0.30,0.32,0.60,0.13,0.40$ and $0.40$, respectively. Moreover, the practical application of the FISTA, G-FISTA, and LC-FISTA algorithms for training of the ELM-AE requires tuning of the $L_f$ Lipschitz constant, and the strong convexity constant $\mu$ must be tuned for the LC-FISTA algorithm. For the convex function $f\left(\mathit{\beta }\right)={\parallel \mathit{H}\mathit{\beta }-\mathit{X}\parallel }_2^2$, we chose values of $L_f={\lambda }_{\max }\left({\mathit{H}}^T\mathit{H}\right)$ and $\mu ={\lambda }_{\min }\left({\mathit{H}}^T\mathit{H}\right)$, where ${\lambda }_{\min }$ and ${\lambda }_{\max }$ are the minimum and maximum proper values of ${\mathit{H}}^T\mathit{H}$. The additional control parameters in G-FISTA were described in Section 2.2.2.

In a second scenario, we evaluated the sensitivity of the regularization parameter $\lambda$ on the performance of our method for the UMist, NORB, Stanford Cars, and Caltech256 datasets. This tuning term was chosen using cross-validation for each algorithm and database.

Table 3. Choice of $\lambda$ for each algorithm and selected database.

Algorithm	Regularization Parameter $\mathit{\lambda }$
	UMist	NORB	SCars	Caltech256
FISTA	0.001	10	0.001	1
G-FISTA	0.001	100	0.01	0.1
LC-FISTA	0.01	100	0.1	0.01

shows the optimal value of $\lambda$ selected of $\{{10}^{-5},{10}^{-4},\dots ,{10}^4,{10}^5\}$, obtained by comparing the RMSE between the generated models, chosen as the $\lambda$ that gives a lower RMSE. The experiment only considered the case $L=2000$ for the selected databases. The other parameters of the FISTA, G-FISTA, and LC-FISTA algorithms follow the same selection criteria of the first scenario. For example, the steplength for FISTA and LC-FISTA has been $\rho =\frac{1}{L_f}$, but for G-FISTA, $\rho =\frac{1.3}{L_f}$, $\xi =0.96$, and $S=1.1$. The Lipschitz constant $L_f$ and the strong convexity parameter $\mu$ of f are taken as the maximum and minimum eigenvalue of the matrix ${\mathit{H}}^T\mathit{H}$. The mathematical analysis and numerical experiments presented in [42,43,44,45,49] motivated the choice of our parameters.

To avoid the problem of overfitting when the number of neurons is increased, we applied a 10-fold cross-validation technique. This control tool allowed us to restrict the training to a certain number of neurons. We then used the test set specified in Table 1 and the metrics defined in Table 2 to evaluate the performance of each algorithm. Figure 1 shows the values of the RMSE for the training, validation, and test sets for the G-FISTA algorithm. A graphical view is not shown for FISTA and LC-FISTA since all three algorithms had a comparable level of accuracy. According to the above analysis, $L=900$ and $L=1300$ were the maximum neuron thresholds that could be assigned in ELM-AE for the MNIST and CIFAR10 databases.

4.4. Representation of Features with Different Numbers of Neurons

The computational complexity and execution time of the proposed method are explored in this section. For this purpose, several cases are considered with varying numbers of neurons in the hidden layer of the ELM-AE. In this phase of the experiments, we selected the measures of training time, RMSE, MAE, $R^2$, IA, TIC, MIN, and MAX for comparison on the test set. These evaluation indicators were obtained as a result of training the ELM-AE with the shrinkage-thresholding class of algorithms introduced in Section 2.2.

Table 4. Training times and RMSE values achieved by FISTA, G-FISTA, and LC-FISTA for training of the ELM-AE.

L	Algorithm	RMSE						Training Time (Seg)
		CIFAR10	UMist	MNIST	NORB	SCars	Caltech256	CIFAR10	UMist	MNIST	NORB	SCars	Caltech256
100	FISTA	0.0751	0.0583	0.0856	0.0385	0.1284	0.1205	21.92	2.92	25.47	22.33	38.22	63.61
	G-FISTA	0.0753	0.0580	0.0856	0.0385	0.1285	0.1205	12.31	2.40	16.73	12.09	18.73	35.73
	LC-FISTA	0.0751	0.0579	0.0855	0.0385	0.1285	0.1205	12.91	2.34	18.97	11.17	20.61	33.03
500	FISTA	0.0242	0.0368	0.0096	0.0193	0.088	0.0836	48.46	11.42	51.24	48.40	83.23	122.15
	G-FISTA	0.0242	0.0373	0.0096	0.0193	0.088	0.0836	26.97	10.27	32.06	26.49	42.16	75.94
	LC-FISTA	0.0242	0.0370	0.0096	0.0193	0.088	0.0837	27.61	9.49	35.92	24.26	42.52	69.93
1000	FISTA	0.0019	0.0288	—	0.0044	0.0718	0.0681	78.80	29.78	—	80.18	153.88	211.51
	G-FISTA	0.0019	0.0298	—	0.0044	0.0718	0.0681	43.14	26.82	—	46.54	80.08	125.85
	LC-FISTA	0.0019	0.0294	—	0.0044	0.0718	0.0681	44.83	25.07	—	42.34	78.60	118.25
1500	FISTA	—	0.0244	—	0.0015	0.0617	0.0589	—	57.79	—	116.54	221.72	285.91
	G-FISTA	—	0.0257	—	0.0015	0.0617	0.0589	—	48.39	—	69.18	117.10	176.85
	LC-FISTA	—	0.0253	—	0.0015	0.0617	0.0589	—	46.48	—	63.09	116.60	162.03
2000	FISTA	—	0.0214	—	0.0003	0.0543	0.0522	—	96.09	—	161.86	297.65	379.08
	G-FISTA	—	0.0229	—	0.0003	0.0543	0.0522	—	78.74	—	95.27	158.85	239.25
	LC-FISTA	—	0.0225	—	0.0003	0.0543	0.0522	—	75.55	—	86.17	162.87	222.65

presents the training times and values of RMSE for the ELM-AE on each of the six datasets, for increasing numbers of neurons in the hidden layer. These experimental results show that the training speeds achieved by G-FISTA and LC-FISTA were lower than that obtained by FISTA. For example, for the CIFAR10 set, with 100 neurons in the hidden layer, the computation time fo rthe ELM trained by FISTA was 21.92 s, while the G-FISTA and LC-FISTA algorithms achieved average values of 12.31 s and 12.91 s. Likewise, when we increased the number of hidden neurons $\left(L\right)$, the training time obtained by G-FISTA and LC-FISTA was approximately half that required by FISTA. At the test stage, the RMSE metric achieved by the G-FISTA and LC-FISTA algorithms proposed in this paper indicated a similar level of error as the current FISTA algorithm. This evaluation metric became smaller as the parameter L was increased. For MNIST and CIFAR10, we obtained evaluation metrics of up to $L=500$ and $L=1000$, respectively.

For each dataset, we show the execution time of the proposed method in Figure 2. A bar chart represents this evaluation measure as a function of the number of neurons. The blue, red, and orange bars indicate the training times obtained by FISTA, G-FISTA, and LC-FISTA, respectively. From the results illustrated in Figure 2, we infer that the proposed G-FISTA and LC-FISTA algorithms are faster than FISTA for training the ELM-AE.

In the following, we demonstrate the efficiency of the proposed method using four evaluation criteria: the MAE, $R^2$ score, IA, and TIC. A summary of the MAE and $R^2$ values is given in

Table 5. MAE and $R^2$ score values achieved by FISTA, G-FISTA and LC-FISTA for training of the ELM-AE.

L	Algorithm	MAE						${\mathit{R}}^2$ Score
		CIFAR10	UMist	MNIST	NORB	SCars	Caltech256	CIFAR10	UMist	MNIST	NORB	SCars	Caltech256
100	FISTA	0.0546	0.0415	0.0489	0.0184	0.0921	0.0837	0.9009	0.9209	0.9241	0.9533	0.7896	0.8426
	G-FISTA	0.0547	0.0413	0.0488	0.0185	0.0921	0.0837	0.9005	0.9217	0.9240	0.9533	0.7895	0.8426
	LC-FISTA	0.0547	0.0412	0.0488	0.0185	0.0921	0.0838	0.9005	0.9218	0.9241	0.9532	0.7896	0.8425
500	FISTA	0.0181	0.0256	0.0046	0.0101	0.0625	0.0566	0.9894	0.9684	0.9991	0.9883	0.8994	0.9241
	G-FISTA	0.0181	0.0258	0.0046	0.0101	0.0625	0.0566	0.9894	0.9677	0.9991	0.9883	0.8993	0.9241
	LC-FISTA	0.0181	0.0256	0.0047	0.0102	0.0625	0.0566	0.9894	0.9681	0.9991	0.9883	0.8994	0.9241
1000	FISTA	0.0014	0.0201	—	0.0018	0.0508	0.0461	0.9999	0.9807	—	0.9994	0.9342	0.9497
	G-FISTA	0.0014	0.0208	—	0.0018	0.0508	0.0461	0.9999	0.9793	—	0.9994	0.9342	0.9497
	LC-FISTA	0.0014	0.0205	—	0.0018	0.0508	0.0461	0.9999	0.9798	—	0.9994	0.9342	0.9497
1500	FISTA	—	0.0172	—	0.0005	0.0440	0.0401	—	0.9861	—	0.9999	0.9515	0.9624
	G-FISTA	—	0.0181	—	0.0005	0.0440	0.0401	—	0.9846	—	0.9999	0.9515	0.9624
	LC-FISTA	—	0.0178	—	0.0005	0.0440	0.0401	—	0.9851	—	0.9999	0.9515	0.9624
2000	FISTA	—	0.0152	—	0.0001	0.0391	0.0358	—	0.9893	—	1.0000	0.9624	0.9704
	G-FISTA	—	0.0163	—	0.0001	0.0391	0.0358	—	0.9878	—	1.0000	0.9624	0.9704
	LC-FISTA	—	0.0160	—	0.0001	0.0391	0.0358	—	0.9882	—	1.0000	0.9624	0.9704

. The closer the value of MAE to zero, and the closer the value of $R^2$ to one, the better the performance of the autoencoder. From the values of the metrics given in Table 5, we can see that the order of magnitude is similar, and there is no significant difference between the shrinkage-thresholding methods, meaning that the proposed method maintains the same level of accuracy as the FISTA algorithm. Taking the UMist database as an example, we see that when $L=1500$, the average MAE values obtained by FISTA, G-FISTA, and LC-FISTA are 0.0172, 0.0181, and 0.0178, while their $R^2$ scores are 0.9861, 0.9846, and 0.9851, respectively. The IA and TIA metrics are also important for evaluating the capability of the proposed learning system. These fit metrics are described in

Table 6. IA and TIA values achieved by FISTA, G-FISTA and LC-FISTA for training of the ELM-AE.

L	Algorithm	IA						TIC
		CIFAR10	UMist	MNIST	NORB	SCars	Caltech256	CIFAR10	UMist	MNIST	NORB	SCars	Caltech256
100	FISTA	0.9734	0.9790	0.9799	0.9879	0.9386	0.9559	0.0700	0.0712	0.1288	0.0247	0.1207	0.0981
	G-FISTA	0.9732	0.9793	0.9799	0.9879	0.9386	0.9559	0.0701	0.0708	0.1288	0.0247	0.1208	0.0981
	LC-FISTA	0.9733	0.9793	0.9800	0.9879	0.9386	0.9559	0.0701	0.0708	0.1288	0.0247	0.1208	0.0981
500	FISTA	0.9973	0.9917	0.9998	0.9971	0.9729	0.9800	0.0228	0.0449	0.0121	0.0123	0.0829	0.0678
	G-FISTA	0.9973	0.9917	0.9998	0.9971	0.9729	0.9800	0.0228	0.0454	0.0121	0.0123	0.0829	0.0678
	LC-FISTA	0.9973	0.9918	0.9998	0.9971	0.9729	0.9800	0.0228	0.0451	0.0121	0.0123	0.0829	0.0678
1000	FISTA	1.0000	0.9951	—	0.9999	0.9827	0.9869	0.0018	0.0350	—	0.0028	0.0668	0.0551
	G-F ISTA	1.0000	0.9948	—	0.9999	0.9827	0.9870	0.0018	0.0363	—	0.0028	0.0668	0.0551
	LC-FISTA	1.0000	0.9949	—	0.9999	0.9827	0.9869	0.0018	0.0358	—	0.0028	0.0669	0.0551
1500	FISTA	—	0.9965	—	1.0000	0.9874	0.9903	—	0.0297	—	9.81 × 10${}^{-4}$	0.0573	0.0476
	G-FISTA	—	0.9961	—	1.0000	0.9874	0.9903	—	0.0313	—	9.79 × 10${}^{-4}$	0.0573	0.0476
	LC-FISTA	—	0.9962	—	1.0000	0.9874	0.9903	—	0.0308	—	9.81 × 10${}^{-4}$	0.0573	0.0476
2000	FISTA	—	0.9973	—	1.0000	0.9903	0.9924	—	0.0261	—	2.16 × 10${}^{-4}$	0.0505	0.0422
	G-FISTA	—	0.9969	—	1.0000	0.9903	0.9924	—	0.0279	—	2.15 × 10${}^{-4}$	0.0505	0.0422
	LC-FISTA	—	0.9970	—	1.0000	0.9903	0.9924	—	0.0273	—	2.16 × 10${}^{-4}$	0.0505	0.0422

. As the average values of the IA and TIA approach one and zero, respectively, the feature representation improves.

Table 7. MIN and MAX values achieved by FISTA, G-FISTA and LC-FISTA for training of the ELM-AE.

L	Algorithm	MIN						MAX
		CIFAR10	UMist	MNIST	NORB	SCars	Caltech256	CIFAR10	UMist	MNIST	NORB	SCars	Caltech256
100	FISTA	0.0093	0.0251	0.022	0.066	0.0204	0.0059	0.1225	0.0743	0.933	0.0531	0.1843	0.2533
	G-FISTA	0.0095	0.0254	0.0224	0.0065	0.0206	0.0061	0.1211	0.0742	0.0953	0.0530	0.1818	0.2505
	LC-FISTA	0.0093	0.0250	0.0227	0.0066	0.0206	0.0060	0.1227	0.0742	0.0917	0.0533	0.1809	0.2528
500	FISTA	0.0028	0.0127	0.0010	0.0038	0.0143	0.0040	0.0563	0.0465	0.0277	0.0346	0.1292	0.1814
	G-FISTA	0.0027	0.0114	0.0009	0.0034	0.0143	0.0040	0.0564	0.0471	0.075	0.0348	0.1283	0.1811
	LC-FISTA	0.0028	0.0112	0.0009	0.0033	0.0144	0.0040	0.0563	0.0465	0.0278	0.0349	0.1290	0.1815
1000	FISTA	0.0006	0.0069	—	0.0005	0.0124	0.0028	0.0175	0.0357	—	0.0176	0.1141	0.1599
	G-FISTA	0.0006	0.0063	—	0.0005	0.0123	0.0028	0.0177	0.0373	—	0.0178	0.1138	0.1605
	LC-FISTA	0.0006	0.0058	—	0.0005	0.0123	0.0028	0.0176	0.0368	—	0.0177	0.1136	0.1602
1500	FISTA	—	0.0038	—	5.3 × 10${}^{-5}$	0.0105	0.0020	—	0.0281	—	0.0084	0.1010	0.1382
	G-FISTA	—	0.0037	—	5.4 × 10${}^{-5}$	0.0105	0.0020	—	0.0303	—	0.0082	0.1010	0.1380
	LC-FISTA	—	0.0033	—	5.4 × 10${}^{-5}$	0.0105	0.0020	—	0.0295	—	0.0082	0.1012	0.1375
2000	FISTA	—	0.0028	—	1.7 × 10${}^{-5}$	0.0095	0.0017	—	0.0252	—	0.0047	0.0950	0.1244
	G-FISTA	—	0.0029	—	1.7 × 10${}^{-5}$	0.0095	0.0017	—	0.0274	—	0.0048	0.0948	0.1249
	LC-FISTA	—	0.0025	—	1.6 × 10${}^{-5}$	0.0094	0.0017	—	0.0267	—	0.0047	0.0950	0.1244

shows the MIN and MAX statistical indices for the proposed model on all the selected databases. This table indicates the minimum and maximum absolute errors for the ELM-AE in the testing stage. From the results of this experimental analysis, we can conclude that the FISTA algorithm has a similar generalization capability as G-FISTA and LC-FISTA.

The best tuning of the parameter $\lambda$ in ELM-AE usually influences its generalization capacity. With the $\lambda$ in Table 3 chosen using cross-validation, we show in

Table 8. The performance of each algorithm for the $\lambda$ chosen in Table 3.

Algorithm	RMSE				Training Time (Minutes)
	UMist	NORB	SCars	Caltech256	UMist	NORB	SCars	Caltech256
FISTA	0.0211	0.0001	0.0537	0.0494	49	21	44	139
G-FISTA	0.0225	0.0001	0.0537	0.0494	40	12	27	85
LC-FISTA	0.0220	0.0001	0.0537	0.0494	37	11	26	70

the performance of our method for the particular case $L=2000$. The values of RMSE and the training time are the two indicators selected to compare the efficiency of FISTA, G-FISTA, and LC-FISTA in ELM-AE. The time shown in Table 8 is the execution time required to select the parameter $\lambda$. Note that G-FISTA and LC-FISTA maintain the reconstruction capability of FISTA but perform better in terms of computational time. In addition, the values of RMSE in Table 4 and Table 8 for the first and second scenarios are comparable measures. Therefore, the $\lambda$ chosen in the first scenario and the additional parameters may be adequate in large-scale application problems since cross-validation is not required.

5. Discussion

The main goal of an autoencoder is to learn a compressed representation of the input and then reconstruct it within a reasonable computation time. This study proposes a robust sparse optimization-based method for training ELM-AE networks. With the control parameters specified in the results section, we see that the training times for G-FISTA and LC-FISTA are less than for FISTA.

Table 9. Average speeds of the G-FISTA and LC-FISTA algorithms compared with FISTA for training of the ELM-AE.

L	Algorithm	Training Time (Seg)
		CIFAR10	UMist	MNIST	NORB	SCars	Caltech256
100	G-FISTA	43.84	17.80	34.31	45.85	50.99	43.82
	LC-FISTA	41.10	19.86	25.52	49.97	46.07	48.07
500	G-FISTA	44.34	10.07	37.43	45.26	49.34	37.83
	LC-FISTA	43.02	16.90	29.89	49.87	48.91	42.75
1000	G-FISTA	45.25	9.93	—	41.95	47.95	40.49
	LC-FISTA	43.10	15.81	—	47.19	48.92	44.09
1500	G-FISTA	—	16.26	—	40.63	47.18	38.14
	LC-FISTA	—	19.57	—	45.86	47.41	43.32
2000	G-FISTA	—	18.05	—	41.14	46.63	36.88
	LC-FISTA	—	21.37	—	46.76	45.28	41.26
Av. time	G-FISTA	44.48%	14.42%	35.87%	42.97%	48.42%	39.43%
Av. time	LC-FISTA	42.41%	18.71%	27.70%	47.93%	47.32%	43.90%

Note: The formula % = (1 − A₁/A₂) × 100 compares the computation time of algorithms A₁ and A₂.

shows these important results in terms of percentages. For the CIFAR10, UMist, MNIST, NORB, SCars, and Caltech256 databases, the average computation times for G-FISTA are 44.48%, 14.42%, 35.87%, 42.97%, 48.42%, and 39.43% faster than for FISTA, respectively. The LC-FISTA algorithm achieved speedups of 42.41%, 18.71%, 27.70%, 47.93%, 47.32%, and 43.90%. In addition, the indicators recorded in Table 4, Table 5, Table 6, Table 7 and Table 8 are coherent, and show comparable performance for all databases. The proposed training scheme is an important automatic learning method since it can help make accurate decisions in shorter periods. Consequently, incorporating fast optimization techniques into the autoencoder architecture can help improve the performance of current models, and particularly the first-order methods of the shrinkage-thresholding class.

The proximal step of FISTA requires the main computational effort, and this also holds true for G-FISTA and LC-FISTA. The additional computation required in G-FISTA and LC-FISTA increases the convergence speed, but the execution cost is higher. However, this problem can be controlled, since G-FISTA and LC-FISTA converge with fewer iterations. In the experimental evaluation described in Section 4, we raised the complexity of the problem by increasing the number of neurons in the hidden layer to get a clearer idea of the performance when solving applications involving higher dimensional datasets. Based on this evaluation criterion, we see that the proposed method can be configured to solve large-scale problems, since the vector operations involved in the ELM-AE architecture and the proposed mechanism are block-separable.

6. Conclusions

This paper has proposed the G-FISTA and LC-FISTA algorithms to reduce the training time of ELM-AEs. The performance of our methods was compared to FISTA, an algorithm of the shrinkage-thresholding class, which is a state-of-the-art method of training ELM-AEs.

Experiments were conducted on six databases: CIFAR10, UMist, MNIST, NORB, Stanford Cars, and Caltech256. The results showed that G-FISTA and LC-FISTA achieved similar computational times for the training of the autoencoder, both of which were significantly lower than FISTA. In numerical terms, G-FISTA and LC-FISTA were 37.60% and 37.99% faster than FISTA in training the ELM-AE (average value obtained from Table 9). Consequently, the advantage main of our study is to maintain the generalization capability of FISTA while the computational speed is improved. A possible disadvantage may be the hyperparameter settings, but many researchers set these values according to existing mathematical analysis in the literature to control runtime. Other advantages and disadvantages we will discuss in future perspectives.

Motivated by technological advances and the speed of convergence of shrinkage-thresholding algorithms, we intend to extend our methodology in further research to a parallel and distributed architecture. This approach will be of great interest for solving current large-scale applications in computer clusters and clouds. In addition, we aim to incorporate iterative schemes of the shrinkage-thresholding class into the architecture of a variational, convolutional deep autoencoder network to solve optimization problems with large datasets.