A New Accelerated Forward–Backward Splitting Algorithm for Monotone Inclusions with Application to Data Classification

Abstract

This paper proposes a new accelerated fixed-point algorithm based on a double-inertial extrapolation technique for solving structured variational inclusion and convex bilevel optimization problems. The underlying framework leverages fixed-point theory and operator splitting methods to address inclusion problems of the form $0\in (A+B)(x)$, where A is a cocoercive operator and B is a maximally monotone operator defined on a real Hilbert space. The algorithm incorporates two inertial terms and a relaxation step via a contractive mapping, resulting in improved convergence properties and numerical stability. Under mild conditions of step sizes and inertial parameters, we establish strong convergence of the proposed algorithm to a point in the solution set that satisfies a variational inequality with respect to a contractive mapping. Beyond theoretical development, we demonstrate the practical effectiveness of the proposed algorithm by applying it to data classification tasks using Deep Extreme Learning Machines (DELMs). In particular, the training processes of Two-Hidden-Layer ELM (TELM) models is reformulated as convex regularized optimization problems, enabling robust learning without requiring direct matrix inversions. Experimental results on benchmark and real-world medical datasets, including breast cancer and hypertension prediction, confirm the superior performance of our approach in terms of evaluation metrics and convergence. This work unifies and extends existing inertial-type forward–backward schemes, offering a versatile and theoretically grounded optimization tool for both fundamental research and practical applications in machine learning and data science.

1. Introduction

The convex bilevel optimization problem plays an important role in real-world applications such as image and signal processing, data classification problems, medical image, machine learning, and so on. Recently, deep learning has become an important tool in many areas, such as image classification, speech recognition, and medical data analysis.

The convex bilevel optimization consists of the following two levels.

The outer-level problem is

$$\underset{x\in \Gamma }{min}\varphi (x),$$

(1)

where $\varphi :\mathcal{H}\to \mathbb{R}$ is a strongly convex and differentiable function over a real Hilbert space $\mathcal{H}$ and $\Gamma$ is the set of solutions to the inner-level problem:

$$\underset{x\in \mathcal{H}}{arg\; min}\{f(x)+g(x)\},$$

(2)

where f is convex and differentiable and $g\in {\Gamma }_0(\mathcal{H})$ is the class of proper, lower semicontinuous, convex functions. The implicit nature of the constraint set $\Gamma$ makes the bilevel problem particularly challenging and well-suited for operator-theoretic approaches.

Various algorithms have been developed to solve Problems (1) and (2). Among these, the Bilevel Gradient Sequential Averaging Method (BiG-SAM, Algorithm 1) was proposed by Sabach and Shtern [1] as follows:

Algorithm 1 BiG-SAM

Input: $x_1\in {\mathbb{R}}^m,{\alpha }_n\in (0,1),{\gamma }_n\in (0,{\displaystyle \frac{1}{L_f}})$ and $s\in (0,{\displaystyle \frac{2}{L_{\varphi }+\sigma }})$ where $L_f$ and $L_{\varphi }$ are the Lipschitz constants of $\nabla f$ and $\nabla \varphi$, respectively. For $n\ge 1$: Compute: $$\left\{\begin{array}{c}\begin{array}{cc}{y}_n\hfill & ={prox}_{{\gamma }_{n}g}(x_n-{\gamma }_n\nabla f(x_n)),\hfill \\ x_{n+1}\hfill & ={\alpha }_n(x_n-s\nabla \varphi (x_n))+(1-{\alpha }_n)y_n,\end{array}\hfill \end{array}\right.$$ where $\nabla f$ and $\nabla \varphi$ are gradients of f and $\varphi$, respectively.

They showed that $x_n\to x\in \Omega$, where $\Omega$ is the set of all solutions to Problem (1).

The inertial technique for accelerating the convergence behavior of the algorithms was first proposed by Polyak [2]. Since then, this technique has been continuously employed in various algorithms.

Shehu et al. [3] designed the inertial Bilevel Gradient Sequential Averaging Method (iBiG-SAM, Algorithm 2) as an extension of Algorithm 1, with inertial technique for improving its convergence rate.

Algorithm 2 iBiG-SAM

Input: $x_0,x_1\in {\mathbb{R}}^m,\alpha \ge 3,{\alpha }_n\in (0,1),{\gamma }_n\in (0,{\displaystyle \frac{2}{L_f}}),s\in (0,{\displaystyle \frac{2}{L_{\varphi }+\sigma }}]$ where $L_f$ and $L_{\varphi }$ are the Lipschitz constants of $\nabla f$ and $\nabla \varphi$, respectively. For $n\ge 1$: Choose: ${\theta }_n\in [0,\overline{{\theta }_n}]$ where $\overline{{\theta }_n}$ is defined by $$\overline{{\theta }_n}:=\left\{\begin{array}{c}\begin{array}{cc}min\{{\displaystyle \frac{n-1}{n+\alpha -1}},{\displaystyle \frac{{\xi }_n}{\parallel x_n-x_{n-1}\parallel }}\},\hfill & \mathrm{if}{x}_n\ne x_{n-1},\hfill \\ {\displaystyle \frac{n-1}{n+\alpha -1}},\hfill & \mathrm{otherwise}.\hfill \end{array}\hfill \end{array}\right.$$ Compute: $$\left\{\begin{array}{c}\begin{array}{cc}{y}_n\hfill & =x_n+{\theta }_n(x_n-x_{n-1}),\hfill \\ t_n\hfill & ={prox}_{{\gamma }_{n}g}(y_n-{\gamma }_n\nabla f(y_n)),\hfill \\ w_n\hfill & =y_n-s\nabla \varphi (y_n),\hfill \\ x_{n+1}\hfill & ={\alpha }_n{w}_n+(1-{\alpha }_n)t_n,\hfill \end{array}\hfill \end{array}\right.$$ where $\nabla f$ and $\nabla \varphi$ are gradients of f and $\varphi$, respectively.

They further demonstrated that the sequence $\{x_n\}$ converges to some $x\in \Omega$ when the control sequence $\{{\alpha }_n\}$ satisfies the following criteria:

$$\underset{n\to \infty }{lim}{\alpha }_n=0 \mathrm{and}\sum _{n=1}^{\infty }{\alpha }_n=\infty .$$

(3)

To further improve the convergence performance of Algorithm 2, Duan and Zhang [4] developed three related methods; namely, the alternated inertial Bilevel Gradient Sequential Averaging Method (aiBiG-SAM, Algorithm 3), multi-step inertial Bilevel Gradient Sequential Averaging Method (miBiG-SAM, Algorithm 4), and multi-step alternative inertial Bilevel Gradient Sequential Averaging Method (amiBiG-SAM, Algorithm 5), which were defined as follows:

Algorithm 3 aiBiG-SAM

Input: $x_0,x_1\in {\mathbb{R}}^m,\alpha \ge 3,\xi >0,{\gamma }_n\in (0,{\displaystyle \frac{2}{L_f}}),{\xi }_n\in (0,1),s\in (0,{\displaystyle \frac{2}{L_{\varphi }+\sigma }}]$ where $L_f$ and $L_{\varphi }$ are the Lipschitz constants of $\nabla f$ and $\nabla \varphi$, respectively. Let $\{{\alpha }_n\}$ be a sequence in $(0,1)$ satisfying (3). For $n\ge 1$: Step 1. Compute: $$y_n=\left\{\begin{array}{c}\begin{array}{cc}{x}_n+{\theta }_n(x_n-x_{n-1}),\hfill & \mathrm{if} n\mathrm{is} \mathrm{odd},\hfill \\ x_n,\hfill & \mathrm{otherwise}.\hfill \end{array}\hfill \end{array}\right.$$ When n is odd, choose ${\theta }_n$ such that $0 \le |{\theta }_n| \le \overline{{\theta }_n}$ with $\overline{{\theta }_n}$ defined by $$\overline{{\theta }_n}:=\left\{\begin{array}{c}\begin{array}{cc}min\{{\displaystyle \frac{n}{n+\alpha -1}},{\displaystyle \frac{{\xi }_n}{\parallel x_n-x_{n-1}\parallel }}\},\hfill & \mathrm{if}{x}_n\ne x_{n-1},\hfill \\ {\displaystyle \frac{n}{n+\alpha -1}},\hfill & \mathrm{otherwise}.\hfill \end{array}\hfill \end{array}\right.$$ When n is even, ${\theta }_n=0.$ Step 2. Compute: $$\left\{\begin{array}{c}\begin{array}{cc}{t}_n\hfill & ={prox}_{{\gamma }_{n}g}(y_n-{\gamma }_n\nabla f(y_n)),\hfill \\ w_n\hfill & =y_n-s\nabla \varphi (y_n),\hfill \\ x_{n+1}\hfill & ={\alpha }_n{w}_n+(1-{\alpha }_n)t_n,\hfill \end{array}\hfill \end{array}\right.$$ where $\nabla f$ and $\nabla \varphi$ are gradients of f and $\varphi$, respectively. Step 3. If $\parallel x_n-x_{n-1}\parallel < \xi$, then stop. Otherwise, set $n=n+1$ and go to Step 1.

Algorithm 4 miBiG-SAM

Input: $x_0,x_1\in {\mathbb{R}}^m,\alpha \ge 3,\xi >0,{\gamma }_n\in (0,{\displaystyle \frac{2}{L_f}}),{\xi }_n\in (0,1),s\in (0,{\displaystyle \frac{2}{L_{\varphi }+\sigma }}]$ where $L_f$ and $L_{\varphi }$ are the Lipschitz constants of $\nabla f$ and $\nabla \varphi$, respectively. Let $\{{\alpha }_n\}$ be a sequence in $(0,1)$ satisfying (3). For $n\ge 1$: Step 1. Given $x_n,x_{n-1},\dots ,x_{n-q+1}$ and compute $$y_n=x_n+\sum _{i\in Q}{\theta }_{i,n}(x_{n-i}-x_{n-1-i}),$$ where $Q=\{0,1,\dots ,q-1\}$. Choose ${\theta }_{i,n}$ such that $0 \le |{\theta }_{i,n}| \le {\overline{\theta }}_n$ with ${\overline{\theta }}_n$ defined by $${\overline{\theta }}_n:=\left\{\begin{array}{c}\begin{array}{cc}min\{{\displaystyle \frac{n}{n+\alpha -1}},{\displaystyle \frac{{\xi }_n}{{\sum }_{i\in Q}\parallel x_{n-i}-x_{n-1-i}\parallel }}\},\hfill & \mathrm{if}\sum _{i\in Q}\parallel x_{n-i}-x_{n-1-i}\parallel \ne 0,\hfill \\ {\displaystyle \frac{n}{n+\alpha -1}},\hfill & \mathrm{otherwise}.\hfill \end{array}\hfill \end{array}\right.$$ Step 2. Compute: $$\left\{\begin{array}{c}\begin{array}{cc}{t}_n\hfill & ={prox}_{{\gamma }_{n}g}(y_n-{\gamma }_n\nabla f(y_n)),\hfill \\ w_n\hfill & =y_n-s\nabla \varphi (y_n),\hfill \\ x_{n+1}\hfill & ={\alpha }_n{w}_n+(1-{\alpha }_n)t_n,\hfill \end{array}\hfill \end{array}\right.$$ where $\nabla f$ and $\nabla \varphi$ are gradients of f and $\varphi$, respectively. Step 3. If $\parallel x_n-x_{n-1}\parallel < \xi$, then stop. Otherwise, set $n=n+1$ and go to Step 1.

Algorithm 5 amiBiG-SAM

Input: $x_0,x_1\in {\mathbb{R}}^m,\alpha \ge 3,\xi >0,{\gamma }_n\in (0,{\displaystyle \frac{2}{L_f}}),{\xi }_n\in (0,1),s\in (0,{\displaystyle \frac{2}{L_{\varphi }+\sigma }}]$ where $L_f$ and $L_{\varphi }$ are the Lipschitz constants of $\nabla f$ and $\nabla \varphi$, respectively. Let $\{{\alpha }_n\}$ be a sequence in $(0,1)$ satisfying (3). For $n\ge 1$: Step 1. Given $x_n,x_{n-1},\dots ,x_{n-q+1}$ and compute $$y_n=\left\{\begin{array}{c}\begin{array}{cc}{x}_n+{\displaystyle \sum _{i\in Q}}{\theta }_{i,n}(x_{n-i}-x_{n-1-i}),\hfill & \mathrm{if} n \mathrm{is} \mathrm{odd},\hfill \\ x_n,\hfill & \mathrm{otherwise},\hfill \end{array}\hfill \end{array}\right.$$ where $Q=\{0,1,\dots ,q-1\}$. When n is odd, choose ${\theta }_{i,n}$ such that $0 \le |{\theta }_{i,n}| \le {\overline{\theta }}_n$ with ${\overline{\theta }}_n$ defined by $${\overline{\theta }}_n:=\left\{\begin{array}{c}\begin{array}{cc}min\{{\displaystyle \frac{n}{n+\alpha -1}},{\displaystyle \frac{{\xi }_n}{{\sum }_{i\in Q}\parallel x_{n-i}-x_{n-1-i}\parallel }}\},\hfill & \mathrm{if}\sum _{i\in Q}\parallel x_{n-i}-x_{n-1-i}\parallel \ne 0,\hfill \\ {\displaystyle \frac{n}{n+\alpha -1}},\hfill & \mathrm{otherwise}.\hfill \end{array}\hfill \end{array}\right.$$ When n is even, ${\theta }_n=0.$ Step 2. Compute: $$\left\{\begin{array}{c}\begin{array}{cc}{t}_n\hfill & ={prox}_{{\gamma }_{n}g}(y_n-{\gamma }_n\nabla f(y_n)),\hfill \\ w_n\hfill & =y_n-s\nabla \varphi (y_n),\hfill \\ x_{n+1}\hfill & ={\alpha }_n{w}_n+(1-{\alpha }_n)t_n,\hfill \end{array}\hfill \end{array}\right.$$ where $\nabla f$ and $\nabla \varphi$ are gradients of f and $\varphi$, respectively. Step 3. If $\parallel x_n-x_{n-1}\parallel < \xi$, then stop. Otherwise, set $n=n+1$ and go to Step 1.

The convergence analysis revealed that Algorithms 3–5 achieve better performance than Algorithms 1 and 2 (see more details in Duan and Zhang [4]).

We note that Algorithms 1–5 were developed based on fixed-point techniques. Subsequently, viscosity approximation methods combined with the fixed-point method and inertial technique were employed to develop accelerated algorithms for solving convex bilevel optimization problems (see [5,6,7]).

The convex minimization problem (2) is one of the most fundamental and crucial problems in applied mathematics, medical image, data science, data classification, and computer science settings.

It is well known that $x^{*}$ is a solution to Problem (2) if, and only if,

$$0\in \nabla f(x^{*})+\partial g(x^{*})$$

(4)

This leads to the more general framework of variational inclusion problems, which unifies many classes of problems by seeking a point $x^{*}$, such that

$$0\in A(x^{*})+B(x^{*})$$

(5)

where $A:\mathcal{H}\to 2^{\mathcal{H}}$ is a monotone operator and B is a Lipschitz continuous mapping. The solution set of Problem (5) is denoted by ${(A+B)}^{-1}(0)$. Variational inclusion problems generalize fixed-point problems, monotone equations, and variational inequalities, and provide a flexible structure for handling nonsmooth terms and constraints.

The variational inclusion Problem (5) can be reformulated as the fixed point equation

$$x^{*}=T{x}^{*},$$

where $T:=J_{\mathit{\gamma }}^B(I-\mathit{\gamma }A)$ and $J_{\gamma }^B(x):={(I+\gamma B)}^{-1}(x)$ for some $\gamma >0$.

Over the years, various iterative schemes have been proposed for solving variational inclusion Problem (5). A well known and extensively studied one is the forward–backward method (FBM), defined by

$$x_1\in \mathcal{H},x_{n+1}=J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)x_n,n\ge 1,$$

where ${\gamma }_n$ is a positive step size.

To improve the performance of the classical forward–backward method in solving monotone inclusion problems, Moudafi and Oliny (2003) [8] proposed an inertial variant known as the Inertial Forward–Backward Algorithm (IFBA). This method incorporates a momentum-like term inspired by inertial techniques and is designed for finding a zero of the sum of two monotone operators. The iterative scheme was given by

$$\begin{array}{cc}\hfill x_0,x_1\in \mathcal{H},y_n& =x_n+{\theta }_n(x_n-x_{n-1}),\hfill \\ \hfill x_{n+1}& =J_{{\gamma }_n}^B(y_n-{\gamma }_{n}A{x}_n),\forall n\ge 1,\hfill \end{array}$$

where ${\gamma }_n\in (0,{\displaystyle \frac{2}{L_A}})$ and $L_A$ denotes the Lipschitz constant of the monotone operator A. The inclusion of the extrapolation parameter ${\theta }_n$ aims to accelerate the convergence rate of proposed algorithm. Under suitable assumptions on ${\theta }_n$, they proved that the generated sequence converges weakly to a solution of the inclusion problem.

Recently, Peeyada [9] proposed the Inertial Mann Forward–Backward Splitting Algorithm (IMFBSA, Algorithm 6) as a refined approach that combines Mann-type iterations with inertial extrapolation, as follows:

Algorithm 6 IMFBSA

Input: $x_0,x_1\in \mathcal{H},\{{\alpha }_n\}\subset (0,1),\{{\gamma }_n\}\subset (0,2\beta ),\{{\theta }_n\}\subset [0,\infty )$ satisfies the following conditions $$0<lim\underset{n\to \infty }{inf}{\gamma }_n\le lim\underset{n\to \infty }{sup}{\gamma }_n<2\beta \mathrm{and}\sum _{n=1}^{\infty }{\theta }_n\parallel x_n-x_{n-1}\parallel <\infty .$$ Step 1. Compute: $$\left\{\begin{array}{c}\begin{array}{cc}{y}_n\hfill & =x_n+{\theta }_n(x_n-x_{n-1}),\hfill \\ z_n\hfill & =y_n+{\eta }_n(x_n-y_n),\hfill \\ x_{n+1}\hfill & =J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n\hfill \end{array}\hfill \end{array}\right.$$ Step 2. Set $n=n+1$ and go to Step 1.

Moreover, several authors have developed the algorithms by using multi-inertial forward–backward schemes for solving variational inclusion problems, which ensure convergence and demonstrate efficiency in practical applications such as image deblurring (see [10,11]). These developments illustrate the increasing interest in designing inertial-type iterative methods for monotone inclusion and convex optimization problems.

Building upon and inspired by the above-mentioned studies, in this work, we aim to propose a new accelerated fixed-point algorithm designed to address variational inclusion and convex bilevel optimization problems, and prove its strong convergence behavior, including comparison its effectiveness in data classification with the existing algorithms.

The structure of this paper is as follows. In Section 2, we introduce some fundamental definitions and key lemmas used in the later sections. The main theoretical contributions of our study are presented in Section 3. In Section 4, we apply the proposed algorithm to data classification problems using breast cancer and hypertension datasets, and compare its performance with other existing methods. Finally, the conclusion of our work is given in Section 5.

2. Preliminaries

Throughout this work, let $\mathcal{H}$ be a real Hilbert space and $T:\mathcal{H}\to \mathcal{H}$ a mapping. We use $x_n\to x$ for strong convergence and $x_n\rightharpoonup x$ for weak convergence.

Definition 1.

A mapping $T:\mathcal{H}\to \mathcal{H}$ is called Lipschitzian if

$$\parallel Tx-Ty\parallel \le L\parallel x-y\parallel ,\forall x,y\in \mathcal{H},$$

for some $L \ge 0$. If $L\in [0,1)$, then T is a k-contraction, and if $L=1$, then T is nonexpansive.

Definition 2.

Let $⌀\ne D\subset \mathcal{H},$ and let $\beta \in {\mathbb{R}}_{++}$. Then, T is β-cocoercive if $\beta T$ is firmly nonexpansive, i.e., 

$$(\forall x\in D)(\forall y\in D)⟨x-y,Tx-Ty⟩ \ge {\beta \parallel Tx-Ty\parallel }^2.$$

Definition 3.

Let $x\in \mathcal{H}$ and $⌀\ne C\subset \mathcal{H}$ be closed and convex. Then, there is a unique point $x^{*}\in C$, such that

$$\parallel x^{*}-x\parallel \le \parallel y-x\parallel ,\forall y\in C.$$

The mapping $P_C:\mathcal{H}\to C$, defined by $P_{C}x=x^{*}$, is called the metric projection onto C.

We conclude this section with several auxiliary lemmas and propositions essential for supporting the main results.

Lemma 1

([12]). Let $x,y\in \mathcal{H}$ and $\lambda \in [0,1]$. Then, the following identities and inequality hold:

(1) 

${\parallel x\pm y\parallel }^2={\parallel x\parallel }^2\pm 2⟨x,y⟩+{\parallel y\parallel }^2;$

(2) 

${\parallel x+y\parallel }^2\le {\parallel x\parallel }^2+2⟨y,x+y⟩;$

(3) 

${\parallel \lambda x+(1-\lambda )y\parallel }^2={\lambda \parallel x\parallel }^2+{(1-\lambda )\parallel y\parallel }^2-\lambda (1-\lambda ){\parallel x-y\parallel }^2.$

Proposition 1.

Let $⌀\ne C\subset \mathcal{H}$ be convex and let $x\in \mathcal{H},x^{*}\in C$. Then,

$$x^{*}=P_{C}x\iff ⟨x-x^{*},y-x^{*}⟩\le 0,\forall y\in C.$$

Proposition 2

([13]). Suppose $\varphi :\mathcal{H}\to \mathbb{R}$ is strongly convex with parameter $\sigma >0$ and continuously differentiable, such that $\nabla \varphi$ is Lipschitz continuous with constant $L_{\varphi }$. Then, the mapping $I-s\nabla \varphi$ is k-contraction for all $0\le s\le {\displaystyle \frac{2}{L_{\varphi }+\sigma }}$, where $k=\sqrt{1-{\displaystyle \frac{2\sigma s{L}_{\varphi }}{\sigma +L_{\varphi }}}}$ and I is the identity mapping.

Lemma 2

([14]). Let $T:\mathcal{H}\to \mathcal{H}$ be a nonexpansive mapping with $Fix(T)\ne ⌀$. If there exists a sequence $\{x_n\}\subset \mathcal{H}$ such that $x_n\rightharpoonup x\in \mathcal{H}$ and $\parallel x_n-T{x}_n\parallel \to 0$, then $x\in Fix(T)$.

Lemma 3

([15]). Let $A:\mathcal{H}\to \mathcal{H}$ be a β-cocoercive mapping and $B:\mathcal{H}\to 2^{\mathcal{H}}$ a maximal monotone mapping. Then, we have

(1) 

$Fix(J_{\gamma }^B(I-\gamma A))={(A+B)}^{-1}(0),\forall \gamma >0$

(2) 

$x\in \mathcal{H},\parallel x-J_{\gamma }^B(I-\gamma A)x\parallel \le 2\parallel x-J_{\overline{\gamma }}^B(I-\overline{\gamma }A)x\parallel ,\forall x\in \mathcal{H},0<\gamma \le \overline{\gamma }$.

Lemma 4

([16]). Let $\{a_n\}\subset {\mathbb{R}}_{+},\{b_n\}\subset \mathbb{R},$ and $\{{\xi }_n\}\subset (0,1)$ satisfy ${\displaystyle \sum _{n=1}^{\infty }}{\xi }_n=\infty$ and

$$a_{n+1}\le (1-{\xi }_n)a_n+{\xi }_n{b}_n,\forall n\in \mathbb{N}.$$

If for every subsequence $\{a_{n_i}\}$ of $\{a_n\}$, such that ${lim\; inf}_{i\to \infty }(a_{n_i+1}-a_{n_i})\ge 0$, we have ${lim\; sup}_{i\to \infty }{b}_{n_i}\le 0$, then ${lim}_{n\to \infty }{a}_n=0$.

3. Main Results

In this section, we introduce the Double Inertial Viscosity Forward–Backward Algorithm (DIVFBA) which is a modification of Algorithm 6, introduced by Peeyada [9], in order to accelerate its convergence by using a double inertial step at the first step and the viscosity approximation method at the final step. It is worth to mentioning that Algorithm 6 obtained only weak convergence, while we aim to prove strong convergence of our proposed algorithm. Moreover, we will compare the performance of our algorithm and the others in data classification in the next section.

Throughout this section, let $S:\mathcal{H}\to \mathcal{H}$ be a k-contraction mapping with $k\in [0,1)$, A be a $\beta$-cocoercive mapping on $\mathcal{H}$, and B be a maximal monotone operator from $\mathcal{H}$ into $2^{\mathcal{H}}$, such that ${(A+B)}^{-1}(0)\ne ⌀$.

We are now ready to present our accelerated fixed-point algorithm (Algorithm 7).

Algorithm 7 DIVFBA

Initialization: Choose $x_1,x_0,x_{-1}\in \mathcal{H},\{{\gamma }_n\}\subset (0,2\beta )$ and $\{{\alpha }_n\},\{{\beta }_n\},\{{\eta }_n\},\{{\tau }_n\}\subset (0,1)$. Take $\{{\xi }_n\},\{{\mu }_n\}\subset (0,\infty )$ and $\{{\rho }_n\}\subset (-\infty ,0)$. Iterative steps: For $n\ge 1$, calculate $x_{n+1}$ as follows: Step 1. Compute the inertial step: $$\begin{array}{cc}{\theta }_n\hfill & =\left\{\begin{array}{cc}min\{{\mu }_n,{\displaystyle \frac{{\xi }_n}{\parallel x_n-x_{n-1}\parallel }}\},\hfill & \mathrm{if}{x}_n\ne x_{n-1};\hfill \\ {\mu }_n,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$ (6) and $$\begin{array}{cc}{\delta }_n\hfill & =\left\{\begin{array}{cc}max\{{\rho }_n,{\displaystyle \frac{-{\xi }_n}{\parallel x_{n-1}-x_{n-2}\parallel }}\},\hfill & \mathrm{if}{x}_{n-1}\ne x_{n-2};\hfill \\ {\rho }_n,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$ (7) Step 2. Compute $$y_n=x_n+{\theta }_n(x_n-x_{n-1})+{\delta }_n(x_{n-1}-x_{n-2}),$$ (8) $$z_n=y_n+{\eta }_n(x_n-y_n),$$ (9) $$v_n=(1-{\beta }_n)x_n+{\beta }_n{J}_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n,$$ (10) $$w_n=(1-{\tau }_n)J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)y_n+{\tau }_n{J}_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)v_n,$$ (11) $$x_{n+1}={\alpha }_{n}S(x_n)+(1-{\alpha }_n)w_n.$$ (12) Set n := n + 1 and return to Step 1.

Theorem 1.

Let $\{x_n\}$ be a sequence generated by Algorithm 7, such that the following additional conditions hold:

(i) 

$\underset{n\to \infty }{lim}{\displaystyle \frac{{\xi }_n}{{\alpha }_n}}=0$,

(ii) 

$\underset{n\to \infty }{lim}{\alpha }_n=0$ and ${\displaystyle \sum _{n=1}^{\infty }}{\alpha }_n=\infty$,

(iii) 

$0<\underset{n\to \infty }{lim\; inf}{\tau }_n\le \underset{n\to \infty }{lim\; sup}{\tau }_n<1$,

(iv) 

$0<\underset{n\to \infty }{lim\; inf}{\beta }_n\le \underset{n\to \infty }{lim\; sup}{\beta }_n<1$,

(v) 

$0<\underset{n\to \infty }{lim\; inf}{\gamma }_n\le \underset{n\to \infty }{lim\; sup}{\gamma }_n<2\beta$.

Then, $x_n\to p\in {(A+B)}^{-1}(0)$, such that $p=P_{{(A+B)}^{-1}(0)}S(p)$.

Proof. 

Let $p\in {(A+B)}^{-1}(0)$. Firstly, we prove that the sequence $\{x_n\}$ is bounded.

From (8), we have

$$\begin{array}{ccc}\parallel y_n-p\parallel \hfill & =& \parallel x_n+{\theta }_n(x_n-x_{n-1})+{\delta }_n(x_{n-1}-x_{n-2})-p\parallel \hfill \\ \hfill & \le & \parallel x_n-p\parallel +{\theta }_n\parallel x_n-x_{n-1}\parallel +|{\delta }_n|·\parallel x_{n-1}-x_{n-2}\parallel .\hfill \end{array}$$

(13)

From (9) and (13), we have

$$\begin{array}{ccc}\parallel z_n-p\parallel \hfill & =& \parallel y_n+{\eta }_n(x_n-y_n)-p\parallel \hfill \\ \hfill & \le & {\eta }_n\parallel x_n-p\parallel +(1-{\eta }_n)\parallel y_n-p\parallel \hfill \\ \hfill & \le & {\eta }_n\parallel x_n-p\parallel +(1-{\eta }_n)\parallel x_n-p\parallel \hfill \\ \hfill & & +(1-{\eta }_n)[{\theta }_n\parallel x_n-x_{n-1}\parallel +|{\delta }_n|·\parallel x_{n-1}-x_{n-2}\parallel ]\hfill \\ \hfill & \le & \parallel x_n-p\parallel +{\theta }_n\parallel x_n-x_{n-1}\parallel +|{\delta }_n|·\parallel x_{n-1}-x_{n-2}\parallel .\hfill \end{array}$$

(14)

From (10) and the nonexpansiveness of $J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)$, we have

$$\begin{array}{ccc}\parallel v_n-p\parallel \hfill & =& \parallel (1-{\beta }_n)x_n+{\beta }_n{J}_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n-p\parallel \hfill \\ \hfill & \le & (1-{\beta }_n)\parallel x_n-p\parallel \hfill \\ \hfill & & +{\beta }_n\parallel J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)p\parallel \hfill \\ \hfill & \le & (1-{\beta }_n)\parallel x_n-p\parallel +{\beta }_n\parallel z_n-p\parallel \hfill \\ \hfill & \le & (1-{\beta }_n)\parallel x_n-p\parallel +{\beta }_n\parallel x_n-p\parallel \hfill \\ \hfill & & +{\beta }_n[{\theta }_n\parallel x_n-x_{n-1}\parallel +|{\delta }_n|·\parallel x_{n-1}-x_{n-2}\parallel ]\hfill \\ \hfill & \le & \parallel x_n-p\parallel +{\theta }_n\parallel x_n-x_{n-1}\parallel +|{\delta }_n|·\parallel x_{n-1}-x_{n-2}\parallel .\hfill \end{array}$$

(15)

From (11), (13), and (15), we have

$$\begin{array}{ccc}\parallel w_n-p\parallel \hfill & =& \parallel (1-{\tau }_n)J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)y_n+{\tau }_n{J}_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)v_n-p\parallel \hfill \\ \hfill & \le & (1-{\tau }_n)\parallel J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)y_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)p\parallel \hfill \\ \hfill & & +{\tau }_n\parallel J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)v_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)p\parallel \hfill \\ \hfill & \le & (1-{\tau }_n)\parallel y_n-p\parallel +{\tau }_n\parallel v_n-p\parallel \hfill \\ \hfill & \le & \parallel x_n-p\parallel +{\theta }_n\parallel x_n-x_{n-1}\parallel +|{\delta }_n|·\parallel x_{n-1}-x_{n-2}\parallel .\hfill \end{array}$$

(16)

From (12), we obtain

$$\begin{array}{ccc}\parallel x_n-p\parallel \hfill & =& \parallel {\alpha }_{n}S(x_n)+(1-{\alpha }_n)w_n-p\parallel \hfill \\ \hfill & \le & {\alpha }_n\parallel S(x_n)-p\parallel +(1-{\alpha }_n)\parallel w_n-p\parallel \hfill \\ \hfill & \le & {\alpha }_n\parallel S(x_n)-S(p)\parallel +{\alpha }_n\parallel S(p)-p\parallel +(1-{\alpha }_n)\parallel w_n-p\parallel \hfill \\ \hfill & \le & {\alpha }_{n}k\parallel x_n-p\parallel +{\alpha }_n\parallel S(p)-p\parallel +(1-{\alpha }_n)\parallel w_n-p\parallel \hfill \\ \hfill & \le & {\alpha }_{n}k\parallel x_n-p\parallel +{\alpha }_n\parallel S(p)-p\parallel \hfill \\ \hfill & & +(1-{\alpha }_n)[\parallel x_n-p\parallel +{\theta }_n\parallel x_n-x_{n-1}\parallel +|{\delta }_n|·\parallel x_{n-1}-x_{n-2}\parallel \hfill \\ \hfill & \le & (1-{\alpha }_n(1-k))\parallel x_n-p\parallel +{\alpha }_n\parallel S(p)-p\parallel \hfill \\ \hfill & & +{\theta }_n\parallel x_n-x_{n-1}\parallel +|{\delta }_n|·\parallel x_{n-1}-x_{n-2}\parallel \hfill \\ \hfill & =& (1-{\alpha }_n(1-k))\parallel x_n-p\parallel \hfill \\ \hfill & & +{\alpha }_n(1-k)[{\displaystyle \frac{1}{1-k}}({\displaystyle \frac{{\theta }_n}{{\alpha }_n}}\parallel x_n-x_{n-1}\parallel +{\displaystyle \frac{|{\delta }_n|}{{\alpha }_n}}·\parallel x_{n-1}-x_{n-2}\parallel +\parallel S(p)-p\parallel )]\hfill \\ \hfill & \le & max\{\parallel x_n-p\parallel ,{\displaystyle \frac{1}{1-k}}({\displaystyle \frac{{\theta }_n}{{\alpha }_n}}\parallel x_n-x_{n-1}\parallel +{\displaystyle \frac{|{\delta }_n|}{{\alpha }_n}}·\parallel x_{n-1}-x_{n-2}\parallel +\parallel S(p)-p\parallel )\}.\hfill \end{array}$$

(17)

By (6) and the condition (i), we have

$${\displaystyle \frac{{\theta }_n}{{\alpha }_n}}\parallel x_n-x_{n-1}\parallel \to 0 \mathrm{as} n\to \infty .$$

Hence, there is $M_1>0$, such that

$${\displaystyle \frac{{\theta }_n}{{\alpha }_n}}\parallel x_n-x_{n-1}\parallel <M_1,\forall n\in \mathbb{N}.$$

(18)

Similarly, we also have that

$${\displaystyle \frac{|{\delta }_n|}{{\alpha }_n}}·\parallel x_{n-1}-x_{n-2}\parallel \to 0 \mathrm{as} n\to \infty ,$$

and

$${\displaystyle \frac{|{\delta }_n|}{{\alpha }_n}}·\parallel x_{n-1}-x_{n-2}\parallel <M_2,\exists M_2>0\forall n\in \mathbb{N}.$$

(19)

From (17), we obtain

$$\begin{array}{ccc}\parallel x_{n+1}-p\parallel \hfill & \le & max\{\parallel x_n-p\parallel ,{\displaystyle \frac{1}{1-k}}(M_1+M_2+\parallel S(p)-p\parallel )\}\hfill \\ \hfill & & ⋮\hfill \\ \hfill & \le & max\{\parallel x_1-p\parallel ,{\displaystyle \frac{1}{1-k}}(M_1+M_2+\parallel S(p)-p\parallel )\}.\hfill \end{array}$$

This implies that $\{x_n\}$ is bounded and so are $\{y_n\},\{z_n\},\{w_n\},\{v_n\}$, and $\{S(x_n)\}$.

By (8), we have

$$\begin{array}{ccc}\parallel y_n{-p\parallel }^2\hfill & =& \parallel x_n+{\theta }_n(x_n-x_{n-1})+{\delta }_n(x_{n-1}-x_{n-2}){-p\parallel }^2\hfill \\ \hfill & =& \parallel x_n{-p\parallel }^2+2{\theta }_n⟨x_n-p,x_n-x_{n-1}⟩\hfill \\ \hfill & & +2{\delta }_n⟨x_n-p,x_{n-1}-x_{n-2}⟩\hfill \\ \hfill & & +\parallel {\theta }_n(x_n-x_{n-1})+{\delta }_n(x_{n-1}-x_{n-2}){\parallel }^2\hfill \\ \hfill & =& \parallel x_n{-p\parallel }^2+2{\theta }_n⟨x_n-p,x_n-x_{n-1}⟩\hfill \\ \hfill & & +2{\delta }_n⟨x_n-p,x_{n-1}-x_{n-2}⟩\hfill \\ \hfill & & +{\theta }_n^2{\parallel x_n-x_{n-1}\parallel }^2+2{\theta }_n{\delta }_n⟨x_n-x_{n-1},x_{n-1}-x_{n-2}⟩\hfill \\ \hfill & & +{\delta }_n^2{\parallel x_{n-1}-x_{n-2}\parallel }^2\hfill \\ \hfill & \le & \parallel x_n{-p\parallel }^2+2{\theta }_n\parallel x_n-p\parallel ·\parallel x_n-x_{n-1}\parallel \hfill \\ \hfill & & +2|{\delta }_n|·\parallel x_n-p\parallel ·\parallel x_{n-1}-x_{n-2}\parallel \hfill \\ \hfill & & +{\theta }_n^2\parallel x_n-x_{n-1}{\parallel }^2 + 2{\theta }_n|{\delta }_n|·\parallel x_n-x_{n-1}\parallel ·\parallel x_{n-1}-x_{n-2}\parallel \hfill \\ \hfill & & +{\delta }_n^2{\parallel x_{n-1}-x_{n-2}\parallel }^2.\hfill \end{array}$$

(20)

From (9) and (20), we have

$$\begin{array}{ccc}\parallel z_n{-p\parallel }^2\hfill & =& \parallel y_n+{\eta }_n(x_n-y_n){-p\parallel }^2\hfill \\ \hfill & \le & {\eta }_n\parallel x_n{-p\parallel }^2+(1-{\eta }_n){\parallel y_n-p\parallel }^2\hfill \\ \hfill & \le & {\eta }_n\parallel x_n{-p\parallel }^2+(1-{\eta }_n)[\parallel x_n{-p\parallel }^2\hfill \\ \hfill & & +2{\theta }_n\parallel x_n-p\parallel ·\parallel x_n-x_{n-1}\parallel + 2|{\delta }_n|·\parallel x_n-p\parallel ·\parallel x_{n-1}-x_{n-2}\parallel \hfill \\ \hfill & & +{\theta }_n^2\parallel x_n-x_{n-1}{\parallel }^2+2{\theta }_n|{\delta }_n|·\parallel x_n-x_{n-1}\parallel ·\parallel x_{n-1}-x_{n-2}\parallel \hfill \\ \hfill & & +{\delta }_n^2\parallel x_{n-1}-x_{n-2}{\parallel }^2]\hfill \\ \hfill & \le & \parallel x_n{-p\parallel }^2+2{\theta }_n\parallel x_n-p\parallel ·\parallel x_n-x_{n-1}\parallel \hfill \\ \hfill & & +2|{\delta }_n|·\parallel x_n-p\parallel ·\parallel x_{n-1}-x_{n-2}\parallel + {\theta }_n^2{\parallel x_n-x_{n-1}\parallel }^2\hfill \\ \hfill & & +2{\theta }_n|{\delta }_n|·\parallel x_n-x_{n-1}\parallel ·\parallel x_{n-1}-x_{n-2}\parallel + {\delta }_n^2{\parallel x_{n-1}-x_{n-2}\parallel }^2.\hfill \end{array}$$

(21)

By the nonexpansiveness of $J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)$ and (21), we have

$$\begin{array}{ccc}\parallel v_n{-p\parallel }^2\hfill & =& \parallel (1-{\beta }_n)x_n+{\beta }_n{J}_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n{-p\parallel }^2\hfill \\ \hfill & =& \parallel (1-{\beta }_n)(x_n-p)+{\beta }_n(J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)p){\parallel }^2\hfill \\ \hfill & \le & (1-{\beta }_n)\parallel x_n{-p\parallel }^2+{\beta }_n{\parallel z_n-p\parallel }^2\hfill \\ \hfill & & -{\beta }_n(1-{\beta }_n){\parallel x_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n\parallel }^2\hfill \\ \hfill & \le & (1-{\beta }_n)\parallel x_n{-p\parallel }^2+{\beta }_n[\parallel x_n{-p\parallel }^2+2{\theta }_n\parallel x_n-p\parallel ·\parallel x_n-x_{n-1}\parallel \hfill \\ \hfill & & +2|{\delta }_n|·\parallel x_n-p\parallel ·\parallel x_{n-1}-x_{n-2}\parallel + {\theta }_n^2{\parallel x_n-x_{n-1}\parallel }^2\hfill \\ \hfill & & +2{\theta }_n|{\delta }_n|·\parallel x_n-x_{n-1}\parallel ·\parallel x_{n-1}-x_{n-2}\parallel + {\delta }_n^2\parallel x_{n-1}-x_{n-2}{\parallel }^2]\hfill \\ \hfill & & -{\beta }_n(1-{\beta }_n){\parallel x_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n\parallel }^2\hfill \\ \hfill & \le & \parallel x_n{-p\parallel }^2+2{\theta }_n\parallel x_n-p\parallel ·\parallel x_n-x_{n-1}\parallel \hfill \\ \hfill & & +2|{\delta }_n|·\parallel x_n-p\parallel ·\parallel x_{n-1}-x_{n-2}\parallel + {\theta }_n^2{\parallel x_n-x_{n-1}\parallel }^2\hfill \\ \hfill & & +2{\theta }_n|{\delta }_n|·\parallel x_n-x_{n-1}\parallel ·\parallel x_{n-1}-x_{n-2}\parallel + {\delta }_n^2{\parallel x_{n-1}-x_{n-2}\parallel }^2\hfill \\ \hfill & & -{\beta }_n(1-{\beta }_n){\parallel x_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n\parallel }^2.\hfill \end{array}$$

(22)

Using (20) and (22), we have

$$\begin{array}{ccc}\parallel w_n{-p\parallel }^2\hfill & =& \parallel (1-{\tau }_n)J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)y_n+{\tau }_n{J}_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)v_n{-p\parallel }^2\hfill \\ \hfill & \le & (1-{\tau }_n){\parallel J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)y_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)p\parallel }^2\hfill \\ \hfill & & +{\tau }_n{\parallel J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)v_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)p\parallel }^2\hfill \\ \hfill & \le & (1-{\tau }_n)\parallel y_n{-p\parallel }^2+{\tau }_n{\parallel v_n-p\parallel }^2\hfill \\ \hfill & \le & \parallel x_n{-p\parallel }^2+2{\theta }_n\parallel x_n-p\parallel ·\parallel x_n-x_{n-1}\parallel \hfill \\ \hfill & & +2|{\delta }_n|·\parallel x_n-p\parallel ·\parallel x_{n-1}-x_{n-2}\parallel + {\theta }_n^2{\parallel x_n-x_{n-1}\parallel }^2\hfill \\ \hfill & & +2{\theta }_n|{\delta }_n|·\parallel x_n-x_{n-1}\parallel ·\parallel x_{n-1}-x_{n-2}\parallel + {\delta }_n^2{\parallel x_{n-1}-x_{n-2}\parallel }^2\hfill \\ \hfill & & -{\tau }_n{\beta }_n(1-{\beta }_n){\parallel x_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n\parallel }^2.\hfill \end{array}$$

(23)

It follows from Lemma 1 and (23), that

$$\begin{array}{ccc}\parallel x_{n+1}{-p\parallel }^2\hfill & =& \parallel {\alpha }_{n}S(x_n)+(1-{\alpha }_n)w_n{-p\parallel }^2\hfill \\ \hfill & =& \parallel {\alpha }_n(S(x_n)-S(p))+(1-{\alpha }_n)(w_n-p)+{\alpha }_n(S(p)-p){\parallel }^2\hfill \\ \hfill & \le & \parallel {\alpha }_n(S(x_n)-S(p))+(1-{\alpha }_n)(w_n-p){\parallel }^2\hfill \\ \hfill & & +2{\alpha }_n⟨S(p)-p,x_{n+1}-p⟩\hfill \\ \hfill & \le & {\alpha }_n\parallel S(x_n)-S(p){\parallel }^2+(1-{\alpha }_n){\parallel w_n-p\parallel }^2\hfill \\ \hfill & & +2{\alpha }_n⟨S(p)-p,x_{n+1}-p⟩\hfill \\ \hfill & \le & {\alpha }_n{k}^2\parallel x_n{-p\parallel }^2+(1-{\alpha }_n){\parallel w_n-p\parallel }^2\hfill \\ \hfill & & +2{\alpha }_n⟨S(p)-p,x_{n+1}-p⟩\hfill \\ \hfill & \le & {\alpha }_{n}k\parallel x_n{-p\parallel }^2+(1-{\alpha }_n)[\parallel x_n{-p\parallel }^2+2{\theta }_n\parallel x_n-p\parallel ·\parallel x_n-x_{n-1}\parallel \hfill \\ \hfill & & +2|{\delta }_n|·\parallel x_n-p\parallel ·\parallel x_{n-1}-x_{n-2}\parallel + {\theta }_n^2{\parallel x_n-x_{n-1}\parallel }^2\hfill \\ \hfill & & +2{\theta }_n|{\delta }_n|·\parallel x_n-x_{n-1}\parallel ·\parallel x_{n-1}-x_{n-2}\parallel + {\delta }_n^2\parallel x_{n-1}-x_{n-2}{\parallel }^2]\hfill \\ \hfill & & +2{\alpha }_n⟨S(p)-p,x_{n+1}-p⟩\hfill \\ \hfill & & -(1-{\alpha }_n){\tau }_n{\beta }_n(1-{\beta }_n){\parallel x_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n\parallel }^2\hfill \\ \hfill & =& (1-{\alpha }_n(1-k))\parallel x_n{-p\parallel }^2+2{\theta }_n\parallel x_n-p\parallel ·\parallel x_n-x_{n-1}\parallel \hfill \\ \hfill & & +2|{\delta }_n|·\parallel x_n-p\parallel ·\parallel x_{n-1}-x_{n-2}\parallel + {\theta }_n^2{\parallel x_n-x_{n-1}\parallel }^2\hfill \\ \hfill & & +2{\theta }_n|{\delta }_n|·\parallel x_n-x_{n-1}\parallel ·\parallel x_{n-1}-x_{n-2}\parallel + {\delta }_n^2{\parallel x_{n-1}-x_{n-2}\parallel }^2\hfill \\ \hfill & & +2{\alpha }_n⟨S(p)-p,x_{n+1}-p⟩\hfill \\ \hfill & & -(1-{\alpha }_n){\tau }_n{\beta }_n(1-{\beta }_n){\parallel x_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n\parallel }^2\hfill \\ \hfill & =& (1-{\alpha }_n(1-k)){\parallel x_n-p\parallel }^2\hfill \\ \hfill & & +{\theta }_n\parallel x_n-x_{n-1}\parallel [2\parallel x_n-p\parallel + {\theta }_n\parallel x_n-x_{n-1}\parallel + 2|{\delta }_n|·\parallel x_{n-1}-x_{n-2}\parallel ]\hfill \\ \hfill & & +|{\delta }_n|·\parallel x_{n-1}-x_{n-2}\parallel [2\parallel x_n-p\parallel + |{\delta }_n|·\parallel x_{n-1}-x_{n-2}\parallel ]\hfill \\ \hfill & & +2{\alpha }_n⟨S(p)-p,x_{n+1}-p⟩\hfill \\ \hfill & & -(1-{\alpha }_n){\tau }_n{\beta }_n(1-{\beta }_n){\parallel x_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n\parallel }^2.\hfill \end{array}$$

(24)

Since

$${\theta }_n\parallel x_n-x_{n-1}\parallel = {\alpha }_n·{\displaystyle \frac{{\theta }_n}{{\alpha }_n}}\parallel x_n-x_{n-1}\parallel \to 0 \mathrm{as} n\to \infty ,$$

and

$$|{\delta }_n|·\parallel x_{n-1}-x_{n-2}\parallel + {\alpha }_n·{\displaystyle \frac{|{\delta }_n|}{{\alpha }_n}}\parallel x_{n-1}-x_{n-2}\parallel \to 0 \mathrm{as} n\to \infty ,$$

there exist the positive constants $M_3$ and $M_4$, such that for all $n\ge 1$,

$${\theta }_n\parallel x_n-x_{n-1}\parallel \le M_3,$$

(25)

$$|{\delta }_n|·\parallel x_{n-1}-x_{n-2}\parallel \le M_4.$$

(26)

We may deduce from (24) that for all $n\ge 1$,

$$\begin{array}{ccc}\parallel x_{n+1}{-p\parallel }^2\hfill & \le & (1-{\alpha }_n(1-k)){\parallel x_n-p\parallel }^2\hfill \\ \hfill & & +5M_5{\theta }_n\parallel x_n-x_{n-1}{\parallel }^2+3M_5|{\delta }_n|·\parallel x_{n-1}-x_{n-2}\parallel \hfill \\ \hfill & & +2{\alpha }_n⟨S(p)-p,x_{n+1}-p⟩\hfill \\ \hfill & & -(1-{\alpha }_n){\tau }_n{\beta }_n(1-{\beta }_n){\parallel x_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n\parallel }^2\hfill \\ \hfill & =& (1-{\alpha }_n(1-k)){\parallel x_n-p\parallel }^2\hfill \\ \hfill & & +{\alpha }_n(1-k)[{\displaystyle \frac{5M_5}{1-k}}·{\displaystyle \frac{{\theta }_n}{{\alpha }_n}}{\parallel x_n-x_{n-1}\parallel }^2\hfill \\ \hfill & & +{\displaystyle \frac{3M_5}{1-k}}·{\displaystyle \frac{|{\delta }_n|}{{\alpha }_n}}·\parallel x_{n-1}-x_{n-2}\parallel +{\displaystyle \frac{2}{1-k}}⟨S(p)-p,x_{n+1}-p⟩]\hfill \\ \hfill & & -(1-{\alpha }_n){\tau }_n{\beta }_n(1-{\beta }_n){\parallel x_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n\parallel }^2,\hfill \end{array}$$

(27)

where $M_5=max\{\underset{n}{sup}\parallel x_n-p\parallel ,M_3,M_4\}$.

From (27), we set

$$a_n:={\parallel x_n-p\parallel }^2,$$

$$b_n:={\displaystyle \frac{1}{1-k}}[5M_5{\displaystyle \frac{{\theta }_n}{{\alpha }_n}}\parallel x_n-x_{n-1}{\parallel }^2+3M_5{\displaystyle \frac{|{\delta }_n|}{{\alpha }_n}}\parallel x_{n-1}-x_{n-2}\parallel +2⟨S(p)-p,x_{n+1}-p⟩]$$

and

$${\epsilon }_n:={\alpha }_n(1-k).$$

Hence, we obtain

$$a_{n+1}=(1-{\epsilon }_n){\alpha }_n+{\epsilon }_n{b}_n.$$

Suppose there is a subsequence $\{a_{n_i}\}$ of $\{a_n\}$, satisfying

$$\underset{i\to \infty }{lim\; inf}(a_{n_i+1}-a_{n_i})\ge 0.$$

It follows from (27) that

$$(1-{\alpha }_n){\tau }_n{\beta }_n(1-{\beta }_n){\parallel x_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n\parallel }^2\le a_n-a_{n+1}+{\alpha }_n(1-k)M,$$

(28)

where $M=\underset{n}{sup}{b}_n.$

By conditions (ii), (iii), and (iv), we obtain

$$\begin{array}{cccc}\hfill & & \hfill \underset{i\to \infty }{lim\; sup}& (1-{\alpha }_n){\tau }_n{\beta }_n(1-{\beta }_n){\parallel x_n-J_{{\mathit{\gamma }}_n}^B(I-{\mathit{\gamma }}_{n}A)z_n\parallel }^2\hfill \\ \hfill & & \hfill \le & \underset{i\to \infty }{lim\; sup}(a_{n_i}-a_{n_i+1}+(1-k)M\underset{i\to \infty }{lim}{\alpha }_{n_i}\hfill \\ \hfill & & \hfill =& -\underset{i\to \infty }{lim\; inf}(a_{n_i}-a_{n_i+1})\hfill \\ \hfill & & \hfill \le & 0,\hfill \end{array}$$

(29)

which implies

$$\parallel x_{n_i}-J_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)z_{n_i}{\parallel }^2=0.$$

(30)

From (8), we have

$$\parallel y_{n_i}-x_{n_i}\parallel + {\theta }_{n_i}\parallel x_{n_i}-x_{n_i-1}\parallel +|{\delta }_{n_i}|·\parallel x_{n_i-1}-x_{n_i-2}\parallel \to 0,$$

(31)

as $i\to \infty$.

From (9) and (31), we have

$$\parallel z_{n_i}-y_{n_i}\parallel + {\eta }_{n_i}\parallel x_{n_i}-y_{n_i}\parallel \to 0,$$

(32)

and

$$\parallel z_{n_i}-x_{n_i}\parallel + (1-{\eta }_{n_i})\parallel y_{n_i}-x_{n_i}\parallel \to 0,$$

(33)

as $i\to \infty .$

From (10), (30), and (32), we have

$$\begin{array}{ccc}\parallel v_{n_i}-z_{n_i}\parallel \hfill & =& \parallel (1-{\beta }_{n_i})x_{n_i}+{\beta }_{n_i}{J}_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)z_{n_i}-z_{n_i}\parallel \hfill \\ \hfill & \le & (1-{\beta }_{n_i})\parallel x_{n_i}-z_{n_i}\parallel + {\beta }_{n_i}\parallel J_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)z_{n_i}-z_{n_i}\parallel \hfill \\ \hfill & \to & 0,\hfill \end{array}$$

(34)

and

$$\begin{array}{ccc}\parallel v_{n_i}-x_{n_i}\parallel \hfill & =& \parallel (1-{\beta }_{n_i})x_{n_i}+{\beta }_{n_i}{J}_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)z_{n_i}-x_{n_i}\parallel \hfill \\ \hfill & \le & {\beta }_{n_i}\parallel x_{n_i}-J_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)z_{n_i}\parallel \hfill \\ \hfill & \to & 0,\hfill \end{array}$$

(35)

as $i\to \infty .$

Using (30), (32), (34), and (35), we obtain

$$\begin{array}{ccc}\parallel w_{n_i}-v_{n_i}\parallel \hfill & =& \parallel (1-{\tau }_{n_i})J_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)y_{n_i}+{\tau }_{n_i}{J}_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)v_{n_i}-v_{n_i}\parallel \hfill \\ \hfill & \le & (1-{\tau }_{n_i})\parallel J_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)y_{n_i}-v_{n_i}\parallel \hfill \\ \hfill & & +{\tau }_{n_i}\parallel J_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)v_{n_i}-v_{n_i}\parallel \hfill \\ \hfill & \le & (1-{\tau }_{n_i})\parallel J_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)y_{n_i}-J_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)z_{n_i}\parallel \hfill \\ \hfill & & +(1-{\tau }_{n_i})\parallel J_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)z_{n_i}-v_{n_i}\parallel \hfill \\ \hfill & & +{\tau }_{n_i}\parallel J_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)v_{n_i}-J_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)z_{n_i}\parallel \hfill \\ \hfill & & +{\tau }_{n_i}\parallel J_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)z_{n_i}-v_{n_i}\parallel \hfill \\ \hfill & \le & (1-{\tau }_{n_i})\parallel y_{n_i}-z_{n_i}\parallel + \parallel J_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)z_{n_i}-v_{n_i}\parallel \hfill \\ \hfill & & +{\tau }_{n_i}\parallel v_{n_i}-z_{n_i}\parallel \hfill \\ \hfill & \le & (1-{\tau }_{n_i})\parallel y_{n_i}-z_{n_i}\parallel + \parallel J_{{\mathit{\gamma }}_{n_i}}^B(I-{\mathit{\gamma }}_{n_i}A)z_{n_i}-x_{n_i}\parallel \hfill \\ \hfill & & +\parallel x_{n_i}-v_{n_i}\parallel + {\tau }_{n_i}\parallel v_{n_i}-z_{n_i}\parallel \hfill \\ \hfill & \to & 0,\hfill \end{array}$$

(36)

as $i\to \infty .$

Using condition (ii), (31), (32), (34), and (36), we obtain

$$\begin{array}{ccc}\parallel x_{n_i+1}-x_{n_i}\parallel \hfill & =& \parallel {\alpha }_{n_i}S(x_{n_i})+(1-{\alpha }_{n_i})w_{n_i}-x_{n_i}\parallel \hfill \\ \hfill & \le & {\alpha }_{n_i}\parallel S(x_{n_i})-x_{n_i}\parallel + (1-{\alpha }_{n_i})\parallel w_{n_i}-x_{n_i}\parallel \hfill \\ \hfill & \le & {\alpha }_{n_i}\parallel S(x_{n_i})-x_{n_i}\parallel + \parallel w_{n_i}-y_{n_i}\parallel + \parallel y_{n_i}-x_{n_i}\parallel \hfill \\ \hfill & \le & {\alpha }_{n_i}\parallel S(x_{n_i})-x_{n_i}\parallel + \parallel w_{n_i}-z_{n_i}\parallel + \parallel z_{n_i}-y_{n_i}\parallel \hfill \\ \hfill & & +\parallel y_{n_i}-x_{n_i}\parallel \hfill \\ \hfill & \le & {\alpha }_{n_i}\parallel S(x_{n_i})-x_{n_i}\parallel + \parallel w_{n_i}-v_{n_i}\parallel + \parallel v_{n_i}-z_{n_i}\parallel \hfill \\ \hfill & & +\parallel z_{n_i}-y_{n_i}\parallel + \parallel y_{n_i}-x_{n_i}\parallel \hfill \\ \hfill & \to & 0,\hfill \end{array}$$

(37)

as $i\to \infty .$

We next show that $\underset{i\to \infty }{lim\; sup}⟨S(p)-p,x_{n_i+1}-p⟩\le 0$.

Let $\{x_{n_{i_j}}\}$ be a subsequence of $\{x_{n_i}\}$, such that

$$\underset{j\to \infty }{lim}⟨S(p)-p,x_{n_{i_j}+1}-p⟩=\underset{i\to \infty }{lim\; sup}⟨S(p)-p,x_{n_i+1}-p⟩.$$

Since $\{x_{n_{i_j}}\}$ is bounded, there exists a subsequence $\{x_{n_{i_{j_k}}}\}$ of $\{x_{n_{i_j}}\}$, such that $x_{n_{i_{j_k}}}\rightharpoonup x\in \mathcal{H}$. Without loss of generality, we may assume that $x_{n_{i_j}}\rightharpoonup x$.

From ${\gamma }_{n_{i_j}}\in (0,2\beta )$, we know that the mapping $J_{{\mathit{\gamma }}_{n_{i_j}}}^B(I-{\mathit{\gamma }}_{n_{i_j}}A)$ is nonexpansive. Due to (30) and (33), the following result is obtained:

$$\begin{array}{ccc}\parallel x_{n_{i_j}}-J_{{\mathit{\gamma }}_{n_{i_j}}}^B(I-{\mathit{\gamma }}_{n_{i_j}}A)x_{n_{i_j}}\parallel \hfill & \le & \parallel x_{n_{i_j}}-J_{{\mathit{\gamma }}_{n_{i_j}}}^B(I-{\mathit{\gamma }}_{n_{i_j}}A)z_{n_{i_j}}\parallel \hfill \\ \hfill & & +\parallel J_{{\mathit{\gamma }}_{n_{i_j}}}^B(I-{\mathit{\gamma }}_{n_{i_j}}A)z_{n_{i_j}}-J_{{\mathit{\gamma }}_{n_{i_j}}}^B(I-{\mathit{\gamma }}_{n_{i_j}}A)x_{n_{i_j}}\parallel \hfill \\ \hfill & \le & \parallel x_{n_{i_j}}-J_{{\mathit{\gamma }}_{n_{i_j}}}^B(I-{\mathit{\gamma }}_{n_{i_j}}A)z_{n_{i_j}}\parallel + \parallel z_{n_{i_j}}-x_{n_{i_j}}\parallel \hfill \\ \hfill & \to & 0\mathrm{as} j\to \infty .\hfill \end{array}$$

(38)

Using Lemmas 2 and 3, we obtain $x\in F(J_{{\mathit{\gamma }}_{n_{i_j}}}^B(I-{\mathit{\gamma }}_{n_{i_j}}A))={(A+B)}^{-1}(0).$

Since $\underset{i\to \infty }{lim}\parallel x_{n_i+1}-x_{n_i}\parallel =0,$ we have $x_{n_{i_j}+1}\rightharpoonup x.$ From $p=P_{{(A+B)}^{-1}(0)}S(p)$, it is implied Proposition 1 that

$$\begin{array}{ccc}\underset{i\to \infty }{lim\; sup}⟨S(p)-p,x_{n_i+1}-p⟩\hfill & =& \underset{j\to \infty }{lim}⟨S(p)-p,x_{n_{i_j}+1}-p⟩\hfill \\ \hfill & =& ⟨S(p)-p,x-p⟩\hfill \\ \hfill & \le & 0.\hfill \end{array}$$

(39)

By Lemma 4, we can conclude that $x_n\to p.$    □

Remark 1.

Note that Algorithm 7 is a modification of Algorithm 6 in order to accelerate its convergence by using a double inertial step at the first step and the viscosity approximation method at the final step. Moreover, our proposed algorithm has a strong convergence result while Algorithm 6 obtained only weak convergence. Furthermore, Algorithm 7 can be applied to solve convex bilevel optimization problems, as seen in Theorem 2, while Algorithm 6 cannot be used to solve such problems.

Next, we employ the Bilevel Double Inertial Forward–Backward Algorithm (BDIFBA, Algorithm 8) to solve the convex bilevel optimization Problem (1), by replacing A and B in Algorithm 7 with $\nabla f$ and $\partial g$, respectively.

Algorithm 8 BDIFBA

Initialization: Choose $x_1,x_0,x_{-1}\in \mathcal{H},\{{\gamma }_n\}\subset (0,2\beta )$ and $\{{\alpha }_n\},\{{\beta }_n\},\{{\eta }_n\},\{{\tau }_n\}\subset (0,1)$. Take $\{{\xi }_n\},\{{\mu }_n\}\subset (0,\infty )$ and $\{{\rho }_n\}\subset (-\infty ,0)$. Iterative steps: For $n\ge 1$, calculate $x_{n+1}$ as follows: Step 1. Compute the inertial step: $$\begin{array}{cc}{\theta }_n\hfill & =\left\{\begin{array}{cc}min\{{\mu }_n,{\displaystyle \frac{{\xi }_n}{\parallel x_n-x_{n-1}\parallel }}\},\hfill & \mathrm{if}{x}_n\ne x_{n-1};\hfill \\ {\mu }_n,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$ (40) and $$\begin{array}{cc}{\delta }_n\hfill & =\left\{\begin{array}{cc}max\{{\rho }_n,{\displaystyle \frac{-{\xi }_n}{\parallel x_{n-1}-x_{n-2}\parallel }}\},\hfill & \mathrm{if}{x}_{n-1}\ne x_{n-2};\hfill \\ {\rho }_n,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$ (41) Step 2. Compute $$y_n=x_n+{\theta }_n(x_n-x_{n-1})+{\delta }_n(x_{n-1}-x_{n-2}),$$ (42) $$z_n=y_n+{\eta }_n(x_n-y_n),$$ (43) $$v_n=(1-{\beta }_n)x_n+{\beta }_n{prox}_{\gamma g}(I-{\gamma }_{n}f)z_n,$$ (44) $$w_n=(1-{\tau }_n){prox}_{{\gamma }_{n}g}(I-{\gamma }_{n}f)y_n+{\tau }_n{prox}_{{\gamma }_{n}g}(I-{\gamma }_{n}f)v_n,$$ (45) $$x_{n+1}={\alpha }_n(I-s\nabla \varphi )(x_n)+(1-{\alpha }_n)w_n.$$ (46) Set n := n + 1 and return to Step 1.

The following result is obtained directly by Theorem 1.

Theorem 2.

Let $\{x_n\}$ be a sequence generated by Algorithm 8 with the same condition as in Theorem 1. Then, $x_n\to p\in \Gamma$ where $p=P_{\Gamma }S(p)$.

Proof. 

Set $S=I-s\nabla \varphi$ in Theorem 1. From Proposition 2, we know that $I-s\nabla \varphi$ is a contraction. By Theorem 1, we obtain that $x_n\to p\in \Gamma ,$ where $p=P_{\Gamma }S(p)$. From Proposition 1, it can be obtained that for any $x\in \Omega$,

$$\begin{array}{ccc}\hfill 0& \ge & ⟨S(p)-p,x-p)⟩\hfill \\ \hfill & =& ⟨(p-s\nabla \varphi (p))-p,x-p⟩\hfill \\ \hfill & =& -s⟨\nabla \varphi (p),x-p⟩\hfill \end{array}$$

(47)

hence $⟨\nabla \varphi (p),x-p⟩\ge 0$, that is, $p\in \Omega .$ □

4. Application

In this section, we apply our proposed algorithm to improve the training of deep learning models by reformulating their training tasks as structured convex optimization problems. Our approach is based on fixed-point theory, which provides strong theoretical guarantees for convergence and solution reliability. This makes the training process more stable, efficient, and robust, especially in the presence of noise or ill-conditioned data.

We focus on a class of models called Extreme Learning Machines (ELM) and their deeper extensions, Two-Hidden-Layer ELM (TELM). These models are known for their fast training and competitive accuracy. Unlike traditional neural networks, ELMs randomly assign hidden layer weights and only compute output weights, typically by solving a least-squares problem.

However, when the hidden layer output matrix is ill-conditioned or the data is noisy, direct pseudoinverse computations become unstable and prone to overfitting. To address this, we reformulate the training process as a convex minimization problem with regularization. This structure naturally fits into the framework of fixed-point problems, allowing us to apply our algorithm without relying on explicit matrix inversion.

4.1. Application to ELM

ELM is a neural network model initially proposed by Huang et al. [17]. ELM is well-known for its rapid training capability and strong generalization performance. By integrating our algorithm into the ELM framework, we aim to boost both optimization efficiency and predictive accuracy.

Let us define the training dataset as $(x_i,t_i)\in {\mathbb{R}}^n\times {\mathbb{R}}^m:i=1,2,\dots ,s$, consisting of s input–target pairs, where $x_i$ denotes the input vector and $t_i$ denotes the associated target output.

ELM is designed for Single-Layer Feedforward Networks (SLFNs) and operates based on the following functional form:

$$o_i=\sum _{j=1}^h{\eta }_{j}G(⟨{\omega }_j,x_i⟩+b_j),i=1,\dots ,s,$$

where $o_i$ is the predicted output, h denotes the number of hidden neurons, $G(·)$ is the activation function, ${\omega }_j$ and ${\eta }_j$ are weight vectors for input and output connections of the j-th hidden node, and $b_j$ is the corresponding bias term.

Let the hidden layer output matrix $\mathbf{H}\in {\mathbb{R}}^{s\times h}$ be defined as

$${\mathbf{H}}_{ij}=G(⟨{\omega }_j,x_i⟩+b_j)i=1,\dots ,s,j=1,\dots ,h.$$

The training objective is to find a solution that best approximates the output target:

$$t_i=\sum _{j=1}^h{\eta }_{j}G(⟨{\omega }_j,x_i⟩+b_j),i=1,2,\dots ,s.$$

which can be compactly written in matrix form as

$$\mathbf{H}u=\mathbf{T},$$

(48)

where $u={[{\eta }_1^T,\dots ,{\eta }_h^T]}^T$ is the output weight vector and $\mathbf{T}={[t_1^T,\dots ,t_s^T]}^T$ is the desired output matrix.

To enhance generalization and reduce overfitting, a LASSO regularization term is introduced. The resulting optimization problem becomes

$$\underset{u}{min}{\parallel \mathbf{H}u-\mathbf{T}\parallel }_2^2+\lambda {\parallel u\parallel }_1,$$

(49)

where ${\parallel ·\parallel }_1$ denotes the $l_1$-norm and $\lambda >0$ is a regularization coefficient that controls sparsity.

4.2. Application to TELM

TELM is an extension of the traditional ELM that improves learning capacity by incorporating two hidden layers. Unlike conventional backpropagation-based multi-layer networks, TELM retains the fast training characteristics of ELM by leveraging analytic solutions in both stages. It is particularly suitable for modeling complex nonlinear relationships in high-dimensional data while avoiding the computational cost of iterative optimization.

A work by Janngam et al. [18] demonstrated that TELM, when trained using their proposed algorithm, not only converges significantly faster than standard ELM but also achieves higher classification accuracy on various medical and benchmark datasets. Additionally, earlier work by Qu et al. [19] showed that TELM consistently outperforms traditional ELM, especially in nonlinear and high-dimensional settings, by yielding better average accuracy with fewer hidden neurons.

These cumulative findings reinforce the choice of TELM as the core learning model for our study, particularly when enhanced with the proposed algorithm.

Let the training set be defined as $\{(x_i,t_i)\in {\mathbb{R}}^n\times {\mathbb{R}}^m:i=1,2,\dots ,s\}$, where $x_i$ is the input vector and $t_i$ is the corresponding target output.

 

Stage 1: Initial Feature Transformation and Output Weights.

To simplify the initialization process, TELM begins by temporarily combining the two hidden layers into a single equivalent hidden layer. The combinated hidden layer matrix $\mathbf{H}$ is defined as

$$\mathbf{H}=G(\mathbf{X}\mathbf{W}+\mathbf{B}),$$

(50)

where $\mathbf{X}\in {\mathbb{R}}^{s\times n}$ is the input matrix, $\mathbf{W}\in {\mathbb{R}}^{n\times h}$ is the randomly initialized weight matrix for the first hidden layer, $\mathbf{B}\in {\mathbb{R}}^{s\times h}$ is the bias matrix, and $G(·)$ is the activation function.

The output weights $u\in {\mathbb{R}}^{h\times m}$ connecting the hidden layer to the output layer are determined based on the linear system:

$$\mathbf{H}u=\mathbf{T},$$

(51)

where $\mathbf{T}\in {\mathbb{R}}^{s\times m}$ is the target matrix.

We find the optimal weight u using Algorithms 7 and 8 for solving the convex optimization problem with LASSO regularization as follows:

$$\underset{u}{min}{\parallel \mathbf{H}u-\mathbf{T}\parallel }_2^2+\lambda {\parallel u\parallel }_1,$$

(52)

where $\lambda >0$ is the regularization parameter that controls model complexity and prevents overfitting.

 

Stage 2: Separation and Refinement of Hidden Layers.

After computing the initial output weights u from the first stage using (52), the two hidden layers are separated to allow independent refinement.

To estimate the expected output of the second hidden layer, denoted as ${\mathbf{H}}_1\in {\mathbb{R}}^{s\times h}$, we express that it satisfies the following equation:

$${\mathbf{H}}_{1}u=\mathbf{T}.$$

(53)

However, rather than computing ${\mathbf{H}}_1$ directly from matrix inversion, we apply our proposed algorithm to solve the following convex optimization problem with LASSO regularization:

$$\underset{{\mathbf{H}}_1}{min}\parallel {\mathbf{H}}_1{u-\mathbf{T}\parallel }_2^2+\lambda {\parallel {\mathbf{H}}_1\parallel }_1,$$

(54)

where $\lambda >0$ is the regularization parameter.

Next, TELM updates the weights and bias between the first and second hidden layers, denoted as ${\mathbf{W}}_1\in {\mathbb{R}}^{h\times h}$ and ${\mathbf{B}}_1\in {\mathbb{R}}^{s\times h}$, respectively, using the expected output ${\mathbf{H}}_1$ from (57). Ideally, the following equation describes the connection between layers:

$${\mathbf{H}}_1=G(\mathbf{H}{\mathbf{W}}_1+{\mathbf{B}}_1).$$

(55)

However, since both ${\mathbf{W}}_1$ and ${\mathbf{B}}_1$ are unknown, solving (55) directly is not feasible. To address this, we reformulate the equation as

$${\mathbf{H}}_1=G({\mathbf{H}}_E{\mathbf{W}}_{\mathrm{HE}}),$$

(56)

where ${\mathbf{H}}_E=[\mathbf{1}\mathbf{H}]\in {\mathbb{R}}^{s\times (h+1)}$ is the extended input matrix and ${\mathbf{W}}_{\mathrm{HE}}={[{\mathbf{B}}_1{\mathbf{W}}_1]}^T\in {\mathbb{R}}^{(h+1)\times h}$ combines the weights and biases into a single matrix.

To estimate ${\mathbf{W}}_{\mathrm{HE}}$, we solve the following convex optimization problem with LASSO regularization:

$$\underset{{\mathbf{W}}_{\mathrm{HE}}}{min}\parallel {\mathbf{H}}_E{\mathbf{W}}_{\mathrm{HE}}-G^{-1}({\mathbf{H}}_1){\parallel }_2^2+\lambda {\parallel {\mathbf{W}}_{\mathrm{HE}}\parallel }_1,$$

(57)

where $G^{-1}(·)$ denotes the inverse of the activation function G, and $\lambda >0$ is the regularization parameter.

Finally, using the estimated ${\mathbf{W}}_{\mathrm{HE}}$ from (57), the refined output of the second hidden layer ${\mathbf{H}}_2$ is computed as

$${\mathbf{H}}_2=G({\mathbf{H}}_E{\mathbf{W}}_{\mathrm{HE}}),$$

(58)

where ${\mathbf{H}}_2\in {\mathbb{R}}^{s\times h}$ represents the updated hidden layer of the second hidden layer after adjusting the weights and biases.

 

Final Stage: Output Layer Update.

Finally, TELM updates the output weight matrix $u_{\mathrm{new}}\in {\mathbb{R}}^{h\times m}$, which connects the second hidden layer to the output layer, by solving

$${\mathbf{H}}_2{u}_{\mathrm{new}}=\mathbf{T}.$$

(59)

To obtain $u_{\mathrm{new}}$, we solve the following convex optimization problem using the LASSO technique:

$$\underset{u_{\mathrm{new}}}{min}\parallel {\mathbf{H}}_2{u}_{\mathrm{new}}{-\mathbf{T}\parallel }_2^2+\lambda {\parallel u_{\mathrm{new}}\parallel }_1,$$

(60)

where $\lambda >0$ is the regularization parameter. Once $u_{\mathrm{new}}$ is obtained, the predicted output matrix $\mathbf{Y}\in {\mathbb{R}}^{s\times m}$ is computed as

$$\mathbf{Y}={\mathbf{H}}_2{u}_{\mathrm{new}}.$$

(61)

This approach enhances numerical stability and improves the model ability to handle high-dimensional or noisy real-world data.

4.2.1. Experiments: Data Classification for Minimization Problems

Data classification is a fundamental task in machine learning, where the objective is to assign each input sample to one of several predefined categories. Common applications include medical diagnosis, object recognition, and fraud detection. In this work, we apply our proposed algorithm to train TELM for practical classification tasks.

To evaluate classification performance, we conducted experiments on three benchmark datasets and one real-world medical dataset. Each dataset was divided into 70% training and 30% testing sets. The details of the datasets are summarized in Table 1.

Table 1. Summary of datasets used in the experiments.

Dataset	Total Samples	Training	Testing	Features	Classes
Breast Cancer	683	478	205	9	2
Heart Disease	303	212	91	13	2
Diabetes	436	305	131	8	2
Hypertension	6108	4275	1833	9	2

• Breast Cancer Dataset: A widely used dataset containing features extracted from digitized images of breast masses, used to classify tumors as benign or malignant.
• Heart Disease Dataset: A standard dataset used to predict the presence of heart disease based on clinical attributes.
• Diabetes Dataset: Contains diagnostic data for predicting the onset of diabetes in patients.
• Hypertension Dataset: A real-world dataset collected by Sripat Medical Center, Faculty of Medicine, Chiang Mai University.

Table 2 summarizes the parameter settings for each algorithm compared in our experiments.

Table 2. Parameter settings for each algorithm.

Parameter	Algorithm 7	Algorithm 6
${\mu }_n$	$1-{\displaystyle \frac{1}{n+1}}$	-
${\rho }_n$	$-{\displaystyle \frac{1}{n}}$	-
${\xi }_n$	${\displaystyle \frac{33\times {10}^{20}}{n^2}}$	-
${\beta }_n$	${\displaystyle \frac{n}{2n+1}}$	-
${\tau }_n$	$0.5+{\displaystyle \frac{1}{2n}}$	-
${\alpha }_n$	${\displaystyle \frac{n}{2n+1}}$	-
${\eta }_n$	${\displaystyle \frac{1}{n+2}}$	${\displaystyle \frac{1}{n+2}}$
${\gamma }_n$	${\displaystyle \frac{1}{L_f}}$	${\displaystyle \frac{1}{L_f}}$
$\overline{{\theta }_n}$	-	${\displaystyle \frac{1}{\parallel x_n-x_{n-1}{\parallel }^5+n^5}}$

In addition, the following settings were consistently applied across all experimental setups:

• Regularization parameter: $\lambda ={10}^{-5}$.
• Activation function: Sigmoid, $g(x)={\displaystyle \frac{1}{1+e^{-x}}}$.
• Number of hidden nodes: $h=30$.
• Contraction mapping: $S={\displaystyle \frac{1}{3}}x+1$.
• In Algorithm 6, ${\theta }_n$ is defined by

  $$\begin{array}{cc}{\theta }_n\hfill & =\left\{\begin{array}{cc}{\displaystyle \frac{\overline{{\theta }_n}}{n^2\parallel x_n-x_{n-1}\parallel }}\hfill & \mathrm{if}{x}_n\ne x_{n-1}\hfill \\ \overline{{\theta }_n},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

To assess and compare the classification performance of each algorithm, we employed four widely used evaluation metrics: accuracy, precision, recall, and F1-score.

Accuracy measures the proportion of correctly classified samples, both positive and negative, relative to the total number of samples. It is computed as

$$\mathrm{Accuracy}(\mathrm{acc})={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}\times 100,$$

(62)

where $TP$ and $TN$ are the true positives and true negatives, respectively; $FP$ is the number of false positives (incorrectly predicting a patient as diseased); and $FN$ is the number of false negatives (failing to detect a diseased patient).

Precision reflects the proportion of true positives among all instances predicted as positive:

$$\mathrm{Precision}(\mathrm{pre})={\displaystyle \frac{TP}{TP+FP}}.$$

(63)

Recall, or sensitivity, represents the proportion of actual positive cases that are correctly identified:

$$\mathrm{Recall}(\mathrm{rec})={\displaystyle \frac{TP}{TP+FN}}.$$

(64)

F1-score is the harmonic mean of precision and recall, providing a balanced measure of model performance, particularly in imbalanced datasets:

$$\mathrm{F}1\mathrm{-score}(\mathrm{F}1)={\displaystyle \frac{2\times pre\times rec}{pre+rec}}.$$

(65)

The performance of each algorithm is analyzed at the 1000th iteration, as presented in Table 3. Four datasets, breast cancer, heart disease, diabetes, and hypertension, were utilized to evaluate and compare the effectiveness of Algorithms 6 and 7 using standard classification metrics including accuracy, precision, recall, and F1-score on both training and testing data.

Table 3. Performance comparison between Algorithms 6 and 7 on each dataset.

Datasets	Algorithm	Accuracy (%)		Precision		Recall		F1
		Train	Test	Train	Test	Train	Test	Train	Test
Breast Cancer	Algorithm 7	97.2995	96.7775	0.9576	0.9552	0.9656	0.9540	0.9616	0.9539
	Algorithm 6	96.8277	96.9246	0.9414	0.9464	0.9698	0.9705	0.9553	0.9570
Heart Disease	Algorithm 7	84.4153	81.4946	0.8363	0.8141	0.8875	0.8717	0.8611	0.8376
	Algorithm 6	77.4129	77.1828	0.7980	0.7941	0.7886	0.8007	0.7925	0.7931
Diabetes	Algorithm 7	92.2782	91.7495	0.9320	0.9279	0.9340	0.9280	0.9330	0.9266
	Algorithm 6	87.6405	87.6321	0.9534	0.9602	0.8256	0.8208	0.8848	0.8811
Hypertension	Algorithm 7	89.0435	88.9163	0.8509	0.8501	0.9241	0.9221	0.8860	0.8846
	Algorithm 6	87.8447	87.4593	0.8259	0.8225	0.9328	0.9282	0.8761	0.8721

The results indicate that Algorithm 7 consistently performs well across all datasets. In particular, in the hypertension dataset, which reflects real-world conditions, Algorithm 7 achieves high accuracy and balanced precision–recall performance. This demonstrates its strong generalization capability and suitability for real-world medical applications that require reliable predictions and low error sensitivity.

To evaluate model performance with respect to both goodness-of-fit and model complexity, we utilize the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC). These criteria are defined as follows:

• Akaike Information Criterion (AIC):

  $$\mathrm{AIC}=2k-2ln(\widehat{L})$$

  (66)

  where k is the number of estimated parameters in the model and $\widehat{L}$ is the maximum value of the likelihood function.
• Bayesian Information Criterion (BIC):

  $$\mathrm{BIC}=kln(n)-2ln(\widehat{L})$$

  (67)

  where n is the number of observations, k is the number of parameters, and $\widehat{L}$ is the maximum likelihood of the model.

Lower AIC and BIC values indicate better models in terms of balancing accuracy and simplicity.

To assess the consistency of the performance of model across multiple trials or datasets, we compute the mean and standard deviation (std) of AIC and BIC values.

• Mean AIC and BIC:

  $$\mathrm{Mean}\mathrm{AIC}={\displaystyle \frac{1}{m}}\sum _{i=1}^m{\mathrm{AIC}}_i,\mathrm{Mean}\mathrm{BIC}={\displaystyle \frac{1}{m}}\sum _{i=1}^m{\mathrm{BIC}}_i$$

  (68)
• Standard Deviation of AIC and BIC:

  $$\mathrm{Std}\mathrm{AIC}=\sqrt{{\displaystyle \frac{1}{m-1}}\sum _{i=1}^m{\left({\mathrm{AIC}}_i-\mathrm{Mean}\mathrm{AIC}\right)}^2}$$

  (69)

  $$\mathrm{Std}\mathrm{BIC}=\sqrt{{\displaystyle \frac{1}{m-1}}\sum _{i=1}^m{\left({\mathrm{BIC}}_i-\mathrm{Mean}\mathrm{BIC}\right)}^2}$$

  (70)

These statistics indicate the central tendency and dispersion of the AIC and BIC scores, where smaller standard deviations imply more stable model performance across different experiments.

To understand how well each algorithm fits the data without too much complexity, we compare their AIC and BIC values, as shown in Table 4. Both AIC and BIC are commonly used to measure how good a model is; lower values mean that the model is more efficient and avoids overfitting.

Table 4. Statistical comparison of Algorithms 6 and 7 based on AIC and BIC values.

Datasets	Algorithm	Train				Test
		AIC Mean	AIC Std	BIC Mean	BIC Std	AIC Mean	AIC Std	BIC Mean	BIC Std
Breast Cancer	Algorithm 7	545.90	57.4635	1872.24	57.4886	591.21	46.2082	1258.38	46.0285
	Algorithm 6	1848.79	31.1606	3175.13	31.2089	737.36	5.8845	1404.53	5.8996
Heart Disease	Algorithm 7	1306.05	13.1770	2821.57	12.8471	895.66	11.8465	1488.29	11.3720
	Algorithm 6	1579.11	5.8859	3094.63	6.1726	921.98	1.3402	1514.61	6.2564
Diabetes	Algorithm 7	930.65	25.1995	2003.17	25.2196	584.11	20.1605	1063.36	20.7454
	Algorithm 6	1600.26	35.6280	2672.77	35.7123	657.04	6.4499	1136.29	7.1045
Hypertension	Algorithm 7	8142.36	63.5788	10,126.00	63.5817	1438.92	74.3951	2763.35	74.4419
	Algorithm 6	14,998.70	334.1825	16,982.34	334.1837	2200.95	38.0503	3525.38	38.0505

The results show that Algorithm 7 gives lower AIC and BIC values than Algorithm 6 for all datasets. This means that Algorithm 7 is simpler and better at handling the data. The difference is most noticeable in the hypertension dataset, which comes from real-world health data. These results confirm that Algorithm 7 is a strong choice for real-world applications, where the model needs to be both accurate and not too complicated.

4.2.2. Application to Convex Bilevel Optimization Problems

The TELM model can also be formulated within the framework of convex bilevel optimization to better capture hierarchical learning structures. In this setting, we interpret the output weight learning (final step of TELM) as the solution to a lower-level convex problem, and the optimization of the hidden transformation weights (e.g., $W_{HE}$) as the upper-level objective.

In our TELM-based learning problem, this bilevel formulation arises naturally:

• The inner problem corresponds to learning the output weights u given the fixed transformation $W_{HE}$, and can be cast as a LASSO-type convex minimization:

  $$f(u) = \parallel {\mathbf{H}}_2{u-\mathbf{T}\parallel }_2^2,g(u)={\lambda }_1{\parallel u\parallel }_1,$$

  (71)

  where ${\mathbf{H}}_2$ is the second hidden layer output and $\mathbf{T}$ is the target.
• The outer problem focuses on optimizing the hidden transformation weights $W_{HE}$ based on the optimal solution $u^{*}(W_{HE})$ from the inner problem. The upper-level loss is given by

  $$\varphi (W_{HE})={\displaystyle \frac{1}{2}}{\parallel W_{HE}\parallel }_2^2.$$

  (72)

Solving this bilevel problem directly is challenging due to the implicit constraint $\Gamma$. However, by leveraging our proposed algorithm and proximal operator techniques, we can solve both levels efficiently and with guaranteed convergence under mild assumptions. This makes TELM highly suitable for structured learning tasks where the learning objectives are nested and interdependent.

To assess the performance of Algorithm 8 in solving convex bilevel optimization problems, we conducted experiments on the same datasets used in the convex optimization setting (see Section 4.2.1). These include the breast cancer, heart disease, diabetes, and hypertension datasets, with a 70%/30% split for training and testing, respectively.

We evaluated classification performance using the same metrics—accuracy, precision, recall, and F1-score—to ensure consistency across experiments.

In this bilevel setting, we compared our method against Algorithm 1 (BiGSAM), Algorithm 2 (iBiGSAM), Algorithm 3 (aiBiGSAM), Algorithm 4 (miBiGSAM), and Algorithm 5 (amiBiGSAM).

All algorithms were configured according to the parameter settings summarized in Table 5, ensuring fair and reproducible evaluation across all methods.

Table 5. The setting of parameters for each algorithms.

Parameters	Algorithm 8	Algorithm 1	Algorithm 2	Algorithm 3	Algorithm 4	Algorithm 5
${\mu }_n$	$1-{\displaystyle \frac{1}{n+1}}$	-	-	-	-	-
${\rho }_n$	$-{\displaystyle \frac{1}{n}}$	-	-	-	-	-
${\eta }_n$	${\displaystyle \frac{1}{n+2}}$	-	-	-	-	-
${\beta }_n$	${\displaystyle \frac{n}{2n+1}}$	-	-	-	-	-
${\tau }_n$	$0.5+{\displaystyle \frac{1}{2n}}$	-	-	-	-	-
${\xi }_n$	${\displaystyle \frac{33\times {10}^{20}}{n^2}}$	-	${\displaystyle \frac{1}{{(n+1)}^2}}$	${\displaystyle \frac{1}{{(n+1)}^2}}$	${\displaystyle \frac{1}{{(n+1)}^2}}$	${\displaystyle \frac{1}{{(n+1)}^2}}$
${\alpha }_n$	${\displaystyle \frac{n}{2n+1}}$	${\displaystyle \frac{1}{n+1}}$	${\displaystyle \frac{1}{n+1}}$	${\displaystyle \frac{1}{n+1}}$	${\displaystyle \frac{1}{n+1}}$	${\displaystyle \frac{1}{n+1}}$
${\gamma }_n$	${\displaystyle \frac{1}{L_f}}$	${\displaystyle \frac{n·{10}^{-5}}{(n+1)·L_F}}$	${\displaystyle \frac{n·{10}^{-5}}{(n+1)·L_F}}$	${\displaystyle \frac{1}{L_F}}$	${\displaystyle \frac{1}{L_F}}$	${\displaystyle \frac{1}{L_F}}$
s	0.001	0.001	0.001	0.001	0.001	0.001
$\alpha$	-	-	3	3	3	3
q	-	-	-	-	4	4

In addition, the following settings were consistently applied across all experimental setups:

• Regularization parameter: $\lambda ={10}^{-5}$.
• Activation function: Sigmoid, $g(x)={\displaystyle \frac{1}{1+e^{-x}}}$.
• Number of hidden nodes: $h=30$.

To evaluate the effectiveness of the proposed algorithm (Algorithm 8), we conducted experiments on four datasets. Each algorithm was trained for 1000 iterations, and the performance was measured in terms of accuracy, precision, recall, and F1-score for both training and testing phases. The comparative results of all algortithms are summarized in Table 6.

Table 6. Performance comparison of algorithms on each dataset.

Datasets	Algorithm	Accuracy (%)		Precision		Recall		F1
		Train	Test	Train	Test	Train	Test	Train	Test
Breast Cancer	Algorithm 8	97.4234	97.3636	0.9455	0.9454	0.9828	0.9833	0.9638	0.9633
	Algorithm 1	96.1933	95.4561	0.9449	0.9495	0.9470	0.9207	0.9453	0.9320
	Algorithm 2	96.8928	96.3427	0.9533	0.9548	0.9582	0.9415	0.9557	0.9473
	Algorithm 3	97.0555	96.7796	0.9490	0.9480	0.9679	0.9625	0.9583	0.9546
	Algorithm 4	97.3320	97.0716	0.9485	0.9447	0.9768	0.9750	0.9624	0.9591
	Algorithm 5	97.2508	96.9246	0.9488	0.9482	0.9740	0.9667	0.9612	0.9567
Heart Disease	Algorithm 8	87.3486	84.7957	0.8627	0.85	0.9131	0.8846	0.8871	0.8635
	Algorithm 1	83.2051	81.1613	0.8531	0.8411	0.8357	0.8114	0.8441	0.8213
	Algorithm 2	86.6150	83.4731	0.8629	0.8419	0.8970	0.8658	0.8794	0.8496
	Algorithm 3	86.6517	83.4731	0.8634	0.8419	0.8970	0.8658	0.8797	0.8496
	Algorithm 4	86.2853	83.7957	0.8532	0.8376	0.9037	0.8779	0.8776	0.8540
	Algorithm 5	86.4311	83.8065	0.8637	0.8428	0.8916	0.8721	0.8774	0.8532
Diabetes	Algorithm 8	94.9031	94.2812	0.9332	0.9254	0.9819	0.9800	0.9569	0.9517
	Algorithm 1	87.3346	86.9609	0.9594	0.9588	0.8145	0.8085	0.8810	0.8730
	Algorithm 2	94.0367	93.5888	0.9186	0.9177	0.9836	0.9760	0.9500	0.9458
	Algorithm 3	94.0367	93.5888	0.9186	0.9177	0.9836	0.9760	0.9500	0.9458
	Algorithm 4	93.5011	91.5275	0.9341	0.9147	0.9544	0.9402	0.9441	0.9264
	Algorithm 5	94.1641	93.8161	0.9208	0.9182	0.9832	0.9800	0.9510	0.9480
Hypertension	Algorithm 8	89.0544	88.9980	0.8714	0.8698	0.8945	0.8955	0.8827	0.8822
	Algorithm 1	65.1008	65.1416	0.6071	0.6075	0.5895	0.5953	0.5934	0.5964
	Algorithm 2	88.0175	88.0157	0.8313	0.8311	0.9282	0.9289	0.8771	0.8772
	Algorithm 3	88.0175	88.0157	0.8313	0.8311	0.9282	0.9289	0.8771	0.8772
	Algorithm 4	87.9284	87.9011	0.8279	0.8278	0.9315	0.9314	0.8767	0.8765
	Algorithm 5	87.9393	87.9666	0.8288	0.8290	0.9303	0.9310	0.8766	0.8770

As shown in Table 6, the proposed algorithm (Algorithm 8) consistently outperforms other methods across all datasets in both training and testing phases. In particular, for the breast cancer and diabetes datasets, Algorithm 8 achieves the highest test accuracy and F1-scores, demonstrating its strong generalization capability and classification performance.

Notably, in the hypertension dataset, which represents real-world medical data with high variability and complexity, the proposed method maintains superior accuracy and F1-score compared to baseline algorithms. This highlights the robustness and practical applicability of Algorithm 8 in real-world clinical settings.

Overall, the results support the effectiveness and stability of the proposed algorithm, making it a promising approach for medical classification tasks across diverse domains.

To statistically evaluate the performance of each algorithm, we computed the AIC and BIC values, including their mean and standard deviation, for both the training and testing phases. The experiments were conducted on four datasets: breast cancer, heart disease, diabetes, and hypertension. The summarized results presented in Table 7 serve to compare the statistical efficiency of each algorithm.

Table 7. Comparison of AIC and BIC scores (mean and standard deviation) for all algorithms.

Datasets	Algorithm	Train				Test
		AIC Mean	AIC Std	BIC Mean	BIC Std	AIC Mean	AIC Std	BIC Mean	BIC Std
Breast Cancer	Algorithm 8	519.3877	56.3868	1845.7279	56.3719	584.6227	33.4939	1251.7889	33.2445
	Algorithm 1	2238.5854	200.3185	3564.9257	200.2692	782.9862	19.9621	1450.1524	20.5936
	Algorithm 2	635.4704	58.1649	1961.8106	58.1312	605.9556	33.9859	1273.1218	34.3455
	Algorithm 3	635.6881	56.0138	1962.0283	55.9914	605.9556	33.9859	1273.1218	34.3455
	Algorithm 4	540.8845	50.3097	1867.2247	50.3208	589.6425	34.4505	1256.8087	34.2693
	Algorithm 5	558.5773	35.3092	1884.9175	35.3339	598.3528	34.6955	1265.5191	34.1824
Heart Disease	Algorithm 8	1238.6305	16.5334	2754.1462	16.5096	887.8190	10.7577	1480.4534	11.6136
	Algorithm 1	1609.6390	3.4581	3125.1548	3.8379	925.7843	1.2751	1518.4187	7.5333
	Algorithm 2	1253.4817	11.9895	2768.9974	11.8911	890.0700	10.7155	1482.7044	11.1995
	Algorithm 3	1253.4817	11.9895	2768.9974	11.8911	890.3966	10.7470	1483.0310	11.5066
	Algorithm 4	1261.5734	12.5241	2777.0892	12.2814	889.8314	11.5517	1482.4658	11.8088
	Algorithm 5	1258.9803	13.1210	2774.4961	12.7994	890.0805	10.7364	1482.7149	11.6097
Diabetes	Algorithm 8	756.0233	21.2332	1828.5392	21.1804	562.4426	25.2120	1041.6909	26.6428
	Algorithm 1	1479.6453	16.5049	2552.1612	16.7075	643.3454	6.2591	1122.5938	7.2984
	Algorithm 2	818.2987	26.2085	1890.8146	26.2338	567.9688	24.2671	1047.2172	25.1373
	Algorithm 3	818.2987	26.2085	1890.8146	26.2338	567.9688	24.2671	1047.2172	25.1373
	Algorithm 4	852.5711	29.2081	1925.0869	29.0782	579.1190	19.4671	1058.3673	20.9319
	Algorithm 5	818.4230	24.7334	1890.9388	24.8507	567.2419	22.9262	1046.4903	23.5958
Hypertension	Algorithm 8	7848.57	82.94	9832.17	82.94	1403.93	75.19	2728.36	75.19
	Algorithm 1	14,416.87	2736.23	16400.47	2736.24	2145.51	286.74	3469.94	286.68
	Algorithm 2	8350.16	63.68	10333.76	63.68	1457.44	71.74	2781.87	71.73
	Algorithm 3	8348.93	61.94	10332.53	61.94	1457.44	71.74	2781.87	71.73
	Algorithm 4	8414.86	61.84	10398.46	61.84	1464.74	69.33	2789.17	69.34
	Algorithm 5	8402.47	62.29	10386.07	62.29	1467.41	63.79	2791.84	63.80

According to the results in Table 7, the proposed algorithm (Algorithm 8) consistently shows lower AIC and BIC values across several datasets. This means that the model fits well and is less likely to overfit the data. In particular, in the hypertension dataset, which contains real and complex medical data, Algorithm 8 achieves the lowest and most consistent scores. This shows that the algorithm can handle real-world situations effectively and gives reliable results.

From Table 6 and Table 7, it is evident that Algorithm 8 consistently outperforms all variants of BiG-SAM, including the improved versions (Algorithms 2–5).

In all datasets considered in this work (see Table 6 and Table 7), Algorithm 8 achieves the highest classification performance and also yields the lowest AIC and BIC scores, suggesting a better model fit with lower complexity. Moreover, its standard deviations are relatively small, indicating robustness and stability across different runs. Therefore, Algorithm 8 can be considered the most effective and reliable algorithm among those evaluated.

5. Conclusions

We proposed the Double Inertial Viscosity Forward–Backward Algorithm (DIVFBA) and the Bilevel Double Inertial Forward–Backward Algorithm (BDIFBA), which is the modification of Algorithm 6, to solve variational inclusion and bilevel optimization problems, respectively. The proposed algorithms ensure strong convergence and achieve higher accuracy and stability compared to existing algorithms. We applied the proposed algorithms to train TELM models, where they consistently outperformed other existing algorithms in terms of evaluation metrics, and statistic values. These results confirm the effectiveness of the proposed algorithms in practical applications.