A New Accelerated Algorithm Based on Fixed Point Method for Convex Bilevel Optimization Problems with Applications

Abstract

A new accelerated common fixed point algorithm is introduced and analyzed for a countable family of nonexpansive mappings and then we apply it to solve some convex bilevel optimization problems. Then, under some suitable conditions, we prove a strong convergence result of the proposed algorithm. As an application, we employ the proposed algorithm for regression and classification problems. Moreover, we compare the performance of our algorithm with others. By numerical experiments, we found that our algorithm has a better performance than the others.

1. Introduction

Let H be a real Hilbert space and $f,g:H\to \mathbb{R}$ be real valued functions. The convex bilevel minimization problem is a special kind of optimization problem where one problem is embedded within another. The outer level is the following constraint minimization problem:

$$\underset{x\in X^{*}}{min}\omega \left(x\right),$$

(1)

where $\omega :{\mathbb{R}}^n\to \mathbb{R}$ is strongly convex with parameter $\sigma >0$ and a continuously differentiable function such that $\nabla \omega$ is Lipschitz continuous with constant $L_{\omega }$ while $X^{*}$ is the nonempty set of minimizers of the inner level problem, given by

$$\underset{x\in {\mathbb{R}}^n}{min}\{f\left(x\right)+g\left(x\right)\},$$

(2)

and sometimes we will use the notation $argmin(f+g)$ for $X^{*}$. The following assumptions are assumed for solving Problem (2).

(i)

$g:{\mathbb{R}}^n\to \mathbb{R}\cup \{\infty \}$ is a proper convex and lower semi-continuous function;

(ii)

$f:{\mathbb{R}}^n\to \mathbb{R}$ is a convex and differentiable function such that $\nabla f$ is a Lipschitz continuous with constant $L_f>0$, that is,

$$\parallel \nabla f\left(x\right)-\nabla f\left(y\right)\parallel \le L_f\parallel x-y\parallel \mathrm{for}\mathrm{all}x,y\in {\mathbb{R}}^n.$$

The solution of (2) can be characterized by Theorem 16.3 of Bauschke and Combettes [1] as follows:

$$\breve{q}\mathrm{is}\mathrm{a}\mathrm{minimizer}\mathrm{of}\mathrm{Problem}\left(2\right)\mathrm{if}\mathrm{and}\mathrm{only}\mathrm{if}0\in \partial g\left(\breve{q}\right)+\nabla f\left(\breve{q}\right),$$

where $\partial g$ is the subdifferential of g and $\nabla f$ is the gradient of f. Moreover, Problem (2) is also characterized by the following fixed point problem:

$$\begin{array}{cc}\hfill \breve{q}\mathrm{is}\mathrm{a}\mathrm{minimizer}\mathrm{of}(f+g)\mathrm{if}\mathrm{and}\mathrm{only}\mathrm{if}\breve{q}& =pro{x}_{cg}(I-c\nabla f)\left(\breve{q}\right),\hfill \end{array}$$

where $c>0$ and $pro{x}_{cg}\left(x\right)=argmi{n}_{y\in H}(g\left(y\right)+\frac{1}{2c}{\parallel x-y\parallel }^2)$. It is also known that $pro{x}_{cg}(I-c\nabla f)$ is a nonexpansive operator when $c\in (0,2/L)$ and $L>0$. The operator $pro{x}_{cg}(I-c\nabla f)$ is called the forward-backward operator of f and g with respect to c. Moreover, it is known that $\breve{q}\in X^{*}$ is a point of minimizer of problem (1) if and only if

$$\begin{array}{c}\hfill ⟨\nabla \omega \left(\breve{q}\right),x-\breve{q}⟩\ge 0\mathrm{for}\mathrm{all}x\in X^{*}.\end{array}$$

(3)

In the past decade, many researchers have proposed methods to find optimal solutions of Problem (2). Lions and Mercier [2] introduced a simple algorithm, called Forward-Backward Splitting (FBS), for solving Problem (2). Their algorithm was given by

$$\begin{array}{c}\hfill x_{n+1}=pro{x}_{c_{n}g}(I-c_n\nabla f)\left(x_n\right),\end{array}$$

(4)

where $c_n$ is the step-size and $c_n\in (0,2/L)$.

In 1964, Polyak [3] firstly introduced the inertial technique for accelerating the rate of convergence of the algorithm. Since then, this technique was widely used for this purpose.

In [4], Beck and Teboulle employed the inertial technique to introduce a fast iterative shrinkage-thresholding algorithm (FISTA) for solving Problem (2) as follows:

$$\begin{array}{cc}\hfill & x_1=y_0\in C,t_1=1,\hfill \\ \hfill & y_n=pro{x}_{\alpha g}(I-\alpha \nabla f)\left(x_n\right),\alpha >0,\hfill \\ \hfill & t_{n+1}=\frac{\sqrt{1+4t_n^2}+1}{2},\hfill \\ \hfill & {\theta }_n=\frac{t_n-1}{t_{n+1}},\hfill \\ \hfill & x_{n+1}=y_n+{\theta }_n(y_n-y_{n-1}).\hfill \end{array}$$

They showed that the convergence behavior of FISTA is better than the others.

Recently, some authors, for instance, Bussaban et al. [5], Puangpee and Suantai [6] and Jailoka et al. [7], employed the inertial technique to introduce common fixed point algorithms for a countable family of nonexpansive operators and established some convergence results under NST-condition (I), NST${}^{☆}$-condition, and the condition (Z). They also applied their algorithms to solving some convex minimization problems.

In 2017, Sabach and Shtern [8] introduced a new method, called Sequential Averaging Method (SAM), for solving a bilevel optimization problem. Such an algorithm was developed from [9] for solving a certain class of fixed point problems. To solve the bilevel optimization Problems (1) and (2), the Bilevel Gradient sequential Averaging Method (BiG-SAM) was proposed in [8]. Their algorithm was defined by Algorithm 1.

Algorithm 1 Bilevel Gradient sequential Averaging Method (BiG-SAM)

(1) Input: $s\in (0,2/(\sigma +L_{\omega }))$, $c\in (0,1/L_f)$, and ${\left\{{\alpha }_k\right\}}_{k\in \mathbb{N}}\subset (0,1].$ (2) Initialization: $x_1\in {\mathbb{R}}^n$. (3) General step: $(k=1,2,...)$: $$\begin{array}{cc}\hfill y_k& =pro{x}_{cg}(x_k-c\nabla f\left(x_k\right)),\hfill \\ \hfill z_k& =x_k-s\nabla \omega \left(x_k\right),\hfill \\ \hfill x_{k+1}& ={\alpha }_k{z}_k+(1-{\alpha }_k)y_k,\hfill \end{array}$$ where $\nabla \omega$ is the gradient of $\omega$.

They proved a strong convergence theorem of the sequence ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ generated by BiG-SAM under some control conditions.

After that, Shehu et al. [10] used the inertial technique for improving the convergence behavior of BiG-SAM. Their algorithm is known as the inertial Bilevel Gradient Sequential Averaging Method (iBiG-SAM). It was defined as follows (Algorithm 2):

Algorithm 2 Inertial Bilevel Gradient sequential Averaging Method (iBiG-SAM)

(1) Input: $\alpha \ge 3$, $s\in (0,2/(\sigma +L_{\omega }))$, and $c\in (0,1/L_f)$. (2) Initialization: $x_0,x_1\in {\mathbb{R}}^n$. (3) Step 1 For $(k=1,2,...)$: $$\begin{array}{c}\hfill {\theta }_n=\left\{\begin{array}{cc}min\{\frac{k}{k+\alpha -1},\frac{{\gamma }_n}{\parallel x_k-x_{k-1}\parallel }\},\hfill & \mathrm{if}{x}_k\ne x_{k-1},\hfill \\ \frac{k}{k+\alpha -1},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$ (4) Step 2 Compute: $$\begin{array}{cc}\hfill w_k& =x_k+{\mu }_k(x_k-x_{k-1}),\hfill \\ \hfill y_k& =pro{x}_{cg}(w_k-c\nabla f\left(w_k\right)),\hfill \\ \hfill z_k& =w_k-s\nabla \omega \left(w_k\right),\hfill \\ \hfill x_{k+1}& ={\alpha }_k{z}_k+(1-{\alpha }_k)y_k,\hfill \end{array}$$ where $\nabla \omega$ is the gradient of $\omega$.

In 2022, Duan and Zhang [11] introduced a new algorithm based on the proximal gradient algorithm for solving a bilevel optimization problem. This algorithm is known as the alternated inertial Bilevel Gradient Sequential Averaging Method (aiBiG-SAM). It was defined as follows (Algorithm 3):

Algorithm 3 The alternated inertial Bilevel Gradient Sequential Averaging Method (aiBiG-SAM)

(1) Input: $\alpha \ge 3$, $s\in (0,2/(\sigma +L_{\omega }))$ and $c\in (0,1/L_f)$. Set $\lambda >0$. (2) Initialization: $x_0,x_1\in {\mathbb{R}}^n$. (3) Step 1 For $(k=1,2,...)$: $$\begin{array}{c}\hfill w_k=\left\{\begin{array}{cc}{x}_k+{\mu }_k(x_k-x_{k-1})),\hfill & \mathrm{if}k\mathrm{is}\mathrm{odd},\hfill \\ x_k,\hfill & \mathrm{if}k\mathrm{is}\mathrm{even}.\hfill \end{array}\right.\end{array}$$ where k is odd, choose ${\mu }_k$ such that $0\le |{\mu }_k|\le {\theta }_n$ with ${\theta }_n$ defined by $$\begin{array}{c}\hfill {\theta }_n=\left\{\begin{array}{cc}min\{\frac{k}{k+\alpha -1},\frac{{\gamma }_k}{\parallel x_k-x_{k-1}\parallel }\},\hfill & \mathrm{if}{x}_k\ne x_{k-1},\hfill \\ \frac{k}{k+\alpha -1},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$ (4) Step 2 Compute: $$\begin{array}{cc}\hfill y_k& =pro{x}_{cg}(w_k-c\nabla f\left(w_k\right)),\hfill \\ \hfill z_k& =w_k-s\nabla \omega \left(w_k\right),\hfill \\ \hfill x_{k+1}& ={\alpha }_k{z}_k+(1-{\alpha }_k)y_k.\hfill \end{array}$$ (5) Step 3 If $\parallel x_k-x_{k-1}\parallel <\lambda$, then stop.

They also proved a strong convergence result of the proposed method.

Motivated by these works, we are interested in proposing a new efficient algorithm for convex bilevel Problems (1) and (2). We establish and prove a convergence theorem of the proposed algorithm under some suitable conditions. We employ it for data prediction and classification. The paper is organized as follows. In Section 2, we describe some notations and useful lemmas for the later sections. In Section 3, we discuss and analyze the convergence of our proposed algorithm. In Section 4, we present applications of the obtained fixed-point results in Section 3 for solving regression and classification problems. Moreover, some numerical experiments on regression and classification problems are also given in Section 4. Finally, we also give conclusions of the paper in Section 5.

2. Preliminaries

Let H be a real Hilbert space with norm $\parallel ·\parallel$ and inner product $⟨·,·⟩$, and C be a nonempty closed convex subset of H. Let $L\in (0,\infty )$. An operator $T:C\to C$ is said to be L-Lipschitz if $∥Tx-Ty∥\le L∥x-y∥\mathrm{for}\mathrm{all}x,y\in C$. If T is Lipschitz continuous with a coefficient $L\in (0,1)$, then T is called a contraction. The operator T is said to be nonexpansive if $L=1$. We use $F\left(T\right)$ to denote the set of fixed points of T, that is, $F\left(T\right):=\{x\in C:x=Tx\}$. The set of all common fixed points of a sequence of nonexpansive operators $\left\{T_n\right\}$ of C into itself is ${\bigcap }_{n=1}^{\infty }F\left(T_n\right)$. For finding a common fixed point of $\left\{T_n\right\}$, Takahashi [12] introduced the NST-condition as the following. Let $\left\{T_n\right\}$ and $\mathsf{\Psi }$ be families of nonexpansive operators of C into itself with $\varnothing \ne F\left(\mathsf{\Psi }\right)\subset {\bigcap }_{n=1}^{\infty }F\left(T_n\right)$, where $F\left(\mathsf{\Psi }\right)={\bigcap }_{T\in \mathsf{\Psi }}F\left(T\right)$. A sequence $\left\{T_n\right\}$ satisfies the NST-condition (I) with $\mathsf{\Psi }$, if for any bounded sequence $\left\{v_n\right\}$ in C,

$$\underset{n\to \infty }{lim}\parallel v_n-T_n{v}_n\parallel =0\mathrm{implies}\underset{n\to \infty }{lim}\parallel v_n-T{v}_n\parallel =0$$

for all $T\in \mathsf{\Psi }$. The sequence $\left\{T_n\right\}$ satisfies the NST-condition (I) with T if $\mathsf{\Psi }=\left\{T\right\}$.

The following Lemma is useful for proving our main result.

Lemma 1 

([5]). Let f be a convex and differentiable function from H into $\mathbb{R}$ with $\nabla f$ as a Lipschitz continuous with constant $L>0$, and g is a proper convex and lower semi-continuous function from H into $\mathbb{R}\cup \{\infty \}$. Let $T_n:=pro{x}_{c_{n}g}(I-c_n\nabla f)$ and $T:=pro{x}_{cg}(I-c\nabla f)$, where $c_n,c\in (0,2/L)$ with $c_n\to c$ as $n\to \infty$. Then $\left\{T_n\right\}$ satisfies the NST-condition (I) with T.

Definition 1 

([13,14]). A sequence $\{T_n:H\to H\}$ with a nonempty common fixed point set is said to satisfy the condition (Z) if $\left\{x_n\right\}$ is a bounded sequence in H such that

$$\underset{n\to \infty }{lim}\parallel x_n-T_n{x}_n\parallel =0,$$

it follows that every weak cluster point of $\left\{x_n\right\}$ belongs to ${\bigcap }_{n=1}^{\infty }F\left(T_n\right)$.

The following remark is obtained by demicloseness of $I-T$ where $T:H\to H$ is the nonexpansive operator.

Remark 1. 

If $\left\{T_n\right\}$ is nonexpansive, the operator satisfies the NST-condition (I) with respect to T where T is the nonexpansive operator. Then $\left\{T_n\right\}$ satisfies the condition (Z).

Note that if $g:{\mathbb{R}}^n\to \mathbb{R}\cup \{\infty \}$ is a proper, lower semi-continuous and convex function, then the $pro{x}_g\left(x\right)$ exists and is unique for all $x\in {\mathbb{R}}^n$; see [15]. We end this part with the following useful lemmas, which will be used in the later section.

Lemma 2 

([16,17]). For any $m,n\in H$ and $\mu \in [0,1]$, the following statements hold:

(1) 

${\parallel m\pm n\parallel }^2={\parallel m\parallel }^2\pm 2⟨m,n⟩+{\parallel n\parallel }^2\mathrm{for}\mathrm{all}m,n\in H$;

(2) 

${\parallel m+n\parallel }^2\le {\parallel m\parallel }^2+2⟨n,m+n⟩\mathrm{for}\mathrm{all}m,n\in H$;

(3) 

${\parallel \mu m+(1-\mu )n\parallel }^2={\mu \parallel m\parallel }^2+{(1-\mu )\parallel n\parallel }^2-\mu (1-\mu ){\parallel m-n\parallel }^2.$

The identity in Lemma 2(3) implies that the following equality holds:

$${\parallel \kappa m+\alpha n+\eta z\parallel }^2={\kappa \parallel m\parallel }^2+{\alpha \parallel n\parallel }^2+{\eta \parallel z\parallel }^2{-\kappa \alpha \parallel m-n\parallel }^2{-\alpha \eta \parallel n-z\parallel }^2-\kappa \eta {\parallel m-z\parallel }^2,$$

for all $m,n,z\in H$ and $\kappa ,\alpha ,\eta \in [0,1]$ with $\kappa +\alpha +\eta =1$.

Lemma 3 

([18]). Let $\left\{a_n\right\}$, $\left\{{\rho }_n\right\}\subset {\mathbb{R}}_{+}$, $\left\{s_n\right\}\subset \mathbb{R}$ and $\left\{{\zeta }_n\right\}\subset [0,1]$ such that

$$a_{n+1}\le (1-{\zeta }_n)a_n+{\zeta }_n{s}_n+{\rho }_n$$

for all $n\in \mathbb{N}$. If the following conditions hold:

(i) 

${\sum }_{n=1}^{\infty }{\zeta }_n=\infty$;

(ii) 

${\sum }_{n=1}^{\infty }{\rho }_n<\infty$;

(iii) 

${lim\; sup}_{n\to \infty }{s}_n\le 0$,

then ${lim}_{n\to \infty }{a}_n=0$.

Lemma 4 

([19]). Let $\left\{{\mathsf{\Phi }}_n\right\}$ be a sequence of real numbers which does not decrease at infinity in the sense that there exists a subsequence $\left\{{\mathsf{\Phi }}_{n_i}\right\}$ of $\left\{{\mathsf{\Phi }}_n\right\}$, satisfying ${\mathsf{\Phi }}_{n_i}<{\mathsf{\Phi }}_{n_i+1}$ for all $i\in \mathbb{N}$. Define the sequence ${\left\{\lambda \left(n\right)\right\}}_{n\ge n_0}$ of integers as follows:

$$\lambda \left(n\right):=max\{k\le n:{\mathsf{\Phi }}_k<{\mathsf{\Phi }}_{k+1}\},$$

where $n_0\in \mathbb{N}$ such that $\{k\le n_0:{\mathsf{\Phi }}_k<{\mathsf{\Phi }}_{k+1}\}\ne \varnothing$. Then the following statements hold:

(i) 

$\lambda \left(n_0\right)\le \lambda (n_0+1)\le \cdots$ and $\lambda \left(n\right)\to \infty$;

(ii) 

${\mathsf{\Phi }}_{\lambda \left(n\right)}\le {\mathsf{\Phi }}_{\lambda \left(n\right)+1}$ and ${\mathsf{\Phi }}_n\le {\mathsf{\Phi }}_{\lambda \left(n\right)+1}$ for all $n\ge n_0$.

Let C be a nonempty closed convex subset of a Hilbert space H and let $P:H\to C$ be a mapping. The metric projection onto C, denoted by $P_C$, is defined for each $x\in H$, $P_{C}x$ and is the unique element in C such that

$$\parallel x-P_{C}x\parallel =\underset{y\in C}{inf}\parallel x-y\parallel .$$

It is known that

$$\breve{q}=P_{C}x\iff ⟨x-\breve{q},y-\breve{q}⟩\le 0,$$

(5)

for all $y\in C$; see [16].

3. Results

Throughout this section, we let $\left\{T_n\right\}$ and $\mathsf{\Psi }$ be families of nonexpansive operators on a real Hilbert space H such that $F\left(\mathsf{\Psi }\right)\subset \mathsf{\Gamma }:={\bigcap }_{n=1}^{\infty }F\left(T_n\right)$ and $S:H\to H$ are a contraction mapping with a constant $k\in (0,1)$.

To find a common fixed point of a countable family of nonexpansive operators in a real Hilbert space, we first propose a new accelerated algorithm. Then, under certain conditions, we show a strong convergence theorem. Now, we are ready to introduce our accelerated algorithm as follows:

Theorem 1. 

Suppose that $\left\{T_n\right\}$ satisfies the condition (Z). Let $\left\{x_n\right\}$ be a sequence generated by Algorithm 4 which satisfies the following conditions:

(i) 

$0<a\le {\kappa }_n\le b<1$ for some $a,b\in \mathbb{R}$;

(ii) 

${lim}_{n\to \infty }{\alpha }_n=0$ and ${\sum }_{n=1}^{\infty }{\alpha }_n=+\infty$;

(iii) 

${lim}_{n\to \infty }{\gamma }_n=0$.

Then $\left\{x_n\right\}$ converges strongly to an element $\breve{q}\in \mathsf{\Gamma }$, where $\breve{q}=P_{\mathsf{\Gamma }}S\left(\breve{q}\right)$.

Algorithm 4 An Inertial Viscosity Modified Picard (IVMP)

Initial. Take $x_0,x_1\in H$ arbitrarily and $t_1=0$. For $n\ge 1$, set $$\begin{array}{c}\hfill {\theta }_n=\left\{\begin{array}{cc}min\{\frac{t_n-1}{t_{n+1}},\frac{{\gamma }_n{\alpha }_n}{\parallel x_n-x_{n-1}\parallel }\},\hfill & \mathrm{if}{x}_n\ne x_{n-1},\hfill \\ \frac{t_n-1}{t_{n+1}},\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$ (6)  where $t_{n+1}=\frac{1+\sqrt{1+4t_n^2}}{2}$. Step 1. Calculate $y_n$, $z_n$ and $x_{n+1}$ using: $$\begin{array}{cc}\hfill & y_n=x_n+{\theta }_n(x_n-x_{n-1}),\hfill \\ \hfill & z_n=(1-{\alpha }_n-{\kappa }_n)y_n+{\alpha }_{n}S\left(y_n\right)+{\kappa }_n{T}_n{y}_n,\hfill \\ \hfill & x_{n+1}=T_n{z}_n,\hfill \end{array}$$  where $\left\{{\alpha }_n\right\}$, $\left\{{\kappa }_n\right\}\subset [0,1]$ with ${\alpha }_n+{\kappa }_n\le 1$. Then, update $n:=n+1$ and return to Step 1.

Proof. 

Let $\breve{q}\in \mathsf{\Gamma }$ be such that $\breve{q}=P_{\mathsf{\Gamma }}S\left(\breve{q}\right)$. By the definition of $y_n$ and $z_n$ in Algorithm 4, for each $n\in \mathbb{N}$, we have

$$\begin{array}{cc}\hfill \parallel y_n-\breve{q}\parallel & =\parallel x_n+{\theta }_n(x_n-x_{n-1})-\breve{q}\parallel \hfill \\ \hfill & \le \parallel x_n-\breve{q}\parallel +{\theta }_n\parallel x_n-x_{n-1}\parallel \hfill \end{array}$$

(7)

and

$$\begin{array}{cc}\hfill \parallel z_n-\breve{q}\parallel & =\parallel (1-{\alpha }_n-{\kappa }_n)y_n+{\alpha }_{n}S\left(y_n\right)+{\kappa }_n{T}_n{y}_n-\breve{q}\parallel \hfill \\ \hfill & =\parallel (1-{\alpha }_n-{\kappa }_n)(y_n-\breve{q})+{\alpha }_n(S\left(y_n\right)-\breve{q})+{\kappa }_n(T_n{y}_n-\breve{q})\parallel \hfill \\ \hfill & \le (1-{\alpha }_n-{\kappa }_n)\parallel (y_n-\breve{q})\parallel +{\alpha }_n\parallel (S\left(y_n\right)-\breve{q})\parallel \hfill \\ \hfill & +{\kappa }_n\parallel (T_n{y}_n-\breve{q})\parallel \hfill \\ \hfill & \le (1-{\alpha }_n-{\kappa }_n)\parallel (y_n-\breve{q})\parallel +{\alpha }_{n}k\parallel y_n-\breve{q}\parallel +{\alpha }_n\parallel S\left(\breve{q}\right)-\breve{q}\parallel \hfill \\ \hfill & +{\kappa }_n\parallel y_n-\breve{q}\parallel \hfill \\ \hfill & =(1-(1-k){\alpha }_n)\parallel y_n-\breve{q}\parallel +{\alpha }_n\parallel S\left(\breve{q}\right)-\breve{q}\parallel \hfill \\ \hfill & \le (1-(1-k){\alpha }_n)(\parallel x_n-\breve{q}\parallel +{\theta }_n\parallel x_n-x_{n-1}\parallel )+{\alpha }_n\parallel S\left(\breve{q}\right)-\breve{q}\parallel \hfill \\ \hfill & \le (1-(1-k){\alpha }_n)\parallel x_n-\breve{q}\parallel +{\alpha }_n\left(\frac{{\theta }_n}{{\alpha }_n}\parallel x_n-x_{n-1}\parallel +\parallel S\left(\breve{q}\right)-\breve{q}\parallel \right).\hfill \end{array}$$

(8)

From (7) and (8), we obtain

$$\begin{array}{cc}\hfill \parallel x_{n+1}-\breve{q}\parallel & =\parallel T_n{z}_n-\breve{q}\parallel \hfill \\ \hfill & \le \parallel z_n-\breve{q}\parallel \hfill \\ \hfill & \le (1-(1-k){\alpha }_n)\parallel x_n-\breve{q}\parallel \hfill \\ \hfill & +{\alpha }_n\left(\frac{{\theta }_n}{{\alpha }_n}\parallel x_n-x_{n-1}\parallel +\parallel S\left(\breve{q}\right)-\breve{q}\parallel \right)\hfill \\ \hfill & =(1-(1-k){\alpha }_n)\parallel x_n-\breve{q}\parallel \hfill \\ \hfill & +(1-k){\alpha }_n\left(\frac{\frac{{\theta }_n}{{\alpha }_n}\parallel x_n-x_{n-1}\parallel +\parallel S\left(\breve{q}\right)-\breve{q}\parallel }{1-k}\right)\hfill \\ \hfill & \le max\left\{\parallel x_n-\breve{q}\parallel ,{\alpha }_n\left(\frac{\frac{{\theta }_n}{{\alpha }_n}\parallel x_n-x_{n-1}\parallel +\parallel S\left(\breve{q}\right)-\breve{q}\parallel }{1-k}\right)\right\}.\hfill \end{array}$$

(9)

Since ${lim}_{n\to \infty }{\gamma }_n=0$, by (6), we get that $\frac{{\theta }_n}{{\alpha }_n}\parallel x_n-x_{n-1}\parallel \to 0\mathrm{as}n\to \infty$. Thus, there is a constant $M_1>0$ such that $\frac{{\theta }_n}{{\alpha }_n}\parallel x_n-x_{n-1}\parallel \le M_1\mathrm{for}\mathrm{all}n\ge 1$. This implies

$$\begin{array}{cc}\hfill \parallel x_{n+1}-\breve{q}\parallel & \le max\left\{\parallel x_n-\breve{q}\parallel ,\frac{M_1+\parallel S\left(\breve{q}\right)-\breve{q}\parallel }{1-k}\right\}\mathrm{for}\mathrm{all}n\ge 1.\hfill \end{array}$$

(10)

Let $M=max\left\{\parallel x_1-\breve{q}\parallel ,\frac{M_1+\parallel S\left(\breve{q}\right)-\breve{q}\parallel }{1-k}\right\}$. We show $\parallel x_n-\breve{q}\parallel \le M$. For $n=1$, we get

$$\begin{array}{cc}\hfill \parallel x_{n+1}-\breve{q}\parallel & \le max\left\{\parallel x_1-\breve{q}\parallel ,\frac{M_1+\parallel S\left(\breve{q}\right)-\breve{q}\parallel }{1-k}\right\}\hfill \\ \hfill & =M.\hfill \end{array}$$

Suppose $\parallel x_n-\breve{q}\parallel \le M$ for some $n\in \mathbb{N}$. It follows from (10) that

$$\begin{array}{cc}\hfill \parallel x_{n+1}-\breve{q}\parallel & \le max\left\{\parallel x_1-\breve{q}\parallel ,\frac{M_1+\parallel S\left(\breve{q}\right)-\breve{q}\parallel }{1-k}\right\},\hfill \\ \hfill & \le max\left\{M,\frac{M_1+\parallel S\left(\breve{q}\right)-\breve{q}\parallel }{1-k}\right\}.\hfill \end{array}$$

Since $M=max\left\{\parallel x_1-\breve{q}\parallel ,\frac{M_1+\parallel S\left(\breve{q}\right)-\breve{q}\parallel }{1-k}\right\}$, we obtain $\frac{M_1+\parallel S\left(\breve{q}\right)-\breve{q}\parallel }{1-k}\le M$ which implies

$$\begin{array}{cc}\hfill \parallel x_{n+1}-\breve{q}\parallel & \le max\left\{M,\frac{M_1+\parallel S\left(\breve{q}\right)-\breve{q}\parallel }{1-k}\right\}\hfill \\ \hfill & =M.\hfill \end{array}$$

By mathematical induction, we conclude that $\parallel x_n-\breve{q}\parallel \le M$ for all $n\in \mathbb{N}$. Thus, $\parallel x_n\parallel \le \parallel x_n-\breve{q}\parallel +\parallel \breve{q}\parallel \le M+\parallel \breve{q}\parallel$ for all $n\in \mathbb{N}$. It follows that $\left\{x_n\right\}$ is bounded, and so are $\left\{y_n\right\},\left\{z_n\right\},\left\{T_n{y}_n\right\},\left\{T_n{z}_n\right\}$ and $\left\{S\left(y_n\right)\right\}$.

For each $n\ge 1$, we have

$$\begin{array}{cc}\hfill \parallel y_n-\breve{q}{\parallel }^2& =\parallel x_n+{\theta }_n(x_n-x_{n-1})-\breve{q}{\parallel }^2\hfill \\ \hfill & \le \parallel x_n-\breve{q}{\parallel }^2+2⟨x_n-\breve{q},{\theta }_n(x_n-x_{n-1})⟩+{\theta }_n^2{\parallel x_n-x_{n-1}\parallel }^2.\hfill \end{array}$$

(11)

By Lemma 2, we get

$$\begin{array}{cc}\hfill \parallel z_n-\breve{q}{\parallel }^2& =\parallel (1-{\alpha }_n-{\kappa }_n)y_n+{\alpha }_{n}S\left(y_n\right)+{\kappa }_n{T}_n{y}_n-\breve{q}{\parallel }^2\hfill \\ \hfill & =\parallel (1-{\alpha }_n-{\kappa }_n)(y_n-\breve{q})+{\alpha }_n(S\left(y_n\right)-\breve{q})+{\kappa }_n(T_n{y}_n-\breve{q}){\parallel }^2\hfill \\ \hfill & \le \parallel (1-{\alpha }_n-{\kappa }_n)(y_n-\breve{q})+{\alpha }_n(S\left(y_n\right)-S\left(\breve{q}\right))+{\kappa }_n(T_n{y}_n-\breve{q}){\parallel }^2\hfill \\ \hfill & +2⟨{\alpha }_n(S\left(\breve{q}\right)-\breve{q}),z_n-\breve{q}⟩\hfill \\ \hfill & \le (1-{\alpha }_n-{\kappa }_n)\parallel y_n-\breve{q}{\parallel }^2+{\alpha }_n\parallel S\left(y_n\right)-S\left(\breve{q}\right){\parallel }^2+{\kappa }_n{\parallel T_n{y}_n-\breve{q}\parallel }^2\hfill \\ \hfill & +2{\alpha }_n⟨S\left(\breve{q}\right)-\breve{q},z_n-\breve{q}⟩\hfill \\ \hfill & \le (1-{\alpha }_n)\parallel y_n-\breve{q}{\parallel }^2+{\alpha }_n{k}^2{\parallel y_n-\breve{q}\parallel }^2\hfill \\ \hfill & +2{\alpha }_n⟨S\left(\breve{q}\right)-\breve{q},z_n-\breve{q}⟩\mathrm{for}\mathrm{all}n\ge 1.\hfill \end{array}$$

It follows from (7) with $0<k<1$ that

$$\begin{array}{cc}\hfill \parallel z_n-\breve{q}{\parallel }^2& \le (1-{\alpha }_n){\parallel y_n-\breve{q}\parallel }^2\hfill \\ \hfill & +{\alpha }_{n}k{\parallel y_n-\breve{q}\parallel }^2+2{\alpha }_n⟨S\left(\breve{q}\right)-\breve{q},z_n-\breve{q}⟩\mathrm{for}\mathrm{all}n\ge 1.\hfill \\ \hfill & \le (1-{\alpha }_n+{\alpha }_{n}k)(\parallel x_n-\breve{q}{\parallel }^2+2⟨x_n-\breve{q},{\theta }_n(x_n-x_{n-1})⟩)\hfill \\ \hfill & +(1-{\alpha }_n+{\alpha }_{n}k)({\theta }_n^2\parallel x_n-x_{n-1}{\parallel }^2)+2{\alpha }_n⟨S\left(\breve{q}\right)-\breve{q},z_n-\breve{q}⟩.\hfill \end{array}$$

(12)

Using (12),

$$\begin{array}{cc}\hfill \parallel x_{n+1}-\breve{q}{\parallel }^2& =\parallel T_n{z}_n-\breve{q}{\parallel }^2\hfill \\ \hfill & \le \parallel z_n-\breve{q}{\parallel }^2\hfill \\ \hfill & \le (1-(1-k){\alpha }_n){\parallel x_n-\breve{q}\parallel }^2\hfill \\ \hfill & +(1-k){\alpha }_n\left[\frac{2{\theta }_n}{(1-k){\alpha }_n}\parallel x_n-\breve{q}\parallel \parallel x_n-x_{n-1}\parallel \right]\hfill \\ \hfill & +(1-k){\alpha }_n\left[\frac{{\theta }_n^2}{(1-k){\alpha }_n}\parallel x_n-x_{n-1}\parallel +\frac{2}{1-k}⟨S\left(\breve{q}\right)-\breve{q},z_n-\breve{q}⟩\right].\hfill \end{array}$$

(13)

From the above inequality, we get

$$a_n:={\parallel x_n-\breve{q}\parallel }^2,{\zeta }_n:=(1-k){\alpha }_n,$$

and

$$s_n:=(1-k){\alpha }_n\left[\frac{2{\theta }_n}{{\alpha }_n(1-k)}\parallel x_n-\breve{q}\parallel \parallel x_n-x_{n-1}\parallel +\frac{{\theta }_n^2}{(1-k){\alpha }_n}\parallel x_n-x_{n-1}\parallel +\frac{2}{1-k}⟨S\left(\breve{q}\right)-\breve{q},z_n-\breve{q}⟩\right].$$

So, we obtain

$$\begin{array}{c}\hfill a_{n+1}\le (1-{\zeta }_n)a_n+{\zeta }_n{s}_n.\end{array}$$

(14)

Now, we consider two cases for the covergence of the sequence $\left\{x_n\right\}$ generated by Algorithm 4.

Case 1. 

There exists a $n_0\in \mathbb{N}$ such that the sequence $\{\parallel x_n-\breve{q}{\parallel \}}_{n\ge n_0}$ is nonincreasing. Since $\{\parallel x_n-\breve{q}\parallel \}$ is bounded from below by zero, ${lim}_{n\to \infty }\parallel x_n-\breve{q}\parallel$ exists. Using assumption ${lim}_{n\to \infty }{\alpha }_n=0$ and ${\sum }_{n=1}^{\infty }{\alpha }_n=+\infty$, we get that ${\sum }_{n=1}^{\infty }(1-k){\alpha }_n=(1-k){\sum }_{n=1}^{\infty }{\alpha }_n=+\infty$. For applying this in Lemma 4, we need to show that

$$\underset{n\to \infty }{lim\; sup}⟨S\left(\breve{q}\right)-\breve{q},z_n-\breve{q}⟩\le 0.$$

Using the fact of Lemma 2(3), we get

$$\begin{array}{cc}\hfill \parallel x_{n+1}-\breve{q}{\parallel }^2& =\parallel T_n{z}_n-\breve{q}{\parallel }^2\hfill \\ \hfill & \le \parallel z_n-\breve{q}{\parallel }^2\hfill \\ \hfill & =\parallel (1-{\alpha }_n-{\kappa }_n)y_n+{\alpha }_{n}S\left(y_n\right)+{\kappa }_n{T}_n{y}_n-\breve{q}{\parallel }^2\hfill \\ \hfill & =\parallel (1-{\alpha }_n-{\kappa }_n)(y_n-\breve{q})+{\alpha }_n(S\left(y_n\right)-\breve{q})+{\kappa }_n(T_n{y}_n-\breve{q}){\parallel }^2\hfill \\ \hfill & =(1-{\alpha }_n-{\kappa }_n)\parallel y_n-\breve{q}{\parallel }^2+{\alpha }_n\parallel S\left(y_n\right)-\breve{q}{\parallel }^2+{\kappa }_n{\parallel T_n{y}_n-\breve{q}\parallel }^2\hfill \\ \hfill & -(1-{\alpha }_n-{\kappa }_n){\alpha }_n\parallel y_n-S\left(y_n\right){\parallel }^2-{\alpha }_n{\kappa }_n{\parallel S\left(y_n\right)-T_n{y}_n\parallel }^2\hfill \\ \hfill & -(1-{\alpha }_n-{\kappa }_n){\kappa }_n{\parallel y_n-T_n{y}_n\parallel }^2\hfill \\ \hfill & \le (1-{\alpha }_n-{\kappa }_n)\parallel y_n-\breve{q}{\parallel }^2+{\alpha }_n\parallel S\left(y_n\right)-\breve{q}{\parallel }^2+{\kappa }_n{\parallel y_n-\breve{q}\parallel }^2\hfill \\ \hfill & -(1-{\alpha }_n-{\kappa }_n){\kappa }_n{\parallel y_n-T_n{y}_n\parallel }^2.\hfill \end{array}$$

This implies that

$$\begin{array}{cc}\hfill (1-{\alpha }_n-{\kappa }_n){\kappa }_n{\parallel y_n-T_n{y}_n\parallel }^2& \le (1-{\alpha }_n)\parallel y_n-\breve{q}{\parallel }^2+{\alpha }_n\parallel S\left(y_n\right)-\breve{q}{\parallel }^2-{\parallel x_{n+1}-\breve{q}\parallel }^2\hfill \\ \hfill & \le (1-{\alpha }_n)(\parallel x_n-\breve{q}{\parallel }^2+2⟨x_n-\breve{q},{\theta }_n(x_n-x_{n-1})⟩)\hfill \\ \hfill & +(1-{\alpha }_n)({\theta }_n^2\parallel x_n-x_{n-1}{\parallel }^2)+{\alpha }_n{\parallel S\left(y_n\right)-\breve{q}\parallel }^2\hfill \\ \hfill & -\parallel x_{n+1}-\breve{q}{\parallel }^2\hfill \\ \hfill & \le (1-{\alpha }_n)(\parallel x_n-\breve{q}{\parallel }^2+2\parallel x_n-\breve{q}\parallel \parallel {\theta }_n(x_n-x_{n-1})\parallel \hfill \\ \hfill & +(1-{\alpha }_n)({\theta }_n^2\parallel x_n-x_{n-1}{\parallel }^2)+{\alpha }_n{\parallel S\left(y_n\right)-\breve{q}\parallel }^2\hfill \\ \hfill & -\parallel x_{n+1}-\breve{q}{\parallel }^2.\hfill \end{array}$$

It follows from the assumption and the convergence of the sequence $\{\parallel x_n-\breve{q}\parallel \}$ and $\{{\theta }_n\parallel x_n-x_{n-1}\parallel \}$ that ${lim}_{n\to \infty }\parallel y_n-T_n{y}_n\parallel =0$. For each $n\ge 1$, we have

$$\begin{array}{cc}\hfill \parallel z_n-T_n{z}_n\parallel & =\parallel z_n-y_n+y_n-T_n{y}_n+T_n{y}_n-T_n{z}_n\parallel \hfill \\ \hfill & \le \parallel z_n-y_n\parallel +\parallel y_n-T_n{y}_n\parallel +\parallel T_n{y}_n-T_n{z}_n\parallel \hfill \\ \hfill & \le \parallel z_n-y_n\parallel +\parallel y_n-T_n{y}_n\parallel +\parallel y_n-z_n\parallel \hfill \\ \hfill & \le 2\parallel z_n-y_n\parallel +\parallel y_n-T_n{y}_n\parallel \hfill \\ \hfill & =2\parallel (1-{\alpha }_n-{\kappa }_n)y_n+{\alpha }_{n}S\left(y_n\right)+{\kappa }_n{T}_n{y}_n-y_n\parallel +\parallel y_n-T_n{y}_n\parallel \hfill \\ \hfill & =2\parallel {\alpha }_n(S\left(y_n\right)-y_n)+{\kappa }_n(T_n{y}_n-y_n)\parallel +\parallel y_n-T_n{y}_n\parallel \hfill \\ \hfill & \le 2{\alpha }_n\parallel S\left(y_n\right)-y_n\parallel +(1+2{\kappa }_n)\parallel y_n-T_n{y}_n\parallel .\hfill \end{array}$$

This implies that ${lim}_{n\to \infty }\parallel z_n-T_n{z}_n\parallel =0$. Let

$$v=\underset{n\to \infty }{lim\; sup}⟨S\left(\breve{q}\right)-\breve{q},z_n-\breve{q}⟩.$$

Since $\left\{z_n\right\}$ is bounded, we can choose a subsequence $\left\{z_{n_k}\right\}$ of $\left\{z_n\right\}$ such that

$$v=\underset{k\to \infty }{lim}⟨S\left(\breve{q}\right)-\breve{q},z_{n_k}-\breve{q}⟩$$

and $z_{m_k}\rightharpoonup w$ for some $w\in H$. It follows from the condition (Z) of $\left\{T_n\right\}$ that $w\in \mathsf{\Gamma }$.

Moreover, using $\breve{q}=P_{\mathsf{\Gamma }}S\left(\breve{q}\right)$, we obtain

$$\begin{array}{cc}\hfill v& =\underset{k\to \infty }{lim}⟨S\left(\breve{q}\right)-\breve{q},z_{n_k}-\breve{q}⟩\hfill \\ \hfill & =⟨S\left(\breve{q}\right)-\breve{q},w-\breve{q}⟩\hfill \\ \hfill & =⟨S\left(\breve{q}\right)-P_{\mathsf{\Gamma }}S\left(\breve{q}\right),w-\breve{q}⟩\hfill \\ \hfill & \le 0.\hfill \end{array}$$

Thus,

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim\; sup}⟨S\left(\breve{q}\right)-\breve{q},z_n-\breve{q}⟩\le 0.\end{array}$$

(15)

It implies by (15) and the fact of $\frac{{\theta }_n}{{\alpha }_n}\parallel x_n-x_{n-1}\parallel \to 0$ that ${lim\; sup}_{n\to \infty }{t}_n\le 0$. From (14), using Lemma 4, we can conclude that $x_n\to \breve{q}$.

Case 2. 

Suppose that sequence ${\{x_n-\breve{q}\}}_{n\ge n_0}$ is not a monotone decreasing sequence for all $n_0$ large enough. Set

$${\mathsf{\Phi }}_n:={\parallel x_n-\breve{q}\parallel }^2.$$

So, there exists a subsequence $\left\{{\mathsf{\Phi }}_{n_j}\right\}$ of $\left\{{\mathsf{\Phi }}_n\right\}$ such that ${\mathsf{\Phi }}_{n_j}\le {\mathsf{\Phi }}_{n_j+1}$ for all $j\in \mathbb{N}$. In this case, we define $\lambda :\{n:n\ge n_0\}\to \mathbb{N}$ by

$$\lambda \left(n\right):=max\{k\in \mathbb{N}:k\le n,{\mathsf{\Phi }}_k<{\mathsf{\Phi }}_{k+1}\}.$$

By Lemma 4, we obtain ${\mathsf{\Phi }}_{\lambda \left(n\right)}\le {\mathsf{\Phi }}_{\lambda \left(n\right)+1}$ for all $n\ge n_0$. Then,

$$\parallel x_{\lambda \left(n\right)}-\breve{q}\parallel \le \parallel x_{\lambda \left(n\right)+1}\parallel \mathrm{for}\mathrm{all}n\ge n_0.$$

The same as the argument in Case 1, we obtain

$$\begin{array}{cc}\hfill \parallel z_{\lambda \left(n\right)}-T_{\lambda \left(n\right)}{z}_{\lambda \left(n\right)}\parallel & \le 2{\alpha }_{\lambda \left(n\right)}\parallel S\left(y_{\lambda \left(n\right)}\right)-y_{\lambda \left(n\right)}\parallel +(1+2{\kappa }_{\lambda \left(n\right)})\parallel y_{\lambda \left(n\right)}-T_{\lambda \left(n\right)}{y}_{\lambda \left(n\right)}\parallel \hfill \end{array}$$

(16)

for all $n\ge n_0$. Hence $\parallel z_{\lambda \left(n\right)}-T_{\lambda \left(n\right)}{z}_{\lambda \left(n\right)}\parallel \to 0$ as $n\to 0.$

Similary, we have ${lim\; sup}_{n\to \infty }⟨S\left(\breve{q}\right)-\breve{q},z_{\lambda \left(n\right)}-\breve{q}⟩\le 0$.

Since ${\mathsf{\Phi }}_{\lambda \left(n\right)}\le {\mathsf{\Phi }}_{\lambda \left(n\right)+1}$ and $0<(1-k){\alpha }_{\lambda \left(n\right)}$, we obtain

$$\begin{array}{cc}\hfill \parallel x_{\lambda \left(n\right)}-\breve{q}{\parallel }^2& \le \parallel x_{\lambda \left(n\right)+1}-\breve{q}{\parallel }^2\hfill \\ \hfill & \le (1-(1-k){\alpha }_{\lambda \left(n\right)}){\parallel x_{\lambda \left(n\right)}-\breve{q}\parallel }^2\hfill \\ \hfill & +{\alpha }_{\lambda \left(n\right)}(1-k)\left[\frac{2{\mathsf{\Phi }}_{\lambda \left(n\right)}}{{\alpha }_{\lambda \left(n\right)}(1-k)}\parallel x_{\lambda \left(n\right)}-\breve{q}\parallel \parallel x_{\lambda \left(n\right)}-x_{\lambda \left(n\right)-1}\parallel \right]\hfill \\ \hfill & +{\alpha }_{\lambda \left(n\right)}(1-k);\left[\frac{{\mathsf{\Phi }}_{\lambda \left(n\right)}^2}{{\alpha }_{\lambda \left(n\right)}(1-k)}\parallel x_{\lambda \left(n\right)}-x_{\lambda \left(n\right)-1}\parallel +\frac{2}{1-k}⟨S\left(\breve{q}\right)-\breve{q},z_{\lambda \left(n\right)}-\breve{q}⟩\right]\hfill \\ \hfill & \le \frac{2{\mathsf{\Phi }}_{\lambda \left(n\right)}}{{\alpha }_{\lambda \left(n\right)}(1-k)}\parallel x_{\lambda \left(n\right)}-\breve{q}\parallel \parallel x_{\lambda \left(n\right)}-x_{\lambda \left(n\right)-1}\parallel \hfill \\ \hfill & +\frac{{\mathsf{\Phi }}_{\lambda \left(n\right)}^2}{{\alpha }_{\lambda \left(n\right)}(1-k)}\parallel x_{\lambda \left(n\right)}-x_{\lambda \left(n\right)-1}\parallel +\frac{2}{1-k}⟨S\left(\breve{q}\right)-\breve{q},z_{\lambda \left(n\right)}-\breve{q}⟩.\hfill \end{array}$$

Since $\frac{{\mathsf{\Phi }}_n}{{\alpha }_n}\parallel x_n-x_{n-1}\parallel \to 0$ and ${lim\; sup}_{n\to \infty }⟨S\left(\breve{q}\right)-\breve{q},z_{\lambda \left(n\right)}-\breve{q}⟩\le 0$, it follows that

$$\underset{n\to \infty }{lim\; sup}{\parallel x_{\lambda \left(n\right)}-\breve{q}\parallel }^2\le 0,$$

and hence $\parallel x_{\lambda \left(n\right)}-\breve{q}\parallel \to 0$ as $n\to \infty$. It implies by

$$\begin{array}{cc}\hfill \parallel x_{\lambda \left(n\right)+1}-\breve{q}{\parallel }^2& \le (1-(1-k){\alpha }_{\lambda \left(n\right)}){\parallel x_{\lambda \left(n\right)}-\breve{q}\parallel }^2\hfill \\ \hfill & +{\alpha }_{\lambda \left(n\right)}(1-k)\left[\frac{2{\mathsf{\Phi }}_{\lambda \left(n\right)}}{{\alpha }_{\lambda \left(n\right)}(1-k)}\parallel x_{\lambda \left(n\right)}-\breve{q}\parallel \parallel x_{\lambda \left(n\right)}-x_{\lambda \left(n\right)-1}\parallel \right]\hfill \\ \hfill & +{\alpha }_{\lambda \left(n\right)}(1-k)\left[\frac{{\mathsf{\Phi }}_{\lambda \left(n\right)}^2}{{\alpha }_{\lambda \left(n\right)}(1-k)}\parallel x_{\lambda \left(n\right)}-x_{\lambda \left(n\right)-1}\parallel \right]\hfill \\ \hfill & +{\alpha }_{\lambda \left(n\right)}(1-k)\left[\frac{2}{1-k}⟨S\left(\breve{q}\right)-\breve{q},z_{\lambda \left(n\right)}-\breve{q}⟩\right]\hfill \end{array}$$

that $\parallel x_{\lambda \left(n\right)+1}-\breve{q}\parallel \to 0$ as $n\to \infty$. Using Lemma 4, we obtain $\parallel x_n-\breve{q}\parallel \le \parallel x_{\lambda \left(n\right)+1}-\breve{q}\parallel \to 0$ as $n\to \infty$. Hence $x_n\to \breve{q}$. The proof is complete.    □

Now, we employ Algorithm 4 for solving Problem (1). We obtain the following result as a consequence of Theorem 1.

Theorem 2. 

Let $\omega :{\mathbb{R}}^n\to \mathbb{R}$ be strongly convex with parameter $\sigma >0$ and a continuously differentiable function such that $\nabla \omega$ is Lipschitz continuous with constant $L_{\omega }$. Suppose that f and g satisfy the assumptions of (2). Let $\left\{c_n\right\}$ be a sequence of positive real numbers in $(0,2/L_f)$ such that $c_n\to c$ as $n\to \infty$ where $c\in (0,2/L_f)$ and let $\left\{x_n\right\}$ be a sequence generated by Algorithm 5. Then $\left\{x_n\right\}$ converges strongly to $\breve{q}\in \mathsf{\Gamma }$.

Algorithm 5 An Inertial Bilevel Gradient Modified Picard (IBiG-MP)

Initial. Take $x_0,x_1\in H$ arbitrarily and $t_1=0$. For $n\ge 1$. Set $$\begin{array}{c}\hfill {\theta }_n=\left\{\begin{array}{cc}min\{\frac{t_n-1}{t_{n+1}},\frac{{\gamma }_n{\alpha }_n}{\parallel x_n-x_{n-1}\parallel }\},\hfill & \mathrm{if}{x}_n\ne x_{n-1},\hfill \\ \frac{t_n-1}{t_{n+1}},\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$ (17)  where $t_{n+1}=\frac{1+\sqrt{1+4t_n^2}}{2}$. Step 1. Calculate $y_n$, $z_n$ and $x_{n+1}$ as follows: $$\begin{array}{cc}\hfill & y_n=x_n+{\theta }_n(x_n-x_{n-1}),\hfill \\ \hfill & z_n=(1-{\alpha }_n-{\kappa }_n)y_n+{\alpha }_n(I-s\nabla \omega )\left(y_n\right)+{\kappa }_{n}pro{x}_{c_{n}g}(I-c_n\nabla f)y_n,\hfill \\ \hfill & x_{n+1}=pro{x}_{c_{n}g}(I-c_n\nabla f)z_n,\hfill \end{array}$$  where $\left\{{\alpha }_n\right\}$, $\left\{{\kappa }_n\right\}\subset [0,1]$ with ${\alpha }_n+{\kappa }_n\le 1$. Then, update $n:=n+1$ and return to Step 1.

Proof. 

Let $T_n=pro{x}_{c_{n}g}(I-c_n\nabla f),n\in \mathbb{N}$ and $T=pro{x}_{cg}(I-c\nabla f)$. By Lemma 1 and Remark 1, we know that $\left\{T_n\right\}$ satisfies the condition (Z). From Theorem 1, we get that $\left\{x_n\right\}$ converges to $\breve{q}\in \mathsf{\Gamma }=arg{min}_{x\in {\mathbb{R}}^n}(f\left(x\right)+g\left(x\right))$. Notice also that $S=I-s\nabla \omega \left(x\right)$ is a k-contraction with parameter $k=\sqrt{1-\frac{2s\sigma L_{\omega }}{\sigma +L_{\omega }}}$, whenever $s\in (0,2/(\sigma +L_{\omega }))$. It remains to show that the variational inequality holds true. By using $\breve{q}=P_{\mathsf{\Gamma }}S\left(\breve{q}\right)$ and (5), for all $z\in \mathsf{\Gamma }$, we obtain

$$\begin{array}{cc}\hfill \breve{q}=P_{\mathsf{\Gamma }}S\left(\breve{q}\right)& \iff ⟨S\left(\breve{q}\right)-\breve{q},z-\breve{q}⟩\le 0\hfill \\ \hfill & \iff ⟨\breve{q}-s\nabla \omega \left(\breve{q}\right)-\breve{q},z-\breve{q}⟩\le 0\hfill \\ \hfill & \iff ⟨s\nabla \omega \left(\breve{q}\right),z-\breve{q}⟩\ge 0\hfill \\ \hfill & \iff s⟨\nabla \omega \left(\breve{q}\right),z-\breve{q}⟩\ge 0\hfill \\ \hfill & \iff ⟨\nabla \omega \left(\breve{q}\right),z-\breve{q}⟩\ge 0\mathrm{for}\mathrm{all}z\in \mathsf{\Gamma }=X^{*}.\hfill \end{array}$$

Thus, $\breve{q}$ is an optimal solution for the Problem (1). □

4. Application

In this section, we employ Algorithm 5 as a machine learning algorithm for regression, a graph of the Sine function and classification of some data by using a model of SLFNs (Single Hidden Layer Feedforward Neural Networks ) and Extreme Learning Machine. The MATLAB computing environment and an Intel Core-i5 gen 8 with 8 GB RAM(Integrated Electronics Corporation, Santa Clara, CA, USA) are used to perform all results.

We first recall a basic knowledge of the extreme learning machine for regression and classification problems. Moreover, we use the propose algorithm for solving these problems and compare the performance among Big-SAM, iBig-SAM and aiBig-SAM.

Extreme learning machine (ELM) [20] is defined as follows: Let $D=\{(x_d,q_d):x_d\in {\mathbb{R}}^n,q_d\in {\mathbb{R}}^m,d=1,2,\cdots ,N\}$ be a training set of N distinct samples, $x_d$ is an input data and $q_d$ is a target. A standard SLFNs with M hidden nodes and activation function $\phi \left(x\right)$ is given by

$$\sum _{j=1}^M{\xi }_j\phi (⟨p_j,x_d⟩+c_j)=o_d,d=1,\cdots ,N,$$

where ${\xi }_j$ is the weight vector connected between the j-th hidden node and the output node, $p_j$ is the weight vector connected between the j-th hidden node and the input node, and $c_j$ is the bias. The aim of SLFNs is to predict these N outputs such that ${\sum }_{d=1}^N∥o_d-q_d∥=0$. That is,

$$\begin{array}{c}\hfill \sum _{j=1}^M{\xi }_j\phi (⟨p_j,x_d⟩+c_j)=q_d,d=1,\cdots ,N.\end{array}$$

(18)

We can rewrite the above system of linear equation by the following matrix equation:

$$\begin{array}{c}\hfill R\xi =Q,\end{array}$$

(19)

where

$$\begin{array}{cc}\hfill & R={\left[\begin{array}{ccc}\phi (⟨p_1,x_1⟩+c_1)& \cdots & \phi (⟨p_M,x_1⟩+c_M)\\ ⋮& \ddots & ⋮\\ \phi (⟨p_1,x_N⟩+c_1)& \cdots & \phi (⟨p_M,x_N⟩+c_M)\end{array}\right]}_{N\times M}\hfill \\ \hfill \\ \hfill & \xi ={[{\xi }_1^T,\dots ,{\xi }_M^T]}_{m\times M}^T,Q={[q_1^T,\dots ,q_N^T]}_{m\times N}^T.\hfill \end{array}$$

The objective of an SLFNs is estimating ${\xi }_j$, $p_j$ and $c_j$ for solving (18) while ELM aims to find only ${\xi }_j$ with randomly $p_j$ and $c_j$.

The Problem (19) can be considered as the following convex minimization problem:

$$\underset{\xi }{min}{∥R\xi -Q∥}_2^2+\lambda {∥\xi ∥}_1,$$

where $\lambda >0$ is called the regularization parameter. In Algorithm 5, we set $f\left(\xi \right)={∥R\xi -Q∥}_2^2$ and $g\left(\xi \right)=\lambda {∥\xi ∥}_1$. We employ Algorithm 5 to solve convex bilevel optimization Problems (1) and (2) while the outer level function is defined by $\omega \left(\xi \right)=\frac{1}{2}{\parallel \xi \parallel }_2^2$.

4.1. Regression a Sine Function

In our experiment for the regression of a graph of the Sine function, we construct a training set by randomly selecting 10 distinct data. we use sigmoid as our activation function. We also set the number of hidden nodes $M=100$, and regularization parameter $\lambda =1\times {10}^{-5}$. In Algorithm 5, we set $T_n=pro{x}_{c_{n}g}(I-c_n\nabla f)$. The Lipschitz constant $L_f$ of gradient f is computed by ${2\parallel R\parallel }^2$. The values indicated in Table 1 are used for all control settings. We evaluate the result by

$$\begin{array}{c}\hfill Meansquarederror\left(MSE\right)=\frac{1}{n}\sum _{d=1}^n{\parallel o_d-q_d\parallel }^2.\end{array}$$

Table 1. Setting parameters of each Algorithm.

Methods	Setting
Algorithm 5	$s=0.01$, $c_n=\frac{1}{L_f}$, $t_1=0$, ${\alpha }_n=\frac{1}{55n}$, ${\kappa }_n=\frac{0.499(n+1)}{n}$, ${\gamma }_n=\frac{55·{10}^{20}}{n}$
BIG-SAM	$s=0.01$, $\alpha =3$, $c_n=\frac{1}{L_f}$, ${\alpha }_n=\frac{2\left(0.1\right)}{1-\frac{2+c_n{L}_f}{4}}$, ${\gamma }_n=\frac{{\alpha }_n}{n^{0.01}}$
iBIG-SAM	$s=0.01$, $\alpha =3$, $c_n=\frac{1}{L_f}$, ${\alpha }_n=\frac{2\left(0.1\right)}{1-\frac{2+c_n{L}_f}{4}}$, ${\gamma }_n=\frac{{\alpha }_n}{n^{0.01}}$
aiBIG-SAM	$s=0.01$, $\alpha =3$, $c_n=\frac{1}{L_f}$, ${\alpha }_n=\frac{1}{k+2}$, ${\gamma }_n=\frac{{\alpha }_n}{n^{0.01}}$

We then obtain results compared to BIG-SAM, iBIG-SAM and aiBIG-SAM as in Figure 1 and Table 2.

Figure 1. (a) A regression of the sine function at 100th step; (b) A regression of the sine function at 500th step.

Table 2. Numerical results for regression of a sine function with 500 iterations.

Methods	MSE	Computational Time
Algorithm 5	0.0011032	0.0216
BIG-SAM	0.3737940	0.0162
iBIG-SAM	0.3738094	0.0182
aiBIG-SAM	0.3685550	0.0172

Table 1 and Figure 1 show that Algorithm 5 gives a better performance to predict a sine function than others while there is little difference in processing time.

4.2. Data Classification

In order to classify datasets, we use four datasets from “https://www.kaggle.com/, accessed on 20 June 2020” and “https://archive.ics.uci.edu/, accessed on 20 June 2020” as follows:

Breast Cancer dataset [21] The dataset contains 11 attributes. In this dataset, we classify 2 classes of data.

Heart Disease UCI dataset [22] The dataset contains 14 attributes. There are two classes of this dataset.

Diabetes dataset [23] The dataset contains 9 attributes. In this dataset, we classify 2 classes of data.

Iris dataset [24] This dataset contains 3 classes of iris plant. The dataset contains 4 attributes. We aim to clasify each type of iris plant (Iris versicolour, Iris virginica and Iris setosa).

Table 3 shows the number of attributes of each dataset and the number of the training set (around $70\%$ of data) and testing set (remainder $30\%$ of data).

Table 3. Training and testing sets of each dataset.

Dataset	Attributes	Sample
		Train	Test
Breast Cancer	11	489	210
Heart Disease	14	213	90
Diabetes	9	538	230
Iris	4	105	45

We set all control parameters the same as in Table 1 in Section 4.1, the number of hidden nodes $M=100$, and activation function is sigmoid. Given a training set for each dataset as mentioned in Table 3, An accuracy of the output data is calculated by

$$\begin{array}{c}\hfill \mathrm{accuracy}=\frac{\mathrm{correct}\mathrm{predicted}\mathrm{data}}{\mathrm{all}\mathrm{data}}\times 100.\end{array}$$

We compare the iteration number, accuracy train and accuracy test of Algorithm 5 with the others on each dataset as seen in Table 4.

Table 4. The iteration number of each algorithm with the best accuracy on each dataset.

Dataset	Algorithm	Iteration No.	Accuracy Train	Accuracy Test
	Algorithm 5	577	96.55	99.50
Breast	BIG-SAM	1500	96.55	98.49
Cancer	iBIG-SAM	1500	96.55	98.49
	aiBIG-SAM	1500	96.55	98.49
	Algorithm 5	185	87.14	82.80
Heart	BIG-SAM	1797	86.19	82.80
Disease	iBIG-SAM	1756	86.19	82.80
	aiBIG-SAM	2501	86.67	82.80
Diabetes	Algorithm 5	100	77.11	81.53
	BIG-SAM	690	76.01	81.08
	iBIG-SAM	695	76.37	81.08
	aiBIG-SAM	1280	76.92	81.18
Iris	Algorithm 5	190	98.10	100.00
	BIG-SAM	781	94.29	95.56
	iBIG-SAM	777	94.29	95.56
	aiBIG-SAM	771	94.29	95.56

From Table 4, Algorithm 5 has a better performance of accuracy than BIG-SAM, iBIG-SAM, aiBIG-SAM in all experiments conducted.

5. Conclusions

We propose a new common fixed point algorithm for a countable family of nonexpansive operators and apply it to solve some convex bilevel optimization problems. We then prove a strong convergence theorem of the proposed algorithm under some suitable conditions. Moreover, we apply our algorithm to solve classification and regression problems. We also give numerical experiments for comparison of the performance of our proposed algorithm with the existing algorithms, the proposed algorithm is more efficient than the existing algorithms in the literature.