Converse Inertial Step Approach and Its Applications in Solving Nonexpansive Mapping

Abstract

In spite of great successes of the inertial step approach (ISA) in various fields, we are investigating the converse inertial step approach (CISA) for the first time. First, the classical Picard iteration for solving nonexpansive mappings converges weakly with CISA integration. Its analysis is based on the newly developed weak quasi-Fejér monotonicity under mild assumptions. We also establish $O(1/k^{\gamma })$ ($\gamma \in (0,1)$) and linear convergence rate under different assumptions. This extends the $O(1/k)$ convergence rate of the Krasnosel’skiĭ–Mann iteration. A generalized version of CISA is then studied. Second, combining CISA with over-relaxed step approach for solving nonexpansive mappings leads to a new algorithm, which not only converges without restrictive assumptions but also allows an inexact calculation in each iteration. Third, with CISA integration, a Backward–Forward splitting algorithm succeeds in accepting a larger step-size, and a Peaceman–Rachford splitting algorithm is guaranteed to converge.

1. Introduction

As is well known, the easy-to-implement inertial step approach (ISA) (difference between the current and the last iterative steps, i.e., $x_k-x_{k-1}$) have played a significant role in accelerating iterative algorithms and have attracted great attention in various fields [1,2,3,4]. It was first introduced by Polyak in the heavy ball method [5] to accelerate the first-order method. Nesterov’s optimal gradient method [6], which improves the convergence rate of the first-order method in minimizing gradient Lipchitz continuous convex function from $O(1/k)$ to $O(1/k^2)$, is in the scheme of ISA. Very recently, based on a slight variant of ISA, Attouch and Peypouquet [7] further improved the convergence rate in [6] from $O(1/k^2)$ to $o(1/k^2)$.

Generally, we can observe that ISA can (nonmonotonically) accelerate the existing convergent algorithms for solving averaged nonexpansive mappings [8], which aims at finding a fixed point of $(1-\lambda )I+\lambda T$, where $\lambda \in (0,1)$, $I:X\to X$ is the identity mapping and $T:X\to X$ is a nonexpansive mapping, which satisfies

$$\parallel Tx-Ty\parallel \le \parallel x-y\parallel ,\forall x,y\in X,$$

and X is a closed convex subset in Hilbert space $\mathcal{H}$ equipped with inner product $\langle x,y\rangle$ and the corresponding norm $\parallel ·\parallel$. Mathematically, $x\in X$ is defined as a fixed point of T if

$$x=Tx,x\in X.$$

(1)

Throughout this paper, the fixed point set of T is denoted by FixT and assumed to be nonempty. There are many applications for this problem [9,10,11,12]. Variational inequalities [13,14,15,16,17,18,19], equilibrium problems [20,21], and complementarity problems [22] can also be seen as special cases for this problem. The extreme case of the above averaged nonexpansive mappings with $\lambda =1$ is known as nonexpansive mappings [8], and there are many popular optimization algorithms in special applications of nonexpansive mappings and averaged nonexpansive mappings, such as the Proximal Point algorithm (PPA) [23,24], Douglas–Rachford splitting (DRS) algorithm [25], Peaceman–Rachford splitting (PRS) algorithm [26], Primal–Dual (PD) algorithms [27,28,29,30], and alternating direction method of multipliers (ADMM) algorithm [31,32,33,34,35]. To the best of our knowledge, there are no references on employing ISA for nonexpansive mappings. A possible reason is that any attempt may result in failure. This motivates our study on the converse inertia step approach (CISA), which is defined to be the difference between the last and current iterative steps ($x_{k-1}-x_k$). In this paper, we introduce CISA, for the first time, to solve nonexpansive mappings (1), and study what benefits we can achieve.

1.1. Difficulties in Solving Nonexpansive Mappings

In practical computations, solving the subproblems involving nonexpansive mappings exactly is often prohibitively expensive or even impossible. This has motivated the development of inexact solution methods, where the subproblems are solved only up to a certain tolerance. However, integrating such inexactness introduces significant challenges. First, the propagation and accumulation of errors must be carefully controlled to ensure the overall convergence of the algorithm. Second, the choice of the inexactness criterion is critical; an overly strict criterion negates the computational benefits of inexact solving, while an overly loose one may lead to divergence or instability. Furthermore, for inertial-type algorithms, the interaction between the extrapolation step and the approximation errors creates additional complexity in the analysis. These difficulties highlight the need for a robust theoretical framework that can accommodate computationally feasible inexact computations without compromising convergence guarantees.

1.2. Related Work

It is well known that the standard Picard iteration $x_{k+1}=T{x}_k$ for solving (1) may not converge, for a counter-example, $T=-I$. In order to guarantee the convergence, various modifications are introduced. A popular one is the Krasnosel’skiĭ–Mann (KM) iteration [36]

$$(\forall k\in \mathbb{N})x_{k+1}=(1-{\lambda }_k)x_k+{\lambda }_{k}T{x}_k,{\lambda }_k\in [0,1].$$

(2)

Under the assumption ${\sum }_{k=0}^{\infty }{\lambda }_k(1-{\lambda }_k)=\infty$, weak convergence of ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ was established in [37]. Different from the KM iteration, Mann [38] proposed another modification approach. All the past iterations ${\left\{x_i\right\}}_{0\le i\le k}$ are convexly combined with a very carefully selected coefficient to build convergent iterative schemes. Moreover, there is some equivalence between the KM iteration and Mann’s method [38], as seen in [37,39]. In order to relax the restrictive assumptions on the coefficients of past iterations ${\left\{x_i\right\}}_{0\le i\le k}$, Combettes and Pennanen [40] improved Mann’s method by combining it with KM iteration. As an extension, Combettes and Glaudin [41] established a unified algorithmic scheme by further combining the method in [40] with the m-layer algorithm [42] for quasi-nonexpansive mapping [8], under restrictive assumptions such as requiring any weak cluster point of the iterative sequences to be a fixed point of T. Moreover, for this algorithm [41], there is no any analysis on the convergence rate.

In order to accelerate the KM iteration, apart from ISA, the over-relaxed step approach (ORSA) (i.e., ${\lambda }_k\ge 1$ in the KM iteration) is a effective way. Corman and Yuan [43] first introduced the ORSA to accelerate the convergence of PPA, under strong assumptions on the resolvent operator. ORSA has also be extended to the Augmented Lagrangian Method (ALM), Alternating Direction Method of Multipliers (ADMM). In [44,45], the authors extended the PRS algorithm to the sum of two maximal strongly monotone operators, using a larger relaxed parameter. It can be regarded as combining the PRS algorithm with ORSA. Very recently, Themelis and Patrinos [46] analyzed Douglas–Rachford splitting with ORSA based on the Douglas–Rachford envelope for the sum of a strongly convex function with gradient Lipchitz continuous and a nonconvex function. Note that all the above-mentioned references [43,44,45,46] require very restrictive assumptions and must be errorless during iterative calculation.

1.3. Contributions

We list in the following the contributions of this paper:

• We show that the classical Picard iteration for solving nonexpansive mappings converges weakly with CISA integration. It leads to the newly proposed CISA algorithm, which only uses the last two rather than the whole past iterations. We introduce a new framework of weak quasi-Fejér monotonicity (see Section 2 for more details) in the convergence analysis. Moreover, our assumptions are much more relaxed than those made in [41]. As a further extension, a generalized version of CISA (GCISA) is presented.
• We establish $O(1/k^{\gamma })$ ($\gamma \in (0,1)$) and linear convergence rates of the CISA algorithm under different assumptions. In particular, our analysis on linear convergence is different from that of the existing analysis [47,48] due to the special structure of CISA.
• By combining CISA with ORSA, we develop the CISA-ORSA algorithm with a better numerical performance. The usual restrictive assumptions required in [43,44,45,46] are no longer needed in our analysis. Moreover, our CISA-ORSA algorithm allows inexact calculation in each iteration.
• With CISA integration, the Backward–Forward splitting (BFS) algorithm [49] could accept a larger step-size and the Peaceman–Rachford splitting (PRS) algorithm [26] could be guaranteed to converge.

1.4. Organization

Section 2 presents some preliminaries and lemmas. Section 3 investigates CISA. We develop the CISA algorithm and analyze the impact of CISA for solving nonexpansive mappings and corresponding convergence rate in Section 3.1. A general version of CISA (GIISA) is studied in Section 3.2. Section 4 explores the relationship between CISA and ORSA. Section 5 combines CISA with BFS algorithm and PRS algorithm, respectively. Conclusions are made in Section 6.

2. Preliminaries

We first give some notations, definitions and lemmas, which are useful for later analysis. Throughout this section, let S be a nonempty convex set in Hilbert space.

Let $\gamma >0$ and ${\mathrm{prox}}_{\gamma f}$ be the proximal mapping of $f:\mathcal{H}\to \mathbb{R}$, defined as

$${\mathrm{prox}}_{\gamma f}\left(x\right):=arg\underset{y\in \mathcal{H}}{min}f\left(y\right)+{\displaystyle \frac{1}{2\gamma }}{\parallel y-x\parallel }^2.$$

Definition 1

(quasi-Fejér monotone [48]). The sequence ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ in Hilbert space is quasi-Fejér monotone with respect to S, if it holds that

$$\parallel x_{k+1}-a\parallel \le \parallel x_k-a\parallel + {\epsilon }_k,k\in \mathbb{N},$$

for all $a\in S$ and ${\left\{{\epsilon }_k\right\}}_{k\in \mathbb{N}}$ satisfying ${\epsilon }_k\ge 0$ and ${\sum }_{k=0}^{\infty }{\epsilon }_k<\infty$.

Lemma 1

(Theorem 5.5 [8]). If the sequence ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ in Hilbert space satisfies the following:

• ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ is quasi-Fejér monotone with respect to S;
• every weak cluster of ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ belongs to S;

then ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ converges weakly to a point of S.

Lemma 2.

Suppose nonnegative sequences ${\left\{a_k\right\}}_{k\in \mathbb{N}}$ and ${\left\{{\epsilon }_k\right\}}_{k\in \mathbb{N}}$ satisfy that $a_{k+1}\le a_k+{\epsilon }_k$ and ${\sum }_{k=0}^{\infty }{\epsilon }_k<\infty$. Then, ${\left\{a_k\right\}}_{k\in \mathbb{N}}$ is convergent.

Denote by dist$(x_k,a)$ the Euclidean distance between $x_k$ and a. For any ${\alpha }_k\in [0,1]$, let $y_k=x_k-{\alpha }_k(x_k-x_{k-1})$ as located in Figure 1. One can verify that

$$\mathrm{dist}(x_{k+1},a)\le \mathrm{dist}(y_k,a)\le (1-{\alpha }_k)\mathrm{dist}(x_k,a)+{\alpha }_k\mathrm{dist}(x_{k-1},a).$$

It is interesting to see that Figure 1 geometrically characterizes the iterative process of the exact CISA algorithm (7) in ${\mathbb{R}}^2$.

Figure 1. The point $x_{k+1}$ falls into the disk centered at a with a radius dist $(y_k,a)$.

Figure 1. The point $x_{k+1}$ falls into the disk centered at a with a radius dist $(y_k,a)$.

Motivated by Figure 1, we make the following extension of quasi-Fejér monotonicity, which is used for later analysis.

Definition 2

(weak quasi-Fejér monotone). The sequence ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ in a Hilbert space is said to be weak quasi-Fejér monotone with respect to S, if for all $a\in S$, there exist sequences ${\left\{{\alpha }_k\right\}}_{k\in \mathbb{N}}\subset [0,1]$ and ${\left\{{\epsilon }_k\right\}}_{k\in \mathbb{N}}\subset [0,\infty )$ with ${\sum }_{k=0}^{\infty }{\epsilon }_k<\infty$ such that

$$\parallel x_{k+1}-a\parallel \le (1-{\alpha }_k)\parallel x_k-a\parallel + {\alpha }_k\parallel x_{k-1}-a\parallel +{\epsilon }_k,\forall k\in \mathbb{N}.$$

Setting ${\alpha }_k\equiv 0$ reduces the definition of weak quasi-Fejér monotonicity to that of quasi-Fejér monotonicity. The following example shows that there is a weak quasi-Fejér monotone sequence ${\left\{x_k\right\}}_{k\in \mathbb{N}}$, which is not quasi-Fejér monotone:

$$\parallel x_k-a\parallel =\mathrm{mod}(k,2)+1,{\alpha }_k=1-\mathrm{mod}(k,2)/2,$$

where mod$(k,2)$ returns the remainder of k divided by 2.

As a further extension, we propose in the following a generalized version of weak quasi-Fejér monotonicity. Throughout this paper, we assume $m\ge 2$.

Definition 3

(m weak quasi-Fejér monotone). The sequence ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ in Hilbert space is m weak quasi-Fejér monotone with respect to S, if it holds that

$$\parallel x_{k+1}-a\parallel \le \sum _{i=k-m+1}^k{\alpha }_{k,i}\parallel x_i-a\parallel +{\epsilon }_k,\sum _{i=k-m+1}^k{\alpha }_{k,i}=1,{\alpha }_{k,i}\in [0,1],$$

(3)

for all $a\in S$, ${\epsilon }_k\ge 0$, and ${\sum }_{k=0}^{\infty }{\epsilon }_k<\infty$.

It is trivial to observe that the definition of quasi-Fejér monotonicity corresponds to that of 2 weak quasi-Fejér monotonicity. Similarly to Lemma 1, which clarifies the relation between quasi-Fejér monotonicity and weak convergence of ${\left\{x_k\right\}}_{k\in \mathbb{N}}$, the following lemma establishes the relation between m weak quasi-Fejér monotonicity and weak convergence of ${\left\{x_k\right\}}_{k\in \mathbb{N}}$.

Lemma 3.

If the sequence ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ in Hilbert space satisfies the following:

• ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ is m weak quasi-Fejér monotone with respect to S;
• every weak cluster of ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ belongs to S;
• for $1\le i\le m-1$,

  $$\sum _{k=m-1}^{\infty }|{\alpha }_{k,k-i}-{\alpha }_{k-i,k-2i}|<\infty .$$

  (4)

Then, ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ converges weakly to a point of S.

Proof. 

It follows from (3) that, for $a\in S$,

$$\parallel x_{k+1}-a\parallel \le \underset{k-m+1\le i\le k}{max}\parallel x_i-a\parallel +{\epsilon }_k\le \cdots \le \underset{1-m\le i\le 0}{max}\parallel x_i-a\parallel +\sum _{i=0}^k{\epsilon }_i.$$

According to ${\sum }_{i=0}^{\infty }{\epsilon }_k<\infty$, then $\{\parallel x_k-a{\parallel \}}_{k\in \mathbb{N}}$ is bounded. Again, (3) implies that

$$\begin{array}{c}\hfill \parallel x_{k+1}-a\parallel \le \parallel x_k-a\parallel +\sum _{i=k-m+1}^{k-1}{\alpha }_{k,i}(\parallel x_i-a\parallel - \parallel x_k-a\parallel )+{\epsilon }_k.\end{array}$$

(5)

Since we can verify that

$$\begin{array}{ccc}& & \sum _{k=m-1}^K\left(\sum _{i=k-m+1}^{k-1}{\alpha }_{k,i}\left(\parallel x_i-a\parallel -\parallel x_k-a\parallel \right)+{\epsilon }_k\right)\hfill \\ & & =\sum _{k=m-1}^K\left(\sum _{i=1}^{m-1}{\alpha }_{k,k-i}\left(\parallel x_{k-i}-a\parallel -\parallel x_k-a\parallel \right)+{\epsilon }_k\right)\hfill \\ & & =\sum _{k=m-1}^K(\sum _{i=1}^{m-1}({\alpha }_{k-i,k-2i}\parallel x_{k-i}-a\parallel -{\alpha }_{k,k-i}\parallel x_k-a\parallel \hfill \\ & & +({\alpha }_{k,k-i}-{\alpha }_{k-i,k-2i})\parallel x_{k-i}-a\parallel )+{\epsilon }_k)\hfill \\ & & =\sum _{i=1}^{m-1}\sum _{j=0}^{i-1}{\alpha }_{m-1-i+j,m-1-2i+j}\parallel x_{m-1-i+j}-a\parallel -\sum _{i=1}^{m-1}\sum _{j=0}^{i-1}{\alpha }_{K-j,K-j-i}\parallel x_{K-j}-a\parallel \hfill \\ & & +\sum _{k=m-1}^K\left(\sum _{i=1}^{m-1}({\alpha }_{k,k-i}-{\alpha }_{k-i,k-2i})\parallel x_{k-i}-a\parallel +{\epsilon }_k\right)<\infty ,K\to \infty .\hfill \end{array}$$

(6)

where the last inequality follows from the boundedness of $\{\parallel x_k-a{\parallel \}}_{k\in \mathbb{N}}$ and ${\sum }_{i=0}^{\infty }{\epsilon }_k$, as well as the assumption (4). Therefore, (5) and (6) yield that ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ is quasi-Fejér monotone with respect to S. Hence, according to Lemma 1, ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ converges weakly to a point of S.          □

Remark 1.

If we select

$${\alpha }_{k,i}=\left\{\begin{array}{cc}{\alpha }_k\hfill & i=k,\hfill \\ (1-{\alpha }_k)\hfill & i=k-1,\hfill \\ 0\hfill & k-m+1\le i<k-1,\hfill \end{array}\right.$$

then the assumption (4) becomes ${\sum }_{k=1}^{\infty }|{\alpha }_k-{\alpha }_{k-1}|<\infty$.

Finally, the following two facts will be used frequently in latter analysis. For $a,b\in \mathcal{H}$, we have the following:

F1.

${\parallel a+b\parallel }^2\le {\parallel a\parallel }^2+2\langle b,a+b\rangle .$

F2.

${\parallel sa+tb\parallel }^2={s(s+t)\parallel a\parallel }^2+{t(s+t)\parallel b\parallel }^2-st{\parallel a-b\parallel }^2,\forall s,t\in \mathbb{R}$.

3. Investigation of CISA

We first study the role of CISA in solving nonexpansive mappings, and then make a generalization of CISA.

3.1. CISA in Solving Nonexpansive Mapping

We first present in the following the CISA algorithm (Algorithm 1) for solving nonexpansive mappings (1):

Algorithm 1 CISA

Input:  Initial points $x_{-1},x_0\in X$; sequence $\left\{{\alpha }_k\right\}\subset [0,1]$; error sequence $\left\{e_k\right\}$   1: for $k=0,1,2,\dots$ do   2:     Step 1: Converse Inertial Extrapolation   3:     $y_k=x_k-{\alpha }_k(x_k-x_{k-1}).$   4:     Step 2: Inexact Operator Evaluation   5:     $x_{k+1}=T\left(y_k\right)+e_k.$   6:     Note: $e_k$ represents computational error in evaluating $T\left(y_k\right)$   7: end for Output:  $x_k$ conditions

$$(x_{-1},x_0\in X)\left\{\begin{array}{cc}\hfill & y_k=x_k-{\alpha }_k(x_k-x_{k-1}),\hfill \\ \hfill & x_{k+1}=T{y}_k+e_k,\hfill \end{array}\right.$$

(7)

where ${\alpha }_k\in [0,1]$ and $e_k$ characterize the error when calculating $T{y}_k$. In spite of the impossibility of applying ISA in solving nonexpansive mappings, we can show that CISA stably guarantees the weak convergence for solving nonexpansive mappings. Our analysis, based on the above newly developed weak quasi-Fejér monotonicity, is under very mild assumptions on ${\alpha }_k$ and $e_k$ rather than strong assumptions in [41]. In addition, in each step, we only use the last two steps $x_k$ and $x_{k-1}$, greatly reducing the cost of storing all the past iterative sequences as in [41]. It is not difficult to verify that Figure 1 geometrically characterizes the exact iterative process of the exact CISA algorithm (7) ($e_k=0$) in ${\mathbb{R}}^2$ if $a\in \mathrm{Fix}T$.

Theorem 1.

Under the assumption that

$$\begin{array}{cc}\hfill & \sum _{k=0}^{\infty }\parallel e_k\parallel <\infty ,\hfill \end{array}$$

(8a)

$$\begin{array}{cc}\hfill & \sum _{k=0}^{\infty }{\alpha }_k(1-{\alpha }_k)=\infty ,\hfill \end{array}$$

(8b)

$$\begin{array}{cc}\hfill & \sum _{k=1}^{\infty }|{\alpha }_k-{\alpha }_{k-1}|<\infty ,\hfill \end{array}$$

(8c)

the sequence ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ generated by CISA algorithm (7) satisfies the following:

(i)

${\left\{x_k\right\}}_{k\in \mathbb{N}}$ is bounded.

(ii)

${\left\{x_k\right\}}_{k\in \mathbb{N}}$ is weak quasi-Fejér monotone with respect to FixT.

(iii)

every weak cluster of ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ belongs to FixT.

(iv)

${\left\{x_k\right\}}_{k\in \mathbb{N}}$ converges weakly to a point of FixT.

Proof. 

(i) Suppose $x_{*}\in \mathrm{Fix}T$ is a fixed point of T, for $k\in \mathbb{N}$, we have

$$\begin{array}{ccc}\hfill \parallel x_{k+1}-x_{*}\parallel & =& \parallel T\left((1-{\alpha }_k)x_k+{\alpha }_k{x}_{k-1}\right)-x_{*}+e_k\parallel \hfill \\ & \le & \parallel T\left((1-{\alpha }_k)x_k+{\alpha }_k{x}_{k-1}\right)-x_{*}\parallel + \parallel e_k\parallel \hfill \\ & \le & \parallel (1-{\alpha }_k)x_k+{\alpha }_k{x}_{k-1}-x_{*}\parallel + \parallel e_k\parallel \hfill \end{array}$$

(9)

$$\begin{array}{ccc}& \le & (1-{\alpha }_k)\parallel x_k-x_{*}\parallel + {\alpha }_k\parallel x_{k-1}-x_{*}\parallel + \parallel e_k\parallel \hfill \\ & \le & max\{\parallel x_k-x_{*}\parallel ,\parallel x_{k-1}-x_{*}\parallel \} + \parallel e_k\parallel \hfill \\ & \le & max\{\parallel x_0-x_{*}\parallel ,\parallel x_{-1}-x_{*}\parallel \}+\sum _{i=0}^k\parallel e_i\parallel \hfill \\ & <& \infty ,\hfill \end{array}$$

(10)

where (9) holds since T is nonexpansive and $x_{*}\in \mathrm{Fix}T$, and the last inequality follows from the assumption (8a). That is, ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ is bounded.

(ii) Weak quasi-Fejér monotone of ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ directly follows from (10) together with the assumption (8a).

(iii) According to the nonexpansiveness of T, we have

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \parallel x_{k+1}-x_{*}{\parallel }^2& = \parallel T\left((1-{\alpha }_k)x_k+{\alpha }_k{x}_{k-1}\right)-x_{*}+e_k{\parallel }^2\hfill \\ & \stackrel{\mathrm{F}1}{\le }{\parallel T\left((1-{\alpha }_k)x_k+{\alpha }_k{x}_{k-1}\right)-x_{*}\parallel }^2+2\langle e_k,x_{k+1}-x_{*}\rangle \hfill \\ & \le \parallel (1-{\alpha }_k)x_k+{\alpha }_k{x}_{k-1}-x_{*}{\parallel }^2+ 2\langle e_k,x_{k+1}-x_{*}\rangle \hfill \\ & \stackrel{\mathrm{F}2}{=}(1-{\alpha }_k)\parallel x_k-x_{*}{\parallel }^2+{\alpha }_k{\parallel x_{k-1}-x_{*}\parallel }^2\hfill \\ & -{\alpha }_k(1-{\alpha }_k){\parallel x_k-x_{k-1}\parallel }^2+2\langle e_k,x_{k+1}-x_{*}\rangle .\hfill \end{array}\end{array}$$

(11)

Rearranging the summation of the above inequalities (11) from $k=0$ to K gives the following inequation:

$$\begin{array}{cc}\hfill & \sum _{k=0}^K{\alpha }_k(1-{\alpha }_k){\parallel x_k-x_{k-1}\parallel }^2\hfill \\ \hfill \le & \sum _{k=0}^K\left((\parallel x_k-x_{*}{\parallel }^2+ {\alpha }_{k-1}\parallel x_{k-1}-x_{*}{\parallel }^2)-(\parallel x_{k+1}-x_{*}{\parallel }^2+ {\alpha }_k\parallel x_k-x_{*}{\parallel }^2)\right)\hfill \\ & +\sum _{k=0}^K\left(({\alpha }_k-{\alpha }_{k-1}){\parallel x_k-x_{*}\parallel }^2+2\langle e_k,x_{k+1}-x_{*}\rangle \right)\hfill \\ \hfill =& (\parallel x_0-x_{*}{\parallel }^2+{\alpha }_0\parallel x_{-1}-x_{*}{\parallel }^2)-(\parallel x_{K+1}-x_{*}{\parallel }^2+{\alpha }_K\parallel x_K-x_{*}{\parallel }^2)\hfill \\ & +\sum _{k=0}^K\left(({\alpha }_k-{\alpha }_{k-1}){\parallel x_k-x_{*}\parallel }^2+2\langle e_k,x_{k+1}-x_{*}\rangle \right).\hfill \end{array}$$

(12)

Notice that

$$\langle e_k,x_{k+1}-x_{*}\rangle \le \parallel e_k\parallel ·\parallel x_{k+1}-x_{*}\parallel .$$

According to the boundedness of ${\left\{x_k\right\}}_{k\in \mathbb{N}}$, assumptions (8a) and (8c), we obtain

$$\sum _{k=0}^{\infty }{\alpha }_k(1-{\alpha }_k){\parallel x_k-x_{k-1}\parallel }^2<\infty .$$

(13)

Combining the assumption (8b) with (13) implies that

$$\underset{k\to \infty }{lim\; inf}{\parallel x_k-x_{k-1}\parallel }^2=0.$$

(14)

Again, based on the nonexpansiveness of T, we have

$$\begin{array}{cc}& \parallel x_{k+1}-x_k{\parallel }^2 ={\parallel T{y}_k-T{y}_{k-1}+e_k-e_{k-1}\parallel }^2\hfill \\ \hfill \stackrel{\mathrm{F}1}{\le }& \parallel y_k-y_{k-1}{\parallel }^2+ 2\langle e_k-e_{k-1},x_{k+1}-x_k\rangle \hfill \\ \hfill =& \parallel (1-{\alpha }_k)(x_k-x_{k-1})+{\alpha }_{k-1}(x_{k-1}-x_{k-2}){\parallel }^2+ 2\langle e_k-e_{k-1},x_{k+1}-x_k\rangle \hfill \\ \hfill \stackrel{\mathrm{F}2}{\le }& (1-{\alpha }_k-{\alpha }_{k-1})\left((1-{\alpha }_k)\parallel x_k-x_{k-1}{\parallel }^2+{\alpha }_{k-1}{\parallel x_{k-1}-x_{k-2}\parallel }^2\right)\hfill \\ & + 2\langle e_k-e_{k-1},x_{k+1}-x_k\rangle \hfill \\ \hfill \le & (1-{\alpha }_k)\parallel x_k-x_{k-1}{\parallel }^2+ {\alpha }_{k-1}\parallel x_{k-1}-x_{k-2}{\parallel }^2+|{\alpha }_k-{\alpha }_{k-1}\left|\right(\parallel x_k-x_{k-1}{\parallel }^2\hfill \\ & + \parallel x_{k-1}-x_{k-2}{\parallel }^2) + 2(\parallel e_k\parallel + \parallel e_{k-1}\parallel )\parallel x_{k+1}-x_k\parallel .\hfill \end{array}$$

(15)

We conclude that $\parallel x_k-x_{k-1}{\parallel }^2$ is convergent by substituting $a_k={\parallel x_k-x_{k-1}\parallel }^2$ and ${\epsilon }_k={\alpha }_{k-1}\parallel x_{k-1}-x_{k-2}{\parallel }^2- {\alpha }_k\parallel x_k-x_{k-1}{\parallel }^2+ |{\alpha }_k-{\alpha }_{k-1}\left|\right(\parallel x_k-x_{k-1}{\parallel }^2+ \parallel x_{k-1}-x_{k-2}{\parallel }^2)+ 2(\parallel e_k\parallel +\parallel e_{k-1}\parallel )\parallel x_{k+1}-x_k\parallel$ into Lemma 2. Thus, according to (14), it holds that $\underset{k\to \infty }{lim}{\parallel x_k-x_{k-1}\parallel }^2=0.$ Therefore, we have

$$\begin{array}{ccc}\hfill \underset{k\to \infty }{lim}\parallel T{x}_k-x_k\parallel & =& \underset{k\to \infty }{lim}\parallel T{x}_k-T{y}_k+T{y}_k-x_k\parallel \hfill \\ & \le & \underset{k\to \infty }{lim}\left({\alpha }_k\parallel x_k-x_{k-1}\parallel + \parallel x_{k+1}-x_k\parallel + \parallel e_k\parallel \right)\hfill \\ & =& 0.\hfill \end{array}$$

Together with the boundedness of ${\left\{x_k\right\}}_{k\in \mathbb{N}}$, it implies that every weak cluster of ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ belongs to FixT.

(iv) According to Lemma 3 with m = 2, it follows from (ii) and (iii) that ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ converges weakly to a point of FixT.          □

As shown in the following corollary, ${\left\{y_k\right\}}_{k\in \mathbb{N}}$ is also a weak quasi-Fejér monotone and thus converges weakly to a point of FixT.

Corollary 1.

Under assumption (8a), the sequence ${\left\{y_k\right\}}_{k\in \mathbb{N}}$ generated by CISA algorithm (7) satisfies the following:

(i)

${\left\{y_k\right\}}_{k\in \mathbb{N}}$ is bounded.

(ii)

${\left\{y_k\right\}}_{k\in \mathbb{N}}$ is weak quasi-Fejér monotone.

Under additional assumptions (8b) and (8c), the following holds:

(iii)

Every weak cluster of ${\left(y_k\right)}_{k\in \mathbb{N}}$ belongs to FixT.

(iv)

${\left(y_k\right)}_{k\in \mathbb{N}}$ converges weakly to a point of FixT.

Proof. 

Observing that CISA algorithm (7) is equivalent to

$$y_{k+1}=(1-{\alpha }_{k+1})T{y}_k+{\alpha }_{k+1}T{y}_{k-1}+(1-{\alpha }_{k+1})e_k+{\alpha }_{k+1}{e}_{k-1},$$

we can prove (i) and (ii) as with Theorem 1.

Proof of (iii) follows from the fact:

$$\parallel T{y}_k-y_k\parallel \le \parallel x_{k+1}-y_k-e_k\parallel \le \parallel x_{k+1}-x_k\parallel + {\alpha }_k\parallel x_k-x_{k-1}\parallel + \parallel e_k\parallel .$$

Proof of (iv) is similar as that in Theorem 1.          □

Remark 2.

Assumptions (8b) and (8c) are easy to satisfy. We list in the following two examples:

• ${\alpha }_k\equiv \alpha \in (0,1)$;
• ${\alpha }_k={\displaystyle \frac{1}{{(k+1)}^p}},p\in (0,1].$

The sublinear convergence of ClSA agorithm (7) is then established.

Theorem 2.

Let ${\left\{z_k\right\}}_{k\in \mathbb{N}}\in \{{\left\{x_k\right\}}_{k\in \mathbb{N}},{\left(y_k\right)}_{k\in \mathbb{N}}\}$ be generated by CISA algorithm (7).

(i)

If ${\alpha }_k\equiv \alpha \in (0,1)$ and ${\sum }_{k=0}^{\infty }k\parallel e_k\parallel <\infty$, then

$$\parallel T{z}_k-z_k{\parallel }^2 =O\left({\displaystyle \frac{1}{k}}\right).$$

(ii)

If ${\alpha }_k={\displaystyle \frac{1}{{(k+1)}^p}}$ for $p\in (0,1]$ and ${\sum }_{k=0}^{\infty }{k}^{1-p}\parallel e_k\parallel <\infty$, then

$$\parallel T{z}_k-z_k{\parallel }^2=\left\{\begin{array}{cc}O\left({\displaystyle \frac{1}{lnk}}\right)\hfill & p=1;\hfill \\ O\left({\displaystyle \frac{1}{k^{1-p}}}\right)\hfill & p\in [{\displaystyle \frac{1}{2}},1);\hfill \\ O\left({\displaystyle \frac{1}{k^p}}\right)\hfill & p\in (0,{\displaystyle \frac{1}{2}}).\hfill \end{array}\right.$$

Proof. 

Rearranging the inequality (15) gives

$$\begin{array}{cc}& \parallel x_{k+1}-x_k{\parallel }^2+ {\alpha }_k{\parallel x_k-x_{k-1}\parallel }^2\hfill \\ \hfill & \le \parallel x_k-x_{k-1}{\parallel }^2+ {\alpha }_{k-1}{\parallel x_{k-1}-x_{k-2}\parallel }^2+2\delta \left(|{\alpha }_k-{\alpha }_{k-1}| + \parallel e_k\parallel + \parallel e_{k-1}\parallel \right),\hfill \end{array}$$

(16)

where $\delta =2\underset{k\in \mathbb{N}}{max}\{\parallel x_k-x_{k-1}\parallel \}$.

Proof of (i). For the case ${\alpha }_k\equiv \alpha \in (0,1)$, by repeatedly applying (16), we have

$$\begin{array}{ccc}& & k\left(\parallel x_{k+1}-x_k{\parallel }^2+ \alpha {\parallel x_k-x_{k-1}\parallel }^2\right)\hfill \\ & & \le k\left(\parallel x_k-x_{k-1}{\parallel }^2+ \alpha {\parallel x_{k-1}-x_{k-2}\parallel }^2\right)+2k\delta \left(\parallel e_k\parallel + \parallel e_{k-1}\parallel \right),\hfill \\ & & \le \sum _{i=0}^{k-1}\left(\parallel x_{i+1}-x_i{\parallel }^2+ \alpha {\parallel x_i-x_{i-1}\parallel }^2\right)+2\delta \sum _{i=1}^{k}i\left(\parallel e_i\parallel + \parallel e_{i-1}\parallel \right)\hfill \\ & & <\infty ,\hfill \end{array}$$

(17)

where the last inequality follows from (13) and ${\sum }_{k=0}^{\infty }k\parallel e_k\parallel <\infty$. Then, it holds that, for $k\to \infty$,

$$\parallel x_{k+1}-x_k{\parallel }^2=O\left(1/k\right),$$

(18)

Substituting (18) into the following two inequalities completes the proof of (i).

$$\begin{array}{ccc}& & \begin{array}{cc}\hfill \parallel T{x}_k-x_k{\parallel }^2& = \parallel T{x}_k-T{y}_k+T{y}_k-x_k{\parallel }^2 ={\parallel T{x}_k-T{y}_k+x_{k+1}-x_k-e_k\parallel }^2\hfill \\ & \le 3\parallel T{x}_k-T{y}_k{\parallel }^2+ 3\parallel x_{k+1}-x_k{\parallel }^2+ 3{\parallel e_k\parallel }^2\hfill \\ & \le 3{\alpha }^2\parallel x_k-x_{k-1}{\parallel }^2+ 3\parallel x_{k+1}-x_k{\parallel }^2+ 3{\parallel e_k\parallel }^2.\hfill \end{array}\hfill \end{array}$$

(19)

$$\begin{array}{ccc}& & \begin{array}{cc}\hfill \parallel T{y}_k-y_k{\parallel }^2& = \parallel x_{k+1}-y_k-e_k{\parallel }^2 ={\parallel x_{k+1}-x_k+\alpha (x_k-x_{k-1})-e_k\parallel }^2\hfill \\ & \le 3{\alpha }^2\parallel x_k-x_{k-1}{\parallel }^2+ 3\parallel x_{k+1}-x_k{\parallel }^2+ 3{\parallel e_k\parallel }^2.\hfill \end{array}\hfill \end{array}$$

(20)

Proof of (ii). For ${\alpha }_k={\displaystyle \frac{1}{{(k+1)}^p}}$ ($p\in (0,1]$), repeatedly applying (16) implies that

$$\begin{array}{cc}\hfill & \left(\sum _{i=1}^k{\displaystyle \frac{1}{{(i+1)}^p}}\right)\left(\parallel x_{k+1}-x_k{\parallel }^2+ {\displaystyle \frac{1}{{(k+1)}^p}}{\parallel x_k-x_{k-1}\parallel }^2\right)\hfill \\ \hfill \le & \sum _{i=0}^{k-1}{\displaystyle \frac{1}{{(i+1)}^p}}\left(\parallel x_{i+1}-x_i{\parallel }^2+ {\displaystyle \frac{1}{{(i+1)}^p}}{\parallel x_i-x_{i-1}\parallel }^2\right)+\hfill \\ & 2\delta \sum _{i=1}^{k}i{\displaystyle \frac{1}{{(i+1)}^p}}\left({\displaystyle \frac{1}{i^p}}-{\displaystyle \frac{1}{{(i+1)}^p}}\right)+2\delta \sum _{i=1}^{k}i{\displaystyle \frac{1}{{(i+1)}^p}}(\parallel e_i\parallel + \parallel e_{i-1}\parallel ).\hfill \end{array}$$

(21)

Dividing both sides of the above inequality by ${\sum }_{i=1}^{k}1/{(i+1)}^p$, we obtain that $\parallel x_{k+1}-x_k{\parallel }^2$ has the same order of k as

$${\displaystyle \frac{{\sum }_{i=1}^{k}i\frac{1}{{(i+1)}^p}\left(\frac{1}{i^p}-\frac{1}{{(i+1)}^p}\right)}{{\sum }_{i=1}^k\frac{1}{{(i+1)}^p}}}=\left\{\begin{array}{cc}O\left({\displaystyle \frac{1}{lnk}}\right)\hfill & p=1;\hfill \\ O\left({\displaystyle \frac{1}{k^{1-p}}}\right)\hfill & p\in [{\displaystyle \frac{1}{2}},1);\hfill \\ O\left({\displaystyle \frac{1}{k^p}}\right)\hfill & p\in (0,{\displaystyle \frac{1}{2}}),\hfill \end{array}\right.$$

when $k\to \infty$. The proof is completed from (19) and (20).          □

Remark 3.

1. It was shown in (Theorem 1 [48]) that the inexact KM iteration converges at the same rate as (i) of Theorem 2 under the assumption that $(k+1)\parallel e_k\parallel$ is summable. According to Theorem 2, this assumption can be relaxed, for example, if we only assume that $\parallel e_k\parallel$ is summable; selecting ${\alpha }_k=1/(k+1)$ will also lead to a convergence (at a rate of $O(1/lnk)$). 2. Specifically, sublinear rate derivation constants are based on $e_k,x_{-1}$ and $x_{-2}$. We can also refer to Equation (13).

We now establish the linear convergence rate for CISA algorithm (7).

Theorem 3.

The sequence ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ generated by CISA algorithm (7) with ${\alpha }_k\equiv \alpha$ and $e_k=0$, if further satisfying that

$$\mu \parallel x_k-x_{*}{\parallel }^2 \le {\parallel T{x}_k-x_k\parallel }^2,x_{*}\in \mathrm{Fix}T,\mu >0,$$

(22)

converges to a point of FixT in finite steps, or linearly converges in the sense that there is at lest one $i_k\in \{0,1,2,3\}$ such that

$$\begin{array}{c}\hfill \parallel x_{k+1}-x_{*}{\parallel }^2+ \parallel x_{k+2}-x_{*}{\parallel }^2 \le \delta (\parallel x_{k-i_k}-x_{*}{\parallel }^2+ \parallel x_{k-i_k+1}-x_{*}{\parallel }^2),\end{array}$$

(23)

where $0<\delta <1.$

Proof. 

The technical proof is presented in Appendix A.          □

Remark 4.

The assumption (22) is called Metric sub-regularity [50] of $T-I$. It was used in the analysis of linear convergence rate of the KM iteration in [47,48]. In linear programming, metric sub-regularity of the KKT solution mapping at a strictly complementary solution guarantees local linear convergence for various first-order optimization methods, such as the augmented Lagrangian method.

3.2. G-CISA: General Converse Inertial Step Approach

In this subsection, we extend CISA to a generalized version for solving nonexpansive mappings (1), denoted by the G-CISA algorithm:

$$\left(x_{1-m},\cdots ,x_{-1},x_0\in X\right)\left\{\begin{array}{c}{y}_k=\sum _{i=k-m+1}^k{\alpha }_{k,i}{x}_i,\hfill \\ x_{k+1}=T{y}_k+e_k,\hfill \end{array}\right.$$

(24)

where ${\alpha }_{k,i}$ satisfied that ${\sum }_{i=k-m+1}^k{\alpha }_{k,i}=1$ with ${\alpha }_{k,i}\ge 0$, and $e_k$ stands for the error during calculating $T{y}_k$. By rewriting $y_k$ as

$$y_k=x_k-\sum _{i=k-m+1}^{k-1}{\alpha }_{k,i}(x_k-x_i),$$

this is exactly the origin of the name of general converse inertial step approach.

The G-CISA algorithm (24) can be compared with Anderson acceleration [51,52] and Energy Direct Inversion on the Iterative Subspace (EDIIS) algorithm [53,54]. The main difference is that Anderson acceleration and EDIIS are based on the optimal choice of ${\alpha }_{k,i}$ to gain excellent numerical performance under strict assumptions, including that T is contracting and Lipschitz differentiable. As shown in the following, G-CISA in solving nonexpansive mappings stably converges under weak assumptions. Our analysis is based on the newly developed m weak quasi-Fejér momotonicity.

Theorem 4.

Under assumption (8a), the sequence ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ generated by G-CISA algorithm (24) satisfies the following:

(i)

${\left\{x_k\right\}}_{k\in \mathbb{N}}$ is bounded.

(ii)

${\left\{x_k\right\}}_{k\in \mathbb{N}}$ is m weak quasi-Fejér monotone with respect to FixT.

Moreover, if there is an $\overline{i}\in \{0,1,..,m-2\}$ such that

$$\sum _{k=0}^{\infty }{\alpha }_{k,k-\overline{i}}{\alpha }_{k,k-\overline{i}-1}=\infty$$

(25)

and for $0\le i\le m-1$,

$$\sum _{k=i}^{\infty }|{\alpha }_{k,k-i}-{\alpha }_{k-i,k-i-1}|<\infty .$$

(26)

Then, the following holds:

(iii)

Every weak cluster of ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ belongs to FixT.

(iv)

${\left\{x_k\right\}}_{k\in \mathbb{N}}$ converges weakly to a point of FixT.

Proof. 

See Appendix B for the proof.         □

Notice that G-CISA algorithm (24) can be reformulated as

$$y_{k+1}=\sum _{i=k-m+2}^{k+1}{\alpha }_{k+1,i}(T{y}_{i-1}+e_{i-1}),k\in \mathbb{N}.$$

We have the following result similar to Corollary 1.

Corollary 2.

Under assumption (8a), the sequence ${\left(y_k\right)}_{k\in \mathbb{N}}$ generated by the G-CISA algorithm (24) satisfies the following:

(i)

${\left(y_k\right)}_{k\in \mathbb{N}}$ is bounded.

(ii)

${\left(y_k\right)}_{k\in \mathbb{N}}$ is m weak quasi-Fejér monotone.

Moreover, under assumptions (25) and (26), we have the following:

(iii)

every weak cluster of ${\left(y_k\right)}_{k\in \mathbb{N}}$ belongs to FixT.

(iv)

${\left(y_k\right)}_{k\in \mathbb{N}}$ converges weakly to a point of FixT.

Remark 5.

Consider the coefficient matrix of $x_k,$

$$C=\left(\begin{array}{ccccc}{\alpha }_{0,-m+1}& {\alpha }_{1,-m+2}& \dots & {\alpha }_{k,k-m+1}& \dots \\ {\alpha }_{0,-m+2}& {\alpha }_{1,-m+3}& \dots & {\alpha }_{k,k-m+2}& \dots \\ ⋮& ⋮& \ddots & ⋮& \ddots \\ {\alpha }_{0,0}& {\alpha }_{1,1}& \dots & {\alpha }_{k,k}& \dots \end{array}\right).$$

Assumption (25) means that there are at least two adjacent rows such that their inner product is divergent. Assumption (26) means that the absolute difference between two adjacent rows is summable. We list in the following two matrices satisfying assumptions (25) and (26).

$$\begin{array}{ccc}& & C=\left(\begin{array}{ccccc}{a}_1& a_1& \dots & a_1& \dots \\ a_2& a_2& \dots & a_2& \dots \\ ⋮& ⋮& \ddots & ⋮& \ddots \\ a_m& a_m& \dots & a_m& \dots \end{array}\right),\sum _{i=1}^m{a}_i=1,a_i>0,\hfill \\ & & C=\left(\begin{array}{ccccc}{\displaystyle \frac{1}{{(k+1)}^p(m-1)}}& {\displaystyle \frac{1}{{(k+1)}^p(m-1)}}& \dots & {\displaystyle \frac{1}{{(k+1)}^p(m-1)}}& \dots \\ {\displaystyle \frac{1}{{(k+1)}^p(m-1)}}& {\displaystyle \frac{1}{{(k+1)}^p(m-1)}}& \dots & {\displaystyle \frac{1}{{(k+1)}^p(m-1)}}& \dots \\ ⋮& ⋮& \ddots & ⋮& \ddots \\ {\displaystyle \frac{1}{{(k+1)}^p(m-1)}}& {\displaystyle \frac{1}{{(k+1)}^p(m-1)}}& \dots & {\displaystyle \frac{1}{{(k+1)}^p(m-1)}}& \dots \\ 1-{\displaystyle \frac{1}{{(k+1)}^p}}& 1-{\displaystyle \frac{1}{{(k+1)}^p}}& \dots & 1-{\displaystyle \frac{1}{{(k+1)}^p}}& \dots \end{array}\right),0<p\le 1.\hfill \end{array}$$

Moreover, if we select ${\alpha }_{k,i}$ as in Remark 1, assumptions (25) and (26) in Theorem 4 reduce to assumptions (8b) and (8c) in Theorem 1, respectively. It turns out that Theorem 1 corresponds to the special case of Theorem 4 with $m=2$.

4. Relation Between CISA and Over-Relaxed Step Approach (ORSA)

As briefly presented in the Introduction section, ORSA is proposed to achieve a better numerical performance under very restrictive assumptions; see [43,44,45,46] for examples. This section shows that in solving nonexpansive mapping with CISA integration, ORSA could converge under much more relaxed assumptions. Moreover, inexact calculation in each iteration is allowed.

We first present the general inexact version of the CISA-ORSA algorithm to solve nonexpansive mapping (1):

$$(x_{-1},x_0\in X)\left\{\begin{array}{c}{y}_k=x_k-{\alpha }_k(x_k-x_{k-1}),\hfill \\ x_{k+1}=(1-{\lambda }_k)y_k+{\lambda }_{k}T{y}_k+e_k,\hfill \end{array}\right.$$

(27)

where ${\alpha }_k\in [0,1]$, ${\lambda }_k\ge 1$, and $e_k$ denotes the error in calculating $T{y}_k$. Notice that ORSA corresponds to the special case ${\alpha }_k\equiv 0$ and $e_k\equiv 0$. Specially, Boţ et al. [55] combine ISA to solve averaged nonexpansive mappings in exact cases, i.e., ${\alpha }_k,-{\lambda }_k\in [-1,0]$ with the setting $e_k\equiv 0$. The following theorem contains the inexact case with ${\alpha }_k\in [0,1]$ and ${\lambda }_k\ge 1$ only under mild assumptions.

Theorem 5.

Let $T:X\to X$ be bounded. For $k\in \mathbb{N}$, assume that

$$\begin{array}{cc}\hfill & \left\{{\alpha }_k\right\}isnon\text{-}increasingin[0,1],\hfill \end{array}$$

(28a)

$$\begin{array}{cc}\hfill & \left\{{\lambda }_k\right\}isnon\text{-}decreasingand{\lambda }_0\ge 1,\hfill \end{array}$$

(28b)

$$\begin{array}{cc}\hfill & \sum _{k=0}^{\infty }\parallel e_k\parallel <\infty ,\hfill \end{array}$$

(28c)

$$\begin{array}{cc}\hfill & {\displaystyle \frac{(1-{\lambda }_k){(1+{\alpha }_k)}^2}{{\lambda }_k}}+{\alpha }_k(1-{\alpha }_k)\ge \delta >0.\hfill \end{array}$$

(28d)

Then, the sequence${\left\{x_k\right\}}_{k\in \mathbb{N}}$generated by CISA-ORSA algorithm (27) converges weakly to a point in FixT.

Proof. 

Let $x_{*}$ be a fixed point in FixT. It follows from the presentation of CISA-ORSA algorithm (27) and the nonexpansiveness of T that

$$\begin{array}{cc}& \parallel x_{k+1}-x_{*}{\parallel }^2={\parallel (1-{\lambda }_k)(y_k-x_{*})+{\lambda }_k(T{y}_k-x_{*})+e_k\parallel }^2\hfill \\ \hfill & \stackrel{\mathrm{F}1}{\le }(1-{\lambda }_k)\parallel y_k-x_{*}{\parallel }^2+ {\lambda }_k\parallel T{y}_k-x_{*}{\parallel }^2- {\lambda }_k(1-{\lambda }_k){\parallel T{y}_k-y_k\parallel }^2+2\langle e_k,x_{k+1}-x_{*}\rangle \hfill \\ \hfill & \le \parallel y_k-x_{*}{\parallel }^2- {\displaystyle \frac{1-{\lambda }_k}{{\lambda }_k}}{\parallel x_{k+1}-y_k-e_k\parallel }^2+2\langle e_k,x_{k+1}-x_{*}\rangle \hfill \\ \hfill & = \parallel y_k-x_{*}{\parallel }^2- {\displaystyle \frac{1-{\lambda }_k}{{\lambda }_k}}{\parallel x_{k+1}-y_k\parallel }^2+\underset{{ϵ}_k}{\underbrace{2\langle e_k,T{y}_k-x_{*}\rangle +\left({\displaystyle \frac{1-{\lambda }_k}{{\lambda }_k}}+2\right){\parallel e_k\parallel }^2}}\hfill \\ \hfill & \stackrel{\mathrm{F}2}{\le }(1-{\alpha }_k)\parallel x_k-x_{*}{\parallel }^2+ {\alpha }_k\parallel x_{k-1}-x_{*}{\parallel }^2- {\alpha }_k(1-{\alpha }_k){\parallel x_k-x_{k-1}\parallel }^2\hfill \\ \hfill & -{\displaystyle \frac{1-{\lambda }_k}{{\lambda }_k}}\left((1+{\alpha }_k)\parallel x_{k+1}-x_k{\parallel }^2+ {\alpha }_k(1+{\alpha }_k){\parallel x_k-x_{k-1}\parallel }^2\right)+{ϵ}_k,\hfill \end{array}$$

(29)

where F2 is used twice in deriving the last inequality. According to assumptions (28a) and (28b), rearranging the above inequality (29) yields that

$$\begin{array}{cc}& \parallel x_{k+1}-x_{*}{\parallel }^2+ {\alpha }_k\parallel x_k-x_{*}{\parallel }^2+ {\displaystyle \frac{(1-{\lambda }_{k+1})(1+{\alpha }_{k+1})}{{\lambda }_{k+1}}}{\parallel x_{k+1}-x_k\parallel }^2\hfill \\ \hfill & \le \parallel x_k-x_{*}\parallel + {\alpha }_{k-1}\parallel x_{k-1}-x_{*}{\parallel }^2+ {\displaystyle \frac{(1-{\lambda }_k)(1+{\alpha }_k)}{{\lambda }_k}}{\parallel x_k-x_{k-1}\parallel }^2\hfill \\ & -\left({\displaystyle \frac{(1-{\lambda }_k){(1+{\alpha }_k)}^2}{{\lambda }_k}}+{\alpha }_k(1-{\alpha }_k)\right){\parallel x_k-x_{k-1}\parallel }^2+{ϵ}_k,\hfill \end{array}$$

(30)

Adding both sides of the above inequality from $k=0$ to ∞, we obtain

$$\begin{array}{cc}& \sum _{k=0}^{\infty }\left({\displaystyle \frac{(1-{\lambda }_k){(1+{\alpha }_k)}^2}{{\lambda }_k}}+{\alpha }_k(1-{\alpha }_k)\right){\parallel x_k-x_{k-1}\parallel }^2\hfill \\ & \le \parallel x_0-x_{*}\parallel + {\alpha }_{-1}\parallel x_{-1}-x_{*}{\parallel }^2+ {\displaystyle \frac{(1-{\lambda }_0)(1+{\alpha }_0)}{{\lambda }_0}}{\parallel x_0-x_{-1}\parallel }^2+\sum _{k=0}^{\infty }{ϵ}_k<\infty ,\hfill \end{array}$$

where the last inequality holds true as follows by using the boundedness of T and assumption (28c):

$$\begin{array}{cc}\hfill & \sum _{k=0}^{\infty }{ϵ}_k\le \sum _{k=0}^{\infty }\left(2\parallel T{y}_k-x_{*}\parallel +\left({\displaystyle \frac{{\lambda }_k-1}{{\lambda }_k}}+2\right)\parallel e_k\parallel \right)\parallel e_k\parallel <\infty .\hfill \end{array}$$

(31)

Then, it follows from assumption (28d) that

$$\underset{k\to \infty }{lim}{\parallel x_k-x_{k-1}\parallel }^2=0.$$

(32)

Substituting (32) and (28c) into the following inequality

$$\begin{array}{cc}\hfill \parallel T{x}_k-x_k\parallel & = \parallel T{x}_k-T{y}_k+T{y}_k-y_k+y_k-x_k\parallel \hfill \\ & \le 2\parallel x_k-y_k\parallel +{\displaystyle \frac{1}{{\lambda }_k}}\parallel x_{k+1}-y_k\parallel + {\displaystyle \frac{1}{{\lambda }_k}}\parallel e_k\parallel \hfill \\ & \le \left(2{\alpha }_k+{\displaystyle \frac{{\alpha }_k}{{\lambda }_k}}\right)\parallel x_k-x_{k-1}\parallel + {\displaystyle \frac{1}{{\lambda }_k}}\parallel x_{k+1}-x_k\parallel + {\displaystyle \frac{1}{{\lambda }_k}}\parallel e_k\parallel \hfill \end{array}$$

yields that

$$\underset{k\to \infty }{lim}{\parallel T{x}_k-x_k\parallel }^2=0.$$

That is, every weak cluster of ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ belongs to FixT. Proof of weak convergence is similar to that of (Lemma 2.39 [8]) and hence omitted.          □

Remark 6.

According to the proof of Theorem 5, suppose $e_k\equiv 0$, then we can remove the boundedness assumption on T.

Remark 7.

Suppose ${\alpha }_k\equiv \alpha \in (0,1)$ and ${\lambda }_k\equiv \lambda$, then${\left\{x_k\right\}}_{k\in \mathbb{N}}$ generated by the CISA-ORSA algorithm (27) converges weakly to a point in FixT under the following simplified assumption:

$${\displaystyle \frac{{(1+\alpha )}^2}{{(1+\alpha )}^2-\alpha (1-\alpha )}}>\lambda \ge 1.$$

Example 1.

Consider the problem of minimizing $f(x,y)=\sqrt{{(x^2+y^2)}^3}$, which has a unique minimizer $x_{*}=(0,0)$. Define a nonexpansive mapping $T_f=2{\mathrm{prox}}_{\tau f}-I$. Set $\tau ={10}^{-2}$. We compare three convergence-guaranteed algorithms, PPA (with an initial point $x_0=(3,-1)$), CISA algorithm (7) and CISA-ORSA algorithm (24) (with the setting $e_k\equiv 0$, $T=T_f$ and initial points $x_{-1}=x_0=(3,-1)$), respectively. We plot in Figure 2 their convergence and observe that in this example, the CISA-ORSA algorithm (24) runs the fastest.

5. Application

In this section, we combine CISA with Backward–Forward splitting (BFS) algorithm [49] and Peaceman–Rachford splitting (PRS) algorithm [26], respectively.

5.1. CISA-BFS Algorithm

We aim at solving the following separable optimization problem

$$\underset{x\in \mathcal{H}}{min}{\theta }_1\left(x\right)+{\theta }_2\left(x\right),$$

(33)

where ${\theta }_{i=1,2}:\mathcal{H}\to \mathbb{R}$ are proper lower semi-continuous convex functions and ${\theta }_1$ is L-Lipschitz differential. This problem has been solved by [9,56,57] and has application in [58,59]. One way to solve (33) is the BFS algorithm presented as

$$x_{k+1}={\mathrm{prox}}_{\gamma {\theta }_2}\left(x_k\right)-\gamma \nabla {\theta }_1\left({\mathrm{prox}}_{\gamma {\theta }_2}\left(x_k\right)\right).$$

The convergence is guaranteed if $0<\gamma <2/L$ (see more detials in [49]). We now introduce the following CISA-BFS algorithm (Algorithm 2):

Algorithm 2 ICISA-BFS

Input:  Initial points $x_{-1},x_0\in \mathcal{H}$; step-size $\gamma >0$; converse inertial parameter $\alpha \in (0,1)$   1: for  $k=0,1,2,\dots$do   2:     Step 1: Converse Inertial Extrapolation   3:     $y_k=x_k-\alpha (x_k-x_{k-1})$,   4:     Step 2: Forward-Backward Update   5:     $x_{k+1}={prox}_{\gamma {\theta }_2}\left(y_k\right)-\gamma \nabla {\theta }_1\left({prox}_{\gamma {\theta }_2}\left(y_k\right)\right)$,   6: end for Output:  $x_{k+1}$

We will show that the benefit is that $\gamma$ could be larger than $2/L$ in CISA-BFS Algorithm 2.

For convenience, let $T_1=I-\gamma \nabla {\theta }_1$ and $T_2={\mathrm{prox}}_{\gamma {\theta }_2}$. Then, CISA-BFS Algorithm 2 can be rewritten as

$$(x_{-1},x_0\in \mathcal{H}),x_{k+1}=T_1\circ T_2{y}_k.$$

We first study a special property of $T_1\circ T_2$.

Lemma 4.

For any $x,y\in \mathcal{H}$, it holds that

$$\begin{array}{cc}\hfill \parallel T_1\circ T_{2}x-T_1\circ T_2{y\parallel }^2\le {\parallel x-y\parallel }^2-s{\parallel (2I-T_2-T_1\circ T_2)x-(2I-T_2-T_1\circ T_2)y\parallel }^2& \\ \hfill +2s\parallel (I-T_1\circ T_2)x-(I-T_1\circ T_2){y\parallel }^2-(1-2s){\parallel (I-T_2)x-(I-T_2)y\parallel }^2,\end{array}$$

where

$$s=-\left(1-{\displaystyle \frac{\gamma L}{2}}\right){\displaystyle \frac{2}{\gamma L}}.$$

(34)

Proof. 

Since $I-{\displaystyle \frac{2}{L}}\nabla {\theta }_1$ is nonexpansive, for all $x,y\in \mathcal{H}$, we have

$$\begin{array}{cc}\hfill \parallel T_{1}x-T_1{y\parallel }^2& = \parallel \left((1-{\displaystyle \frac{\gamma L}{2}})+{\displaystyle \frac{\gamma L}{2}}(I-{\displaystyle \frac{2}{L}}\nabla {\theta }_1)\right)x-\left((1-{\displaystyle \frac{\gamma L}{2}})+{\displaystyle \frac{\gamma L}{2}}(I-{\displaystyle \frac{2}{L}}\nabla {\theta }_1)\right)y{\parallel }^2\hfill \\ \hfill & =(1-{\displaystyle \frac{\gamma L}{2}}){\parallel x-y\parallel }^2+{\displaystyle \frac{\gamma L}{2}}\parallel (I-{\displaystyle \frac{2}{L}}\nabla {\theta }_1)x-(I-{\displaystyle \frac{2}{L}}\nabla {\theta }_1)y{\parallel }^2\hfill \\ \hfill & -(1-{\displaystyle \frac{\gamma L}{2}}){\displaystyle \frac{\gamma L}{2}}\parallel {\displaystyle \frac{2}{L}}\left(\nabla {\theta }_1\left(x\right)-\nabla {\theta }_1\left(y\right)\right){\parallel }^2\hfill \\ \hfill & \le {\parallel x-y\parallel }^2+s{\parallel (I-T_1)x-(I-T_1)y\parallel }^2.\hfill \end{array}$$

(35)

As $T_2$ is firmly nonexpansive, it holds that

$$\parallel T_{2}x-T_2{y\parallel }^2\le {\parallel x-y\parallel }^2-{\parallel (I-T_2)x-(I-T_2)y\parallel }^2,\forall x,y\in \mathcal{H}.$$

(36)

Then, we have

$$\begin{array}{cc}& \parallel T_1\circ T_{2}x-T_1\circ T_2{y\parallel }^2\stackrel{\left(35\right)}{\le } \parallel T_{2}x-T_2{y\parallel }^2+s{\parallel (T_2-T_1\circ T_2)x-(T_2-T_1\circ T_2)y\parallel }^2\hfill \\ & \stackrel{\left(36\right)}{\le }{\parallel x-y\parallel }^2-\parallel (I-T_2)x-(I-T_2){y\parallel }^2+s{\parallel (T_2-T_1\circ T_2)x-(T_2-T_1\circ T_2)y\parallel }^2\hfill \\ & ={\parallel x-y\parallel }^2-s{\parallel (2I-T_2-T_1\circ T_2)x-(2I-T_2-T_1\circ T_2)y\parallel }^2\hfill \\ & + 2s\parallel (I-T_1\circ T_2)x-(I-T_1\circ T_2){y\parallel }^2-(1-2s){\parallel (I-T_2)x-(I-T_2)y\parallel }^2,\hfill \end{array}$$

where the last equation is obtained by taking $a=(I-T_2)x-(I-T_2)y$ and $b=(I-T_1\circ T_2)x-(I-T_1\circ T_2)y$ into the identical equation $2(a^2+b^2)={(a+b)}^2+{(a-b)}^2$. The proof is complete.         □

Now, we show that CISA-BFS Algorithm 2 converges with a larger step-size.

Theorem 6.

Let $S_{*}$ be the solution set of (33) and assumed to be nonempty. The sequence  ${\left\{x_k\right\}}_{k\in \mathbb{N}}$  generated by CISA-BFS Algorithm 2 converges weakly to a point in $S_{*}$, if the following condition holds:

$${\displaystyle \frac{2}{L}}\le \gamma <{\displaystyle \frac{2}{L}}{\displaystyle \frac{{(1+\alpha )}^2}{{(1+\alpha )}^2-\frac{1}{2}\alpha (1-\alpha )}},\alpha \in (0,1).$$

(37)

Proof. 

Let $x_{*}\in S_{*}$ so that $T{x}_{*}=x_{*}$. Taking $y=x_{*}$ and $x=y_k$ into Lemma 4 yields

$$\begin{array}{cc}\hfill \parallel x_{k+1}-x_{*}{\parallel }^2 \le & \parallel y_k-x_{*}{\parallel }^2+ 2s\parallel y_k-x_{k+1}{\parallel }^2- s{\parallel (2I-T_2-T)y_k-(I-T_2)x_{*}\parallel }^2\hfill \\ \hfill & -(1-2s)\parallel (I-T_2)y_k-(I-T_2)x_{*}{\parallel }^2,\hfill \end{array}$$

(38)

where s is defined in (34). Then, according to (37) and (38), we have

$$\parallel x_{k+1}-x_{*}{\parallel }^2 \le \parallel y_k-x_{*}{\parallel }^2+ 2s{\parallel y_k-x_{k+1}\parallel }^2.$$

The remaining proof for the weak convergence of ${\left\{x_k\right\}}_{k\in \mathbb{N}}$ is similar to that of Theorem 5 with $e_k\equiv 0$, ${\alpha }_k\equiv \alpha$ and $({\lambda }_k-1)/{\lambda }_k=2s$.          □

Remark 8.

Theorem 6 is not a corollary of Theorem 5. Actually, in the case $\gamma >2/L$, we can not rewrite $T_1\circ T_2$ as

$$T_1\circ T_2=(1-\lambda )I+\lambda T^{\prime },$$

where $\lambda \ge 1$ and $T^{\prime }$ is a nonexpansive mapping. That is, CISA-BFS Algorithm 2 with $\gamma >2/L$ does not satisfy the conditions assumed in Theorem 5.

Remark 9.

Suppose ${\theta }_2=0$, then $T_2=I$. CISA-BFS Algorithm 2 reduces to

$$(x_{-1},x_0\in \mathcal{H})\left\{\begin{array}{c}{y}_k=x_k-\alpha (x_k-x_{k-1}),\hfill \\ x_{k+1}=y_k-\gamma \nabla {\theta }_1\left(y_k\right).\hfill \end{array}\right.$$

(39)

It follows from (38) and $T_2=I$ that

$$\parallel x_{k+1}-x_{*}{\parallel }^2 \le \parallel y_k-x_{*}{\parallel }^2+ s{\parallel y_k-x_{k+1}\parallel }^2.$$

Similar to the proof for Theorem 6, condition (37) can be improved to

$${\displaystyle \frac{2}{L}}\le \gamma <{\displaystyle \frac{2}{L}}{\displaystyle \frac{{(1+\alpha )}^2}{{(1+\alpha )}^2-\alpha (1-\alpha )}}.$$

Interestingly, based on an approach similar to (39), Alecsa, László, and Viorel [60] proposed a gradient method to minimize nonconvex and gradient Lipchitz continuous function with an extended step-size from $(0,1/L)$ to $(0,2/L)$. Comparing with the work in [60], we can extend the step-size of gradient method to larger than $2/L$ by CISA in convex case.

5.2. CISA-PRS Algorithm

In this subsection, we consider the following separable convex optimization problem with linear constraint.

$$\underset{x,y\in \mathcal{H}}{min}\{{\theta }_1\left(x\right)+{\theta }_2\left(y\right)|Ax+By=b\}$$

(40)

where ${\theta }_{i=1,2}:\mathcal{H}\to \mathbb{R}$ are proper lower semi-continuous convex functions, A and B are both linear operators. This problem address optimization problems across various domains, including business optimization [61], sensor technology [62], and data science [63,64,65]. The classical Peaceman–Rachford splitting [26] method for solving (40) reads as follows:

$$\left\{\begin{array}{cc}& x_{k+1}\in arg\underset{x\in \mathcal{H}}{min}{\theta }_1\left(x\right)+{\theta }_2\left(y_k\right)-{\lambda }_k^T(Ax+B{y}_k-b)+{\displaystyle \frac{\beta }{2}}{\parallel Ax+B{y}_k-b\parallel }^2,\hfill \\ & {\lambda }_{k+{\displaystyle \frac{1}{2}}}={\lambda }_k-\beta (A{x}_k+B{y}_k-b),\hfill \\ & y_{k+1}\in arg\underset{y\in \mathcal{H}}{min}{\theta }_1\left(x_{k+1}\right)+{\theta }_2\left(y\right)-{\lambda }_{k+{\displaystyle \frac{1}{2}}}^T(A{x}_{k+1}+By-b)+{\displaystyle \frac{\beta }{2}}{\parallel A{x}_{k+1}+By-b\parallel }^2,\hfill \\ & {\lambda }_{k+1}={\lambda }_{k+{\displaystyle \frac{1}{2}}}-\beta (A{x}_{k+1}+B{y}_{k+1}-b).\hfill \end{array}\right.$$

However, there is no convergence guaranteed for PRS in the general situation. To overcome this shortcoming, He et al. propose the following strictly contractive Peaceman–Rachford splitting [66] method for (40) with a relaxation factor $r\in (0,1)$ in dual variable updating:

$$\left\{\begin{array}{cc}& x_{k+1}\in arg\underset{x\in \mathcal{H}}{min}{\theta }_1\left(x\right)+{\theta }_2\left(y_k\right)-{\lambda }_k^T(Ax+B{y}_k-b)+{\displaystyle \frac{\beta }{2}}{\parallel Ax+B{y}_k-b\parallel }^2,\hfill \\ & {\lambda }_{k+{\displaystyle \frac{1}{2}}}={\lambda }_k-r\beta (A{x}_k+B{y}_k-b),\hfill \\ & y_{k+1}\in arg\underset{y\in \mathcal{H}}{min}{\theta }_1\left(x_{k+1}\right)+{\theta }_2\left(y\right)-{\lambda }_{k+{\displaystyle \frac{1}{2}}}^T(A{x}_{k+1}+By-b)+{\displaystyle \frac{\beta }{2}}{\parallel A{x}_{k+1}+By-b\parallel }^2,\hfill \\ & {\lambda }_{k+1}={\lambda }_{k+{\displaystyle \frac{1}{2}}}-r\beta (A{x}_{k+1}+B{y}_{k+1}-b).\hfill \end{array}\right.$$

In this subsection, as an alternative modification to guarantee the convergence, we propose the following CISA–Peaceman–Rachford splitting (CISA-PRS) algorithm (Algorithm 3) for solving (40):

Algorithm 3 CISA-PRS

Input:  $(y_{-1},{\lambda }_{-1}),(y_0,{\lambda }_0)\in \mathcal{H}\times \mathcal{H}$; parameter $\beta >0$; sequence $\left\{{\alpha }_k\right\}\subset [0,1]$   1: for  $k=0,1,2,\dots$  do   2:     Step 1: Converse Inertial Extrapolation   3:     ${({\overline{y}}_k,{\overline{\lambda }}_k)}^T={(y_k,{\lambda }_k)}^T-{\alpha }_k{(y_k-y_{k-1},{\lambda }_k-{\lambda }_{k-1})}^T$,   4:     Step 2: Primal Update (x-subproblem)   5:     $x_{k+1}\in arg\underset{x\in \mathcal{H}}{min}{\theta }_1\left(x\right)+{\theta }_2\left({\overline{y}}_k\right)-{\overline{\lambda }}_k^T(Ax+B{\overline{y}}_k-b)+{\displaystyle \frac{\beta }{2}}{\parallel Ax+B{\overline{y}}_k-b\parallel }^2$,   6:     Step 3: Intermediate Dual Update   7:     ${\lambda }_{k+{\displaystyle \frac{1}{2}}}={\overline{\lambda }}_k-\beta (A{x}_k+B{\overline{y}}_k-b)$,   8:     Step 4: Primal Update (y-subproblem)   9:     $y_{k+1}\in arg\underset{y\in \mathcal{H}}{min}{\theta }_1\left(x_{k+1}\right)+{\theta }_2\left(y\right)-{\lambda }_{k+{\displaystyle \frac{1}{2}}}^T(A{x}_{k+1}+By-b)+{\displaystyle \frac{\beta }{2}}{\parallel A{x}_{k+1}+By-b\parallel }^2$,   10:     Step 5: Dual Update   11:     ${\lambda }_{k+1}={\lambda }_{k+{\displaystyle \frac{1}{2}}}-\beta (A{x}_{k+1}+B{y}_{k+1}-b)$,   12: end for Output:  Sequence $\left\{(x_k,y_k,{\lambda }_k)\right\}$ converges to a solution of (40)

Define $z={(y,\lambda )}^T$ and ${\parallel z\parallel }_M=\sqrt{\langle z,Mz\rangle }$ with M being symmetric positive semidefinite. Then CISA-PRS Algorithm 3 can be rewritten in short as

$$\left\{\begin{array}{c}{\overline{z}}_k=z_k-{\alpha }_k(z_k-z_{k-1}),\hfill \\ z_{k+1}=T{\overline{z}}_k.\hfill \end{array}\right.$$

As shown in [66], the above operator T satisfies that, for $k\in \mathbb{N},$

$$\parallel T{\overline{z}}_k-z_{*}{\parallel }_M^2 \le \parallel {\overline{z}}_k-z_{*}{\parallel }_M^2,\parallel T{\overline{z}}_{k+1}-T{\overline{z}}_k{\parallel }_M^2 \le {\parallel {\overline{z}}_{k+1}-{\overline{z}}_k\parallel }_M^2.$$

(41)

where $M={\displaystyle \frac{1}{2}}\left(\begin{array}{cc}\beta B^{T}B& -B^T\\ -B& {\displaystyle \frac{1}{\beta }}I\end{array}\right)⪰0$. Moreover, according to (Lemma 3.2 [66]), if $\parallel T{\overline{z}}_k-{\overline{z}}_k{\parallel }_M^2 =0,$ then $\left(x_{k+1},y_{k+1},{\overline{\lambda }}_k+\beta (A{x}_{k+1}-B{\overline{y}}_k-b)\right)$ is a solution of (40). Therefore, we are sufficient to give the convergence rate of $\parallel T{\overline{z}}_k-{\overline{z}}_k{\parallel }_M^2$.

Theorem 7.

Suppose the solution of (40) is nonempty. Let${\left({\overline{z}}_k\right)}_{k\in \mathbb{N}}$  be generated from CISA-PRS Algorithm 3.

(i)

If ${\alpha }_k\equiv \alpha \in (0,1),$ then $\parallel T{\overline{z}}_k-{\overline{z}}_k{\parallel }_M^2=O(1/k)$.

(ii)

If ${\alpha }_k=1/{(k+1)}^p$ for $p\in (0,1]$, then

$$\parallel T{\overline{z}}_k-{\overline{z}}_k{\parallel }_M^2=\left\{\begin{array}{cc}O\left({\displaystyle \frac{1}{lnk}}\right),\hfill & p=1,\hfill \\ O\left({\displaystyle \frac{1}{k^{1-p}}}\right),\hfill & p\in [{\displaystyle \frac{1}{2}},1),\hfill \\ O\left({\displaystyle \frac{1}{k^p}}\right),\hfill & p\in (0,{\displaystyle \frac{1}{2}}).\hfill \end{array}\right.$$

Proof. 

The proof is similar to that of Theorem 2, with an extended norm ${\parallel ·\parallel }_M$ satisfying (41). □

6. Conclusions

We introduce and investigate the converse inertial step approach (CISA) in solving nonexpansive mappings. The weak convergence of the CISA algorithm is first established based on a newly developed weak quasi-Fejér monotonicity under mild assumptions. We also establish sublinear and linear convergence under different assumptions. As a further extension, we then study the generalized version of CISA. The second contribution of this paper is to design a more general inexact CISA-ORSA algorithm to achieve a better numerical performance without making restrictive assumptions. Finally, with CISA ing, BFS algorithm accepts a larger step-size and PRS algorithm is guaranteed to converge. More applications of CISA are expected in the area of nonconvex optimization.

Based on the present study, several promising directions for future research emerge. First, extending the CISA framework to nonconvex optimization problems is of significant interest, as many modern applications in machine learning and signal processing involve nonconvex objectives. Second, investigating the applicability of CISA in real time and online settings, such as adaptive control systems and online learning, where its convergence properties could be highly beneficial, represents a critical next step. Additionally, exploring stochastic variants of CISA and its integration with large-scale distributed computing frameworks are important avenues for further development.