The Extended Second APG Method for Constrained DC Problems

Abstract

In this paper, we develop the extended proximal gradient algorithm with Nesterov’s second acceleration (${\mathrm{EAPG}}_s$) for constrained difference-of-convex (DC) optimization problems. ${\mathrm{EAPG}}_s$ has two key links to existing methods: it extends ${\mathrm{APG}}_s$ (for unconstrained DC problems) by adopting the constraint handling idea from Auslender’s ESQM, and serves as a variant of ${\mathrm{ESQM}}_e$ with extrapolation replaced by Nesterov’s second acceleration. Under basic assumptions, we establish the subsequential convergence of ${\mathrm{EAPG}}_s$. By introducing a restart technique and leveraging the Kurdyka–Łojasiewicz (KL) property of a suitable potential function, we further prove its global convergence, analyze its convergence rate, and do so under weaker conditions than those for ${\mathrm{APG}}_s$. Additionally, we propose ${\mathrm{EAPG}}_{sr}$ by adding practical restart criteria to ${\mathrm{EAPG}}_s$. Numerical experiments verify the criteria’s efficiency and show that ${\mathrm{EAPG}}_{sr}$ performs well against state-of-the-art methods for constrained and unconstrained DC problems.

1. Introduction

In this paper, we discuss the following difference-of-convex (DC) optimization problem with smooth inequality constraints and simple geometric constraints:

$$\begin{array}{cccc}\hfill & \underset{x\in {\mathbb{R}}^n}{min}\hfill & \hfill & F\left(x\right):=f\left(x\right)+P_1\left(x\right)-P_2\left(x\right)\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\hfill & \hfill & g_i\left(x\right)\le 0,i=1,\dots ,m,\hfill \\ \hfill & \hfill & & x\in C,\hfill \end{array}$$

(1)

where all functions are real-valued and defined on ${\mathbb{R}}^n$. Specifically, f and $g_i$ ($i=1,\dots ,m$) are differentiable; $P_1$ and $P_2$ are convex, with $P_1$ admitting an efficiently computable proximal operator; $C\subset {\mathbb{R}}^n$ is a nonempty closed convex set; and the feasible set $C\cap \mathcal{F}$ is nonempty (here, $\mathcal{F}:=\{x\in {\mathbb{R}}^n:g_i\left(x\right)\le 0,i=1,\dots ,m\}$).

Problem (1) belongs to one of the three types of DC problems summarized by [1], which is widely applied in various fields such as joint chance constrained programs [2], multicast network design problems [3], sparsity constrained optimization problems [4], and bilevel optimization problems [5]. As a result, it has garnered significant attention from researchers (see [1,6,7] and the references therein). Problem (1) encompasses two commonly used and extensively studied special cases:

• Unconstrained DC problems, discussed in [1,8,9,10,11,12,13,14,15] and related references:

  $$\begin{array}{cccc}\hfill & \underset{x\in {\mathbb{R}}^n}{min}\hfill & \hfill & F\left(x\right):=f\left(x\right)+P_1\left(x\right)-P_2\left(x\right).\hfill \end{array}$$

  (2)
  This corresponds to setting $g_i\equiv 0$ (for all i), $C={\mathbb{R}}^n$, while retaining the original assumptions on f, $P_1$, and $P_2$.
• DC problems with $f\equiv 0$, analyzed in [16,17,18,19] and references therein:

  $$\begin{array}{cccc}\hfill & \underset{x\in {\mathbb{R}}^n}{min}\hfill & \hfill & P_1\left(x\right)-P_2\left(x\right)\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\hfill & \hfill & g_i\left(x\right)\le 0,i=1,\dots ,m,\hfill \\ \hfill & \hfill & & x\in C.\hfill \end{array}$$

  (3)

Both Problems (2) and (3) arise naturally in numerous applications. For instance, Yin et al. [17] show that compressed sensing problems can be formulated as Problem (3): here, $P_1-P_2$ acts as a sparsity-inducing regularizer (e.g., the difference of ${\ell }_1$- and ${\ell }_2$-norms), and the (unique) constraint involves estimation error. Alternatively, the same compressed sensing problem can be cast as Problem (2), where $P_1-P_2$ remains unchanged and f serves as a penalty term for the original constraint.

Notably, while Problems (2) and (3) often model the same applications, they differ in key trade-offs: Problem (3) enforces (approximate) satisfaction of constraints, but developing constraint-aware algorithms is more challenging; Problem (2) benefits from a richer set of unconstrained solvers, yet its solutions may violate the original constraints—leading to larger deviations from the true optimal solution. Given these trade-offs, extending successful methods for Problem (2) to Problems (1) and (3) is of significant value.

DC programming and DC algorithms were first introduced by Pham Dinh [15] (for a comprehensive survey of results before 2015, see [1] and references therein). The core idea of DC algorithms (DCA) is to approximate the concave component of the DC function with an affine function in each iteration, yielding a convex subproblem that can be solved via existing convex optimization techniques. Subsequent challenges in DCA research include improving algorithm efficiency, conducting convergence analyses, and deriving convergence rate guarantees.

Among DC algorithms, line search methods are fundamental (see [20] for a categorized overview). These methods typically do not require strict conditions on f, $g_1,\dots ,g_m$ (e.g., differentiability) or $P_1$ (e.g., efficient proximal computation), making them applicable to a broad range of DC problems (including those with fewer restrictions than Problem (1)). However, this generality comes at a cost: line search methods cannot leverage favorable properties of functions that arise in practical applications, limiting their potential efficiency.

For Problem (2) in the case where f has a Lipschitz-continuous gradient, integrating proximal mappings and Nesterov’s acceleration techniques yields notable performance improvements. Proximal gradient methods and acceleration strategies were originally developed for convex optimization (see [21] for a detailed review). Examples include FISTA [22] (a proximal gradient method with Nesterov’s first acceleration) and IGA [23] (a proximal gradient method with Nesterov’s second acceleration). Subsequent extensions to DC problems—such as pDCAe [24] (extending FISTA) and ${\mathrm{APG}}_s$ [25,26] (extending IGA)—have demonstrated the effectiveness of these convex optimization ideas in the DC setting. The main iteration of IGA generates

$$\begin{array}{cc}\hfill y^k=& {\theta }_k{z}^k+(1-{\theta }_k)x^k,\hfill \\ \hfill z^{k+1}=& \underset{z\in {\mathbb{R}}^n}{argmin}\{P_1\left(z\right)+\langle \nabla f\left(y^k\right),z\rangle +{\displaystyle \frac{1}{2}}{\theta }_k{L}_f\parallel z-z^k{\parallel }^2\},\hfill \\ \hfill x^{k+1}=& {\theta }_k{z}^{k+1}+(1-{\theta }_k)x^k\hfill \end{array}$$

for the convex version of Problem (2) (i.e., when f is convex and $P_2=0$), where $\left\{{\theta }_k\right\}$ is the sequence of acceleartion parameters, and $L_f$ denotes the Lipschitz constant of $\nabla f$. For the general case of Problem (2), ${\mathrm{APG}}_s$ reformulate the above subproblem by subtracting ${\zeta }^k(\in \partial P_2\left(x^k\right))$ from $\nabla f\left(y^k\right)$.

For Problem (1), Lu proposed a sequential convex programming (SCP) method in [27], where each iteration is obtained by solving a constrained convex programming problem. It was shown that any accumulation point of the sequence generated by SCP is a stationary point under Slater’s condition. However, the convergence and convergence rate of the entire generated sequence remain unknown. Recently, Yu et al. further studied the SCP method with monotone line search (denoted as ${\mathrm{SCP}}_{ls}$) in [28], successfully establishing global convergence guarantees for the proposed algorithm and quantitatively estimating its convergence rate.

Furthermore, for nonlinear programming problems where C is the entire space, Sequential Quadratic Programming (SQP) (see [16,29]) is one of the most successful methods. The SQP algorithm solves a subproblem involving linear inequalities at each iteration. Later, attention turned to modifying SQP algorithms by constructing Sequential Quadratically Constrained Quadratic Programming (SQCQP) methods (see [30,31]), where each iteration solves a subproblem with convex quadratic inequalities. However, Solodov pointed out in [32] that a major drawback of SQP is that global convergence statements rely on the boundedness of the sequence of primal problems constructed by the algorithm—an assumption that is not easily justified. To address this, he introduced a safeguard into the line search procedure, developing an SQP method where the primal sequence is proven bounded when the feasible set is bounded and $g_i$ is convex. This drawback also persists in SQCQP methods.

In [18], Auslender proposed the Extended Sequential Quadratic Method (ESQM) to overcome this critical limitation of existing SQP and SQCQP methods. ESQM achieves global convergence without such boundedness assumptions, enhancing its versatility for a wider range of optimization problems. To improve the convergence rate, Zhang et al. developed a variant of ESQM (denoted as ${\mathrm{ESQM}}_e$ [19]) that incorporates Nesterov’s extrapolation technique, achieving empirical acceleration for Problem (3).

Building on the proven effectiveness of ESQM_e for constrained DC optimization and the well-established advantages of APGs for solving Problem (2), this paper extends APGs to solve (1) by adopting the constraint handling strategy from Auslender’s ESQM. The resulting algorithm is termed the extended proximal gradient algorithm with Nesterov’s second acceleration technique (${\mathrm{EAPG}}_s$). This algorithm also serves as a variant of ${\mathrm{ESQM}}_e$, where the extrapolation step is replaced with Nesterov’s second acceleration.

For ${\mathrm{EAPG}}_s$, we can prove its subsequential convergence properties under basic assumptions. However, analyzing its global convergence and convergence rate faces a key obstacle: the lack of information about the subdifferential properties of F along the sequence $\left\{x^k\right\}$ generated by ${\mathrm{EAPG}}_s$. For APGs applied to Problem (2), this obstacle is circumvented by assuming $P_1$ is Lipschitz differentiable—though this condition is rarely satisfied in practical applications.

For Problem (1), the presence of constraints introduces non-differentiable components in the subproblem, making this obstacle insurmountable and posing a significant challenge. To overcome this, inspired by the restart technique introduced by O’Donoghue and Candès [33], we integrate this technique into ${\mathrm{EAPG}}_s$ and find it effective for both theoretical analysis and practical computation. Theoretically, we construct a suitable potential function and assume it satisfies the Kurdyka–Łojasiewicz (KL) property, along with additional differentiability conditions for each $g_i$ in (1), thereby establishing the convergence of the entire sequence and its convergence rate. Practically, we introduce efficient restart criteria to develop a practical variant (${\mathrm{EAPG}}_{sr}$), which is validated through numerical experiments.

The remainder of this paper is structured as follows: Section 2 presents preliminary concepts and mathematical foundations essential to our analysis. Section 3 formally introduces the ${\mathrm{EAPG}}_s$ algorithm. Section 4 establishes its subsequential convergence properties. Section 5 introduces the theoretical variant of ${\mathrm{EAPG}}_s$ with a restart technique, proves its global convergence, estimates its convergence rate, and presents the practical variant ${\mathrm{EAPG}}_{sr}$. Finally, Section 6 demonstrates the practical performance of ${\mathrm{EAPG}}_{sr}$ through comprehensive numerical experiments.

2. Notation and Preliminaries

In this paper, we use the following standard notation:

• $\mathbb{R}$ and ${\mathbb{R}}_{+}$ denote the sets of real numbers and nonnegative real numbers, respectively.
• ${\mathbb{R}}^n$ and ${\mathbb{R}}_{+}^n$ denote the n-dimensional Euclidean space and its nonnegative orthant, respectively.
• $\mathbb{N}$ denotes the set of positive integers.
• For $x\in \mathbb{R}$, $x_{+}:=max\{x,0\}$.
• For $p\ge 1$, ${\parallel ·\parallel }_p$ denotes the $l_p$-norm on ${\mathbb{R}}^n$; in particular, $\parallel ·\parallel$ is used exclusively to represent the $l_2$-norm (${\parallel ·\parallel }_2$).
• For $x,y\in {\mathbb{R}}^n$, $\langle x,y\rangle$ denotes their inner product.
• Given a nonempty set $D\subseteq {\mathbb{R}}^n$, the distance from $x\in {\mathbb{R}}^n$ to D is defined as $dist(x,D):=inf\{\parallel x-z\parallel \mid z\in D\}$.

For extended real-valued functions $f:{\mathbb{R}}^n\to (-\infty ,+\infty ]$, we adopt the following definitions:

1.

A function f is proper if its domain $domf:=\{x\mid f(x)<+\infty \}$ is nonempty.

2.

A proper function f is closed if it is lower semicontinuous at every $x\in {\mathbb{R}}^n$, i.e., $f\left(x\right)\le {lim\; inf}_{z\to x}f\left(z\right)$.

3.

A proper closed function f is level bounded if all its lower level sets $\{x\in {\mathbb{R}}^n\mid f\left(x\right)\le a\}$ are bounded for every $a\in \mathbb{R}$.

4.

For a sequence $\left\{x^k\right\}\subseteq {\mathbb{R}}^n$, $x^k\stackrel{f}{\to }x$ (as $k\to \infty$) means $x^k\to x$ (in ${\mathbb{R}}^n$) and $f\left(x^k\right)\to f\left(x\right)$.

Definition 1

([34] Definition 8.3). For a proper closed function f, the regular subdifferential of $f:{\mathbb{R}}^n\to \mathbb{R}\cup \{+\infty \}$ at $x\in domf$ is defined by

$$\begin{array}{c}\hfill \widehat{\partial }f\left(x\right):=\left\{\widehat{x}\in {\mathbb{R}}_n:\underset{z\to x,z\ne x}{lim\; inf}{\displaystyle \frac{f\left(z\right)-f\left(x\right)-\langle \widehat{x},z-x\rangle }{\parallel z-x\parallel }}\ge 0\right\}.\end{array}$$

(4)

The (general) subdifferential of f at $x\in domf$ is defined by

$$\begin{array}{c}\hfill \partial f\left(x\right):=\left\{\widehat{x}:\exists x^k\stackrel{f}{\to }x,{\widehat{x}}^k\to \widehat{x}\mathit{with}{\widehat{x}}^k\in \widehat{\partial }f\left(x^k\right)\mathit{for}\mathit{each}k\right\},\end{array}$$

(5)

we write $\mathit{dom}\partial f:=\{x:\partial f(x)\ne \varnothing \}$.

Note that if f is convex, then the general subdifferential and regular subdifferential of f at $x\in \mathrm{dom}f$ reduce to the (classical) subdifferential ([34] Proposition 8.12), which is given by

$$\partial f\left(x\right)=\{\widehat{x}:f\left(y\right)\ge f\left(x\right)+\langle \widehat{x},y-x\rangle \forall y\in {\mathbb{R}}^n\}.$$

(6)

For a nonempty closed set $D\subseteq {\mathbb{R}}^n$, the indicator function ${\delta }_D$ is defined by

$${\delta }_D\left(x\right)=\left\{\begin{array}{cc}0\hfill & x\in D,\hfill \\ \infty \hfill & x\notin D.\hfill \end{array}\right.$$

(7)

The normal cone of D at $x\in D$ is defined by ${\mathcal{N}}_D\left(x\right):=\partial {\delta }_D\left(x\right)$.

Next, we recall several key definitions that will be used in the subsequent analysis. First, we introduce the constraint qualification for problem (1)—which was also adopted in [18,19]—followed by the (associated) first-order optimality conditions for (1).

Definition 2

(RCQ). We say that the Robinson constraint qualification holds at an $x\in {\mathbb{R}}^n$ for (1) if the following statement holds:

$$\begin{array}{c}\hfill RCQ\left(x\right):\exists y\in C\mathit{such}\mathit{that}{g}_i\left(x\right)+\langle \nabla g_i\left(x\right),y-x\rangle <0\forall i=1,\dots m.\end{array}$$

(8)

Definition 3

(Critical point). For (1), we say that x is a critical point of (1) if $x\in C$ and there exists $\lambda =({\lambda }_1,{\lambda }_2,\dots ,{\lambda }_m)\in {\mathbb{R}}_{+}^m$ such that $(x,\lambda )$ satisfies the following conditions:

(i) 

$g_i\left(x\right)\le 0\forall i=1,\dots ,m$,

(ii) 

${\lambda }_i{g}_i\left(x\right)=0\forall i=1,\dots ,m$,

(iii) 

$0\in \nabla f\left(x\right)+\partial P_1\left(x\right)-\partial P_2\left(x\right)+{\sum }_{i=1}^m{\lambda }_i\nabla g_i\left(x\right)+{\mathcal{N}}_C\left(x\right)$.

Using arguments analogous to those in ([28] Section 2), one can show the following: if the RCQ(x) holds for all $x\in C\cap \mathcal{F}$, then every local minimizer of (1) is a critical point of (1)—provided that Assumption 1 (presented in the next section) holds and $P_1$ is continuous at x.

It is further straightforward to verify that if $g_1,\dots ,g_m$ are convex and the Slater condition is satisfied (i.e., there exists $\tilde{x}\in C$ such that $g_i\left(\tilde{x}\right)<0$ for all $i=1,2,\dots ,m$), then RCQ(x) holds for all $x\in C$.

Numerous functions are known to satisfy the Kurdyka–Łojasiewicz (KL) property. For example, proper closed semi-algebraic functions satisfy the KL property with some exponent $\beta \in [0,1)$ (see [35]). The KL property plays a crucial role in the convergence analysis of many first-order methods, and its exponent is particularly significant for establishing convergence rates (for further details, refer to [25,26,36,37,38,39,40,41,42] and the references therein).

First, for $\eta >0$, we define ${\mathrm{\Theta }}_{\eta }$ as the class of all continuous concave functions $\phi :[0,\eta )\to [0,+\infty )$ satisfying $\phi \left(0\right)=0$, where $\phi$ is continuously differentiable on $(0,\eta )$ with ${\phi }^{\prime }>0$ (see ([43] Section 2)).

Definition 4

((KL property and KL function) ([43] Section 2)). Let $h:{\mathbb{R}}^n\to \mathbb{R}\cup \{+\infty \}$ be a proper closed function.

(i) 

For $\tilde{x}\in \mathit{dom}\partial h:=\{x\in {\mathbb{R}}^n:\partial h\left(x\right)\ne \varnothing \}$, if there exist a neighborhood $\mathcal{O}$ of $\tilde{x}$, $\eta \in (0,+\infty ]$ and function $\phi \in {\mathrm{\Theta }}_{\eta }$ such that for all $x\in \mathcal{O}\cap \{x\in {\mathbb{R}}^n:h\left(\tilde{x}\right)<h\left(x\right)<h\left(\tilde{x}\right)+\eta \}$, it holds that

$${\phi }^{\prime }(h\left(x\right)-h\left(\tilde{x}\right))dist(0,\partial h\left(x\right))\ge 1,$$

(9)

then h is said to have the Kurdyka-Lojasiewicz (KL) property at $\tilde{x}$.

(ii) 

If h satisfies the KL property at each point of $\mathit{dom}\partial h$, then h is called a KL function.

Definition 5

((KL exponent) ([43] Section 2)). Suppose that $h:{\mathbb{R}}^n\to \mathbb{R}\cup \{+\infty \}$ is a proper closed function satisfying the KL property at $\tilde{x}\in \mathit{dom}\partial h$ with $\phi \left(s\right)=\rho s^{1-\beta }$ for some $\rho >0$ and $\beta \in [0,1)$. Then, h is said to have the KL property at $\tilde{x}$ with an exponent β. If h is a KL function and has the same exponent β at any $\tilde{x}\in \mathit{dom}\partial h$, then h is said to be a KL function with the exponent β.

Lemma 1

((Uniformized KL property) ([39] Lemma 6)). Suppose that $h:{\mathbb{R}}^n\to \mathbb{R}\cup \{+\infty \}$ is a proper closed function and Γ is a compact set. If $h\equiv \zeta$ on Γ for some constant ζ and satisfies the KL property at each point of Γ , then there exist $\epsilon >0$, $\eta >0$ and $\phi \in {\mathrm{\Theta }}_{\eta }$ such that

$${\phi }^{\prime }(h\left(x\right)-\zeta )dist(0,\partial h\left(x\right))\ge 1$$

(10)

for all $x\in \{x\in {\mathbb{R}}^n:dist(x,\mathrm{\Gamma })<\epsilon \}\cap \{x\in {\mathbb{R}}^n:\zeta <h\left(x\right)<\zeta +\eta \}$.

3. Algorithmic Framework

From this section to Section 5, we always suppose that the following conditions are fulfilled.

Assumption 1.

(i) 

f is a differentiable (possibly nonconvex) function and its gradient $\nabla f$ is Lipschitz continuous with Lipschitz constant $L_f\ge 0$, and $l_f\in [0,L_f]$ is such that $f(·)+{\displaystyle \frac{1}{2}}{l}_f{\parallel ·\parallel }^2$ is convex.

(ii) 

All $g_i(i=1,2,\dots ,m)$ are differentiable functions with Lipschitz continuous gradients. We use $L_g$ to denote the common Lipschitz continuity modulus of $\nabla g_1,\dots ,\nabla g_m$, and let $l_g\in [0,L_g]$ be such that $g_i(·)+{\displaystyle \frac{1}{2}}{l}_g\parallel ·\parallel$ be convex for all $i=1,2,\dots ,m$.

(iii) 

At least one of $L_f$ and $L_g$ is positive.

(iv) 

The function $P_1$ is proper, convex and lower semicontinuous; the function $P_2$ is continuous and convex.

(v) 

Either (a) C is compact, or (b) all $g_i=0$ and F is level bounded.

The above assumptions all constitute basic conditions for DC problems. The Lipschitz continuity of gradients is a necessary requirement for nearly all proximal gradient-type algorithms (see, e.g., [19,21,23,25,26,42]). Since we can always select a larger Lipschitz modulus, condition (iii) is easily satisfied. Finally, level boundedness frequently arises in the context of problem (2) (see, e.g., [25,26,36,42]), whereas the compactness of C is required in [19]. Either condition guarantees the existence of an optimal solution to problem (2).

The algorithm we study in this paper is presented as Algorithm 1 below; here and throughout, for notational simplicity, for each $u,w\in {\mathbb{R}}^n$, we define

$$\begin{array}{c}\hfill {\mathrm{lin}}_{g_i}(u,w):=g_i\left(w\right)+\langle \nabla g_i\left(w\right),u-w\rangle \forall i=1,\dots ,m,\end{array}$$

(11)

$$g_0:=0(\mathrm{which}\mathrm{implies}\mathrm{that}{\mathrm{lin}}_{g_0}(u,w)\equiv 0),$$

(12)

$$\mathrm{\Psi }(u,w):=\underset{i=1,\dots ,m}{max}{\left[{\mathrm{lin}}_{g_i}(u,w)\right]}_{+}\left(=\underset{i=0,1,\dots ,m}{max}\left\{{\mathrm{lin}}_{g_i}(u,w)\right\}\right).$$

(13)

Algorithm 1${\mathrm{EAPG}}_s$ for solving Problem (1)

Initialization: Choose $\left\{{\theta }_k\right\}\subseteq (0,1]$, $d>0$, ${\alpha }_0>0$, $x^0,z^0\in C$. For$k=0,1,\dots$, take ${\xi }^k\in \partial P_2\left(x^k\right)$, and compute $$\begin{array}{cc}\hfill y^k=& {\theta }_k{z}^k+(1-{\theta }_k)x^k,\hfill \\ \hfill z^{k+1}=& \underset{z\in C}{argmin}E\left(z\right):=P_1\left(z\right)+\langle \nabla f\left(y^k\right)-{\zeta }^k,z\rangle \hfill \end{array}$$ (14) $$\begin{array}{cc}\hfill & +{\alpha }_k\mathrm{\Psi }(z,y^k)+{\displaystyle \frac{1}{2}}{\theta }_k({\alpha }_k{L}_g+L_f){\parallel z-z^k\parallel }^2,\hfill \end{array}$$ (15) $$\begin{array}{cc}\hfill x^{k+1}=& {\theta }_k{z}^{k+1}+(1-{\theta }_k)x^k,\hfill \end{array}$$ (16) $$\begin{array}{cc}\hfill {\alpha }_{k+1}=& \left\{\begin{array}{cc}{\alpha }_k,\hfill & \mathrm{if}\mathrm{\Psi }(z^{k+1},y^k)\le 0,\hfill \\ {\alpha }_k+d,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$ (17) End for

We refer to our algorithm as the extended proximal gradient algorithm with Nesterov’s second acceleration technique (${\mathrm{EAPG}}_s$), where “Nesterov’s second acceleration technique” corresponds to Equations (14) and (16). To guarantee the convergence properties of Algorithm 1 in Section 4, we impose the following assumption on the acceleration parameters $\left\{{\theta }_k\right\}$:

Assumption 2.

With ${inf}_k{\theta }_k>0$, there exists a constant $\delta \in (0,1)$ satisfying

$$(L_g+l_g){\gamma }_k^2\le L_g(1-\delta ),$$

(18)

$$(L_f+l_f){\gamma }_k^2\le L_f(1-\delta ),$$

(19)

where ${\gamma }_k:={\displaystyle \frac{{\theta }_k(1-{\theta }_{k-1})}{{\theta }_{k-1}}}$, for $k\ge 1$.

Remark 1.

We can provide concrete examples for selecting the acceleration parameters $\left\{{\theta }_k\right\}$ and the constant δ. Define

$${\tau }_f:=\left\{\begin{array}{cc}{\displaystyle \frac{L_f+l_f}{L_f}},\hfill & if{L}_f>0,\hfill \\ 1,\hfill & if{L}_f=0,\hfill \end{array}\right.{\tau }_g:=\left\{\begin{array}{cc}{\displaystyle \frac{L_g+l_g}{L_g}},\hfill & if{L}_g>0,\hfill \\ 1,\hfill & if{L}_g=0,\hfill \end{array}\right.$$

and let $\tau :=max\{{\tau }_f,{\tau }_g\}$.

(i) 

Constant Acceleration Parameters: Set ${\theta }_k\equiv \theta$ for some constant $\theta \in (0,1]$ satisfying ${(1-\theta )}^2<{\tau }^{-1}$. Choose the constant $\delta =1-\tau {(1-\theta )}^2$.

(ii) 

Variable Acceleration Parameters: For a preselected positive integer K, let ${\theta }_k={\vartheta }_k$ for all $k<K$, and ${\theta }_k={\vartheta }_K$ for all $k\ge K$. Here, $\left\{{\vartheta }_k\right\}$ represents the classical parameter sequence introduced by Nesterov (see [21,44]), where

$${\vartheta }_0=1,{\vartheta }_{k+1}={\displaystyle \frac{\sqrt{{\vartheta }_k^4+4{\vartheta }_k^2}-{\vartheta }_k^2}{2}}.$$

The integer K is chosen such that ${(1-{\vartheta }_K)}^2<{\tau }^{-1}$. Let $\delta :=1-\tau {(1-{\vartheta }_K)}^2$. Noticing that the sequence $\left\{{\vartheta }_k\right\}$ is decreasing (as shown in [21]), it follows that $\left\{{\theta }_k\right\}$ is also decreasing. Additionally, we have:

$$1-\tau {\gamma }_k^2\ge 1-\tau {(1-{\theta }_{k-1})}^2\ge \delta ,$$

which establishes the desired inequalities (18) and (19).

The subproblem in (15) admits a unique solution, as it involves a strongly convex objective function over a nonempty closed convex feasible set, while an iterative solver is generally required for this subproblem, we refer readers to ([45] Appendix A) for an efficient routine to solve the subproblem in (15)—specifically for cases where $m=1$ and $P_1$ takes certain forms.

The following lemmas are useful for proving results in subsequent sections. First, the conclusions of Lemma 2 are identical to their counterparts in ([19] Lemma 3.1), with only minor differences in parameter specifications.

Lemma 2.

Suppose that the sequece $\left\{z^k\right\}\subseteq C$ is generated in Algorithm 1. Then the following statements hold:

(i) 

Problem (15) has a unique solution.

(ii) 

$z^{k+1}$ is the minimizer of the subproblem in (15) if and only if there exist ${\lambda }_i^k\ge 0$ for all $i\in I_k\left(z^{k+1}\right)$ such that ${\sum }_{i\in I_k\left(z^{k+1}\right)}{\lambda }_i^k=1$ and

$$\begin{array}{cc}\hfill 0\in & \partial P_1\left(z^{k+1}\right)+\nabla f\left(y^k\right)-{\xi }^k+{\alpha }_k\sum _{i\in I_k\left(z^{k+1}\right)}{\lambda }_i^k\nabla g_i\left(y^k\right)\hfill \\ \hfill & +{\theta }_k({\alpha }_k{L}_g+L_f)(z^{k+1}-z^k)+{\mathcal{N}}_C\left(z^{k+1}\right),\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill I_k\left(z\right):=& \left\{\iota \in \{0,1,\dots ,m\}:{\mathrm{lin}}_{g_{\iota }}(z,y^k)=\mathrm{\Psi }(z,y^k)\right\}.\hfill \end{array}$$

(20)

The next lemma is based on Equation (4.2) in [19].

Lemma 3.

For any $x,y,y^{\prime }\in {\mathbb{R}}^n$,

$$\mathrm{\Psi }(x,y)-\mathrm{\Psi }(x,y^{\prime })\le {\displaystyle \frac{L_g}{2}}\parallel x-y^{\prime }{\parallel }^2+{\displaystyle \frac{l_g}{2}}{\parallel x-y\parallel }^2.$$

The inequality specified in Lemma 4 has appeared in the proofs of several previous works (e.g., (26) in [26]); here, we provide a concise proof.

Lemma 4.

For any $x,x^{\prime },y\in {\mathbb{R}}^n$,

$$\begin{array}{cc}\hfill & f\left(x\right)-f\left(x^{\prime }\right)\le \langle \nabla f\left(y\right),x-x^{\prime }\rangle +{\displaystyle \frac{1}{2}}{L}_f{\parallel x-y\parallel }^2+{\displaystyle \frac{1}{2}}{l}_f{\parallel x^{\prime }-y\parallel }^2.\hfill \end{array}$$

(21)

Proof. 

By the Lipschitz continuity of $\nabla f$ and ([44] Lemma 1.2.3), we have

$$f\left(x\right)\le f\left(y\right)+\langle \nabla f\left(y\right),x-y\rangle +{\displaystyle \frac{1}{2}}{L}_f{\parallel x-y\parallel }^2.$$

(22)

On the other hand, since $f(·)+{\displaystyle \frac{1}{2}}{l}_f{\parallel ·-y\parallel }^2$ is a convex function with gradient $\nabla f\left(y\right)$ at y, the following inequality holds:

$$f\left(x^{\prime }\right)+{\displaystyle \frac{1}{2}}{l}_f{\parallel x^{\prime }-y\parallel }^2\ge f\left(y\right)+\langle \nabla f\left(y\right),x^{\prime }-y\rangle .$$

(23)

Then (21) follows from (22) and (23).    □

To simplify our discussion in Section 4, Section 5 and Section 6, we denoted by

$${\Delta }_k:=x^k-x^{k-1},k=1,2,\dots$$

(24)

Based on (14) and (16), we have

$$\begin{array}{cc}\hfill z^{k+1}-x^k& ={\displaystyle \frac{1}{{\theta }_k}}{\Delta }_{k+1},\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill x^k-z^k& =-{\displaystyle \frac{{\gamma }_k}{{\theta }_k}}{\Delta }_k,\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill z^{k+1}-z^k& ={\displaystyle \frac{1}{{\theta }_k}}(x^{k+1}-y^k)={\displaystyle \frac{1}{{\theta }_k}}{\Delta }_{k+1}-{\displaystyle \frac{{\gamma }_k}{{\theta }_k}}{\Delta }_k,\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill x^k-y^k& ={\theta }_k(x^k-z^k)=-{\gamma }_k{\Delta }_k,\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill z^{k+1}-y^k& ={\displaystyle \frac{1}{{\theta }_k}}{\Delta }_{k+1}-{\gamma }_k{\Delta }_k.\hfill \end{array}$$

(29)

4. Convergence Properties

In this section, we analyze the convergence properties of Algorithm 1. A central element of our analysis is the following auxiliary function:

$$\begin{array}{cc}\hfill Q(x,x^{\prime },y,\alpha )=& {\alpha }^{-1}[F\left(x\right)-\overline{m}]+\mathrm{\Psi }(x,y)+{\displaystyle \frac{1}{2}}{L}_g{\parallel x-y\parallel }^2+{\displaystyle \frac{1}{2}}(L_g+{\alpha }^{-1}{L}_f){\parallel x-x^{\prime }\parallel }^2,\hfill \end{array}$$

(30)

where

$$\overline{m}:=inf\left\{F\left(x\right):x\in C\right\}.$$

(31)

Theorem 1

(Vanishing successive changes). Consider the Problem (1) under Assumptions 1 and 2. Let $\{x^k,y^k,z^k,{\alpha }_k\}$ be generated by Algorithm 1. Then the following statements hold:

(i) 

For $k\ge 1$, it holds that

$$H_k-H_{k+1}\ge {\displaystyle \frac{\delta }{2}}(L_g+{\alpha }_k^{-1}{L}_f){\parallel {\Delta }_k\parallel }^2,$$

(32)

where

$$\begin{array}{cc}\hfill H_k& :=Q(x^k,x^{k-1},y^{k-1},{\alpha }_k).\hfill \end{array}$$

(33)

(ii) 

${\sum }_{k=0}^{\infty }{\parallel {\Delta }_k\parallel }^2<\infty$, ${lim}_{k\to \infty }\parallel {\Delta }_k\parallel =0$, and    ${lim}_{k\to \infty }\parallel z^{k+1}-x^k\parallel ={lim}_{k\to \infty }\parallel x^k-y^k\parallel ={lim}_{k\to \infty }$$\parallel z^{k+1}-z^k\parallel ={lim}_{k\to \infty }\parallel z^{k+1}-y^k\parallel =0$.

(iii) 

The sequence $\left\{x^k\right\}\subset C$ and is bounded.

Proof. 

(i)

Since $E\left(z\right)$ is a strongly convex function with parameter ${\theta }_k({\alpha }_k{L}_g+L_f)$ and $z^{k+1}$ denotes its minimizer over the set C, the 3-Point Property ([46] Lemma 3.2) yields

$$E\left(z^{k+1}\right)<E\left(x^k\right)-{\displaystyle \frac{1}{2}}{\theta }_k({\alpha }_k{L}_g+L_f){\parallel z^{k+1}-x^k\parallel }^2,$$

(34)

which is equivalent to:

$$\begin{array}{cc}\hfill P_1\left(z^{k+1}\right)\le & P_1\left(x^k\right)+\langle {\xi }^k-\nabla f\left(y^k\right),z^{k+1}-x^k\rangle +{\alpha }_k[\mathrm{\Psi }(x^k,y^k)-\mathrm{\Psi }(z^{k+1},y^k)]\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{\theta }_k({\alpha }_k{L}_g+L_f)\left[\parallel x^k-z^k{\parallel }^2-\parallel z^{k+1}-z^k{\parallel }^2-{\parallel z^{k+1}-x^k\parallel }^2\right].\hfill \end{array}$$

(35)

Substituting the equalities (25)–(27) into the above inequality, we obtain

$$\begin{array}{cc}\hfill P\left(z^{k+1}\right)\le & P_1\left(x^k\right)+{\displaystyle \frac{1}{{\theta }_k}}\langle {\xi }^k-\nabla f\left(y^k\right),{\Delta }_{k+1}\rangle +{\alpha }_k[\mathrm{\Psi }(x^k,y^k)-\mathrm{\Psi }(z^{k+1},y^k)]\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{\theta }_k({\alpha }_k{L}_g+L_f)\left[{\displaystyle \frac{{\gamma }_k^2}{{\theta }_k^2}}\parallel {\Delta }_k{\parallel }^2-{\displaystyle \frac{1}{{\theta }_k^2}}\parallel x^{k+1}-y^k{\parallel }^2-{\displaystyle \frac{1}{{\theta }_k^2}}{\parallel {\Delta }_{k+1}\parallel }^2\right].\hfill \end{array}$$

(36)

By virtue of (16) and convexity of $P_1$, it follows that

$$\begin{array}{cc}\hfill P_1\left(x^{k+1}\right)\le & P_1\left(x^k\right)+{\theta }_k[P_1\left(z^{k+1}\right)-P_1\left(x^k\right)]\hfill \\ \hfill \le & P_1\left(x^k\right)+\langle {\xi }^k,{\Delta }_{k+1}\rangle -\langle \nabla f\left(y^k\right),{\Delta }_{k+1}\rangle \hfill \\ \hfill & +{\theta }_k{\alpha }_k[\mathrm{\Psi }(x^k,y^k)-\mathrm{\Psi }(z^{k+1},y^k)]\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}({\alpha }_k{L}_g+L_f)\left[{\gamma }_k^2\parallel {\Delta }_k{\parallel }^2-\parallel x^{k+1}-y^k{\parallel }^2-{\parallel {\Delta }_{k+1}\parallel }^2\right].\hfill \end{array}$$

(37)

Combining this result with two key inequalities:

$$-P_2\left(x^{k+1}\right)+P_2\left(x^k\right)\le \langle {\xi }^k,{\Delta }_{k+1}\rangle ,$$

(38)

which holds due to the convexity of $P_2$ and the fact that ${\xi }^k\in \partial P_2\left(x^k\right)$;

$$\begin{array}{cc}\hfill & f\left(x^{k+1}\right)-f\left(x^k\right)\hfill \\ \hfill \le & \langle \nabla f\left(y^k\right),x^{k+1}-x^k\rangle +{\displaystyle \frac{1}{2}}{L}_f\parallel x^{k+1}-y^k{\parallel }^2+{\displaystyle \frac{1}{2}}{l}_f{\parallel x^k-y^k\parallel }^2,\hfill \end{array}$$

(39)

which is derived from Lemma 4 by substituting $x,x^{\prime },y$ with $x^{k+1},x^k,y^k$, respectively, we arrive at

$$\begin{array}{cc}\hfill & F\left(x^{k+1}\right)-F\left(x^k\right)\hfill \\ \hfill \le & {\displaystyle \frac{1}{2}}{L}_f\parallel x^{k+1}-y^k{\parallel }^2+{\displaystyle \frac{1}{2}}{l}_f{\parallel x^k-y^k\parallel }^2+{\theta }_k{\alpha }_k[\mathrm{\Psi }(x^k,y^k)-\mathrm{\Psi }(z^{k+1},y^k)]\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}({\alpha }_k{L}_g+L_f)\left[{\gamma }_k^2\parallel {\Delta }_k{\parallel }^2-\parallel x^{k+1}-y^k{\parallel }^2-{\parallel {\Delta }_{k+1}\parallel }^2\right].\hfill \end{array}$$

(40)

Thus,

$$\begin{array}{cc}\hfill & {\displaystyle \frac{F\left(x^{k+1}\right)-\overline{m}}{{\alpha }_{k+1}}}+\mathrm{\Psi }(x^{k+1},y^k)-\left[{\displaystyle \frac{F\left(x^k\right)-\overline{m}}{{\alpha }_k}}+\mathrm{\Psi }(x^k,y^{k-1})\right]\hfill \\ \hfill \le & {\alpha }_k^{-1}\left[F\left(x^{k+1}\right)-F\left(x^k\right)\right]+\mathrm{\Psi }(x^{k+1},y^k)-\mathrm{\Psi }(x^k,y^{k-1})\hfill \\ \hfill \le & \mathrm{\Psi }(x^{k+1},y^k)-\mathrm{\Psi }(x^k,y^{k-1})+{\theta }_k\mathrm{\Psi }(x^k,y^k)-{\theta }_k\mathrm{\Psi }(z^{k+1},y^k)\hfill \\ \hfill & +{\displaystyle \frac{1}{2{\alpha }_k}}{L}_f\parallel x^{k+1}-y^k{\parallel }^2+{\displaystyle \frac{1}{2{\alpha }_k}}{l}_f{\parallel x^k-y^k\parallel }^2\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}\left(L_g+{\displaystyle \frac{1}{{\alpha }_k}}{L}_f\right)\left[{\gamma }_k^2\parallel {\Delta }_k{\parallel }^2-\parallel x^{k+1}-y^k{\parallel }^2-{\parallel {\Delta }_{k+1}\parallel }^2\right],\hfill \end{array}$$

(41)

where the first inequality follows from the setting of ${\alpha }_{k+1}$ in Algorithm 1 (which ensures ${\alpha }_{k+1}\ge {\alpha }_k$), and the second inequality is a consequence of (40).

Next, leveraging the convexity of $\mathrm{\Psi }(·,y^k)$, we have

$$\mathrm{\Psi }(x^{k+1},y^k)\le {\theta }_k\mathrm{\Psi }(z^{k+1},y^k)+(1-{\theta }_k)\mathrm{\Psi }(x^k,y^k).$$

(42)

Additionally, by Lemma 3 (with $x,y,y^{\prime }$ replaced by $x^k,y^k,y^{k-1}$, respectively),

$$\mathrm{\Psi }(x^k,y^k)\le \mathrm{\Psi }(x^k,y^{k-1})+{\displaystyle \frac{1}{2}}{L}_g\parallel x^k-y^{k-1}{\parallel }^2+{\displaystyle \frac{1}{2}}{l}_g{\parallel x^k-y^k\parallel }^2.$$

(43)

Combining these two inequalities and (41) gives

$$\begin{array}{cc}\hfill & {\displaystyle \frac{F\left(x^{k+1}\right)-\overline{m}}{{\alpha }_{k+1}}}+\mathrm{\Psi }(x^{k+1},y^k)-\left[{\displaystyle \frac{F\left(x^k\right)-\overline{m}}{{\alpha }_k}}+\mathrm{\Psi }(x^k,y^{k-1})\right]\hfill \\ \hfill \le & {\displaystyle \frac{1}{2}}{L}_g\parallel x^k-y^{k-1}{\parallel }^2+{\displaystyle \frac{1}{2}}{l}_g\parallel x^k-y^k{\parallel }^2+{\displaystyle \frac{1}{2{\alpha }_k}}{L}_f{\parallel x^{k+1}-y^k\parallel }^2\hfill \\ \hfill & +{\displaystyle \frac{1}{2{\alpha }_k}}{l}_f{\parallel x^k-y^k\parallel }^2+{\displaystyle \frac{1}{2}}\left(L_g+{\displaystyle \frac{1}{{\alpha }_k}}\right)\left[{\gamma }_k^2\parallel {\Delta }_k{\parallel }^2-\parallel x^{k+1}-y^k{\parallel }^2-{\parallel {\Delta }_{k+1}\parallel }^2\right].\hfill \end{array}$$

(44)

Together with the definition of $H_k$ in (33), this implies

$$\begin{array}{cc}\hfill & H_{k+1}-H_k\hfill \\ \hfill \le & {\displaystyle \frac{1}{2}}{L}_g\parallel x^{k+1}-y^k{\parallel }^2+{\displaystyle \frac{1}{2}}\left(L_g+{\alpha }_{k+1}^{-1}{L}_f\right)\parallel {\Delta }_{k+1}{\parallel }^2-{\displaystyle \frac{1}{2}}{L}_g{\parallel x^k-y^{k-1}\parallel }^2\hfill \\ \hfill & -{\displaystyle \frac{1}{2}}(L_g+{\alpha }_k^{-1}{L}_f)\parallel {\Delta }_k{\parallel }^2+{\displaystyle \frac{1}{2}}{L}_g\parallel x^k-y^{k-1}{\parallel }^2+{\displaystyle \frac{1}{2}}{l}_g{\parallel x^k-y^k\parallel }^2\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}{\alpha }_k^{-1}{L}_f\parallel x^{k+1}-y^k{\parallel }^2+{\displaystyle \frac{1}{2}}{\alpha }_k^{-1}{l}_f{\parallel x^k-y^k\parallel }^2\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}\left(L_g+{\alpha }_k^{-1}{L}_f\right)\left[{\gamma }_k^2\parallel {\Delta }_k{\parallel }^2-\parallel x^{k+1}-y^k{\parallel }^2-{\parallel {\Delta }_{k+1}\parallel }^2\right]\hfill \\ \hfill =& {\displaystyle \frac{1}{2}}\left({\alpha }_{k+1}^{-1}-{\alpha }_k^{-1}\right)L_f{\parallel {\Delta }_{k+1}\parallel }^2\hfill \\ \hfill & +{\displaystyle \frac{1}{2}}\left[(L_g+l_g){\gamma }_k^2-L_g+{\alpha }_k^{-1}\left((L_f+l_f){\gamma }_k^2-L_f\right)\right]{\parallel {\Delta }_k\parallel }^2\hfill \\ \hfill \le & -{\displaystyle \frac{1}{2}}(L_g+{\alpha }_k^{-1}{L}_f)\delta {\parallel {\Delta }_k\parallel }^2,\hfill \end{array}$$

(45)

where the equality relies on (28), and the final inequality holds due to ${\alpha }_{k+1}\ge {\alpha }_k$ and Assumption 2.

(ii)

From (45), we deduce

$$\begin{array}{c}\hfill \sum _{k=1}^t{\displaystyle \frac{\delta }{2}}{\parallel {\Delta }_k\parallel }^2\le \sum _{k=1}^t(H_k-H_{k+1})=H_1-H_{t+1}\le H_1-\underset{k\to \infty }{lim\; inf}{H}_k<+\infty .\end{array}$$

We directly conclude that ${\sum }_{k=0}^{\infty }{\parallel {\Delta }_k\parallel }^2<\infty$ and ${lim}_{k\to \infty }\parallel {\Delta }_k\parallel =0$. Since Assumption 2 guarantees ${inf}_k{\theta }_k>0$, combining this with ${lim}_{k\to \infty }\parallel {\Delta }_k\parallel =0$ and Equations (25) and (27)–(29), we further obtain the remaining limit conclusions.

(iii)

According to (15) and (16), and the convexity of C, $x^k\in C$ for each k. If C is compact, as part (a) of Assumption 1(iv) holds, the sequence $\left\{x_k\right\}$ is obviously bounded. Otherwise, we have all $g_i=0$ and F is level bounded. Thus all ${\alpha }_k={\alpha }_0$ in our algorithm. From (48), we observe that $F\left(x^k\right)\le {\alpha }_0{H}_k+\overline{m}\le {\alpha }_0{H}_1+\overline{m}<+\infty$. So $\left\{x^k\right\}$ is bounded by the level boundedness of F.

   □

In Algorithm 1, if the penalty parameters $\left\{{\alpha }_k\right\}$ are unbounded, the influence of the objective function on subproblem (15) will diminish. Consequently, we cannot guarantee the critical point property for any cluster point of $\left\{x^k\right\}$.

To establish the boundedness of $\left\{{\alpha }_k\right\}$, the following assumption is critical. First, introduced in ([18] Assumption (A1)) for analyzing ESQM, this assumption was also adopted in [19] for ${\mathrm{ESQM}}_e$.

Assumption 3.

For (1), the $RCQ\left(x\right)$ holds at every $x\in C\cap \mathcal{F}$, and for every $x\in C\setminus \mathcal{F}$, there cannot exist $u_i,i\in I\left(x\right)$, such that

$$\begin{array}{c}\hfill u_i\ge 0\forall i\in I\left(x\right),\sum _{i\in I\left(x\right)}{u}_i=1,〈\sum _{i\in I\left(x\right)}{u}_i\nabla g_i\left(x\right),z-x〉\ge 0\forall z\in C,\end{array}$$

(46)

where $I\left(x\right):=\left\{l\in \{1,\dots ,m\}:g_l\left(x\right)={max}_{i=1,\dots ,m}{\left[g_i\left(x\right)\right]}_{+}\right\}$.

Remark 2.

(i) 

As shown in ([18] Remark 2.1), if Assumption 3 holds, then for any $x\in C$, there exist no $u_i$ (for $i\in I\left(x\right)$) that satisfy (46).

(ii) 

As shown in ([18] Remark 2.2), if RCQ(x) holds for all $x\in C$, then Assumption 3 is satisfied.

Theorem 2

(Boundedness of the penalty parameters $\left\{{\alpha }_k\right\}$). Consider (1) and suppose that Assumption 1–3 hold. Let $\left\{(x^k,y^k,z^k,{\alpha }_k)\right\}$ be generated by Algorithm 1. Then the sequence $\left\{{\alpha }_k\right\}$ is bounded above, i.e., there exists $K_0\in \mathbb{N}$ such that ${\alpha }_k={\alpha }_{K_0}$ whenever $k\ge K_0$.

Proof. 

Suppose on the contrary that $\left\{{\alpha }_k\right\}$ is unbounded above. By the definition of ${\alpha }_k$ in Algorithm 1, there exists a subsequence of positive integers $\left\{k_j\right\}$ such that

$$\mathrm{\Psi }(z^{k_j+1},y^{k_j})>0.$$

(47)

Moreover, we have ${lim}_{k\to \infty }{\alpha }_k=\infty$ and ${lim}_{k\to \infty }{\alpha }_k^{-1}=0$.

Recalling the definitions of $I_k(·)$ in (20) and $g_0$ in (12), we have $0\notin I_{k_j}\left(z^{k_j+1}\right)$ and

$${lin}_{g_i}(z^{k_j+1},y^{k_j})>0\forall i\in I_{k_j}\left(z^{k_j+1}\right),\forall j.$$

Now, in view of the finiteness of $\left\{I_{k_j}\left(z^{k_j+1}\right)\right\}$ (since $I_{k_j}\left(z^{k_j+1}\right)\subseteq \{1,\dots ,m\}$ for all j), by passing to a further subsequence if necessary, we deduce that there exists a nonempty subset $I_0\subseteq \{1,\dots ,m\}$ such that $I_{k_j}\left(z^{k_j+1}\right)\equiv I_0$ for all j. That is, for all $i\in I_0$,

$${lin}_{g_i}(z^{k_j+1},y^{k_j})=\mathrm{\Psi }(z^{k_j+1},y^{k_j})>0\forall j.$$

(48)

In addition, from Lemma 2(ii), we have that for each $k_j$, there exist ${\lambda }_i^{k_j}\ge 0$ for each $i\in I_{k_j}\left(z^{k_j+1}\right)\equiv I_0$, such that ${\sum }_{i\in I_0}{\lambda }_i^{k_j}=1$ and

$$\begin{array}{cc}\hfill 0\in & {\alpha }_{k_j}^{-1}[\partial P_1\left(z^{k_j+1}\right)+\nabla f\left(y^{k_j}\right)-{\xi }^{k_j}]+{\theta }_{k_j}(L_g+{\alpha }_{k_j}^{-1}{L}_f)(z^{k_j+1}-z^{k_j})\hfill \\ \hfill & +\sum _{i\in I_0}{\lambda }_i^{k_j}\nabla g_i\left(y^{k_j}\right)+{\mathcal{N}}_C\left(z^{k_j+1}\right).\hfill \end{array}$$

(49)

Since the sequences $\left\{x^{k_j}\right\}\subseteq C$ and $\left\{{\lambda }_i^{k_j}\right\}$ (for each $i\in I_0$) are bounded, by passing to a further subsequence if necessary, we assume that ${lim}_{j\to \infty }{x}^{k_j}=x^{*}$ for some $x^{*}$ and that for each $i\in I_0$, ${lim}_{j\to \infty }{\lambda }_i^{k_j}={\overline{\lambda }}_i$ for some ${\overline{\lambda }}_i$. Then $x^{*}\in C$, ${\overline{\lambda }}_i\ge 0$ (for each $i\in I_0$), ${\sum }_{i\in I_0}{\overline{\lambda }}_i=1$ and $I_0\subseteq \{\iota \in \{0,1,\cdots ,m\}:g_{\iota }\left(x^{*}\right)={max}_{i=0,1,\dots ,m}{g}_i\left(x^{*}\right)\}$. Since $0\notin I_0$, we see that

$$\begin{array}{c}\hfill I_0\subseteq I\left(x^{*}\right),\end{array}$$

where $I\left(x\right)$ was defined in Assumption 3. Passing to the limit in (49), and noting that ${lim}_{j\to \infty }{\alpha }_{k_j}^{-1}=0$, ${lim}_{k\to \infty }\parallel z^{k+1}-z^k\parallel =0$ (thanks to Theorem 1(iii)), and the fact that $\{\partial P_1\left(x^{k_j+1}\right)\}$, $\{\nabla f\left(y^{k_j}\right)\}$, and $\left\{{\xi }^{k_j}\right\}$ are uniformly bounded (thanks to the boundedness of $\left\{x^k\right\}$ and $\left\{y^k\right\}$, the convexity of $P_1$, $P_2$ and ([47] Theorem 24.7), and the Lipschitz continuity of $\{\nabla f\}$), we have upon invoking the closedness of $x\mapsto {\mathcal{N}}_C\left(x\right)$ that

$$0\in \sum _{i\in I_0}{\overline{\lambda }}_i\nabla g_i\left(x^{*}\right)+{\mathcal{N}}_C\left(x^{*}\right),$$

which implies that

$$〈\sum _{i\in I_0}{\overline{\lambda }}_i\nabla g_i\left(x^{*}\right),x-x^{*}〉\ge 0\forall x\in C.$$

Since $I_0\subseteq I\left(x^{*}\right)$, this contradicts Assumption 3 in view of Remark 2(i). We complete the proof.    □

Theorem 3

(Subsequential convergence). Consider (1) and suppose that Assumptions 1–3 hold. Let $\left\{x^k\right\}$ be generated by Algorithm 1. Then, for any accumulation point $\overline{x}$ of $\left\{x^k\right\}$, there exists ${\overline{\lambda }}_i\ge 0$ for each $i\in \tilde{I}\left(\overline{x}\right)$ such that ${\sum }_{i\in \tilde{I}\left(\overline{x}\right)}{\overline{\lambda }}_i=1$ and

$$\begin{array}{c}\hfill 0\in \partial P_1\left(\overline{x}\right)+\nabla f\left(\overline{x}\right)-\partial P_2\left(\overline{x}\right)+{\alpha }_{K_0}\sum _{i\in \tilde{I}\left(\overline{x}\right)}{\overline{\lambda }}_i\nabla g_i\left(\overline{x}\right)+{\mathcal{N}}_C\left(\overline{x}\right),\end{array}$$

(50)

where $\tilde{I}\left(\overline{x}\right):=\{\iota \in \{0,1,\cdots ,m\}:g_{\iota }\left(\overline{x}\right)={max}_{i=0,1,\dots ,m}\left\{g_i\left(\overline{x}\right)\right\}\}$, and ${\alpha }_{K_0}$ is defined in Theorem 2; moreover, $\overline{x}$ is a critical point of Problem (1).

Proof. 

Suppose that $\overline{x}$ is an accumulation point of $\left\{x^k\right\}$, so there exists a convergent subsequence $\left\{x^{k_j}\right\}$ such that ${lim}_{j\to \infty }{x}^{k_j}=\overline{x}$. Let $\left\{{\xi }^k\right\}$ be the sequence generated in Algorithm 1, and let $\left\{{\lambda }_i^k\right\}$ (for $i\in I_{k_j}\left(z^{k_j+1}\right)$) be the sequence specified in Lemma 2(ii). Note first that for all j, $I_{k_j}\left(z^{k_j+1}\right)\subseteq \{0,1,\dots ,m\}$, which implies the set $\left\{I_{k_j}\left(z^{k_j+1}\right)\right\}$ is finite. By passing to a further subsequence if necessary, we may assume there exists a nonempty subset $I_0\subseteq \{0,1,\dots ,m\}$ such that $I_{k_j}\left(z^{k_j+1}\right)\equiv I_0$. From Lemma 2(ii), it follows that

$$\begin{array}{cc}\hfill 0\in & \partial P_1\left(z^{k_j+1}\right)+\nabla f\left(y^{k_j}\right)-{\xi }^{k_j}+{\theta }_{k_j}({\alpha }_{k_j}{L}_g+L_f)(z^{k_j+1}-z^{k_j})\hfill \\ \hfill & +{\alpha }_{k_j}\sum _{i\in I_0}{\lambda }_i^{k_j}\nabla g_i\left(y^{k_j}\right)+{\mathcal{N}}_C\left(z^{k_j+1}\right)\hfill \\ \hfill \mathrm{and}& \sum _{i\in I_0}{\lambda }_i^{k_j}=1,{\lambda }_i^{k_j}\ge 0\forall i\in I_{k_j}\left(z^{k_j+1}\right)\equiv I_0.\hfill \end{array}$$

(51)

Moreover, for each $i\in I_{k_j}\left(z^{k_j+1}\right)\equiv I_0$, the sequence $\left\{{\lambda }_i^{k_j}\right\}$ consists of nonnegative numbers bounded above by 1, hence is bounded. As for $\left\{{\xi }^k\right\}$, its boundedness is guaranteed by the fact that $P_2$ is convex, together with ([47] Theorem 24.7). By passing to another further subsequence if needed, we may assume without loss of generality that ${lim}_{j\to \infty }{\lambda }_i^{k_j}={\overline{\lambda }}_i\ge 0$ for each $i\in I_0$, and ${lim}_{j\to \infty }{\xi }^{k_j}=\overline{\xi }$. Additionally, Theorem 2 implies that for all $k_j\ge K_0$, ${\alpha }_{k_j}={\alpha }_{K_0}$ and

$$\mathrm{\Psi }(z^{k_j+1},y^{k_j})=0.$$

(52)

Taking the limit as $j\to \infty$ in both (51) and (52)—and recalling Theorem 1(ii) along with the closedness of $\partial P_1$, $\partial P_2$ and ${\mathcal{N}}_C$—we obtain

$$\begin{array}{cc}\hfill 0& \in \nabla f\left(\overline{x}\right)+\partial P_1\left(\overline{x}\right)-\partial P_2\left(\overline{x}\right)+{\alpha }_{K_0}\sum _{i\in I_0}{\overline{\lambda }}_i\nabla g_i\left(\overline{x}\right)+{\mathcal{N}}_C\left(\overline{x}\right),\hfill \\ \hfill & \sum _{i\in I_0}{\overline{\lambda }}_i=1,{\overline{\lambda }}_i\ge 0\forall i\in I_0,\hfill \end{array}$$

(53)

and

$$\mathrm{\Psi }(\overline{x},\overline{x})=0,$$

(54)

where the above inequality is equivalent to (noting that $\mathrm{\Psi }(\overline{x},\overline{x})={max}_{i=0,1,\dots ,m}\left\{g_i\left(\overline{x}\right)\right\}$ by (11) and (13))

$$g_i\left(\overline{x}\right)\le 0\forall i=1,\dots ,m.$$

(55)

Furthermore, from the definition of $I_{k_j}\left(z^{k_j+1}\right)$ in (20) (and since $I_{k_j}\left(z^{k_j+1}\right)\equiv I_0$) and Theorem 1(ii), we have

$$\begin{array}{cc}\hfill I_0\subseteq \tilde{I}\left(\overline{x}\right):=& \left\{\iota \in \{0,1,\cdots ,m\}:g_{\iota }\left(\overline{x}\right)=\underset{i=0,1,\dots ,m}{max}\left\{g_i\left(\overline{x}\right)\right\}\right\}.\hfill \end{array}$$

(56)

Noting that the above inclusion, we define ${\overline{\lambda }}_i=0$ for $i\in \tilde{I}\left(\overline{x}\right)\setminus I_0$. With this definition, the inclusion (50) follows directly from (53).

Finally, define ${\widehat{\lambda }}_i:={\alpha }_{N_0}{\overline{\lambda }}_i\ge 0$ for all $i\in I_0\cap \{1,\cdots ,m\}$, and ${\widehat{\lambda }}_i=0$ for all $i\in \{1,\cdots ,m\}\setminus I_0$. By (55) and the fact that $I_0\subseteq \tilde{I}\left(\overline{x}\right)$ (see (56)), we have

$${\widehat{\lambda }}_i{g}_i\left(\overline{x}\right)=0\forall i=1,\dots ,m.$$

(57)

To verify this, observe that for each $i\in I_0$, we have $g_i\left(\overline{x}\right)=0$, and for each $i\notin I_0$, we have ${\widehat{\lambda }}_i=0$. Note also that $\nabla g_0\left(\overline{x}\right)=0$ (since $g_0\equiv 0$). Using the definition of ${\widehat{\lambda }}_i$ and (53), we find

$$0\in \partial P_1\left(\overline{x}\right)+\nabla f\left(\overline{x}\right)-\partial P_2\left(\overline{x}\right)+\sum _{i=1}^m{\widehat{\lambda }}_i\nabla g_i\left(\overline{x}\right)+{\mathcal{N}}_C\left(\overline{x}\right).$$

(58)

Combining (55), (57), (58) and the definition of $\widehat{\lambda }$ above, we conclude that $\overline{x}$ is a critical point of (1).    □

5. ${\mathrm{EAPG}}_s$ with the Restart Technique

To discuss the global convergence and convergence rate of the sequence $\left\{x^k\right\}$ generated by proximal-gradient-type algorithms (e.g., the algorithms proposed in [19,25,26,42,48] and numerous other related methods), the following conditions are typically indispensable: (1) An inequality analogous to Theorem 1(i), which involves an auxiliary sequence (e.g., $\left\{H_k\right\}$) that depends on $\left\{x^k\right\}$, $\left\{F\left(x^k\right)\right\}$ and potentially other sequences; (2) Certain boundedness properties of the subgradients of the objective function within the subproblem (e.g., with respect to $\left\{x^k\right\}$ for the function $E(·)$); (3) The Kurdyka–Łojasiewicz (KL) property of an auxiliary function H associated with $\left\{H_k\right\}$. However, for proximal gradient algorithms incorporating the second acceleration technique, analyzing condition (2) poses a significant challenge. The core reason lies in the structural design of these methods: in such algorithms, $z^k$ (rather than $x^k$) serves as the minimizer of subproblem (15), while $x^k$ is merely a linear combination of $z^k$ and $x^{k-1}$. Consequently, although we can derive bounds for the gradient of $E(·)$ at $z^k$, we lack sufficient information to characterize the subdifferential $\partial E$ at $x^k$.

The solution approach adopted in [25,26,36] relies on imposing Lipschitz continuity assumptions on the gradients of all involved functions. Nevertheless, this assumption is overly restrictive: in practical scenarios, the function $P_1$ is often nondifferentiable (e.g., $P_1={\parallel ·\parallel }_1$, the $l_1$-norm), and it is even inapplicable to Algorithm 1—since the function $\mathrm{\Psi }$ in this algorithm fails to be differentiable everywhere.

To address the aforementioned difficulty, it is essential to revisit the working mechanism of Nesterov’s second acceleration method for proximal gradient algorithms, which was first investigated in [23]. Specifically, ([23] Theorem 5.1) demonstrates that if the objective function is strongly convex and differentiable with a Lipschitz-continuous gradient, then $x^k$ always yields a better performance (in terms of objective function value) than $z^k$. For nonconvex problems, however, this conclusion no longer holds—we cannot guarantee that $x^k$ outperforms $z^k$.

Given the possibility that $z^k$ may be superior to $x^k$ in nonconvex settings, continuing the acceleration operation (which relies on $x^k$ as the update base) may not be optimal. Thus, by drawing inspiration from the restart strategy used in Nesterov’s first acceleration method [33], we propose the following algorithm (Algorithm 2):

Algorithm 2$$ for solving (1) with the restart technique (Theoretical version).

Initialization: $x^{(1,0)}=x^0\in C$, $z^{(1,0)}=z^0\in C$, ${\alpha }_{(1,0)}={\alpha }_0>0$, $d>0$. for$s=1,2,\dots$ do carry out Algorithm 1 with initial values $x^{(s,0)}$, $z^{(s,0)}$, ${\alpha }_{(s,0)}$ and preselected parameters $\left\{{\theta }_{(s,k^{\prime })}\right\}$ satisfying Assumption 4 to generate the sequence $\left\{(x^{(s,k^{\prime })},y^{(s,k^{\prime })},z^{(s,k^{\prime })},{\alpha }_{(s,k^{\prime })})\right\}$ until $$\begin{array}{c}\hfill Q(x^{(s,k^{\prime })},x^{(s,k^{\prime }-1)},y^{(s,k^{\prime }-1)},{\alpha }_{(s,k^{\prime })})>Q(z^{(s,k^{\prime })},x^{(s,k^{\prime }-1)},y^{(s,k^{\prime }-1)},{\alpha }_{(s,k^{\prime })})\end{array}$$ (59) where Q is defined in (30). Denote the above $k^{\prime }$ by $N_s$. Set: $x^{(s+1,0)}=z^{(s+1,0)}=z^{(s,N_s)}$, ${\alpha }_{(s+1,0)}={\alpha }_{(s,N_s)}$. end for

Assumption 4.

With ${inf}_{(s,k^{\prime })}\theta (s,k^{\prime })>0$, there exists a constant $\delta \in (0,1)$ such that

$$\begin{array}{cc}\hfill (L_g+l_g){\gamma }_{(s,k^{\prime })}^2& \le L_g(1-\delta ),\hfill \\ \hfill (L_f+l_f){\gamma }_{(s,k^{\prime })}^2& \le L_f(1-\delta ),\hfill \end{array}$$

(60)

where ${\gamma }_{(s,k^{\prime })}:={\theta }_{(s,k^{\prime })}(1-{\theta }_{(s,k^{\prime }-1)}){\theta }_{(s,k^{\prime }-1)}^{-1}$ for $k^{\prime }\ge 1$.

In the subsequent discussion of this section, for simplicity, we use $\left\{x^k\right\}$ to denote the sequence

$$\left\{x^{(s,k^{\prime })}\right\}:x^{(1,0)},\dots ,x^{(1,N_1-1)},x^{(2,0)},\dots ,x^{(2,N_2-1)},x^{(3,0)}\dots$$

generated by Algorithm 2. Similarly, we use $\left\{y^k\right\}$, $\left\{z^k\right\}$, $\left\{{\alpha }_k\right\}$, $\left\{{\theta }_k\right\}$, and $\left\{{\gamma }_k\right\}$ to represent $\left\{y^{(s,k^{\prime })}\right\}$, $\left\{z^{(s,k^{\prime })}\right\}$, $\left\{{\alpha }_{(s,k^{\prime })}\right\}$, $\left\{{\theta }_{(s,k^{\prime })}\right\}$, and $\left\{{\gamma }_{(s,k^{\prime })}\right\}$, respectively. We also use $k=k(s,k^{\prime })$ to denote the correspondence mapping $(s,k^{\prime })$ to k. As will be shown below, these sequences satisfy the same results as those established in Section 4.

Remark 3.

For the sequences generated by Algorithm 2, we have the following relations:

(i) 

When $k=k(s,k^{\prime })$ for $1\le k^{\prime }\le N_s-2$, the equality Equations (25)–(29) remain valid.

(ii) 

When $k=k(s,N_s-1)$, we have $x^{k+1}=y^{k+1}=z^{k+1}$, Equations (26) and (28) still hold, and

$$\begin{array}{cc}\hfill z^{k+1}-x^k& ={\Delta }_{k+1},\hfill \\ \hfill z^{k+1}-z^k& ={\Delta }_{k+1}-{\displaystyle \frac{{\gamma }_k}{{\theta }_k}}{\Delta }_k,\hfill \\ \hfill z^{k+1}-y^k& ={\Delta }_{k+1}-{\gamma }_k{\Delta }_k.\hfill \end{array}$$

(iii) 

When $k=k(s+1,0)$, we have $x^k=y^k=z^k$, Equation (25) holds, and we have

$$\begin{array}{cc}\hfill x^k-z^k& =x^k-y^k=0,\hfill \\ \hfill z^{k+1}-y^k& =z^{k+1}-z^k=z^{k+1}-x^k={\displaystyle \frac{1}{{\theta }_k}}{\Delta }_{k+1}.\hfill \end{array}$$

Lemma 5.

The results established in Theorems 1–3 remain valid for the sequence $\{x^k,y^k,z^k,{\alpha }_k\}$ generated by Algorithm 2, provided the same conditions are satisfied with Assumption 2 replaced by Assumption 4.

Proof. 

First, we prove that (32) remains valid for Algorithm 2. From Theorem 1(i), we know that for $s\ge 1$ and $1\le k^{\prime }\le N_s-1$, the following holds:

$$\begin{array}{cc}\hfill & Q\left(x^{(s,k^{\prime })},x^{(s,k^{\prime }-1)},y^{(s,k^{\prime }-1)},{\alpha }_{(s,k^{\prime })}\right)-Q\left(x^{(s,k^{\prime }+1)},x^{(s,k^{\prime })},y^{(s,k^{\prime })},{\alpha }_{(s,k^{\prime }+1)}\right)\hfill \\ \hfill \ge & {\displaystyle \frac{\delta }{2}}\left(L_g+{\alpha }_{(s,k^{\prime })}^{-1}{L}_f\right){∥x^{(s,k^{\prime })}-x^{(s,k^{\prime }-1)}∥}^2.\hfill \end{array}$$

This inequality leads to the conclusion that (32) is valid in two cases:

(i) When $k=k(s,k^{\prime })$ with $1\le k^{\prime }\le N_s-2$.

(ii) When $k=k(s,N_s-1)$. Here, we note that $x^{k+1}=x^{(s+1,0)}=z^{(s,N_s)}$, ${\alpha }_{k+1}={\alpha }_{(s+1,0)}={\alpha }_{(s,N_s)}$, and that (59) holds for $k^{\prime }=N_s$ due to the setup of Algorithm 2.

Now, we need to address the case where $k=k(s+1,0)$. In this situation, $y^k=z^k=x^k$, and (35) still holds for $z^{k+1}$, being equivalent to:

$$\begin{array}{cc}\hfill P_1\left(z^{k+1}\right)\le & P_1\left(x^k\right)+\langle \xi -\nabla f\left(x^k\right),z^{k+1}-x^k\rangle -{\alpha }_k\mathrm{\Psi }(z^{k+1},y^k)-{\theta }_k({\alpha }_k{L}_g+L_f){\parallel z^{k+1}-x^k\parallel }^2.\hfill \end{array}$$

By Remark 3(iii), the equality (25) is still satisfied. Thus, the above inequality implies the following:

$$\begin{array}{cc}\hfill P_1\left(z^{k+1}\right)\le & P_1\left(x^k\right)+{\displaystyle \frac{1}{{\theta }_k}}\langle \xi -\nabla f\left(x^k\right),{\Delta }_{k+1}\rangle -{\alpha }_k\mathrm{\Psi }(z^{k+1},y^k)-{\displaystyle \frac{1}{{\theta }_k}}({\alpha }_k{L}_g+L_f){\parallel {\Delta }_{k+1}\parallel }^2.\hfill \end{array}$$

Since $P_1\left(x^{k+1}\right)\le P_1\left(x^k\right)+{\theta }_k[P_1\left(z^{k+1}\right)-P_1\left(x^k\right)]$, we can obtain the following:

$$\begin{array}{cc}\hfill P_1\left(x^{k+1}\right)\le & P_1\left(x^k\right)+\langle \xi -\nabla f\left(x^k\right),{\Delta }_{k+1}\rangle -{\theta }_k{\alpha }_k\mathrm{\Psi }(z^{k+1},y^k)-({\alpha }_k{L}_g+L_f){\parallel {\Delta }_{k+1}\parallel }^2.\hfill \end{array}$$

Combining this with (38) and (39) (noting that $x^k-y^k=0$), the above inequality gives

$$\begin{array}{cc}\hfill F\left(x^{k+1}\right)-F\left(x^k\right)\le & -{\theta }_k{\alpha }_k\mathrm{\Psi }(z^{k+1},y^k)-{\displaystyle \frac{1}{2}}(2{\alpha }_k{L}_g+L_f){\parallel {\Delta }_{k+1}\parallel }^2.\hfill \end{array}$$

(61)

Consequently,

$$\begin{array}{cc}\hfill & {\displaystyle \frac{F\left(x^{k+1}\right)-\overline{m}}{{\alpha }_{k+1}}}+\mathrm{\Psi }(x^{k+1},y^k)-\left[{\displaystyle \frac{F\left(x^k\right)-\overline{m}}{{\alpha }_k}}+\mathrm{\Psi }(x^k,y^{k-1})\right]\hfill \\ \hfill \le & {\alpha }_k^{-1}[F\left(x^{k+1}\right)-F\left(x^k\right)]+\mathrm{\Psi }(x^{k+1},y^k)-\mathrm{\Psi }(x^k,y^{k-1})\hfill \\ \hfill \le & \mathrm{\Psi }(x^{k+1},y^k)-{\theta }_k\mathrm{\Psi }(z^{k+1},y^k)-\mathrm{\Psi }(x^k,y^{k-1})-{\displaystyle \frac{1}{2}}(2L_g+{\alpha }_k^{-1}{L}_f){\parallel {\Delta }_{k+1}\parallel }^2\hfill \\ \hfill \le & -{\displaystyle \frac{1}{2}}(2L_g+{\alpha }_k^{-1}{L}_f){\parallel {\Delta }_{k+1}\parallel }^2,\hfill \end{array}$$

(62)

where the first inequality arises from the fact that ${\alpha }_k\le {\alpha }_{k+1}$, the second one follows from (61), and the last one is derived from (42) along with the facts that $\mathrm{\Psi }(x^k,y^k)=0$ and $\mathrm{\Psi }(x^k,y^{k-1})\ge 0$. From (62), we can deduce that   

$$\begin{array}{cc}\hfill H_{k+1}-H_k& \le -{\displaystyle \frac{1}{2}}(2L_g+{\alpha }_k^{-1}{L}_f)\parallel {\Delta }_{k+1}{\parallel }^2+{\displaystyle \frac{L_g}{2}}\parallel x^{k+1}-y^k{\parallel }^2+{\displaystyle \frac{1}{2}}(L_g+{\alpha }_{k+1}^{-1}{L}_f){\parallel {\Delta }_{k+1}\parallel }^2\hfill \\ \hfill & -{\displaystyle \frac{L_g}{2}}\parallel x^k-y^{k-1}{\parallel }^2-{\displaystyle \frac{1}{2}}(L_g+{\alpha }_k^{-1}{L}_f){\parallel {\Delta }_k\parallel }^2\hfill \\ \hfill & ={\displaystyle \frac{1}{2}}({\alpha }_{k+1}^{-1}-{\alpha }_k^{-1})L_f\parallel {\Delta }_{k+1}{\parallel }^2-{\displaystyle \frac{L_g}{2}}\parallel x^k-y^{k-1}{\parallel }^2-{\displaystyle \frac{1}{2}}(L_g+{\alpha }_k^{-1}{L}_f){\parallel {\Delta }_k\parallel }^2\hfill \\ \hfill & \le -{\displaystyle \frac{1}{2}}(L_g+{\alpha }_k^{-1}{L}_f){\parallel {\Delta }_k\parallel }^2\hfill \\ \hfill & \le -{\displaystyle \frac{\delta }{2}}(L_g+{\alpha }_k^{-1}{L}_f){\parallel {\Delta }_k\parallel }^2,\hfill \end{array}$$

where the equality holds because $x^{k+1}-y^k={\Delta }_{k+1}$. This completes the proof of the result in Theorem 1(i).

The remaining parts of the proof are straightforward. Specifically, the other results in Theorems 1–3 follow directly from (32), and their original proofs remain unchanged.    □

Applying Theorem 3 to the sequences generated by Algorithm 2, there exists an integer $K_0$ and a positive number $\widehat{\alpha }$ such that

$${\alpha }_k=\widehat{\alpha },\forall k\ge K_0.$$

(63)

Our discussion regarding the remaining convergence properties of Algorithm 2 will rely on the following auxiliary function:

$$\begin{array}{cc}\hfill H(z,x,y)& :=Q(z,x,y,\widehat{\alpha })+{\delta }_C\left(z\right)\hfill \\ \hfill & ={\widehat{\alpha }}^{-1}\left[F\left(z\right)-\overline{m}\right]+\mathrm{\Psi }(z,y)+{\displaystyle \frac{L_g}{2}}{\parallel z-y\parallel }^2+{\displaystyle \frac{L}{2}}{\parallel z-x\parallel }^2+{\delta }_C\left(z\right),\hfill \end{array}$$

(64)

where $L:=L_g+{\widehat{\alpha }}^{-1}{L}_f$. It is worth noting that for all $k\ge K_0$,

$$H_k=H(x^k,x^{k-1},y^{k-1}),$$

and in light of the setup of Algorithm 2, we have

$$H_k\le {\widehat{H}}_k,$$

(65)

where ${\widehat{H}}_k:=H(z^k,x^{k-1},y^{k-1})$.

Lemma 6.

Suppose that Assumptions 1, 3, and 4 hold, let $\left\{x^k\right\}$, $\left\{y^k\right\}$, and $\left\{z^k\right\}$ be generated by Algorithm 2. Let Λ and Ω denote the sets of accumulation points of $\left\{x^k\right\}$ and $\left\{(z^{k+1},y^k,x^k)\right\}$, respectively. Then the following assertions hold

(i) 

Λ is a nonempty compact set.

(ii) 

$\mathrm{\Omega }=\{(\overline{x},\overline{x},\overline{x}):\overline{x}\in \mathrm{\Lambda }\}$ is also nonempty and compact.

(iii) 

The limit $\omega :={lim}_{k\to \infty }{H}_k$ exists.

(iv) 

If $P_1$ is continuous on Ω , we have $H\equiv \omega$ on Ω , and ${lim}_{k\to \infty }{\widehat{H}}_k=\omega$.

Proof. 

(i)

The nonemptiness and compactness of $\mathrm{\Lambda }$ follows directly from the boundedness of $\left\{\left(x^k\right)\right\}$, as stated in Theorem 1(iii).

(ii)

The representation $\mathrm{\Omega }=\{(\overline{x},\overline{x},\overline{x}):\overline{x}\in \mathrm{\Lambda }\}$ is a consequence of Theorem 1(ii). Consequently, the properties of nonemptiness and compactness are inherited from $\mathrm{\Lambda }$ to $\mathrm{\Omega }$.

(iii)

By Theorem 1(i), the sequence $\left\{H_k\right\}$ is nonincreasing. Furthermore, it follows from the definition of $H_k$ in (33) that $H_k$ is always non-negative. Then $\omega :={lim}_{k\to \infty }{H}_k$ exists.

(iv)

Finally, we assume that $P_1$ is continuous on $\mathrm{\Omega }$, which implies the continuity of H on $\mathrm{\Omega }$. For any $\overline{x}\in \mathrm{\Lambda }$, let $\left\{x^{k_j}\right\}$ be a subsequence converging to $\overline{x}$. By Theorem 1(ii), both $\left\{y^{k_j}\right\}$ and $\left\{x^{k_j+1}\right\}$ also converge to $\overline{x}$. Thus,

$$H(\overline{x},\overline{x},\overline{x})=\underset{j\to \infty }{lim}H(x^{k_j+1},x^{k_j},y^{k_j})=\underset{j\to \infty }{lim}{H}_{k_j}=\omega .$$

(66)

Now, suppose for contradiction that $\left\{{\widehat{H}}_k\right\}$ does not converge to $\omega$. By (65) and (66), there exist a subsequence $\left\{H(z^{k_j+1},x^{k_j},y^{k_j})\right\}$ and a positive number $ϵ$ such that

$$H(z^{k_j+1},x^{k_j},y^{k_j})\ge \omega +ϵ.$$

(67)

By passing to a subsequence if necessary, we may assume that $\left\{x^{k_j}\right\}$ converges to some $\overline{x}\in \mathrm{\Lambda }$. Consequently, both $\left\{y^{k_j}\right\}$ and $\left\{z^{k_j+1}\right\}$ also converge to $\overline{x}$. By the continuity of H, we have

$$\underset{j\to \infty }{lim}H(z^{k_j+1},x^{k_j},y^{k_j})=H(\overline{x},\overline{x},\overline{x})=\omega ,$$

(68)

which contradicts (67). Therefore, ${lim}_{k\to \infty }H(z^{k+1},x^k,y^k)$$=\omega$.

   □

Next, we introduce an assumption to help derive an upper bound for $dist\left(\right(0,0,0)$, $\partial H(z^{k+1},x^k,y^k))$. This assumption is widely adopted in proximal-gradient-type algorithms and is generally satisfied in numerous applications (e.g., the algorithms and problems presented in [19,28,42] and the references therein).

Assumption 5.

Each $g_i$ in (1) is twice continuously differentiable. The function $P_2$ is differentiable with locally Lipschitz continuous gradient on an open set $U_0$ containing $\mathcal{X}$, where $\mathcal{X}$ is the set of critical points of (1).

Lemma 7.

Suppose that Assumptions 1 and 3–5 hold, $P_1$ is continuous, and let $\left\{(z^{k+1},x^k,y^k)\right\}$ be generated by Algorithm 2. Then there exist a positive constant $A_1$ and a positive integer $K_1$ such that for all $k\ge K_1$, we have

$$dist((0,0,0),\partial H(z^{k+1},x^k,y^k))\le A_1(\parallel {\Delta }_{k+1}\parallel +\parallel {\Delta }_k\parallel ).$$

(69)

Proof. 

By Lemma 5 and Theorem 3, $\mathrm{\Lambda }\subset \mathcal{X}(\subset U_0)$. Noticing Lemma 6(i), the local Lipschitz continuity of $\nabla P_2$ ensures the existence of a bounded open neighborhood $U_1$ of $\mathrm{\Lambda }$ such that $\nabla P_2$ is Lipschitz continuous on $U_1$, with the corresponding Lipschitz constant denoted by $L_2$.

Leveraging the boundedness of $\left\{x^k\right\}$ and the limit results ${lim}_{k\to \infty }\parallel z^{k+1}-x^k\parallel =0$ and ${lim}_{k\to \infty }\parallel x^k-y^k\parallel =0$ (stated in Theorem 1(ii, iii)), there exists a positive integer $K^{\prime }$ such that $z^{k+1},x^k,y^k\in U_1$ for all $k\ge K^{\prime }$. Let $K_1:=max\{K_0,K^{\prime }\}$. For any $k\ge K_1$, it follows that ${\alpha }_k=\widehat{\alpha }$; substituting this into Lemma 2(ii) yields the following:

$$\begin{array}{cc}\hfill & 0\in {\widehat{\alpha }}^{-1}\left[\nabla f\left(y^k\right)-\nabla P_2\left(x^k\right)\right]+{\widehat{\alpha }}^{-1}\partial P_1\left(z^{k+1}\right)\hfill \\ \hfill & +\sum _{i\in I_k\left(z^{k+1}\right)}{\lambda }_i^k\nabla g_i\left(y^k\right)+{\theta }_{k}L(z^{k+1}-z^k)+{\mathcal{N}}_C\left(z^{k+1}\right),\hfill \end{array}$$

(70)

which can be rearranged to the following:

$$v_1^k\in V^k,$$

(71)

where

$$\begin{array}{cc}\hfill v_1^k:=& {\widehat{\alpha }}^{-1}\left[\nabla f\left(z^{k+1}\right)-\nabla f\left(y^k\right)\right]-{\widehat{\alpha }}^{-1}\left[\nabla P_2\left(z^{k+1}\right)-\nabla P_2\left(x^k\right)\right]\hfill \\ \hfill & +L_g(z^{k+1}-y^k)+L(z^{k+1}-x^k)-{\theta }_{k}L(z^{k+1}-z^k)\hfill \end{array}$$

(72)

and

$$\begin{array}{cc}\hfill V^k:=& {\widehat{\alpha }}^{-1}\left[\nabla f\left(z^{k+1}\right)+\partial P_1\left(z^{k+1}\right)-\nabla P_2\left(z^{k+1}\right)\right]+{\mathcal{N}}_C\left(z^{k+1}\right)+\sum _{i\in I_k\left(z^{k+1}\right)}{\lambda }_i^k\nabla g_i\left(y^k\right)\hfill \\ \hfill & +L_g(z^{k+1}-y^k)+L(z^{k+1}-x^k).\hfill \end{array}$$

(73)

Next, we analyze the subdifferential $\partial H(z^{k+1},x^k,y^k)$. For simplicity, we decompose the function H into three components:

$$\begin{array}{cc}\hfill & H^a(z,x,y):={\widehat{\alpha }}^{-1}[F\left(z\right)-\overline{m}]+{\delta }_C\left(z\right),\hfill \\ \hfill & H^b(z,x,y):=\mathrm{\Psi }(z,y),\hfill \\ \hfill & H^c(z,x,y):={\displaystyle \frac{L_g}{2}}{\parallel z-y\parallel }^2+{\displaystyle \frac{L}{2}}{\parallel z-x\parallel }^2.\hfill \end{array}$$

(74)

By ([49] Theorem 8.6) and ([49] Corollary 10.9) (respectively), we have

$$\begin{array}{cc}\hfill \partial H(z^{k+1},x^k,y^k)& \supset \widehat{\partial }H(z^{k+1},x^k,y^k)\hfill \\ \hfill & \supset \widehat{\partial }{H}^a(z^{k+1},x^k,y^k)+\widehat{\partial }{H}^b(z^{k+1},x^k,y^k)+\widehat{\partial }{H}^c(z^{k+1},x^k,y^k).\hfill \end{array}$$

(75)

We now compute the regular subdifferentials of these three components:

1. For $H^a$:

$$\begin{array}{cc}\hfill \widehat{\partial }{H}^a(z^{k+1},x^k,y^k)& =\left[\begin{array}{c}\widehat{\partial }[{\widehat{\alpha }}^{-1}F(·)+{\delta }_C(·)]\left(z^{k+1}\right)\\ 0\\ 0\end{array}\right]\supset \left[\begin{array}{c}{\widehat{\alpha }}^{-1}\widehat{\partial }F\left(z^{k+1}\right)+{\mathcal{N}}_C\left(z^{k+1}\right)\\ 0\\ 0\end{array}\right],\hfill \end{array}$$

(76)

where the equality follows from ([49] Proposition 10.5), and the inclusion is derived from ([49] Corollary 10.9, Equation 10(6), Proposition 8.12, Exercise 8.14). Furthermore, by ([49] Exercise 8.8(c), Proposition 8.12), we obtain:

$$\begin{array}{cc}\hfill \widehat{\partial }F\left(z^{k+1}\right)& =\nabla f\left(z^{k+1}\right)+\widehat{\partial }{P}_1\left(z^{k+1}\right)-\nabla P_2\left(z^{k+1}\right)\hfill \\ \hfill & =\nabla f\left(z^{k+1}\right)+\partial P_1\left(z^{k+1}\right)-\nabla P_2\left(z^{k+1}\right).\hfill \end{array}$$

(77)

2. For $H^b$: We have

$$\begin{array}{cc}\hfill \widehat{\partial }{H}^b(z^{k+1},x^k,y^k)& =\partial H^b(z^{k+1},x^k,y^k)\ni \left[\begin{array}{c}\sum _{i\in I_k\left(z^{k+1}\right)}{\lambda }_i^k\nabla g_i\left(y^k\right)\\ 0\\ \sum _{i\in I_k\left(z^{k+1}\right)}{\lambda }_i^k{\nabla }^2{g}_i\left(y^k\right)(z^{k+1}-y^k)\end{array}\right],\hfill \end{array}$$

(78)

where the equality follows from ([49] Example 7.28, Corollary 8.11), and the inclusion is deduced from ([49] Exercise 8.31).

3. For $H^c$: By ([49] Exercise 8.8(a)), we have

$$\widehat{\partial }{H}^c(z^{k+1},x^k,y^k)=\left[\begin{array}{c}{L}_g(z^{k+1}-y^k)+L(z^{k+1}-x^k)\\ -L(z^{k+1}-x^k)\\ -L_g(z^{k+1}-y^k)\end{array}\right].$$

(79)

Combining the relations (75)–(79) gives the following:

$$\left[\begin{array}{c}{v}_1^k\\ v_2^k\\ v_3^k\end{array}\right]\in \left[\begin{array}{c}{V}^k\\ v_2^k\\ v_3^k\end{array}\right]\subset \partial H(z^{k+1},x^k,y^k),$$

(80)

where

$$\begin{array}{cc}\hfill v_2^k& :=-L(z^{k+1}-x^k),\hfill \\ \hfill v_3^k& :=\sum _{i\in I_k\left(z^{k+1}\right)}{\lambda }_i^k{\nabla }^2{g}_i\left(y^k\right)(z^{k+1}-y^k)-L_g(z^{k+1}-y^k).\hfill \end{array}$$

(81)

Thus, from (80), we have

$$dist\left((0,0,0),\partial H(z^{k+1},x^k,y^k)\right)\le \parallel v_1^k\parallel +\parallel v_2^k\parallel +\parallel v_3^k\parallel .$$

(82)

Furthermore, in light of

$$\left\{\begin{array}{c}\parallel v_1^k\parallel \le \left({\widehat{\alpha }}^{-1}{L}_f+L_g\right)\parallel z^{k+1}-y^k\parallel +({\widehat{\alpha }}^{-1}{L}_2+L)\parallel z^{k+1}-x^k\parallel +{\theta }_{k}L\parallel z^{k+1}-z^k\parallel ,\hfill \\ \parallel v_2^k\parallel =L\parallel z^{k+1}-x^k\parallel ,\hfill \\ \parallel v_3^k\parallel \le 2L_g\parallel z^{k+1}-y^k\parallel \hfill \end{array}\right.$$

(83)

(where the last inequality follows from the Lipschitz continuity of $\nabla g_i$ and ([49] Theorem 9.7)), and by virtue of Remark 3, we arrive at Equation (69).

   □

Now we present our global convergence analysis under the Kurdyka-Lojasiewicz (KL) assumption of H.

Theorem 4.

Under the same conditions as in Lemma 7, and assuming that H is a KL function, we have ${\sum }_{k=1}^{\infty }\parallel {\Delta }_k\parallel <+\infty$ and that $\left\{x^k\right\}$ converges globally to a critical point of Problem (1).

Proof. 

Our proof is divided into two cases.

Case 1. There exists an integer $\widehat{K}>0$ such that $H_{\widehat{K}}=\omega$. Since $\left\{H_k\right\}$ converges non-increasingly to $\omega$, it follows that $H_k=\omega$ for all $k\ge \widehat{K}$. Substituting this into Equation (32) further yields ${\Delta }_k=0$ for all such k, which implies the finite convergence of $\left\{x^k\right\}$.

Case 2. $H_k>\omega$ for all k. From Lemma 6, we recall two key properties: $\mathrm{\Omega }$ is a compact set, and $H\equiv \omega$ on $\mathrm{\Omega }$. Given that H is a KL function, Lemma 1 guarantees the existence of $\phi \in {\mathrm{\Theta }}_{\eta }$, $\epsilon >0$, and $\eta >0$ such that

$${\phi }^{\prime }(H(z,x,y)-\omega )dist\left((0,0,0),\partial H(z,x,y)\right)\ge 1$$

(84)

holds for all $(z,x,y)$ satisfying

$$dist\left((z,x,y),\mathrm{\Omega }\right)<\epsilon \mathrm{and}\omega <H(z,x,y)<\omega +\eta .$$

(85)

By Lemma 6(iv), we know that ${lim}_{k\to \infty }{\widehat{H}}_k=\omega$. Additionally, since $\left\{(z^k,x^{k-1},y^{k-1})\right\}$ is a bounded sequence and $\mathrm{\Omega }$ is its accumulation set, there exists an integer $K_2\ge K_1$ such that for all $k\ge K_2$: $dist\left((z^k,x^{k-1},y^{k-1}),\mathrm{\Omega }\right)<\epsilon$, $\omega <{\widehat{H}}_k<\omega +\eta$, and therefore,

$${\phi }^{\prime }({\widehat{H}}_k-\omega )dist\left((0,0,0),\partial H(z^k,x^{k-1},y^{k-1})\right)\ge 1.$$

(86)

For the remainder of this proof, we assume $k\ge K_2$. Combining inequality (86) with Lemma 7 leads to

$${\displaystyle \frac{1}{{\phi }^{\prime }({\widehat{H}}_k-\omega )}}\le A_1(\parallel {\Delta }_{k-1}\parallel +\parallel {\Delta }_k\parallel ),\forall k\ge K_2.$$

(87)

On the other hand, leveraging the mean value theorem, the decreasing property of ${\phi }^{\prime }$ (a direct consequence of $\phi$ being concave), and the relations $H_{k+1}\le H_k\le {\widehat{H}}_k$ (from Equations (32) and (65)), we further derive that

$$\phi (H_k-\omega )-\phi (H_{k+1}-\omega )\ge {\phi }^{\prime }({\widehat{H}}_k-\omega )(H_k-H_{k+1}).$$

(88)

Define ${\nu }_{k,t}:=\phi (H_k-\omega )-\phi (H_t-\omega )$. Using Equations (32), (88), and (87), respectively, we get

$$\begin{array}{c}\hfill {\displaystyle \frac{\delta L}{2}}\parallel {\Delta }_k{\parallel }^2\le H_k-H_{k+1}\le {\displaystyle \frac{{\nu }_{k,k+1}}{{\phi }^{\prime }({\widehat{H}}_k-\omega )}}\le A_1(\parallel {\Delta }_{k-1}\parallel +\parallel {\Delta }_k\parallel ){\nu }_{k,k+1}.\end{array}$$

(89)

Since ${\phi }^{\prime }>0$ and $H_k\ge H_{k+1}$, it follows that ${\nu }_{k,k+1}\ge 0$. Applying the AM-GM inequality $\sqrt{ab}\le {\displaystyle \frac{a+b}{2}}$ to the result above, we obtain

$$\begin{array}{c}\hfill \parallel {\Delta }_k\parallel \le \sqrt{{\displaystyle \frac{1}{2}}(\parallel {\Delta }_{k-1}\parallel +\parallel {\Delta }_k\parallel )}\sqrt{{\displaystyle \frac{4A_1}{\delta L}}{\nu }_{k,k+1}}\le {\displaystyle \frac{1}{4}}(\parallel {\Delta }_{k-1}\parallel +\parallel {\Delta }_k\parallel )+{\displaystyle \frac{2A_1}{\delta L}}{\nu }_{k,k+1},\end{array}$$

(90)

this yields

$$\parallel {\Delta }_k\parallel \le {\displaystyle \frac{1}{2}}(\parallel {\Delta }_{k-1}\parallel -\parallel {\Delta }_k\parallel )+{\displaystyle \frac{4A_1}{\delta L}}{\nu }_{k,k+1}.$$

(91)

Summing inequality (91) over $k=K_2,K_2+1,\dots ,t$ (for any $t\ge K_2$) gives

$$\begin{array}{c}\hfill \sum _{k=K_2}^t\parallel {\Delta }_k\parallel \le {\displaystyle \frac{1}{2}}(\parallel {\Delta }_{K_2-1}\parallel -\parallel {\Delta }_t\parallel )+{\displaystyle \frac{4A_1}{\delta L}}{\nu }_{K_2,t+1}\le {\displaystyle \frac{1}{2}}\parallel {\Delta }_{K_2-1}\parallel +{\displaystyle \frac{4A_1}{\delta L}}\phi (H_{K_2}-\omega ),\end{array}$$

(92)

where the final inequality holds because $\phi \ge 0$. This result implies ${\sum }_{k=0}^{\infty }\parallel {\Delta }_k\parallel <+\infty$, which in turn shows that $\left\{x^k\right\}$ is a Cauchy sequence and hence converges to some $\overline{x}$. By Theorem 3, $\overline{x}$ is a critical point of Problem (1).    □

Lemma 8

([48] Lemma 10). Let ${\left\{{\mathrm{\Lambda }}_k\right\}}_{k\in \mathbb{N}}$ be a nonincreasing sequence in ${\mathbb{R}}_{+}$ converging to 0 and there exists $\overline{k}\ge \overline{l}\ge 0$ such as for all $k\ge \overline{k}$, ${\mathrm{\Lambda }}_k^{2a}\le m({\mathrm{\Lambda }}_{k-\overline{l}}-{\mathrm{\Lambda }}_k)$, where m is a nonnegative constant and $a\in [0,1)$. Then, the following statements hold.

(i) 

If $a=0$, then $\left\{{\mathrm{\Lambda }}_k\right\}$ converges finite time.

(ii) 

If $a\in (0,{\displaystyle \frac{1}{2}}]$, there exists ${\mu }_1>0$ and $\tau \in [0,1)$ such that for all $k\ge \overline{k}$, ${\mathrm{\Lambda }}_k\le {\mu }_1{\tau }^k$.

(iii) 

If $a\in ({\displaystyle \frac{1}{2}},1)$, there exists ${\mu }_2>0$ such that for all $k\ge \overline{k}+\overline{l}$, ${\mathrm{\Lambda }}_k\le {\mu }_2{(k-\overline{l}+1)}^{-{\displaystyle \frac{1}{2a-1}}}$.

Theorem 5.

Under the same conditions as in Lemma 7, suppose further that H is a KL function, where the function φ in the KL inequality takes the form $\phi \left(s\right)=\rho s^{1-\beta }$ for some constants $\beta \in [0,1)$ and $\rho >0$. Let $\left\{x^k\right\}$ be the sequence generated by Algorithm 2, and let $\overline{x}$ be its limit. Then the following assertions hold:

(i) 

If $\beta =0$, then Algorithm 2 terminates after finitely many iterations.

(ii) 

If $\beta \in (0,{\displaystyle \frac{1}{2}}]$, there exists constants $c_1>0$ and $\tau \in [0,1)$ such that $\parallel x^k-\overline{x}\parallel \le c_1{\tau }^k$.

(iii) 

If $\beta \in ({\displaystyle \frac{1}{2}},1)$, there exists constants $c_2>0$ and $\tau \in [0,1)$ such that $\parallel x^k-\overline{x}\parallel \le c_2{k}^{-{\displaystyle \frac{1-\beta }{2\beta -1}}}$.

Proof. 

For the case $\beta =0$, we have $\phi \left(s\right)=\rho s$, ${\phi }^{\prime }\left(s\right)=\rho$. We aim to show that $H_k=\omega$ for for sufficiently large k, and consequently, (i) holds by virtue of Case 1 in the proof of Theorem 4. Suppose, for contradiction, that $H_k>\omega$ for all k. Then, by (65), ${\widehat{H}}_k>\omega$. Inequality (87) yields

$${\displaystyle \frac{1}{\rho }}\le A_1(\parallel {\Delta }_{k-1}\parallel +\parallel {\Delta }_k\parallel )$$

(93)

which contradicts the fact that ${lim}_{k\to \infty }\parallel {\Delta }_k\parallel =0$ from Theorem 1(ii).

Next, consider the case where $\beta \in (0,1)$. For the remainder of this proof, we assume $k\ge K_2$, where $K_2$ is defined in the proof of Theorem 4. First, utilizing the inequality (87) and the expression ${\phi }^{\prime }\left(s\right)=\rho (1-\beta )s^{-\beta }$, we derive

$$\begin{array}{c}\hfill {(H_k-\omega )}^{\beta }\le {({\widehat{H}}_k-\omega )}^{\beta }\le \rho (1-\beta )A_1(\parallel {\Delta }_{k-1}\parallel +\parallel {\Delta }_k\parallel ).\end{array}$$

(94)

Raising the above inequality to the power of ${\displaystyle \frac{1-\beta }{\beta }}>0$ and noting that $\phi \left(s\right)=\rho s^{1-\beta }$, we obtain

$$\begin{array}{c}\hfill \phi (H_k-\omega )=\rho {(H_k-\omega )}^{1-\beta }\le \rho {\left[\rho (1-\beta )A_1\right]}^{{\displaystyle \frac{1-\beta }{\beta }}}(\parallel {\Delta }_{k-1}\parallel +\parallel {\Delta }_k{\parallel )}^{{\displaystyle \frac{1-\beta }{\beta }}}.\end{array}$$

(95)

Define a nonincreasing sequence $R_k:={\sum }_{i=k+1}^{\infty }\parallel {\Delta }_i\parallel$. By Theorem 4, each $R_k$ is finite. It follows that

$$\begin{array}{cc}\hfill R_{k+1}\le R_k& =\sum _{i=k+1}^{\infty }\parallel {\Delta }_i\parallel \hfill \\ \hfill & \stackrel{\left(a\right)}{\le }{\displaystyle \frac{1}{2}}\parallel {\Delta }_k\parallel +{\displaystyle \frac{4A_1}{\delta L}}\phi (H_{k+1}-\omega )\hfill \\ \hfill & \stackrel{\left(b\right)}{\le }{\displaystyle \frac{1}{2}}\parallel {\Delta }_k\parallel +A_2(\parallel {\Delta }_k+\parallel {\Delta }_{k+1}{\parallel \parallel )}^{{\displaystyle \frac{1-\beta }{\beta }}}\hfill \\ \hfill & ={\displaystyle \frac{1}{2}}(R_{k-1}-R_k)+A_2{(R_{k-1}-R_{k+1})}^{{\displaystyle \frac{1-\beta }{\beta }}},\hfill \end{array}$$

(96)

where $A_2={\displaystyle \frac{4A_1}{\delta L}}\rho {\left[\rho (1-\beta )A_1\right]}^{{\displaystyle \frac{1-\beta }{\beta }}}$, the inequality $\left(a\right)$ is derived analogously to (92), and inequality $\left(b\right)$ follows from (95). With (96) established, we now prove (ii) and (iii) separately.

(ii) For $\beta \in (0,{\displaystyle \frac{1}{2}}]$, we have ${\displaystyle \frac{1-\beta }{\beta }}\ge 1$. Recalling that ${lim}_{k\to \infty }\parallel {\Delta }_k\parallel =0$, we observe that $R_{k-1}-R_{k+1}\to 0$ as $k\to \infty$. Thus, there exists $K_3>K_2$ such that $R_{k-1}-R_{k+1}<1$ for all $k\ge K_3$. Hence

$$R_{k+1}\le (A_2+{\displaystyle \frac{1}{2}})(R_{k-1}-R_{k+1}),\forall k\ge K_3.$$

(97)

Applying Lemma 8(ii) to $\left\{R_k\right\}$ with $a={\displaystyle \frac{1}{2}}$ and $\overline{l}=2$, we conclude that (ii) holds.

(iii) For $\beta \in ({\displaystyle \frac{1}{2}},1)$, we have ${\displaystyle \frac{1-\beta }{\beta }}<1$. Similarly, there exists $K_4>K_2$ such that

$$R_{k+1}\le (A_2+{\displaystyle \frac{1}{2}}){(R_{k-1}-R_{k+1})}^{{\displaystyle \frac{1-\beta }{\beta }}},\forall k\ge K_4.$$

(98)

Applying Lemma 8(iii) to $\left\{R_k\right\}$ with $a={\displaystyle \frac{\beta }{2(1-\beta )}}$ and $\overline{l}=2$, we establish (iii) and complete the proof.    □

From the preceding results (Lemma 5 to Theorem 4), we observe the efficacy of the restart technique in the convergence analysis of the ${\mathrm{EAPG}}_s$ method. However, the restart criterion given by (59) merely serves as a sufficient condition for $z^{k+1}$ to outperform $x^{k+1}$, and this condition is overly restrictive, making it difficult to satisfy. In fact, when implementing Algorithm 2 on most problems presented in Section 6, at most one restart occurred. In experiments where no restart was triggered, Algorithm 2 was entirely identical to Algorithm 1, with the sole exception that we explicitly ensured condition (65)—along with the subsequent theoretical results—held for Algorithm 2. Thus, Algorithm 2 should be regarded more as a theoretical construct than a practical implementation.

Practically speaking, however, the restart technique can indeed enhance the efficiency of our algorithms, as demonstrated in Section 6. This indicates the necessity of developing a practical criterion to replace (59), thereby maximizing opportunities for effective restarts. Drawing inspiration from the fixed restarting algorithm ([33] Algorithm 3), the $d_k$ criterion (discussed following Equation (75) in [26]), and the inner product criterion in [19], we define

$$d_k={\displaystyle \frac{{\alpha }_k^{-1}\left[F\left(x^k\right)-F\left(x^{k+1}\right)\right]+G_k-G_{k+1}}{\parallel {\Delta }_k{\parallel }^2}},$$

(99)

where

$$G_k=\mathrm{\Psi }(x^k,y^{k-1})+{\displaystyle \frac{L_g}{2}}\parallel x^k-y^{k-1}{\parallel }^2+{\displaystyle \frac{L_g+{\alpha }_k^{-1}{L}_f}{2}}{\parallel {\Delta }_k\parallel }^2,$$

(100)

and propose the following algorithm (Algorithm 3):

Algorithm 3 (${\mathrm{EAPG}}_{sr}$) ${\mathrm{EAPG}}_s$ with Restart Technique (Practical Version)

Initialization: Given $x^0,z^0\in C$, ${\alpha }_0>0$, $d>0$, a positive integer $N_0$, and $\left\{{\theta }_k\right\}$ as defined in Remark A1(ii). Determining the restart interval: Execute Algorithm 1 with initial values $x^0,z^0,{\alpha }_0$ and parameters $\{{\theta }_0,{\theta }_1,\cdots \}$ for N steps, where N denotes the first step k satisfying $k\ge N_0$ and $d_k>d_{k-1}$. Set $x^0=z^0=z^N$ and ${\alpha }_0={\alpha }_N$. for $s=1,2,\dots$ do Execute Algorithm 1 with initial values $x^0,z^0,{\alpha }_0$ and parameters $\{{\theta }_0,{\theta }_1,\cdots \}$ until either $\langle y^{k-1}-z^k,z^k-z^{k-1}\rangle >0$ or $k=N$. Set $x^0=z^0=z^k$ and ${\alpha }_0={\alpha }_k$. end for

6. Numerical Experiments

In this section, we conduct numerical experiments to assess the performance of Algorithm 3. The design of these experiments is intended to demonstrate three key aspects, as elaborated below:

1.

In Section 6.1 and Section 6.2, we verify the computational efficiency of Algorithm 3 against the IPOPT solver [50,51] and three state-of-the-art methods—namely ${\mathrm{SCP}}_{\mathrm{ls}}$ [28], ${\mathrm{ESQM}}_{\mathrm{b}}$ (a basic variant of ESQM derived by fixing ${\beta }_k\equiv 0$ in ${\mathrm{ESQM}}_{\mathrm{e}}$), and ${\mathrm{ESQM}}_{\mathrm{e}}$ [19]—when solving the optimization problem formulated in (3).

2.

In Section 6.3, we evaluate three key metrics: the effectiveness of Algorithm 3’s $d_k$-criterion for optimal restart interval identification, Algorithm 3’s efficiency versus Algorithms 1 and 2, and its overall performance relative to multiple modified variants.

3.

In Section 6.4, we validate the efficacy of Algorithm 3 on the unconstrained DC problem specified in (2), with comparisons drawn to the IPOPT solver and three established algorithms for unconstrained DC problems: GIST [52], pDCAe [53], and ${\mathrm{APG}}_s$ [26] (the foundational prototype of Algorithm 3).

From Section 6.1 and Section 6.2, the numerical experiments focus on the following compressed sensing optimization problem:

$$\begin{array}{cc}\hfill \underset{x\in {\mathbb{R}}^n}{min}& {\parallel x\parallel }_1-\mu \parallel x\parallel \hfill \\ \hfill \mathrm{s}.\mathrm{t}.& h(Ax-b)\le \sigma ,\hfill \\ \hfill & {\parallel x\parallel }_{\infty }\le M,\hfill \end{array}$$

(101)

where

–

$\mu \in [0,1)$;

–

$A\in {\mathbb{R}}^{q\times n}$ has full row rank;

–

$b\in {\mathbb{R}}^q$;

–

$M={(1-\mu )}^{-1}\left(\parallel A^{†}{b\parallel }_1-\mu \parallel A^{†}b\parallel \right)$;

–

$h:{\mathbb{R}}^q\to {\mathbb{R}}_{+}$ is an analytic function with Lipschitz-continuous gradient (modulus $L_h$), $h\left(0\right)=0$, and $\sigma \in (0,h(-b\left)\right)$.

This problem is equivalent to the following model:

$$\begin{array}{cc}\hfill \underset{x\in {\mathbb{R}}^n}{min}& {\parallel x\parallel }_1-\mu {\parallel x\parallel }_2\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& h(Ax-b)\le \sigma ,\hfill \end{array}$$

(102)

initially introduced in [54] and further explored in [17,55] for sparse signal recovery.

Problem (101) is a special case of (3) with $P_1\left(x\right)={\parallel x\parallel }_1$, $P_2\left(x\right)=\mu \parallel x\parallel$, $m=1$, $g_1\left(x\right)=h(Ax-b)-\sigma$, and $C=\{x:\parallel x{\parallel }_{\infty }\le M\}$. Additionally, since A has full row rank and $h\left(0\right)=0<\sigma$, we have $A^{†}b\in C\cap \{x:g_1\left(x\right)<0\}\ne \varnothing$. It is straightforward to verify that Assumption 1 holds for this problem.

In the following subsections, we conduct experiments on Problem (101) with different selections of h. All numerical experiments were performed on a computer with an Intel(R) Core(TM) i5-8265U processor and 8.00 GB of memory, running the Windows 10 operating system. The experiments were implemented using MATLAB R2021a.

6.1. Compressed Sensing with $h(·)={\displaystyle \frac{1}{2}}{\parallel ·\parallel }^2$

We first consider the Problem (101) with $h(·)={\displaystyle \frac{1}{2}}{\parallel ·\parallel }^2$, which transforms (101) into:

$$\begin{array}{cc}\hfill \underset{x\in {\mathbb{R}}^n}{min}& {\parallel x\parallel }_1-\mu \parallel x\parallel \hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\displaystyle \frac{1}{2}}{\parallel Ax-b\parallel }^2\le \sigma ,\hfill \\ \hfill & {\parallel x\parallel }_{\infty }\le M.\hfill \end{array}$$

(103)

Note that h is convex, so $g_1$ is also convex. Thus, we can set $L_g={\parallel A\parallel }^2$ and $l_g=0$. Furthermore, since $A^{†}b\in C\cap \{x:g_1\left(x\right)<0\}$, the Slater condition holds for the above problem. Based on the discussion following Definition 3 and Remark 2, Assumption 3 is satisfied.

Since the function H in (64) (corresponding to Problem (103)) is clearly semi-algebraic—and therefore a KL function—we can apply Theorem 5 to deduce the convergence of the entire sequence $\left\{x^k\right\}$ generated by Algorithm 3 for solving (103).

• Details of the Five Algorithms

We detail the setup of the five algorithms below:

(i)

Initialization and Stopping Criteria: For ${\mathrm{SCP}}_{\mathrm{ls}}$, ${\mathrm{ESQM}}_b$, and ${\mathrm{ESQM}}_e$, we adopt the same initial points as specified in [19]: specifically, $x^0=A^{†}b$ for ${\mathrm{SCP}}_{\mathrm{ls}}$, and $x^0=0$ for both ${\mathrm{ESQM}}_b$ and ${\mathrm{ESQM}}_e$. For ${\mathrm{EAPG}}_{\mathrm{sr}}$, the initial point is set to $x^0=z^0=0$. For IPOPT, we introduce slack variables u and v to reformulate Problem (103) as follows:

$$\begin{array}{cc}\hfill \underset{u,v\in {\mathbb{R}}^n}{min}& {\mathbf{1}}^{\top }u+{\mathbf{1}}^{\top }v-\mu \parallel u-v\parallel \hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\displaystyle \frac{1}{2}}{\parallel A(u-v)-b\parallel }^2\le \sigma ,\hfill \\ \hfill & u,v\ge 0,\hfill \end{array}$$

(104)

where $\mathbf{1}$ denotes the all-ones vector (i.e., a vector with each component equal to 1). The corresponding initial points are set to $u^0={\left[A^{†}b\right]}_{+}+0.001\mathbf{1}$ and $v^0={[-A^{†}b]}_{+}+0.001\mathbf{1}$.

All algorithms except IPOPT terminate when either the relative iterate difference satisfies ${\displaystyle \frac{\parallel x^{k+1}-x^k\parallel }{max\{1,\parallel x^{k+1}\parallel \}}}\le ϵ$ (with $ϵ>0$ to be specified in subsequent sections) or the maximum number of iterations (3000) is reached. For IPOPT, the convergence tolerance is configured to the same value of $ϵ$ and the maximum number of iterations is set to 1000.

(ii)

Parameter Settings: The parameters for ${\mathrm{SCP}}_{\mathrm{ls}}$ follow [28], while those for ${\mathrm{ESQM}}_b$ and ${\mathrm{ESQM}}_e$ follow [19]. For ${\mathrm{EAPG}}_{\mathrm{sr}}$, we set ${\alpha }_0=1$ and $d=1$ (consistent with [19]) and $N_0=20$. Since $g_1$ is convex (implying $l_g=0$), any positive integer K is valid for the acceleration parameters $\left\{{\theta }_k\right\}$ defined in Remark 1(ii); here, we set $K=150$. Notably, in practice, the restart period N observed in experiments was consistently less than 100. Thus, the experimental performance would remain unchanged if we select any $K\ge 100$.

The subproblems of these algorithms are solved following the procedures outlined in the appendices of [28,45]. In each subproblem, the computational complexity of evaluating $\nabla g_1$ and ${\zeta }^k$ is $O\left(qn\right)$ and $O\left(n\right)$, respectively. Additionally, solving the optimization problem (15) incurs a complexity of $O\left(n\right)$. Consequently, the overall computational complexity of each subproblem is $O\left(qn\right)$.

All settings of IPOPT are set to their default values except for the tolerance parameter and maximum number of iterations.

• Experimental Setup for Random Instances

We tested Algorithm 3 on random instances of Problem (103), generated as follows:

1.

Generate $A\in {\mathbb{R}}^{q\times n}$ with independent and identically distributed (i.i.d.) standard Gaussian entries, then normalize A such that each column has unit norm.

2.

Randomly select a subset $T\subseteq \{1,2,\cdots ,n\}$ of size p, and generate a p-sparse vector $x_{\mathrm{orig}}$ with i.i.d. standard Gaussian entries on T.

3.

Set $b=A{x}_{\mathrm{orig}}+0.01·\widehat{n}$, where $\widehat{n}$ is a random vector with i.i.d. standard Gaussian entries; set $\sigma =0.5{\sigma }_1^2$, where ${\sigma }_1=1.1·\parallel 0.01·\widehat{n}\parallel$.

• Experimental Parameters and Result Metrics

In our numerical tests:

–

We set $\mu =0.99$ in Problem (105).

–

We considered parameter triples $(q,n,p)=(720i,2560i,$$160i)$ for $i\in \{2,4,6,8,10\}$.

–

For each i, 20 random instances were generated (as above), and results were averaged over these 20 instances.

Computational results for $ϵ={10}^{-4}$ and $ϵ={10}^{-6}$ are presented in

Table 1. Computational results for Problem (103) with $ϵ={10}^{-4}$.

	Method	i = 2	i = 4	i = 6	i = 8	i = 10
Time	${\mathrm{t}}_{QR}$	0.818	5.449	18.567	43.191	91.172
(sec)	${\mathrm{t}}_{A^{†}b}$	0.008	0.034	0.081	0.149	0.264
	${\mathrm{t}}_{\parallel A\parallel }$	1.321	2.062	6.486	13.789	27.855
	IPOPT	4.141	12.439	30.015	43.383	88.684
	${\mathrm{SCP}}_{ls}$	3.804	15.112	31.714	56.368	92.386
	${\mathrm{ESQM}}_b$	15.321	65.436	143.914	262.850	427.229
	${\mathrm{ESQM}}_e$	0.976	4.133	9.150	16.533	26.966
	${\mathrm{EAPG}}_{sr}$	0.986	3.963	8.812	16.132	25.874
Iter	IPOPT	108	134	146	162	180
	${\mathrm{SCP}}_{ls}$	212	222	217	218	225
	${\mathrm{ESQM}}_b$	1705	1777	1786	1793	1816
	${\mathrm{ESQM}}_e$	108	112	113	112	113
	${\mathrm{EAPG}}_{sr}$	101	102	104	103	104
RecErr	IPOPT	0.048687	0.051378	0.050837	0.051987	0.052050
	${\mathrm{SCP}}_{ls}$	0.049113	0.051792	0.051330	0.052505	0.052568
	${\mathrm{ESQM}}_b$	0.066395	0.071510	0.071093	0.072476	0.072915
	${\mathrm{ESQM}}_e$	0.048744	0.051471	0.050940	0.052131	0.052169
	${\mathrm{EAPG}}_{sr}$	0.049920	0.053142	0.052955	0.053912	0.053758
Residual	IPOPT	$1.99\times {10}^{-6}$	$1.91\times {10}^{-6}$	$1.97\times {10}^{-6}$	$1.98\times {10}^{-6}$	$2.10\times {10}^{-6}$
	${\mathrm{SCP}}_{ls}$	$-5.69\times {10}^{-7}$	$-5.62\times {10}^{-7}$	$-6.09\times {10}^{-7}$	$-5.63\times {10}^{-7}$	$-6.93\times {10}^{-7}$
	${\mathrm{ESQM}}_b$	$6.79\times {10}^{-7}$	$5.73\times {10}^{-7}$	$5.79\times {10}^{-7}$	$5.63\times {10}^{-7}$	$5.49\times {10}^{-7}$
	${\mathrm{ESQM}}_e$	$1.22\times {10}^{-7}$	$1.17\times {10}^{-7}$	$1.03\times {10}^{-7}$	$1.08\times {10}^{-7}$	$1.03\times {10}^{-7}$
	${\mathrm{EAPG}}_{sr}$	$4.82\times {10}^{-7}$	$6.22\times {10}^{-7}$	$5.90\times {10}^{-7}$	$6.08\times {10}^{-7}$	$6.12\times {10}^{-7}$

and

Table 2. Computational results for Problem (103) with $ϵ={10}^{-6}$.

	Method	i = 2	i = 4	i = 6	i = 8	i = 10
Time	${\mathrm{t}}_{QR}$	0.883	5.840	18.999	43.567	90.874
(sec)	${\mathrm{t}}_{A^{†}b}$	0.009	0.035	0.078	0.149	0.264
	${\mathrm{t}}_{\parallel A\parallel }$	1.365	2.199	6.517	13.888	27.822
	IPOPT	4.781	13.733	29.939	52.517	100.201
	${\mathrm{SCP}}_{ls}$	4.388	17.595	36.548	65.808	107.272
	${\mathrm{ESQM}}_b$	23.894	103.784	232.600	423.241	690.270
	${\mathrm{ESQM}}_e$	1.679	7.989	18.182	34.184	57.380
	${\mathrm{EAPG}}_{sr}$	1.584	7.392	14.545	27.413	46.253
Iter	IPOPT	127	152	165	181	200
	${\mathrm{SCP}}_{ls}$	242	256	251	254	260
	${\mathrm{ESQM}}_b$	2683	2792	2867	2866	2902
	${\mathrm{ESQM}}_e$	187	214	223	231	240
	${\mathrm{EAPG}}_{sr}$	161	168	168	171	169
RecErr	IPOPT	0.048687	0.051378	0.050836	0.051986	0.052049
	${\mathrm{SCP}}_{ls}$	0.048695	0.051389	0.050848	0.051997	0.052060
	${\mathrm{ESQM}}_b$	0.048820	0.051618	0.050987	0.052167	0.052220
	${\mathrm{ESQM}}_e$	0.048689	0.051379	0.050840	0.051985	0.052049
	${\mathrm{EAPG}}_{sr}$	0.048708	0.051406	0.050868	0.052010	0.052074
Residual	IPOPT	$1.77\times {10}^{-6}$	$1.74\times {10}^{-6}$	$1.73\times {10}^{-6}$	$1.73\times {10}^{-6}$	$1.75\times {10}^{-6}$
	${\mathrm{SCP}}_{ls}$	$-5.69\times {10}^{-10}$	$-5.61\times {10}^{-10}$	$-6.08\times {10}^{-10}$	$-5.52\times {10}^{-10}$	$-6.78\times {10}^{-10}$
	${\mathrm{ESQM}}_b$	$1.01\times {10}^{-10}$	$3.07\times {10}^{-10}$	$1.05\times {10}^{-10}$	$1.36\times {10}^{-10}$	$1.30\times {10}^{-10}$
	${\mathrm{ESQM}}_e$	$5.83\times {10}^{-11}$	$2.06\times {10}^{-11}$	$3.19\times {10}^{-11}$	$2.43\times {10}^{-11}$	$-1.96\times {10}^{-11}$
	${\mathrm{EAPG}}_{sr}$	$3.63\times {10}^{-11}$	$-2.59\times {10}^{-12}$	$9.27\times {10}^{-11}$	$6.51\times {10}^{-11}$	$3.74\times {10}^{-11}$

, respectively. The metrics reported include:

–

$t_{\mathrm{QR}}$: Time to compute the QR decomposition of $A^T$.

–

$t_{\parallel A\parallel }$: Time to compute ${\parallel A\parallel }^2$.

–

$t_{A^{†}b}$: Time to compute $x^0=A^{†}b$ using the QR factorization of $A^T$.

–

CPU time of each algorithm.

–

Iter: Number of iterations.

–

RecErr $:={\displaystyle \frac{\parallel x^{*}-x_{\mathrm{orig}}\parallel }{max\{1,\parallel x_{\mathrm{orig}}\parallel \}}}$: Recovery error (where $x^{*}$ is the approximate solution from the algorithm).

–

Residual $:={\displaystyle \frac{\parallel A{x}^{*}{-b\parallel }^2-{\sigma }_1^2}{{\sigma }_1^2}}$: Residual of the constraint violation.

• Key Observations from Results

From Table 1 and Table 2, we observe two main results:

1.

${\mathrm{EAPG}}_{\mathrm{sr}}$ achieves the fastest computation speed among the five algorithms.

2.

The recovery errors (RecErr) and residuals of all five methods are comparable.

6.2. Compressed Sensing with Lorentzian Norm

Next, we consider Problem (101) with Lorentzian norm, which transforms (101) into

$$\begin{array}{cc}\hfill \underset{x\in {\mathbb{R}}^n}{min}& {\parallel x\parallel }_1-\mu \parallel x\parallel \hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\parallel Ax-b\parallel }_{LL2,\gamma }\le \sigma ,\hfill \\ \hfill & {\parallel x\parallel }_{\infty }\le M.\hfill \end{array}$$

(105)

The Lorentzian norm is defined as follows [56]:

$${\parallel y\parallel }_{LL2,\gamma }:=\sum _{i=1}^{q}log\left(1+{\displaystyle \frac{y_i^2}{{\gamma }^2}}\right),$$

(106)

where $\gamma >0$.

As proven in ([19] Subsection 6.2), Assumption 3 holds.

• Details of the Five Algorithms

The setup of the five algorithms is detailed below:

(i)

Initialization and Stopping Criteria: All algorithms use the same initialization and stopping criteria as in Section 6.1.

(ii)

Parameter Settings:

–

For ${\mathrm{SCP}}_{\mathrm{ls}}$, parameters follow the settings in [28].

–

For the other three algorithms, we set $L_g={\displaystyle \frac{{2\parallel A\parallel }^2}{{\gamma }^2}}$, $l_g={\displaystyle \frac{{\parallel A\parallel }^2}{4{\gamma }^2}}$, ${\alpha }_0=1.1\gamma$, $d={\displaystyle \frac{{\gamma }^2}{{150\parallel A\parallel }^2}}$—consistent with [19].

–

For the acceleration parameters $\left\{{\theta }_k\right\}$ of ${\mathrm{EAPG}}_{\mathrm{sr}}$, since ${\displaystyle \frac{L_g}{L_g+l_g}}={\displaystyle \frac{8}{9}}$, it is straightforward to verify that $K\le 31$. However, in practice, because all iterates $\left\{x^k\right\}$ lie within a bounded local region, the theoretical results may remain valid for larger values of K. As the next subsection will demonstrate, we can consistently use a large K and adaptively determine the restart period N (where $N<K$) to enhance the performance of Algorithm 3. In fact, the restart period N observed in experiments was consistently less than 100. Thus, selecting any $K\ge 100$ would ensure both consistent and improved experimental performance. For the lower bound of N, we also set $N_0=20$, consistent with the setting in Section 6.1.

The subproblems of these algorithms are solved following the procedures outlined in the appendices of [28,45]. Consistent with the experimental results presented in Section 6.1, the overall computational complexity of each subproblem is also $O\left(qn\right)$.

• Experimental Setup for Random Instances

Random instances are generated as follows:

1.

Generate A, subset T, size p, and sparse vector $x_{\mathrm{orig}}$ using the same method as in Section 6.1.

2.

Set $b=A{x}_{\mathrm{orig}}+0.01·\tilde{n}$, where ${\tilde{n}}_i\sim \mathrm{Cauchy}(0,1)$. Specifically, ${\tilde{n}}_i$ is generated as $tan\left(\pi \left({\widehat{n}}_i-{\displaystyle \frac{1}{2}}\right)\right)$, with $\widehat{n}$ being a random vector with i.i.d. entries uniformly sampled from $[0,1]$ (note: corrected the ambiguous “with $\tilde{n}$ being” to “with $\widehat{n}$ being” to avoid variable confusion).

3.

Set $\sigma =1.05·\parallel 0.01·\tilde{n}{\parallel }_{LL2,\gamma }$ with $\gamma =0.055$.

• Experimental Parameters and Result Metrics

In the numerical tests:

–

We set $\mu =0.99$ in Problem (110).

–

We considered parameter triples $(q,n,p)=(720i,2560i,80i)$ for $i\in \{2,4,6,8,10\}$.

–

For each i, 20 random instances were generated, and results were averaged over these instances (consistent with Section 6.1).

Computational results for $ϵ={10}^{-4}$ and $ϵ={10}^{-6}$ are presented in

Table 3. Computational results for Problem (105) with $ϵ={10}^{-4}$.

	Method	i = 2	i = 4	i = 6	i = 8	i = 10
Time (sec)	${\mathrm{t}}_{QR}$	0.763	5.485	18.964	46.317	90.568
	${\mathrm{t}}_{A^{†}b}$	0.008	0.038	0.092	0.167	0.269
	${\mathrm{t}}_{\parallel A\parallel }$	1.280	2.201	6.614	14.669	27.917
	IPOPT	13.380	25.354	31.578	50.119	122.681
	${\mathrm{SCP}}_{ls}$	2.861	11.071	15.250	51.427	67.749
	${\mathrm{ESQM}}_b$	9.879	41.459	89.737	180.831	279.228
	${\mathrm{ESQM}}_e$	1.803	7.540	16.394	32.972	50.844
	${\mathrm{EAPG}}_{sr}$	1.541	6.269	13.451	26.205	42.090
Iter	IPOPT	334	268	162	151	191
	${\mathrm{SCP}}_{ls}$	183	181	117	207	180
	${\mathrm{ESQM}}_b$	1136	1149	1146	1195	1163
	${\mathrm{ESQM}}_e$	204	207	208	217	209
	${\mathrm{EAPG}}_{sr}$	170	169	165	167	170
RecErr	IPOPT	0.762756	3051.699141	0.084689	0.084819	15.288138
	${\mathrm{SCP}}_{ls}$	0.081013	0.081612	0.084733	0.083314	0.086727
	${\mathrm{ESQM}}_b$	0.086207	0.087180	0.090687	0.089185	0.092889
	${\mathrm{ESQM}}_e$	0.080836	0.081385	0.084550	0.083104	0.086500
	${\mathrm{EAPG}}_{sr}$	0.081517	0.081882	0.085460	0.083959	0.087379
Residual	IPOPT	$1.21\times {10}^0$	$2.03\times {10}^0$	$2.22\times {10}^{-7}$	$2.03\times {10}^{-7}$	$1.96\times {10}^0$
	${\mathrm{SCP}}_{ls}$	$-4.88\times {10}^{-8}$	$-4.03\times {10}^{-8}$	$-6.11\times {10}^{-8}$	$-3.70\times {10}^{-8}$	$-4.19\times {10}^{-8}$
	${\mathrm{ESQM}}_b$	$9.40\times {10}^{-8}$	$9.44\times {10}^{-8}$	$9.19\times {10}^{-8}$	$9.43\times {10}^{-8}$	$9.11\times {10}^{-8}$
	${\mathrm{ESQM}}_e$	$3.21\times {10}^{-8}$	$2.71\times {10}^{-8}$	$1.87\times {10}^{-8}$	$2.41\times {10}^{-8}$	$3.15\times {10}^{-8}$
	${\mathrm{EAPG}}_{sr}$	$2.98\times {10}^{-8}$	$2.41\times {10}^{-8}$	$3.26\times {10}^{-8}$	$3.20\times {10}^{-8}$	$2.31\times {10}^{-8}$

and

Table 4. Computational results for Problem (105) with $ϵ={10}^{-6}$.

	Method	i = 2	i = 4	i = 6	i = 8	i = 10
Time (sec)	${\mathrm{t}}_{QR}$	0.910	5.989	19.148	44.312	91.606
	${\mathrm{t}}_{A^{†}b}$	0.009	0.034	0.081	0.146	0.263
	${\mathrm{t}}_{\parallel A\parallel }$	1.440	2.323	6.580	14.279	28.077
	IPOPT	16.414	27.392	35.748	56.783	135.875
	${\mathrm{SCP}}_{ls}$	3.357	12.381	18.140	52.182	72.401
	${\mathrm{ESQM}}_b$	14.465	59.228	130.443	243.864	385.851
	${\mathrm{ESQM}}_e$	2.457	9.712	21.284	40.449	65.420
	${\mathrm{EAPG}}_{sr}$	2.503	9.822	20.052	37.168	59.899
Iter	IPOPT	343	277	172	161	201
	${\mathrm{SCP}}_{ls}$	198	196	134	225	196
	${\mathrm{ESQM}}_b$	1555	1580	1604	1650	1632
	${\mathrm{ESQM}}_e$	259	258	260	272	276
	${\mathrm{EAPG}}_{sr}$	261	259	243	249	250
RecErr	IPOPT	0.762755	3051.699141	0.084688	0.084818	15.288138
	${\mathrm{SCP}}_{ls}$	0.080891	0.081482	0.084545	0.083153	0.086576
	${\mathrm{ESQM}}_b$	0.080938	0.081531	0.084597	0.083204	0.086629
	${\mathrm{ESQM}}_e$	0.080887	0.081479	0.084540	0.083149	0.086571
	${\mathrm{EAPG}}_{sr}$	0.080896	0.081489	0.084538	0.083148	0.086569
Residual	IPOPT	$1.21\times {10}^0$	$2.03\times {10}^0$	$1.75\times {10}^{-7}$	$1.71\times {10}^{-7}$	$1.96\times {10}^0$
	${\mathrm{SCP}}_{ls}$	$-4.60\times {10}^{-11}$	$-2.96\times {10}^{-11}$	$-2.97\times {10}^{-11}$	$-4.15\times {10}^{-11}$	$-3.98\times {10}^{-11}$
	${\mathrm{ESQM}}_b$	$1.07\times {10}^{-11}$	$1.03\times {10}^{-11}$	$9.55\times {10}^{-12}$	$9.66\times {10}^{-12}$	$9.25\times {10}^{-12}$
	${\mathrm{ESQM}}_e$	$3.40\times {10}^{-12}$	$6.13\times {10}^{-12}$	$4.77\times {10}^{-12}$	$3.88\times {10}^{-12}$	$1.69\times {10}^{-12}$
	${\mathrm{EAPG}}_{sr}$	$6.00\times {10}^{-12}$	$4.99\times {10}^{-12}$	$5.67\times {10}^{-12}$	$5.66\times {10}^{-12}$	$-9.82\times {10}^{-12}$

, respectively. The reported metrics are identical to those in Section 6.1:

–

$t_{\mathrm{QR}}$: Time to compute the QR decomposition of $A^T$.

–

$t_{\parallel A\parallel }$: Time to compute ${\parallel A\parallel }^2$.

–

$t_{A^{†}b}$: Time to compute $x^0=A^{†}b$ using the QR factorization of $A^T$.

–

CPU time of each algorithm.

–

Iter: Number of iterations.

–

RecErr $:={\displaystyle \frac{\parallel x^{*}-x_{\mathrm{orig}}\parallel }{max\{1,\parallel x_{\mathrm{orig}}\parallel \}}}$: Recovery error (where $x^{*}$ is the approximate solution from the algorithm).

–

Residual $:={\displaystyle \frac{\parallel A{x}^{*}{-b\parallel }_{LL2,\gamma }-\sigma }{\sigma }}$: Residual of the Lorentzian norm constraint violation.

• Key Observations from Results  

From Table 3 and Table 4, we observe a pattern consistent with that in Table 1 and Table 2:

1.

${\mathrm{EAPG}}_{\mathrm{sr}}$ frequently demonstrates the fastest convergence speed among the five.

2.

The recovery errors (RecErr) and residuals of all five methods are comparable.

6.3. Analysis on the Settings of Algorithm 3

The preceding experiments have demonstrated the efficiency of Algorithm 3 compared to other algorithms. In this subsection, we explain the rationality of the settings in Algorithm 3 by illustrating the following conclusions with experimental results:

(i)

The restart period N determined by Algorithm 3 is a good approximation of the optimal fixed restart period for Algorithm 4.

(ii)

The restart scheme of Algorithm 3 outperforms the following alternative schemes:

• Algorithm 1 and Algorithm 2, both with $\left\{{\theta }_k\right\}$ defined in Remark 1(ii) and with K set to 30 and 100, respectively.
• Variant (a) of Algorithm 3: Restarts only based on the $d_k$-criterion (without determining N or using the inner product criterion).
• Variant (b) of Algorithm 3: Algorithm 3 with the inner product criterion removed.
• Variant (c) of Algorithm 3: Determines the restart interval using both the $d_k$-criterion and the inner product criterion (consistent with Algorithm 3).
• Variant (d) of Algorithm 3: Algorithm 3 with the $d_k$-criterion replaced by the inner product criterion.
• Variant (e) of Algorithm 3: (Algorithm 3 Incorporating the Armijo Step Size Rule): Replace Equation (16) with the step size selection strategy outlined below:
  Let ${\tilde{x}}^{k+1}={\theta }_k{z}^{k+1}+(1-{\theta }_k)x^k$. If ${\theta }_k=1$ or $F\left({\tilde{x}}^{k+1}\right)\ge F\left(x^k\right)$, set $x^{k+1}={\tilde{x}}^{k+1}$ directly. Otherwise, compute

  $$x^{k+1}={\tilde{x}}^{k+1}+{\beta }^p(z^{k+1}-{\tilde{x}}^{k+1})$$

  where p denotes the smallest non-negative integer satisfying the inequality

  $${\displaystyle \frac{F\left({\tilde{x}}^{k+1}+{\beta }^p(z_{k+1}-{\tilde{x}}^{k+1})\right)-F\left({\tilde{x}}^{k+1}\right)}{{\beta }^p(1-{\theta }_k)}}\le c{\displaystyle \frac{F\left({\tilde{x}}^{k+1}\right)-F\left(x^k\right)}{{\theta }_k}}.$$

  In the above criterion, the parameters are fixed as $c=0.1$ and $\beta =0.5$.

To verify conclusion (i), we tested the fixed restarting version of Algorithm 1 on Problem (103) using a single dataset (where $i=2$) for each $N\in \{1,2,\dots ,50\}$ (with $ϵ={10}^{-4}$):

Algorithm 4 Fixed restarting with period N

Initialization: Given $x^0,z^0\in C$, ${\alpha }_0>0$, $d>0$, a positive integer N, and $\left\{{\theta }_k\right\}$ as defined in Remark A1(ii). for$s=1,2,\dots$ do Execute Algorithm 1 for N steps, with initial values $x^0,z^0,{\alpha }_0$ and parameters $\{{\theta }_0,{\theta }_1,\dots \}$. Set $x^0=z^0=z^N$ and ${\alpha }_0={\alpha }_N$. end for

Figure 1 presents the number of iterations for $N\in \{10,11,\dots ,50\}$; note that iterations for $N\in \{1,2,\dots ,9\}$ are excessively large (ranging from 1776 to 315). From Figure 1, the optimal fixed restart period is identified as 22. By contrast, when applying Algorithm 3 to the same problem and dataset, the $d_k$-criterion yields a restart period $N=24$ (see Figure 2).

Similarly, for Problem (105), we conducted the same comparison: the optimal fixed restart period is 42 (see Figure 3); the $d_k$-criterion yields a restart period $N=61$ (see Figure 4), which is also among the approximately optimal fixed restart periods. Consistent findings were observed across other datasets, confirming that the N determined by Algorithm 3 approximates the optimal fixed restart period well.

We also note that the lower bound $N_0$ is necessary. Without $N_0$, a small percentage of datasets may result in an extremely small restart period N, which in turn leads to a significantly higher number of iterations.

To verify conclusion (ii), we generated 20 random instances (following the same procedure as in Section 6.1 and Section 6.2) with $i=2$ and $ϵ={10}^{-4}$. The averaged computational results are presented in

Table 5. Computational results for Problem (103) with $i=2,ϵ={10}^{-4}$.

	Time	Iter	RecErr	Residual
Algorithm 1 (K = 30)	3.901	302	0.048687	$9.72\times {10}^{-7}$
Algorithm 2 (K = 30)	2.179	184	0.048684	$4.36\times {10}^{-10}$
Algorithm 1 (K = 100)	10.828	825	0.048687	$9.91\times {10}^{-7}$
Algorithm 2 (K = 100)	4.339	360	0.048689	$2.09\times {10}^{-8}$
Algorithm 3	1.029	101	0.049920	$4.82\times {10}^{-7}$
Variant (a)	1.827	173	0.048779	$7.19\times {10}^{-7}$
Variant (b)	1.009	101	0.049920	$4.82\times {10}^{-7}$
Variant (c)	1.010	101	0.049920	$4.82\times {10}^{-7}$
Variant (d)	1.376	135	0.049629	$2.74\times {10}^{-7}$
Variant (e)	1.298	129	0.049143	$5.64\times {10}^{-7}$

and

Table 6. Computational results for Problem (105) with $i=2,ϵ={10}^{-4}$.

	Time	Iter	RecErr	Residual
Algorithm 1 (K = 30)	3.983	349	0.080888	$9.78\times {10}^{-8}$
Algorithm 2 (K = 30)	1.998	205	0.080925	$3.81\times {10}^{-9}$
Algorithm 1 (K = 100)	12.048	935	0.080888	$9.89\times {10}^{-8}$
Algorithm 2 (K = 100)	3.821	361	0.080886	$6.69\times {10}^{-9}$
Algorithm 3	1.697	170	0.081517	$2.98\times {10}^{-8}$
Variant (a)	7.611	762	0.081102	$7.24\times {10}^{-8}$
Variant (b)	2.280	204	0.080623	$1.56\times {10}^{-8}$
Variant (c)	2.261	204	0.080623	$1.56\times {10}^{-8}$
Variant (d)	2.124	201	0.080910	$1.96\times {10}^{-8}$
Variant (e)	1.791	187	0.080645	$6.61\times {10}^{-8}$

, respectively. In both tables, Algorithm 3 consistently requires the fewest iterations, while the recovery errors of all algorithms are comparable. This confirms that Algorithm 3 is more effective than the other variants.

We note here that Algorithm 2 constitutes a genuine improvement over Algorithm 1; however, in the majority of test cases, only a single restart is triggered when implementing Algorithm 2. Thus, in practical scenarios, Algorithm 3 is able to identify more appropriate restart points and achieve more substantial performance gains.

On the other hand, Variant (e) delivers moderate performance across all variants of Algorithm 3. This observation indicates that line search techniques possess considerable potential for boosting the performance of algorithms such as ${\mathrm{EAPG}}_s$. Further research into the constraints governing acceleration parameters—analogous to the momentum conditions for generalized acceleration parameters in Nesterov’s first acceleration-augmented proximal gradient-type algorithms [57]—could enable line search to yield more efficient performance improvements.

6.4. Experiments on Unconstrained DC Problems

In this subsection, we demonstrate that the ${\mathrm{EAPG}}_{sr}$ algorithm yields notable performance improvements over IPOPT, GIST, pDCAe, and ${\mathrm{APG}}_s$ when applied to unconstrained DC problems.

We consider the support vector machine (SVM) model with training data ${\left\{(x_i,y_i)\right\}}_{i=1}^m\subseteq {\mathbb{R}}^n\times \{-1,1\}$, as proposed in [58]:

$$\underset{b\in \mathbb{R},w\in {\mathbb{R}}^n}{min}F(b,w)=f_1(b,w)-f_2(b,w)-f_3(b,w)+P_1(b,w),$$

(107)

where $P_1(b,w)={\lambda }_1{\parallel w\parallel }_1+{\displaystyle \frac{{\parallel w\parallel }_2^2}{2}}+{\displaystyle \frac{b^2}{2}}$ (with ${\lambda }_1$ denoting the regularization parameter). For $j=1,2,3$, $f_j(b,w)={\displaystyle \frac{1}{m}}{\sum }_{i=1}^m{\ell }_j\left[y_i(b+w^{\top }{x}_i)\right]$, and the smooth convex loss functions ${\ell }_j$ are defined as

$${\ell }_1\left(t\right)=\left\{\begin{array}{cc}{\displaystyle \frac{4}{5}}-t,\hfill & \mathrm{if}t<{\displaystyle \frac{3}{5}},\hfill \\ {\displaystyle \frac{5}{4}}{(1-t)}^2,\hfill & \mathrm{if}{\displaystyle \frac{3}{5}}\le t<1,\hfill \\ {\displaystyle \frac{5}{8}}{(1-t)}^2,\hfill & \mathrm{if}1\le t<{\displaystyle \frac{7}{5}},\hfill \\ -{\displaystyle \frac{1}{2}}\left({\displaystyle \frac{6}{5}}-t\right),\hfill & \mathrm{if}t\ge {\displaystyle \frac{7}{5}},\hfill \end{array}\right.$$

(108)

$${\ell }_2\left(t\right)=\left\{\begin{array}{cc}-t-{\displaystyle \frac{1}{5}},\hfill & \mathrm{if}t\le -{\displaystyle \frac{2}{5}},\hfill \\ {\displaystyle \frac{5}{4}}{t}^2,\hfill & \mathrm{if}-{\displaystyle \frac{2}{5}}<t\le 0,\hfill \\ 0,\hfill & \mathrm{if}t\ge 0,\hfill \end{array}\right.$$

(109)

$${\ell }_3\left(t\right)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}t\le {\displaystyle \frac{8}{5}},\hfill \\ {\displaystyle \frac{5}{8}}{\left(t-{\displaystyle \frac{8}{5}}\right)}^2,\hfill & \mathrm{if}{\displaystyle \frac{8}{5}}\le t<2,\hfill \\ -{\displaystyle \frac{1}{2}}\left({\displaystyle \frac{9}{5}}-t\right),\hfill & \mathrm{if}t\ge 2.\hfill \end{array}\right.$$

(110)

Implementation Details of the Five Algorithms

We specify the setup of the five competing algorithms in detail below:

(i)

Objective Function Formulation:

• For GIST, the objective is cast as $F=f+P_1$, where $f=f_1-f_2-f_3$;
• For pDCAe, F is decomposed into $F=f+P_1-P_2$, with $f=f_1$ and $P_2=f_2+f_3$;
• For ${\mathrm{APG}}_s$ and ${\mathrm{EAPG}}_{sr}$, the decomposition takes the form $F=f+P_1-P_2$, where $f=f_1-f_2$ and $P_2=f_3$;
• For IPOPT, to accommodate its requirement for differentiable objectives, we introduce non-negative slack variables $u,v$ such that $w=u-v$, thereby reformulating $F(b,w)$ into a differentiable form.

(ii)

Initialization and Termination Criteria: For each dataset, a total of 21 initial points $(b^0,w^0)$ are used uniformly across all algorithms except IPOPT: one zero vector, plus 5 independently sampled vectors from the normal distribution $\mathcal{N}(0,{\sigma }^{2}I)$ for each $\sigma \in \{1,2,4,8\}$. All algorithms terminate if either the relative iterate difference satisfies

$${\displaystyle \frac{\parallel (b^{k+1};w^{k+1})-(b^k;w^k)\parallel }{max\{1,\parallel (b^k;w^k)\parallel \}}}<{10}^{-6}$$

or the iteration count reaches the upper limit of 3000.

For IPOPT, corresponding to each of the aforementioned initial points $(b^0,w^0)$, we initialize the slack variables as $u^0={\left[w^0\right]}_{+}+0.001\mathbf{1}$ and $v^0={[-w^0]}_{+}+0.001\mathbf{1}$. The convergence tolerance is set to ${10}^{-6}$, with the maximum number of iterations configured to 1000.

(iii)

Parameter Configurations: The parameters for GIST, pDCAe, and ${\mathrm{APG}}_s$ are set in accordance with their respective original studies [26,52,53]. For ${\mathrm{EAPG}}_{sr}$, we fix the restart parameter $N_0=20$. All IPOPT settings are retained at their default values, with the only exception being the convergence tolerance (adjusted as specified above) and the maximum number of iterations.

A total of 8 real-world datasets are selected from the UCI Machine Learning Repository [59] for testing.

• Experimental Parameters and Evaluation Metrics

In our numerical experiments, the following parameter specifications and performance metrics are adopted:

–

The regularization parameter in Problem (107) is set to ${\lambda }_1=1\times {10}^{-3}$;

–

The Lipschitz constant L for all algorithms is taken as

$$L={\displaystyle \frac{5}{2}}{\displaystyle \frac{1}{m}}\sum _{i=1}^m{y}_i^2(1+\parallel x_i{\parallel }^2),$$

which is derived from ([58] Proposition 1).

Computational results are summarized in

Table 7. Comparison of five methods on 8 datasets.

	Iter
Dataset	IPOPT	GIST	pDCAe	${\mathbf{APG}}_{\mathit{s}}$	${\mathbf{EAPG}}_{\mathit{sr}}$
Australian	138	678	343	179	148
Banknote	46	2859	3000	78	76
Blood	45	290	157	79	74
German	110	2636	215	356	288
Glass	101	1401	206	174	126
Hepatitis	116	1269	230	227	187
Landmines	37	490	3000	69	70
Tic	66	67	147	120	107
	Fval
Dataset	IPOPT	GIST	pDCAe	${\mathbf{APG}}_{\mathit{s}}$	${\mathbf{EAPG}}_{\mathit{sr}}$
Australian	0.417258	4.222325	0.753193	0.417256	0.417256
Banknote	0.524224	0.524248	0.749316	0.524223	0.524223
Blood	0.559554	0.559553	0.565832	0.559553	0.559553
German	0.590471	8.832912	0.618164	0.590463	0.590463
Glass	0.373864	0.373862	0.566074	0.374673	0.374403
Hepatitis	0.415421	0.415417	0.518707	0.415417	0.415417
Landmines	0.521345	0.521344	0.774479	0.521344	0.521344
Tic	0.653866	0.653864	0.656695	0.653864	0.653864
	Time (s)
Dataset	IPOPT	GIST	pDCAe	${\mathbf{APG}}_{\mathit{s}}$	${\mathbf{EAPG}}_{\mathit{sr}}$
Australian	0.345	0.973	0.155	0.064	0.053
Banknote	0.132	5.630	2.422	0.053	0.051
Blood	0.100	0.210	0.067	0.027	0.026
German	0.331	3.020	0.167	0.207	0.160
Glass	0.169	1.487	0.064	0.057	0.046
Hepatitis	0.194	1.011	0.048	0.050	0.044
Landmines	0.068	0.271	0.695	0.032	0.027
Tic	0.142	0.084	0.159	0.084	0.076

, with the following metrics reported for each algorithm:

–

Iter: Number of iterations required to reach convergence;

–

Fval: Final value of the objective function at termination;

–

Time: Total CPU time (in seconds) consumed during the optimization process.

The experimental results in Table 7 consistently verify three core advantages of the ${\mathrm{EAPG}}_{sr}$ algorithm:

1.

It attains the second-lowest iteration count in most test cases.

2.

It achieves optimal objective function values across most datasets.

3.

It incurs the shortest computational time among all competing methods.

These findings confirm that ${\mathrm{EAPG}}_{sr}$ also exhibits superior performance in solving unconstrained DC problems.

7. Conclusions

In this paper, we extend the proximal gradient method with Nesterov’s second acceleration technique (${\mathrm{APG}}_s$; see [23,25,26,44])—originally designed for unconstrained DC problems—to an extended version (${\mathrm{EAPG}}_s$) for constrained DC problems. This extension draws on the constraint handling idea from the extended sequential quadratic method (ESQM) introduced in [18].

We establish the subsequential convergence of the entire sequence under appropriate assumptions. Additionally, by incorporating a restart technique, we further derive a global convergence result. Notably, this global convergence requires weaker assumptions on the function $P_1$ compared to those in [25,26].

Guided by our theoretical analysis, we further propose a practical variant of ${\mathrm{EAPG}}_s$ (dubbed ${\mathrm{EAPG}}_{sr}$, detailed in Algorithm 3) with efficient restart criteria. Numerical experiments demonstrate that, in most cases, ${\mathrm{EAPG}}_{sr}$ achieve high-quality solutions to the test problems with fewer iterations and lower CPU time.

Our core theoretical contributions are summarized as follows:

(1)

Extending the ${\mathrm{APG}}_s$ method to the setting of constrained DC problems, filling the gap between unconstrained DC optimization and constrained DC problem solving.

(2)

Deriving a global convergence result for the restart-augmented ${\mathrm{EAPG}}_s$ framework.

(3)

Weakening the original regularity requirements on the function $P_1$ that underpin the convergence of the baseline ${\mathrm{APG}}_s$ method.

From a numerical perspective, ${\mathrm{EAPG}}_{sr}$ demonstrates two key practical merits:

(a)

Providing a competitive new approach for solving constrained DC problems, with performance comparable to or exceeding state-of-the-art methods.

(b)

Delivering significant performance gains (attributed to the embedded restart technique) over the baseline ${\mathrm{APG}}_s$ method when applied to unconstrained DC problems.

Building on the advances of this work, several promising avenues for future research are identified:

(i)

Identifying the conditions under which Inequality (65) holds for Algorithm 1.

(ii)

Conducting a theoretical analysis of the restart criteria for Algorithm 3.

(iii)

Developing efficient solvers for subproblems involving more generalized forms of $P_1$ and cases with $m>1$.

(iv)

Exploring techniques (e.g., adaptive line search rules) to identify optimal acceleration parameters, further enhancing the algorithm’s computational efficiency.