Incremental Weak Subgradient Methods for Non-Smooth Non-Convex Optimization Problems

Abstract

Non-smooth, non-convex optimization problems frequently arise in modern machine learning applications, yet solving them efficiently remains a challenge. This paper addresses the minimization of functions of the form $f\left(x\right)={\sum }_{i=1}^m{f}_i\left(x\right)$ where each component is Lipschitz continuous but potentially non-smooth and non-convex. We extend the incremental subgradient method by incorporating weak subgradients, resulting in a framework better suited for non-convex objectives. We provide a comprehensive convergence analysis for three step size strategies: constant, diminishing, and a novel dynamic approach. Our theoretical results show that all variants converge to a neighborhood of the optimal solution, with the size of this neighborhood governed by the weak subgradient parameters. Numerical experiments on classification tasks with non-convex regularization, evaluated on the Breast Cancer Wisconsin dataset, demonstrate the effectiveness of the proposed approach. In particular, the dynamic step size method achieves superior practical performance, outperforming both classical and diminishing step size variants in terms of accuracy and convergence speed. These results position the incremental weak subgradient framework as a promising tool for scalable and efficient optimization in machine learning settings involving non-convex objectives.

1. Introduction

Optimization problems involving the minimization of a sum of component functions arise in numerous applications spanning machine learning, neural network training, and large-scale data analysis. These problems take the general form

$$\underset{x\in X}{min}f\left(x\right):=\sum _{i=1}^m{f}_i\left(x\right).$$

(1)

Assume that $f_i:{\mathbb{R}}^d\to \mathbb{R},\forall i=1,\dots ,m$ is $L_i$-Lipschitz continuous on X and it is not necessarily convex and smooth. X is also a nonempty, closed, and convex subset of ${\mathbb{R}}^d$. While traditional subgradient methods are well established for convex optimization, their extension to non-convex settings requires more sophisticated tools. Although widely applicable, traditional subgradient methods suffer from significant numerical instability near non-differentiable points, leading to erratic updates and slow convergence rates [1]. To address the limitations of classical subgradient methods in non-smooth optimization, we adopt the framework of weak subgradients, a relaxed notion of the standard subdifferential. This approach offers several advantages:

• Numerical stability: Weak subgradients mitigate the high variance of standard subgradients near non-differentiable points, enabling more stable and smoother convergence trajectories [2].
• Scalability: In large-scale problems like empirical risk minimization, they support efficient incremental or stochastic updates without requiring full subgradient computations [3].
• Adaptive step sizes: Their synergy with dynamic or diminishing step size schemes enhances both convergence speed and robustness in practice [4].
• Non-smooth structures: Particularly effective in settings with non-smooth regularizers (e.g., ${\mathcal{\ell }}_1$), piecewise-linear losses (e.g., hinge loss), or discontinuous activations, where classical subgradients are ill defined [5,6].

One such advancement is the weak subgradient concept introduced by Azimov and Gasimov [7], which replaces the supporting hyperplanes used in convex analysis with supporting conic surfaces to the graph of a function. This approach enables the handling of non-convex functions without requiring convexity assumptions.

A weak subgradient is defined as a pair $(v,c)\in {\mathbb{R}}^d\times {\mathbb{R}}_{+}$ such that the graph of a superlinear function ${\eta }_{(v,c)}\left(x\right)=\langle v,x\rangle -c\parallel x\parallel$ supports the epigraph of the function under consideration at a given point [8]. This construction significantly expands the class of functions that can be effectively analyzed, including many non-smooth and non-convex functions that arise in practical applications.

The relationship between directional derivatives and weak subdifferentials established by Kasimbeyli and Mammadov [9] provides a theoretical foundation for developing computational methods for non-convex optimization. Their work showed that under certain conditions, the directional derivative can be expressed as a “sup” relation using elements from the weak subdifferential, which generalizes the well-known “max-relation” from convex analysis.

The computational burden of processing large objective functions led to the development of incremental methods, which process component functions sequentially rather than evaluating the entire objective at each iteration. The incremental subgradient algorithm was first systematically studied by Nedic and Bertsekas [1], who established convergence properties under various step size rules. Their work was motivated by applications in neural network training [10,11], image reconstruction, and machine learning problems where the objective function naturally decomposes into a sum of many component functions. Earlier incremental gradient methods for smooth problems [12,13,14] provided important foundations, but extensions to non-smooth settings required more sophisticated analysis.

A critical factor affecting the performance of subgradient methods is the step size selection strategy. Traditional approaches include: constant step sizes, which typically converge to a neighborhood of the optimal solution whose size depends on the step magnitude [15]; diminishing step sizes satisfying the conditions, which guarantee asymptotic convergence but often exhibit slow practical performance [16]; Polyak step sizes, which utilize estimates of the optimal function value to determine appropriate step lengths [16].

A significant advancement came with Goffin and Kiwiel’s [17] introduction of dynamic step size rules, which automatically adapt based on the algorithm’s progress. This approach was extended to incremental settings by Nedic and Bertsekas [1], where it demonstrated superior empirical performance compared to other step size strategies. Yang and Wang [18] further refined this approach, proposing two modified dynamic step size rules that demonstrated improved convergence properties and practical performance for separable convex optimization problems.

While convex optimization benefits from well-established theory and methods, non-convex problems present additional challenges due to the potential presence of multiple local minima and saddle points. The concept of weak subgradients offers a more suitable framework for non-convex optimization, providing a generalization that accommodates the complex geometry of non-convex functions while maintaining useful theoretical properties. Recent work by Wang [19] has explored subgradient algorithms on Riemannian manifolds with lower bounded curvatures, while Nesterov [20] contributed with primal–dual subgradient methods, establishing important theoretical foundations. Yao et al. [21] explored projected subgradient algorithms for pseudomonotone equilibrium problems, demonstrating the breadth of applications for advanced subgradient techniques.

This paper addresses a significant gap in the literature by extending incremental subgradient methods to non-smooth, non-convex optimization problems using weak subgradients. Our primary contributions include the following: (1) Development of an incremental weak subgradient framework that accommodates non-convex component functions while maintaining theoretical convergence guarantees. (2) Comprehensive analysis of three-step size strategies—constant, diminishing, and a novel dynamic approach—establishing that all variants converge to a neighborhood of optimal solutions, with the neighborhood size determined by weak subgradient parameters. (3) Introduction of a modified dynamic step size algorithm with practical enhancements that improve robustness and convergence behavior, particularly for challenging optimization landscapes. (4) Empirical validation through extensive numerical experiments on classification tasks with non-convex regularization demonstrates our approach’s practical effectiveness compared to conventional methods.

Our theoretical analysis builds upon and extends the work of Yang and Wang [18] on incremental subgradient methods with dynamic step sizes for convex optimization, adapting their insights to the more challenging non-convex setting through the use of weak subgradients. The proposed methods maintain the computational efficiency of incremental approaches while accommodating the additional complexity introduced by non-convexity.

We establish that all three variants of our algorithm—with constant, diminishing, and dynamic step sizes—converge to a neighborhood of the optimal solution, with the size of this neighborhood determined by the parameters of the weak subgradients. Notably, our dynamic step size approach demonstrates superior practical performance by automatically adjusting to problem characteristics during the optimization process.

The theoretical contributions of this paper build upon extensive literature, extending the analysis to accommodate both the incremental processing of component functions and the use of weak subgradients for non-convex optimization. By establishing convergence guarantees for constant, diminishing, and dynamic step size strategies in this context, we bridge an important gap in the existing theory while providing practical algorithms for challenging real-world problems.

The remainder of this paper is organized as follows: Section 2 introduces preliminary concepts and properties of weak subgradients. Section 3 presents the incremental weak subgradient algorithm and analyzes its convergence properties under different step size strategies. Section 4 provides experimental results on classification tasks with non-convex regularization, and Section 5 concludes with a discussion of implications and future research directions.

2. Preliminaries

In this section, an exploration of the weak subdifferential and its fundamental properties is undertaken. While all of the theorems presented may not be directly utilized in the current proofs, they serve to establish a foundational comprehension of the weak subdifferential. For a deeper insight into this topic, readers are encouraged to consult references [7,9,22] for additional information.

Definition 1.

Let $f:\mathbb{S}\to (-\infty ,+\infty ]$ be a given function, where $\mathbb{S}$ is a nonempty and compact subset of $\mathbb{R}$, and $\overline{x}\in \mathbb{S}$. A pair $(v,c)\in {\mathbb{R}}^d\times {\mathbb{R}}_{+}$ is called a weak subgradient of f at $\overline{x}$ on $\mathbb{S}$ if

$$f\left(x\right)\ge f\left(\overline{x}\right)+\langle v,x-\overline{x}\rangle -c\parallel x-\overline{x}\parallel ,\forall x\in \mathbb{S}.$$

The set

$${\partial }_{\mathbb{S}}^{w}f\left(\overline{x}\right)=\left\{(v,c)\in {\mathbb{R}}^d\times {\mathbb{R}}_{+}:f\left(x\right)\ge f\left(\overline{x}\right)+\langle v,x-\overline{x}\rangle -c\parallel x-\overline{x}\parallel ,\forall x\in \mathbb{S}\right\}$$

of all weak subgradients of f at $\overline{x}$ is called the weak subdifferential of f at $\overline{x}$ on $\mathbb{S}$.

Definition 2.

A function $f:\mathbb{S}\to (-\infty ,+\infty ]$ is called locally lower Lipschitz at $\overline{x}\in \mathbb{S}$ if there exists a non-negative number L (Lipschitz constant), and a neighborhood $N\left(\overline{x}\right)$ of $\overline{x}$ such that

$$f\left(x\right)-f\left(\overline{x}\right)\ge -L\parallel x-\overline{x}\parallel ,\forall x\in N\left(\overline{x}\right).$$

(2)

If the above inequality holds true for all $x\in \mathbb{S}$, then f is called lower Lipschitz at $\overline{x}$ with Lipschitz constant L.

Theorem 1.

Let $f:\mathbb{S}\to (-\infty ,+\infty ]$ be a given function.

• Let $f\left(\overline{x}\right)$ be lower semi-continuous at $\overline{x}\in \mathbb{S}$. Then, f is weakly subdifferentiable at $\overline{x}$ if there is lower locally Lipschitz at $\overline{x}$ and there exists a point $\tilde{x}$ such that f is subdifferentiable there.
• The function f is weakly subdifferentiable at $\overline{x}$ if f is lower Lipschitz at $\overline{x}$.
• The function f is weakly subdifferentiable at $\overline{x}$ if there is lower locally Lipschitz at $\overline{x}$ and bounded below.
• The function f is weakly subdifferentiable at $\overline{x}$ if f is a lower locally Lipschitz at $\overline{x}$, and there exist numbers $p\ge 0$ and q such that $f\left(x\right)\ge$$-p\parallel x\parallel +q$ for all $x\in \mathbb{S}$.
• If f is a positively homogeneous function bounded from below on some neighborhood of $0_{{\mathbb{R}}^d}$, then f is weakly subdifferentiable at $0_{{\mathbb{R}}^d}$.

Theorem 2.

The weak subdifferential ${\partial }^{w}f\left(·\right)$ of a function f is convex and closed.

3. Incremental Subgradient Method for Non-Smooth Non-Convex Optimization

The incremental weak subgradient method addresses problem (3) by processing the component functions sequentially in cycles. The following algorithm presents the core update procedure for a single cycle.

The critical element that determines the performance of Algorithm 1 is the choice of step size ${\alpha }_n$. In this section, we analyze three strategies for selecting the step size: constant, diminishing, and a novel dynamic approach. For each strategy, we establish convergence properties and characterize the quality of the solutions obtained. We use the notation

$$d_X=\mathrm{diam}\left(X\right)=\underset{x_1,x_2}{max}∥x_1-x_2∥,f^{*}=f\left(x^{*}\right)=\underset{x\in X}{inf}f\left(x\right)$$

At each iteration n, the weak subgradient approximation algorithm [8] can be used. This algorithm is based on the approximate representation ${\partial }_X^w{f}_i\left({\phi }_{i-1,n}\right)$ of the weak subdifferential, which assumes that there exist positive numbers $D_i$ and $K_i$ such that

$$\begin{array}{cc}\hfill \parallel g_{i,n}\parallel & \le D_i\forall n\ge 0i=1,\dots ,m,\hfill \\ \hfill c_{i,n}& \le K_i.\hfill \end{array}$$

Algorithm 1 Incremental weak subgradient update step.

Require:  Current point $x_n$, step size ${\alpha }_n$ 1: Set ${\phi }_0=x_n$ 2: for $i=1$ to m do 3:    Select $(g_{i,n},c_{i,n})\in {\partial }_X^w{f}_i\left({\phi }_{i-1,n}\right)$ such that $\parallel g_{i,n}\parallel$ is minimized 4:    ${\phi }_{i,n}=P_X({\phi }_{i-1,n}-{\alpha }_n{g}_{i,n})$ 5: end for 6: return ${\phi }_{m,n}$

Lemma 1.

Let $\left\{x_n\right\}$ be the sequence generated by the incremental subgradient method. Then, for all $n\ge 0$ and $y\in X$, we have

$$\parallel x_{n+1}{-y\parallel }^2\le {\parallel x_n-y\parallel }^2-2{\alpha }_n\left(f\left(x_n\right)-f\left(y\right)\right)+M_1{\alpha }_n^2+M_2{\alpha }_n$$

where $M_1:=2{\displaystyle \sum _{i=1}^m}\left\{(L_i+K_i){\sum }_{j=1}^{i-1}{D}_j{D}_i^2\right\}$ and $M_2:=2d_X{\displaystyle \sum _{i=1}^m}{K}_i$.

Proof. 

Using the nonexpansion property of the projection, the weak subgradient boundedness, and the weak subgradient inequality for each component function $f_i$, we obtain for all $y\in X$

$$\begin{array}{cc}\hfill \parallel {\phi }_{i,n}{-y\parallel }^2& = \parallel P_X({\phi }_{i-1,n}-{\alpha }_n{g}_{i,n}){-y\parallel }^2\hfill \\ \hfill & \le \parallel {\phi }_{i-1,n}-{\alpha }_n{g}_{i,n}{-y\parallel }^2\hfill \\ \hfill & \le \parallel {\phi }_{i-1,n}{-y\parallel }^2-2{\alpha }_n{g}_{i,n}^T({\phi }_{i-1,n}-y)+{\alpha }_n^2{\parallel g_{i,n}\parallel }^2\hfill \\ \hfill & \le \parallel {\phi }_{i-1,n}{-y\parallel }^2-2{\alpha }_n{g}_{i,n}^T({\phi }_{i-1,n}-y)+{\alpha }_n^2{D}_i^2\hfill \\ \hfill & \le \parallel {\phi }_{i-1,n}{-y\parallel }^2-2{\alpha }_n\left(f_i\left({\phi }_{i-1,n}\right)-f_i\left(y\right)-c_{i,n}\parallel y-{\phi }_{i-1,n}\parallel \right)+{\alpha }_n^2{D}_i^2.\hfill \end{array}$$

Since $\parallel {\phi }_{i-1,n}-x_n\parallel \le {\alpha }_n{\sum }_{j=1}^i{D}_{j}i=1,\dots ,m,n\ge 0$, by adding these inequalities over $i=1,\dots ,m$, we have, for all $y\in X$ and n,

$$\begin{array}{cc}\hfill \parallel x_{n+1}{-y\parallel }^2& =\parallel {\phi }_{m,n}{-y\parallel }^2\hfill \\ \hfill & \le \parallel x_n{-y\parallel }^2-2{\alpha }_n\sum _{i=1}^m\left\{f_i\left({\phi }_{i-1,n}\right)-f_i\left(y\right)-c_{i,n}\parallel y-{\phi }_{i-1,n}\parallel \right\}+{\alpha }_n^2\sum _{i=1}^m{D}_i^2\hfill \\ \hfill & =\parallel x_n{-y\parallel }^2-2{\alpha }_n\left(f\left(x_n\right)-f\left(y\right)+\sum _{i=1}^m\left\{f_i\left({\phi }_{i-1,n}\right)-f_i\left(x_n\right)-c_{i,n}\parallel {\phi }_{i-1,n}-y\parallel \right\}\right)+{\alpha }_n^2\sum _{i=1}^m{D}_i^2\hfill \\ \hfill & \le \parallel x_n{-y\parallel }^2+2{\alpha }_n\left(f\left(y\right)-f\left(x_n\right)+\sum _{i=1}^m\left\{L_i\parallel {\phi }_{i-1,n}-x_n\parallel +K_i(\parallel {\phi }_{i-1,n}-x_n\parallel +\parallel x_n-y\parallel )\right\}\right)\hfill \\ \hfill & +{\alpha }_n^2\sum D_i^2\hfill \\ \hfill & \le \parallel x_n{-y\parallel }^2-2{\alpha }_n\left(f\left(x_n\right)-f\left(y\right)\right)+{\alpha }_n^2\left(2\sum _{i=1}^m\left\{(L_i+K_i)\sum _{j=1}^{i-1}{D}_j{D}_i^2\right\}\right)+2d_X\sum _{i=1}^m{K}_i{\alpha }_n\hfill \\ \hfill & =\parallel x_n{-y\parallel }^2-2{\alpha }_n\left(f\left(x_n\right)-f\left(y\right)\right)+M_1{\alpha }_n^2+M_2{\alpha }_n\hfill \end{array}$$

where $M_1:=2{\displaystyle \sum _{i=1}^m}\left\{(L_i+K_i){\sum }_{j=1}^{i-1}{D}_j{D}_i^2\right\}$ and $M_2:=2d_X{\displaystyle \sum _{i=1}^m}{K}_i$.    □

3.1. Convergence Analysis for the Constant Step Size

We start the convergence analysis with the constant step size rule.

Proposition 1.

For the sequence $\left\{x_n\right\}$ generated by the weak incremental subgradient method with a constant step size ${\alpha }_n=\alpha$, we have

$$\underset{n\to \infty }{lim\; inf}f\left(x_n\right)\le f^{*}+{\displaystyle \frac{\alpha M_1+M_2}{2}}$$

Proof. 

If the result does not hold, there must exist an epsilon > 0 such that

$$\underset{n\to \infty }{lim\; inf}f\left(x_n\right)>f^{*}+{\displaystyle \frac{\alpha M_1+M_2}{2}}+2ϵ$$

Let $\widehat{y}\in X$ be such that

$$\underset{n\to \infty }{lim\; inf}f\left(x_n\right)\ge f\left(\widehat{y}\right)+{\displaystyle \frac{\alpha M_1+M_2}{2}}+2ϵ$$

and let $k_0$ be large enough so that for all $n\ge k_0$,

$$f\left(x_n\right)-f\left(\widehat{y}\right)\ge {\displaystyle \frac{\alpha M_1+M_2}{2}}+ϵ$$

By adding the preceding two relations, we obtain, for all $n\ge k_0$,

$$f\left(x_n\right)-f\left(\widehat{y}\right)\ge {\displaystyle \frac{\alpha M_1+M_2}{2}}+ϵ$$

Using Lemma 1 for the case where $y=\widehat{y}$ together with the above relation, we obtain, for all $n\ge k_0$,

$$\begin{array}{cc}\hfill \parallel x_{n+1}-\widehat{y}{\parallel }^2-2\alpha ϵ& \le \parallel x_{n-1}-\widehat{y}{\parallel }^2-2\alpha ϵ\hfill \\ \hfill & \le \dots \le \parallel x_{k_0}-\widehat{y}{\parallel }^2-2(n+1-k_0)\alpha ϵ\hfill \end{array}$$

which cannot hold for k sufficiently large, so it is a contradiction.    □

3.2. Convergence Analysis for the Diminishing Step Size

Proposition 2.

Assume that the step size ${\alpha }_n$ satisfies

$$\underset{n\to \infty }{lim}{\alpha }_n=0\sum _{n=0}^{\infty }{\alpha }_n=\infty .$$

Then, for the sequence $\left\{x_n\right\}$ generated by the weak incremental method, we have

$$lim\; inff\left(x_n\right)\le f^{*}+{\displaystyle \frac{M_2}{2}}.$$

Proof. 

Assume, to arrive at a contradition, that the above inequality does not hold, so there exists an $ϵ>0$ such that

$$lim\; inff\left(x_n\right)>f\left(\widehat{y}\right)+{\displaystyle \frac{M_2}{2}}+ϵ$$

Let $k_0$ be large enough so that for all $n\ge k_0$, we have

$$f\left(x_n\right)-f\left(\widehat{y}\right)\ge {\displaystyle \frac{M_2}{2}}+ϵ$$

Again, by using Lemma 1, in a similar way used in the proof of Proposition 1, we obtain

$$\begin{array}{cc}\hfill \parallel x_{n+1}-\widehat{y}{\parallel }^2& \le \parallel x_n-\widehat{y}{\parallel }^2-2{\alpha }_n\left({\displaystyle \frac{M_2}{2}}+ϵ\right)+M_1{\alpha }_n^2+M_2{\alpha }_n\hfill \\ \hfill & = \parallel x_n-\widehat{y}{\parallel }^2-{\alpha }_n{M}_2-2{\alpha }_nϵ+M_1{\alpha }_n^2+M_2{\alpha }_n\hfill \\ \hfill & = \parallel x_n-\widehat{y}{\parallel }^2-{\alpha }_n(2ϵ-{\alpha }_n{M}_1)\hfill \end{array}$$

Because ${\alpha }_n\to 0$, and $\left\{g_n\right\}$ is bounded, without loss of generality, we may assume that $k_0$ is large enough so that $2ϵ-{\alpha }_n{M}_1<ϵ$ for all $n\ge k_0$, implying that

$$\parallel x_{n+1}-\widehat{y}{\parallel }^2 \le \parallel x_n-\widehat{y}{\parallel }^2-{\alpha }_nϵ\le \dots \le {\parallel x_{k_0}-\widehat{y}\parallel }^2-ϵ\sum _{j=k_0}^n{\alpha }_j$$

Since ${\displaystyle \sum _{j=k_0}^n}{\alpha }_j=\infty$, this relation cannot hold for n sufficiently large, so it is a contradiction.    □

Proposition 3.

Assume that the step size ${\alpha }_n$ satisfies

$$\sum {\alpha }_n=\infty \sum {\alpha }_n^2<\infty$$

Then, for the sequence $\left\{x_n\right\}$ generated by the weak incremental method, we have

$$lim\; inff\left(x_n\right)\le f^{*}+{\displaystyle \frac{M_2}{2}}.$$

Proof. 

Assume, to arrive at a contradition, that the above inequality does not hold, so there exists an $ϵ>0$ such that

$$lim\; inff\left(x_n\right)>f\left(\widehat{y}\right)+{\displaystyle \frac{M_2}{2}}+ϵ$$

Let $k_0$ be large enough so that for all $n\ge k_0$, we have

$$f\left(x_n\right)-f\left(\widehat{y}\right)\ge {\displaystyle \frac{M_2}{2}}+ϵ$$

Again, by using Lemma 1, in a similar way used in the proof of Proposition 1, we obtain

$$\begin{array}{cc}\hfill \parallel x_{n+1}-\widehat{y}{\parallel }^2& \le \parallel x_n-\widehat{y}{\parallel }^2-2{\alpha }_nϵ+M_1{\alpha }_n^2\hfill \\ \hfill & \le \parallel x_{n-1}-\widehat{y}{\parallel }^2-2({\alpha }_n+{\alpha }_{n-1})ϵ+M_1({\alpha }_n^2+{\alpha }_{n-1}^2)\hfill \\ \hfill & \le \dots \le \parallel x_0-\widehat{y}{\parallel }^2-2\left(\sum _{j=0}^n{\alpha }_j\right)ϵ+M_1\left(\sum _{j=0}^n{\alpha }_j^2\right)\hfill \end{array}$$

According to the assumption, we reach a contradiction.    □

3.3. Convergence Analysis for the Dynamic Step Size

Now, we propose a dynamic step size strategy for the incremental weak subgradient method that adapts based on the progress of the optimization process. This approach offers advantages over both constant and diminishing step size methods by automatically adjusting the step size based on the current function value and a target level.

In Algorithm 2, the step size ${\alpha }_n$ is determined dynamically using the following formula:

$${\alpha }_n=max\left\{{\gamma }_n{\displaystyle \frac{f\left(x_n\right)-f_{\mathrm{lev}}^n}{C^2}},{\alpha }_{min}\right\}$$

(3)

where

• $f\left(x_n\right)$ is the current function value at iteration n;
• $f_{\mathrm{lev}}^n=f_{\mathrm{rec}}^n-{\delta }_l$ is the target level, with $f_{\mathrm{rec}}^n$ being the best function value found so far;
• ${\gamma }_n\in (0,\overline{\gamma })$ where $\overline{\gamma }<2$ is a control parameter;
• ${\alpha }_{min}>0$ is a minimum step size threshold.

Algorithm 2 Incremental weak subgradient method with dynamic step size.

Require:  Initial point $x_0\in X$, initial target level parameter ${\delta }_0>0$, minimum target level ${\delta }_{min}>0$ 1: Set $n=0$, $l=0$, $f_{rec}^{-1}=+\infty$. 2: while stopping criterion not satisfied do 3:    Compute $f\left(x_n\right)$ 4:    if $f\left(x_n\right)<f_{rec}^{n-1}$ then 5:      $f_{rec}^n=f\left(x_n\right)$, $x_{rec}^n=x_n$ 6:    else 7:      $f_{rec}^n=f_{rec}^{n-1}$, $x_{rec}^n=x_{rec}^{n-1}$ 8:    end if 9:    if $(0_{{\mathbb{R}}^d},0_{\mathbb{R}})\in {\partial }^{w}f\left(x_n\right)$ then 10:      STOP 11:    end if 12:    if ${\delta }_l<{\delta }_{tol}$ then 13:      RETURN $x_{rec}^n$ {Early termination when target level is sufficiently small} 14:    end if 15:    if $f\left(x_n\right)\le f_{rec}^{n-1}-{\displaystyle \frac{1}{2}}{\delta }_l$ then 16:      ${\delta }_{l+1}={\delta }_l$, $l=l+1$ 17:    else 18:      ${\delta }_{l+1}=max\{{\displaystyle \frac{{\delta }_0}{l+1}},{\delta }_{min}\}$, $l=l+1$ 19:    end if 20:    Set $f_{lev}^n=f_{rec}^n-{\delta }_l$ 21:    Set $C^2={\sum }_{i=1}^m{D}_i^2$ {Define C based on weak subgradient bounds} 22:    Consider step size ${\alpha }_n=max\{{\gamma }_n{\displaystyle \frac{f\left(x_n\right)-f_{lev}^n}{C^2}},{\alpha }_{min}\}$, where $0<\underline{\gamma }\le {\gamma }_n\le \overline{\gamma }<2$ 23:    Calculate ${\phi }_{m,n}$ according Algorithm 1 and then set $x_{n+1}={\phi }_{m,n}$ 24:    Set $n=n+1$ 25: end while 26: return $x_{rec}^n$

This adaptive step size strategy offers several advantages over traditional approaches:

• Problem-aware adaptation: The term $f\left(x_n\right)-f_{\mathrm{lev}}^n$ allows the step size to automatically adjust based on the gap between the current function value and our target level. This enables larger steps when far from the target (accelerating progress) and smaller, more careful steps as we approach it (preventing overshooting).
• Gradient normalization: The denominator $C^2$ normalizes the step size relative to the magnitudes of the weak subgradients, ensuring appropriate scaling regardless of the problem’s conditioning.
• Lower bound guarantee: The minimum step size ${\alpha }_{min}$ prevents the algorithm from stalling in challenging regions of non-convex functions, such as plateaus or shallow local minima.
• Theoretical convergence: The parameter ${\gamma }_n<2$ ensures the method maintains theoretical convergence guarantees while ${\alpha }_{min}$ provides practical robustness.

We have enhanced the standard method with several practical improvements:

• A minimum target level ${\delta }_{min}$ to prevent excessive step size reduction;
• Early termination based on tolerance ${\delta }_{\mathrm{tol}}$;
• Explicit definition of parameter C based on weak subgradient bounds;
• A minimum step size ${\alpha }_{min}$ for numerical stability.

Together, these modifications provide robustness for non-convex optimization while maintaining theoretical convergence guarantees. Unlike constant step sizes (which cannot adapt to varying problem difficulty) or diminishing step sizes (which decrease regardless of progress), our dynamic approach intelligently responds to the local geometry of the objective function, balancing theoretical convergence guarantees with practical efficiency for non-smooth, non-convex optimization problems.

The main reason for the selection approach to sequence ${\left\{{\delta }_l\right\}}_{l\in \mathbb{N}}$ in non-convex optimization is to address the fundamentally different landscape characteristics compared to convex problems. In convex optimization, progress tends to be relatively predictable and monotonic, with the function value consistently decreasing as we approach the global minimum. Standard methods with fixed or simply diminishing step sizes work well because the landscape does not contain local minima or saddle points to complicate the trajectory.

Non-convex functions, however, present several unique challenges:

• Multiple local minima that can trap optimization algorithms;
• Saddle points where progress may temporarily stall;
• Regions of varying curvature requiring different step sizes;
• Plateaus and steep valleys that demand different exploration strategies.

The dynamic delta selection addresses these challenges by enabling the algorithm to automatically sense and adapt to the local geometry of the function. When the algorithm encounters steep descents, it maintains the current target level to capitalize on rapid progress. When it hits challenging regions like plateaus or approaches local minima, it reduces the target level to facilitate more careful exploration.

This adaptive mechanism acts as a form of “landscape sensing” that is not necessary in convex optimization but becomes crucial for efficiently navigating the complex topography of non-convex functions. The addition of a minimum target level ${\delta }_{min}$ further enhances this approach by ensuring the algorithm maintains sufficient exploration capability to escape shallow local minima—a concern that does not exist in the convex case.

We now establish several important theoretical properties of this algorithm, starting with the behavior of the target level parameter.

Lemma 2 (Finiteness of Parameters).

Let $\left\{x_n\right\}$ be the sequence generated by Algorithm 2. Then, exactly one of the following statements holds:

• The algorithm terminates with $(0_{{\mathbb{R}}^d},0_{\mathbb{R}})\in {\partial }^{w}f\left(x_n\right)$;
• The algorithm terminates with ${\delta }_l<{\delta }_{tol}$;
• $l\to \infty$ and $lim{inf}_{l\to \infty }{\delta }_l={\delta }_{min}$.

Proof. 

From the structure of Algorithm 2, there are only two explicit termination conditions:

• Lines 9–11: The algorithm terminates if $(0_{{\mathbb{R}}^d},0_{\mathbb{R}})\in {\partial }^{w}f\left(x_n\right)$, which indicates that a necessary condition for optimality holds.
• Lines 12–14: The algorithm terminates if ${\delta }_l<{\delta }_{tol}$, which is the early termination criterion based on the target level parameter becoming sufficiently small.

If neither of these conditions is met, the algorithm continues executing. We will now prove that in this case, $l\to \infty$ and $lim{inf}_{l\to \infty }{\delta }_l={\delta }_{min}$.

Assume, by contradiction, that the algorithm does not terminate due to either condition above, and that l remains bounded (i.e., $l<\infty$) with ${\delta }_l>{\delta }_{min}$ for all l beyond some index. Since l is incremented in each iteration where the algorithm does not terminate, this means the algorithm must terminate after finitely many iterations, contradicting our assumption. Thus, if the algorithm does not terminate due to the explicit conditions, we must have $l\to \infty$.

Now, we need to show that $lim{inf}_{l\to \infty }{\delta }_l={\delta }_{min}$. According to the update rule in lines 15–19 of the algorithm, particularly line 17,

$${\delta }_{l+1}=max\{{\displaystyle \frac{{\delta }_0}{l+1}},{\delta }_{min}\}$$

(4)

For sufficiently large l, either ${\delta }_{l+1}={\delta }_l$ when progress is significant or ${\delta }_{l+1}$ is determined by the formula above. Observe that for any $l>{\displaystyle \frac{{\delta }_0}{{\delta }_{min}}}-1$, we have ${\displaystyle \frac{{\delta }_0}{l+1}}<{\delta }_{min}$, which means ${\delta }_{l+1}={\delta }_{min}$ for all sufficiently large l. Therefore, the sequence $\left\{{\delta }_l\right\}$ eventually becomes constant at ${\delta }_{min}$, which implies $lim{inf}_{l\to \infty }{\delta }_l={\delta }_{min}$.

However, to be thorough, we will also prove that ${\delta }_l$ cannot remain above ${\delta }_{min}$ indefinitely. Assume that there exists $\overline{\delta }>{\delta }_{min}$ such that ${\delta }_l\ge \overline{\delta }$ for all l.

From the step size calculation in line 21 of the algorithm and the fact that $f_{lev}^n=f_{rec}^n-{\delta }_l$, with our redefined $C^2={\sum }_{i=1}^m{D}_i^2$,

$${\alpha }_n\ge max\{{\gamma }_n{\displaystyle \frac{f\left(x_n\right)-f_{rec}^n+\overline{\delta }}{{\sum }_{i=1}^m{D}_i^2}},{\alpha }_{min}\}$$

(5)

We now consider two cases:

Case 1: $f\left(x_n\right)-f_{rec}^n\ge -{\displaystyle \frac{\overline{\delta }}{2}}$ for infinitely many n.

For these iterations:

$${\alpha }_n\ge max\{{\gamma }_n{\displaystyle \frac{\overline{\delta }/2}{{\sum }_{i=1}^m{D}_i^2}},{\alpha }_{min}\}\ge max\{{\displaystyle \frac{\underline{\gamma }\overline{\delta }}{2{\sum }_{i=1}^m{D}_i^2}},{\alpha }_{min}\}=\mu >0$$

(6)

Using Lemma 1 with $y=x^{*}$, for the iterations where ${\alpha }_n\ge \mu$:

$$\parallel x_{n+1}-x^{*}{\parallel }^2\le {\parallel x_n-x^{*}\parallel }^2-2\mu (f\left(x_n\right)-f^{*})+M_1{\mu }^2+M_2\mu$$

(7)

If $f\left(x_n\right)>f^{*}+{\displaystyle \frac{M_1\mu +M_2}{2}}+\epsilon$ for some $\epsilon >0$ and infinitely many n, then

$$\parallel x_{n+1}-x^{*}{\parallel }^2\le {\parallel x_n-x^{*}\parallel }^2-\mu \epsilon$$

(8)

Applying this inequality recursively for all such iterations would eventually make $\parallel x_n-x^{*}{\parallel }^2$ negative, which is a contradiction.

Therefore, for infinitely many iterations, we must have

$$f\left(x_n\right)\le f^{*}+{\displaystyle \frac{M_1\mu +M_2}{2}}+\epsilon$$

(9)

Since this holds for any $\epsilon >0$, we have ${lim\; inf}_{n\to \infty }f\left(x_n\right)\le f^{*}+{\displaystyle \frac{M_1\mu +M_2}{2}}$, where $\mu =max\{{\displaystyle \frac{\underline{\gamma }\overline{\delta }}{2{\sum }_{i=1}^m{D}_i^2}},{\alpha }_{min}\}$.

Given that $f\left(x_n\right)-f_{rec}^n\ge -{\displaystyle \frac{\overline{\delta }}{2}}$ for infinitely many n in this case, we know that $f_{rec}^n\le f\left(x_n\right)+{\displaystyle \frac{\overline{\delta }}{2}}$ for these iterations. Combined with the bound on $lim\; inff\left(x_n\right)$, this implies that for infinitely many iterations,

$$f_{rec}^n\le f^{*}+{\displaystyle \frac{M_1\mu +M_2}{2}}+{\displaystyle \frac{\overline{\delta }}{2}}+\epsilon$$

(10)

For sufficiently small $\epsilon$, this means $f\left(x_n\right)\le f_{rec}^n-{\displaystyle \frac{\overline{\delta }}{2}}$ for infinitely many n, which would cause ${\delta }_l$ to remain constant by the condition in line 15. This contradicts our assumption that $l\to \infty$.

Case 2: $f\left(x_n\right)-f_{rec}^n<-{\displaystyle \frac{\overline{\delta }}{2}}$ for all but finitely many n.

This means $f\left(x_n\right)<f_{rec}^n-{\displaystyle \frac{\overline{\delta }}{2}}$ for almost all n. According to the update rule in line 15, this would cause ${\delta }_l$ to remain constant. Since we have established that $l\to \infty$, this is a contradiction.

Therefore, our assumption that ${\delta }_l\ge \overline{\delta }>{\delta }_{min}$ for all l must be false. Combined with our earlier observation that ${\delta }_l={\delta }_{min}$ for all sufficiently large l, we can conclude that $lim{inf}_{l\to \infty }{\delta }_l={\delta }_{min}$.

Thus, we have proven that exactly one of the three statements in the lemma must hold: either the algorithm terminates with $(0_{{\mathbb{R}}^d},0_{\mathbb{R}})\in {\partial }^{w}f\left(x_n\right)$, or it terminates with ${\delta }_l<{\delta }_{tol}$, or $l\to \infty$ and $lim{inf}_{l\to \infty }{\delta }_l={\delta }_{min}$.    □

Theorem 3 (Convergence to Neighborhood).

Let $\left\{x_n\right\}$ be the sequence generated by Algorithm 2. Then,

$$lim\underset{n\to \infty }{inf}f\left(x_n\right)\le f^{*}+{\displaystyle \frac{M_2}{2}}$$

(11)

where $M_2$ is as defined in Lemma 1.

Proof. 

We prove by contradiction. Assume the inequality does not hold. Then, there exists $\epsilon >0$, such that

$$lim\underset{n\to \infty }{inf}f\left(x_n\right)>f^{*}+{\displaystyle \frac{M_2}{2}}+\epsilon$$

(12)

By Lemma 2, exactly one of three cases must occur:

Case 1: The algorithm terminates with $(0_{{\mathbb{R}}^d},0_{\mathbb{R}})\in {\partial }^{w}f\left(x_n\right)$.

In this case, $x_n$ is an optimal point, so $f\left(x_n\right)=f^{*}$. This directly contradicts our assumption that $lim{inf}_{n\to \infty }f\left(x_n\right)>f^{*}+{\displaystyle \frac{M_2}{2}}+\epsilon$.

Case 2: The algorithm terminates with ${\delta }_l<{\delta }_{tol}$.

If the algorithm terminates after finitely many iterations, $lim{inf}_{n\to \infty }f\left(x_n\right)$ is not well defined. However, we can interpret our contradiction assumption as meaning $f\left(x_n\right)>f^{*}+{\displaystyle \frac{M_2}{2}}+\epsilon$ for the final iterate $x_n$. We will show this is impossible.

Case 3: The algorithm continues indefinitely with $l\to \infty$ and $lim{inf}_{l\to \infty }{\delta }_l={\delta }_{min}$.

This is the main case to analyze. Let $\widehat{y}\in X$ be a point such that $f\left(\widehat{y}\right)\le f^{*}+{\displaystyle \frac{\epsilon }{3}}$ (such a point exists by definition of $f^{*}$).

From our contradiction assumption, there exists an index $k_0$ such that for all $n\ge k_0$,

$$f\left(x_n\right)>f^{*}+{\displaystyle \frac{M_2}{2}}+{\displaystyle \frac{2\epsilon }{3}}$$

(13)

This implies that

$$f\left(x_n\right)-f\left(\widehat{y}\right)>\left(f^{*}+{\displaystyle \frac{M_2}{2}}+{\displaystyle \frac{2\epsilon }{3}}\right)-\left(f^{*}+{\displaystyle \frac{\epsilon }{3}}\right)={\displaystyle \frac{M_2}{2}}+{\displaystyle \frac{\epsilon }{3}}$$

(14)

Using Lemma 1 with $y=\widehat{y}$,

$$\begin{array}{cc}\hfill \parallel x_{n+1}-\widehat{y}{\parallel }^2& \le \parallel x_n-\widehat{y}{\parallel }^2-2{\alpha }_n\left(f\left(x_n\right)-f\left(\widehat{y}\right)\right)+M_1{\alpha }_n^2+M_2{\alpha }_n\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill & < \parallel x_n-\widehat{y}{\parallel }^2-2{\alpha }_n\left({\displaystyle \frac{M_2}{2}}+{\displaystyle \frac{\epsilon }{3}}\right)+M_1{\alpha }_n^2+M_2{\alpha }_n\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & = \parallel x_n-\widehat{y}{\parallel }^2-{\displaystyle \frac{2\epsilon {\alpha }_n}{3}}+M_1{\alpha }_n^2\hfill \end{array}$$

(17)

Let $N=\left\{n_j\right|j\in \mathbb{N},\mathrm{where}l\mathrm{increases}\mathrm{at}\mathrm{iteration}{n}_j\}$. Since we are in Case 3, where $l\to \infty$, this set is infinite.

From the step size calculation with our redefined $C^2={\sum }_{i=1}^m{D}_i^2$, and given that $f_{lev}^n=f_{rec}^n-{\delta }_l$, with ${\delta }_l$ approaching ${\delta }_{min}$, there are infinitely many iterations where

$${\alpha }_n\ge {\alpha }_{min}>0$$

(18)

For sufficiently large $n\ge n_0$, we can choose ${\alpha }_{min}$ small enough that

$$M_1{\alpha }_n<{\displaystyle \frac{\epsilon }{3}}$$

(19)

For these iterations, the inequality becomes

$$\parallel x_{n+1}-\widehat{y}{\parallel }^2\le \parallel x_n-\widehat{y}{\parallel }^2-{\displaystyle \frac{2\epsilon {\alpha }_n}{3}}+{\displaystyle \frac{\epsilon {\alpha }_n}{3}}={\parallel x_n-\widehat{y}\parallel }^2-{\displaystyle \frac{\epsilon {\alpha }_n}{3}}$$

(20)

Applying this inequality recursively for all iterations from $n_0$ to N,

$$\parallel x_{N+1}-\widehat{y}{\parallel }^2<{\parallel x_{n_0}-\widehat{y}\parallel }^2-{\displaystyle \frac{\epsilon }{3}}\sum _{n=n_0}^N{\alpha }_n$$

(21)

Since ${\alpha }_n\ge {\alpha }_{min}>0$ for all n, and N can be arbitrarily large, the sum ${\sum }_{n=n_0}^N{\alpha }_n$ can be made arbitrarily large as $N\to \infty$. This would eventually make $\parallel x_{N+1}-\widehat{y}{\parallel }^2$ negative, which contradicts the non-negativity of squared norms.

Therefore, our initial assumption must be false in all three cases, and we have

$$lim\underset{n\to \infty }{inf}f\left(x_n\right)\le f^{*}+{\displaystyle \frac{M_2}{2}}$$

(22)

This confirms that the convergence guarantee holds even with the redefined $C^2={\sum }_{i=1}^m{D}_i^2$, although the practical convergence behavior may differ due to the different step size calculations.    □

Proposition 4. (Convergence Rate).

Let $\left\{x_n\right\}$ be the sequence generated by Algorithm 2. Let $\left\{f_{rec}^n\right\}$ be the sequence of recorded best function values. Then, for any $\epsilon >0$, there exists an iteration number $N\left(\epsilon \right)$ such that

$$f_{rec}^n\le f^{*}+{\displaystyle \frac{M_2}{2}}+\epsilon \forall n\ge N\left(\epsilon \right)$$

(23)

Furthermore, if the algorithm parameters are chosen such that ${\delta }_l=max\{{\displaystyle \frac{{\delta }_0}{l}},{\delta }_{min}\}$ and ${\delta }_{min}$ is sufficiently small, then

$$f_{rec}^n-f^{*}-{\displaystyle \frac{M_2}{2}}=O\left({\displaystyle \frac{1}{\sqrt{n}}}\right)$$

(24)

Proof. 

From Theorem 3, we know that ${lim\; inf}_{n\to \infty }f\left(x_n\right)\le f^{*}+{\displaystyle \frac{M_2}{2}}$. Since $f_{rec}^n$ is a non-increasing sequence that tracks the best function value found so far, and $f_{rec}^n\le f\left(x_n\right)$ whenever $f\left(x_n\right)$ improves upon the previous best value, we have

$$\underset{n\to \infty }{lim}{f}_{rec}^n\le f^{*}+{\displaystyle \frac{M_2}{2}}$$

(25)

This implies that for any $\epsilon >0$, there exists $N\left(\epsilon \right)$ such that $f_{rec}^n\le f^{*}+{\displaystyle \frac{M_2}{2}}+\epsilon$ for all $n\ge N\left(\epsilon \right)$.

For the convergence rate, we need to analyze the case where the algorithm does not terminate finitely. By Lemma 2, this means $l\to \infty$ and $lim{inf}_{l\to \infty }{\delta }_l={\delta }_{min}$.

When ${\delta }_{min}$ is chosen to be sufficiently small, the target level update becomes effectively ${\delta }_l={\displaystyle \frac{{\delta }_0}{l}}$ for most iterations.

From the step size calculation (3) with $C^2={\sum }_{i=1}^m{D}_i^2$, to establish the claimed rate, we analyze the recurrence relation for $\parallel x_n-x^{*}{\parallel }^2$ from the inequality in Lemma 1. When ${\delta }_l={\displaystyle \frac{{\delta }_0}{l}}$, the step sizes ${\alpha }_n$ are related to ${\displaystyle \frac{1}{\sqrt{n}}}$ in order of magnitude.

Let $\left\{n_k\right\}$ be the sequence of iterations where a new best function value is found, i.e., $f_{rec}^{n_k}<f_{rec}^{n_k-1}$. For these iterations, we have $f\left(x_{n_k}\right)=f_{rec}^{n_k}$.

For each k, we can bound the progress as

$$\begin{array}{cc}\hfill \parallel x_{n_k}-x^{*}{\parallel }^2& \le \parallel x_0-x^{*}{\parallel }^2-2\sum _{j=0}^{n_k-1}{\alpha }_j(f\left(x_j\right)-f^{*})+\sum _{j=0}^{n_k-1}(M_1{\alpha }_j^2+M_2{\alpha }_j)\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill & \le \parallel x_0-x^{*}{\parallel }^2-2\sum _{j=0}^{n_k-1}{\alpha }_j(f_{rec}^j-f^{*})+\sum _{j=0}^{n_k-1}(M_1{\alpha }_j^2+M_2{\alpha }_j)\hfill \end{array}$$

(27)

Due to the relationship between ${\delta }_l$ and ${\displaystyle \frac{1}{l}}$, and consequently between ${\alpha }_n$ and ${\displaystyle \frac{1}{\sqrt{n}}}$, we can show that

$$\sum _{j=0}^{n-1}{\alpha }_j\sim O\left(\sqrt{n}\right)\mathrm{and}\sum _{j=0}^{n-1}{\alpha }_j^2\sim O\left(1\right)$$

(28)

Rearranging the inequality and using the fact that $\parallel x_{n_k}-x^{*}{\parallel }^2\ge 0$,

$$2\sum _{j=0}^{n_k-1}{\alpha }_j(f_{rec}^j-f^{*})\le {\parallel x_0-x^{*}\parallel }^2+\sum _{j=0}^{n_k-1}(M_1{\alpha }_j^2+M_2{\alpha }_j)$$

(29)

Since $f_{rec}^j$ is non-increasing, we have

$$2(f_{rec}^{n_k}-f^{*})\sum _{j=0}^{n_k-1}{\alpha }_j\le {\parallel x_0-x^{*}\parallel }^2+M_1\sum _{j=0}^{n_k-1}{\alpha }_j^2+M_2\sum _{j=0}^{n_k-1}{\alpha }_j$$

(30)

Dividing both sides by $2{\sum }_{j=0}^{n_k-1}{\alpha }_j$,

$$f_{rec}^{n_k}-f^{*}\le {\displaystyle \frac{\parallel x_0-x^{*}{\parallel }^2+M_1{\sum }_{j=0}^{n_k-1}{\alpha }_j^2}{2{\sum }_{j=0}^{n_k-1}{\alpha }_j}}+{\displaystyle \frac{M_2}{2}}$$

(31)

Using our earlier asymptotic bounds on the sums of step sizes, and noting that $n_k\le n$ for recording the best value at iteration n, we obtain

$$f_{rec}^n-f^{*}-{\displaystyle \frac{M_2}{2}}\le {\displaystyle \frac{\parallel x_0-x^{*}{\parallel }^2+O\left(1\right)}{2·O\left(\sqrt{n}\right)}}=O\left({\displaystyle \frac{1}{\sqrt{n}}}\right)$$

(32)

This establishes the claimed $O\left({\displaystyle \frac{1}{\sqrt{n}}}\right)$ convergence rate for the optimality gap $f_{rec}^n-f^{*}-{\displaystyle \frac{M_2}{2}}$.

The rate matches the best known rate for subgradient methods with diminishing step sizes, while the dynamic approach offers practical advantages through its automatic adaptation to the problem difficulty. The specific value of $C^2={\sum }_{i=1}^m{D}_i^2$ affects the constants in the convergence rate but not the asymptotic order.    □

The theoretical analysis demonstrates that the modified dynamic step size strategy achieves the same asymptotic convergence guarantees as the diminishing step size approach, with additional robustness provided by the minimum target level and step size parameters. As we will show in our numerical experiments, the dynamic approach offers significant practical advantages by automatically adjusting to the optimization landscape, leading to faster convergence and better solution quality across a variety of test problems.

4. Numerical Results

In this section, we present the results of numerical experiments conducted to evaluate the performance of incremental weak subgradient algorithms in non-smooth and non-convex settings. We begin by comparing the results with those obtained using state-of-the-art methods for standard non-convex benchmark problems. Although the proposed algorithm is inherently distributed, evaluating its behavior in a sequential setting still offers valuable insights, particularly regarding verifying the correctness and assessing convergence. Next, we analyze the algorithm’s performance on a binary classification task. Finally, we investigate the scalability of a preliminary parallel implementation to identify strengths and potential bottlenecks. All the tests were run with MATLAB R2024a on an Apple Mac equipped with an M3 Pro chip, featuring a 12-core CPU, 18-core GPU, and 36 GB of unified memory.

4.1. Benchmarking Optimization Methods

This subsection compares the proposed subgradient-based algorithms and classical and modern optimization solvers designed for non-smooth and non-convex problems. The baseline methods include MATLAB’s built-in fminsearch and patternsearch functions, representing well-established derivative-free approaches. Furthermore, we consider two popular adaptive gradient-based optimizers: Adam [23] and Adagrad [24], both widely used in large-scale optimization due to their ability to adjust learning rates during training.

Step size strategies for the subgradient-based methods are defined as follows:

• Constant step: ${\alpha }_k=0.01$
• Diminishing step: ${\alpha }_k={\displaystyle \frac{{\alpha }_0}{1+\beta k}}$, with ${\alpha }_0=0.1$, $\beta =0.5$
• Dynamic step: An adaptive strategy based on an incremental weak subgradient framework, where the subgradient bound C from definition (3) is not fixed a priori but adaptively estimated. Specifically, we maintain a vector $\mathbf{D}\in {\mathbb{R}}^m$, where each $D_i$ stores the maximum observed norm of the weak subgradient for the corresponding component function $f_i$. Initially, $D_i=1$ for all i, and during optimization, it is updated as follows:

  $$D_i=max\{D_i,\parallel g_i\parallel \}$$

  (33)

  where $g_i$ denotes the current weak subgradient of component i. The global subgradient bound is then computed as $C^2={\sum }_{i=1}^m{D}_i^2$. We fix ${\delta }_0=0.01,{\gamma }_0=0.5,ϵ={10}^{-6},{\alpha }_{min}={10}^{-5},{\alpha }_{max}={10}^{-1}$.

The evaluation is conducted across three benchmark problems:

• Rastrigin.

  $$f\left(x\right)=10n+\sum _{i=1}^n\left|x_i^2-10cos\left(2\pi x_i\right)\right|.$$

  (34)
  This function is a non-smooth variant of the classical Rastrigin function, characterized by a multimodal landscape with many local minima.
• Rosenbrock

  $$f\left(x\right)=\sum _{i=1}^{n-1}\left|100{(x_{i+1}-x_i^2)}^2+{(1-x_i)}^2\right|.$$

  (35)
  This is a non-smooth version of the Rosenbrock function, presenting a curved narrow valley with a challenging landscape.
• Smoothly clipped absolute deviation (SCAD)

  $$f\left(x\right):=\sum _{i=1}^n{\rho }_{\lambda ,a}\left(x_i\right)$$

  (36)

  where ${\rho }_{\lambda ,a}$ is defined as

  $${\rho }_{\lambda ,a}\left(x_i\right)=\left\{\begin{array}{cc}\lambda |x_i|\hfill & \mathrm{if}|x_i| \le \lambda ,\hfill \\ {\displaystyle \frac{2a\lambda |x_i|-x_i^2-{\lambda }^2}{2(a-1)}}\hfill & \mathrm{if}\lambda < |x_i| \le a\lambda ,\hfill \\ {\displaystyle \frac{(a+1){\lambda }^2}{2}}\hfill & \mathrm{if}|x_i| >a\lambda ,\hfill \end{array}\right.$$

  (37)

  with parameters $a=3.7$, $\lambda =0.01$.

Each problem is defined over a high-dimensional space with $n=1000$, and all experiments start from a randomly generated initial point $x_0\sim \mathcal{N}(0,I_n)$. The maximum number of iterations is set to 1000.

The results presented in

Table 1. Performance of optimization methods on three non-smooth and non-convex benchmark solvers. The table reports computation time (in seconds) and final objective value for each method and problem.

Problem	Method	Time [s]	${\mathit{f}}_{\mathbf{finale}}$
Ragistrin	Constant	0.061	$1.52\times {10}^4$
	Diminishing	0.037	$1.63\times {10}^4$
	Dynamic	0.036	$1.63\times {10}^4$
	Adam	15.470	$1.00\times {10}^4$
	Adagrad	16.378	$1.00\times {10}^4$
	Fminsearch	1.267	$1.64\times {10}^4$
	Patternsearch	22.766	$1.54\times {10}^4$
Rosembroack	Constant	0.044	$4.74\times {10}^6$
	Diminishing	0.021	$3.18\times {10}^5$
	Dynamic	0.026	$3.82\times {10}^5$
	Adam	3.056	$8.34\times {10}^4$
	Adagrad	3.053	$1.40\times {10}^5$
	Fminsearch	1.386	$4.49\times {10}^5$
	Patternsearch	10.194	$3.55\times {10}^5$
SCAD	Constant	0.026	$2.32\times {10}^{-1}$
	Diminishing	0.020	$2.32\times {10}^{-1}$
	Dynamic	0.015	$2.32\times {10}^{-1}$
	Adam	2.146	$2.27\times {10}^{-1}$
	Adagrad	2.140	$2.27\times {10}^{-1}$
	Fminsearch	2.135	$2.32\times {10}^{-1}$
	Patternsearch	12.383	$1.91\times {10}^{-1}$

demonstrate the effectiveness of the proposed subgradient-based methods, particularly the dynamic step size strategy, compared to classical and modern optimization algorithms. While adaptive gradient methods such as Adam and Adagrad achieve lower final objective values in some cases, their computational cost is considerably higher, especially noticeable in large-scale scenarios. The dynamic weak subgradient method achieves competitive final objective values with substantially reduced computation times relative to Adam and Adagrad. This efficiency gain is crucial for large-dimensional problems, where scalability and computational budget are often limiting factors. Derivative-free methods like fminsearch and patternsearch show reliable performance but suffer from significantly longer runtimes, rendering them less practical for high-dimensional problems. Moreover, by adaptively estimating subgradient bounds during optimization, the weak subgradient framework enables the dynamic method to adjust step sizes intelligently, balancing convergence speed and stability. This feature highlights its robustness and suitability for non-smooth, non-convex problems where traditional gradient-based methods might struggle. These results position the dynamic weak subgradient approach as a competitive and computationally efficient alternative to classical and adaptive gradient methods, making it particularly attractive for large-scale non-smooth optimization tasks.

4.2. Application of Incremental Weak Subgradient Methods to Classification

To validate the practical relevance of our theoretical framework, we apply the proposed incremental weak subgradient methods to a binary classification task with non-convex SCAD regularization. This setup reflects the challenges typical of modern machine learning, where optimization landscapes are often both non-smooth and non-convex.

We conducted experiments using the Breast Cancer Wisconsin dataset [25], a widely adopted benchmark for predictive breast cancer diagnosis, publicly available through the UCI Machine Learning Repository (https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic), accessed on 1 January 2025). The dataset comprises 569 instances, including 357 benign (62.7%) and 212 malignant (37.3%) cases. Each instance is described by 30 numerical features of cell nuclei (e.g., mean radius, texture, perimeter), extracted from digitized images of breast tissue biopsies. The classification task aims to distinguish between benign and malignant tumors. We transformed the dataset by (1) scaling features with different magnitudes, (2) introducing correlations between selected features, and (3) adding outliers to 5% of samples.

The classification task is formulated as a regularized empirical risk minimization problem:

$$\underset{w\in {\mathbb{R}}^d}{min}f\left(w\right):=\sum _{i=1}^m\mathcal{\ell }(y_i,x_i^{T}w)+\lambda \sum _{j=1}^d\rho \left(w_j\right)$$

(38)

where ℓ is the hinge loss function for binary classification:

$$\ell (y_i,x_i^{T}w)=max(0,1-y_i{x}_i^{T}w)$$

(39)

and the SCAD penalty $\rho$ promotes sparsity while avoiding the excessive penalization of large coefficients that occurs with conventional $L_1$ regularization, making it particularly valuable for feature selection in high-dimensional problems [26,27].

For the Breast Cancer dataset with approximately 400 training samples, the initial value of C is $\sqrt{400}=20$, which then adapts throughout training.

The effectiveness of the dynamic step size method is closely tied to the careful tuning of its hyperparameters. We employed grid-based cross-validation to explore the following parameter ranges: ${\delta }_0\in \{0.05,0.1,0.5,1,2,5,10,25,50\}$, $\gamma \in \{0.1,0.5,1.0,1.5,2.0\}$, $\lambda \in \{0.0001,0.001,0.01,0.05,0.1,0.5\}$, and $c_{\mathrm{init}}\in \{0.1,0.3,0.5,0.7,0.9\}$. The optimal configuration identified was ${\delta }_0=5.0$, $\gamma =1.5$, $\lambda =0.01$, and $c_{\mathrm{init}}=0.1$, which yielded the best validation performance. All methods were initialized consistently using random weights scaled by 0.01 to ensure comparability. For the computation of weak subgradients, we implemented an adaptive epsilon selection strategy, dynamically testing values $ϵ\in \{{10}^{-6},{10}^{-5},{10}^{-4},{10}^{-3}\}$ to maintain numerical stability throughout the optimization process. This meticulous hyperparameter tuning is crucial for the dynamic step size method to effectively adjust to the local geometry of the optimization landscape, thereby enhancing convergence speed and overall performance.

Performance evaluation was performed using five-fold cross-validation, with 70% of data used for training and 30% for testing. Early stopping was employed when the change in objective function value fell below ${10}^{-6}$ or after 200 iterations.

Table 2. Performance comparison of incremental weak subgradient methods for non-convex classification.

Method	Accuracy (%)	Conv. Time (s)	Rel. Improvement (%)
Case 1	59.64	1.43	–
Case 2	53.22	1.87	−10.76
Case 3	71.93	1.21	+17.84

presents the comparative performance of our three incremental weak subgradient methods on the classification task.

The results demonstrate that the dynamic step size strategy significantly enhances the performance of subgradient methods in the classification setting. The dynamic approach achieves 71.93% accuracy, representing a 17.84% improvement over the constant step size method and a 28.58% improvement over the diminishing step size approach. These results align with our theoretical analysis and highlight the importance of adaptive step sizing for non-convex classification problems.

Performance Analysis by Method:

• Constant Step Size (Case 1): As established in Proposition 1, this method converges to a neighborhood of the optimal solution, but the neighborhood size depends on the fixed step magnitude. This limits fine-tuning near the optimum in the complex SCAD-regularized classification landscape.
• Diminishing Step Size (Case 2): While Proposition 2 guarantees asymptotic convergence, the rigid step reduction schedule cannot adapt to the varying curvature and difficulty regions in the non-convex objective function, resulting in suboptimal classification performance.
• Dynamic Step Size (Case 3): Theorem 3 demonstrates that our adaptive approach provides both theoretical convergence guarantees and superior adaptivity to problem characteristics. The method automatically adjusts step sizes based on the local geometry of the SCAD-regularized hinge loss, enabling effective navigation of the non-convex optimization landscape.

The dynamic method demonstrates the shortest convergence time (1.21 s), indicating computational efficiency and improved accuracy. The adaptation range was measured at $18.4\times$ from maximum to minimum step size, illustrating the method’s ability to adjust to varying problem difficulties during optimization automatically. This adaptivity, combined with the $O(1/\sqrt{n})$ convergence rate established in Proposition 4, explains why the dynamic approach outperforms alternatives on challenging non-convex classification problems.

The accuracy results in Table 2 highlight the relative effectiveness of our dynamic step size method, though they remain below the 95%+ accuracy typically reported for the Breast Cancer Wisconsin dataset. This performance gap stems from several factors that point to promising directions for future research:

• Model Simplicity: We focused on a linear model with hinge loss and SCAD regularization to emphasize optimization aspects. Applying the weak subgradient framework to more expressive models (e.g., kernels, neural networks, ensembles) could yield better classification results.
• Challenging Preprocessing: Our deliberately complex preprocessing—featuring outliers, correlated features, and uneven scaling—complicates optimization. Future work could explore robust or adaptive preprocessing techniques to preserve difficulty while improving performance.
• Adaptive Hyperparameters: Parameters such as $c_{\mathrm{init}}$, $\gamma$, and ${\delta }_0$ could be tuned dynamically during training using adaptive schemes or meta-learning, enhancing both convergence and generalization.
• Optimization Enhancements: Incorporating momentum, variance reduction, or second-order approximations may accelerate convergence and improve final accuracy by better exploiting the problem’s structure.

These directions could significantly enhance the performance and applicability of weak subgradient methods in complex non-convex settings, helping bridge the gap with state-of-the-art classifiers.

5. First Scalability Evaluation in a Parallel Setting

In this work, we consider a preliminary parallel version of the incremental weak subgradient algorithm to explore the potential of the method, with a focus on analyzing its weak scalability. This approach evaluates how effectively the algorithm distributes a fixed computational workload per core as the total problem size increases proportionally with the number of processors. Such analysis provides a realistic and informative baseline, especially for large-scale optimization problems where computational resources are scaled accordingly. Moreover, it helps identify parallel overheads and potential inefficiencies in the current implementation before moving on to more demanding scalability studies, such as strong scaling. We tested the dynamic variant of the incremental weak subgradient method on the minimization of the SCAD function to assess weak scalability. The problem is decomposed into 10 fixed blocks (functions), each working on approximately $n/10$ variables. In our experiments, a constant workload of 500 variables per core was assigned, ensuring a fixed computational burden per processor. This configuration aligns with the principles of weak scalability by isolating the effects of increasing problem size under parallel execution.

The algorithm was parallelized using MATLAB’s parfor construct, which enables concurrent evaluation of weak subgradients for each component function. Each parallel worker computes the weak subgradient of a single component function independently. These partial gradients are then communicated back to the main process, where they are aggregated to form the full subgradient vector. This aggregation introduces synchronization at each iteration, resulting in communication overhead that tends to increase with the number of workers. As the optimization problem is unconstrained, the projection operator reduces to the identity map, eliminating additional costs associated with constraint handling.

Table 3. Execution times and time per variable for weak scalability test using the SCAD function decomposed into 10 fixed blocks.

Cores	Problem Dimension (n)	Execution Time [s]	Time per Variable [ms]
1	500	1.1911	2.38
2	1000	1.2654	1.27
4	2000	1.4477	0.72
8	4000	2.1513	0.54
12	6000	2.4738	0.41

reports the wall-clock execution times for increasing numbers of cores and corresponding problem sizes. As the problem dimension grows from 500 variables on a single core to 6000 variables on twelve cores, total execution time increases from 3.17 s to 23.58 s. While this growth is expected, it is not strictly proportional to the problem size, revealing the impact of synchronization and communication costs. Nevertheless, the average time per variable decreases from approximately 6.33 ms to 3.93 ms, indicating that the cost of parallelization is effectively amortized as the problem size increases. These results highlight the algorithm’s promising weak scalability properties, as it maintains near-constant per-variable efficiency despite larger workloads.

These preliminary results demonstrate that, even in its current form, the parallel implementation of the proposed method is well suited for weakly scalable problems with separable structure. The total execution time increases moderately with more cores, from 1.19 s on a single core to 2.47 s on twelve cores. This increase is expected due to overhead from parallel execution, such as communication and synchronization. However, the time per variable decreases significantly—from 2.38 ms at 1 core down to 0.41 ms at 12 cores—indicating that the algorithm achieves good weak scalability. The decreasing per-variable cost demonstrates that the parallel implementation effectively amortizes overhead as the problem size and computational resources grow. Future work may address these by introducing asynchronous updates to reduce synchronization delays, adopting adaptive step size strategies to improve convergence speed, and implementing dynamic load balancing to compensate for variability in subfunction evaluation times. Additionally, detailed profiling of communication and synchronization costs would provide valuable insight into performance bottlenecks, facilitating more refined optimizations on larger parallel systems.

6. Conclusions

We have extended incremental subgradient methods to handle non-smooth, non-convex optimization problems using weak subgradients. Our theoretical analysis confirms that all three variants—constant, diminishing, and dynamic step size strategies—converge to a neighborhood of the optimal solution, with the size of this neighborhood influenced by weak subgradient parameters.

Among these, the dynamic step size method stands out by automatically adapting to the local geometry of the objective function, leading to superior performance. Experimental results on classification tasks with non-convex SCAD regularization show that the dynamic method significantly outperforms classical approaches, achieving 71.93% accuracy compared to 59.64% and 53.22% for constant and diminishing step sizes, respectively.

Notably, the method’s strong point lies in its fast convergence and ease of implementation, making it a practical and computationally efficient alternative to classical and adaptive gradient methods. However, performance depends on careful hyperparameter tuning.

Future work will explore stochastic extensions, momentum incorporation, and scalability to large distributed systems. Overall, this approach offers a promising tool for solving complex non-convex optimization problems in machine learning and related domains, particularly when speed and simplicity of implementation are key considerations.