Dimension-Independent Convergence Rate for Adagrad with Heavy-Ball Momentum

Abstract

In this study, we analyze the convergence rate of Adagrad with momentum for non-convex optimization problems. We establish the first dimension-independent convergence rate under the $(L_0,L_1)$-smoothness assumption, which is a generalization of the standard L-smoothness. We show the $\mathcal{O}\left(\sqrt{1/T}\right)$ convergence rate under bounded noise in stochastic gradients, where the bound can scale with the current optimality gap and gradient norm.

1. Introduction

For a differentiable function $f:{\mathbb{R}}^d\to {\mathbb{R}}^1$ from the d-dimensional Euclidean space to the one-dimensional Euclidean space with $d\ge 1$, we consider the following minimization problem:

$$\underset{x\in {\mathbb{R}}^d}{min}f\left(x\right).$$

(1)

Stochastic iterative algorithms that leverage first-order derivative information, such as stochastic gradient descent (SGD), are popular tools for solving this problem (1). The step size heavily influences the convergence properties of these methods. However, tuning and adjusting this parameter during model training can be time-consuming and computationally expensive. To mitigate these challenges, various adaptive step size algorithms have been proposed.

One such adaptive algorithm, Adagrad (and its variants), has demonstrated strong empirical performance. To theoretically understand its performance, the convergence rates of Adagrad have been studied; however, most of the existing analyses focus on the in-expectation convergence rate, which cannot describe the success of a single run (or at most a few runs) of Adagrad in practice. Understanding the convergence property of these few runs requires a high-probability convergence guarantee in which the convergence rate has logarithmic dependency on the failure probability, and hence, recent research has increasingly focused on achieving high probability bounds.

In addition, the existing convergence rates for Adagrad are dimension-dependent. For instance, Hong and Lin [1] and Liu et al. [2] report rates that scale with $d^2$ where d denotes the input dimension. However, when optimizing with a high-dimensional input, as in deep learning, Adagrad shows a faster (or at least comparable) convergence speed compared to that of SGD, which has a dimension-independent convergence rate in terms of both expectation and high probability. This discrepancy between theory and practice implies that the theoretical convergence rates of Adagrad can be significantly improved, especially for dimension independence and high probability. 

In this work, we derive the first high-probability and dimension-independent convergence rates for Adagrad with momentum. Specifically, under $(L_0,L_1)$-smoothness, which is a generalization of traditional L-smoothness, and bounded gradient noise where the bound can scale with the current gradient norm and optimality gap, we show the $\mathcal{O}\left(\sqrt{1/T}\right)$ convergence rate of Adagrad with momentum for non-convex stochastic optimization problems (1), where T denotes the number of iterations.

2. Related Works

2.1. Convergence Analysis of Adagrad

Several works have analyzed Adagrad and its variants in the context of convex optimization, with extensions to variational inequality problems—for example, Kavis et al. [3], Bach and Levy [4], and Ene et al. [5]. For non-convex optimization, Li and Orabona [6] first analyzed the convergence rate of a modified version of Adagrad that did not use the latest gradient to compute the step size, deviating from the original algorithm. Subsequent works (e.g., Hong and Lin [1], Liu et al. [2], Défossez et al. [7], Kavis et al. [8], Wang et al. [9], Faw et al. [10], Attia and Koren [11], etc.) also studied the convergence rate for Adagrad (or Adagrad-Norm) and provided error bounds on ${\displaystyle \frac{1}{T}}{\sum }_{t=1}^T{\parallel \nabla f\left(x_t\right)\parallel }_2^2$ that scaled with the input dimension d.

In contrast to Adagrad or Adagrad-Norm, the convergence behavior of Adagrad with momentum has received relatively little attention (e.g., Hong and Lin [1], Li and Orabona [6], and Défossez et al. [7]). Empirically, in SGD with heavy-ball momentum, it is observed that smaller momentum factors $\beta$ (see $m_t$ in Algorithm 1) often lead to better training results. As a result, practitioners commonly set $\beta$ to a small constant (e.g., 0.01) or allow it to decrease over time. However, the theoretical understanding of this empirical observation is quite limited. For instance, Hong and Lin [1] derived a convergence rate inversely proportional to ${\beta }^2$. Défossez et al. [7] improve this rate; however, their rate is still inversely proportional to $\beta$. On the other hand, our convergence rates for Adagrad with momentum are not inversely proportional to $\beta$. A summary of these results is presented in Table 1.

Table 1. Summary of related works. ADM refers to Adagrad with momentum, ADS refers to Adagrad. $(L_0,0)$-smoothness denotes the standard smoothness assumption, and $(L_0,L_1)$-smoothness is its relaxed version (see Assumption 2). We only reveal the total number of Adagrad iterations T and the input dimension d in $\mathcal{O}(·)$ for the convergence rate.

Reference	Alg.	Smoothness	Convergence Rate
Hong and Lin [1]	ADM	$(L_0,L_1)$	$\mathcal{O}\left({\displaystyle \frac{d^{2}logT}{T}}+{\displaystyle \frac{dlogT}{\sqrt{T}}}\right)$
Liu et al. [2]	ADS	$(L_0,0)$	$\mathcal{O}\left({\displaystyle \frac{dlogT}{\sqrt{T}}}\right)$
Li and Orabona [6]	ADM	$(L_0,0)$	$\mathcal{O}\left({\displaystyle \frac{dlogT}{\sqrt{T}}}\right)$
Défossez et al. [7]	ADM	$(L_0,0)$	$\mathcal{O}\left({\displaystyle \frac{dlogT}{\sqrt{T}}}\right)$
Wang et al. [9]	ADS	$(L_0,0)$	$\mathcal{O}\left({\displaystyle \frac{dlogT}{\sqrt{T}}}\right)$
Ours (Theorem 1)	ADM	$(L_0,L_1)$	$\mathcal{O}\left({\displaystyle \frac{1}{\sqrt{T}}}\right)$

2.2. Stopping Time in Optimization

In the literature on stochastic approximation, stopping times have been widely employed either as analytical tools (Faw et al. [10], Patel et al. [12], and Patel [13]) or as components of the algorithm design (Ene et al. [5]). In most of these works, stopping time is used to test for the proximity to a stationary point or to ensure a sufficient decrease in the objective function. The notable exceptions of Li et al. [14] and Li et al. [15] utilize the stopping time to bound the sub-optimality gap, $f\left(x\right)-{inf}_{x}f\left(x\right)$. By leveraging the reverse direction of the Polyak–Lojasiewicz inequality, it follows that the gradient norm ${\parallel \nabla f\left(x\right)\parallel }_2$ can also be bounded. Using this idea, we used a stopping time analysis to explore scenarios in which the sub-optimality gap and gradient norm remained bounded. This approach integrates the stopping times as a practical mechanism for algorithm control and a theoretical framework for bounding the key metrics during optimization.

3. Preliminaries

3.1. Notations

For $x,y\in {\mathbb{R}}^d$,  $x^{\odot 2}$, $\sqrt{x}$ and $x\odot y$ denote the coordinate-wise square, square root, and coordinate-wise multiplication, respectively. The Euclidean norm and the standard inner product are denoted by $\parallel ·\parallel$ and $⟨·,·⟩$, respectively. For a positive semi-definite matrix $A\in {\mathbb{R}}^{n\times n}$ and $x\in {\mathbb{R}}^n$, ${\parallel x\parallel }_A^2$ denotes the quadratic form $x^{T}Ax$. For a vector $x=(x_1,\dots ,x_d)\in {\mathbb{R}}^d$ and a scalar value $z\in \mathbb{R}$, we use $z/x$ to denote $(z/x_1,\dots ,z/x_d)$ if $x_i\ne 0$ for all $i\in \{1,\dots ,x\}$ and use $z+x$ to denote $(z+x_1,\dots ,z+x_d)$. For symmetric matrices $A,B\in {\mathbb{R}}^{n\times n}$, we say $A⪯B$ if $B-A$ is positive semi-definite. For a matrix $A\in {\mathbb{R}}^{n\times n}$, let $\parallel A\parallel$ be the spectral norm of A.

Finally, for $x\in \mathbb{R}$, $\lfloor x\rfloor$ denotes the floor function, which maps x to the greatest integer less than or equal to x.

3.2. Problem Setup and Assumptions

Throughout this paper, we consider an algorithm for Adagrad with (heavy-ball) momentum (Algorithm 1) applied to a non-convex objective function (1).

Algorithm 1 Adagrad with heavy-ball momentum

Require:  A learning rate $\eta >0$, momentum coefficient $0<\beta <1$, small constant $\lambda >0$ to prevent division by zero, initial parameter $x_1\in {\mathbb{R}}^d$. 1: for $t=1$ to T do, 2:     Draw a stochastic gradient $g_t$ such that $\mathbb{E}\left[g_t\right|g_1,\dots ,g_{t-1}]=f\left(x_t\right)$. 3:     Update the momentum vector $m_t\leftarrow \left\{\begin{array}{ll}{g}_t,& \mathrm{if}t=1,\\ (1-\beta )m_{t-1}+\beta g_t,& \mathrm{if}t>1\end{array}\right.$. 4:     Compute the next iterate $x_{t+1}\leftarrow x_t-{\displaystyle \frac{\eta }{\sqrt{{\sum }_{k=1}^t{g}_k^{\odot 2}+\lambda }}}\odot m_t$. 5: end for

Throughout this paper, we assume that the objective function f in (1) is bounded below.

Assumption A1. 

f is bounded below by its finite infimum $f^{🟉}:=f\left(x^{🟉}\right)>-\infty$.

We also assume that f is the $(L_0,L_1)$-smoothness for some $L_0,L_1>0$.

Assumption A2. 

f is $(L_0,L_1)$-smooth, $i.e.,$ f is differentiable, and for any $x,y\in {\mathbb{R}}^d$ satisfying $\parallel x-y\parallel \le 1/L_1$,

$$\parallel \nabla f\left(x\right)-\nabla f\left(y\right)\parallel \le (L_0+L_1\parallel \nabla f\left(y\right)\parallel )\parallel x-y\parallel .$$

(2)

For a twice-differentiable function f, Assumption 2 is strictly weaker than the standard L-smoothness, as the L-smoothness condition is equivalent to the $(L,0)$-smoothness condition and there are functions that are $(L_0,L_1)$-smooth for some $L_0,L_1$ but not L-smooth for all L (see Lemma A1 and Zhang et al. [16]). Empirical evidence shows that many practical objective functions satisfy (2) while they do not satisfy the L-smoothness assumption (e.g., large language models [17]). For a more detailed discussion, see Appendix A.

We consider the following assumption on the noise in the stochastic gradients.

Assumption A3. 

${\sigma }_0,{\sigma }_1,{\sigma }_2>0$ exist such that for each $t\in \mathbb{N}$,

$$\parallel g_t-\nabla f\left(x_t\right)\parallel \le {\sigma }_0+{\sigma }_1{\parallel \nabla f\left(x_t\right)\parallel }^2+{\sigma }_2(f\left(x_t\right)-f^{🟉}).$$

Assumption 3 relaxes the standard bounded noise assumption by allowing the error bound on the stochastic gradient noise to increase with the gradient norm square and the optimality gap. For further details on the stochastic noise assumptions, please refer to Khaled and Richtárik [18].

4. The High-Probability and Dimension-Independent Convergence Rate of Adagrad with Momentum

We are now ready to present our main results. In this section, we present the convergence rate (Theorem 1) and iteration complexity (Corollary 1) of Adagrad with momentum to find an $ϵ$-stationary point. We now introduce our high-probability convergence analysis of Algorithm 1 under Assumptions 1–3.

Theorem 1. 

Let $x_t$ be generated by Adagrad with heavy-ball momentum under Assumptions 1–3. Let $\Delta =f\left(x_1\right)-f^{🟉}$, $\iota =\sqrt{log\left(1/\delta \right)}$, $\sigma ={\sigma }_0+{\sigma }_1{M}^2+{\sigma }_{2}G$, and

$$\begin{array}{cc}\hfill & G=29·max\left\{\Delta ,{\displaystyle \frac{1}{L_1\sqrt{\lambda }}}\right\},M={\displaystyle \frac{4L_{1}G+\sqrt{16L_1^2{G}^2+24L_{0}G}}{3}},\hfill \\ \hfill & 0<\beta \le min\left\{1,{\displaystyle \frac{1}{M^2}},{\displaystyle \frac{1}{{\sigma }^2\iota }},{\displaystyle \frac{1}{{\sigma }^2}}\right\},L=L_0+L_{1}M,\eta \le min\left\{{\displaystyle \frac{1}{L_1}},{\displaystyle \frac{\sqrt{\lambda }{\beta }^2}{6L}},{\displaystyle \frac{\sqrt{\beta }\sigma }{L}}\right\}.\hfill \end{array}$$

Then, for any natural number $t\in [1,T]$ with $T\in \mathbb{N}$ such that $T\le 1/{\beta }^2$ and for any $\delta \in (0,1)$, it holds with probability of at least $1-\delta$ that

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{T}}\sum _{t=1}^T{\parallel \nabla f\left(x_t\right)\parallel }^2& \le {\displaystyle \frac{2\Delta {\beta }^2(M+\sigma )}{\eta \sqrt{T}}}+{\displaystyle \frac{2\Delta {\beta }^2\sqrt{\lambda }}{\eta T}}\hfill \\ \hfill & +\left({\displaystyle \frac{2{\beta }^2(M+\sigma )}{\sqrt{\lambda T}}}+{\displaystyle \frac{2{\beta }^2(M+\sigma )}{T}}\right)\left[1+4\beta {\sigma }^2+4{\beta }^2{\sigma }^2+18(1-\beta )\beta {\sigma }^2\iota \right].\hfill \end{array}$$

Compared to prior works analyzing the convergence rate of Adagrad with momentum [1,6,7], Theorem 1 offers several key improvements. Notably, the error bounds in prior works scale with $1/\beta$ and d, whereas our bound does not. Furthermore, our bounds do not deteriorate for a smaller $\beta$ value, which aligns well with empirical observations: a smaller $\beta$ often yields better training results. Furthermore, when compared to [1], which derives its results from the same assumptions as ours, our convergence rate has no $logT$ factor. By using Theorem 1, we can also compute the iteration complexity of Adagrad with momentum to obtain an $ϵ$-stationary point.

Corollary 1. 

Let $x_t$ be generated by Adagrad with heavy-ball momentum under Assumptions 1–3. Let $ϵ\in (0,1)$, $\Delta =f\left(x_1\right)-f^{🟉}$, $\iota =\sqrt{log\left(1/\delta \right)}$, and

$$\begin{array}{cc}\hfill & G=29·max\left\{\Delta ,{\displaystyle \frac{1}{L_1\sqrt{\lambda }}}\right\},\hfill \\ \hfill & M={\displaystyle \frac{4L_{1}G+\sqrt{16L_1^2{G}^2+24L_{0}G}}{3}},\hfill \\ \hfill & \sigma ={\sigma }_0+{\sigma }_1{M}^2+{\sigma }_{2}G,L=L_0+L_{1}M\hfill \\ \hfill & 0<\beta \le min\left\{{\displaystyle \frac{1}{2}},{\displaystyle \frac{6L}{L_1\sqrt{\lambda }}},{\displaystyle \frac{6\sigma }{\sqrt{\lambda }}},{\displaystyle \frac{1}{M^2}},{\displaystyle \frac{1}{{\sigma }^2\iota }},{\displaystyle \frac{1}{{\sigma }^2}},{\displaystyle \frac{1}{M}},{\displaystyle \frac{1}{\sigma }},{\displaystyle \frac{1}{\sigma \iota }}\right.\hfill \\ \hfill & \left.{\displaystyle \frac{\sqrt{\lambda }{ϵ}^2}{144\Delta L(M+\sigma )}},{\displaystyle \frac{ϵ}{12}}\sqrt{{\displaystyle \frac{1}{L\Delta }}},{\displaystyle \frac{\sqrt[4]{\lambda }ϵ}{12\sqrt{M+\sigma }}},{\displaystyle \frac{{ϵ}^2}{144}}\right\},\hfill \\ \hfill & \eta ={\displaystyle \frac{\sqrt{\lambda }{\beta }^2}{6L}},T=\lfloor 1/{\beta }^2\rfloor .\hfill \end{array}$$

Then, for any natural number $t\in [1,T]$ and for any $\delta \in (0,1)$, it holds with probability of at least $1-\delta$ that

$$\begin{array}{c}\hfill {\displaystyle \frac{1}{T}}\sum _{t=1}^T{\parallel \nabla f\left(x_t\right)\parallel }^2\le {ϵ}^2.\end{array}$$

Under a constant failure probability $\delta$, Corollary 1 implies that choosing $\beta =\Theta \left({ϵ}^2\right)$ and $T=\mathcal{O}\left({ϵ}^{-4}\right)$ suffices to find an $ϵ$-stationary point x such that $\parallel \nabla f\left(x\right)\parallel \le ϵ$. Here, we hide all terms other than $ϵ$ in the big-O and big-$\Theta$ notations.

5. Proof of Theorem 1

To prove Theorem 1, we first introduce the stopping time that is used to bound the function value and the gradient norm as Adagrad iterates. We then present key lemmas (Lemmas 1–3) and derive Theorem 1 in Section 5.1. We describe the high-level idea behind our proof in Section 5.2. We present all of the technical lemmas in Section 5.3 and the proofs of Lemmas 1–3 in Section 5.4, Section 5.5 and Section 5.6, respectively.

5.1. Proof of Theorem 1

We define the stopping time as

$$\tau =min\left\{t\right|f\left(x_t\right)-f^{🟉}>G\}\wedge (T+1)$$

where $a\wedge b$ denotes $min(a,b)$. In other words, the optimality gap is bounded by G until time $\tau$. We note that this also implies bounded gradients under the $(L_0,L_1)$-smoothness (see Lemma 6 in Section 5.3).

Using the definition of the stopping time $\tau$, we introduce the following key lemmas to prove Theorem 1.

Lemma 1. 

Suppose that Assumptions 1–3 hold. Furthermore, the definitions of $G,M,L,$ and σ, as well as the conditions of η, β, and T, are identical to those in Theorem 1. Then, the iterates ${\left\{x_t\right\}}_{t<\tau }$ generated by Algorithm 1 satisfy the following:

$$\begin{array}{c}\hfill f\left(x_{\tau }\right)-f^{🟉}\le f\left(x_1\right)-f^{🟉}+{\displaystyle \frac{\eta {\beta }^2}{\sqrt{\lambda }}}\sum _{t=1}^{\tau -1}{\parallel {ϵ}_t\parallel }^2,\end{array}$$

and

$$\begin{array}{cc}\hfill \sum _{t=1}^{\tau -1}{\parallel \nabla f\left(x_t\right)\parallel }^2\le & {\displaystyle \frac{2{\beta }^2\Delta \sqrt{{(M+\sigma )}^2(\tau -1)+\lambda }}{\eta }}\hfill \\ & +{\displaystyle \frac{2{\beta }^4\sqrt{{(M+\sigma )}^2(\tau -1)+\lambda }}{\sqrt{\lambda }}}\sum _{t=1}^{\tau -1}{\parallel {ϵ}_t\parallel }^2\hfill \end{array}$$

where ${ϵ}_t=m_t-\nabla f\left(x_t\right)$.

Lemma 2. 

Suppose that Assumptions 1–3 hold. Furthermore, the definitions of $G,M,L,\sigma$, as well as the conditions of η, β, and T, are identical to those in Theorem 1. Then, for any $\delta \in (0,1)$, with probability at least $1-\delta$,

$$\begin{array}{cc}\hfill \sum _{t=1}^{\tau -1}{\parallel {ϵ}_t\parallel }^2\le & {\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^2(\tau -2)+2\beta {\sigma }^2(\tau -2)+18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2\hfill \end{array}$$

where ${ϵ}_t=m_t-\nabla f\left(x_t\right)$.

Lemma 3. 

Suppose that Assumptions 1–3 hold. Furthermore, the definitions of $G,M,L,$ and σ, as well as the conditions of η, β, and T, are identical to those in Theorem 1. Suppose that

$$\begin{array}{cc}\hfill f\left(x_{\tau }\right)-f^{🟉}& \le f\left(x_1\right)-f^{🟉}+{\displaystyle \frac{\eta {\beta }^2}{\sqrt{\lambda }}}[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^2(\tau -2)+2\beta {\sigma }^2(\tau -2)\hfill \\ \hfill & +18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2].\hfill \end{array}$$

Then, it holds that

$$\tau =T+1.$$

We now prove Theorem 1 using Lemmas 1–3.

Proof of Theorem 1. 

According to Lemmas 1–3, for any $\delta \in (0,1)$, with a probability of at least $1-\delta$, we have

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{T}}\sum _{t=1}^T{\parallel \nabla f\left(x_t\right)\parallel }^2& \le {\displaystyle \frac{2\Delta {\beta }^2\sqrt{{(M+\sigma )}^{2}T+\lambda }}{\eta T}}+{\displaystyle \frac{2{\beta }^4\sqrt{{(M+\sigma )}^{2}T+\lambda }}{\sqrt{\lambda }T}}\left[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^{2}T+2\beta {\sigma }^{2}T\right.\hfill \\ \hfill & \left.+18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2\right]\hfill \\ \hfill & \le {\displaystyle \frac{2\Delta {\beta }^2\sqrt{{(M+\sigma )}^{2}T+\lambda }}{\eta T}}+{\displaystyle \frac{2{\beta }^2\sqrt{{(M+\sigma )}^{2}T+\lambda }}{\sqrt{\lambda }T}}[2\beta {\sigma }^2+1+2\beta {\sigma }^2\hfill \\ \hfill & +18(1-\beta ){\sigma }^2\beta \sqrt{\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\beta }^2{\sigma }^2]\hfill \end{array}$$

(3)

where the second inequality comes from $\beta \le 1/M^2$, $T\le 1/{\beta }^2$.

By applying $\sqrt{a+b}\le \sqrt{a}+\sqrt{b}$ to the RHS, we obtain the bound in Theorem 1. □

5.2. The High-Level Idea Behind the Proof of Theorem 1

In this section, we illustrate the main idea behind our proof and how we use the technical lemmas in Section 5.3. Our proof mainly relies on the stopping time result (Lemma 3), which enables us to bound the optimality gap until the end of Adagrad with momentum. This enables us to bound the gradient norm (Lemma 7) and treat the $(L_0,L_1)$-smoothness as the standard smoothness assumption (Lemma 5). Furthermore, under the bounded gradient norm (and the bounded number of iterations T), we can derive the upper and lower bounds on the adaptive step size (Lemma 7). Using these observations and analyzing the bias between the update vector $m_t$ and the true gradient (Lemma 2), we can derive our dimension-independent convergence rate.

5.3. Technical Lemmas

We introduce the following technical lemmas.

Lemma 4. 

The iterates ${\left\{x_t\right\}}_{t\in T}$ generated by Algorithm 1 satisfy the condition of for all $t\ge 1$,

$$\parallel x_{t+1}-x_t\parallel \le \eta \sqrt{\beta }\le \eta .$$

Moreover, for any function $f:{\mathbb{R}}^d\to \mathbb{R}$ satisfying Assumption 2, it holds that

$$\parallel \nabla f\left(x_{t+1}\right)-\nabla f\left(x_t\right)\parallel \le (L_0+L_1\parallel \nabla f\left(x_t\right)\parallel )\eta \sqrt{\beta }\le (L_0+L_1\parallel \nabla f\left(x_t\right)\parallel )\eta .$$

Proof. 

According to the definition of Algorithm 1, we have

$$\begin{array}{cc}\hfill \parallel x_{t+1}-x_t\parallel & =\eta ∥{\displaystyle \frac{\beta {\sum }_{k=1}^t{(1-\beta )}^{t-k}{g}_k}{\sqrt{{\sum }_{k=1}^t{g}_k^{\odot 2}+\lambda }}}∥\le \eta ∥{\displaystyle \frac{\beta {\sum }_{k=1}^t{(1-\beta )}^{t-k}{g}_k}{\sqrt{{\sum }_{k=1}^t{(1-\beta )}^{t-k}{g}_k^{\odot 2}}}}∥\hfill \\ \hfill & \le \eta \beta ∥{\displaystyle \frac{\sqrt{{\sum }_{k=1}^t{(1-\beta )}^{t-k}{g}_k^{\odot 2}}\sqrt{{\sum }_{k=1}^t\frac{{(1-\beta )}^{2t-2k}}{{(1-\beta )}^{t-k}}}}{\sqrt{{\sum }_{k=1}^t{(1-\beta )}^{t-k}{g}_k^{\odot 2}}}}∥\hfill \\ \hfill & =\eta \beta \left|\sqrt{\sum _{k=1}^t{(1-\beta )}^{t-k}}\right|=\eta \beta \left(\sqrt{{\displaystyle \frac{1}{\beta }}-{\displaystyle \frac{{(1-\beta )}^t}{\beta }}}\right)\hfill \\ \hfill & \le \eta \sqrt{\beta }.\hfill \end{array}$$

Here, the second inequality follows from Cauchy’s inequality, and for the last equality, we use the sum formula of the geometric sequence. □

Lemma 5 

(Lemma from Faw et al. [10] and Zhang et al. [16]). For any function $f:{\mathbb{R}}^d\to \mathbb{R}$ satisfying Assumption 2, the sequence of iterates ${\left\{x_t\right\}}_{t\in T}$ generated by Algorithm 1 with $\eta \le 1/L_1$ satisfy the condition of for all $t\ge 1$,

$$f\left(x_{t+1}\right)\le f\left(x_t\right)+⟨\nabla f\left(x_t\right),x_{t+1}-x_t⟩+{\displaystyle \frac{L_0+L_1\parallel \nabla f\left(x_t\right)\parallel }{2}}{\parallel x_{t+1}-x_t\parallel }^2.$$

Lemma 6. 

For any function $f:{\mathbb{R}}^d\to \mathbb{R}$ satisfying Assumptions 1–2, the following inequality holds:

$${\parallel \nabla f\left(x\right)\parallel }^2\le {\displaystyle \frac{8(L_0+L_1\parallel \nabla f\left(x\right)\parallel )}{3}}(f\left(x\right)-f^{🟉}).$$

Proof. 

Let $y=x-{\displaystyle \frac{1}{2(L_0+L_1\parallel \nabla f\left(x\right)\parallel )}}\nabla f\left(x\right)$. Then, we have

$$\begin{array}{c}\hfill \parallel y-x\parallel ={\displaystyle \frac{\parallel \nabla f\left(x\right)\parallel }{2L_0+2L_1\parallel \nabla f\left(x\right)\parallel }}\le {\displaystyle \frac{\parallel \nabla f\left(x\right)\parallel }{2L_1\parallel \nabla f\left(x\right)\parallel }}={\displaystyle \frac{1}{2L_1}}\le {\displaystyle \frac{1}{L_1}}.\end{array}$$

This implies that

$$\begin{array}{cc}\hfill f^{🟉}-f\left(x\right)\le f\left(y\right)-f\left(x\right)& \le ⟨\nabla f\left(x\right),y-x⟩+{\displaystyle \frac{L_0+L_1\parallel \nabla f\left(x\right)\parallel }{2}}{\parallel y-x\parallel }^2\hfill \\ \hfill & =-{\displaystyle \frac{3}{8(L_0+L_1\parallel \nabla f\left(x\right)\parallel )}}{\parallel \nabla f\left(x\right)\parallel }^2\hfill \end{array}$$

where the second inequality is from Lemma 5. □

Lemma 7. 

Suppose that Assumptions 1–3 hold. Furthermore, the definitions of $G,M,L$, and σ, as well as the conditions of η, β, and T, are identical to those in Theorem 1. Suppose that $f\left(x_t\right)-f^{*}\le G$ for all $t<\tau$ for a given τ. Then, it holds that

$$\begin{array}{cc}\hfill & \parallel \nabla f\left(x_t\right)\parallel \le M,\hfill \\ \hfill & f\left(x_{t+1}\right)\le f\left(x_t\right)+⟨\nabla f\left(x_t\right),x_{t+1}-x_t⟩+{\displaystyle \frac{L}{2}}{\parallel x_{t+1}-x_t\parallel }^2,\hfill \\ \hfill & \parallel g_t\parallel \le M+\sigma ,\hfill \\ \hfill & {\displaystyle \frac{\eta }{\sqrt{{(M+\sigma )}^{2}t+\lambda }}}\mathbf{I}⪯H_t:=\mathrm{diag}\left({\displaystyle \frac{\eta }{\sqrt{{\sum }_{k=1}^t{g}_k^{\odot 2}+\lambda }}}\right)⪯{\displaystyle \frac{\eta }{\sqrt{\lambda }}}\mathbf{I}\hfill \end{array}$$

where $\mathbf{I}$ denotes the $d\times d$ identity matrix and diag$\left(v\right)$ for $v\in {\mathbb{R}}^n$ denotes the diagonal matrix whose diagonal entries are given by the components of the vector v.

Proof. 

According to Lemma 6, we have the following inequalities:

$$\begin{array}{cc}\hfill & {\parallel \nabla f\left(x_t\right)\parallel }^2\le {\displaystyle \frac{8(L_0+L_1\parallel \nabla f\left(x_t\right)\parallel )}{3}}(f\left(x_t\right)-f^{🟉})\hfill \\ \hfill & {\displaystyle \frac{3{\parallel \nabla f\left(x_t\right)\parallel }^2}{8(L_0+L_1\parallel \nabla f\left(x_t\right)\parallel )}}\le f\left(x_t\right)-f^{🟉}.\hfill \end{array}$$

Define the function $g\left(k\right):={\displaystyle \frac{3k^2}{8(L_0+L_{1}k)}}$ over $k\ge 0$. It is straightforward to verify that $g^{\prime }\left(k\right)\ge 0$ for all $k\ge 0$, which implies that $g\left(k\right)$ is an increasing function and is therefore invertible. Consequently, $g^{-1}$ is also an increasing function and is defined as follows:

$$g^{-1}\left(y\right)={\displaystyle \frac{4L_{1}y+\sqrt{16L_1^2{y}^2+24L_{0}y}}{3}}.$$

If $f\left(x\right)-f^{🟉}\le G$, then

$$\parallel \nabla f\left(x\right)\parallel \le g^{-1}(f\left(x\right)-f^{🟉})\le g^{-1}\left(G\right)=M.$$

This implies that for all $t<\tau$, we have $f\left(x_t\right)-f^{🟉}\le G$ and $\parallel \nabla f\left(x_t\right)\parallel \le M$. Then, $L_0+L_1\parallel \nabla f\left(x_t\right)\parallel \le L_0+L_{1}M$. Using this bound, we can apply the descent lemma (refer to Lemma 5) as the starting point for the proof. Specifically, for all $t<\tau$, if $\eta \le 1/L_1$, the descent lemma gives

$$\begin{array}{cc}\hfill f\left(x_{t+1}\right)& \le f\left(x_t\right)+⟨\nabla f\left(x_t\right),x_{t+1}-x_t⟩+{\displaystyle \frac{L_0+L_1\parallel \nabla f\left(x_t\right)\parallel }{2}}{\parallel x_{t+1}-x_t\parallel }^2\hfill \\ \hfill & \le f\left(x_t\right)+⟨\nabla f\left(x_t\right),x_{t+1}-x_t⟩+{\displaystyle \frac{L}{2}}{\parallel x_{t+1}-x_t\parallel }^2.\hfill \end{array}$$

Next, according to Assumption 3, for all $t<\tau$, it holds that

$$\parallel g_t\parallel \le \parallel g_t-\nabla f\left(x_t\right)\parallel +\parallel \nabla f\left(x_t\right)\parallel \le M+\sigma .$$

□

Lemma 8 

(Young’s inequality with $ϵ$). For any $ϵ>0$ and $a,b\in \mathbb{R}$, we have

$$ab\le ϵa^2+{\displaystyle \frac{b^2}{4ϵ}}.$$

Proof. 

Observe that

$${\left(\sqrt{ϵ}a-{\displaystyle \frac{b}{2\sqrt{ϵ}}}\right)}^2=ϵa^2-ab+{\displaystyle \frac{b^2}{4ϵ}}\ge 0$$

By rearranging the terms, we obtain the desired result. □

Lemma 9. 

For any $k>0$ and $a,b\in {\mathbb{R}}^n$, we have

$${\parallel a+b\parallel }^2\le (1+k){\parallel a\parallel }^2+\left(1+{\displaystyle \frac{1}{k}}\right){\parallel b\parallel }^2.$$

Proof. 

We want to show

$$(1+k){\parallel a\parallel }^2+\left(1+{\displaystyle \frac{1}{k}}\right){\parallel b\parallel }^2-({\parallel a\parallel }^2+{\parallel b\parallel }^2)-2⟨a,b⟩\ge 0,$$

which is equivalent to

$$k{\parallel a\parallel }^2+{\displaystyle \frac{1}{k}}{\parallel b\parallel }^2-2⟨a,b⟩\ge 0.$$

Since

$${∥\sqrt{k}a-{\displaystyle \frac{1}{\sqrt{k}}}b∥}^2=k{\parallel a\parallel }^2+{\displaystyle \frac{1}{k}}{\parallel b\parallel }^2-2⟨a,b⟩\ge 0,$$

we obtain the desired results. □

Lemma 10 

(The Azuma–Hoeffding inequality). Let $Z_1,Z_2,\dots ,Z_T$ be a martingale difference sequence with respect to a filtration $F_{t-1}$. Suppose that there is a constant b such that for any t,

$$\mathbb{P}\left(\right|Z_t|\le b)=1.$$

Then, for any positive integer T and for any $\delta >0$, it holds with a probability of at least $1-\delta$ that

$${\displaystyle \frac{1}{T}}\sum _{t=1}^T{Z}_t\le b\sqrt{{\displaystyle \frac{2log(1/\delta )}{T}}}.$$

5.4. Proof of Lemma 1

For all $t<\tau$, under the condition $\eta \le 1/L_1$, the following holds according to Lemma 7:

$$\begin{array}{cc}\hfill f\left(x_{t+1}\right)& \le f\left(x_t\right)+⟨\nabla f\left(x_t\right),x_{t+1}-x_t⟩+{\displaystyle \frac{L}{2}}{\parallel x_{t+1}-x_t\parallel }^2\hfill \\ \hfill & =f\left(x_t\right)-\nabla f{\left(x_t\right)}^T{H}_t{m}_t+{\displaystyle \frac{L}{2}}{m}_t^T{H}_t^2{m}_t\hfill \\ \hfill & \le f\left(x_t\right)-\nabla f{\left(x_t\right)}^T{H}_t(m_t-\nabla f\left(x_t\right))-{\parallel \nabla f\left(x_t\right)\parallel }_{H_t}^2+{\displaystyle \frac{\eta L}{2\sqrt{\lambda }}}{\parallel m_t\parallel }_{H_t}^2\hfill \\ \hfill & =f\left(x_t\right)-{\parallel \nabla f\left(x_t\right)\parallel }_{H_t}^2-\nabla f{\left(x_t\right)}^T{H}_t{ϵ}_t+{\displaystyle \frac{\eta L}{2\sqrt{\lambda }}}{\parallel \nabla f\left(x_t\right)+{ϵ}_t\parallel }_{H_t}^2\hfill \\ \hfill & \underset{\left(a\right)}{\le }f\left(x_t\right)-{\displaystyle \frac{2}{3{\beta }^2}}{\parallel \nabla f\left(x_t\right)\parallel }_{H_t}^2+{\displaystyle \frac{3{\beta }^2}{4}}{\parallel {ϵ}_t\parallel }_{H_t}^2+{\displaystyle \frac{\eta L}{\sqrt{\lambda }}}({\parallel \nabla f\left(x_t\right)\parallel }_{H_t}^2+{\parallel {ϵ}_t\parallel }_{H_t}^2)\hfill \\ \hfill & \underset{\left(b\right)}{\le }f\left(x_t\right)-{\displaystyle \frac{1}{2{\beta }^2}}{\parallel \nabla f\left(x_t\right)\parallel }_{H_t}^2+{\beta }^2{\parallel {ϵ}_t\parallel }_{H_t}^2\hfill \end{array}$$

where ${ϵ}_t:=m_t-\nabla f\left(x_t\right)$. For $\left(a\right)$, we apply Lemma 8 with $ϵ={\displaystyle \frac{1}{3{\beta }^2}}$ and Cauchy’s inequality. For $\left(b\right)$, we utilize the step size condition $\eta \le {\displaystyle \frac{\sqrt{\lambda }{\beta }^2}{6L}}\le {\displaystyle \frac{\sqrt{\lambda }}{6L{\beta }^2}}$ for $0<\beta <1$.

Then, according to the lower bound and upper bound of $H_t$ in Lemma 7, we have

$$\begin{array}{cc}\hfill & f\left(x_{t+1}\right)\le f\left(x_t\right)-{\displaystyle \frac{\eta }{2{\beta }^2\sqrt{{(M+\sigma )}^{2}t+\lambda }}}{\parallel \nabla f\left(x_t\right)\parallel }^2+{\displaystyle \frac{\eta {\beta }^2}{\sqrt{\lambda }}}{\parallel {ϵ}_t\parallel }^2.\hfill \end{array}$$

By rearranging the above inequality, we have

$$\begin{array}{cc}\hfill & {\displaystyle \frac{\eta }{2{\beta }^2\sqrt{{(M+\sigma )}^{2}t+\lambda }}}{\parallel \nabla f\left(x_t\right)\parallel }^2\le f\left(x_t\right)-f\left(x_{t+1}\right)+{\displaystyle \frac{\eta {\beta }^2}{\sqrt{\lambda }}}{\parallel {ϵ}_t\parallel }^2.\hfill \end{array}$$

This implies that for all $t<\tau$, it holds that

$$\begin{array}{cc}\hfill {\displaystyle \frac{\eta }{2{\beta }^2\sqrt{{(M+\sigma )}^2(\tau -1)+\lambda }}}{\parallel \nabla f\left(x_t\right)\parallel }^2& \le {\displaystyle \frac{\eta }{2{\beta }^2\sqrt{{(M+\sigma )}^{2}t+\lambda }}}{\parallel \nabla f\left(x_t\right)\parallel }^2\hfill \\ & \le f\left(x_t\right)-f\left(x_{t+1}\right)+{\displaystyle \frac{\eta {\beta }^2}{\sqrt{\lambda }}}{\parallel {ϵ}_t\parallel }^2.\hfill \end{array}$$

Take a summation from $t=1$ to $\tau -1$. Then,

$${\displaystyle \frac{\eta }{2{\beta }^2\sqrt{{(M+\sigma )}^2(\tau -1)+\lambda }}}\sum _{t=1}^{\tau -1}{\parallel \nabla f\left(x_t\right)\parallel }^2\le f\left(x_1\right)-f^{🟉}-(f\left(x_{\tau }\right)-f^{🟉})+{\displaystyle \frac{\eta {\beta }^2}{\sqrt{\lambda }}}\sum _{t=1}^{\tau -1}{\parallel {ϵ}_t\parallel }^2$$

From the above equation, the bound in Lemma 1 follows

$$\begin{array}{c}\hfill f\left(x_{\tau }\right)-f^{🟉}\le f\left(x_1\right)-f^{🟉}+{\displaystyle \frac{\eta {\beta }^2}{\sqrt{\lambda }}}\sum _{t=1}^{\tau -1}{\parallel {ϵ}_t\parallel }^2\end{array}$$

and

$$\begin{array}{cc}\hfill \sum _{t=1}^{\tau -1}{\parallel \nabla f\left(x_t\right)\parallel }^2\le & {\displaystyle \frac{2{\beta }^2\sqrt{{(M+\sigma )}^2(\tau -1)+\lambda }}{\eta }}(f\left(x_1\right)-f^{🟉})\hfill \\ \hfill & +{\displaystyle \frac{2{\beta }^4\sqrt{{(M+\sigma )}^2(\tau -1)+\lambda }}{\sqrt{\lambda }}}\sum _{t=1}^{\tau -1}{\parallel {ϵ}_t\parallel }^2.\hfill \end{array}$$

5.5. Proof of Lemma 2

${ϵ}_t$ can be represented as follows:

$$\begin{array}{cc}\hfill {ϵ}_t& =m_t-\nabla f\left(x_t\right)\hfill \\ \hfill & =\underset{(🟉)}{\underbrace{(1-\beta )({ϵ}_{t-1}+\nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right))}}+\beta (g_t-\nabla f\left(x_t\right)).\hfill \end{array}$$

This representation highlights how ${ϵ}_t$ is recursively defined in terms of the momentum term $\left(m_t\right)$, the gradient differences, and the stochastic gradient noise. First, for all $t<\tau$, by using $\eta \le 1/L_1$, Assumption 2, and Lemma 7, we have

$$\begin{array}{cc}\hfill {\parallel \nabla f\left(x_{t-1}\right)-f\left(x_t\right)\parallel }^2& \le {(L_0+L_1\parallel \nabla f\left(x_t\right)\parallel )}^2{\parallel x_t-x_{t-1}\parallel }^2\le L^2{\parallel x_t-x_{t-1}\parallel }^2\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill & \le {\displaystyle \frac{{\eta }^2{L}^2}{\lambda }}{\parallel m_{t-1}\parallel }^2\le {\displaystyle \frac{2{\eta }^2{L}^2}{\lambda }}({\parallel \nabla f\left(x_{t-1}\right)\parallel }^2+{\parallel {ϵ}_{t-1}\parallel }^2).\hfill \end{array}$$

(5)

Using this, we can bound the term $(🟉)$ as follows:

$$\begin{array}{cc}\hfill & {\parallel (1-\beta ){ϵ}_{t-1}+(1-\beta )(\nabla f\left(x_{t-1}\right)-f\left(x_t\right))\parallel }^2\hfill \\ & \underset{\left(a\right)}{\le }{(1-\beta )}^2(1+\beta ){\parallel {ϵ}_{t-1}\parallel }^2+{(1-\beta )}^2(1+{\displaystyle \frac{1}{\beta }}){\parallel \nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right)\parallel }^2\hfill \\ \hfill & \underset{\left(b\right)}{\le }(1-\beta ){\parallel {ϵ}_{t-1}\parallel }^2+{\displaystyle \frac{1}{\beta }}{\parallel \nabla f\left(x_{t-1}\right)-f\left(x_t\right)\parallel }^2\hfill \\ \hfill & \underset{\left(c\right)}{\le }(1-\beta ){\parallel {ϵ}_{t-1}\parallel }^2+{\displaystyle \frac{2{\eta }^2{L}^2}{\beta \lambda }}({\parallel \nabla f\left(x_{t-1}\right)\parallel }^2+{\parallel {ϵ}_{t-1}\parallel }^2)\hfill \\ \hfill & \underset{\left(d\right)}{\le }(1-{\displaystyle \frac{\beta }{2}}){\parallel {ϵ}_{t-1}\parallel }^2+{\displaystyle \frac{2{\eta }^2{L}^2}{\beta \lambda }}{\parallel \nabla f\left(x_{t-1}\right)\parallel }^2\hfill \\ \hfill & \underset{\left(e\right)}{\le }(1-{\displaystyle \frac{\beta }{2}}){\parallel {ϵ}_{t-1}\parallel }^2+{\displaystyle \frac{{\beta }^2}{2}}{\parallel \nabla f\left(x_{t-1}\right)\parallel }^2.\hfill \end{array}$$

In $\left(a\right)$, we use Lemma 9 with $k=\beta$. $\left(b\right)$ follows from the following inequalities:

$$\begin{array}{cc}\hfill & {(1-\beta )}^2(1+\beta )=(1-\beta )(1-{\beta }^2)\le (1-\beta ),\hfill \\ \hfill & {(1-\beta )}^2(1+{\displaystyle \frac{1}{\beta }})={\displaystyle \frac{1}{\beta }}{(1-\beta )}^2(1+\beta )\le {\displaystyle \frac{1}{\beta }}(1-\beta )\le {\displaystyle \frac{1}{\beta }}.\hfill \end{array}$$

$\left(c\right)$ is due to (4). $\left(d\right)$ is derived from the step size condition $\eta \le {\displaystyle \frac{\sqrt{\lambda }{\beta }^2}{6L}}\le {\displaystyle \frac{\sqrt{\lambda }\beta }{2L}}$. Finally, $\left(e\right)$ is due to the step size condition $\eta \le {\displaystyle \frac{\sqrt{\lambda }{\beta }^2}{6L}}\le {\displaystyle \frac{\sqrt{\lambda }{\beta }^{3/2}}{2L}}$. Hence,

$$\begin{array}{cc}\hfill {\parallel {ϵ}_t\parallel }^2& ={\parallel (1-\beta ){ϵ}_{t-1}+(1-\beta )(\nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right))\parallel }^2+{\beta }^2{\parallel g_t-\nabla f\left(x_t\right)\parallel }^2\hfill \\ \hfill & +2(1-\beta )\beta ⟨{ϵ}_{t-1}+\nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right),g_t-\nabla f\left(x_t\right)⟩\hfill \\ \hfill & \le (1-{\displaystyle \frac{\beta }{2}}){\parallel {ϵ}_{t-1}\parallel }^2+{\displaystyle \frac{{\beta }^2}{2}}{\parallel \nabla f\left(x_{t-1}\right)\parallel }^2+{\beta }^2{\parallel g_t-\nabla f\left(x_t\right)\parallel }^2\hfill \\ & +2(1-\beta )\beta ⟨{ϵ}_{t-1}+\nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right),g_t-\nabla f\left(x_t\right)⟩.\hfill \end{array}$$

Then,

$$\begin{array}{cc}\hfill {\displaystyle \frac{\beta }{2}}{\parallel {ϵ}_{t-1}\parallel }^2\le & {\parallel {ϵ}_{t-1}\parallel }^2-{\parallel {ϵ}_t\parallel }^2+{\displaystyle \frac{{\beta }^2}{2}}{\parallel \nabla f\left(x_{t-1}\right)\parallel }^2+{\beta }^2{\parallel g_t-\nabla f\left(x_t\right)\parallel }^2\hfill \\ & +2(1-\beta )\beta ⟨{ϵ}_{t-1}+\nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right),g_t-\nabla f\left(x_t\right)⟩.\hfill \end{array}$$

Multiply $2/\beta$ and take the summation from $t=2$ to $\tau -1$. Then, we obtain

$$\begin{array}{cc}\hfill \sum _{t=2}^{\tau -1}{\parallel {ϵ}_{t-1}\parallel }^2=\sum _{t=1}^{\tau -2}{\parallel {ϵ}_t\parallel }^2\le & {\displaystyle \frac{2}{\beta }}(\parallel {ϵ}_1{\parallel }^2-\parallel {ϵ}_{\tau -1}{\parallel }^2)+\beta \sum _{t=2}^{\tau -1}\parallel \nabla f\left(x_{t-1}\right){\parallel }^2+2\beta \sum _{t=2}^{\tau -1}{\parallel g_t-\nabla f\left(x_t\right)\parallel }^2\hfill \\ & +4(1-\beta )\sum _{t=2}^{\tau -1}⟨{ϵ}_{t-1}+\nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right),g_t-\nabla f\left(x_t\right)⟩\hfill \\ \hfill \le & {\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^2(\tau -2)+2\beta {\sigma }^2(\tau -2)\hfill \\ & +4(1-\beta )\sum _{t=2}^{\tau -1}⟨{ϵ}_{t-1}+\nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right),g_t-\nabla f\left(x_t\right)⟩\hfill \end{array}$$

where we use Lemma 7 for the last inequality. To handle the term ${\sum }_{t=2}^{\tau -1}⟨{ϵ}_{t-1}+\nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right),g_t-\nabla f\left(x_t\right)⟩$, we apply the Azuma–Hoeffding inequality (Lemma 10). First, we observe that since $\tau -1\le T$ by its definition,

$$\begin{array}{cc}\hfill & \sum _{t=2}^{\tau -1}⟨{ϵ}_{t-1}+\nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right),g_t-\nabla f\left(x_t\right)⟩\hfill \\ \hfill & =\sum _{t=2}^T⟨({ϵ}_{t-1}+\nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right)){\mathbf{1}}_{\tau \ge t},g_t-\nabla f\left(x_t\right)⟩.\hfill \end{array}$$

Let $X_t=⟨({ϵ}_{t-1}+\nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right)){\mathbf{1}}_{\tau \ge t},g_t-\nabla f\left(x_t\right)⟩$ for $t\in \{2,\dots ,T\}$. We now show $\parallel {ϵ}_t\parallel \le 2\sigma$ for all $t<\tau$ through mathematical induction on t. When $t=1$, according to Lemma 7, it holds that

$$\parallel {ϵ}_1\parallel =\parallel g_1-\nabla f\left(x_1\right)\parallel \le \sigma \le 2\sigma$$

Furthermore, according to Lemma 4, we have

$$\parallel \nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right)\parallel \le L\eta \sqrt{\beta }\le \beta \sigma$$

where the last inequality comes from $\eta \le {\displaystyle \frac{\sqrt{\beta }\sigma }{L}}$. Recall that

$${ϵ}_t=(1-\beta )({ϵ}_{t-1}+\nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right))+\beta (g_t-\nabla f\left(x_t\right)).$$

Now, suppose that $\parallel {ϵ}_{t-1}\parallel \le 2\sigma$. Then, we have

$$\begin{array}{cc}\hfill \parallel {ϵ}_t\parallel & \le (1-\beta )\parallel {ϵ}_{t-1}\parallel +(1-\beta )\parallel \nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right)\parallel +\beta \parallel g_t-\nabla f\left(x_t\right)\parallel \hfill \\ \hfill & \le (2-\beta )\sigma +\parallel \nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right)\parallel \hfill \\ \hfill & \le (2-\beta )\sigma +\beta \sigma =2\sigma \hfill \end{array}$$

where we use the induction hypothesis and $\parallel g_t-\nabla f\left(x_t\right)\parallel \le \sigma$ for the second inequality and $\parallel \nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right)\parallel \le \beta \sigma$ for the last inequality. Hence, it holds that

$$\begin{array}{cc}\hfill |X_t|& \le \parallel {ϵ}_{t-1}+\nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right)\parallel \parallel g_t-\nabla f\left(x_t\right)\parallel \hfill \\ \hfill & \le \sigma \parallel {ϵ}_{t-1}+\nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right)\parallel \hfill \\ \hfill & \le \sigma (\parallel {ϵ}_{t-1}\parallel +\parallel \nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right)\parallel )\hfill \\ \hfill & \le 3{\sigma }^2\hfill \end{array}$$

where we use $\parallel \nabla f\left(x_{t-1}\right)-\nabla f\left(x_t\right)\parallel \le \beta \sigma$ and $\beta \le 1$ for the last inequality. Note that $\mathbb{E}\left[X_t\right|X_1,\dots ,X_{t-1}]=0$ by its definition. This implies that ${\left\{X_t\right\}}_{t\le T}$ is a martingale difference sequence. Now, we apply the Azuma–Hoeffding inequality (Lemma 10) to obtain the following: with a probability of at least $1-\delta$,

$$\sum _{t=2}^T{X}_t\le 3{\sigma }^2\sqrt{2T\left(log{\displaystyle \frac{1}{\delta }}\right)}\le {\displaystyle \frac{9}{2}}{\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}.$$

Therefore, with a probability of at least $1-\delta$,

$$\begin{array}{cc}\hfill \sum _{t=1}^{\tau -2}{\parallel {ϵ}_t\parallel }^2\le & {\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^2(\tau -2)+2\beta {\sigma }^2(\tau -2)+18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}.\hfill \end{array}$$

With ${\parallel {ϵ}_{\tau -1}\parallel }^2\le 4{\sigma }^2$, which is from $\parallel {ϵ}_{\tau -1}\parallel \le 2\sigma$, we obtain

$$\begin{array}{cc}\hfill \sum _{t=1}^{\tau -1}{\parallel {ϵ}_t\parallel }^2\le & {\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^2(\tau -2)+2\beta {\sigma }^2(\tau -2)+18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2.\hfill \end{array}$$

5.6. Proof of Lemma 3

According to Lemmas 1 and 2, with a probability of at least $1-\delta$, we have

$$\begin{array}{cc}\hfill f\left(x_{\tau }\right)-f^{🟉}& \le f\left(x_1\right)-f^{🟉}+{\displaystyle \frac{\eta {\beta }^2}{\sqrt{\lambda }}}[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^2(\tau -2)+2\beta {\sigma }^2(\tau -2)\hfill \\ \hfill & +18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2]\hfill \\ \hfill & \le f\left(x_1\right)-f^{🟉}+{\displaystyle \frac{{\beta }^2}{\sqrt{\lambda }{L}_1}}[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^{2}T+2\beta {\sigma }^{2}T\hfill \\ \hfill & +18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2]\hfill \\ \hfill & =:I_1\hfill \end{array}$$

where the last inequality is from $\eta \le 1/L_1$ and $\tau \le T+1$. If $\tau \le T$, then according to the definition of $\tau$, we can immediately derive the following lower bound on $f\left(x_{\tau }\right)$: $f\left(x_{\tau }\right)-f^{🟉}>G$. We now show $\tau =T+1$ by showing that this lower bound exceeds the upper bound $I_1$, which results in a contradiction. Specifically, we show that $I_1/G<1$ as follows:

$$\begin{array}{cc}\hfill {\displaystyle \frac{I_1}{G}}\le {\displaystyle \frac{\Delta }{G}}& +{\displaystyle \frac{{\beta }^2}{\sqrt{\lambda }{L}_{1}G}}\left[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^{2}T+2\beta {\sigma }^{2}T+18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2\right]\hfill \\ \hfill & \le {\displaystyle \frac{\Delta }{G}}+{\displaystyle \frac{2}{\sqrt{\lambda }{L}_{1}G}}+{\displaystyle \frac{1}{\sqrt{\lambda }{L}_{1}G}}+{\displaystyle \frac{2}{\sqrt{\lambda }{L}_{1}G}}+{\displaystyle \frac{18}{\sqrt{\lambda }{L}_{1}G}}+{\displaystyle \frac{4}{\sqrt{\lambda }{L}_{1}G}}\hfill \\ \hfill & ={\displaystyle \frac{28}{29}}<1.\hfill \end{array}$$

For the second inequality, we use the conditions $T\le 1/{\beta }^2$ and $\beta \le min\left\{1,{\displaystyle \frac{1}{{\sigma }^2}},{\displaystyle \frac{1}{M^2}},{\displaystyle \frac{1}{{\sigma }^2\iota }}\right\}$. For the last equality, we use $G\ge {\displaystyle \frac{29}{L_1\sqrt{\lambda }}}$ and $G\ge 29\Delta$. Since $I_1/G<1$ (i.e., $I_1<G$), which contradicts our assumption that $\tau \le T$, it holds that $\tau =T+1$.

6. Proof of Corollary 1

First, we restate the hyper-parameter conditions in Corollary 1 for convenience as follows:

$$\begin{array}{cc}\hfill & \eta ={\displaystyle \frac{\sqrt{\lambda }{\beta }^2}{6L}},T=⌊{\displaystyle \frac{1}{{\beta }^2}}⌋,\hfill \\ \hfill & 0<\beta \le min\left\{{\displaystyle \frac{1}{2}},{\displaystyle \frac{6L}{L_1\sqrt{\lambda }}},{\displaystyle \frac{6\sigma }{\sqrt{\lambda }}},{\displaystyle \frac{1}{M^2}},{\displaystyle \frac{1}{{\sigma }^2\iota }},{\displaystyle \frac{1}{{\sigma }^2}},{\displaystyle \frac{1}{M}},{\displaystyle \frac{1}{\sigma }},{\displaystyle \frac{1}{\sigma \iota }}\right.\hfill \\ \hfill & \left.{\displaystyle \frac{\sqrt{\lambda }{ϵ}^2}{144\Delta L(M+\sigma )}},{\displaystyle \frac{ϵ}{12}}\sqrt{{\displaystyle \frac{1}{L\Delta }}},{\displaystyle \frac{\sqrt[4]{\lambda }ϵ}{12\sqrt{M+\sigma }}},{\displaystyle \frac{{ϵ}^2}{144}}\right\}.\hfill \end{array}$$

Note that since $\beta \le min\{{\displaystyle \frac{6L}{L_1\sqrt{\lambda }}},{\displaystyle \frac{6\sigma }{\sqrt{\lambda }}}\}$, we have $\eta ={\displaystyle \frac{\sqrt{\lambda }{\beta }^2}{6L}}\le min\{{\displaystyle \frac{1}{L_1}},{\displaystyle \frac{\sqrt{\beta }\sigma }{L}}\}$, i.e., the condition on $\eta$ in Lemmas 1–3 is satisfied. According to Lemmas 1–3, we derive the following inequality as in (3): for any $\delta \in (0,1)$,

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{T}}\sum _{t=1}^T{\parallel \nabla f\left(x_t\right)\parallel }^2& \le {\displaystyle \frac{2\Delta {\beta }^2\sqrt{{(M+\sigma )}^{2}T+\lambda }}{\eta T}}\hfill \\ \hfill & +{\displaystyle \frac{2{\beta }^4\sqrt{{(M+\sigma )}^{2}T+\lambda }}{\sqrt{\lambda }T}}[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^{2}T+2\beta {\sigma }^{2}T\hfill \\ \hfill & +18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2]\hfill \\ \hfill & \le {\displaystyle \frac{2\Delta {\beta }^2(M+\sigma )}{\eta \sqrt{T}}}+{\displaystyle \frac{2\Delta {\beta }^2\sqrt{\lambda }}{\eta T}}\hfill \\ \hfill & +\left({\displaystyle \frac{2{\beta }^4(M+\sigma )}{\sqrt{\lambda T}}}+{\displaystyle \frac{2{\beta }^4}{T}}\right)[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^{2}T+2\beta {\sigma }^{2}T\hfill \\ \hfill & +18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2]\hfill \end{array}$$

with a probability of at least $1-\delta$ where the second inequality is from the inequality $\sqrt{a+b}\le \sqrt{a}+\sqrt{b}$. First, we bound the following terms:

$${\displaystyle \frac{2\Delta {\beta }^2(M+\sigma )}{\eta \sqrt{T}}}+{\displaystyle \frac{2\Delta {\beta }^2\sqrt{\lambda }}{\eta T}}.$$

Since $T=\lfloor 1/{\beta }^2\rfloor$, we have

$${\displaystyle \frac{1}{{\beta }^2}}-1\le T\le {\displaystyle \frac{1}{{\beta }^2}}.$$

This implies that

$${\displaystyle \frac{1}{T}}\le {\displaystyle \frac{{\beta }^2}{1-{\beta }^2}}={\displaystyle \frac{{\beta }^2}{(1-\beta )(1+\beta )}}\le {\displaystyle \frac{{\beta }^2}{1-\beta }}\le 2{\beta }^2$$

(6)

where the last inequality holds because $1-\beta \ge 1/2$. By using $\eta =\sqrt{\lambda }{\beta }^2/6L$ and (6), we obtain

$$\begin{array}{cc}\hfill & {\displaystyle \frac{2\Delta {\beta }^2(M+\sigma )}{\eta \sqrt{T}}}+{\displaystyle \frac{2\Delta {\beta }^2\sqrt{\lambda }}{\eta T}}\le {\displaystyle \frac{18L\Delta \beta (M+\sigma )}{\sqrt{\lambda }}}+24L\Delta {\beta }^2\le {\displaystyle \frac{42}{144}}{ϵ}^2\hfill \end{array}$$

where the last inequality is due to the following condition on $\beta$:

$$\beta \le min\left\{{\displaystyle \frac{\sqrt{\lambda }{ϵ}^2}{144\Delta L(M+\sigma )}},{\displaystyle \frac{ϵ}{12}}\sqrt{{\displaystyle \frac{1}{L\Delta }}}\right\}.$$

Next, we bound the following term:

$$\begin{array}{c}\hfill {\displaystyle \frac{2{\beta }^4(M+\sigma )}{\sqrt{\lambda T}}}\left[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^{2}T+2\beta {\sigma }^{2}T+18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2\right].\end{array}$$

Using the condition $T=\lfloor 1/{\beta }^2\rfloor \le 1/{\beta }^2$ and (6), we obtain

$$\begin{array}{cc}\hfill & {\displaystyle \frac{2{\beta }^4(M+\sigma )}{\sqrt{\lambda T}}}\left[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^{2}T+2\beta {\sigma }^{2}T+18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2\right]\hfill \\ \hfill \le & {\displaystyle \frac{3{\beta }^5(M+\sigma )}{\sqrt{\lambda }}}\left[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^{2}T+2\beta {\sigma }^{2}T+18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2\right]\hfill \\ \hfill \le & {\displaystyle \frac{3{\beta }^2(M+\sigma )}{\sqrt{\lambda }}}\left[2{\beta }^2{\sigma }^2+{\beta }^2{M}^2+2{\beta }^2{\sigma }^2+18{\beta }^2{\sigma }^2\iota +4{\beta }^3{\sigma }^2\right].\hfill \end{array}$$

Then, according to the following condition of $\beta$

$$\beta \le min\left\{{\displaystyle \frac{1}{2}},{\displaystyle \frac{1}{\sigma }},{\displaystyle \frac{1}{M}},{\displaystyle \frac{1}{\sigma \iota }},{\displaystyle \frac{\sqrt[4]{\lambda }ϵ}{12\sqrt{M+\sigma }}}\right\},$$

we obtain

$$\begin{array}{c}\hfill {\displaystyle \frac{3{\beta }^2(M+\sigma )}{\sqrt{\lambda }}}\left[2{\beta }^2{\sigma }^2+{\beta }^2{M}^2+2{\beta }^2{\sigma }^2+18{\beta }^2{\sigma }^2\iota +4{\beta }^3{\sigma }^2\right]\le {\displaystyle \frac{75{\beta }^2(M+\sigma )}{\sqrt{\lambda }}}\le {\displaystyle \frac{75}{144}}{ϵ}^2.\end{array}$$

Lastly, we bound the following term:

$$\begin{array}{c}\hfill {\displaystyle \frac{2{\beta }^4}{T}}\left[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^{2}T+2\beta {\sigma }^{2}T+18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2\right].\end{array}$$

Using (6), we obtain

$$\begin{array}{cc}\hfill & {\displaystyle \frac{2{\beta }^4}{T}}\left[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^{2}T+2\beta {\sigma }^{2}T+18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2\right]\hfill \\ \hfill & \le 4{\beta }^6\left[{\displaystyle \frac{2}{beta}}{\sigma }^2+\beta M^{2}T+2\beta {\sigma }^{2}T+18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2\right].\hfill \end{array}$$

Then, according to the following condition of $\beta$

$$\beta \le min\left\{{\displaystyle \frac{1}{2}},{\displaystyle \frac{1}{\sigma }},{\displaystyle \frac{1}{M}},{\displaystyle \frac{1}{\sigma \iota }},{\displaystyle \frac{{ϵ}^2}{144}}\right\},$$

it holds that

$$\begin{array}{cc}\hfill & 4{\beta }^6\left[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^{2}T+2\beta {\sigma }^{2}T+18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2\right]\hfill \\ \hfill & \le {\beta }^4\left[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^{2}T+2\beta {\sigma }^{2}T+18(1-\beta ){\sigma }^2\sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2\right]\hfill \\ \hfill & \le 25\beta \le {\displaystyle \frac{25}{144}}{ϵ}^2.\hfill \end{array}$$

Therefore, for any $\delta \in (0,1)$, with a probability of at least $1-\delta$, we have

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{T}}\sum _{t=1}^T{\parallel \nabla f\left(x_t\right)\parallel }^2& \le {\displaystyle \frac{2\Delta {\beta }^2(M+\sigma )}{\eta \sqrt{T}}}+{\displaystyle \frac{2\Delta {\beta }^2\sqrt{\lambda }}{\eta T}}\hfill \\ \hfill & +\left({\displaystyle \frac{2{\beta }^4(M+\sigma )}{\sqrt{\lambda T}}}+{\displaystyle \frac{2{\beta }^4}{T}}\right)[{\displaystyle \frac{2}{\beta }}{\sigma }^2+\beta M^{2}T+2\beta {\sigma }^{2}T\hfill \\ \hfill & +12(1-\beta )\sigma \sqrt{T\left(log{\displaystyle \frac{1}{\delta }}\right)}+4{\sigma }^2]\hfill \\ \hfill & \le {\displaystyle \frac{142}{144}}{ϵ}^2\le {ϵ}^2.\hfill \end{array}$$

7. Conclusions

In this paper, we proved the dimension-independent convergence rates of Adagrad with momentum under the $(L_0,L_1)$-smoothness assumption. We demonstrated that Adagrad with momentum achieves convergence to a stationary point at a rate of $\mathcal{O}\left(\sqrt{1/T}\right)$ that does not scale with the dimension and $1/\beta$. We believe that our results can improve the theoretical understanding of adaptive gradient methods with momentum.