Online Gradient Descent for Kernel-Based Maximum Correntropy Criterion

Abstract

In the framework of statistical learning, we study the online gradient descent algorithm generated by the correntropy-induced losses in Reproducing kernel Hilbert spaces (RKHS). As a generalized correlation measurement, correntropy has been widely applied in practice, owing to its prominent merits on robustness. Although the online gradient descent method is an efficient way to deal with the maximum correntropy criterion (MCC) in non-parameter estimation, there has been no consistency in analysis or rigorous error bounds. We provide a theoretical understanding of the online algorithm for MCC, and show that, with a suitable chosen scaling parameter, its convergence rate can be min–max optimal (up to a logarithmic factor) in the regression analysis. Our results show that the scaling parameter plays an essential role in both robustness and consistency.

1. Introduction

Regression analysis is an important problem in many fields of science. The traditional least squares method may be the most used algorithm for regression in practice. However, it only relies on the mean squared error and belongs to second-order statistics, whose optimality depends heavily on the assumption of Gaussian noise. Thus, it usually performs poorly when the noise is not normally distributed. Alterative approaches have been proposed to deal with outliers or heavy-tailed distributions. A generalized correlation function named correntropy [1] is introduced as a substitute for the least squares loss, and the maximum correntropy criterion (MCC) [2,3,4,5] is used to improve robustness in situations of non-Gaussian and heavy-tailed error distributions. Recently, MCC has been succeeded in many real applications, e.g., wind power forecasting and pattern recognition [6,7].

In the standard framework of statistical learning, let $X\in {\mathcal{R}}^n$ be an explanatory variable with values taken in a compact metric space $(\mathcal{X},d),$Y be a real response variable with $Y\in \mathcal{Y}\subset \mathcal{R}.$ Here we investigate the application of MCC in the following regression model

$$\begin{array}{c}\hfill Y=f_{\rho }(X)+ϵ,\mathbb{E}(ϵ|X=x)=0,\end{array}$$

where $ϵ$ is the noise and $f_{\rho }(x)$ is the regression function, defined as the conditional mean $\mathbb{E}(Y|X=x)$ at each $x\in \mathcal{X}.$ The purpose of regression is to estimate the unknown target function $f_{\rho }$ according to the sample $\mathbf{z}={\{z_i=(x_i,y_i)\}}_{i=1}^T,$ which is drawn independently from the underlying unknown probability distribution $\rho$ on $\mathcal{Z}:=\mathcal{X}\times \mathcal{Y}.$ For a hypothesis function $f:\mathcal{X}\to \mathcal{Y},$ with the scaling parameter $\sigma >0,$ the correntropy between $f(X)$ and Y is defined by $V_{\sigma }(f):=\mathbb{E}G\left(\frac{{(f(X)-Y)}^2}{2{\sigma }^2}\right)$ where $G(u)$ is the Exponential function $exp\left\{-u\right\},u\in \mathbb{R}.$ For the given sample $\mathbf{z},$ the empirical form of $V_{\sigma }$ is ${\widehat{V}}_{\sigma }(f):=\frac{1}{T}{\sum }_{i=1}^{T}G\left(\frac{{(f(x_i)-y_i)}^2}{2{\sigma }^2}\right).$ When applied to regression problems, MCC intends to maximize the empirical correntropy ${\widehat{V}}_{\sigma }$ over a certain underlying hypothesis space $\mathcal{H}$, that is

$$\begin{array}{c}\hfill f_{\mathbf{z},\mathcal{H}}:=arg\underset{f\in \mathcal{H}}{max}{\widehat{V}}_{\sigma }(f).\end{array}$$

(1)

MCC in regression problems has shown its efficiency for cases when the noises are non-Gaussian, and also with large outliers, see [8,9,10]. It also has drawn much attention in the signal processing, machine learning and optimization communities [2,5,11,12,13,14].

Let $K:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ be a Mercer kernel, i.e., a continuous, symmetric and positive semi-definite function. We say that K is a positive semi-definite, if for any finite set $\left\{u_1,\cdots ,u_m\right\}\subset \mathcal{X}$ and $m\in \mathbb{N},$ the matrix ${\left(K(u_i,u_j)\right)}_{i,j=1}^m$ is positive semi-definite. An RKHS $({\mathcal{H}}_K,\parallel ·{\parallel }_K)$ associated with the Mercer kernel K is defined as the completion of the linear span of the functions set $\{K_x:=K(x,·),x\in \mathcal{X}\}.$ It has the reproducing property

$$\begin{array}{c}\hfill f(x)={\langle f,K_x\rangle }_K\end{array}$$

(2)

for any $f\in {\mathcal{H}}_K$ and $x\in \mathcal{X}.$ Since $\mathcal{X}$ is compact, the RKHS ${\mathcal{H}}_K$ is contained in $C(\mathcal{X}),$ the space of continuous functions on $\mathcal{X}$ with the norm ${\displaystyle {\parallel f\parallel }_{\infty }:=\underset{x\in \mathcal{X}}{sup}|f(x)|.}$ Moreover, if $\mathcal{X}$ is a Euclidean ball in ${\mathbb{R}}^n$ with some $\alpha >\frac{n}{2},$ then the Sobolev space $H^{\alpha }(\mathcal{X})$ is an RKHS. For more families of RKHS in statistical learning, one can refer to [15]. Denote $\kappa :={sup}_{x\in \mathcal{X}}\sqrt{K(x,x),}$ then, by the reproducing property (2), there holds

$$\begin{array}{c}\hfill {\parallel f\parallel }_{\infty }\le \kappa {\parallel f\parallel }_K,foranyf\in {\mathcal{H}}_K.\end{array}$$

(3)

Denote ${\ell }_{\sigma }:\mathbb{R}\times \mathbb{R}\to \mathbb{R}$ as the correntropy induced regression loss, given by

$$\begin{array}{c}\hfill {\ell }_{\sigma }(u,v):={\sigma }^2\left(1-G\left(\frac{{(u-v)}^2}{2{\sigma }^2}\right)\right)={\sigma }^2\left(1-exp\left\{-\frac{{(u-v)}^2}{2{\sigma }^2}\right\}\right).\end{array}$$

Associated with this regression loss ${\ell }_{\sigma }$ and the RKHS ${\mathcal{H}}_K,$ MCC for regression (1) in the context of learning theory is reformulated as

$$\begin{array}{c}\hfill f_{\mathbf{z}}:=arg\underset{f\in {\mathcal{H}}_K}{min}\frac{1}{T}\sum _{i=1}^T{\ell }_{\sigma }(f(x_i),y_i).\end{array}$$

(4)

Notice that ${\ell }_{\sigma }$ is not convex, MCC algorithms are usually implemented by various gradient descent methods [14,16,17]. In this paper, we take the online gradient descent method as follows to solve the above optimization scheme (4) since it is scalable to large datasets and applicable to situations where the samples are presented in sequence.

Definition 1.

Given the sample $\mathbf{z}={\{z_i=(x_i,y_i)\}}_{i=1}^T,$ the online gradient descent method for MCC is defined by $f_1=0,$ and

$$\begin{array}{c}\hfill f_{t+1}=f_t-\eta {\ell }_{\sigma }^{\prime }(f_t(x_t),y_t)K_{x_t},t\in \mathbb{N},\end{array}$$

(5)

where $\eta >0$ is the step size and ${\ell }_{\sigma }^{\prime }$ denotes the derivative of ${\ell }_{\sigma }$ with respect to the first variable.

In the literature, most MCC algorithms have been implemented for linear models and cannot be applied to analysis of data with nonlinear structures. Kernel methods provide efficient non-parametric learning algorithms for dealing with nonlinear features. So, RKHS are used in this work as hypothesis spaces in the design of learning algorithms.

An online algorithm for MCC has been used in practical applications for more than one decade, but there still is a lack of the theoretical guarantee or strict analysis for its asymptotical convergence. Because the optimization problem arising from MCC is not convex, the global optimization convergence of the online algorithm (5) for MCC is not unconditionally guaranteed. This also makes the theoretical analysis for MCC essentially difficult. In fact, vast numerical studies show that MCC can lead robust estimators while keeping convenient convergence properties. Thus, our goal is to fill the gap between the theoretical analysis and the optimization process so that the output function of the online algorithm (5) can converge to a global minima while the existing work can not ensure the global optimization of this output. To this end, we study the approximation ability of $f_{T+1}$ generated by (5) at the T-iteration to the regression function $f_{\rho }.$ We derive the explicit error rate for (5) with suitable choice of step sizes, which is competitive with those in the regression analysis. In this work, we show that the scaling parameter $\sigma$ plays an important role in providing robustness and a fast convergence rate.

2. Preliminaries and Main Results

We begin with some preliminaries and notations. Throughout the paper, we assume that the unknown distribution $\rho$ on $\mathcal{Z}=\mathcal{X}\times \mathcal{Y}$ can be decomposed into the marginal distribution ${\rho }_{\mathcal{X}}$ on $\mathcal{X}$ and the conditional distribution $\rho (·|x)$ at each $x\in \mathcal{X}.$ We also require that $|Y|<M$ almost surely for some $M>1.$ In the regression analysis, the approximation power of $f_{T+1}$ by (5) is usually measured in terms of the mean squared error in $L_{{\rho }_{\mathcal{X}}}^2$-metric $\parallel f_{T+1}-f_{\rho }{\parallel }_{\rho },$ that is defined as ${\parallel ·\parallel }_{\rho }={\parallel ·\parallel }_{L_{{\rho }_{\mathcal{X}}}^2}:={\left({\int }_{\mathcal{X}}{|·|}^{2}d{\rho }_{\mathcal{X}}\right)}^{\frac{1}{2}}.$

To present our main result for the error bound of $f_{T+1}-f_{\rho }$, the assumption on the target function $f_{\rho }$ will be given as below. Define an integral operator $L_K:L_{{\rho }_{\mathcal{X}}}^2⟶L_{{\rho }_{\mathcal{X}}}^2$ associated with the kernel K by

$$\begin{array}{c}\hfill L_K(f):={\int }_{\mathcal{X}}f(x)K_{x}d{\rho }_{\mathcal{X}},f\in L_{{\rho }_{\mathcal{X}}}^2.\end{array}$$

By the reproducing property (2) of ${\mathcal{H}}_K,$ for any $f\in {\mathcal{H}}_K,$ it can be expressed as

$$\begin{array}{c}\hfill L_K(f)={\int }_{\mathcal{X}}{\langle f,K_x\rangle }_K{K}_{x}d{\rho }_{\mathcal{X}}.\end{array}$$

(6)

Since K is a Mercer kernel, $L_K$ is compact and positive. Denote $L_K^r$ as the r-th power of $L_K,$ then it is well defined for any $r>0$ by the spectral theorem. Let ${\{{\lambda }_i\}}_{i\ge 1}$ be the eigenvalues of $L_K,$ arranged in decreasing order. The corresponding eigenfunctions ${\{{\varphi }_i\}}_{i\ge 1}$ form an orthonormal basis of $L_{{\rho }_{\mathcal{X}}}^2$ space. Hence, the regularity space $L_K^r(L_{{\rho }_{\mathcal{X}}}^2)$ is expressed as [18]

$$\begin{array}{c}\hfill L_K^r(L_{{\rho }_{\mathcal{X}}}^2):=\left\{f=\sum _{i=1}^{\infty }{\lambda }_j^r{a}_i{\varphi }_i:{\parallel L_K^{-r}f\parallel }_{\rho }=\sum _{i=1}^{\infty }{a}_i^2<\infty \right\}.\end{array}$$

It implies that for any $r_1>r_2>0$, there holds $L_K^{r_1}(L_{{\rho }_{\mathcal{X}}}^2)\subset L_K^{r_2}(L_{{\rho }_{\mathcal{X}}}^2).$ In particular, we know that $L_K^r(L_{{\rho }_{\mathcal{X}}}^2)\subseteq {\mathcal{H}}_K$ for any $r\ge \frac{1}{2}$ and $L_K^{\frac{1}{2}}(L_{{\rho }_{\mathcal{X}}}^2)={\mathcal{H}}_K$ satisfying

$$\begin{array}{c}\hfill {\parallel f\parallel }_K={\parallel L_K^{-\frac{1}{2}}f\parallel }_{\rho },\forall f\in {\mathcal{H}}_K.\end{array}$$

(7)

Throughout the paper, the regularity assumption holds for $f_{\rho }$, i.e.,

$$\begin{array}{c}\hfill f_{\rho }=L_K^r(g),forsomer>0andg\in L_{{\rho }_{{\mathcal{X}}^2}}^2,\end{array}$$

(8)

and $\parallel L^{-r}{f}_{\rho }{\parallel }_{\rho }={\parallel g\parallel }_{\rho }.$

This assumption is called the source condition [19] in inverse problems and it characterizes the smoothness of the target function $f_{\rho }.$ Obviously, the larger the parameter r is, the higher the regularity of $f_{\rho }$ is. The general source conditions considered in inverse problems usually take the form of

$$\begin{array}{c}\hfill f_{\rho }=\psi (L_K)h,forsomeh\in {\mathcal{H}}_K\end{array}$$

(9)

where $\psi$ is non-decreasing and $\psi (0)=0,$ called the index function. It is clear that when $r>\frac{1}{2},$ The above assumption is a special case of (9) with $\psi (L_K)=L_K^{r-\frac{1}{2}}$ and $h=L_K^{\frac{1}{2}}g.$ It should be pointed that our analysis in this work also can applied to more general cases by taking source conditions (9).

We are now in a position to state our convergence rate for (5) in $L_{{\rho }_{\mathcal{X}}}^2$-space as well as in ${\mathcal{H}}_K$ by choosing the step size $\eta :=\eta (T).$ For brevity, let $\kappa =1$ without losing generality and denote the expectation ${\mathbb{E}}_{z_1,\cdots ,z_t}$ as ${\mathbb{E}}_t$ for each $t\in \mathbb{N}.$

Theorem 1.

Define ${\{f_t\}}_{t=1}^{T+1}$ by (5). Suppose that the assumption (8) holds for $r>0.$ Take $\eta =T^{-\frac{2r}{2r+1}}$ and $T>{\left(24{\left({(1/2e)}^{1/2}+1\right)}^{2}log(T)\right)}^{\frac{2r+1}{2r}},$ then

$$\begin{array}{c}\hfill {\mathbb{E}}_T\left[\parallel f_{T+1}-f_{\rho }{\parallel }_{\rho }^2\right]\le Cmax\left\{T^{-\frac{2r}{2r+1}}log(T),T^{\frac{5}{2r+1}}{\sigma }^{-4}\right\}\end{array}$$

(10)

and if $r>\frac{1}{2},$

$$\begin{array}{c}\hfill {\mathbb{E}}_T\left[\parallel f_{T+1}-f_{\rho }{\parallel }_K^2\right]\le C^{\prime }max\left\{T^{-\frac{2r-1}{2r+1}},T^{\frac{5}{2r+1}}{\sigma }^{-4}\right\}\end{array}$$

(11)

where the constants $C,C^{\prime }$ are independent of $T,\sigma ,$ and will be given in the proof.

Remark 1.

Besides the error $\parallel f_{T+1}-f_{\rho }{\parallel }_{\rho }$, the error bound (11) in ${\mathcal{H}}_K$-norm is also given if $r>\frac{1}{2}$, i.e., $f_{\rho }\in {\mathcal{H}}_K.$ By (3), it leads the pointwise convergence of $f_{T+1}$ to $f_{\rho }$ since for each $u\in \mathcal{X}$, $|f_{T+1}(u)-f_{\rho }(u)|\le \parallel f_{T+1}-f_{\rho }{\parallel }_K.$ Compared with the global error $\parallel |f_{T+1}-f_{\rho }{\parallel }_{\rho },$ the error rate in ${\mathcal{H}}_K$ characterizes the local performance of (5) and is much stronger. Furthermore [18], when the kernel K lies in $C^{\alpha }(\mathcal{X}\times \mathcal{X})$ for some $\alpha >0,$ its associated RKHS ${\mathcal{H}}_K$ can be embedded into $C^{\alpha /2}(\mathcal{X})$, whose partial derivative up to order $\alpha /2$ are continuous with ${\parallel f\parallel }_{C^{\alpha /2}(\mathcal{X})}={\sum }_{|s|\le \frac{\alpha }{2}}{\parallel D^{\frac{\alpha }{2}}f\parallel }_{\infty }.$ So, the convergence in ${\mathcal{H}}_K$ implies that $f_{T+1}$ will converge to $f_{\rho }$ in $C^{\frac{\alpha }{2}},$ that ensures the convergence of the derivatives of $f_{T+1}$ to those of $f_{\rho }.$

Remark 2.

It has been proved in [20] that the min–max optimal rate for regression problems is of order $O\left(T^{-\frac{2r}{2r+s}}\right)$ when there exists constants $C_s>0,$$0<s\le 1$ such that the following effective dimension condition holds, i.e.,

$$\begin{array}{c}\hfill Trace({(L_K+\lambda I)}^{-1}{L}_K)\le C_s{\lambda }^{-s},forany\lambda >0,\end{array}$$

where $Trace(·)$ denotes the trace of the operator. This condition measures the complexity [15,20,21] of ${\mathcal{H}}_K$ with respect to the marginal distribution ${\rho }_{\mathcal{X}}.$ It is always satisfied with $s=1$ by taking the constant $C_s=Trace(L_K).$ Hence, the min–max optimal rate for capacity-independent cases is of order $O\left(T^{-\frac{2r}{2r+1}}\right)$ by taking a universal parameter $s=1.$

When $\sigma \ge T^{\frac{2r+5}{4(2r+1)}}$, we see that our convergence rate in $L_{{\rho }_{\mathcal{X}}}^2$-norm is of order $O\left(T^{-\frac{2r}{2r+1}}log(T)\right).$ Thus, it is nearly optimal in the capacity-independent sense that up to a logarithmic factor, it matches the min–max optimal rate above. We also find that the convergence rates (10) and (11) keep decreasing as the regularity parameter r increases. Hence, the online algorithm (5) does not suffer from the saturation phenomenon existing in Tikhonov regularization schemes [22], where the error rate of the estimators will not improve if r is out of the range $(0,1].$ This again shows the advantage of the online algorithm (5).

Remark 3.

Recent paper [2] investigated the approximation ability of the empirical scheme (4) over general hypothesis spaces $\mathcal{H}$. This work shows that with a complexity parameter $0<\beta \le 2,$ their error rate is of order $O(T^{-\frac{2}{2+\beta }})$ if the scaling parameter $\sigma =T^{\frac{1}{2+\beta }}.$ To be fair, do not take the capacity of $\mathcal{H}$ into consideration by taking $\beta =2.$ Then, their order reduces to $O(T^{-\frac{1}{2}}),$ which is far from capacity-independent optimality and inferior to our rates.

In the work [17], iterative regularization techniques (alternatively called early stopping) are taken to solve the optimization problems associated with general robust losses including the correntropy induced loss ${\ell }_{\sigma },$ where the whole sample $\mathbf{z}$ are presented at each iteration. In their analysis, under the polynomial decay of the eigenvalues $\{{\lambda }_i\},$ that is, there exists some constants $C_b>0$ and $b\ge 1$ such that

$$\begin{array}{c}\hfill {\lambda }_i\le c_b{i}^{-b},\forall i\ge 1,\end{array}$$

the obtained rate is $O(T^{-\frac{2br}{2br+1}})$ if $r\ge \frac{1}{2}$, else, it is $O(T^{-\frac{2br}{b+1}}).$ This decay is also a measurement for the complexity of ${\mathcal{H}}_K,$ please refer to [21]. Recall that the compactness of $\mathcal{X}$ implies that ${\sum }_i{\lambda }_i<\infty$ and ${\lambda }_i\le c{i}^{-1}$ for some $c>0.$ So, their rate for capacity-independent cases is $O(T^{-\frac{2r}{2r+1}})$ if $r\ge \frac{1}{2}$, else, it is $O(T^{-r}).$ We can see that our results in (10) are superior in the case $0<r<\frac{1}{2}.$ It shows in theory that the online algorithm (5) for MCC can achieve better approximation rate when $f_{\rho }$ is not in ${\mathcal{H}}_K.$

Remark 4.

It is easy to check that the roots of the second derivative of ${\ell }_{\sigma }$ is $\pm \sigma ,$ i.e., when $|f(x)-y|<\sigma ,$ this loss is convex and behaves as the least squares loss; when $|f(x)-y|\ge \sigma$, the loss function becomes concave and rapidly tends to be flat as the value of $|f(x)-y|$ goes to infinity. It implies that ${\ell }_{\sigma }$ satisfies the redescending property, and with a suitable chosen scaling parameter σ, ${\ell }_{\sigma }$ can reject gross outliers while keeping a prediction accuracy. In Theorem 1, we observe that σ should be large enough to guarantee the nice convergence, which coincides with the work in [2]. They also pointed that too small σ may prevent the estimator to converge to $f_{\rho }.$ In a recent paper [23], correntropy with small σ is interpreted as modal regression. According to the above discussions and empirical studies [2,14,17], we conclude that the value of σ would determine the learning target and a moderate σ may be more appropriate for balancing the convergence and robustness in practice.

Based on the above remarks, we see that the convergence rate of online kernel-based MCC is comparable to that of the least squares that has appeared in the literature [24]. Meanwhile, MCC’s redescending property will produce robustness to various outliers including sub-Gaussain, Student’s t-distribution, and Cauchy distribution. These all shows the superiority of MCC in a variety of applications, such as clustering, classification and feature selection [14]. At the end of this section, we would like to point out that although our work is carried out under the boundness condition of $\mathcal{Y},$ it can be extended to more general situations such as the moment conditions [20].

3. Proofs of Main Result

In this section, we prove our main results in Theorem 1. First, we derive the uniform bound for the iteration sequence ${\{f_t\}}_{t=1}^{T+1}$ by (5).

Lemma 1.

Define ${\{f_t\}}_{t=1}^{T+1}$ by (5). If $0<\eta \le 1,$ then

$$\begin{array}{c}\hfill \parallel f_t{\parallel }_K\le M{\eta }^{\frac{1}{2}}{(t-1)}^{\frac{1}{2}},t\in \mathbb{N}.\end{array}$$

(12)

Proof. 

We prove (12) by induction. It is trivial that (12) holds for $t=1.$ Suppose (12) holds for $t\ge 2.$ Notice that ${\ell }_{\sigma }^{\prime }(f_t(x_t),y_t)=G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right)\left[f_t(x_t)-y_t\right].$ Write (12) as $f_{t+1}=f_t-\eta H_t$ where $H_t=G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right)\left[f_t(x_t)-y_t\right]K_{x_t}.$ Then by (2),

$$\begin{array}{c}\parallel f_{t+1}{\parallel }_K^2=\parallel f_t{\parallel }_K^2-2\eta {\langle f_t,H_t\rangle }_K+{\eta }^2{\parallel H_t\parallel }_K^2\hfill \\ =\parallel f_t{\parallel }_K^2-2\eta G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right)\left[f_t(x_t)-y_t\right]f_t(x_t)+{\eta }^2{\parallel H_t\parallel }_K^2\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill \parallel H_t{\parallel }_K^2& =G\left(-\frac{{(f_t(x_t)-y_t)}^2}{{\sigma }^2}\right){\left[f_t(x_t)-y_t\right]}^2{K}_{(x_t,x_t)}\hfill \\ & \le G\left(-\frac{{(f_t(x_t)-y_t)}^2}{{\sigma }^2}\right){\left[f_t(x_t)-y_t\right]}^2.\hfill \end{array}$$

Then, we have

$$\begin{array}{c}\hfill \parallel f_{t+1}{\parallel }_K^2\le {\parallel f_t\parallel }_K^2+\eta \left\{\eta G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right){\left[f_t(x_t)-y_t\right]}^2-2(f_t(x_t)-y_t)f_t(x_t)\right\}G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right).\end{array}$$

For the part of the above inequality, we have

$$\begin{array}{c}\eta G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right){\left[f_t(x_t)-y_t\right]}^2-2(f_t(x_t)-y_t)f_t(x_t)\hfill \\ =\left(\eta G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right)-2\right){\left((f_t(x_t)-y_t)-\frac{y_t}{\eta G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right)-2}\right)}^2\hfill \\ +\frac{y_t^2}{2-\eta G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right)}.\hfill \end{array}$$

Since $\eta \le 1,$ it follows that $\eta G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right)-2<0$ and $2-\eta G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right)\ge 1.$ Recall that $|y|\le M$ for all $y\in \mathcal{Y},$ then

$$\begin{array}{c}\hfill \eta G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right){\left[f_t(x_t)-y_t\right]}^2-2(f_t(x_t)-y_t)f_t(x_t)\le \frac{y_t^2}{2-\eta G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right)}\le M^2.\end{array}$$

Based on the above analysis,

$$\begin{array}{c}\hfill \parallel f_{t+1}{\parallel }_K^2\le \parallel f_t{\parallel }_K^2+\eta M^{2}G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right)\le {\parallel f_t\parallel }_K^2+\eta M^2\le M^2\eta (t-1)+\eta M^2=M^2\eta t.\end{array}$$

Then the proof is completed. □

Next, we will establish a proposition which is crucial to prove the convergence rates in Theorem 1. It is closely related to the generalization error of $f_t.$ Define the generalization error$\mathcal{E}(f)$ for any measurable function $f:\mathcal{X}\to \mathbb{R}$ by

$$\begin{array}{c}\hfill \mathcal{E}(f)={\int }_{\mathcal{Z}}{(f(x)-y)}^{2}d\rho .\end{array}$$

The regression function $f_{\rho }$ that we want to learn or approximate is a minimizer of $\mathcal{E}(f)$ , that is

$$\begin{array}{c}\hfill f_{\rho }=argmin\{\mathcal{E}(f):fisameasurablefunctionfrom\mathcal{X}to\mathcal{Y}\}.\end{array}$$

A simple computation yields the relation for $f:\mathcal{X}\to \mathbb{R}$

$$\begin{array}{c}\hfill \parallel f-f_{\rho }{\parallel }_{\rho }^2=\mathcal{E}(f)-\mathcal{E}(f_{\rho }).\end{array}$$

(13)

For brevity, set the operator ${\pi }_k^t(L_K):={\prod }_{j=k}^t(I-\eta L_K)$ for $k,t\in \mathbb{N}$ and ${\pi }_{t+1}^t(L_K):=I.$

Proposition 1.

Define ${\{f_t\}}_{t=1}^{T+1}$ by (5). If the step size $0<\eta <1,$ then we have

$$\begin{array}{cc}\hfill {\mathbb{E}}_T\left[\parallel f_{T+1}-f_{\rho }{\parallel }_{\rho }^2\right]& \le 2{∥{\pi }_1^T(L_K)f_{\rho }∥}_{\rho }^2+2{\eta }^2\sum _{t=1}^T\frac{2{\left({(1/2e)}^{1/2}+1\right)}^2}{1+\eta (T-t)}{\mathbb{E}}_{t-1}\left[\mathcal{E}(f_t)\right]\hfill \\ & +2{\mathbb{E}}_T\left[{∥\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K){\Delta }_t∥}_{\rho }^2\right],\hfill \end{array}$$

(14)

furthermore, if $f_{\rho }\in {\mathcal{H}}_K,$

$$\begin{array}{cc}\hfill {\mathbb{E}}_T\left[\parallel f_{T+1}-f_{\rho }{\parallel }_K^2\right]& \le 2{∥{\pi }_1^T(L_K)f_{\rho }∥}_K^2+2{\eta }^2\sum _{t=1}^T{\mathbb{E}}_{t-1}\left[\mathcal{E}(f_t)\right]+2{\mathbb{E}}_T\left[{∥\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K){\Delta }_t∥}_K^2\right],\hfill \end{array}$$

(15)

where ${\Delta }_t$ is defined in the proof.

Proof. 

Denote

$$\begin{array}{cc}\hfill {\Delta }_t& =\left[G(0)-G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right)\right]\left[f_t(x_t)-y_t\right]K_{x_t}\hfill \\ & =\left[1-G\left(-\frac{{(f_t(x_t)-y_t)}^2}{2{\sigma }^2}\right)\right]\left[f_t(x_t)-y_t\right]K_{x_t}\hfill \end{array}$$

(16)

and define a random variable $\xi (f_t,z_t):=L_K(f_t-f_{\rho })-(f_t(x_t)-y_t)K_{x_t.}$

By (5), we have that for any $t\in \mathbb{N},$

$$\begin{array}{c}{f}_{t+1}-f_{\rho }=f_t-f_{\rho }-\eta \left[f_t(x_t)-y_t\right]K_{x_t}+\eta {\Delta }_t=\left(I-\eta L_K\right)(f_t-f_{\rho })+\eta \xi (f_t,z_t)+\eta {\Delta }_t.\hfill \end{array}$$

Applying the above equality iteratively from $t=T$ to $t=1,$ we get that by $f_1=0,$

$$\begin{array}{c}\hfill f_{T+1}-f_{\rho }=-{\pi }_1^T(L_K)f_{\rho }+\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K)\xi (f_t,z_t)+\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K){\Delta }_t.\end{array}$$

(17)

It follows from the elementary inequality that $\parallel g_1+g_2{\parallel }_{\rho }^2\le 2\parallel g_1{\parallel }_{\rho }^2+2{\parallel g_2\parallel }_{\rho }^2$ for any $g_1,g_2\in L_{{\rho }_{\mathcal{X}}}^2,$ that

$$\begin{array}{c}\hfill {\mathbb{E}}_T\left[\parallel f_{T+1}-f_{\rho }{\parallel }_{\rho }^2\right]\le 2{\mathbb{E}}_T\left[{∥-{\pi }_1^T(L_K)f_{\rho }+\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K)\xi (f_t,z_t)∥}_{\rho }^2\right]+2{\mathbb{E}}_T\left[{∥\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K){\Delta }_t∥}_{\rho }^2\right].\end{array}$$

(18)

To prove (14), we consider the part of the first term on the right-hand side of (18)

$$\begin{array}{c}{\mathbb{E}}_T\left[{∥-{\pi }_1^T(L_K)f_{\rho }+\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K)\xi (f_t,z_t)∥}_{\rho }^2\right]={∥{\pi }_1^T(L_K)f_{\rho }∥}_{\rho }^2\hfill \\ +{\mathbb{E}}_T\left[{∥\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K)\xi (f_t,z_t)∥}_{\rho }^2\right]-2{\mathbb{E}}_T\left[{〈{\pi }_1^T(L_K)f_{\rho },\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K)\xi (f_t,z_t)〉}_{\rho }\right].\hfill \end{array}$$

(19)

Observe that $f_t$ is only dependent on $\{z_1,\cdots ,z_{t-1}\}$, not on $z_t.$ Thus, by the fact that ${\int }_{\mathcal{X}}yd\rho =f_{\rho }(x),$ we have

$$\begin{array}{c}\hfill {\mathbb{E}}_{z_t}\left[\xi (f_t,z_t)\right]=0,t=1,\cdots ,T.\end{array}$$

(20)

We consider the second term on the right-hand side of (19). It can be rewritten as

$$\begin{array}{c}\hfill {\mathbb{E}}_T\left[{∥\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K)\xi (f_t,z_t)∥}_{\rho }^2\right]={\eta }^2\sum _{t=1}^T\sum _{l=1}^T{\mathbb{E}}_T{〈{\pi }_{t+1}^T(L_K)\xi (f_t,z_t),{\pi }_{l+1}^T(L_K)\xi (f_l,z_l)〉}_{\rho }.\end{array}$$

When $t<l\le T,$ by (20),

$$\begin{array}{c}{\mathbb{E}}_T{〈{\pi }_{t+1}^T(L_K)\xi (f_t,z_t),{\pi }_{l+1}^T(L_K)\xi (f_l,z_l)〉}_{\rho }={\mathbb{E}}_{l-1}{\mathbb{E}}_{z_l}{〈{\pi }_{t+1}^T(L_K)\xi (f_t,z_t),{\pi }_{l+1}^T(L_K)\xi (f_l,z_l)〉}_{\rho }\hfill \\ ={\mathbb{E}}_{l-1}{〈{\pi }_{l+1}^T(L_K){\pi }_{t+1}^T(L_K)\xi (f_t,z_t),{\mathbb{E}}_{z_l}\xi (f_l,z_l)〉}_{\rho }=0.\hfill \end{array}$$

Obviously, the above equality holds for $l<t\le T.$ So, with (7), we get

$$\begin{array}{c}{\eta }^2\sum _{t=1}^T\sum _{l=1}^T{\mathbb{E}}_T{〈{\pi }_{t+1}^T(L_K)\xi (f_t,z_t),{\pi }_{l+1}^T(L_K)\xi (f_l,z_l)〉}_{\rho }\hfill \\ ={\eta }^2\sum _{t=1}^T{\mathbb{E}}_t\left[{∥{\pi }_{t+1}^T(L_K)\xi (f_t,z_t)∥}_{\rho }^2\right]\le {\eta }^2\sum _{t=1}^T{∥{\pi }_{t+1}^T(L_K)L_K^{\frac{1}{2}}∥}^2{\mathbb{E}}_t\left[{∥L_K^{-\frac{1}{2}}\xi (f_t,z_t)∥}_{\rho }^2\right]\hfill \\ ={\eta }^2\sum _{t=1}^T{∥{\pi }_{t+1}^T(L_K)L_K^{\frac{1}{2}}∥}^2{\mathbb{E}}_t\left[{∥\xi (f_t,z_t)∥}_K^2\right].\hfill \end{array}$$

To bound ${\mathbb{E}}_t\left[{∥\xi (f_t,z_t)∥}_K^2\right],$ we have

$$\begin{array}{c}{\mathbb{E}}_t\left[{∥\xi (f_t,z_t)∥}_K^2\right]={\mathbb{E}}_t\left[{∥(f_t(x_t)-y_t)K_{x_t}∥}_K^2\right]-{∥{\mathbb{E}}_t\left[(f_t(x_t)-y_t)K_{x_t}\right]∥}_K^2\hfill \\ \le {\mathbb{E}}_{t-1}{\mathbb{E}}_{z_t}\left[{∥(f_t(x_t)-y_t)K_{x_t}∥}_K^2\right]\le {\mathbb{E}}_{t-1}{\mathbb{E}}_{z_t}\left[{(f_t(x_t)-y_t)}^2\right]={\mathbb{E}}_{t-1}\left[\mathcal{E}(f_t)\right]\hfill \end{array}$$

where the last inequality is derived from (3). Applying Lemma A1 with $\beta =\frac{1}{2}$, $l=t+1$ and $k=T,$ we have

$$\begin{array}{c}\sum _{t=1}^T{∥{\pi }_{t+1}^T(L_K)L_K^{\frac{1}{2}}∥}^2=\sum _{t=1}^{T-1}{∥{\pi }_{t+1}^T(L_K)L_K^{\frac{1}{2}}∥}^2+{∥{\pi }_{T+1}^T(L_K)L_K^{\frac{1}{2}}∥}^2\hfill \\ \le \sum _{t=1}^{T-1}\frac{2{\left({(1/2e)}^{1/2}+1\right)}^2}{1+\eta (T-t)}+1\le \sum _{t=1}^T\frac{2{\left({(1/2e)}^{1/2}+1\right)}^2}{1+\eta (T-t)}.\hfill \end{array}$$

Based on the above analysis, we have

$$\begin{array}{c}\hfill {\mathbb{E}}_T\left[{∥\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K)\xi (f_t,z_t)∥}_{\rho }^2\right]\le {\eta }^2\sum _{t=1}^T\frac{2{\left({(1/2e)}^{1/2}+1\right)}^2}{1+\eta (T-t)}{\mathbb{E}}_{t-1}\left[\mathcal{E}(f_t)\right].\end{array}$$

(21)

Now, we estimate the last term on the right-hand side of (19). Using (20) again, we have

$$\begin{array}{c}{\mathbb{E}}_T\left[{〈{\pi }_1^T(L_K)f_{\rho },\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K)\xi (f_t,z_t)〉}_{\rho }\right]\hfill \\ ={〈{\pi }_1^T(L_K)f_{\rho },\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K){\mathbb{E}}_{t-1}{\mathbb{E}}_{z_t}\left[\xi (f_t,z_t)\right]〉}_{\rho }=0.\hfill \end{array}$$

(22)

Plugging (21) and (22) into (19), we get

$$\begin{array}{c}{\mathbb{E}}_T\left[{∥-{\pi }_1^T(L_K)f_{\rho }+\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K)\xi (f_t,z_t)∥}_{\rho }^2\right]={∥{\pi }_1^T(L_K)f_{\rho }∥}_{\rho }^2\hfill \\ +{\eta }^2\sum _{t=1}^T\frac{2{\left({(1/2e)}^{1/2}+1\right)}^2}{1+\eta (T-t)}{\mathbb{E}}_{t-1}\left[\mathcal{E}(f_t)\right].\hfill \end{array}$$

(23)

This together with (18) yields the desired conclusion (14).

Now we turn to bound $f_{T+1}-f_{\rho }$ in ${\mathcal{H}}_K$-norm. By (17) again, we have

$$\begin{array}{c}\hfill {\mathbb{E}}_T\left[\parallel f_{T+1}-f_{\rho }{\parallel }_K^2\right]\le 2{\mathbb{E}}_T\left[{∥-{\pi }_1^T(L_K)f_{\rho }+\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K)\xi (f_t,z_t)∥}_K^2\right]+2{\mathbb{E}}_T\left[{∥\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K){\Delta }_t∥}_K^2\right].\end{array}$$

Following the similar procedure in estimating (14), we also get

$$\begin{array}{c}\hfill {\mathbb{E}}_T\left[\parallel f_{T+1}-f_{\rho }{\parallel }_K^2\right]\le 2{∥{\pi }_1^T(L_K)f_{\rho }∥}_K^2+2{\eta }^2\sum _{t=1}^T{∥{\pi }_{t+1}^T(L_K)∥}^2{\mathbb{E}}_{t-1}\left[\mathcal{E}(f_t)\right]+2{\mathbb{E}}_T\left[{∥\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K){\Delta }_t∥}_K^2\right].\end{array}$$

Noticing that $∥{\pi }_{t+1}^T(L_K)∥\le 1,$ then the bound (15) is obtained. □

Based on the error bounds of $f_{T+1}-f_{\rho }$ in Proposition 1, we need to estimate the generalization error $\mathcal{E}(f_t).$

Lemma 2.

Define ${\{f_t\}}_{t=1}^{T+1}$ by (5). If

$$\begin{array}{c}\hfill 0<\eta \le min\left\{1,\frac{1}{8}{\left({(1/2e)}^{1/2}+1\right)}^{-2}{(log(et)+1)}^{-1}\right\},\end{array}$$

(24)

then for $t\ge 2,$

$$\begin{array}{c}\hfill {\mathbb{E}}_{t-1}\left[\mathcal{E}(f_t)\right]\le 2\mathcal{E}(f_{\rho })+4{\parallel f_{\rho }\parallel }_{\rho }^2+64{\eta }^2{\sigma }^{-4}{(t-1)}^2{\left(\underset{1\le k\le t-1}{sup}\{\parallel f_k{\parallel }_K,M\}\right)}^6.\end{array}$$

(25)

Proof. 

We shall prove (25) by induction. Obviously, (25) holds for $t=2.$ Suppose (25) holds for $t\ge 2.$ Applying (14) with $T=t$, then

$$\begin{array}{cc}\hfill {\mathbb{E}}_t\left[\parallel f_{t+1}-f_{\rho }{\parallel }_{\rho }^2\right]& \le 2{∥{\pi }_1^t(L_K)f_{\rho }∥}_{\rho }^2+2{\eta }^2\sum _{k=1}^t\frac{2{\left({(1/2e)}^{1/2}+1\right)}^2}{1+\eta (t-k)}{\mathbb{E}}_{k-1}\left[\mathcal{E}(f_k)\right]+2{\mathbb{E}}_t\left[{∥\eta \sum _{k=1}^t{\pi }_{k+1}^t(L_K){\Delta }_k∥}_{\rho }^2\right]\hfill \\ & \le 2{∥{\pi }_1^t(L_K)f_{\rho }∥}_{\rho }^2+2{\eta }^2\sum _{k=1}^t\frac{2{\left({(1/2e)}^{1/2}+1\right)}^2}{1+\eta (t-k)}{\mathbb{E}}_{k-1}\left[\mathcal{E}(f_k)\right]+2{\eta }^2{\left(\sum _{k=1}^t\parallel {\pi }_{k+1}^t(L_K)\parallel \parallel {\Delta }_k{\parallel }_{\infty }\right)}^2.\hfill \end{array}$$

(26)

Since the Gaussian G is Lipschitz continuous, we have that for each $1\le k\le T,$

$$\begin{array}{cc}\hfill \parallel {\Delta }_k{\parallel }_{\infty }& \le {∥\left[G(0)-G\left(-\frac{{(f_k(x_k)-y_t)}^2}{2{\sigma }^2}\right)\right]\left[f_k(x_k)-y_k\right]K_{x_k}∥}_{\infty }\hfill \\ & \le \left|\left[G(0)-G\left(-\frac{{(f_k(x_k)-y_k)}^2}{2{\sigma }^2}\right)\right]\left[f_t(x_k)-y_k\right]\right|{\parallel K_{x_k}\parallel }_K\hfill \\ & \le \frac{{(f_k(x_k)-y_k)}^2}{2{\sigma }^2}\left|f_k(x_k)-y_k\right|\le \frac{(\parallel f_k{{\parallel }_{\infty }+M)}^3}{2{\sigma }^2}\le \frac{(\parallel f_k{{\parallel }_K+M)}^3}{2{\sigma }^2}\hfill \end{array}$$

where the last inequality is derived from (3).

Notice that by $0<\eta \le 1,$ there holds $\parallel {\pi }_k^t(L_K)\parallel \le {\prod }_{l=k}^t\parallel I-\eta L_K\parallel \le 1$ for each $1\le k\le t\le T.$ Then the last term on the right-hand side of (26) is bounded as

$$\begin{array}{c}\hfill 2{\eta }^2{\left(\sum _{k=1}^t\parallel {\pi }_{k+1}^t(L_K)\parallel \parallel {\Delta }_k{\parallel }_{\infty }\right)}^2\le 32{\eta }^2{\sigma }^{-4}{t}^2{\left(\underset{1\le k\le t}{sup}\{\parallel f_k{\parallel }_K,M\}\right)}^6.\end{array}$$

(27)

For the first term $2{∥{\pi }_1^t(L_K)f_{\rho }∥}_{\rho }^2$, it is easy to get that $2{∥{\pi }_1^t(L_K)f_{\rho }∥}_{\rho }^2\le 2{\parallel f_{\rho }\parallel }_{\rho }^2.$

Putting the above estimates into (26) and using the relation (13) with $f=f_{t+1},$ we have

$$\begin{array}{c}{\mathbb{E}}_t\left[\mathcal{E}(f_{t+1})\right]={\mathbb{E}}_t\left[\parallel f_{t+1}-f_{\rho }{\parallel }_{\rho }^2\right]+\mathcal{E}(f_{\rho })\hfill \\ \le \mathcal{E}(f_{\rho })+2{\parallel f_{\rho }\parallel }_{\rho }^2+32{\eta }^2{\sigma }^{-4}{t}^2{\left(\underset{1\le k\le t}{sup}\{\parallel f_k{\parallel }_K,M\}\right)}^6+2{\eta }^2\sum _{k=1}^t\frac{2{\left({(1/2e)}^{1/2}+1\right)}^2}{1+\eta (t-k)}{\mathbb{E}}_{k-1}\left[\mathcal{E}(f_k)\right]\hfill \\ \le \mathcal{E}(f_{\rho })+2{\parallel f_{\rho }\parallel }_{\rho }^2+32{\eta }^2{\sigma }^{-4}{t}^2{\left(\underset{1\le k\le t}{sup}\{\parallel f_k{\parallel }_K,M\}\right)}^6\hfill \\ +2{\eta }^2\sum _{k=1}^t\frac{2{\left({(1/2e)}^{1/2}+1\right)}^2}{1+\eta (t-k)}\left(2\mathcal{E}(f_{\rho })+4{\parallel f_{\rho }\parallel }_{\rho }^2+64{\eta }^2{\sigma }^{-4}{(t-1)}^2{\left(\underset{1\le k\le t-1}{sup}\{\parallel f_k{\parallel }_K,M\}\right)}^6\right).\hfill \end{array}$$

(28)

By the restriction (24) of $\eta$ and Lemma A3, we know that

$$\begin{array}{c}\hfill 2{\eta }^2\sum _{k=1}^t\frac{2{\left({(1/2e)}^{1/2}+1\right)}^2}{1+\eta (t-k)}\le 4\eta {\left({(1/2e)}^{1/2}+1\right)}^2(log(et)+1)\le \frac{1}{2}.\end{array}$$

Plugging it into (28), we have

$$\begin{array}{c}{\mathbb{E}}_t\left[\mathcal{E}(f_{t+1})\right]\le \mathcal{E}(f_{\rho })+2{\parallel f_{\rho }\parallel }_{\rho }^2+32{\eta }^2{\sigma }^{-4}{t}^2{\left(\underset{1\le k\le t}{sup}\{\parallel f_k{\parallel }_K,M\}\right)}^6\hfill \\ +\frac{1}{2}\left(2\mathcal{E}(f_{\rho })+4{\parallel f_{\rho }\parallel }_{\rho }^2+64{\eta }^2{\sigma }^{-4}{(t-1)}^2{\left(\underset{1\le k\le t-1}{sup}\{\parallel f_k{\parallel }_K,M\}\right)}^6\right)\hfill \\ \le 2\mathcal{E}(f_{\rho })+4{\parallel f_{\rho }\parallel }_{\rho }^2+64{\eta }^2{\sigma }^{-4}{t}^2{\left(\underset{1\le k\le t}{sup}\{\parallel f_k{\parallel }_K,M\}\right)}^6.\hfill \end{array}$$

Then the proof is completed. □

With these preliminaries in place, we shall prove our main results.

Proof of Theorem 1.

We shall prove Theorem 1 by Proposition 1. First, we will use (14) to estimate the error rate for (5) in $L_{{\rho }_{\mathcal{X}}}^2$-space. For the first term on the right-hand side of (14), applying Lemma A2 with $f=f_{\rho }$ and $\eta =T^{-\frac{2r}{2r+1}}$, we have that

$$\begin{array}{c}\hfill {∥{\pi }_1^T(L_K)f_{\rho }∥}_{\rho }^2\le 4{\left({(r/e)}^r+1\right)}^2\parallel L_K^{-r}{f}_{\rho }{\parallel }_{\rho }^2{T}^{-\frac{2r}{2r+1}}=4{\left({(r/e)}^r+1\right)}^2{\parallel g\parallel }_{\rho }^2{T}^{-\frac{2r}{2r+1}}.\end{array}$$

For the second term on the right-hand side of (14), the choice of $\eta$ and T in Theorem 1 implies that the restriction (24) holds. Then we can put the bound (12) into (25) and get that for $t\ge 2$

$$\begin{array}{c}{\mathbb{E}}_{t-1}\left[\mathcal{E}(f_t)\right]\le 2\mathcal{E}(f_{\rho })+4{\parallel f_{\rho }\parallel }_{\rho }^2+64M^6{\sigma }^{-4}{\eta }^5{(t-1)}^5\hfill \\ \le \left(2\mathcal{E}(f_{\rho })+4{\parallel f_{\rho }\parallel }_{\rho }^2+64M^6\right)(1+{\sigma }^{-4}{\eta }^5{(t-1)}^5)\hfill \\ \le \left(2\mathcal{E}(f_{\rho })+4{\parallel f_{\rho }\parallel }_{\rho }^2+64M^6\right)(1+{\sigma }^{-4}{\eta }^5{T}^5):=c_{M,\rho }(1+{\sigma }^{-4}{T}^{\frac{5}{2r+1}}).\hfill \end{array}$$

This together with Lemma A3 yields that

$$\begin{array}{cc}\hfill {\eta }^2\sum _{t=1}^T\frac{2{\left({(1/2e)}^{1/2}+1\right)}^2}{1+\eta (T-t)}{\mathbb{E}}_{t-1}\left[\mathcal{E}(f_t)\right]& \le 2{\left({(1/2e)}^{1/2}+1\right)}^2{c}_{M,\rho }\eta (log(eT)+1)(1+{\sigma }^{-4}{T}^{\frac{5}{2r+1}})\hfill \\ & \le 4{\left({(1/2e)}^{1/2}+1\right)}^2{c}_{M,\rho }log(T)(T^{-\frac{2r}{2r+1}}+{\sigma }^{-4}{T}^{\frac{5-2r}{2r+1}}).\hfill \end{array}$$

Finally, we bound the last term on the right-hand side of (14). Notice that

$$\begin{array}{c}\hfill {∥\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K){\Delta }_t∥}_{\rho }\le \eta \sum _{t=1}^T\parallel {\pi }_{t+1}^t(L_K)\parallel \parallel {\Delta }_t{\parallel }_{\infty }.\end{array}$$

Then, using the estimate (27) and the bound (12) of $\{f_t\}$, we have

$$\begin{array}{c}{\mathbb{E}}_T\left[{∥\eta \sum _{t=1}^T{\pi }_{t+1}^T(L_K){\Delta }_t∥}_{\rho }^2\right]\le {\eta }^2{\left(\sum _{t=1}^T\parallel {\pi }_{t+1}^T(L_K)\parallel \parallel {\Delta }_t{\parallel }_{\infty }\right)}^2\hfill \\ \le 16{\eta }^2{\sigma }^{-4}{t}^2{\left(\underset{1\le t\le T}{sup}\{\parallel f_t{\parallel }_K,M\}\right)}^6\le 16M^6{\sigma }^{-4}{T}^{\frac{5}{2r+1}}.\hfill \end{array}$$

Based on the above analysis, the conclusion (10) is obtained by taking

$$C=8{\left({(r/e)}^r+1\right)}^2{\parallel g\parallel }_{\rho }^2+16{\left({(1/2e)}^{1/2}+1\right)}^2{c}_{M,\rho }+32M^6.$$

Similarity, we can get the conclusion (11) by taking

$$C^{\prime }=8{\left({((2r-1)/2e)}^{r-\frac{1}{2}}+1\right)}^2{\parallel g\parallel }_{\rho }^2+8c_{M,\rho }+32M^6.$$

 □