Stochastic Gradient Descent for Kernel-Based Maximum Correntropy Criterion

Abstract

Maximum correntropy criterion (MCC) has been an important method in machine learning and signal processing communities since it was successfully applied in various non-Gaussian noise scenarios. In comparison with the classical least squares method (LS), which takes only the second-order moment of models into consideration and belongs to the convex optimization problem, MCC captures the high-order information of models that play crucial roles in robust learning, which is usually accompanied by solving the non-convexity optimization problems. As we know, the theoretical research on convex optimizations has made significant achievements, while theoretical understandings of non-convex optimization are still far from mature. Motivated by the popularity of the stochastic gradient descent (SGD) for solving nonconvex problems, this paper considers SGD applied to the kernel version of MCC, which has been shown to be robust to outliers and non-Gaussian data in nonlinear structure models. As the existing theoretical results for the SGD algorithm applied to the kernel MCC are not well established, we present the rigorous analysis for the convergence behaviors and provide explicit convergence rates under some standard conditions. Our work can fill the gap between optimization process and convergence during the iterations: the iterates need to converge to the global minimizer while the obtained estimator cannot ensure the global optimality in the learning process.

1. Introduction

Non-Gaussian noise and outliers are ubiquitous in many real-life applications, and robustness analysis plays an important role in the fields of signal processing and machine learning [1,2,3,4,5,6,7,8,9]. For regression problems, the classical least squares (LS) method is the most widely used tool. The LS method refers to the framework of second-order statistics due to its minimization of the variance of the prediction error, and the success of the LS method depends heavily on the assumption of Gaussianity. However, the LS method has poor performance when data may be contaminated by non-Gaussian noise or outliers. This motivates the application of MCC into robust regression problems as the correntropy loss can capture high-order moment information contained in the data. Due to its robustness to non-Gaussian noise, MCC has found wide applications in solving optimization problems in various scientific and engineering areas such as signal processing, regression analysis, feature selection, and data clustering; see [10,11,12,13,14,15,16,17,18,19,20,21] and the references therein.

Stochastic gradient descent (SGD) is a popular optimization method to solve large-scale computation problems due to its low memory requirement, computational complexity, and promising learning performances; see [22,23,24]. As an iterative method, SGD calculates a gradient randomly based on a random event and updates iterates along the direction of the negative gradient. Theoretical properties of SGD are well studied for optimization with both convex and strongly convex loss functions, but the existing theoretical results in the non-convex setting are not mature. Recently, non-convex optimization with SGD has attracted significant interest from researchers regarding the development of deep neural nets. In the paper, we employ SGD to solve the optimization problems induced by kernel-based MCC. As the correntropy loss is not convex, a rigorous theoretical foundation for applying SGD to MCC is not clear and will be investigated in the work.

Kernel methods are efficient tools for dealing with non-linear features in models. Here, the learning process of the kernel MCC method is associated with a Mercer kernel $K$. The function $K:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ is said to be a Mercer kernel if it is continuous, symmetric and positive semidefinite on $\mathcal{X}\times \mathcal{X}$. It induces a reproducing kernel Hilbert space (RKHS) ${\mathcal{H}}_K$ [25], which is defined to be the completion of the linear span of $\{K_x=K(·,x),x\in \mathcal{X}\}$ with the inner product ${\langle K(·,x),K(·,u)\rangle }_K=K(x,u),\forall x,u\in \mathcal{X}$. and the reproducing property

$$\begin{array}{c}\hfill f\left(x\right)={\langle K(·,x),f\rangle }_K,\forall f\in {\mathcal{H}}_K,x\in \mathcal{X}.\end{array}$$

(1)

So, the following holds

$$\begin{array}{c}\hfill {\parallel f\parallel }_{\infty }\le {\displaystyle \underset{x\in \mathcal{X}}{sup}}\sqrt{K(x,x)}{\parallel f\parallel }_K,\forall f\in {\mathcal{H}}_K.\end{array}$$

(2)

In this paper, we attempt to present the convergence analysis of the kernel version of SGD applied to the correntropy loss,

$${\varphi }_{\sigma }\left(u\right)=exp\left\{-{\displaystyle \frac{u^2}{{\sigma }^2}}\right\},u\in \mathbb{R},$$

where $\sigma >0$ is the robustness parameter. Assume $X\in {\mathbb{R}}^d$ is a vector of explanatory variables, $Y\in \mathbb{R}$ is the response variable. Given a prediction function f, the correntropy loss ${\varphi }_{\sigma }$ provides a measure of the error prediction error $E=f\left(X\right)-Y,$ that is

$$\begin{array}{c}\hfill {\varphi }_{\sigma }\left(E\right)=exp\left\{-{\displaystyle \frac{{(f\left(X\right)-Y)}^2}{{\sigma }^2}}\right\},\sigma >0.\end{array}$$

(3)

The purpose of the maximum correntropy criterion is to find a good predictor by maximizing (3). Given a set of independent and identically distributed (i.i.d.) observations, $\mathbf{z}={\left\{(x_i,y_i)\right\}}_{i=1}^T,$ the implementation of MCC by SGD in an RKHS ${\mathcal{H}}_K$ is defined with an initial value of $f_1=0$ and the updating rule

$$\begin{array}{c}\hfill f_{t+1}=f_t-{\eta }_t{\displaystyle \frac{2u_t}{{\sigma }^2}}exp\left\{-{\displaystyle \frac{u_t^2}{{\sigma }^2}}\right\}{K}_{x_t},t=1,\cdots ,T\end{array}$$

(4)

where $u_t=f\left(x_t\right)-y_t$ and ${\eta }_t>0$ are the step size.

In a previous work [7,9], an analysis of the implementation of MCC mainly focuses on the convergence of $\left\{f_t\right\}$ to the regression function $\mathbf{E}\left(Y\right|X=x)$, which is the value of the conditional mean of the output Y. However, in a non-parametric context, the target function of MCC varies with the choice of the robustness parameter $\sigma .$ In the following paper [26,27], the authors established a sound theoretical foundation for MCC and provided a full analysis of the prediciton ability of MCC. They showed that MCC indeed belonged to the robust regression scheme when $\sigma$ was large enough, but when $\sigma \to 0$, the target function of MCC was a modal regression fucntion; that is, ${\displaystyle \underset{y\in Y}{max}p\left(y\right|X=x)}$ ($p(·|X=x)$ denotes the condtional density function). Thus, this paper will investigate the convegence ability of SGD applied to MCC for any $\sigma >0.$

The contributions of the work are as follows:

(a)

This work establishes the theoretical foundations for SGD applied to MCC. Some important convergence properties are provided, in which the role of the robustness parameter $\sigma$ is presented.

(b)

We introduce the Polyak–Łojasiewicz (PL) condition to derive the explicit convergence rates of algorithm (4). A global linear convergence rate is achieved when the step size ${\eta }_t$ is chosen properly.

The rest of the paper is organized as follows. In Section 2, after introducing necessary notations and assumptions that underlie our analysis, we provide and discuss our main theorems. Some discussions on the kernelized MCC are provided in Section 3. We provide the simulations in Section 4. Section 5 is devoted to the technical proof. We conclude this paper with Section 6.

2. Main Results

This paper investigates the convergence of algorithm (4) in a non-parametric context. Some necessary notations and assumptions are given. Let $\rho$ be a Borel probability measure on the product space $\mathcal{Z}:=\mathcal{X}\times \mathcal{Y}$. The samples $\mathbf{z}={\left\{z_i:=(x_i,y_i)\right\}}_{i=1}^T\subset \mathcal{Z}$ are drawn independently according to $\rho .$ Let ${\rho }_{\mathcal{X}}$ be the marginal distribution of $\rho$ on $\mathcal{X}$, and $\rho \left(y\right|x)$ be the conditional distribution on $\mathcal{Y}$ given $x\in \mathcal{X}$. Associated with $({\varphi }_{\sigma },\rho )$, the generalization error$\mathcal{E}$ for $f:\mathcal{X}\to \mathcal{Y}$ is given by

$$\begin{array}{c}\hfill \mathcal{E}\left(f\right)={\int }_{\mathcal{Z}}{\varphi }_{\sigma }(f\left(x\right)-y)d\rho ,\end{array}$$

which is well defined for any measurable function f. We assume that there exists a function $f^{\ast }$ that maximizes the expected correntropy error. In this sense, the goal of MCC learning is to find an approximation of $f^{\ast }$ through only the training set $\mathbf{z}={\left\{z_i:=(x_i,y_i)\right\}}_{i=1}^T$ when measure $\rho$ is unknown. The prediction ability to algorithm (4) is estimated by $\mathcal{E}\left(f^{\ast }\right)-\mathcal{E}\left(f_{T+1}\right)$. In the rest of the work, we denote the gradient of the functional $\mathcal{E}\left(f\right)$ in a RKHS ${\mathcal{H}}_K$ by $\nabla \mathcal{E}\left(f\right)$.

Theorem 1. 

Define $\left\{f_t\right\}$ by algorithm (4) and take the step size ${\sum }_{t=1}^{\infty }{\eta }_t^2<\infty$. Then, the following statements holds.

(a) 

The generalization error $\mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_{t+1}\right)$ is uniformly bounded. More precisely, there is some constant C, such that

$$\mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_{t+1}\right)\le C(1+{\sigma }^{-4}),t=1,\cdots ,T.$$

(b) 

There is some constant $C>0$, such that

$$\underset{t=1,\cdots ,T}{inf}\mathbf{E}{\parallel \nabla \mathcal{E}\left(f_t\right)\parallel }_K^2\le C(1+{\sigma }^{-4}){\left(\sum _{t=1}^T{\eta }_t\right)}^{-1},$$

where the constant C will be given explicitly in the proof.

Remark 1. 

Statement (a) tells us that the error $\mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_t\right)$ converges to a bounded random variable without any boundness requirement on $f_t$ appearing in previous work [22,28]. That is because of the so-called redescending property of the correntropy loss. We say that loss $\ell (·)$ satisfies the redescending property if ${\ell }^{\prime }(·)$ is non-decreasing near the origin, but decreasing towards 0 far from the origin. At present, there are a great deal of papers to describe the redescending property in robustness analysis [29,30].

Statement (b) shows that after T iterations, there exists a single iterate, such that its gradient converges to 0 if we choose the step size ${\sum }_{t=1}^{\infty }{\eta }_t=\infty .$ Explicit rates are provided in the following corollary by instantiating the step sizes.

Corollary 1. 

Define $\left\{f_t\right\}$ by algorithm (4), we have

(a) 

If ${\eta }_t=t^{-1+ϵ}$ for any $0<ϵ<1$, then

$$\begin{array}{c}\hfill {\displaystyle \underset{t=1,\cdots ,T}{inf}}\mathbf{E}{\parallel \nabla \mathcal{E}\left(f_t\right)\parallel }_K^2=\mathcal{O}\left(T^{-1+ϵ}\right).\end{array}$$

(b) 

If ${\eta }_t=t^{-{\displaystyle \frac{1}{2}}}logt$, then

$$\begin{array}{c}\hfill {\displaystyle \underset{t=1,\cdots ,T}{inf}}\mathbf{E}{\parallel \nabla \mathcal{E}\left(f_t\right)\parallel }_K^2=\mathcal{O}\left(T^{-{\displaystyle \frac{1}{2}}}logT\right).\end{array}$$

As we see, the convergence rate of algorithm (4) varies with the step size and can achieve the order $\mathcal{O}\left(T^{-1}\right)$, as presented in statement (a).

In the following, we improve our convergence analysis by imposing the Polyak–Łojasiewicz (PL) condition, which has been explored to show fast convergence rates for SGD without convexity or strong convexity, see [31,32].

Assumption 1. 

We say that $\mathcal{E}\left(f\right)$ satisfies Polyak–Łojasiewicz (PL) condition if the following holds for some $\mu >0$

$$\begin{array}{c}\hfill \mathcal{E}\left(f^{\ast }\right)-\mathcal{E}\left(f\right)\le {\mu }^{-1}{\parallel \nabla \mathcal{E}\left(f\right)\parallel }_{\infty }^2\end{array}$$

(5)

for any measurable function $f:\mathcal{X}\to \mathcal{Y}.$

This condition was originally introduced by Polyak [33] and is commonly used in non-convex gradient descent analysis [32]. It simply requires that the gradient grows faster than a quadratic distance between the generalization error, and is a sufficient condition to guarantee a global linear convergence rate for the gradient descent. In this work, we use the PL condition to provide a full analysis of SGD for MCC implementation.

Theorem 2. 

Define $\left\{f_t\right\}$ by algorithm (4) and suppose (5) holds.

(a) 

If ${\sum }_{t=1}^{\infty }{\eta }_t^2<\infty$ and ${\sum }_{t=1}^{\infty }{\eta }_t=\infty$, then

$$\underset{t\to \infty }{lim}\mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_{t+1}\right)=0.$$

(b) 

If ${\sum }_{t=1}^{\infty }{\eta }_t^2<\infty$ and ${\sum }_{t=1}^{\infty }{\eta }_t=\infty$, then

$$\underset{t\to \infty }{lim}{\parallel \nabla \mathcal{E}\left(f_t\right)\parallel }_K^2=0.$$

(c) 

If there exists some $t_0\in \mathbb{N}$, such that for any $t\ge t_0$, ${\eta }_t\le {\displaystyle \frac{{\sigma }^2}{4}}$, then

$$\begin{array}{c}\hfill \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_{t+1}\right)\le {\displaystyle \prod _{j=1}^t}\left(1-{\displaystyle \frac{{\eta }_j{\kappa }^{-2}\mu }{2}}\right)\left(\mathcal{E}\left(f^{\ast }\right)-\mathcal{E}\left(f_1\right)\right).\end{array}$$

Remark 2. 

Statements (a) and (b) establish sufficient conditions to guarantee almost certain convergence in SGD applied to MCC. Statement (c) tells us that with a constant step size of ${\eta }_t\equiv {\displaystyle \frac{{\sigma }^2}{4}}$, algorithm (4) has a global linear convergence rate $\mathcal{O}\left({\left(1-{\displaystyle \frac{{\sigma }^2{\kappa }^{-2}\mu }{8}}\right)}^T\right)$. It also shows that the value of the robustness parameter σ plays an important role in the choice of stepsize for deriving fast convergence rates.

3. Discussions

Recent theoretical works on the global solutions to SDG generated by nonconvex losses are limited. Inspired by the work of [34,35], MCC is based on a smooth loss that can be utilized in the theoretical analysis and sufficient convergence conditions can be derived for MCC. Thus, a fine theoretical foundation kernel-based MCC is established in the work, allthough no convexity property assumption has been imposed on the optimization problems. We show that the usual bounded gradient assumptions on nonconvex learning can be removed without affecting the convergence rates. By using a condition referred to as PL inequality, we find that the last iteration of SGD with kernel-based MCC can enjoy fast convergence speeds. To the best of our knowledge, this work provides the first theoretical analysis of MCC that establishes a global linear convergence rate.

We review some work related to the kernelized MCC. In linear additive noise models, the studies by [7,26,35] investigated regression models associated with correntropy-induced losses, demonstrating that the scale parameter in the learning process balances the convergence rates of the regression model with its robustness. However, their approaches rely solely on batch algorithms, which are limited in streaming data scenarios and large-scale computational settings. Moreover, the convergence rates achieved in their work are noticeably inferior to those established in our study (see Theorem 2).

We turn to the theoretical works on the gradient descent in training kernelized MCC. In the works of [10,21,36], the authors focussed on the learning performance and convergence ability in terms of the target functions, defined as the conditional mean $\mathbf{E}\left(Y\right|X=x).$ But, it has been proved in [26] that in MCC models, for a good generalization ability and robustness properties, the minimizer of the generalization error $\mathcal{E}\left(f\right)$ is not necessarily $\mathbf{E}\left(Y\right|X=x)$. Our work does not impose extra conditions on the target functions and can apply to general learning schemes. In addition, our analysis relaxes thethe requirement for the choice of $\sigma$, which must be sufficiently large in [10,21,36].

4. Simulation Validation

In this section, we provide some experimental results to confirm our theoretical results. These experiments are simulated from the regression model $y_i=f^{\ast }\left(x_i\right)+ϵ$, where $f^{\ast }\left(x_i\right)=x_i(1-x_i)$ and the random samples $x_i$ are drawn independently according to the uniform distribution $U[-1,2]$. To carry out the experiments, the noise $ϵ$ is generated as follows.

• Gaussion noise: $ϵ$∼$N(0,0.1^2);$
• outlier noise: $90\%$ noise ∼$N(0,0.1^2)$, $10\%$ noise ∼$U[-1,1];$
• skewed noise: $ϵ=z_1-z_2$, where $z_1,z_2\sim Exp\left(0.2\right);$
• GMM noise: $90\%$ noise ∼$N(0,0.1^2)$, $10\%$ noise is generated by the following density probability

$$\begin{array}{c}\hfill p\left(ϵ\right)={\displaystyle \frac{1}{2\sqrt{2\pi }\times 0.1}}exp\left\{-{\displaystyle \frac{{(ϵ-0.5)}^2}{2\times 0.1^2}}\right\}+{\displaystyle \frac{1}{2\sqrt{2\pi }\times 0.1}}exp\left\{-{\displaystyle \frac{{(ϵ+0.5)}^2}{2\times 0.1^2}}\right\}.\end{array}$$

We generate a dataset of 1000 samples for the above four types of noise. See Figure 1.

We chose the Gaussian kernel function $G(x,u)=exp\left\{-{\displaystyle \frac{{\parallel x-u\parallel }^2}{2}}\right\}$ for the kernel MCC. We applied SGD to the kernel MCC with stepsizes of $\eta =0.1$ and $\sigma =0.1.$ For each type of noise, we ran the SGD for T = 1000 times. The generalization errors $\mathcal{E}\left(f_t\right)$ with iterations t are plotted in Figure 2. It is observed that for different types of noise, the generalization errors decreased rapidly as iterations progressed. This shows the efficiency of SGD for MCC in dealing with non-Gaussain noise. This coincides with our theory.

5. Proofs

Now, we provide the proofs for our main results.

Proof of Theorem 1. 

As ${\varphi }_{\sigma }^{\prime }\left(u\right)=-{\displaystyle \frac{2u}{{\sigma }^2}}exp\left\{-{\displaystyle \frac{u^2}{{\sigma }^2}}\right\}$, we know

$$\begin{array}{c}\hfill |{\varphi }_{\sigma }^{\prime }\left(u\right)-{\varphi }_{\sigma }^{\prime }\left(v\right)|\le {\displaystyle \frac{4}{{\sigma }^2}}|u-v|,u,v\in \mathbb{R}.\end{array}$$

(6)

It follows that for any $u,v\in \mathbb{R},$

$$\begin{array}{c}\hfill -{\varphi }_{\sigma }\left(u\right)\le -{\varphi }_{\sigma }\left(v\right)+{\varphi }_{\sigma }^{\prime }\left(v\right)(u-v)+{\displaystyle \frac{2}{{\sigma }^2}}{|u-v|}^2.\end{array}$$

(7)

Applying the inequality above with $u=f_{t+1}\left(x\right)-y$, $v=f_t\left(x\right)-y$, then by (1)

$$\begin{array}{cc}\hfill -{\varphi }_{\sigma }(f_{t+1}\left(x\right)-y)& \le -{\varphi }_{\sigma }(f_t\left(x\right)-y)+{\varphi }_{\sigma }^{\prime }(f_t\left(x\right)-y)(f_{t+1}\left(x\right)-f_t\left(x\right))+{\displaystyle \frac{2}{{\sigma }^2}}{|f_{t+1}\left(x\right)-f_t\left(x\right)|}^2\hfill \\ \hfill & \le {\varphi }_{\sigma }(f_t\left(x\right)-y)+{\langle f_{t+1}-f_t,{\varphi }_{\sigma }^{\prime }(f_t\left(x\right)-y)K_x\rangle }_K+{\displaystyle \frac{2}{{\sigma }^2}}{|f_{t+1}\left(x\right)-f_t\left(x\right)|}^2.\hfill \end{array}$$

(8)

Taking the expectation with respect to $z=(x,y)$ on both sides, and using (4)

$$\begin{array}{cc}\hfill -\mathcal{E}\left(f_{t+1}\right)& \le -\mathcal{E}\left(f_t\right)-{\langle f_{t+1}-f_t,-\nabla \mathcal{E}\left(f_t\right)\rangle }_K+{\displaystyle \frac{2{\eta }_t^2}{{\sigma }^2}}{\int }_{\mathcal{Z}}{\left|{\displaystyle \frac{2u_t}{{\sigma }^2}}exp\left\{-{\displaystyle \frac{u_t^2}{{\sigma }^2}}\right\}{K}_{x_t}\right|}^{2}d\rho \hfill \\ \hfill & =-\mathcal{E}\left(f_t\right)-{\langle -{\eta }_t{\varphi }_{\sigma }^{\prime }(f\left(x_t\right)-y_t)K_{x_t},-\nabla \mathcal{E}\left(f_t\right)\rangle }_K+{\displaystyle \frac{2{\eta }_t^2}{{\sigma }^2}}{\int }_{\mathcal{Z}}|{\varphi }_{\sigma }^{\prime }\left(f_{(}{x}_t\right)-y_t{)K_{x_t}|}^{2}d\rho .\hfill \end{array}$$

(9)

For the term $|{\varphi }_{\sigma }^{\prime }\left(f_{(}{x}_t\right)-y_t{\left)K_{x_t}\right|}^2$, using (2), we know

$$\begin{array}{c}\hfill |{\varphi }_{\sigma }^{\prime }\left(f_{(}{x}_t\right)-y_t)K_{x_t}{|}^2\le {\kappa }^2|{\varphi }_{\sigma }^{\prime }\left(f_{(}{x}_t\right)-y_t{\left)\right|}^2={\displaystyle \frac{4u_t^2{\kappa }^2}{{\sigma }^4}}exp\left\{-{\displaystyle \frac{2u_t^2}{{\sigma }^2}}\right\}\le {\displaystyle \frac{2{\kappa }^2}{{\sigma }^2}}.\end{array}$$

(10)

where the last inequality is obtained by $t{e}^{-t}\le 1$ for any $t>0.$ Putting it into (9) yields

$$\begin{array}{c}\hfill -\mathcal{E}\left(f_{t+1}\right)\le -\mathcal{E}\left(f_t\right)-{\langle -{\eta }_t{\varphi }_{\sigma }^{\prime }(f\left(x_t\right)-y_t)K_{x_t},-\nabla \mathcal{E}\left(f_t\right)\rangle }_K+{\displaystyle \frac{4{\kappa }^2{\eta }_t^2}{{\sigma }^4}}.\end{array}$$

Noting that $f_t$ is independent of $z_t$, then by taking the conditional expectation with respect to $z_t$, we derive

$$\begin{array}{c}\hfill -\mathbf{E}\mathcal{E}\left(f_{t+1}\right)\le -\mathcal{E}\left(f_t\right)-{\eta }_t{\parallel \nabla \mathcal{E}\left(f_t\right)\parallel }_K^2+{\displaystyle \frac{4{\kappa }^2{\eta }_t^2}{{\sigma }^4}}\end{array}$$

Taking the conditional expectation with respect to $z_{t-1},z_{t-1},\cdots ,z_1$ sequentially yields

$$\begin{array}{c}\hfill -\mathbf{E}\mathcal{E}\left(f_{t+1}\right)\le -\mathbf{E}\mathcal{E}\left(f_t\right)-{\eta }_t\mathbf{E}{\parallel \nabla \mathcal{E}\left(f_t\right)\parallel }_K^2+{\displaystyle \frac{4{\kappa }^2{\eta }_t^2}{{\sigma }^4}}.\end{array}$$

(11)

This implies that

$$\begin{array}{c}\hfill \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_{t+1}\right)\le \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_t\right)+{\displaystyle \frac{4{\kappa }^2{\eta }_t^2}{{\sigma }^4}}.\end{array}$$

An application of the inequality recursively provides

$$\mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_{t+1}\right)\le \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_1\right)+{\displaystyle \frac{4{\kappa }^2}{{\sigma }^4}}\sum _{j=1}^t{\eta }_j^2.$$

So, we can obtain that for any $t\in \mathbb{N},$

$$\begin{array}{cc}\hfill \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_{t+1}\right)& \le \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_1\right)+{\displaystyle \frac{4{\kappa }^2}{{\sigma }^4}}{\displaystyle \sum _{j=1}^t}{\eta }_j^2\hfill \\ \hfill & =C(1+{\sigma }^{-4})<\infty \hfill \end{array}$$

(12)

where $C:=max\{1,4{\kappa }^2{\sum }_{j=1}^{\infty }{\eta }_j^2\}$ and the last inequality is obtained by $\mathcal{E}\left(f\right)\le 1$ for any measurable function $f:\mathcal{X}\to \mathcal{Y}.$

Then, statement (a) holds.

Now, we provide statement (b). Inequality (11) also provides

$$\begin{array}{c}\hfill {\eta }_t\mathbf{E}{\parallel \nabla \mathcal{E}\left(f_t\right)\parallel }_K^2\le \mathbf{E}\mathcal{E}\left(f_{t+1}\right)-\mathbf{E}\mathcal{E}\left(f_t\right)+{\displaystyle \frac{4{\kappa }^2}{{\sigma }^4}}{\eta }_t^2,t=1,\cdots ,T.\end{array}$$

A summation of the aforementioned inequality then implies

$$\begin{array}{cc}\hfill {\displaystyle \sum _{j=1}^t}{\eta }_t\mathbf{E}{\parallel \nabla \mathcal{E}\left(f_j\right)\parallel }_K^2& \le {\displaystyle \sum _{j=1}^t}\mathbf{E}\mathcal{E}\left(f_{j+1}\right)-\mathbf{E}\mathcal{E}\left(f_j\right)+{\displaystyle \frac{4{\kappa }^2}{{\sigma }^4}}{\displaystyle \sum _{j=1}^t}{\eta }_j^2\le \mathbf{E}\mathcal{E}\left(f_{t+1}\right)+{\displaystyle \frac{4{\kappa }^2}{{\sigma }^4}}{\displaystyle \sum _{j=1}^t}{\eta }_j^2\hfill \\ \hfill & \le 1+{\displaystyle \frac{4{\kappa }^2}{{\sigma }^4}}\sum _{j=1}^t{\eta }_j^2:=C(1+{\sigma }^{-4})<\infty .\hfill \end{array}$$

So, we obtain

$$\underset{j=1,\cdots ,t}{inf}\mathbf{E}{\parallel \nabla \mathcal{E}\left(f_j\right)\parallel }_K^2\sum _{j=1}^t{\eta }_j\le C(1+{\sigma }^{-4}).$$

Then, statement (b) holds. □

Proof of Corollary 1. 

We know that for any $t\in \mathbb{N},$

$$\begin{array}{c}\hfill {\displaystyle \frac{1}{1-\gamma }}\left[{(t+1)}^{1-\gamma }-1\right]\le \sum _{j=1}^t{j}^{-\gamma }\le {\displaystyle \frac{1}{1-\gamma }}{t}^{1-\gamma },\gamma \in (0,1).\end{array}$$

Using statement (b) of Theorem 1 with $\gamma =1-ϵ$ implies that statement (a) holds.

Statement (b) can be similarly proven. □

Proof of Theorem 2. 

For the proof of statement (a), we use estimate (11). Adding $\mathcal{E}\left(f^{\ast }\right)$ on both sides provides

$$\begin{array}{c}\hfill \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_{t+1}\right)\le \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_t\right)-{\eta }_t\mathbf{E}{\parallel \nabla \mathcal{E}\left(f_t\right)\parallel }_K^2+{\displaystyle \frac{4{\kappa }^2{\eta }_t^2}{{\sigma }^4}}.\end{array}$$

Note that the relation (2), then $\parallel \nabla \mathcal{E}\left(f_t\right){\parallel }_{\infty }\le \kappa {\parallel \nabla \mathcal{E}\left(f_t\right)\parallel }_K$ This together with (5) yields

$$\begin{array}{cc}\hfill \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_{t+1}\right)& \le \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_t\right)-{\kappa }^{-2}{\eta }_t\mathbf{E}{\parallel \nabla \mathcal{E}\left(f_t\right)\parallel }_{\infty }^2+{\displaystyle \frac{4{\kappa }^2{\eta }_t^2}{{\sigma }^4}}\hfill \\ \hfill & \le \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_t\right)-{\kappa }^{-2}\mu {\eta }_t\left(\mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_t\right)\right)+{\displaystyle \frac{4{\kappa }^2{\eta }_t^2}{{\sigma }^4}}\hfill \\ \hfill & \le (1-{\mu }^{-2}{\eta }_t)\left(\mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_t\right)\right)+{\displaystyle \frac{4{\kappa }^2{\eta }_t^2}{{\sigma }^4}}.\hfill \end{array}$$

Using the inequality recursively from $t=1$ to T, then

$$\begin{array}{cc}\hfill \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_{T+1}\right)& \le {\displaystyle \sum _{t=1}^T}(1-{\kappa }^{-2}{\eta }_t)\left(\mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_1\right)\right)+{\displaystyle \frac{4{\kappa }^2}{{\sigma }^4}}{\displaystyle \sum _{t=1}^T}{\eta }_t^2{\displaystyle \prod _{j=t+1}^T}(1-{\kappa }^{-2}{\eta }_j)\hfill \\ \hfill & \le exp\left\{-{\kappa }^{-2}{\displaystyle \sum _{t=1}^T}{\eta }_t\right\}\left(\mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_1\right)\right)+{\displaystyle \frac{4{\kappa }^2}{{\sigma }^4}}{\displaystyle \sum _{t=1}^T}{\eta }_t^2{\displaystyle \prod _{j=t+1}^T}(1-{\kappa }^{-2}{\eta }_j)\hfill \end{array}$$

The part of the first term ${lim}_{T\to \infty }exp\left\{-{\kappa }^{-2}{\sum }_{t=1}^T{\eta }_t\right\}=0$ due to ${\sum }_{t=1}^{\infty }{\eta }_t=\infty$. The part of the second term ${\sum }_{t=1}^T{\eta }_t^2{\prod }_{j=t+1}^T$ goes to zero, as $T\to \infty$ due to Lemma 11 in [37].

The the proof of (a) is complete.

For the proof of statement (b), we recall an elementary inequality in [24]. For any positive functional $\ell \left(f\right),f\in {\mathcal{H}}_K$, if the gradient of ℓ is L-Lipschitz continuous with respect to ${\mathcal{H}}_K$, then

$$\begin{array}{c}\hfill {\parallel \Delta \ell \left(u\right)\parallel }_K^2\le 2L\ell \left(u\right).\end{array}$$

(13)

Let $\ell \left(f\right)=\mathcal{E}\left(f^{\ast }\right)-\mathcal{E}\left(f\right)$, then by (6), we obtain

$$\begin{array}{c}\hfill {\ell }^{\prime }\left(f\right)={\int }_{\mathcal{Z}}-{\varphi }_{\sigma }^{\prime }(f\left(x\right)-y)K_{x}d\rho \end{array}$$

and for any $f,g\in {\mathcal{H}}_K$, it follows that

$$|{\ell }^{\prime }\left(f\right)-{\ell }^{\prime }\left(g\right)|\le {\int }_{\mathbf{z}}{\displaystyle \frac{4}{{\sigma }^2}}|f\left(x\right)-g\left(x\right)|K_{x}d\rho \le {\displaystyle \frac{4{\kappa }^2}{{\sigma }^2}}{\parallel f-g\parallel }_K$$

by using (2). The gradient of the functional $\ell \left(f\right)$ is ${\displaystyle \frac{4{\kappa }^2}{{\sigma }^2}}$-Lipschitz continuous in ${\mathcal{H}}_K.$

Note that $\nabla \mathcal{E}\left(f\right)=\nabla \left(\mathcal{E}\left(f\right)-\mathcal{E}\left(f^{\ast }\right)\right)={\int }_{\mathbf{z}}-{\varphi }_{\sigma }^{\prime }(f\left(x\right)-y)K_{x}d\rho$ as $\nabla \mathcal{E}\left(f^{\ast }\right)=0$. Then, we obtain

$$\begin{array}{c}\hfill \parallel \nabla \left(\mathcal{E}\left(f\right)-\mathcal{E}\left(f^{\ast }\right)\right){\parallel }_K^2={\parallel \nabla \mathcal{E}\left(f\right)\parallel }_K^2\le {\displaystyle \frac{8{\kappa }^2}{{\sigma }^2}}\left(\mathcal{E}\left(f^{\ast }\right)-\mathcal{E}\left(f\right)\right)\end{array}$$

(14)

using (13) with $L={\displaystyle \frac{4{\kappa }^2}{{\sigma }^2}}$. Replacing f with $f_t$, then from statement (a), we can derive

$$\begin{array}{c}\hfill \underset{t\to \infty }{lim}{\parallel \nabla \mathcal{E}\left(f_t\right)\parallel }_K^2\le {\displaystyle \frac{8{\kappa }^2}{{\sigma }^2}}\underset{t\to \infty }{lim}\left(\mathcal{E}\left(f^{\ast }\right)-\mathcal{E}\left(f_t\right)\right)=0.\end{array}$$

Then, the proof of statement (b) is complete.

For the proof of statement (c), we take the expectation with respect to $z_1,\cdots ,z_t$ and add $\mathcal{E}\left(f^{\ast }\right)$ on both sides of (9). So, we have

$$\begin{array}{cc}\hfill \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_{t+1}\right)& \le \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_t\right)-{\eta }_t\mathbf{E}\parallel \nabla \mathcal{E}\left(f_t\right){\parallel }_K^2+{\displaystyle \frac{2{\eta }_t^2}{{\sigma }^2}}\mathbf{E}{\parallel \nabla \mathcal{E}\left(f_t\right)\parallel }_K^2.\hfill \end{array}$$

(15)

When there exists $t_0$, such that for any $t\ge t_0$, ${\eta }_t\le {\displaystyle \frac{{\sigma }^2}{4}}$, we have ${\displaystyle \frac{2{\eta }_t^2}{{\sigma }^2}}\le {\displaystyle \frac{1}{2}}{\eta }_t$ and

$$\begin{array}{c}\hfill \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_{t+1}\right)\le \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_t\right)-{\displaystyle \frac{{\eta }_t}{2}}\mathbf{E}{\parallel \nabla \mathcal{E}\left(f_t\right)\parallel }_K^2,t>t_0.\end{array}$$

(16)

Using (2) and (5) again, we obtain

$$\mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_{t+1}\right)\le \mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_t\right)-{\displaystyle \frac{{\eta }_t{\kappa }^{-2}}{2}}\mathbf{E}{\parallel \nabla \mathcal{E}\left(f_t\right)\parallel }_K^2\le \left(1-{\displaystyle \frac{{\eta }_t{\kappa }^{-2}\mu }{2}}\right)\left(\mathcal{E}\left(f^{\ast }\right)-\mathbf{E}\mathcal{E}\left(f_t\right)\right).$$

Applying the estimate iteratively from $t=1$ to T, we obtain the conclusion for statement (c).

The proof is finished. □

6. Conclusions and Future Works

This paper studies the stochastic gradient method with correntropy loss functions. More precisely, we study how convergence properties can be achieved through a suitable choice of step size in robustness learning. We also use the PL condition to provide a global linear convergence rate. These results refine the previous work and provide a theoretical foundation for SGD with MCC.

Two related questions are worthwhile for future research. First, our analysis provides very useful insights on the application of SGD for MCC. The simulation is also consistent with our theory. However, the optimal selection of $\sigma$ is unknown in practice. It is necessary to develop empirically applicable parameter selection strategies for optimal $\sigma .$ Secondly, with the development of deep learning, various stochastic gradient methods are continuously emerging that accelerate the training progress, such as Adam [38], AdaGrad [39], RMSProp [40], and AMSGrad [41]. Non-convexity [42,43,44] of the MCC loss function will bring major challenges in the theoretical analysis. The techniques used in this paper may also be applied to other stochastic gradient methods. Meanwhile, our error bounds already imply linear convergence rates. It is yet unknown whether the non-convexity can be overcome in the error analysis and similar convergence results can be derived in other stochastic gradient methods. This will be considered in future research.