Gradient Learning under Tilted Empirical Risk Minimization

Abstract

Gradient Learning (GL), aiming to estimate the gradient of target function, has attracted much attention in variable selection problems due to its mild structure requirements and wide applicability. Despite rapid progress, the majority of the existing GL works are based on the empirical risk minimization (ERM) principle, which may face the degraded performance under complex data environment, e.g., non-Gaussian noise. To alleviate this sensitiveness, we propose a new GL model with the help of the tilted ERM criterion, and establish its theoretical support from the function approximation viewpoint. Specifically, the operator approximation technique plays the crucial role in our analysis. To solve the proposed learning objective, a gradient descent method is proposed, and the convergence analysis is provided. Finally, simulated experimental results validate the effectiveness of our approach when the input variables are correlated.

1. Introduction

Data-driven variable selection aims to select informative features related with the response in high-dimensional statistics and plays a critical role in many areas. For example, if the milk production of dairy cows can be predicted by the blood biochemical indexes, then the doctors are eager to know which indexes can drive the milk production because each of them is independently measured with additional burden. Therefore, an explainable and interpretable system to select the effective variables is critical to convince domain experts. Currently, the methodologies on variable selection methods can be roughly divided into three categories including linear models [1,2,3], nonlinear additive models [4,5,6], and partial linear models [7,8,9]. Although achieving promising performance in some applications, these methods mentioned above still suffer from two main limitations. Firstly, the target function of these methods is restricted on the assumption of specific structures. Secondly, these methods cannot revive how the coordinates vary with respect to each other. As an alternative, Mukherjee and Zhou [10] proposed the gradient learning (GL) model, which aims to learn the gradient functions and enjoys the model-free property.

Despite the empirical success [11,12,13], there are still some limitations of the GL model, such as high computational cost, lacking the sparsity in high-dimensional data and lacking the robustness to complex noises. To this end, several variants of the GL model have been devoted to developing alternatives for individual purposes. For example, Dong and Zhou [14] proposed a stochastic gradient descent algorithm for learning the gradient and demonstrated that the gradient estimated by the algorithm converges to the true gradient. Mukherjee et al. [15] provided an algorithm to reduce dimension on manifolds for high-dimensional data with few observations. They obtained generalization error bounds of the gradient estimates and revealed that the convergence rate depends on the intrinsic dimension of the manifold. Borkar et al. [16] combined ideas from Spall’s Simultaneous Perturbation Stochastic Approximation with compressive sensing and proposed to learn the gradient with few function evaluations. Ye et al. [17] originally proposed a sparse GL model to further address the sparsity for high-dimensional variable selection of the estimated sparse gradients. He et al. [18] developed a three-step sparse GL method which allows for efficient computation, admits general predictor effects, and attains desirable asymptotic sparsistency. Following the research direction of robustness, Guinney et al. [19] provided a multi-task model which are efficient and robust for high-dimensional data. In addition, Feng et al. [20] provided a robust gradient learning (RGL) framework by introducing a robust regression loss function. Meanwhile, a simple computational algorithm based on gradient descent was provided, and the convergence of the proposed method is also analyzed.

Despite rapid progress, the GL model and its extensions mentioned above are established under the framework of empirical risk minimization (ERM). While enjoying the nice statistical properties, ERM usually performs poorly in situations where average performance is not an appropriate surrogate for the problem of interest [21]. Recently, a novel framework, named tilted empirical risk minimization (TERM), is proposed to flexibly address the deficiencies in ERM [21]. By using a new loss named t-tilted loss, it has been shown that TERM (1) can increase or decrease the influence of outliers, respectively, to enable fairness or robustness; (2) has variance reduction properties that can benefit generalization; and (3) can be viewed as a smooth approximation to a superquantile method. Considering these strength, we propose to investigate the GL under the framework of TERM. The main contributions of this paper can be summarized as follows:

• New learning objective. We propose to learn the gradient function under the framework of TERM. Specifically, the t-tilted loss is embedded into the GL model. To the best of our knowledge, it may be the first endeavor in this topic.
• Theoretical guarantees. For the new learning objective, we estimate the generalization bound by error decomposition and operator approximation technique, and further provide the theoretical consistency and the convergence rate. To be specific, the convergence rate can recover the result of traditional GL as t tends 0 [10].
• Efficient computation. A gradient descent method is provided to solve the proposed learning objective. By showing the smoothness and strongly convex of the learning objective, the convergence to the optimal solution is proved.

The rest of this paper is organized as follows: Section 2 proposes the GL with t-tilted loss (TGL) and states the main theoretical results on the asymptotic estimation. Section 3 provides the computational algorithm and its convergence analysis. Numerical experiments on synthetic data sets will be implemented in Section 4. Finally, Section 5 closes this paper with some conclusions.

2. Learning Objective

In this section, we introduce TGL and provide the main theoretical results on the asymptotic estimation.

2.1. Gradient Learning with t-Tilted Loss

Let X be a compact subset of ${\mathbb{R}}^n$ and $Y\in \mathbb{R}$. Assume that $\rho$ is a probability measure on $Z:=X\times Y$. It induces the marginal distribution ${\rho }_X$ on X and conditional distributions $\rho (·|x)$ at $x\in X$. Denote $L_{{\rho }_X}^2$ as the $L^2$ space with the metric ${\parallel f\parallel }_{\rho }=({\int }_X{\left|f\left(x\right)\right|}^{2}d{\rho }_X{)}^{1/2}$. In addition, the regression function $f_{\rho }:X\to Y$ associated with $\rho$ is defined as

$$f_{\rho }\left(x\right)={\int }_{Y}yd\rho \left(y\right|x),x\in X.$$

For $x={(x^1,x^2,\cdots ,x^n)}^{\mathsf{T}}\in X$, the gradient of $f_{\rho }$ is the vector of functions (if the partial derivatives exist)

$$\nabla f_{\rho }={\left(\frac{\partial f_{\rho }}{\partial x^1},\frac{\partial f_{\rho }}{\partial x^2},\dots ,\frac{\partial f_{\rho }}{\partial x^n}\right)}^{\mathsf{T}}.$$

The relevance between the l-th coordinate and $f_{\rho }$ can be evaluated via the norm of its partial derivative $\parallel \frac{\partial f_{\rho }}{\partial x^l}\parallel$, where a large value implies a large change in the function $f_{\rho }$ with respect to a sensitive change in the l-th coordinate. This fact gives an intuitive motivation for the GL. In terms of Taylor series expansion, the following equation holds:

$$f_{\rho }\left(x\right)\approx f_{\rho }\left(\tilde{x}\right)+\nabla f_{\rho }\left(\tilde{x}\right)·(x-\tilde{x}),$$

(1)

for $x\approx \tilde{x}$ and $x,\tilde{x}\in X$. Inspired by (1), we denote the weighted square loss of $\overrightarrow{f}$ as

$$V(\overrightarrow{f},z,\tilde{z}=\omega (x,\tilde{x}){(\tilde{y}-y+\overrightarrow{f}{\left(\tilde{x}\right)}^{\mathsf{T}}(x-\tilde{x}))}^2,\overrightarrow{f}\in {\left(L_{{\rho }_X}^2\right)}^n,z,\tilde{z}\in Z,$$

(2)

where the restriction $x\approx \tilde{x}$ will be enforced by weights $\omega (x,\tilde{x})$ given by $\frac{1}{s^{n+2}}{e}^{-|x-\tilde{x}{|}^2/2s^2}$ with a constant $0<s\le 1$, see, e.g., [10,11,19]. Then, the expected risk of $\overrightarrow{f}$ can be given by

$$\mathcal{E}(\overrightarrow{f})={\int }_Z{\int }_{Z}V(\overrightarrow{f},z,\tilde{z})d\rho \left(z\right)d\rho \left(\tilde{z}\right).$$

(3)

As mentioned in [21], the $\overrightarrow{f}$ defined in (3) usually performs poorly in situations where average performance is not an appropriate surrogate. Inspired from [21], for $t\in {\mathbb{R}}^{\setminus 0}$, we address the deficiencies by introducing the t-tilted loss and define the expected risk of $\overrightarrow{f}$ with t-tilted loss as

$$\mathcal{E}(\overrightarrow{f},t)=\frac{1}{t}log{\int }_Z{\int }_Z{e}^{tV(\overrightarrow{f},z,\tilde{z})}d\rho \left(z\right)d\rho \left(\tilde{z}\right).$$

(4)

Remark 1.

Note that $t\in {\mathbb{R}}^{\setminus 0}$ is a real-valued hyperparameter, and it can encompass a family of objectives which can address the fairness ($t>0$) or robustness ($t<0$) by different choices. In particular, it recovers the expected risk (3) as $t\to 0$.

On this basis, the GL with t-tilted loss is formulated as the following regularization scheme:

$${\overrightarrow{f}}_{\lambda ,t}=\underset{\overrightarrow{f}\in {\mathcal{H}}_K^n}{argmin}\{\mathcal{E}(\overrightarrow{f},t)+\lambda \parallel \overrightarrow{f}{\parallel }_K^2\},$$

(5)

where $\lambda >0$ is a regularization parameter. Here, $K:X\times X\to \mathbb{R}$ is a Mercer kernel that is continuous, symmetric, and positive semidefinite [22,23] and ${\mathcal{H}}_K$ induced by K be an RKHS defined as the closure of the linear span of the set of functions $\{K_x:=K(x,·):x\in X\}$ with the inner product ${\langle ·,·\rangle }_K$ satisfying ${\langle K_x,K_{\tilde{x}}\rangle }_K=K(x,\tilde{x})$. The reproducing property takes the form ${\langle K_x,f\rangle }_K=f\left(x\right),\forall x\in X,\forall f\in {\mathcal{H}}_K.$ Then, we denote ${\mathcal{H}}_K^n$ as an n-fold RKHS with the inner product

$$\begin{array}{c}\hfill {\langle \overrightarrow{f},\overrightarrow{h}\rangle }_K=\sum _{l=1}^n{\langle f^l,h^l\rangle }_K,\overrightarrow{f}={(f^1,f^2,\dots ,f^n)}^{\mathsf{T}},\overrightarrow{h}={(h^1,h^2,\dots ,h^n)}^{\mathsf{T}}\in {\mathcal{H}}_K^n,\end{array}$$

and norm $\parallel \overrightarrow{f}{\parallel }_K^2={\langle \overrightarrow{f},\overrightarrow{f}\rangle }_K$.

2.2. Main Results

This subsection states our main theoretical results on the asymptotic estimation of $\parallel {\overrightarrow{f}}_{\lambda ,t}-\nabla f_{\rho }{\parallel }_{\rho }$ on the space ${\left(L_{{\rho }_X}^2\right)}^n$ with norm $\parallel \overrightarrow{f}{\parallel }_{\rho }=({\sum }_{l=1}^n\parallel f^l{{\parallel }_{\rho }^2)}^{1/2}$. Before proceeding, we provide some necessary assumptions which have been used extensively in machine learning literature, e.g., [24,25].

Assumption 1.

Supposing that $\nabla f_{\rho }\in {\mathcal{H}}_K^n$ and the kernel K is $C^3$, there exists a constant $c_{\upsilon }>0$ such that

$$|f_{\rho }\left(x\right)-f_{\rho }\left(\tilde{x}\right)-\nabla f_{\rho }{\left(\tilde{x}\right)}^{\mathsf{T}}(x-\tilde{x})|\le c_{\upsilon }{\left|x-\tilde{x}\right|}^2,\forall x,\tilde{x}\in X.$$

(6)

Assumption 2.

Assume $\left|y\right|\le M$, $\left|x\right|\le M_X$ almost surely. Suppose that, for some $\varsigma \in (0,\frac{2}{3})$, $c_l,c_h>0$, the marginal distribution ${\rho }_X$ satisfies

$${\rho }_X(\{x\in X:\underset{\tilde{x}\in {\mathbb{R}}^n\setminus X}{inf}\left|x-\tilde{x}\right|\le s\})\le c_h^2{s}^{4\varsigma },\forall s>0,$$

(7)

and the density $p\left(z\right)$ of $d\rho \left(z\right)$ exists and satisfies

$$c_l\le p\left(z\right)\le c_h,\left|p\left(z\right)-p\left(\tilde{z}\right)\right|\le c_h{\left|z-\tilde{z}\right|}^{\varsigma },\forall z,\tilde{z}\in Z.$$

(8)

Taking the functional derivatives of (5), we know that ${\overrightarrow{f}}_{\lambda ,t}$ can be expressed in terms of the following integral operator on the space ${\left(L_{{\rho }_X}^2\right)}^n$.

Definition 1.

Let integral operator $L_{K,s}:{\left(L_{{\rho }_X}^2\right)}^n\to {\left(L_{{\rho }_X}^2\right)}^n$ be defined by

$$\begin{array}{c}\hfill L_{K,s}\overrightarrow{f}={\int }_Z{\int }_Z\varphi (z,\tilde{z})\omega (x,\tilde{x})(\overrightarrow{f}{\left(\tilde{x}\right)}^{\mathsf{T}}(x-\tilde{x}))K_{\tilde{x}}(x-\tilde{x})d\rho \left(\tilde{z}\right)d\rho \left(z\right),\end{array}$$

(9)

where

$$\varphi (z,\tilde{z})={({\int }_Z{\int }_Z{e}^{tV({\overrightarrow{f}}_{\lambda ,t},u,v)}d\rho \left(u\right)d\rho \left(v\right))}^{-1}{e}^{tV({\overrightarrow{f}}_{\lambda ,t},z,\tilde{z})}.$$

The operator $L_{K,s}$ has its range in ${\mathcal{H}}_K^n$. It can also be regarded as a positive operator on ${\mathcal{H}}_K^n$. We shall use the same notion for the operators on these two different domains. Given the definition of integral operator $L_{K,s}$, we can write ${\overrightarrow{f}}_{\lambda ,t}$ in the following equation.

Theorem 1.

Given the integral operator $L_{K,s}$, we have the following relationship:

$$\begin{array}{c}\hfill {\overrightarrow{f}}_{\lambda ,t}={(L_{K,s}+\lambda I)}^{-1}{\overrightarrow{f}}_{\rho ,s},\end{array}$$

(10)

where ${\overrightarrow{f}}_{\rho ,s}={\int }_Z{\int }_Z\varphi (z,\tilde{z})\omega (x,\tilde{x})\left(f_{\rho }\left(x\right)-f_{\rho }\left(\tilde{x}\right)\right)K_{\tilde{x}}(x-\tilde{x})d\rho \left(\tilde{z}\right)d\rho \left(z\right)$, and I is the identity operator.

Proof of Theorem 1.

To solve the scheme (5), we take the functional derivative with respect to $\overrightarrow{f}$, apply it to an element $\delta \overrightarrow{f}$ of ${\mathcal{H}}_K^n$ and set it equal to 0. We obtain

$$\begin{array}{c}\hfill {\int }_Z{\int }_Z\varphi (z,\tilde{z})\omega (x,\tilde{x})(\tilde{y}-y+{\overrightarrow{f}}_{\lambda ,t}{\left(\tilde{x}\right)}^{\mathsf{T}}(x-\tilde{x}))\delta \overrightarrow{f}{\left(\tilde{x}\right)}^{\mathsf{T}}(x-\tilde{x})d\rho \left(\tilde{z}\right)d\rho \left(z\right)+\lambda {\langle {\overrightarrow{f}}_{\lambda ,t},\delta \overrightarrow{f}\rangle }_K=0.\end{array}$$

Since it holds for any $\delta \overrightarrow{f}\in {\mathcal{H}}_K^n$, it is trivial to obtain

$$\begin{array}{c}\hfill {\int }_Z{\int }_Z\varphi (z,\tilde{z})\omega (x,\tilde{x})(\tilde{y}-y+{\overrightarrow{f}}_{\lambda ,t}{\left(\tilde{x}\right)}^{\mathsf{T}}(x-\tilde{x}))K_{\tilde{x}}(x-\tilde{x})d\rho \left(\tilde{z}\right)d\rho \left(z\right)+\lambda {\overrightarrow{f}}_{\lambda ,t}=0\end{array}$$

and

$$\begin{array}{c}\hfill \lambda {\overrightarrow{f}}_{\lambda ,t}+L_{K,s}{\overrightarrow{f}}_{\lambda ,t}={\overrightarrow{f}}_{\rho ,s}.\end{array}$$

The desired result follows by shifting items.    □

On this basis, we propose to bound the error $\parallel {\overrightarrow{f}}_{\lambda ,t}-\nabla f_{\rho }{\parallel }_{\rho }$ by a functional analysis approach and present the error decomposition as following proposition. The proof is straightforward and omitted for brevity.

Proposition 1.

For the ${\overrightarrow{f}}_{\lambda ,t}$ defined in (5), it holds that

$$\parallel {\overrightarrow{f}}_{\lambda ,t}-\nabla f_{\rho }{\parallel }_{\rho }\le \parallel {\overrightarrow{f}}_{\lambda ,t}-\nabla f_{\rho }+\lambda {(L_{K,s}+\lambda I)}^{-1}\nabla f_{\rho }{\parallel }_{\rho }+{\parallel \lambda {(L_{K,s}+\lambda I)}^{-1}\nabla f_{\rho }\parallel }_{\rho }.$$

(11)

In the sequel, we focus on bounding $\parallel {\overrightarrow{f}}_{\lambda ,t}-\nabla f_{\rho }+\lambda {(L_{K,s}+\lambda I)}^{-1}\nabla f_{\rho }{\parallel }_{\rho }$ and $\parallel \lambda {(L_{K,s}+\lambda I)}^{-1}\nabla f_{\rho }{\parallel }_{\rho }$, respectively. Before we embark on the proof, we single out a important property regarding $\varphi (z,\tilde{z})$ that will be useful in later proofs.

Lemma 1.

Under the Assumptions 1 and 2, there exists $B_t$ and $A_t$ dependent on t satisfying

$$B_t=e^{-8\left|t\right|(M^2+C_K{M}_X)}\le \varphi (z,\tilde{z})\le A_t=e^{8\left|t\right|(M^2+C_K{M}_X)}.$$

(12)

Proof of Lemma 1.

Since the kernel K is $C^3$ and ${\overrightarrow{f}}_{\lambda ,t}\in {\mathcal{H}}_K^n$, we know from Zhou [26] that $f_{\lambda ,t}^l$ is $C^1$ for each l. There exists a constant $C_K$ satisfying $|{\overrightarrow{f}}_{\lambda ,t}{\left(x\right)|}^2\le C_K,\forall x\in X$. Hence, using Cauchy inequality, we have

$$\begin{array}{cc}\hfill V({\overrightarrow{f}}_{\lambda ,t},z,\tilde{z})& =\omega (\tilde{x},x){(\tilde{y}-y+{\overrightarrow{f}}_{\lambda ,t}{\left(\tilde{x}\right)}^{\mathsf{T}}(x-\tilde{x}))}^2\hfill \\ & \le 2(4M^2+|{\overrightarrow{f}}_{\lambda ,t}\left(\tilde{x}\right){|}^2|x-\tilde{x}{|}^2)\hfill \\ & \le 8(M^2+C_K{M}_X).\hfill \end{array}$$

By a direct computation, we obtain

$$\begin{array}{c}\hfill e^{-8\left|t\right|(M^2+C_K{M}_X)}\le {\left({\int }_Z{\int }_Z{e}^{tV({\overrightarrow{f}}_{\lambda ,t},u,v)}d\rho \left(u\right)d\rho \left(v\right)\right)}^{-1}{e}^{tV({\overrightarrow{f}}_{\lambda ,t},z,\tilde{z})}\le e^{8\left|t\right|(M^2+C_K{M}_X)}.\end{array}$$

The desired result follows.    □

Denote $\kappa ={sup}_{x\in X}K(x,x)$ and the moments of the Gaussian as $J_p={\int }_{{\mathbb{R}}^n}{e}^{-\frac{{\left|x\right|}^2}{2}}{\left|x\right|}^{p}dx$, $p=1,2,3,\dots$, we establish the following Lemma.

Lemma 2.

Under Assumptions 1 and 2, we have

$$\begin{array}{c}\hfill \parallel {\overrightarrow{f}}_{\lambda ,t}-\nabla f_{\rho }+\lambda {(L_{K,s}+\lambda I)}^{-1}\nabla f_{\rho }{\parallel }_K\le 2M\frac{s}{\lambda }\kappa c_{\upsilon }{c}_h{J}_3{A}_t.\end{array}$$

(13)

Proof of Lemma 2.

Taking notice of (10), it follows that

$$\begin{array}{cc}\hfill & {\overrightarrow{f}}_{\lambda ,t}-\nabla f_{\rho }+\lambda {(L_{K,s}+\lambda I)}^{-1}\nabla f_{\rho }={(L_{K,s}+\lambda I)}^{-1}({\overrightarrow{f}}_{\rho ,s}-L_{K,s}\nabla f_{\rho }).\hfill \end{array}$$

Then, we have

$$\begin{array}{cc}\hfill \parallel {\overrightarrow{f}}_{\lambda ,t}-\nabla f_{\rho }+\lambda {(L_{K,s}+\lambda I)}^{-1}\nabla f_{\rho }{\parallel }_K\le & \parallel {(L_{K,s}+\lambda I)}^{-1}{\parallel }_K{\parallel {\overrightarrow{f}}_{\rho ,s}-L_{K,s}\nabla f_{\rho }\parallel }_K\hfill \\ \hfill \le & \frac{1}{\lambda }{\parallel {\overrightarrow{f}}_{\rho ,s}-L_{K,s}\nabla f_{\rho }\parallel }_K.\hfill \end{array}$$

We note that

$$\begin{array}{cc}\hfill & J_p{s}^{p-2}={\int }_{{\mathbb{R}}^n}\omega (x,\tilde{x})|x-\tilde{x}{|}^{p}d\tilde{x}={\int }_{{\mathbb{R}}^n}\frac{1}{s^{n+2}}{e}^{\frac{-{\left|x-\tilde{x}\right|}^2}{2s^2}}{|x-\tilde{x}|}^{p}d\tilde{x},p=2,3,\cdots .\hfill \end{array}$$

From Assumptions 1 and 2, we have

$$\parallel {\overrightarrow{f}}_{\rho ,s}-L_{K,s}\nabla f_{\rho }{\parallel }_K\le {\int }_Z{\int }_Z\omega (x,\tilde{x}){\left|x-\tilde{x}\right|}^3\varphi (z,\tilde{z}){\parallel K_{\tilde{x}}\parallel }_K{c}_{\upsilon }d\rho \left(z\right)d\rho \left(\tilde{z}\right)\le 2Ms\kappa c_{\upsilon }{c}_h{J}_3{A}_t.$$

The desired result follows.    □

As for $\parallel \lambda {(L_{K,s}+\lambda I)}^{-1}\nabla f_{\rho }{\parallel }_{\rho }$, the multivariate mean value theorem ensures that there exists $R_t\left(\tilde{z}\right)=\varphi \left(\tilde{z},{\eta }_z\right),{\eta }_z\in {\mathbb{R}}^n\times Y$, such that

$$\begin{array}{cc}\hfill & {\int }_Z{\int }_{{\mathbb{R}}^n\times Y}{e}^{-\frac{{\left|x-\tilde{x}\right|}^2}{2s^2}}\frac{|x-\tilde{x}{|}^2}{s^{2+n}}\varphi (z,\tilde{z})K_{\tilde{x}}\overrightarrow{f}\left(\tilde{x}\right)p\left(\tilde{z}\right)dzd\rho \left(\tilde{z}\right)\hfill \\ \hfill =& {\int }_Z{\int }_{{\mathbb{R}}^n\times Y}{e}^{-\frac{{\left|x-\tilde{x}\right|}^2}{2s^2}}\frac{|x-\tilde{x}{|}^2}{s^{2+n}}{R}_t\left(\tilde{z}\right)K_{\tilde{x}}\overrightarrow{f}\left(\tilde{x}\right)p\left(\tilde{z}\right)dzd\rho \left(\tilde{z}\right).\hfill \end{array}$$

(14)

From (14), we can define the integral operator associated with the Mercer kernel K which is related to $L_{K,s}$. Using Lemma 16 and Lemma 18 in [10], we establish the following Lemma.

Lemma 3.

Under the Assumption 2, denote $c_{\rho }={\left(2M{A}_t{\kappa }^2{c}_h(2J_{2+\varsigma }+J_4+c_h{J}_2)\right)}^{\frac{1}{\varsigma }}$ and $V_p={\int }_Z{\left(p\left(z\right)\right)}^2{R}_t\left(z\right)dz$. For any $0<s\le \mathrm{m}\mathrm{i}\mathrm{n}\{c_{\rho }{\lambda }^{\frac{1}{\varsigma }},1\}$, we have

$$\begin{array}{c}\hfill \parallel \lambda {(L_{K,s}+\lambda I)}^{-1}\nabla f_{\rho }{\parallel }_{\rho }\le 2\sqrt{\lambda }{\left(V_{p}n{\left(2\pi \right)}^{\frac{n}{2}}M\right)}^{-\frac{1}{2}}{\parallel L_K^{-\frac{1}{2}}\nabla f_{\rho }\parallel }_{\rho },\end{array}$$

(15)

where $L_K$ is a positive operator on ${\left(L_{{\rho }_X}^2\right)}^n$ defined by

$$L_K\overrightarrow{f}={\int }_Z{K}_x\overrightarrow{f}\left(x\right)\frac{p\left(z\right)R_t\left(z\right)}{V_p}d\rho \left(z\right),\overrightarrow{f}\in {\left(L_{\rho }^2\right)}^n.$$

Proof of Lemma 3.

To estimate (15), we need to consider the convergence of $L_{K,s}$ as $s\to 0$. Denote the stepping stone

$$\begin{array}{c}\hfill \overrightarrow{g}={\int }_Z{\int }_Z\omega (x,\tilde{x})(x-\tilde{x})R_t\left(\tilde{z}\right)K_{\tilde{x}}{(x-\tilde{x})}^{\mathsf{T}}\overrightarrow{f}\left(\tilde{x}\right)p\left(\tilde{z}\right)dzd\rho \left(\tilde{z}\right),\end{array}$$

we deduce that

$$\begin{array}{cc}\hfill \parallel L_{K,s}\overrightarrow{f}-2M{V}_{p}n{\left(2\pi \right)}^{\frac{n}{2}}{L}_K\overrightarrow{f}{\parallel }_K& \le \parallel L_{K,s}\overrightarrow{f}-\overrightarrow{g}+\overrightarrow{g}-2M{V}_{p}n{\left(2\pi \right)}^{\frac{n}{2}}{L}_K\overrightarrow{f}{\parallel }_K\hfill \\ & \le \parallel L_{K,s}\overrightarrow{f}-\overrightarrow{g}{\parallel }_K+{\parallel \overrightarrow{g}-2M{V}_{p}n{\left(2\pi \right)}^{\frac{n}{2}}{L}_K\overrightarrow{f}\parallel }_K.\hfill \end{array}$$

Using the multivariate mean value theorem, there exists $z_{\zeta },z_{\sigma }\in {\mathbb{R}}^n\times Y$, such that

$$\begin{array}{cc}\hfill \parallel L_{K,s}\overrightarrow{f}-\overrightarrow{g}{\parallel }_K=& \parallel p\left(z_{\zeta }\right){\int }_Z{\int }_{{\mathbb{R}}^n\times Y}{R}_t\left(\tilde{z}\right)\omega (x,\tilde{x})(\overrightarrow{f}{\left(\tilde{x}\right)}^{\mathsf{T}}(x-\tilde{x}))K_{\tilde{x}}(x-\tilde{x})dzd\rho \left(\tilde{z}\right)\hfill \\ & -{\int }_Z{\int }_Z\omega (x,\tilde{x})(x-\tilde{x})R_t\left(\tilde{z}\right)K_{\tilde{x}}{(x-\tilde{x})}^{\mathsf{T}}\overrightarrow{f}\left(\tilde{x}\right)p\left(\tilde{z}\right)dzd\rho \left(\tilde{z}\right){\parallel }_K\hfill \\ \hfill \le & \parallel p\left(z_{\zeta }\right){\int }_Z{\int }_{{\mathbb{R}}^n\times Y}{R}_t\left(\tilde{z}\right)\omega (x,\tilde{x})(\overrightarrow{f}{\left(\tilde{x}\right)}^{\mathsf{T}}(x-\tilde{x}))K_{\tilde{x}}(x-\tilde{x})dzd\rho \left(\tilde{z}\right)\hfill \\ & -{\int }_Z{\int }_{{\mathbb{R}}^n\times Y}{R}_t\left(\tilde{z}\right)\omega (x,\tilde{x})(\overrightarrow{f}{\left(\tilde{x}\right)}^{\mathsf{T}}(x-\tilde{x}))K_{\tilde{x}}(x-\tilde{x})p\left(z\right)dzd\rho \left(\tilde{z}\right){\parallel }_K\hfill \\ & +\parallel {\int }_Z{\int }_Z{R}_t\left(\tilde{z}\right)\omega (x,\tilde{x})(\overrightarrow{f}{\left(\tilde{x}\right)}^{\mathsf{T}}(x-\tilde{x}))K_{\tilde{x}}(x-\tilde{x})(p\left(z\right)-p\left(\tilde{z}\right))dzd\rho \left(\tilde{z}\right){\parallel }_K\hfill \\ \hfill \le & \parallel p\left(z_{\zeta }\right)-p\left(z_{\sigma }\right){\int }_Z{\int }_{{\mathbb{R}}^n\times Y}{R}_t\left(\tilde{z}\right)\omega (x,\tilde{x})()\overrightarrow{f}{\left(\tilde{x}\right)}^{\mathsf{T}}(x-\tilde{x})K_{\tilde{x}}(x-\tilde{x})dzd\rho \left(\tilde{z}\right){\parallel }_K\hfill \\ & +\parallel {\int }_Z{\int }_Z{R}_t\left(\tilde{z}\right)\omega (x,\tilde{x})(\overrightarrow{f}{\left(\tilde{x}\right)}^{\mathsf{T}}(x-\tilde{x}))K_{\tilde{x}}(x-\tilde{x})(p\left(z\right)-p\left(\tilde{z}\right))dzd\rho \left(\tilde{z}\right){\parallel }_K\hfill \\ \hfill \le & 4M{s}^{\varsigma }\kappa c_h{J}_{2+\varsigma }{\parallel \overrightarrow{f}\parallel }_{\rho }{A}_t.\hfill \end{array}$$

Noticing $n{\left(2\pi \right)}^{\frac{n}{2}}=J_2$, we have

$$\begin{array}{cc}\hfill 2V_{p}n{\left(2\pi \right)}^{\frac{n}{2}}M{L}_K\overrightarrow{f}=& {\int }_Z{\int }_{{\mathbb{R}}^n\times Y}\omega (x,\tilde{x})R_t\left(\tilde{z}\right)K_{\tilde{x}}\overrightarrow{f}\left(\tilde{x}\right){(x-\tilde{x})}^{\mathsf{T}}(x-\tilde{x})p\left(\tilde{z}\right)dzd\rho \left(\tilde{z}\right).\hfill \end{array}$$

Then, by (7), we can obtain the following conclusion from Lemma 16 in [10] when $0\le s\le 1$,

$$\begin{array}{cc}\parallel \overrightarrow{g}-2M{V}_{p}n{\left(2\pi \right)}^{\frac{n}{2}}{L}_K\overrightarrow{f}{\parallel }_K\hfill & \le \parallel {\int }_Z{\int }_{({\mathbb{R}}^n\times Y)\setminus Z}\omega (x,\tilde{x})R_t\left(\tilde{z}\right)K_{\tilde{x}}\overrightarrow{f}\left(\tilde{x}\right)p\left(\tilde{z}\right){|x-\tilde{x}|}^{2}dzd\rho \left(\tilde{z}\right){\parallel }_K\hfill \\ & \le 2M{c}_{\rho }{A}_t{\int }_X{\int }_{{\mathbb{R}}^n\setminus X}\omega (x,\tilde{x})K_{\tilde{x}}|\overrightarrow{f}\left(\tilde{x}\right)\left|\right|(x-\tilde{x}){|}^{2}dxd{\rho }_X\left(\tilde{x}\right)\hfill \\ & \le 2M{s}^{\varsigma }\kappa c_h(J_4+c_h{J}_2){\parallel \overrightarrow{f}\parallel }_{\rho }{A}_t.\hfill \end{array}$$

Combining the above two estimates, there holds for any $0\le s\le 1$,

$$\begin{array}{c}\hfill \parallel L_{K,s}-2M{V}_{p}n{\left(2\pi \right)}^{\frac{n}{2}}{L}_K{\parallel }_K\le 2M{A}_t{\kappa }^2{c}_h{s}^{\varsigma }(2J_{2+\varsigma }+J_4+c_h{J}_2).\end{array}$$

(16)

Using Lemma 18 in [10] and (16), the desired result follows.    □

Since the measure $d\tilde{\rho }={\int }_Y\frac{p\left(z\right)R_t\left(z\right)}{V_p}d\rho$ is probability one on X, we know that the operator $L_K$ can be used to define the reproducing kernel Hilbert space [22]. Let $L_K^{1/2}$ be the $\frac{1}{2}$-th power of the positive operator $L_K$ on ${\left(L_{\tilde{\rho }}^2\right)}^n$ with norm $\parallel \overrightarrow{f}{\parallel }_{\tilde{\rho }}=({\sum }_{l=1}^n\parallel f^l{{\parallel }_{\tilde{\rho }}^2)}^{1/2}$ having a range in ${\mathcal{H}}_K^n$, where $\parallel f^l{\parallel }_{\tilde{\rho }}=({\int }_X|f^l\left(x\right){{|}^{2}d\tilde{\rho })}^{1/2}$. Then, ${\mathcal{H}}_K^n$ is the range of $L_K^{1/2}:$

$$\begin{array}{c}\hfill \parallel \overrightarrow{f}{\parallel }_{\tilde{\rho }}={\parallel L_K^{1/2}\overrightarrow{f}\parallel }_K,\overrightarrow{f}\in {\left(L_{\tilde{\rho }}^2\right)}^n.\end{array}$$

(17)

The assumption we shall use is $\parallel L_K^{-1/2}\nabla f_{\rho }{\parallel }_{\tilde{\rho }}<\infty$. It means that $\nabla f_{\rho }$ lies in the range of $L_K^{1/2}$. Finally, we can give the upper bound of the error $\parallel {\overrightarrow{f}}_{\lambda ,t}-\nabla f_{\rho }{\parallel }_{\rho }$.

Theorem 2.

Under the Assumptions 1 and 2, choose $\lambda =m^{-\frac{\tau }{n+2+3\tau }}$ and $s={\left(\kappa c_h\right)}^{\frac{2}{\zeta }}{m}^{-\frac{1}{n+2+3\tau }}$. For any $m\ge {\left(\kappa c_h\right)}^{2(n+2+3\tau )/\tau }$, there exists a constant $C_{\rho ,K}$ such that we have

$$\parallel {\overrightarrow{f}}_{\lambda ,t}-\nabla f_{\rho }{\parallel }_{\rho }\le C_{\rho ,K}\frac{A_t}{\sqrt{B_t}}{\left(\frac{1}{m}\right)}^{\frac{\zeta }{2n+4+6\zeta }}.$$

(18)

Proof of Theorem 2.

Using Cauchy inequality, for $\overrightarrow{f}={(f^1,f^2,\dots ,f^n)}^{\mathsf{T}}\in {\left(L_{{\rho }_X}^2\right)}^n$, we have

$$\begin{array}{cc}\hfill {\int }_X{\left(f^l\left(x\right)\right)}^{2}d{\rho }_X\left(x\right)& \le {\left({\int }_Z{\left(f^l\left(x\right)\right)}^2\frac{p\left(z\right)R_t\left(z\right)}{V_p}d\rho \left(z\right)\right)}^{\frac{1}{2}}{\left({\int }_Z{\left(f^l\left(x\right)\right)}^2\frac{V_p}{p\left(z\right)R_t\left(z\right)}d\rho \left(z\right)\right)}^{\frac{1}{2}}\hfill \\ & \le \sqrt{\frac{V_p}{c_l{B}_t}}{\left({\int }_Z{\left(f^l\left(x\right)\right)}^2\frac{p\left(z\right)R_t\left(z\right)}{V_p}d\rho \left(z\right)\right)}^{\frac{1}{2}}{\left({\int }_X{\left(f^l\left(x\right)\right)}^{2}d{\rho }_X\left(x\right)\right)}^{\frac{1}{2}}.\hfill \end{array}$$

It means that

$$\begin{array}{c}\hfill {\left({\int }_X{\left(f^l\left(x\right)\right)}^{2}d{\rho }_X\left(x\right)\right)}^{\frac{1}{2}}\le \sqrt{\frac{V_p}{c_l{B}_t}}{\left({\int }_Z{\left(f^l\left(x\right)\right)}^2\frac{p\left(z\right)R_t\left(z\right)}{V_p}d\rho \left(z\right)\right)}^{\frac{1}{2}}.\end{array}$$

According to the definitions of $\parallel f^l{\parallel }_{\rho }$ and $\parallel f^l{\parallel }_{\tilde{\rho }}$, it is trivial to obtain

$$\begin{array}{c}\hfill \parallel \overrightarrow{f}{\parallel }_{\rho }\le \sqrt{\frac{V_p}{c_l{B}_t}}{\parallel \overrightarrow{f}\parallel }_{\tilde{\rho }}.\end{array}$$

(19)

Since $s={\left(\kappa c_h\right)}^{\frac{2}{\zeta }}{\lambda }^{\frac{1}{\zeta }},\lambda ={\left(\frac{1}{m}\right)}^{\frac{\zeta }{n+2+3\zeta }}$, we see from the fact $J_2>1$ that the restriction $0<s\le min\{c_{\rho }{\lambda }^{\frac{1}{\zeta }},1\}$ in Lemma 3 is satisfied for $m\ge {\left(\kappa c_h\right)}^{2(n+2+3\tau )/\tau }$. Then, combining Lemmas 2 and 3, Equation (17) and inequality (19), we have

$$\begin{array}{cc}\hfill \parallel {\overrightarrow{f}}_{\lambda ,t}-\nabla f_{\rho }{\parallel }_{\rho }\le & \parallel {\overrightarrow{f}}_{\lambda ,t}-\nabla f_{\rho }+\lambda {(L_{K,s}+\lambda I)}^{-1}\nabla f_{\rho }{\parallel }_{\rho }+{\parallel \lambda {(L_{K,s}+\lambda I)}^{-1}\nabla f_{\rho }\parallel }_{\rho }\hfill \\ \hfill \le & \kappa \parallel {\overrightarrow{f}}_{\lambda ,t}-\nabla f_{\rho }+\lambda {(L_{K,s}+\lambda I)}^{-1}\nabla f_{\rho }{\parallel }_K+{\parallel \lambda {(L_{K,s}+\lambda I)}^{-1}\nabla f_{\rho }\parallel }_{\rho }\hfill \\ \hfill \le & 2M\frac{s}{\lambda }{\kappa }^2{c}_{\upsilon }{c}_h{J}_3{A}_t+2\sqrt{\frac{V_p}{c_l{B}_t}}\sqrt{\lambda }{\left(M{V}_{p}n{\left(2\pi \right)}^{\frac{n}{2}}\right)}^{-\frac{1}{2}}{\parallel \nabla f_{\rho }\parallel }_K\hfill \\ \hfill \le & C_{\rho ,K}\frac{A_t}{\sqrt{B_t}}{\left(\frac{1}{m}\right)}^{\frac{\zeta }{2n+4+6\zeta }},\hfill \end{array}$$

where $C_{\rho ,K}=\left({\left(2\kappa c_h\right)}^{\frac{2}{\zeta }}+2\right)max\{M{\kappa }^2{c}_{\upsilon }{c}_h{J}_3,\sqrt{\frac{V_p}{c_l}}{\left(M{V}_{p}n{\left(2\pi \right)}^{\frac{n}{2}}\right)}^{-\frac{1}{2}}{C}_K\}$.    □

Remark 2.

Theorem 2 shows when $m\to +\infty$, $\parallel {\overrightarrow{f}}_{\lambda ,t}-\nabla f_{\rho }{\parallel }_{\rho }\to 0$. This means that the scheme (5) is consistent. In addition, $A_t$ and $B_t$ tend to 1 as t tends 0, we can see that the convergence rate of Scheme (5) is $-\frac{\zeta }{2n+4+6\zeta }$, which is consistent with previous result in [10]. It means that the proposed method can be regarded as an extension of traditional GL.

3. Computing Algorithm

In this section, we present the GL model under TERM and propose to use the gradient descent algorithm to find the minimizer. Finally, the convergence of the proposed algorithm is also guaranteed.

Given a set of observations $\mathbf{z}={\left\{z_i=(x_i,y_i)\right\}}_{i=1}^m\in Z^m$ independently drawn according to $\rho$ and assume that the RKHS are rich that the kernel matrix $\mathbf{K}={\left(K(x_i,x_j)\right)}_{i,j=1}^m$ is strictly positive definite [27]. According to the Representer Theorem of kernel methods [28], we assert the approximation of ${\overrightarrow{f}}_{\lambda ,t}$ has the following form: ${\sum }_{i=1}^m{c}_i{K}_{x_i},c_i={(c_i^1,\dots ,c_i^n)}^{\mathsf{T}}\in {\mathbb{R}}^n.$ Let $c={(c_1^{\mathsf{T}},\cdots ,c_m^{\mathsf{T}})}^{\mathsf{T}}\in {\mathbb{R}}^{mn}$, the empirical version of (4) is formulated as follows:

$$\begin{array}{c}\hfill c_{\mathbf{z},\lambda }:=\underset{c\in {\mathbb{R}}^{mn}}{argmin}\left\{{\mathcal{E}}_{\mathbf{z}}(c,t)+\lambda \parallel \sum _{i=1}^m{c}_i{K}_{x_i}{\parallel }_K^2\right\},\end{array}$$

(20)

where

$${\mathcal{E}}_{\mathbf{z}}(c,t)=\frac{1}{t}log\left(\frac{1}{m^2}\sum _{i,j=1}^{m}exp\left\{t\omega (x_i,x_j){(y_i-y_j+\sum _{p=1}^{m}K(x_p,x_i){\widehat{x}}_{ij}{c}_p)}^2\right\}\right),$$

with ${\widehat{x}}_{ij}={(x_j-x_i)}^{\mathsf{T}}$. For simplicity, we denote

$$V_{\mathbf{z}}(c,z_i,z_j)=\omega (x_i,x_j){(y_i-y_j+\sum _{p=1}^{m}K(x_p,x_i){\widehat{x}}_{ij}{c}_p)}^2$$

and

$${\varphi }_{\mathbf{z}}(c,z_i,z_j)=exp\left\{t\left(V_{\mathbf{z}}(c,z_i,z_j)-{\mathcal{E}}_{\mathbf{z}}(c,t)\right)\right\}.$$

The gradients of ${\mathcal{E}}_{\mathbf{z}}(c,t)$ and $\parallel {\sum }_{i=1}^m{c}_i{K}_{x_i}{\parallel }_K^2$ at c are given by

$$\begin{array}{cc}\hfill {\nabla }_c{\mathcal{E}}_{\mathbf{z}}(c,t)=& \frac{1}{m^2}\sum _{i,j=1}^m{\varphi }_{\mathbf{z}}(c,z_i,z_j)2\omega (x_i,x_j)(y_i-y_j+\sum _{p=1}^{m}K(x_p,x_i){\widehat{x}}_{ij}{c}_p)\times \hfill \\ & {\left(K(x_1,x_i){\widehat{x}}_{ij},\cdots ,K(x_m,x_i){\widehat{x}}_{ij}\right)}^{\mathsf{T}},\hfill \end{array}$$

and

$${\nabla }_c\parallel \sum _{i=1}^m{c}_i{K}_{x_i}{\parallel }_K^2=2\sum _{i=1}^m{\left(K(x_i,x_1)c_i^{\mathsf{T}},\cdots ,K(x_i,x_m)c_i^{\mathsf{T}}\right)}^{\mathsf{T}}.$$

Correspondingly, scheme (20) can be solved via the following gradient method:

$$c^k=c^{k-1}-\alpha \left({\nabla }_c{\mathcal{E}}_{\mathbf{z}}(c^{k-1},t)+\lambda {\nabla }_c\parallel \sum _{i=1}^m{c}_{i,k-1}{K}_{x_i}{\parallel }_K^2\right),$$

(21)

where $c^k={(c_{1,k}^{\mathsf{T}},\cdots ,c_{m,k}^{\mathsf{T}})}^{\mathsf{T}}\in {\mathbb{R}}^{mn}$ is the calculated solution at iteration k, and $\alpha$ is the step-size. The detailed gradient descent scheme is stated in Algorithm 1. To prove the convergence, we introduce the following lemma derived from Theorem 1 in [29].

Lemma 4.

When $h\left(c\right)$ has an γ-Lipschitz continuous gradient (γ-smoothness) and is μ-strongly convex, for the basic unconstrained optimization problem $c^{*}=argminh\left(c\right)$, the gradient descent algorithm $c^k=c^{k-1}-\frac{1}{\gamma }\nabla h\left(c^{k-1}\right)$ with a step-size of $1/\gamma$ has a global linear convergence rate

$$h\left(c^k\right)-h\left(c^{*}\right)\le {\left(1-\frac{\mu }{\gamma }\right)}^k\left(h\left(c^0\right)-h\left(c^{*}\right)\right).$$

Algorithm 1 Gradient descent for the Gradient Learning under TERM

From Lemma 4, we obtain the following conclusion which states that the proposed algorithm converges to (20) by choosing a suitable step size $\alpha$.

Theorem 3.

Denote $L(c,t)={\mathcal{E}}_{\mathbf{z}}(c,t)+\lambda {\parallel {\sum }_{i=1}^m{c}_i{K}_{x_i}\parallel }_K^2$, ${\beta }_{max},{\beta }_{min}$ are the maximum and minimum eigenvalues of kernel matrix $\mathit{K}$, respectively. There exist $\mu \in {\mathbb{R}}^{+}$ and $\gamma \in {\mathbb{R}}^{+}$ dependent on t such that $L(c^k,t)$ is γ-smoothness and μ-strongly convex for any $t>(-n\lambda {\beta }_{min}/64(M^2+C_K{M}_X)M_X^{2}m{\kappa }^4)$. In addition, let the minimizer $c_{\mathbf{z},\lambda }$ defined in scheme (20) and $\left\{c^k\right\}$ be the sequence generated by Algorithm 1 with $\alpha =1/\gamma$, we have

$$\begin{array}{c}\hfill L(c^k,t)-L(c_{\mathbf{z},\lambda },t)\le {\left(1-\frac{\mu }{\gamma }\right)}^k\left(L(c^0,t)-L(c_{\mathbf{z},\lambda },t)\right).\end{array}$$

(22)

Proof of Theorem 3.

Note that the strong convexity and the smoothness are related to the Hessian Matrix, and we provide the proof by dividing the Hessian Matrix into three parts:

$$\begin{array}{cc}\hfill {\nabla }_{c{c}^{\mathsf{T}}}^{2}L(c,t)=& \underset{E_1}{\underbrace{\frac{t}{m^2}\sum _{i,j=1}^m{\varphi }_{\mathbf{z}}(c,z_i,z_j)\left({\nabla }_c{V}_{\mathbf{z}}(c,z_i,z_j)-{\nabla }_c{\mathcal{E}}_{\mathbf{z}}(c,t)\right){\nabla }_c{V}_{\mathbf{z}}{(c,z_i,z_j)}^{\mathsf{T}}}}\hfill \\ & +\underset{E_2}{\underbrace{\frac{1}{m^2}\sum _{i,j=1}^m{\varphi }_{\mathbf{z}}(c,z_i,z_j){\nabla }_{c{c}^{\mathsf{T}}}^2{V}_{\mathbf{z}}(c,z_i,z_j)}}+\underset{E_3}{\underbrace{\lambda {\nabla }_{c{c}^{\mathsf{T}}}^2{\parallel \sum _{i=1}^m{c}_i{K}_{x_i}\parallel }_K^2}}.\hfill \end{array}$$

(23)

(1) Estimation on $E_1$: Note that $m^2{\nabla }_c{\mathcal{E}}_{\mathbf{z}}(c,t)={\sum }_{i,j=1}^m{\varphi }_{\mathbf{z}}(c,z_i,z_j){\nabla }_c{V}_{\mathbf{z}}(c,z_i,z_j)$ and ${\sum }_{i,j=1}^m{\varphi }_{\mathbf{z}}(c,z_i,z_j)=m^2$. It follows that

$$\begin{array}{c}\hfill \sum _{i,j=1}^m{\varphi }_{\mathbf{z}}(c,z_i,z_j)({\nabla }_c{V}_{\mathbf{z}}(c,z_i,z_j)-{\nabla }_c{\mathcal{E}}_{\mathbf{z}}(c,t)){\nabla }_c^{\mathsf{T}}{\mathcal{E}}_{\mathbf{z}}(c,t)=0.\end{array}$$

Hence, we can get the following equation:

$$\begin{array}{c}\hfill E_1=\frac{t}{m^2}\sum _{i,j=1}^m{\varphi }_{\mathbf{z}}(c,z_i,z_j)({\nabla }_c{V}_{\mathbf{z}}(c,z_i,z_j)-{\nabla }_c{\mathcal{E}}_{\mathbf{z}}(c,t)){({\nabla }_c{V}_{\mathbf{z}}(c,z_i,z_j)-{\nabla }_c{\mathcal{E}}_{\mathbf{z}}(c,t))}^{\mathsf{T}}.\end{array}$$

(24)

Similar to the proof of Lemma 1, for $i,j=1,\cdots ,m$, it directly follows that

$$\begin{array}{c}\hfill \omega (x_i,x_j)(y_i-y_j+\sum _{p=1}^{m}K(x_p,x_i){\widehat{x}}_{ij}{c}_p)\le 2\sqrt{2(M^2+C_K{M}_X)}.\end{array}$$

Note that, for $i,j=1,\cdots ,m$, ${\nabla }_c{V}_{\mathbf{z}}(c,z_i,z_j){\nabla }_c{V}_{\mathbf{z}}{(c,z_i,z_j)}^{\mathsf{T}}$ has a sole eigenvalue, it means

$$\begin{array}{cc}\hfill {\nabla }_c{V}_{\mathbf{z}}(c,z_i,z_j){\nabla }_c{V}_{\mathbf{z}}{(c,z_i,z_j)}^{\mathsf{T}}⪯& 32(M^2+C_K{M}_X)M_X^{2}m{\kappa }^4{I}_{mn},\hfill \end{array}$$

(25)

and we have

$$\begin{array}{c}\hfill {({\nabla }_c{V}_{\mathbf{z}}(c,z_i,z_j)-{\nabla }_c{\mathcal{E}}_{\mathbf{z}}(c,t))}^{\mathsf{T}}({\nabla }_c{V}_{\mathbf{z}}(c,z_i,z_j)-{\nabla }_c{\mathcal{E}}_{\mathbf{z}}(c,t))\le 128(M^2+C_K{M}_X)M_X^{2}m{\kappa }^4.\end{array}$$

It means that the maximum eigenvalue of $E_1$ is $128t(M^2+C_K{M}_X)M_X^{2}m{\kappa }^4$. Then, the following inequations are satisfied

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{0}_{mn}⪯E_1⪯128t(M^2+C_K{M}_X)M_X^{2}m{\kappa }^4{I}_{mn},t>0;\\ 128t(M^2+C_K{M}_X)M_X^{2}m{\kappa }^4{I}_{mn}⪯E_1⪯0_{mn},t<0,\end{array}\right.\end{array}$$

(26)

where $0_{mn}$ is the $mn\times mn$ matrix with all elements zero.

(2) Estimation on $E_2$: Note that ${\nabla }_{c{c}^{\mathsf{T}}}^2{V}_{\mathbf{z}}(c,z_i,z_j)$ can be rewritten as

$$\begin{array}{c}\hfill 2\omega (x_i,x_j)\left(K(x_1,x_i){\widehat{x}}_{ij},\cdots ,K(x_m,x_i){\widehat{x}}_{ij}\right){\left(K(x_1,x_i){\widehat{x}}_{ij},\cdots ,K(x_m,x_i){\widehat{x}}_{ij}\right)}^{\mathsf{T}}.\end{array}$$

Similar to (25), we have ${\nabla }_{c{c}^{\mathsf{T}}}^2{V}_{\mathbf{z}}(c,z_i,z_j)⪯2{\kappa }^4{M}_x^2{I}_{mn}$. It follows

$$\begin{array}{c}\hfill 0_{mn}⪯E_2⪯2{\kappa }^4{M}_x^2{I}_{mn}.\end{array}$$

(27)

(3) Estimation on $E_3$: By a direct computation, we have

$$\begin{array}{c}\hfill E_3=2\lambda \left[\begin{array}{cccc}{I}_{n}K(x_1,x_1)& I_{n}K(x_1,x_2)& \cdots & I_{n}K(x_1,x_m)\\ I_{n}K(x_2,x_1)& I_{n}K(x_2,x_2)& \cdots & I_{n}K(x_2,x_m)\\ ⋮& ⋮& \ddots & ⋮\\ I_{n}K(x_m,x_1)& I_{n}K(x_m,x_2)& \cdots & I_{n}K(x_m,x_m)\end{array}\right].\end{array}$$

Setting $Q={(q_{11},q_{21},\cdots ,q_{n1},\cdots ,q_{1m},q_{2m},\cdots ,q_{nm})}^{\mathsf{T}}\in {\mathbb{R}}^{mn}$, we deduce that

$$\begin{array}{c}\hfill Q^{\mathsf{T}}{E}_{3}Q=2\lambda \sum _{l=1}^n\sum _{i=1}^m\sum _{j=1}^{m}K(x_i,x_j)q_{li}{q}_{lj}.\end{array}$$

Note that the matrix of quadratic form ${\sum }_{i=1}^m{\sum }_{j=1}^{m}K(x_i,x_j)q_{li}{q}_{lj}$ is $\mathbf{K}$, then we can obtain

$$\begin{array}{c}\hfill 2\lambda n{\beta }_{min}{I}_{mn}⪯E_3⪯2\lambda n{\beta }_{max}{I}_{mn}.\end{array}$$

(28)

Combining (26), (27) and (28), there exist two constants

$$\mu =min\{2n\lambda {\beta }_{min}+128t(M^2+C_K{M}_X)M_X^{2}m{\kappa }^4,2n\lambda {\beta }_{min}\}$$

and

$$\gamma =max\{128t(M^2+C_K{M}_X)M_X^{2}m{\kappa }^4+2n\lambda {\beta }_{max},2{\kappa }^4{M}_x^2+2n\lambda {\beta }_{max}\}$$

satisfying that

$$\begin{array}{c}\hfill \mu I_{mn}⪯{\nabla }_{c{c}^{\mathsf{T}}}^{2}L(c,t)⪯\gamma I_{mn}.\end{array}$$

Note $\mu >0$ as $t>-n\lambda {\beta }_{min}/64(M^2+C_K{M}_X)M_X^{2}m{\kappa }^4$, and it means that $L(c,t)$ is $\gamma$-smoothness and $\mu$-strongly convex. The desired result follows by Lemma 4. □

4. Simulation Experiments

In this section, we carry out simulation studies with the TGL model ($t<0$ for robust) on a synthetic data set in the robust variable selection problem. Let the observation data set $\mathbf{z}={\left\{z_i=(x_i,y_i)\right\}}_{i=1}^m$ with $x_i=(x_i^1,\cdots ,x_i^n)$ be generated by the following linear equations:

$$y_i=x_i·w+ϵ,$$

where $ϵ$ represents the outliers or noises. To be specific, three different noises are used: Cauchy noise with the location parameter $a=2$ and scale parameter $b=4$, Chi-square noise with 5 DOF scaled by 0.01 and Gaussian noise $\mathcal{N}(0,0.3)$. Three different proportions of outliers including $0\%$, $20\%$, or $40\%$ are drawn from the Gaussian noise $\mathcal{N}(0,100)$. Meanwhile, we consider two different cases with $(m,n)=(50,50),(30,80)$ corresponding to m = n and m < n, respectively. The weighted vector $w=(w^1,\cdots ,w^n)$ over different dimensions is constructed as follows:

$w^l=2+0.5sin\left(\frac{2\pi l}{10}\right)$, for $l=1,\cdots ,N_n$ and 0, otherwise.

Here, $N_n=30$ means the number of effective variables. Two situations including uncorrelated variables $x\sim \mathcal{N}(0_n,I_n)$ and correlated variables $x\sim \mathcal{N}(0_n,{\Sigma }_n)$ are implemented for x, where the covariance matrix ${\Sigma }_n$ is given with the $(l,p)$th entry $0.5^{|l-p|}$.

For the variable selection algorithms, we perform the TGL with $t=6\times {10}^{-6},-1,-10$ and compare the traditional GL model [10] and RGL model [20]. For the GL and TGL models, $N_n$ variables are selected by ranking

$$\begin{array}{c}\hfill r_l=\frac{\parallel f_{\mathbf{z},\lambda }^l{\parallel }_K^2}{{\sum }_{p=1}^n{\parallel f_{\mathbf{z},\lambda }^p\parallel }_K^2},l=1,\cdots ,n.\end{array}$$

For the RGL model, $N_n$ variables are selected by ranking

$$\begin{array}{c}\hfill r_l=\frac{{\sum }_{i=1}^m{\left(c_i^l\right)}^2}{{\sum }_{q=1}^n{\sum }_{i=1}^m{\left(c_i^q\right)}^2},l=1,\cdots ,n.\end{array}$$

A model selecting more effective variables $(\le N_n)$ means a better algorithm.

We repeat experiments for 30 times with the observation set $\mathbf{z}$ generated in each circumstance. The average selected effective variables for different circumstances are reported in

Table 1. Variable selection results for different circumstances.

	Methods	Uncorrelated Variables			Correlated Variables
		$\mathbf{0}\%$	$\mathbf{20}\%$	$\mathbf{40}\%$	$\mathbf{0}\%$	$\mathbf{20}\%$	$\mathbf{40}\%$
Cauchy noise	$GL$	$28.70$	$24.27$	$19.03$	$20.27$	$17.53$	$16.53$
$(m,n)=(50,50)$	$RGL$	$29.00$	$\mathbf{26}.\mathbf{57}$	$\mathbf{27}.\mathbf{7}$	$20.80$	$15.40$	$14.16$
	$TG{L}_{t=6\times {10}^{-6}}$	$\mathbf{29}.\mathbf{63}$	$24.06$	$18.04$	$20.67$	$17.00$	$16.23$
	$TG{L}_{t=-1}$	$29.53$	$26.07$	$26.00$	$\mathbf{21}.\mathbf{07}$	$\mathbf{17}.\mathbf{6}$	$\mathbf{17}.\mathbf{13}$
	$TG{L}_{t=-10}$	$29.53$	$24.23$	$24.03$	$16.93$	$15.78$	$15.67$
Chi-square noise	$GL$	$29.40$	$24.73$	$20.37$	$18.40$	$17.93$	$16.03$
$(m,n)=(50,50)$	$RGL$	$29.63$	$\mathbf{26}.\mathbf{90}$	$\mathbf{27}.\mathbf{60}$	$19.90$	$16.10$	$14.67$
	$TG{L}_{t=6\times {10}^{-6}}$	$\mathbf{29}.\mathbf{84}$	$24.4$	$20.90$	$18.20$	$17.30$	$17.20$
	$TG{L}_{t=-1}$	$29.14$	$24.56$	$25.18$	$\mathbf{21}.\mathbf{10}$	$\mathbf{18}.\mathbf{77}$	$\mathbf{17}.\mathbf{93}$
	$TG{L}_{t=-10}$	$25.13$	$24.10$	$24.93$	$20.83$	$17.10$	$16.60$
Gaussian noise	$GL$	$28.83$	$25.16$	$20.13$	$18.04$	$16.70$	$15.93$
$(m,n)=(50,50)$	$RGL$	$\mathbf{29}.\mathbf{40}$	$\mathbf{26}.\mathbf{70}$	$\mathbf{27}.\mathbf{20}$	$19.87$	$16.40$	$14.36$
	$TG{L}_{t=6\times {10}^{-6}}$	$29.23$	$25.23$	$20.20$	$18.37$	$17.76$	$16.3$
	$TG{L}_{t=-1}$	$27.63$	$26.20$	$25.90$	$21.06$	$\mathbf{18}.\mathbf{40}$	$\mathbf{17}.\mathbf{90}$
	$TG{L}_{t=-10}$	$22.9$	$25.23$	$25.06$	$\mathbf{21}.\mathbf{43}$	$17.13$	$16.23$
Cauchy noise	$GL$	$29.60$	$11.33$	$12.30$	$11.93$	$11.57$	$10.97$
$(m,n)=(30,80)$	$RGL$	$\mathbf{29}.\mathbf{87}$	$\mathbf{29}.\mathbf{97}$	$\mathbf{29}.\mathbf{93}$	$16.50$	$\mathbf{16}.\mathbf{97}$	$\mathbf{15}.\mathbf{20}$
	$TG{L}_{t=6\times {10}^{-6}}$	$28.47$	$10.67$	$10.49$	$11.13$	$11.03$	$10.93$
	$TG{L}_{t=-1}$	$27.06$	$20.67$	$11.3$	$\mathbf{17}.\mathbf{08}$	$14.4$	$11.56$
	$TG{L}_{t=-10}$	$16.66$	$16.23$	$15.12$	$13.97$	$13.92$	$13.54$
Chi-square noise	$GL$	$29.83$	$11.47$	$12.57$	$12.57$	$11.67$	$11.33$
$(m,n)=(30,80)$	$RGL$	$\mathbf{29}.\mathbf{93}$	$\mathbf{29}.\mathbf{93}$	$\mathbf{29}.\mathbf{71}$	$\mathbf{19}.\mathbf{87}$	$\mathbf{18}.\mathbf{80}$	$\mathbf{17}.\mathbf{50}$
	$TG{L}_{t=6\times {10}^{-6}}$	$29.03$	$11.10$	$12.90$	$12.50$	$10.87$	$11.43$
	$TG{L}_{t=-1}$	$29.37$	$23.60$	$23.53$	$16.08$	$14.4$	$11.40$
	$TG{L}_{t=-10}$	$28.17$	$23.33$	$23.23$	$13.97$	$13.92$	$13.54$
Gaussian noise	$GL$	$\mathbf{29}.\mathbf{77}$	$11.83$	$12.27$	$12.92$	$12.44$	$11.54$
$(m,n)=(30,80)$	$RGL$	$29.70$	$\mathbf{29}.\mathbf{93}$	$\mathbf{29}.\mathbf{93}$	$\mathbf{19}.\mathbf{73}$	$13.67$	$9.83$
	$TG{L}_{t=6\times {10}^{-6}}$	$28.47$	$10.67$	$10.49$	$13.06$	$9.79$	$8.73$
	$TG{L}_{t=-1}$	$27.06$	$20.67$	$11.3$	$16.08$	$\mathbf{14}.\mathbf{4}$	$11.90$
	$TG{L}_{t=-10}$	$16.66$	$16.23$	$15.12$	$13.97$	$13.92$	$\mathbf{13}.\mathbf{54}$

, and the optimal results are marked in bold. Several useful conclusions can be drawn from Table 1.

(1) When the input variables are uncorrelated, the three models have similar performance under different noise conditions and can provide satisfactory variable selection results (approaching $N_n$) without outliers. However, the performance degrades severely for GL and a little for TGL ($t<0$ for robust) with the increasing proportions of outliers, especially in case $(m,n)=(30,80)$. In contrast, RGL can always provide satisfying performance. This is consistent with the previous phenomenon [20].

(2) When the input variables are correlated, the three models also have similar performance under different noise conditions but only can select partial effective variables ranging from $N_n/3$ to $2N_n/3$. In general, they degrade slowly with the increasing proportions of outliers and perform better in case $(m,n)=(50,50)$ than in $(m,n)=(30,80)$. Specifically, the TGL model with $t=-1$ gives slightly better selection results than GL and RGL in case $(m,n)=(50,50)$. It supports the superiority of TGL to some extent.

(3) It is worth noting that the TGL model with $t=6\times {10}^{-6}$ has similar performance to GL. This phenomenon supports the theoretical conclusion that TGL recovers the GL as $t\to 0$ and the algorithmic effectiveness that the proposed gradient descent method can converge to the minimizer.

(4) Noting that the TGL model with different parameters t has great differences in the variable selection results, we further conduct some simulation studies to investigate the influence. Figure 1 shows the variable selection results of different parameters t ranging from $-100$ to $-0.1$. We can see that the satisfying performance can be achieved when the parameter t is near $-1$. It does not turn out well when $\left|t\right|$ is too large. This coincides with our previous discussion that $L(c,t)$ is strongly convex with limited t.

5. Conclusions

In this paper, we have proposed a new learning objective TGL by embedding the t-tilted loss into the GL model. On the theoretical side, we have established its consistency and provided the convergence rate with the help of error decomposition and operator approximation technique. On the practical side, we have proposed a gradient descent method to solve the learning objective and provided the convergence analysis. Simulated experiments have verified the theoretical conclusion that TGL recovers the GL as $t\to 0$ and the algorithmic effectiveness that the proposed gradient descent method can converge to the minimizer. In addition, they also demonstrated the superiority of TGL when the input variables are correlated. Along the line of the present work, several open problems deserve further research—for example, using the random feature approximation to scale up the kernel methods [30] and learning with data-dependent hypothesis space to achieve a tighter error bound [31]. These problems are under our research.