On the M-Estimator under Third Moment Condition

Abstract

Estimating the expected value of a random variable by data-driven methods is one of the most fundamental problems in statistics. In this study, we present an extension of Olivier Catoni’s classical M-estimators of the empirical mean, which focus on the heavy-tailed data by imposing more precise inequalities on exponential moments of Catoni’s estimator. We show that our works behave better than Catoni‘s both in practice and theory. The performances are illustrated in the simulation and real data.

1. Introduction

In this study, we focused on estimating the mean $m=\mathbb{E}X$ of a real random variable X, supposing that $X_1,\dots ,X_n$ are independent and identically distributed drawn from X. It is well known that the empirical mean ${\hat{m}}_n=n^{-1}{\sum }_{i=1}^n{X}_i$ is the most popular estimator of m, and theoretical properties have been thoroughly studied [1].

However, recent works have concentrated more on the performance of the estimator when the distribution is heavy-tailed (the second moment or fourth moment of the distribution does not exist), which is becoming more and more common in many research fields (see, e.g., Embrechts, Klüppelberg, and Mikosch [2]). When the data have a heavy tail, the traditional method such as the empirical mean performs poorly, and appropriate robust estimators are required, which drives related research on M-estimator (generalizations of Maximum Likelihood estimator) for correction of the outliers (Huber [3]).

There has been a renewed interest in the area of robust statistics over the last several decades. Nemirovsky and Yudin [4], Hsu and Sabato [5], and Jerrum et al. [6] proposed various forms of Median-of-means (MOM) estimators to handle data in different situations. They called for dividing the data into several groups with equal size and then calculating the empirical mean within each group, finally taking the median of these empirical means as the formal MOM estimator, which reduces the impact of heavy-tailed data. Tukey and McLaughlin [7] and Huber and Ronchetti [8] tried to improve the performance of the empirical mean by using a truncation of X (they name it truncated mean), which removed part of the sample containing $\gamma n$ maximum and minimum values depending on the parameter $\gamma \in (0,1)$ and then averaged the remaining values to improve the robustness. Catoni [9], Audibert, and Catoni [10] studied the properties of M-estimation for regression problems. The relevant works about robust techniques in various fields are summarized in Bartlett and Mendelson [11], Maronna [12], Bubeck, and Lugosi [13].

Recently, Catoni [14] modified the empirical mean to a new robust estimator. It is easy to observe that the empirical mean is the solution of the following estimation equation

$$\sum _{i=1}^n\left(X_i-\mu \right)=0.$$

(1)

If we change the form of Equation (1) to

$$\sum _{i=1}^n\varphi \left[\alpha \left(X_i-\mu \right)\right]=0.$$

(2)

The solution of (2) is called Catoni’s mean estimator, where $\varphi :\mathbb{R}\to \mathbb{R}$ is a non-decreasing differentiable truncation function such that for any $x\in \mathbb{R}$, $-log\left(1-x+x^2/2\right)$ $\le \varphi \left(x\right)\le log\left(1+x+x^2/2\right)$, and $\alpha$ is a parameter to ensure the existence of the estimator. We denote Catoni’s mean estimator by ${\tilde{m}}_{n,\alpha }$. The main purpose of the truncation function is to make $\varphi \left(x\right)$ grow slower than x, and then the effect of outliers due to heavy tails in X will be diminished. Although $\varphi \left(x\right)$ is not the derivative of some explicit error function, it still can be considered as an influence function in robust theory.

By a mild assumption that the variance $v=\mathbb{E}\left[{(X-m)}^2\right]$ of the distribution exists and choosing the parameter $\alpha$ to optimize the bounds, Catoni [14] obtained the following performance of ${\tilde{m}}_{n,\alpha }$.

Theorem 1.

Let $X_1,\dots ,X_n$ be independent, identically distributed random variables, which are drawn from X. We assume that the mean m and variance v of X exist. For any $x\in {\mathbb{R}}_{+}$, and positive integer n such that $n>2x$. Catoni’s mean estimator ${\tilde{m}}_{n,\alpha }$ with parameter $\alpha =\sqrt{\frac{2x}{nv\left(1+\frac{2x}{n-2x)}\right)}}$ satisfies,

$$\mathbb{P}\left\{\left|{\tilde{m}}_{n,\alpha }-m\right|\ge \sqrt{\frac{2vx}{n-2x}}\right\}\le 2e^{-x}.$$

(3)

Moreover, if we choose α to be independent from x as follows, and assume $n>2\left(1+x\right)$, $\alpha =\sqrt{\frac{2}{nv}},$ then

$$\mathbb{P}\left\{\left|{\tilde{m}}_{n,\alpha }-m\right|\ge \frac{1+x}{1-\left(1+x\right)/n}\sqrt{\frac{v}{2n}}\right\}\le 2e^{-x}.$$

(4)

The method of Catoni [14] is widely promoted as a robust estimator by Brownlees, Joly, and Lugosi [15], Minsker [16], and Wang et al. [17]. We need to point out here that the parameter $\alpha$ is the solution of the equation where the derivative of Catoni’s estimator’s deviation with respect to $\alpha$ equals to 0. When $v=0$, the Catoni’s estimator’s deviation is 0, and no specific $\alpha$ is needed. This also holds for Theorem 2.

The main contribution of this article is to improve Catoni’s estimator under the assumption of the third moment condition, and we named it the third-moment Catoni estimator. Starting from the adjustment of the truncation function denoted by $\psi \left(x\right)$ in our work, as Figure 1 shows, the influence function with the third moment performs closer to the true value than the original one of Catoni’s. We obtained a more precise exponential moment upper bound, which leads to a better error bound.

Simultaneously, our work had a better performance for the samples drawn from the t-distribution, which is common in many fields of research(see Jones and Faddy [18]). As a special case of the heavy-tailed distribution, the third moment of the t-distribution exists, which satisfies our assumptions about the distribution. We present the superiority of our estimator in a Monte Carlo simulation. We also show the performance of the proposed estimator under a skewed normal distribution to evaluate the adaptability of the estimator to other distributions.

The rest of the article is organized as follows. In Section 2, we introduce the main result of the third-moment Catoni’s estimator. A Monte Carlo simulation is provided in Section 3 to compare the performance of the proposed estimator with Catoni’s estimator for t-distribution. Section 4 examines the performance of the proposed estimator on real data.

2. Main Result

Let ${\left(X_i\right)}_{i=1}^n$ denote an i.i.d. sample drawn from the distribution of X. Let m, v, and s be the mean, variance, and third central moment of X, respectively, which is $\mathbb{E}\left(X\right)=m,\mathbb{E}\left[{(X-m)}^2\right]=v,\mathrm{and} \mathbb{E}\left[{(X-m)}^3\right]$$=s$.

The influence function $\psi \left(x\right)$ here should be considered wider than the original function as Catoni’s to obtain a more accurate exponential moment. In this study, we assumed that

$$\psi \left(x\right)=\left\{\begin{array}{cc}log\left(1+x+x^2/2+x^3/6\right),\hfill & x\ge 0\hfill \\ -log\left(1-x+x^2/2-x^3/6\right),\hfill & x<0.\hfill \end{array}\right.$$

(5)

Our mean estimator ${\hat{m}}_{n,\alpha }$ is the unique solution of $R_{n,\alpha }\left(\mu \right)=0$, where

$$R_{n,\alpha }\left(\mu \right)=\sum _{i=1}^n\psi \left(\alpha \left(X_i-\mu \right)\right).$$

(6)

Next, we present our main result that bounds the ${\hat{m}}_{n,\alpha }-m$ with the appropriate choice of negative parameter $\alpha$:

Theorem 2.

Let $X_1,\dots ,X_n$ be independent, identically distributed random variables with finite mean m, variance v, and third central moment s. For any $x>0$, the error bound between the estimator and the empirical mean satisfies

$$\mathbb{P}\left\{\left|{\hat{m}}_{n,\alpha }-m\right|\ge 2\left(\sqrt[3]{\frac{q}{2}+\sqrt{\Delta }}+\sqrt[3]{\frac{q}{2}-\sqrt{\Delta }}\right)\right\}\le 2e^{-x},$$

(7)

where

$$\Delta ={\left(\frac{q}{2}\right)}^2+{\left(\frac{p}{3}\right)}^3,p=\frac{3+3v{\alpha }^2}{{\alpha }^2},q=\frac{n{\alpha }^{3}s+6x-4n}{n{\alpha }^3}.$$

Under some technical assumptions that will be mentioned in the following corollary, we have the following upper bound on the probability of the exponential tail:

Corollary 1.

Let $X_1,\dots ,X_n$ be independent, identically distributed random variables with finite mean m, variance v and third central moment s. For any $x>0$ and assume that $n>\frac{3}{2}\left(1+x\right)$ and $-\sqrt{\frac{4n^3{v}^3}{729}}⩽s⩽\sqrt{\frac{4n^3{v}^3}{729}}$,

$$\mathbb{P}\left\{\left|{\hat{m}}_{n,\alpha }-m\right|\ge (1+x)\sqrt{\frac{v}{n}}\right\}\le 2e^{-x}.$$

(8)

Remark 1.

It is obvious that with the assumption that n is a positive integer and satisfies $n>\frac{3}{2}\left(1+x\right)$ and $-\sqrt{\frac{4n^3{v}^3}{729}}⩽s⩽\sqrt{\frac{4n^3{v}^3}{729}}$, then

$$\frac{1+x}{1-\left(1+x\right)/n}\sqrt{\frac{v}{2n}}\ge (1+x)\sqrt{\frac{v}{n}},$$

By assuming that $\alpha <0$, we obtained a better estimator bias than (4) in Catoni’s result.

Remark 2.

When the sample was small, our result was still valid with a small s. We might consider the following example. Let $X_1,\dots ,X_n$ be independent, identically distributed random variables, which are drawn from X. Assuming that the mean $m=0.01$, variance $v=1$, $x=1$, $n=4$, and whenever $-\sqrt{\frac{4n^3{v}^3}{729}}⩽s⩽\sqrt{\frac{4n^3{v}^3}{729}}$ such as $s=0.2$, which satisfies the assumption we have

$$\mathbb{P}\left(\right|{\hat{m}}_{n,\alpha }-1|\ge 1)\le \frac{2}{e}.$$

For the convenience of proof, we first present the following lemma (Cardano formula); refer to Høyrup’s [19] for more details.

Lemma 1.

For any general cubic equation of the form $x^3+px+q=0$, one of the roots over the field of real numbers has the form:

$$x=\sqrt[3]{-\frac{q}{2}+\sqrt{\Delta }}+\sqrt[3]{-\frac{q}{2}-\sqrt{\Delta }},$$

(9)

where the discriminant of the root Δ is as follows, when $\Delta >0$, the cubic equation has one real root; the cubic equation has three real roots when $\Delta \le 0$.

$$\Delta =\frac{q^2}{4}+\frac{p^3}{27}.$$

Proof of Theorem 2.

Due to the inequality (5) about the $\psi \left(x\right)$, we have the following exponential moment inequality of $R_{n,\alpha }\left(\mu \right)$, for all $\mu \in \mathbb{R}$:

$$\begin{array}{cc}\hfill & \mathbb{E}\left[{\mathrm{e}}^{R_{n,\alpha }\left(\mu \right)}\right]\le {\left(\mathbb{E}\left[1+\alpha (X-\mu )+\frac{{\alpha }^2{(X-\mu )}^2}{2}+\frac{{\alpha }^3{(X-\mu )}^3}{6}\right]\right)}^n\hfill \\ \hfill & ={\left(1+\mathbb{E}\left[\alpha (X-\mu )\right]+\mathbb{E}\left[\frac{{\alpha }^2{(X-\mu )}^2}{2}\right]+\mathbb{E}\left[\frac{{\alpha }^3{(X-\mu )}^3}{6}\right]\right)}^n,\hfill \end{array}$$

(10)

with a brief calculation, we have $E{[X-\mu ]}^2=v+{(m-\mu )}^2$ and $E{[X-\mu ]}^3={(m-\mu )}^3+3v(m-\mu )+s$; so, the inequality (10) can be bounded by the following term:

$$exp\left(n\alpha (m-\mu )+\frac{n{\alpha }^2\left(v+{(m-\mu )}^2\right)}{2}+\frac{n{\alpha }^3}{6}\left({(m-\mu )}^3+3v(m-\mu )+s\right)\right).$$

Similarly,

$$\begin{array}{cc}\hfill & \mathbb{E}\left[{\mathrm{e}}^{-R_{n,\alpha }\left(\mu \right)}\right]\le {\left(\mathbb{E}\left[1-\alpha (X-\mu )+\frac{{\alpha }^2{(X-\mu )}^2}{2}-\frac{{\alpha }^3{(X-\mu )}^3}{6}\right]\right)}^n\hfill \\ \hfill & ={\left(1-\mathbb{E}\left[\alpha (X-\mu )\right]+\mathbb{E}\left[\frac{{\alpha }^2{(X-\mu )}^2}{2}\right]-\mathbb{E}\left[\frac{{\alpha }^3{(X-\mu )}^3}{6}\right]\right)}^n\hfill \\ \hfill & ={\left(1-\alpha (m-\mu )+\frac{{\alpha }^2\left(v+{(m-\mu )}^2\right)}{2}-\frac{{\alpha }^3}{6}\left({(m-\mu )}^3+3v(m-\mu )+s\right)\right)}^n\hfill \\ \hfill & \le exp\left(-n\alpha (m-\mu )+\frac{n{\alpha }^2\left(v+{(m-\mu )}^2\right)}{2}-\frac{n{\alpha }^3}{6}\left({(m-\mu )}^3+3v(m-\mu )+s\right)\right).\hfill \end{array}$$

(11)

Let

$$A_1=n\alpha (m-\mu )+\frac{n{\alpha }^2\left(v+{(m-\mu )}^2\right)}{2}+\frac{n{\alpha }^3}{6}\left({(m-\mu )}^3+3v(m-\mu )+s\right),$$

$$A_2=-n\alpha (m-\mu )+\frac{n{\alpha }^2\left(v+{(m-\mu )}^2\right)}{2}-\frac{n{\alpha }^3}{6}\left({(m-\mu )}^3+3v(m-\mu )+s\right),$$

whenever $X_i$ has a finite third moment s. We can obtain from the Markov inequality that for any $\mu \in \mathbb{R}$ and $x\in {\mathbb{R}}_{+}$,

$$\begin{array}{cc}\hfill & \mathbb{P}\left\{R_{n,\alpha }\left(\mu \right)\ge A_1+x\right\}\hfill \\ \hfill & =\mathbb{P}\left\{exp\left(R_{n,\alpha }\left(\mu \right)\right)\ge exp\left(A_1+x\right)\right\}\hfill \\ \hfill & \le \mathbb{E}\left[{\mathrm{e}}^{R_{n,\alpha }\left(\mu \right)}\right]/exp\left(A_1+x\right)\hfill \\ \hfill & \le e^{-x}.\hfill \end{array}$$

(12)

In the same way, we have

$$\mathbb{P}\left\{-R_{n,\alpha }\left(\mu \right)\ge A_2+x\right\}\le e^{-x}.$$

(13)

Then, as shown in Figure 2.

We can control the estimator ${\hat{m}}_{n,\alpha }$ by the roots of the cubic equation as follows:

$$\begin{array}{cc}\hfill & C_{+}\left(\mu \right)=A_1+x=0,\hfill \\ \hfill & C_{-}\left(\mu \right)=-A_2-x=0.\hfill \end{array}$$

(14)

Equation (13) above can be regarded as a cubic equation about $m-\mu$. To solve (13), we first convert it into a standard-form one-dimensional cubic equation by letting $y_n=m-\mu -\frac{1}{\alpha },(n=1,2)$, and then we obtain the following equations:

$$\begin{array}{cc}\hfill & y_1^3+\frac{3+3v{\alpha }^2}{{\alpha }^2}{y}_1+\frac{n{\alpha }^{3}s+6x-4n}{n{\alpha }^3}=0,\hfill \\ \hfill & y_2^3+\frac{3+3v{\alpha }^2}{{\alpha }^2}{y}_2-\frac{n{\alpha }^{3}s+6x-4n}{n{\alpha }^3}=0.\hfill \end{array}$$

(15)

For any $\alpha \in {\mathbb{R}}_{-}$, according to Lemma 1, since $(3+3v{\alpha }^2)/{\alpha }^2$ is always positive, $\Delta$ is always greater than 0. In this case, our equation has one real root and two imaginary roots, which means we can control the ${\hat{m}}_{n,\alpha }$ by the root of (13) as follows:

$${\mu }_{+}=m-\frac{1}{\alpha }+\sqrt[3]{\frac{q}{2}-\sqrt{\Delta }}+\sqrt[3]{\frac{q}{2}+\sqrt{\Delta }},$$

$${\mu }_{-}=m-\frac{1}{\alpha }-\sqrt[3]{\frac{q}{2}+\sqrt{\Delta }}-\sqrt[3]{\frac{q}{2}-\sqrt{\Delta }},$$

where the $\Delta$, p, and q are the same as above. We can easily obtain from the formula above that $R_{n,\alpha }\left({\mu }_{+}\right)\le 0$, implying that ${\hat{m}}_{\alpha ,n}<{\mu }_{+}$ with probability at least $1-e^{-x}$, since $R_{n,\alpha }\left(\mu \right)$ is a non-increasing function. Similarly, ${\hat{m}}_{\alpha ,n}>{\mu }_{-}$ with probability at least $1-e^{-x}$. Then, by choosing the parameter $\alpha$, we can derive the performance of the estimator ${\hat{m}}_{\alpha ,n}$ for the bias of the mean m. That is, with probability at least $1-2e^{-x}$, we have

$${\mu }_{-}<{\hat{m}}_{\alpha ,n}<{\mu }_{+}.$$

The proof of Theorem 2 is completed. □

Proof of Corollary 1. 

In fact, the right-hand side of (7) can be bounded as follows without limiting the sign of s:

$$\begin{array}{cc}\hfill \left|{\hat{m}}_{n,\alpha }-m\right|& \le 2\left(\left|\sqrt[3]{\frac{q}{2}+\sqrt{\Delta }}+\sqrt[3]{\frac{q}{2}-\sqrt{\Delta }}\right|\right)\hfill \\ \hfill & <4\left|\sqrt[3]{\frac{n{\alpha }^{3}s+6x-4n}{2n{\alpha }^3}}\right|\hfill \\ \hfill & =4\left|\sqrt[3]{-\frac{2}{{\alpha }^3}+\frac{3x}{n{\alpha }^3}+\frac{s}{2}}\right|,\hfill \end{array}$$

(16)

with the assumption $n>\frac{3}{2}\left(1+x\right)$, which is weaker than Catoni’s, (16) can be bounded by

$$\begin{array}{cc}\hfill & 4\left|\sqrt[3]{\frac{2}{{\alpha }^3}-\frac{2}{{\alpha }^3}+\frac{s}{2}}\right|\hfill \\ \hfill & =4\left|\sqrt[3]{\frac{s}{2}}\right|.\hfill \end{array}$$

(17)

Moreover, assuming that $-\sqrt{\frac{4n^3{v}^3}{729}}⩽s⩽\sqrt{\frac{4n^3{v}^3}{729}}$, we can obtain that (17) is bounded by $(1+x)\sqrt{\frac{v}{n}}$; then, (8) holds. □

3. Simulation

In this section, we considered the performance of the estimator with respect to the t-distribution on applications by Monte Carlo simulation exercise results. We focused on the performance of the estimator in $L_1$ regression. Our data were simulated from a linear model generated from a t-distribution regressed by our proposed estimator; we measured the loss of the regression by the minimization of the $L_1$ norm.

The details of the simulation are as follows: we considered n independent, identically distributed real random variables pairs $\left(X_1,Y_1\right),\left(X_2,Y_2\right)\dots$, $\left(X_n,Y_n\right)$ where $X_i$ take their values in ${\mathbb{R}}^3$ while $Y_i$ in $\mathbb{R}$, and the explanatory variables $X_i$ are drawn from a multivariate normal distribution with 0 mean, and variance is a three-dimensional identity matrix. The response variable $Y_i$ is generated as follows:

$$Y_i=X_i^T\theta +{ϵ}_i,$$

(18)

where the parameter vector $\theta$ is set to be $(0.25,-0.25,0.50)$, and ${ϵ}_i$ is an error term with zero mean and unit variance, which is drawn from a Student t-distribution. Our main goal was to estimate the parameter $\theta$ by minimizing the $L_1$ risk

$$\mathbb{E}\left|Y-X_i^T\theta \right|,$$

and then we defined the the $L_1$ estimators ${\hat{\theta }}_1$, the classical Catoni mean estimator ${\hat{\theta }}_2$, and the third-moment Catoni’s estimator ${\hat{\theta }}_3$ as follows

$$\begin{array}{cc}\hfill & {\hat{\theta }}_1=arg\underset{\theta }{min}{\hat{R}}_1\left(\theta \right)=arg\underset{\theta }{min}\frac{1}{n}\sum _{i=1}^n\left|Y_i-X_i^T\theta \right|,\hfill \\ \hfill & {\hat{\theta }}_2=arg\underset{\theta }{min}{\hat{R}}_2\left(\mu \right)=arg\underset{\theta }{min}\frac{1}{n\alpha }\sum _{i=1}^n\varphi \left(\alpha \left(\left|Y_i-X_i^T\theta \right|-\mu \right)\right)=0,\hfill \\ \hfill & {\hat{\theta }}_3=arg\underset{\theta }{min}{\hat{R}}_3\left(\mu \right)=arg\underset{\theta }{min}\frac{1}{n\alpha }\sum _{i=1}^n\psi \left(\alpha \left(\left|Y_i-X_i^T\theta \right|-\mu \right)\right)=0,\hfill \end{array}$$

(19)

where the ${\hat{R}}_2\left(\mu \right)$, ${\hat{R}}_3\left(\mu \right)$ is the root of the right side of the equation, respectively; $\varphi \left(x\right)$ is the widest choice defined in Catoni’s result, the parameter $\alpha =1$, which is the same as Brownless’s work; $\psi \left(x\right)$ was set as above; and the parameter $\alpha =-1$. The measures for the performance of the estimator are as follows:

$$\begin{array}{cc}\hfill & R\left({\hat{\theta }}_1\right)-R\left(\theta \right)=\mathbb{E}\left|Y-X^T{\hat{\theta }}_1\right|-\mathbb{E}\left|Y-X^T\theta \right|,\hfill \\ \hfill & R\left({\hat{\theta }}_2\right)-R\left(\theta \right)=\mathbb{E}\left|Y-X^T{\hat{\theta }}_2\right|-\mathbb{E}\left|Y-X^T\theta \right|,\hfill \\ \hfill & R\left({\hat{\theta }}_3\right)-R\left(\theta \right)=\mathbb{E}\left|Y-X^T{\hat{\theta }}_3\right|-\mathbb{E}\left|Y-X^T\theta \right|.\hfill \end{array}$$

(20)

The simulation experiments repeated with different sample sizes, which ranged from 50 to 1000 and with degrees of freedom of the t-distribution ranging from 1 to 7. Each set of the sample size experiments was replicated 1000 times, and for each replication, we evaluated the performance of the regression by the mean of the sample $\left(X_1^{\prime },Y_1^{\prime }\right),\left(X_2^{\prime },Y_2^{\prime }\right),\dots ,\left(X_m^{\prime },Y_m^{\prime }\right)$—that is, i.i.d.with the sample $\left(X_1,Y_1\right),\left(X_2,Y_2\right)\dots$, $\left(X_n,Y_n\right)$. We used the following equation to evaluate the performance of the regression, which called excess risk.

$$\begin{array}{cc}\hfill & \tilde{R}\left({\hat{\theta }}_1\right)=\frac{1}{m}\sum _{i=1}^m{\left|Y_i^{\prime }-Z_i^T{\hat{\theta }}_1\right|}^2,\hfill \\ \hfill & \tilde{R}\left({\hat{\theta }}_2\right)=\frac{1}{m}\sum _{i=1}^m{\left|Y_i^{\prime }-Z_i^T{\hat{\theta }}_2\right|}^2,\hfill \\ \hfill & \tilde{R}\left({\hat{\theta }}_3\right)=\frac{1}{m}\sum _{i=1}^m{\left|Y_i^{\prime }-Z_i^T{\hat{\theta }}_3\right|}^2.\hfill \end{array}$$

(21)

Figure 3 displays the performance of the excess risk for three estimators when $n=500$ and the degrees of freedom of the t-distribution ranged from 1 to 7; we can obtain that the proposed estimator performs better than the other estimators, which means more stability on the outliers.

The results of the Monte Carlo simulation including the performance of the estimator for different n are presented in

Table 1. The excess risk of the $L_1$, Catoni, and third-moment Catoni regression estimator for different degrees of freedom and sample size n.

		$\mathit{n}=50$	$\mathit{n}=100$	$\mathit{n}=250$	$\mathit{n}=500$	$\mathit{n}=1000$
$d=\mathbf{1}$	$L_1$	$8.79$	$5.91$	$5.75$	$4.15$	$3.67$
	C	$7.53$	$4.63$	$4.90$	$4.07$	$3.49$
	$C_3$	$7.46$	$4.06$	$4.84$	$4.06$	$3.38$
$d=\mathbf{3}$	$L_1$	$1.51$	$1.34$	$1.22$	$1.15$	$1.14$
	C	$1.38$	$1.27$	$1.20$	$1.15$	$1.12$
	$C_3$	$1.27$	$1.21$	$1.14$	$1.10$	$1.08$
$d=\mathbf{5}$	$L_1$	$1.09$	$1.11$	$1.08$	$1.08$	$1.07$
	C	$1.08$	$1.13$	$1.09$	$1.10$	$1.08$
	$C_3$	$1.06$	$1.08$	$1.03$	$1.04$	$1.04$
$d=\mathbf{7}$	$L_1$	$1.08$	$1.02$	$1.05$	$1.01$	$1.00$
	C	$1.05$	$0.94$	$1.03$	$1.00$	$0.98$
	$C_3$	$0.97$	$0.94$	$0.90$	$0.85$	$0.86$

, and we also compared the performance between the proposed estimator and other estimators with various risks in

Table 2. Comparisons of the performance between the proposed estimator and other estimators with various risks.

	${\mathit{L}}_1$	C	${\mathit{C}}_3$
$ER$	4.1561	4.0747	4.0628
$RB$	0.0398	0.0385	0.0383
$SMSE$	0.0970	0.0952	0.0947

where sample size $n=500$ and degrees of freedom $d=1$; the $L_1$ represents the general $L_1$ regression; the C and $C_3$ denote the original Catoni estimator and our third-moment Catoni estimator, respectively; and the ER, RB, and SMSE represents the excess risk, relative risk ($\frac{|\parallel \overline{\hat{\theta }}{\parallel }_2-{\parallel \theta \parallel }_2|}{{\parallel \theta \parallel }_2}$, with $\overline{\hat{\theta }}=\frac{1}{1000}{\sum }_{j=1}^{1000}{\hat{\theta }}^{\left(j\right)}$), and the square root of the mean square error ($\sqrt{MSE}=\sqrt{\frac{1}{1000}{\sum }_{j=1}^{1000}[\parallel \hat{\theta }{\left(j\right)\parallel }_2-{\parallel \theta \parallel }_2{]}^2}$).

We can derive from the table that when the distribution has a heavy tail, our estimator performs better in most cases than the other two estimators, and the excess risk of the estimator decreases as the sample size increases. At the same time, with the degrees of freedom of the t-distribution rising, the tail of the t-distribution becomes thinner, which becomes closer to the normal distribution, and the performance of all procedures on excess risk is significantly improved; additionally, the proposed estimator also performs well for different risks.

We also examined the performance of the third-moment Catoni estimator on a skewed normal distribution in

Table 3. The excess risk of the $L_1$, Catoni, and third-moment Catoni regression estimator on a skewed normal distribution.

		$\mathit{n}=50$	$\mathit{n}=100$	$\mathit{n}=250$	$\mathit{n}=500$	$\mathit{n}=1000$
$s=\mathbf{1}$	$L_1$	$0.847$	$0.829$	$0.820$	$0.807$	$0.784$
	C	$0.865$	$0.844$	$0.825$	$0.785$	$0.779$
	$C_3$	$0.859$	$0.837$	$0.823$	$0.789$	$0.781$
$s=\mathbf{3}$	$L_1$	$0.857$	$0.833$	$0.809$	$0.819$	$0.798$
	C	$0.861$	$0.842$	$0.829$	$0.835$	$0.828$
	$C_3$	$0.861$	$0.843$	$0.827$	$0.835$	$0.824$
$s=\mathbf{5}$	$L_1$	$0.831$	$0.825$	$0.812$	$0.792$	$0.782$
	C	$0.856$	$0.855$	$0.850$	$0.839$	$0.828$
	$C_3$	$0.855$	$0.855$	$0.848$	$0.827$	$0.822$

; the model still follows (18) where $ϵ$ follows a skewed normal distribution with shape parameter $\alpha =1,3,5$ and with other settings unchanged. We can draw conclusions from the table that the bias of the improved estimator is still smaller than the original one. However, the deviation in the estimator did not display a significant difference as the shape parameter $\alpha$ changed. We suppose that this results from the tail behavior of the skew normal distribution in that the existence of its fourth moment conflicts with the usual assumption that the fourth moment of heavy tail distribution does not exist. At the same time, neither Catoni’s estimator nor our estimator performed better than the estimator obtained by L1 regression.

4. Empirical Analysis

In this section, we used the proposed procedure to research the dataset “tumor cell resistance to death,” an artificial dataset consisting of two different types of tumor cells A and B, and the experiment records their resistance to different doses of experimental drugs. The explanatory variable $X_i$ here is the dose of the drug, and the response variable $Y_i$ is the score representing the resistance to death, ranging from 0 to 4. These data are available in the R lqr package; Galarza et al. [20] have studied these data by the quantile regression method.

In Figure 4, Figure 5, Figure 6 and Figure 7, we display the QQplot and the log-QQplot about the scores for cell A and cell B, and it can be seen that the distribution of both cells lacks normality; however, the normality is satisfied between cells and log-scores; besides, the boxplot and the bee colony diagram in Figure 8 and Figure 9 shows that both cell A and cell B have heavy-tails, which allows us to focus on the following regression model:

$$log\left(Y_i\right)={\beta }_0+{\beta }_1{X}_i,$$

where $Y_i$ and $X_i$ are defined before. Our focus was estimating the parameters ${\beta }_0$ and ${\beta }_1$, the solution of the following equation:

$${\hat{r}}_{\beta }\left(u\right)=\frac{1}{n\alpha }\sum _{i=1}^n\psi \left(\alpha \left(\left|log\left(Y_i\right)-{\beta }_0-X_i^{\top }{\beta }_1\right|-u\right)\right)=0.$$

Let ${\hat{R}}_C\left(\beta \right)$ denote the solution of the ${\hat{r}}_{\beta }\left(u\right)=0$; then, the Catoni regression estimator of ${\beta }_0$ and ${\beta }_1$ is in the form as follows:

$$arg\underset{{\beta }_0,{\beta }_1}{min}{\hat{R}}_C\left(\beta \right).$$

Moreover, we compared the proposed estimator with the classical OLS estimator in Figure 10 and Figure 11. The residuals plots are shown in Figure 12, Figure 13, Figure 14 and Figure 15, from which we can draw the conclusion that the distribution of the residual of the three-order Catoni regression performs more uniformly; furthermore, the Mean Squared Error of the third-moment Catoni regression and OLS regression was 0.1120, 0.1255 for cell A and 0.2268, 0.2335 for cell B, respectively, which indicates that the proposed method is a better regression.

5. Discussion

Estimating the mean of random variables is a classical issue in statistics [1], and it has been well studied in classical statistics; however, with the discovery of heavy-tailed distribution in many research fields, its existence is an important challenge in statistics. When the data have heavy tails, the traditional estimators such as the empirical mean usually perform poorly. Therefore, how to find an appropriate robust procedure is a well-known problem and has aroused great interest. A new estimator based on reconstructing the structure of the empirical mean was proposed by Catoni, which has excellent theoretical properties on the bias.

The Catoni’s estimator is based on the existence of the variance v of the random variable. Therefore, with a weaker assumption on the moment conditions, it is an interesting issue whether the estimator has a better performance. In this study, we assumed that the third moment s of the data exists, and a more accurate upper bound of the exponential moment was obtained, which motivates an estimator with a better bias. To a certain extent, the assumption reduces the robustness to outliers, but it has a minimal effect in heavy tails distribution (the fourth moment does not exist). In future work, we have the following goals: first, we believe that our method can be applied as an improved mean estimator to any relevant model as long as the third moment of the distribution has good theoretical properties and wide application; second, it is an interesting idea to discuss and compare the bias bound of the proposed estimator with the minimax bound; finally, the estimation of the variance in regression models is very important in statistical inference. Considering that the deviation of the estimator given in our main theoretical results from the true value can be regarded as the confidence interval based on the known variance; therefore, the proposed estimator is not suitable for the estimation of variance, but it is an interesting issue how a proper variance estimator affects the bias of our estimator; additionally, we will consider variance estimation under heavy-tailed distributions in later work.