Transfer Learning for Moderate–Dimensional Ridge-Regularized Robust Linear Regression

Abstract

This paper studies transfer learning for ridge-regularized robust linear regression in the moderate–dimensional regime, where the number of predictors is of the same order as the sample size and the regression coefficients are not assumed to be sparse. We propose Trans-RR, which combines a robust ridge estimator from a source study with a robust ridge correction based on the target study. Under mild assumptions, we characterize the asymptotic estimation error of the proposed estimator and show that leveraging source data can substantially improve estimation accuracy relative to the traditional single-study ridge-regularized robust estimator. To guard against negative transfer when the source study is not sufficiently informative, we further propose an adaptive aggregation of Trans-RR with the single-task estimator that selects the mixing weight by cross-validation. Simulation studies and a real-data analysis support the theory and illustrate the transition between positive and negative transfer as the discrepancy between the source and target studies varies.

1. Introduction

Modern statistical analyses often involve several related datasets collected from different studies, populations, or experiments. A basic question is how to use these datasets together to improve prediction and estimation for a study of interest. Transfer learning addresses this question by borrowing useful information from related tasks. It is now standard in machine learning and has been successful in applications such as natural language processing, remote sensing, and computer vision [1,2,3]. In statistics, it has also become an important tool for improving performance in multi-study problems.

In many contemporary applications, regression problems arise in a moderate–dimensional regime, where the number of predictors is of the same order as the sample size and sparsity is often not a reasonable structural assumption. At the same time, heavy-tailed errors or outlying observations may substantially affect estimation accuracy. This setting arises naturally in multisite metabolomics studies, where many metabolites are measured simultaneously, between-cohort heterogeneity is often present, and outliers or other data contamination can be an important practical concern [4]. These features make transfer learning particularly challenging: related source studies may contain useful information, but effective borrowing requires methods that are robust to contamination and suitable for moderate–dimensional, non-sparse settings.

Existing theoretical work on transfer learning covers several important settings. For linear regression, ref. [5] studied data-enriched regression in a fixed-dimensional setting, and ref. [6] analyzed linear models with a shared low-dimensional representation across tasks. In the high-dimensional sparse regime, ref. [7] considered transfer learning with proxy data, while ref. [8] established prediction and estimation guarantees for sparse linear regression. For high-dimensional generalized linear models, refs. [9,10] developed transfer learning methods with theoretical guarantees. Transfer learning has also been studied for nonparametric classification [11], nonparametric regression [12], and settings with unreliable source data [13]. However, these works do not apply to the moderate–dimensional robust setting considered here.

Robust regression has, in contrast, been extensively studied in the single-study setting. For classical M-estimation, a substantial body of work has established asymptotic results when $p/n\to 0$ while $p\to \infty$, including [14,15,16,17,18,19]. When $p/n\to \kappa \in (0,1)$, robust regression has a qualitatively different asymptotic behavior [20]. When $p/n\to \kappa >0$, ref. [21] proposed a ridge-regularized robust estimator to address the nonexistence of the ordinary robust estimator. However, these results do not directly extend to the transfer learning framework when related source data are available.

Motivated by these gaps, we study transfer learning under a moderate–dimensional robust linear model with one target study and one related source study. We allow the predictor dimension to be of the same order as the target and source sample sizes, do not impose sparsity assumptions on the regression coefficients, and permit heavy-tailed errors. Within this framework, our goal is to leverage information from the source study to improve the estimation performance of traditional single-task approaches.

Our main contribution is to develop and analyze a transfer learning method for moderate–dimensional ridge-regularized robust linear regression. First, we propose Trans-RR, a transfer learning procedure for ridge-regularized robust linear regression. It combines a robust ridge fit on the source study with a robust ridge correction on the target study and is designed for non-sparse coefficients. Second, we derive an asymptotic characterization of the ${\mathcal{l}}_2$ risk of the resulting estimator under mild assumptions on the design and error distributions. The theory shows how auxiliary source data can improve estimation accuracy relative to the single-study ridge-regularized robust estimator, while also clarifying the possibility of negative transfer. Third, we propose an adaptive aggregation of Trans-RR with the single-task estimator that selects the mixing weight by cross-validation, providing a data-driven safeguard against negative transfer. Fourth, we conduct simulation studies and a real-data analysis to examine the practical performance of the proposed methods, including their sensitivity to tuning choices and to the identity–covariance assumption underlying the theory.

The rest of the paper is organized as follows. Section 2 introduces the model setup and the proposed algorithm. Section 3 presents the technical assumptions, the theoretical results, and the adaptive aggregation against negative transfer. Section 4 presents simulation studies to evaluate the performance of the proposed methods. Section 5 applies the proposed methods to a real-data example. The proofs of the main theoretical result as well as the lemmas are included in Appendix D and Appendix E.

Notation. Denote by ${\mathrm{I}}_m$ the $m\times m$ identity matrix. Let ${\mathbf{0}}_m\in {\mathbb{R}}^m$ and ${\mathbf{1}}_m\in {\mathbb{R}}^m$ be the vectors of zeros and ones, respectively. For a vector $\mathit{v}={(v_1,\dots ,v_m)}^{\top }$, the ${\mathcal{l}}_2$ norms are $\parallel \mathit{v}\parallel ={({\sum }_{i=1}^m{v}_i^2)}^{1/2}$, whereas ${\parallel \mathit{v}\parallel }_{\infty }={\max }_{1\le k\le p}\left|v_k\right|$. For an $m\times m$ matrix $\mathit{A}={\left\{a_{ij}\right\}}_{1\le i,j\le m}$, denote by ${\lambda }_{\max }(\mathit{A})$ and ${\lambda }_{\min }(\mathit{A})$ the maximum and minimum eigenvalues of $\mathit{A}$, respectively. The $L_2$ norm of $\mathit{A}$ is defined as $\parallel \mathit{A}\parallel ={\left\{{\lambda }_{\max }({\mathit{A}}^{\top }\mathit{A})\right\}}^{1/2}$.

2. Methodology

2.1. Problem Setup

We consider a transfer learning problem with one target study and one related source study. In the target study, we observe n samples ${\mathit{x}}_i\in {\mathbb{R}}^p$ and $y_i\in \mathbb{R}$, $i=1,\dots ,n$, generated from

$$y_i={\mathit{x}}_i^{\top }{\mathit{\beta }}_0+{ϵ}_i,$$

(1)

where ${ϵ}_i,i=1,\dots ,n$, are independently distributed errors and ${\mathit{\beta }}_0\in {\mathbb{R}}^p$ is the unknown regression parameter of interest.

In addition to the target data, we observe $n_1$ samples $({\mathit{x}}_i^{(1)},y_i^{(1)})$, $i=1,\dots ,n_1$, from the source study satisfying

$$y_i^{(1)}={({\mathit{x}}_i^{(1)})}^{\top }{\mathit{w}}_0+{ϵ}_i^{(1)},$$

(2)

where ${\mathit{w}}_0\in {\mathbb{R}}^p$ is the regression parameter for the source study and ${ϵ}_i^{(1)},i=1,\dots ,n_1$, are independently distributed errors. Throughout, both ${ϵ}_i$ and ${ϵ}_i^{(1)}$ are allowed to be heavy-tailed.

We work in the moderate–dimensional regime, where p is of the same order as both n and $n_1$, with $p/n\to \kappa \in (0,\infty )$ and $p/n_1\to {\kappa }_1\in (0,\infty )$. We also do not impose sparsity assumptions on ${\mathit{\beta }}_0$ or ${\mathit{w}}_0$. Let ${\mathit{\delta }}_0={\mathit{\beta }}_0-{\mathit{w}}_0$ denote the source–target discrepancy and let $h=\parallel {\mathit{\delta }}_0\parallel$ measure its size. Smaller values of h correspond to a more informative source study and hence a greater potential for useful transfer.

2.2. Trans-RR Algorithm

Based on this setup, we now introduce the proposed transfer learning algorithm, referred to as Trans-RR. Following the general two-stage strategy used in [7,8,9], the core of the procedure consists of two estimation steps. The first step estimates the source coefficient vector ${\mathit{w}}_0$ from the source data. The second step estimates the source–target discrepancy ${\mathit{\delta }}_0={\mathit{\beta }}_0-{\mathit{w}}_0$ from the target data, after which the two estimates are combined. Algorithm 1 summarizes the procedure.    

Algorithm 1: Trans-RR algorithm

Input: target data ${\left\{({\mathit{x}}_i,y_i)\right\}}_{i=1}^n$ and source data ${\left\{({\mathit{x}}_i^{(1)},y_i^{(1)})\right\}}_{i=1}^{n_1}$  Output: the estimated coefficient vector $\widehat{\mathit{\beta }}$  Step 1. Compute $$\widehat{\mathit{w}}=\underset{\mathit{w}\in {\mathbb{R}}^p}{\mathrm{arg\; min}}\left[{\displaystyle \frac{1}{n_1}}{\displaystyle \sum _{i=1}^{n_1}}\tilde{\rho }\left\{y_i^{(1)}-{({\mathit{x}}_i^{(1)})}^{\top }\mathit{w}\right\}+{\displaystyle \frac{{\tau }_1}{2}}{\parallel \mathit{w}\parallel }^2\right]$$ (3)  for some constant ${\tau }_1$.  Step 2. Compute $$\widehat{\mathit{\delta }}=\underset{\mathit{\delta }\in {\mathbb{R}}^p}{\mathrm{arg\; min}}\left[{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\rho \left\{y_i-{\mathit{x}}_i^{\top }(\widehat{\mathit{w}}+\mathit{\delta })\right\}+{\displaystyle \frac{\tau }{2}}{\parallel \mathit{\delta }\parallel }^2\right]$$ (4)  for some constant $\tau$.  Step 3. Let $$\widehat{\mathit{\beta }}=\widehat{\mathit{w}}+\widehat{\mathit{\delta }}.$$ (5)  Step 4. Output $\widehat{\mathit{\beta }}$.

The idea behind the construction is straightforward. Step 1 computes a ridge-regularized robust estimator from the source study. Step 2 then estimates the discrepancy relative to the source-stage fit by solving a second ridge-regularized robust regression problem on the target study. The final estimator is obtained by combining these two pieces, namely $\widehat{\mathit{\beta }}=\widehat{\mathit{w}}+\widehat{\mathit{\delta }}$. The main difference between our procedure and those of [7,8,9] is that we use ridge/${\mathcal{l}}_2$ regularization in both steps, whereas they use ${\mathcal{l}}_1$ regularization. This choice is motivated by our diffuse-coefficient setting: the regression parameters ${\mathit{\beta }}_0$ and ${\mathit{w}}_0$ have many small coordinates and are not well approximated by sparse vectors. In this setting, lasso-based methods are not well-suited to the problem, whereas ridge penalization is natural. Another difference is that we use robust loss functions rather than the quadratic loss, which makes the procedure less sensitive to outliers and heavy-tailed errors.

Remark 1. 

When robustness to heavy-tailed errors or outliers is needed, Huber-type loss functions are natural candidates for ρ and $\tilde{\rho }$. Specific choices of ρ and $\tilde{\rho }$ under our theoretical framework are discussed in Section 3. The regularization parameters τ and ${\tau }_1$ may be selected by standard data-driven tuning methods such as cross-validation.

3. Theoretical Results

This section introduces the assumptions for the analysis and then presents the main asymptotic error results for Trans-RR.

3.1. Technical Assumptions

We study the estimation error of the estimator in Algorithm 1 under the following assumptions. We state the assumptions separately for the target study (Assumption 1) and the source study (Assumption 2), since the two stages are based on different samples. The two sets of conditions are largely parallel.

Assumption 1. 

(a) 

$p/n\to \kappa \in (0,\infty )$.

(b) 

Suppose ρ is an even and convex function. Assume that $\psi ={\rho }^{\prime }$ is bounded and ${\psi }^{\prime }$ is Lipschitz and bounded. Moreover, we assume that $\mathrm{sign}(\psi (x))=\mathrm{sign}(x)$ and that $\rho (x)\ge \rho (0)=0$ for all $x\in \mathbb{R}$.

(c) 

Assume that there exist independent variables ${\lambda }_i$’s and ${\mathcal{X}}_i$’s such that ${\mathit{x}}_i={\lambda }_i{\mathcal{X}}_i$. Suppose that ${\mathcal{X}}_i$’s are i.i.d. with independent entries, and they have mean ${\mathbf{0}}_p$ and $\mathrm{cov}({\mathcal{X}}_i)={\mathrm{I}}_p$. Suppose there exist ${\mathrm{c}}_n$ and $C_n$ that vary with n, where $1/{\mathrm{c}}_n=\mathrm{O}(\mathrm{polyLog}(n))$ and $C_n$ is bounded in n, such that for any convex 1-Lipschitz function G of ${\mathcal{X}}_i$, $P(|G({\mathcal{X}}_i)-m_G|>t)\le C_n\exp (-{\mathrm{c}}_n{t}^2)$ holds for all $t>0$, where $m_G$ is the median of $G({\mathcal{X}}_i)$. We require the same assumption to hold for the columns of the $n\times p$ design matrix $\mathcal{X}$. Additionally, we assume that the coordinates of ${\mathcal{X}}_i$ have moments of all orders, and the k-th moment of the entries of ${\mathcal{X}}_i$ is assumed to be uniformly bounded independently of n and p for all k.

(d) 

Suppose that ${\lambda }_i$’s are independent, with $\mathrm{E}({\lambda }_i^2)=1$, $\mathrm{E}({\lambda }_i^4)$ being bounded, and ${\sup }_{1\le i\le n}\left|{\lambda }_i\right|$ growing at most like $C_{\lambda }{(\log n)}^k$ for some k. ${\lambda }_i$’s may have finitely many possible distributions.

(e) 

Suppose that ${ϵ}_i$’s are independent and are also independent of ${\mathcal{X}}_i$’s and ${\lambda }_i$’s. They may have finitely many possible distributions, each with a density that is differentiable, symmetric, and unimodal. If $f_i$ is the density of one such distribution, we assume that ${\lim }_{x\to \infty }x{f}_i(x)=0$.

(f) 

The fraction of occurrences for each possible combination of distributions for $({ϵ}_i,{\lambda }_i)$ has a limit as $n\to \infty$.

(g) 

There exist constants $C_{\mathit{\beta }}$ and $\mathrm{e}>1/3$ such that $\parallel {\mathit{\beta }}_0\parallel \le C_{\mathit{\beta }}$ and $\parallel {\mathit{\beta }}_0{\parallel }_{\infty } = \mathrm{O}(n^{-\mathrm{e}})$.

Assumption 2. 

(a) 

$p/n_1\to {\kappa }_1\in (0,\infty )$.

(b) 

$\tilde{\rho }$ and $\tilde{\psi }$ satisfy Assumption 1(b).

(c) 

${\mathit{x}}_i^{(1)}$, ${\mathcal{X}}_i^{(1)}$’s and ${\lambda }_i^{(1)}$’s satisfy Assumption 1(c).

(d) 

${\lambda }_i^{(1)}$’s satisfy Assumption 1(d).

(e) 

${ϵ}_i^{(1)}$’s, ${\mathcal{X}}_i^{(1)}$’s and ${\lambda }_i^{(1)}$’s satisfy Assumption 1(e).

(f) 

${\lambda }_i^{(1)}$’s and ${ϵ}_i^{(1)}$’s satisfy Assumption 1(f).

(g) 

$\parallel {\mathit{w}}_0{\parallel }_2$ remains bounded. Furthermore, $\parallel {\mathit{w}}_0{\parallel }_{\infty }=\mathrm{O}(n_1^{-\mathrm{e}})$, where $\mathrm{e}>1/3$.

For Assumptions 1(b) and 2(b), it is quite common in robust statistics to require $\psi$ to be bounded. For example, the Huber loss

$${\rho }_H(x)=\{\begin{array}{cc}{\displaystyle \frac{x^2}{2}}\hfill & \mathrm{if}\left|x\right|\le \delta ,\\ \delta ·\left(\left|x\right|-{\displaystyle \frac{1}{2}}\delta \right)\hfill & \mathrm{otherwise}.\end{array}$$

is chosen to grow linearly to infinity, which reduces the influence of outliers on the resulting regression estimator. Although the Huber loss does not fully satisfy the assumptions because it is not differentiable at $\left|x\right|=\delta$, these assumptions hold for a smoothed approximation such as

$${\rho }_{\eta }(x)=\{\begin{array}{cc}{\displaystyle \frac{x^2}{2}}\hfill & \mathrm{if}\left|x\right|\le \delta -\eta ,\hfill \\ \left(\delta -{\displaystyle \frac{\eta }{2}}\right)·\left|x\right|+{\displaystyle \frac{{(\delta -|x|)}^3}{6\eta }}+C_{\rho }\hfill & \mathrm{if}\left|x\right|\in (\delta -\eta ,\delta ),\hfill \\ \left(\delta -{\displaystyle \frac{\eta }{2}}\right)·\left|x\right|+C_{\rho }\hfill & \mathrm{if}\left|x\right|\ge \delta ,\hfill \end{array}$$

(6)

where $C_{\rho }=-{\eta }^2/6+\eta \delta /2-{\delta }^2/2$. The corresponding ${\psi }_{\eta }$ is given by

$${\psi }_{\eta }(x)=\{\begin{array}{cc}x\hfill & \mathrm{if}\left|x\right|\le \delta -\eta ,\hfill \\ \mathrm{sign}(x)·\left\{\delta -{\displaystyle \frac{\eta }{2}}-{\displaystyle \frac{{(\delta -|x|)}^2}{2\eta }}\right\}\hfill & \mathrm{if}\left|x\right|\in (\delta -\eta ,\delta ),\hfill \\ \mathrm{sign}(x)·\left(\delta -{\displaystyle \frac{\eta }{2}}\right)\hfill & \mathrm{if}\left|x\right|\ge \delta .\hfill \end{array}$$

(7)

Another example that fully satisfies the assumptions is the pseudo-Huber loss function from [22,23], defined by

$${\rho }_P(x)={\delta }^2\left(\sqrt{1+{\displaystyle \frac{x^2}{{\delta }^2}}}-1\right).$$

(8)

The assumptions on ${\mathit{x}}_i$’s and ${\mathit{x}}_i^{(1)}$’s, in particular that they have mean ${\mathbf{0}}_p$ and covariance matrix ${\mathrm{I}}_p$, are common in the study of M-estimators for linear models. These assumptions have been used in the low-dimensional regime $p/n\to 0$ studied in [14,15,24], in the moderate–dimensional regime $p/n\to \kappa \in (0,1)$ considered in [20], and in the regime $p/n\to \kappa >0$ analyzed in [21,25].

The concentration assumption on ${\mathcal{X}}_i$’s and ${\mathcal{X}}_i^{(1)}$’s is weaker than the Gaussian assumptions often imposed in robust statistics. This assumption has also been studied in [21,26] and holds for a broad class of distributions. Corollary 4.10 in [27] demonstrates that our assumptions are satisfied if ${\mathcal{X}}_i$ has independent entries bounded by $1/(2\sqrt{c_1})$ for some $c_1>0$. Additionally, Theorem 2.7 in [27] shows that the assumptions hold when ${\mathcal{X}}_i$ has independent entries with density $f_k$, $1\le k\le p$, such that $f_k(x)=\exp (-u_k(x))$ and $u_k^{\prime \prime }(x)\ge \sqrt{c_2}$ for some $c_2>0$. In particular, this condition holds when ${\mathcal{X}}_i$ has i.i.d. $N(0,1)$ entries, where $c_2=1$. As will be seen in the proof, the functions G that arise in our analysis are either linear functions or square roots of quadratic forms. A similar discussion applies to the ${\mathcal{X}}_i^{(1)}$’s.

The introduction of ${\lambda }_i$ and ${\lambda }_i^{(1)}$, as also considered in [20,21], is used to induce a nonspherical geometry on the predictors. Although the assumption $\mathrm{E}({\lambda }_i^2)=1$ can be relaxed to the requirement that $\mathrm{E}({\lambda }_i^2)$ be uniformly bounded, it remains statistically important because it guarantees that $\mathrm{cov}({\mathit{x}}_i)={\mathrm{I}}_p$ in all the models we consider. This construction shows that many models can share the same covariance $\mathrm{cov}({\mathit{x}}_i)$ while exhibiting substantially different estimation errors for $\widehat{\mathit{\beta }}$. This contrasts with the low-dimensional setting studied in [28], where $\mathrm{cov}({\mathit{x}}_i)$ is the key quantity for robust regression. A similar discussion applies to ${\lambda }_i^{(1)}$’s and ${\mathit{x}}_i^{(1)}$’s.

In Assumptions 1(e) and 2(e), no moment restriction is imposed on the ${ϵ}_i$’s and ${ϵ}_i^{(1)}$’s. For instance, smooth symmetric log-concave densities satisfy all of these assumptions; see [29,30]. Furthermore, the Cauchy distribution also satisfies these conditions; see Theorem 1.6 in [31]. This makes the framework compatible with heavy-tailed errors, which are of particular interest in robust regression.

Assumptions 1(g) and 2(g) impose a non-sparse structure on ${\mathit{\beta }}_0$ and ${\mathit{w}}_0$, meaning that these vectors cannot be well approximated by sparse vectors in ${\mathcal{l}}_2$ norm. This setting is common in moderate–dimensional statistics and contrasts with the sparse regime, where only a small fraction of coefficients are substantial. Under these assumptions, the proposed Trans-RR estimator may outperform lasso-based methods.

We now turn to the target-stage result and the resulting error characterization for Trans-RR.

3.2. Asymptotic Characterization of Estimation Error

Our main theorem characterizes the asymptotic ${\mathcal{l}}_2$ error of the Trans-RR estimator. Recall that $\widehat{\mathit{\beta }}$ is defined in (5), and let $\tau >0$ be fixed as n and p vary. To state the result, let $\mathrm{prox}(c\rho )$ denote the proximal mapping of the function $c\rho$, see [32]. It is given by

$$\mathrm{prox}(c\rho )(x)=\underset{y\in \mathbb{R}}{\mathrm{arg\; min}}\left\{c\rho (y)+\frac{1}{2}{(x-y)}^2\right\},x\in \mathbb{R}.$$

(9)

When $\rho$ is differentiable, $\mathrm{prox}(c\rho )(x)$ is the unique $y\in \mathbb{R}$ satisfying $y+c\psi (y)=x$, with $\psi ={\rho }^{\prime }$. $\mathrm{prox}(c\rho )(x)$ can be viewed as a shrinkage of x toward the minimizer of $\rho$, with the amount of shrinkage depending on c and $\rho$. The proximal mapping is a standard object in convex analysis and convex optimization (see [33] for a review of its analytic properties and an introduction to proximal gradient algorithms). As explained in [34], the system in Theorem 1 can be reformulated in terms of $\mathrm{prox}({(c_{\rho }(\kappa )\rho )}^{\ast })$, where $f^{\ast }(u)={\sup }_{y\in \mathbb{R}}\{uy-f(y)\}$ denotes the Fenchel–Legendre conjugate of f.

Under Assumptions 1 and 2, the estimation error admits the following limit.

Theorem 1. 

Under Assumptions 1 and 2, conditional on the source-stage estimator $\widehat{\mathit{w}}$, which is independent of the target sample, we have $\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0\parallel \to r_{\rho }(\kappa )$ in probability, where $r_{\rho }(\kappa )$ is deterministic for the given value of $\widehat{\mathit{w}}$. Let $W_i={ϵ}_i+r_{\rho }(\kappa ){\lambda }_i{Z}_i$, where $Z_i$ is a standard normal random variable independent of ${ϵ}_i$ and ${\lambda }_i$. Then there exists a constant $c_{\rho }(\kappa )$ such that

$$\{\begin{array}{cc}\underset{n\to \infty }{\lim }{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left\{{\left[\mathrm{prox}\left\{c_{\rho }(\kappa ){\lambda }_i^2\rho \right\}\right]}^{\prime }(W_i)\right\}\hfill & =1-\kappa +\tau c_{\rho }(\kappa ),\hfill \\ \kappa \left[\underset{n\to \infty }{\lim }{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left({\displaystyle \frac{{[W_i-\mathrm{prox}\left\{c_{\rho }(\kappa ){\lambda }_i^2\rho \right\}(W_i)]}^2}{{\lambda }_i^2}}\right)\right]+{\tau }^2{\parallel {\mathit{\beta }}_0-\widehat{\mathit{w}}\parallel }^2{c}_{\rho }^2(\kappa )\hfill & ={\kappa }^2{r}_{\rho }^2(\kappa ).\hfill \end{array}$$

(10)

The proof of Theorem 1 is given in Appendix E. Here and below, the dependence of $r_{\rho }(\kappa )$ and $c_{\rho }(\kappa )$ on $\widehat{\mathit{w}}$ is suppressed for notational simplicity.

Theorem 1 shows that the asymptotic error $r_{\rho }(\kappa )$ depends on the source study only through the discrepancy $\parallel {\mathit{\beta }}_0-\widehat{\mathit{w}}\parallel$. To investigate when positive transfer occurs, Section 4.2 numerically computes $r_{\rho }$ as a function of $\parallel {\mathit{\beta }}_0-\widehat{\mathit{w}}\parallel$ under the smoothed Huber loss (6) in three simulation cases. The resulting curves are monotonically increasing across the displayed range. By the triangle inequality,

$$\parallel {\mathit{\beta }}_0-\widehat{\mathit{w}}\parallel \le \parallel {\mathit{\beta }}_0-{\mathit{w}}_0\parallel +\parallel \widehat{\mathit{w}}-{\mathit{w}}_0\parallel .$$

The right-hand side is the sum of the population gap between the source and target coefficients and the source-stage estimation error. Transfer is therefore expected to help when the source-stage estimator is accurate and close to the target coefficient, and to hurt when either the population gap $\parallel {\mathit{\beta }}_0-{\mathit{w}}_0\parallel$ or the source-stage estimation error $\parallel \widehat{\mathit{w}}-{\mathit{w}}_0\parallel$ is large. In practice, Section 3.3 develops an adaptive Trans-RR estimator to avoid negative transfer.

Unlike several recent transfer learning analyses, such as [7,8,9], our theory does not impose direct structural restrictions on ${\mathit{\delta }}_0$, the difference between the target and source coefficients. Theorem 1 also shows that the performance of $\widehat{\mathit{\beta }}$ depends on the distribution of the ${\lambda }_i$’s in the representation ${\mathit{x}}_i={\lambda }_i{\mathcal{X}}_i$ from Assumption 1(c). Thus, in the moderate–dimensional regime, the geometry of the target predictors encoded by ${\lambda }_i$ materially affects the estimation error. This again contrasts with low-dimensional robust regression, where $\mathrm{cov}({\mathit{x}}_i)$ is the dominant quantity.

When ${\lambda }_i^2=1$ for all i and the errors ${ϵ}_i$ are i.i.d., Theorem 1 simplifies as follows.

Corollary 1. 

Under the same assumptions as in Theorem 1, if ${\lambda }_i^2=1$ for all i and the errors ${ϵ}_i$ are i.i.d., then, conditional on $\widehat{\mathit{w}}$, we have $\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0\parallel \to r_{\rho }(\kappa )$ in probability, where $r_{\rho }(\kappa )$ is deterministic for the given value of $\widehat{\mathit{w}}$. Let $w=ϵ+r_{\rho }(\kappa )z$, where ϵ has the same distribution as the ${ϵ}_i$’s and z is a standard normal random variable independent of ϵ. Then there exists a constant $c_{\rho }(\kappa )$ such that

$$\{\begin{array}{cc}\mathrm{E}\left\{{\left[\mathrm{prox}\left\{c_{\rho }(\kappa )\rho \right\}\right]}^{\prime }(w)\right\}\hfill & =1-\kappa +\tau c_{\rho }(\kappa ),\hfill \\ \kappa \mathrm{E}\left({[w-\mathrm{prox}\left\{c_{\rho }(\kappa )\rho \right\}(w)]}^2\right)+{\tau }^2{\parallel {\mathit{\beta }}_0-\widehat{\mathit{w}}\parallel }^2{c}_{\rho }^2(\kappa )\hfill & ={\kappa }^2{r}_{\rho }^2(\kappa ).\hfill \end{array}$$

Corollary 1 shows that, under a homogeneous target design, the general characterization in Theorem 1 reduces to a simpler scalar system. This special case is useful for interpretation and will also serve as a convenient benchmark in the simulation study.

Remark 2. 

The limits on the left-hand side of (10) exist because Assumption 1(f) guarantees convergence of the proportions associated with each pair $(\mathcal{L}({ϵ}_i),\mathcal{L}({\lambda }_i))$, where $\mathcal{L}({ϵ}_i)$ and $\mathcal{L}({\lambda }_i)$ denote the laws of ${ϵ}_i$ and ${\lambda }_i$. For the second equation in (10), the ratio can be interpreted through the identity

$${\displaystyle \frac{{[x-\mathrm{prox}\left\{c_{\rho }(\kappa ){\lambda }^2\rho \right\}(x)]}^2}{{\lambda }^2}}=c_{\rho }^2(\kappa ){\lambda }^2{\psi }^2(\mathrm{prox}\left\{c_{\rho }(\kappa ){\lambda }^2\rho \right\}(x))$$

which is well defined when $\lambda =0$. Equivalently, (10) can be written as

$$\{\begin{array}{cc}\underset{n\to \infty }{\lim }{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left\{{\left[\mathrm{prox}\left\{c_{\rho }(\kappa ){\lambda }_i^2\rho \right\}\right]}^{\prime }(W_i)\right\}\hfill & =1-\kappa +\tau c_{\rho }(\kappa ),\hfill \\ \kappa \left[\underset{n\to \infty }{\lim }{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left\{c_{\rho }^2(\kappa ){\lambda }_i^2{\psi }^2(\mathrm{prox}\left\{c_{\rho }(\kappa ){\lambda }_i^2\rho \right\}(W_i))\right\}\right]+{\tau }^2{\parallel {\mathit{\beta }}_0-\widehat{\mathit{w}}\parallel }^2{c}_{\rho }^2(\kappa )\hfill & ={\kappa }^2{r}_{\rho }^2(\kappa ).\hfill \end{array}$$

This representation shows that the expectation in (10) is well defined, and Assumption 1(f) ensures that the relevant limits exist.

3.3. Adaptive Aggregation Against Negative Transfer

We now develop an adaptive aggregation of Trans-RR with the single-task estimator to address negative transfer. Specifically, let ${\widehat{\mathit{\beta }}}_{st}$ denote the single-task ridge-regularized robust estimator on the target sample,

$${\widehat{\mathit{\beta }}}_{st}=\underset{\mathit{\beta }\in {\mathbb{R}}^p}{\mathrm{arg\; min}}\left[{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\rho (y_i-{\mathit{x}}_i^{\top }\mathit{\beta })+{\displaystyle \frac{{\tau }_{st}}{2}}{\parallel \mathit{\beta }\parallel }^2\right]$$

(11)

for some constant ${\tau }_{st}$. Theorem 1 and the discussion above show that negative transfer may happen when the target–source discrepancy $\parallel {\mathit{\beta }}_0-{\mathit{w}}_0\parallel$ or the source-stage estimation error is too large. We therefore propose the adaptive Trans-RR estimator to avoid negative transfer, defined for $\theta \in [0,1]$ by

$${\widehat{\mathit{\beta }}}_{\mathrm{ada}}(\theta )=\theta \widehat{\mathit{\beta }}+(1-\theta ){\widehat{\mathit{\beta }}}_{st}.$$

(12)

Intuitively, ${\widehat{\mathit{\beta }}}_{\mathrm{ada}}(\theta )$ recovers Trans-RR at $\theta =1$ and the single-task estimator at $\theta =0$. Including both endpoints allows the procedure to fall back to either Trans-RR or the single-task estimator, while interior values allow partial transfer. We select the mixing weight $\theta$ from data by cross-validation and write ${\widehat{\mathit{\beta }}}_{\mathrm{ada}}:={\widehat{\mathit{\beta }}}_{\mathrm{ada}}(\widehat{\theta })$ for the resulting estimator.

We tune the ridge penalties before selecting $\theta$. The source penalty ${\tau }_1$ is tuned by cross-validation on the source sample, yielding $\widehat{\mathit{w}}$ via (3). With $\widehat{\mathit{w}}$ fixed, the target penalty $\tau$ is tuned by cross-validation on the target sample through (4). The single-task penalty ${\tau }_{st}$ is tuned by cross-validation on the target sample. To select $\theta$, we then use a K-fold partition ${\left\{{\mathcal{V}}_k\right\}}_{k=1}^K$ of the target sample, drawn independently of the partitions used to tune $\tau$ and ${\tau }_{st}$. Here, ${\mathcal{V}}_k$ denotes the validation index set of fold k.

For each $k=1,\dots ,K$, let ${\widehat{\mathit{\beta }}}^{(-k)}$ be the output of Algorithm 1 applied to the full source data and $\{({\mathit{x}}_i,y_i):i\notin {\mathcal{V}}_k\}$ at the tuned $({\tau }_1,\tau )$. Let ${\widehat{\mathit{\beta }}}_{st}^{(-k)}$ be the solution of (11) on $\{({\mathit{x}}_i,y_i):i\notin {\mathcal{V}}_k\}$ at the tuned ${\tau }_{st}$. We reuse ${\tau }_1$, $\tau$, and ${\tau }_{st}$ across all K folds rather than re-tuning per fold, which would multiply the penalty-tuning cost by K. We then choose

$$\widehat{\theta }=\underset{\theta \in \mathrm{\Theta }}{\mathrm{arg\; min}}{\displaystyle \frac{1}{n}}{\displaystyle \sum _{k=1}^K}\sum _{i\in {\mathcal{V}}_k}L\left(y_i-{\mathit{x}}_i^{\top }[\theta {\widehat{\mathit{\beta }}}^{(-k)}+(1-\theta ){\widehat{\mathit{\beta }}}_{st}^{(-k)}]\right),$$

(13)

where $L:\mathbb{R}\to {\mathbb{R}}_{\ge 0}$ is a validation loss and $\mathrm{\Theta }\subset [0,1]$ is a finite candidate set. A natural default is the absolute-error loss $L(t)=\left|t\right|$, which matches the criterion used to tune the ridge penalties. We refer to ${\widehat{\mathit{\beta }}}_{\mathrm{ada}}$ as the adaptive Trans-RR estimator, denoted Trans-RR-Ada in the following numerical experiments. Algorithm 2 summarizes the procedure.    

Algorithm 2: Adaptive Trans-RR algorithm

Input: target data ${\left\{({\mathit{x}}_i,y_i)\right\}}_{i=1}^n$, source data ${\left\{({\mathit{x}}_i^{(1)},y_i^{(1)})\right\}}_{i=1}^{n_1}$, fold count K,     candidate set $\mathrm{\Theta }\subset [0,1]$, validation loss L  Output: the adaptive estimator ${\widehat{\mathit{\beta }}}_{\mathrm{ada}}$ and the selected weight $\widehat{\theta }$  Step 1. Tune ${\tau }_1$, $\tau$, and ${\tau }_{st}$ by cross-validation on the corresponding samples.  Compute $\widehat{\mathit{\beta }}$ from Algorithm 1 and ${\widehat{\mathit{\beta }}}_{st}$ from (11).  Step 2. Draw a K-fold partition ${\left\{{\mathcal{V}}_k\right\}}_{k=1}^K$ of the target indices, independent of the  partitions used in Step 1.  Step 3. For each $k=1,\dots ,K$, let ${\widehat{\mathit{\beta }}}^{(-k)}$ be the output of Algorithm 1 applied to the  full source data and $\{({\mathit{x}}_i,y_i):i\notin {\mathcal{V}}_k\}$ at the tuned $({\tau }_1,\tau )$. Let ${\widehat{\mathit{\beta }}}_{st}^{(-k)}$ be the  solution of (11) on $\{({\mathit{x}}_i,y_i):i\notin {\mathcal{V}}_k\}$ at the tuned ${\tau }_{st}$.  Step 4. Compute $$\widehat{\theta }=\underset{\theta \in \mathrm{\Theta }}{\mathrm{arg\; min}}{\displaystyle \frac{1}{n}}{\displaystyle \sum _{k=1}^K}\sum _{i\in {\mathcal{V}}_k}L\left(y_i-{\mathit{x}}_i^{\top }[\theta {\widehat{\mathit{\beta }}}^{(-k)}+(1-\theta ){\widehat{\mathit{\beta }}}_{st}^{(-k)}]\right).$$  Step 5. Output ${\widehat{\mathit{\beta }}}_{\mathrm{ada}}=\widehat{\theta }\widehat{\mathit{\beta }}+(1-\widehat{\theta }){\widehat{\mathit{\beta }}}_{st}$ and $\widehat{\theta }$.

3.4. Applicability and Limitations

Theorem 1 and Corollary 1 are derived under Assumptions 1 and 2, which include three structural conditions: identity covariance $\mathrm{cov}({\mathit{x}}_i)={\mathit{I}}_p$, the moderate–dimensional regime $p/n\to \kappa \in (0,\infty )$, and a twice-differentiable robust loss. Under these assumptions, the asymptotic ${\mathcal{l}}_2$ estimation error of Trans-RR equals the deterministic limit $r_{\rho }(\kappa )$, in agreement with the numerical results of Section 4. When any of these assumptions fails, Theorem 1 no longer applies.

The adaptive aggregation of Section 3.3, by contrast, is constructed without invoking these structural assumptions. Its mixing weight $\widehat{\theta }$ is selected by cross-validation on the target sample and serves as a data-driven safeguard against negative transfer. Section 4.4 provides numerical support under AR(1)-correlated predictors across all three noise cases: a transition between positive and negative transfer is observed near $h=1$, and Trans-RR-Ada continues to track the better of the two base estimators.

4. Simulation

In this section, we conduct numerical studies to support the theoretical results. We set the dimension of both target and source data to be $p\in \{200,400,800\}$. We set $n=p,p/4$ and $n_1=2p,p/2$, corresponding to moderate–dimensional settings with $\kappa =1,4$ and ${\kappa }_1=1/2,2$. To generate data, we set ${\mathit{x}}_i={\lambda }_i{\mathcal{X}}_i$ and ${\mathit{x}}_i^{(1)}={\lambda }_i^{(1)}{\mathcal{X}}_i^{(1)}$, where ${\mathcal{X}}_i$ and ${\mathcal{X}}_i^{(1)}$ have i.i.d. $N(0,1)$ entries. We consider three different cases for the choices of ${\lambda }_i$’s, ${ϵ}_i$’s, ${\lambda }_i^{(1)}$’s and ${ϵ}_i^{(1)}$’s:

• Case I: ${\lambda }_i=1$ for $i=1,\dots ,n$ and ${\lambda }_j^{(1)}=1$ for $j=1,\dots ,n_1$. The target errors ${ϵ}_i$ are i.i.d. $N(0,1)$, and the source errors ${ϵ}_j^{(1)}$ are i.i.d. $N(0,2^2)$.
• Case II: The variables ${\lambda }_i$ and ${\lambda }_j^{(1)}$ are i.i.d. Unif$(0,\sqrt{3})$, while ${ϵ}_i$ and ${ϵ}_j^{(1)}$ are i.i.d. $Cauchy(0,1)$ and $Cauchy(0,2)$, respectively.
• Case III: In both the target and source studies, half of the observations are generated as in Case $\mathrm{I}$ and the other half are generated as in Case $\mathrm{II}$.

Case $\mathrm{I}$ is a standard Gaussian setup for linear regression. Case $\mathrm{II}$ features a non-Gaussian design and heavy-tailed errors. Case $\mathrm{III}$ is a mixture of the two cases and is designed to test the effectiveness of our theoretical results under non-identical ${\mathit{x}}_i$’s and ${ϵ}_i$’s.

4.1. Validity of Theoretical Results

We first evaluate the validity of the proposed scalar $r_{\rho }$ in Theorem 1. For each setting, we generate ${\mathit{\beta }}^{\ast }$ and ${\mathit{w}}^{\ast }$ once with i.i.d. Unif$(0,1)$ entries and set ${\mathit{\beta }}_0={\mathit{\beta }}^{\ast }/\sqrt{n}$ and ${\mathit{w}}_0={\mathit{w}}^{\ast }/\sqrt{n}$. This construction yields diffuse coefficients whose Euclidean norms remain bounded as n grows. These coefficient vectors are fixed across the 1000 replications, while the target and source samples are regenerated in each replicate. In each replicate, we first compute $\widehat{\mathit{w}}$ from the source sample and then obtain $\widehat{\mathit{\beta }}$ by applying Algorithm 1, using the smoothed Huber loss (6) for both $\tilde{\rho }$ and $\rho$ with parameters $\delta =1.35$ and $\eta =0.1$. We fix $\tau ={\tau }_1=1$ and repeat each setup 1000 times.

Figure 1 presents boxplots of the estimation error $\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0{\parallel }^2$ for cases $\mathrm{I}$–$\mathrm{III}$ and $\kappa =1,4$. The red point in each boxplot marks the theoretical value $r_{\rho }^2$, obtained by numerically solving the system in Theorem 1 under the corresponding simulation specification. We observe that the empirical distribution of $\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0{\parallel }^2$ is centered close to this value, and its dispersion decreases as n and p become larger. Table 1 shows the mean and standard deviation (SD) of $\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0{\parallel }^2$ (denoted as ${\widehat{r}}^2$) and the corresponding $r_{\rho }^2$ for each setup. As dimensionality increases, that is, as $\kappa$ increases from 1 to 4, both the mean error and its variability increase, indicating that estimation becomes more difficult in more challenging moderate–dimensional regimes. The average estimation error also grows with heavier-tailed errors, highlighting the difficulty of estimation under such conditions. Results under case $\mathrm{III}$ demonstrate that Theorem 1 is effective in handling non-identical ${\mathit{x}}_i$’s and ${ϵ}_i$’s. Overall, the findings in Figure 1 and Table 1 align well with the theoretical predictions of Theorem 1.

Figure 1. Boxplot of $\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0{\parallel }^2$ over 1000 simulations. The red point in each boxplot represents the theoretical value $r_{\rho }^2$ from Theorem 1. Panels from top to bottom are for $\kappa =1,4$, respectively, while panels from left to right are for cases $\mathrm{I},\mathrm{II},\mathrm{III}$, respectively.

Table 1. Mean and SD (in parentheses) of $\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0{\parallel }^2$ (denoted as ${\widehat{r}}^2$) and the corresponding $r_{\rho }^2$ over 1000 simulations. Rows are indexed by $(p,n)$, with $n_1=2p$ when $\kappa =1$ and $n_1=p/2$ when $\kappa =4$.

($\mathit{p},\mathit{n}$)	Case $\mathbf{I}$		Case $\mathbf{II}$		Case $\mathbf{III}$
	${\widehat{\mathit{r}}}^{\mathbf{2}}$	${\mathit{r}}_{\mathit{\rho }}^{\mathbf{2}}$	${\widehat{\mathit{r}}}^{\mathbf{2}}$	${\mathit{r}}_{\mathit{\rho }}^{\mathbf{2}}$	${\widehat{\mathit{r}}}^{\mathbf{2}}$	${\mathit{r}}_{\mathit{\rho }}^{\mathbf{2}}$
$\kappa =1$
(200, 200)	0.3653 (0.0318)	0.3649	0.7163 (0.0738)	0.7204	0.4683 (0.0478)	0.4685
(400, 400)	0.3472 (0.0208)	0.3477	0.6970 (0.0549)	0.6923	0.5076 (0.0368)	0.5065
(800, 800)	0.3603 (0.0151)	0.3598	0.7206 (0.0374)	0.7212	0.4989 (0.0238)	0.4986
$\kappa =4$
(200, 50)	1.3415 (0.0734)	1.3419	2.8222 (0.3311)	2.8216	2.2410 (0.2407)	2.2415
(400, 100)	1.3565 (0.0531)	1.3544	2.4896 (0.2363)	2.4996	1.8087 (0.1590)	1.8102
(800, 200)	1.5261 (0.0427)	1.5247	2.7226 (0.1693)	2.7219	2.0342 (0.1153)	2.0301

4.2. Theoretical Estimation Error Curves

We take $\rho$ to be our recommended smoothed Huber loss (6) with $(\delta ,\eta )=(1.35,0.1)$ and fix $\kappa =1$. We numerically solve the associated scalar system while varying the discrepancy term $\parallel {\mathit{\beta }}_0-\widehat{\mathit{w}}\parallel$ that enters the second equation of Theorem 1. Figure 2 plots the resulting curves of $r_{\rho }$ for five values of $\tau$ under cases $\mathrm{I}$–$\mathrm{III}$. In all three cases, the curves are monotonically increasing over the displayed range, so a larger $\parallel {\mathit{\beta }}_0-\widehat{\mathit{w}}\parallel$ corresponds to a larger asymptotic estimation error in this setting.

Figure 2. Theoretical curves of $r_{\rho }$ as a function of $\parallel {\mathit{\beta }}_0-\widehat{\mathit{w}}\parallel$ for five values of $\tau$ under cases $\mathrm{I}$–$\mathrm{III}$, obtained by numerically solving Corollary 1. The three panels correspond to cases $\mathrm{I}$, $\mathrm{II}$, and $\mathrm{III}$, respectively.

4.3. Comparison with Existing Methods

To evaluate when transfer is beneficial, we compare our method with several competing procedures across the three scenarios described above. We set $p=400$, $n=p$ and $n_1=2p$. We generate ${\mathit{\beta }}_0={\mathit{\beta }}^{\ast }/\parallel {\mathit{\beta }}^{\ast }\parallel$, where ${\mathit{\beta }}^{\ast }={({\mathit{\beta }}_1^{\ast },\dots ,{\mathit{\beta }}_p^{\ast })}^{\top }$ has i.i.d. Unif$(0,1)$ entries. To control the transfer strength $h=\parallel {\mathit{\delta }}_0\parallel$, we set

$${\mathit{\delta }}_0=\exp (c_d)·{\mathbf{1}}_p/\sqrt{p},c_d\in \{-2.0,-1.5,-1.0,-0.5,0,0.5,1.0\},$$

and define the source coefficient by ${\mathit{w}}_0={\mathit{\beta }}_0-{\mathit{\delta }}_0$. By varying $c_d$ from $-2.0$ to $1.0$, we obtain h ranging from approximately $0.135$ to $2.718$, providing a comprehensive evaluation across different levels of source-target similarity. For each value of $c_d$, the pair $({\mathit{\beta }}_0,{\mathit{w}}_0)$ is fixed across the 500 replications, and only the data are regenerated.

We compare four ridge-type methods (Single-RR, Trans-RR, Trans-RR-Ada, and Pooled-RR) and two lasso baselines (Single-Lasso and Trans-Lasso):

• Single-RR: The single-task estimator ${\widehat{\mathit{\beta }}}_{st}$ in (11), fit to the target sample alone.
• Trans-RR: The two-stage estimator $\widehat{\mathit{\beta }}=\widehat{\mathit{w}}+\widehat{\mathit{\delta }}$ in (5), computed by Algorithm 1.
• Trans-RR-Ada: The adaptive aggregate ${\widehat{\mathit{\beta }}}_{\mathrm{ada}}$ in (12), computed by Algorithm 2 with $K=5$, absolute-error loss $L(t)=\left|t\right|$, and weight grid $\mathrm{\Theta }=\{0,0.1,\dots ,0.9,1\}$.
• Pooled-RR: The same robust ridge fit applied to the concatenation of the source and target samples.
• Single-Lasso: The lasso on the target sample, with its regularization parameter chosen by $five$-fold cross-validation.
• Trans-Lasso: The two-stage transfer-lasso of [8], in which a cross-validated source-stage lasso estimates ${\mathit{w}}_0$ and a cross-validated target-stage lasso fits the residual $y_i-{\mathit{x}}_i^{\top }\widehat{\mathit{w}}$ on the target sample.

The four ridge-type methods all use the smoothed Huber loss (6) with $(\delta ,\eta )=(1.35,0.1)$. Each ridge penalty is tuned by $five$-fold cross-validation under the mean absolute error criterion, over the grid $\{3^a:a=-2,-3/2,\dots ,2\}$. For Trans-RR, the source penalty ${\tau }_1$ in (3) is tuned on the source sample, yielding $\widehat{\mathit{w}}$ as the minimizer of (3) at the tuned ${\tau }_1$. With $\widehat{\mathit{w}}$ held fixed, the target penalty $\tau$ in (4) is then tuned on the target sample.

Performance is summarized by the relative estimation error $\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0{\parallel }^2/{\parallel {\mathit{\beta }}_0\parallel }^2$. Figure 3 presents boxplots of these errors on a logarithmic scale, for varying values of h across cases $\mathrm{I}$–$\mathrm{III}$. We report all six estimators under Case $\mathrm{I}$. In Cases $\mathrm{II}$ and $\mathrm{III}$, the noise is heavy-tailed, and the lasso methods are not designed for robustness. Their fits failed to converge to stable estimates, so we restrict these two cases to the ridge-type methods.

Figure 3. Boxplots of relative estimation errors (log scale) across 500 replications for varying $\parallel {\mathit{\beta }}_0-{\mathit{w}}_0\parallel$ under cases $\mathrm{I}$–$\mathrm{III}$, with $p=400$ and $n=400$. Case $\mathrm{I}$ includes all six methods, while cases $\mathrm{II}$ and $\mathrm{III}$ include only the four ridge-type procedures. Panels from top to bottom are for cases $\mathrm{I},\mathrm{II},\mathrm{III}$, respectively.

Under Case $\mathrm{I}$, Trans-RR compares favorably with both lasso baselines, consistent with the non-sparse structure assumed throughout the paper. Among the ridge-type methods, Trans-RR outperforms Pooled-RR for small and moderate h across all three cases. Pooled-RR fits the source and target observations together as if they shared one coefficient, so its error reflects two mismatches: the gap between ${\mathit{\beta }}_0$ and ${\mathit{w}}_0$, and the difference between source and target noise levels. As h decreases, the gap between Pooled-RR and Trans-RR narrows, in line with the theoretical expectation that smaller h indicates greater similarity between the two domains.

More importantly, the comparison with Single-RR reveals the transition between positive and negative transfer. When h is small, Trans-RR achieves the lowest relative error among the ridge-type methods. As h grows, its advantage shrinks and eventually reverses. In our experiments, this turnover occurs near $h=1$. This is consistent with the numerical evidence in Section 3, since transfer is more favorable when the source-stage estimator is closer to the target coefficient. The same qualitative pattern appears in Cases $\mathrm{II}$ and $\mathrm{III}$, where the transition near $h=1$ persists under heavy-tailed errors and heterogeneous designs.

Trans-RR-Ada provides a data-driven safeguard against this negative transfer. Figure 3 shows that it tracks the better of Single-RR and Trans-RR for every value of h. For small h, Trans-RR-Ada essentially coincides with Trans-RR, while for large h it coincides with Single-RR. At the transition $h=1$, where Single-RR and Trans-RR are comparable, Trans-RR-Ada stays close to the better one.

4.4. Robustness to Non-Identity Covariance

The asymptotic theory in Section 3 assumes $\mathrm{cov}({\mathit{x}}_i)={\mathit{I}}_p$. To verify that our methods remain effective under non-identity covariance, we re-run the comparison of Section 4.3 under AR(1) covariance ${\sum }_{jk}={\rho }^{|j-k|}$ with $\rho =0.6$, across all three noise cases $\mathrm{I}$, $\mathrm{II}$, and $\mathrm{III}$. All other settings, including the four ridge-type methods Single-RR, Trans-RR, Trans-RR-Ada, and Pooled-RR and the tuning protocol, match Section 4.3.

Figure 4 reports the resulting boxplots. The qualitative pattern of the i.i.d. comparison persists in all three cases. For small h, Trans-RR achieves the lowest error among the ridge-type methods. As h grows, its advantage shrinks and eventually reverses, with the transition again occurring near $h=1$. Trans-RR-Ada tracks the better of Single-RR and Trans-RR across all values of h. The negative-transfer transition and the safeguard role of Trans-RR-Ada are therefore not specific to the identity–covariance assumption underlying Theorem 1. This suggests that Trans-RR may remain effective under non-identity covariance.

Figure 4. Robustness check under AR(1) correlated predictors. Boxplots of relative estimation errors (log scale) across 500 replications for varying $\parallel {\mathit{\beta }}_0-{\mathit{w}}_0\parallel$ under cases $\mathrm{I}$–$\mathrm{III}$ with ${\mathit{x}}_i\sim N(\mathbf{0},{\sum }_{\rho })$, where ${\sum }_{\rho ,jk}={\rho }^{|j-k|}$ and $\rho =0.6$. Panels from top to bottom are for cases $\mathrm{I}$, $\mathrm{II}$, and $\mathrm{III}$, respectively. The four ridge-type procedures, Single-RR, Trans-RR, Trans-RR-Ada, and Pooled-RR are shown.

4.5. Sensitivity to Tuning Choices

The comparison in Figure 3 fixes the smoothed Huber loss with $(\delta ,\eta )=(1.35,0.1)$ and selects each ridge penalty by $five$-fold cross-validation under the mean absolute error criterion, on a common geometric grid of nine values from $1/9$ to 9. This subsection examines whether the qualitative findings of that comparison are stable under perturbations to four tuning choices: the smoothed Huber parameters $(\delta ,\eta )$, the cross-validation criterion, the robust loss family, and the ridge penalty grid. Each perturbation re-runs the full simulation with $M=500$ replications per setting.

4.5.1. Choice of $(\delta ,\eta )$

The default $\delta =1.35$ is the standard Huber tuning. The smoothing parameter $\eta =0.1$ closely approximates the unsmoothed Huber loss and keeps ${\rho }_{\eta }$ twice continuously differentiable, as required by Assumption 1(b). In the sensitivity experiment, we vary $\delta$ over $\{1.0,1.35,2.0\}$ and $\eta$ over $\{0.05,0.10,0.20\}$, giving nine $(\delta ,\eta )$ pairs, and re-run the full simulation across the three cases and seven discrepancy values h for every pair. Within each $(\mathrm{case},h)$ combination, varying the $(\delta ,\eta )$ pair over the nine settings changes each method’s mean error by less than $7\%$ (median $1.5\%$) of its mean. The four-method ranking matches the default $(1.35,0.10)$ ranking in 171 of the 189 $(\mathrm{case},h,(\delta ,\eta ))$ cells. The qualitative findings are therefore stable under this perturbation. Table 2 displays the four method means as a $(\delta ,\eta )$ heatmap at $h=1$ in each of the three cases. Within each $3\times 3$ grid, the entries vary only slightly and the four-method ordering is the same in every cell, illustrating the two findings stated above. Appendix B reports the heatmaps at the other six values of h, where the same pattern holds.

Table 2. Sensitivity of relative estimation error to the smoothed Huber parameters $(\delta ,\eta )$ at the transition discrepancy $h=1$. Each entry reports Single-RR/Trans-RR/Trans-RR-Ada/Pooled-RR mean estimation error over $M=500$ replications. The three blocks correspond to the three error distributions of Section 4.3.

Case I (Gaussian Errors)
$\delta \setminus \eta$	$0.05$	$0.10$	$0.20$
$1.00$	0.653/0.655/0.651/0.883	0.654/0.656/0.652/0.883	0.656/0.658/0.654/0.883
$1.35$	0.644/0.647/0.642/0.882	0.645/0.647/0.642/0.882	0.646/0.648/0.643/0.882
$2.00$	0.641/0.642/0.637/0.885	0.640/0.642/0.637/0.885	0.641/0.643/0.637/0.885
Case II (Cauchy Errors)
$\delta \setminus \eta$	$0.05$	$0.10$	$0.20$
$1.00$	0.884/0.893/0.884/0.952	0.884/0.892/0.884/0.952	0.884/0.893/0.884/0.952
$1.35$	0.883/0.893/0.884/0.951	0.883/0.892/0.883/0.951	0.883/0.892/0.884/0.951
$2.00$	0.890/0.899/0.891/0.951	0.889/0.899/0.890/0.951	0.889/0.899/0.890/0.951
Case III (Mixture Errors)
$\delta \setminus \eta$	$0.05$	$0.10$	$0.20$
$1.00$	0.779/0.786/0.779/0.921	0.780/0.787/0.779/0.921	0.780/0.787/0.779/0.921
$1.35$	0.781/0.788/0.781/0.920	0.781/0.787/0.780/0.920	0.781/0.787/0.780/0.921
$2.00$	0.794/0.801/0.793/0.924	0.793/0.800/0.792/0.924	0.792/0.800/0.791/0.923

4.5.2. Choice of Cross-Validation Criterion

The default selects each ridge penalty by $five$-fold cross-validation under the mean absolute error criterion. We re-run the simulation with every cross-validation loss changed to mean squared error, keeping the smoothed Huber loss for estimation. Table 3 compares the mean estimation errors under the two criteria across all cases and values of h.

Table 3. Sensitivity of relative estimation error to the cross-validation criterion. All settings are as in Figure 3, except that every cross-validation loss (used to select ${\tau }_1$, $\tau$, ${\tau }_{\mathrm{st}}$, ${\tau }_{\mathrm{p}}$, and $\theta$) is changed from MAE to MSE. Each entry reports the default (MAE-CV) and the MSE-CV mean estimation error, in the format “default/MSE-CV”, over $M=500$ replications.

Case	h	Single-RR	Trans-RR	Trans-RR-Ada	Pooled-RR
$\mathrm{I}$	$0.135$	0.645/0.644	0.550/0.546	0.550/0.546	0.600/0.601
	$0.223$	0.645/0.644	0.565/0.561	0.566/0.561	0.628/0.629
	$0.368$	0.645/0.644	0.588/0.585	0.589/0.585	0.678/0.681
	$0.607$	0.645/0.644	0.616/0.614	0.617/0.615	0.764/0.770
	$1.000$	0.645/0.644	0.647/0.645	0.642/0.641	0.882/0.888
	$1.649$	0.645/0.644	0.791/0.793	0.645/0.644	0.990/0.991
	$2.718$	0.645/0.644	1.639/1.646	0.645/0.644	1.261/1.479
$\mathrm{II}$	$0.135$	0.883/1.772	0.816/2.582	0.819/1.895	0.811/1.324
	$0.223$	0.883/1.772	0.826/2.597	0.829/1.904	0.829/1.326
	$0.368$	0.883/1.772	0.840/2.618	0.843/1.915	0.855/1.353
	$0.607$	0.883/1.772	0.860/2.640	0.862/1.920	0.898/1.401
	$1.000$	0.883/1.772	0.892/2.745	0.883/1.963	0.951/1.497
	$1.649$	0.883/1.772	1.021/2.994	0.887/2.020	1.005/1.679
	$2.718$	0.883/1.772	1.898/3.827	0.883/2.108	1.123/2.223
$\mathrm{III}$	$0.135$	0.781/1.108	0.693/1.319	0.692/1.094	0.706/0.944
	$0.223$	0.781/1.108	0.707/1.334	0.707/1.102	0.729/0.966
	$0.368$	0.781/1.108	0.727/1.358	0.727/1.114	0.770/0.993
	$0.607$	0.781/1.108	0.754/1.395	0.754/1.138	0.834/1.054
	$1.000$	0.781/1.108	0.787/1.473	0.780/1.161	0.920/1.155
	$1.649$	0.781/1.108	0.941/1.749	0.783/1.194	0.998/1.368
	$2.718$	0.781/1.108	1.985/2.646	0.781/1.274	1.181/1.961

Under Case $\mathrm{I}$, the two criteria yield nearly identical mean errors and the same four-method ranking at every h. Under Cases $\mathrm{II}$ and $\mathrm{III}$, the mean errors of all four methods become substantially larger, and two qualitative findings of Section 4.3 no longer hold. First, Trans-RR has a larger mean error than Single-RR at every h in both heavy-tailed cases, so the range on which transfer helps disappears entirely. Second, Trans-RR-Ada no longer approximates min(Single-RR, Trans-RR) in most settings of the heavy-tailed cases. This may result from the heavy tails of Cauchy errors, under which MSE-based cross-validation is more sensitive to extreme residuals than MAE-based cross-validation. MAE-CV is therefore the appropriate default for heavy-tailed errors, and the conclusions of Section 4.3 are specific to this choice.

4.5.3. Choice of Robust Loss

In order to assess the sensitivity to the specific form of the robust loss, we replace the smoothed Huber loss in (6) with the pseudo-Huber loss ${\rho }_P(t;\delta )={\delta }^2\left(\sqrt{1+{(t/\delta )}^2}-1\right)$ at the same $\delta =1.35$. Table 4 compares the mean estimation errors under the two losses across all cases and values of h.

Table 4. Sensitivity of relative estimation error to the choice of robust loss. All settings are as in Figure 3, except that the smoothed Huber loss is replaced by the pseudo-Huber loss ${\rho }_P(t;\delta )={\delta }^2(\sqrt{1+{(t/\delta )}^2}-1)$ with $\delta =1.35$ (the smoothing parameter $\eta$ is no longer needed). Each entry reports the default (smoothed Huber) and the pseudo-Huber mean estimation error, in the format “default/pseudo-Huber”, over $M=500$ replications.

Case	h	Single-RR	Trans-RR	Trans-RR-Ada	Pooled-RR
$\mathrm{I}$	$0.135$	0.645/0.645	0.550/0.545	0.550/0.546	0.600/0.595
	$0.223$	0.645/0.645	0.565/0.561	0.566/0.562	0.628/0.625
	$0.368$	0.645/0.645	0.588/0.585	0.589/0.586	0.678/0.675
	$0.607$	0.645/0.645	0.616/0.615	0.617/0.616	0.764/0.763
	$1.000$	0.645/0.645	0.647/0.647	0.642/0.643	0.882/0.883
	$1.649$	0.645/0.645	0.791/0.797	0.645/0.645	0.990/0.995
	$2.718$	0.645/0.645	1.639/1.642	0.645/0.645	1.261/1.299
$\mathrm{II}$	$0.135$	0.883/0.891	0.816/0.828	0.819/0.830	0.811/0.823
	$0.223$	0.883/0.891	0.826/0.836	0.829/0.839	0.829/0.839
	$0.368$	0.883/0.891	0.840/0.850	0.843/0.852	0.855/0.865
	$0.607$	0.883/0.891	0.860/0.869	0.862/0.872	0.898/0.905
	$1.000$	0.883/0.891	0.892/0.900	0.883/0.891	0.951/0.954
	$1.649$	0.883/0.891	1.021/1.023	0.887/0.895	1.005/1.007
	$2.718$	0.883/0.891	1.898/1.871	0.883/0.891	1.123/1.129
$\mathrm{III}$	$0.135$	0.781/0.793	0.693/0.703	0.692/0.703	0.706/0.713
	$0.223$	0.781/0.793	0.707/0.717	0.707/0.718	0.729/0.738
	$0.368$	0.781/0.793	0.727/0.740	0.727/0.740	0.770/0.777
	$0.607$	0.781/0.793	0.754/0.765	0.754/0.765	0.834/0.842
	$1.000$	0.781/0.793	0.787/0.798	0.780/0.791	0.920/0.925
	$1.649$	0.781/0.793	0.941/0.950	0.783/0.794	0.998/1.002
	$2.718$	0.781/0.793	1.985/1.972	0.781/0.793	1.181/1.199

The mean errors are nearly identical to those under the smoothed Huber loss across all 21 settings. The four-method ranking matches the default in nearly every setting, and the few mismatches involve methods whose mean errors are essentially tied. The negative-transfer transition stays at the same value of h in every case, and Trans-RR-Ada continues to track min(Single-RR, Trans-RR) as closely as under the default. The qualitative findings of Section 4.3 are insensitive to this choice, consistent with Theorem 1.

4.5.4. Choice of Ridge Penalty Grid

The four ridge penalties ${\tau }_1$, $\tau$, ${\tau }_{\mathrm{st}}$, and ${\tau }_{\mathrm{p}}$ are tuned by $five$-fold cross-validation over a common geometric grid of nine values from $1/9$ to 9. We probe the sensitivity to this grid in two complementary ways: extending the grid to wider values, and forcing all four penalties to a common fixed value with no cross-validation.

We first extend the grid to thirteen values from $1/27$ to 27 on the same geometric scale, which contains the default grid as a strict subset. Table 5 compares the mean estimation errors under the default and the wide grid across all cases and values of h. The impact on estimation is negligible: each method’s mean error barely changes across settings, the four-method ranking is unchanged except at settings with near-tied means, the negative-transfer transition does not move, and Trans-RR-Ada continues to track min(Single-RR, Trans-RR) as closely as under the default. The default grid is therefore wide enough on both sides, and the qualitative findings of Section 4.3 do not depend on the choice of upper or lower endpoint.

Table 5. Sensitivity of relative estimation error to the ridge-penalty cross-validation grid. The default grid contains 9 values from $1/9$ to 9 on a geometric scale. The wide grid extends this to 13 values from $1/27$ to 27 on the same geometric scale, and contains the default grid as a strict subset. All other settings are as in Figure 3, with $M=500$ replications per cell. Each entry reports Single-RR/Trans-RR/Trans-RR-Ada/Pooled-RR mean estimation error.

Case	h	Default Grid (9 pts, [1/9, 9])	Wide Grid (13 pts, [1/27, 27])
$\mathrm{I}$	$0.135$	0.645/0.550/0.550/0.600	0.645/0.550/0.550/0.600
$\mathrm{I}$	$0.223$	0.645/0.565/0.566/0.628	0.645/0.565/0.566/0.628
$\mathrm{I}$	$0.368$	0.645/0.588/0.589/0.678	0.645/0.589/0.590/0.678
$\mathrm{I}$	$0.607$	0.645/0.616/0.617/0.764	0.645/0.618/0.619/0.764
$\mathrm{I}$	$1.000$	0.645/0.647/0.642/0.882	0.645/0.648/0.642/0.883
$\mathrm{I}$	$1.649$	0.645/0.791/0.645/0.990	0.645/0.791/0.645/0.990
$\mathrm{I}$	$2.718$	0.645/1.639/0.645/1.261	0.645/1.639/0.645/1.261
$\mathrm{II}$	$0.135$	0.883/0.816/0.819/0.811	0.887/0.824/0.826/0.811
$\mathrm{II}$	$0.223$	0.883/0.826/0.829/0.829	0.887/0.835/0.838/0.829
$\mathrm{II}$	$0.368$	0.883/0.840/0.843/0.855	0.887/0.850/0.852/0.856
$\mathrm{II}$	$0.607$	0.883/0.860/0.862/0.898	0.887/0.869/0.871/0.900
$\mathrm{II}$	$1.000$	0.883/0.892/0.883/0.951	0.887/0.896/0.888/0.955
$\mathrm{II}$	$1.649$	0.883/1.021/0.887/1.005	0.887/1.021/0.891/1.006
$\mathrm{II}$	$2.718$	0.883/1.898/0.883/1.123	0.887/1.899/0.887/1.120
$\mathrm{III}$	$0.135$	0.781/0.693/0.692/0.706	0.782/0.694/0.694/0.706
$\mathrm{III}$	$0.223$	0.781/0.707/0.707/0.729	0.782/0.709/0.709/0.729
$\mathrm{III}$	$0.368$	0.781/0.727/0.727/0.770	0.782/0.731/0.731/0.770
$\mathrm{III}$	$0.607$	0.781/0.754/0.754/0.834	0.782/0.758/0.758/0.834
$\mathrm{III}$	$1.000$	0.781/0.787/0.780/0.920	0.782/0.789/0.781/0.922
$\mathrm{III}$	$1.649$	0.781/0.941/0.783/0.998	0.782/0.941/0.783/0.999
$\mathrm{III}$	$2.718$	0.781/1.985/0.781/1.181	0.782/1.985/0.782/1.180

We next force all four ridge penalties to a common fixed value $\tau \in \{1/3,1,3,9\}$, removing the ridge cross-validation entirely. Trans-RR-Ada’s mixing weight $\theta$ is still selected by $five$-fold cross-validation on the target sample (Algorithm 2). Table 6 reports the mean estimation errors at the four fixed $\tau$ values across all cases and values of h. At every fixed $\tau$, the mean error of Trans-RR grows monotonically with h in all three cases, consistent with the theoretical risk curves of Figure 2. The absolute levels and the four-method ranking do vary substantially across the four $\tau$ values, but the qualitative dependence on h predicted by the theory is preserved at every $\tau$.

Table 6. Sensitivity of relative estimation error to the ridge penalty value when cross-validation tuning is disabled. All four ridge penalties (Single-RR’s ${\tau }_{\mathrm{st}}$, Trans-RR’s ${\tau }_1$ and $\tau$, Pooled-RR’s ${\tau }_{\mathrm{p}}$) are forced to a common fixed value from $\{1/3,1,3,9\}$. Trans-RR-Ada’s mixing weight $\theta$ is still selected by 5-fold cross-validation on the target sample (Algorithm 2). All other settings are as in Figure 3, with $M=500$ replications per cell. Each entry reports Single-RR/Trans-RR/Trans-RR-Ada/Pooled-RR mean estimation error.

Case	h	τ = 1/3	τ = 1	τ = 3	τ = 9
$\mathrm{I}$	$0.135$	0.742/0.769/0.687/0.619	0.628/0.523/0.525/0.605	0.742/0.620/0.620/0.773	0.881/0.814/0.814/0.906
$\mathrm{I}$	$0.223$	0.742/0.781/0.696/0.645	0.628/0.538/0.538/0.625	0.742/0.631/0.632/0.783	0.881/0.820/0.820/0.910
$\mathrm{I}$	$0.368$	0.742/0.804/0.709/0.694	0.628/0.565/0.561/0.661	0.742/0.651/0.652/0.801	0.881/0.829/0.829/0.917
$\mathrm{I}$	$0.607$	0.742/0.851/0.728/0.790	0.628/0.616/0.596/0.726	0.742/0.687/0.690/0.833	0.881/0.846/0.846/0.929
$\mathrm{I}$	$1.000$	0.742/0.953/0.745/0.990	0.628/0.714/0.629/0.848	0.742/0.752/0.740/0.890	0.881/0.876/0.876/0.951
$\mathrm{I}$	$1.649$	0.742/1.185/0.743/1.423	0.628/0.913/0.628/1.074	0.742/0.866/0.742/0.986	0.881/0.924/0.881/0.986
$\mathrm{I}$	$2.718$	0.742/1.739/0.742/2.321	0.628/1.287/0.628/1.436	0.742/1.033/0.742/1.113	0.881/0.987/0.881/1.030
$\mathrm{II}$	$0.135$	2.026/2.443/2.014/1.143	0.978/0.964/0.930/0.789	0.861/0.778/0.784/0.850	0.924/0.877/0.878/0.936
$\mathrm{II}$	$0.223$	2.026/2.459/2.019/1.167	0.978/0.979/0.938/0.804	0.861/0.787/0.793/0.857	0.924/0.881/0.882/0.939
$\mathrm{II}$	$0.368$	2.026/2.490/2.027/1.211	0.978/1.005/0.953/0.833	0.861/0.803/0.810/0.870	0.924/0.888/0.889/0.943
$\mathrm{II}$	$0.607$	2.026/2.551/2.037/1.297	0.978/1.055/0.972/0.885	0.861/0.832/0.836/0.894	0.924/0.900/0.903/0.952
$\mathrm{II}$	$1.000$	2.026/2.681/2.045/1.475	0.978/1.152/0.988/0.983	0.861/0.885/0.865/0.937	0.924/0.923/0.922/0.968
$\mathrm{II}$	$1.649$	2.026/2.971/2.042/1.848	0.978/1.342/0.984/1.161	0.861/0.977/0.864/1.008	0.924/0.959/0.927/0.994
$\mathrm{II}$	$2.718$	2.026/3.615/2.034/2.569	0.978/1.663/0.978/1.420	0.861/1.096/0.861/1.093	0.924/1.000/0.925/1.022
$\mathrm{III}$	$0.135$	1.286/1.442/1.246/0.827	0.784/0.713/0.704/0.686	0.798/0.693/0.694/0.808	0.902/0.844/0.844/0.920
$\mathrm{III}$	$0.223$	1.286/1.456/1.253/0.852	0.784/0.729/0.715/0.705	0.798/0.703/0.704/0.817	0.902/0.849/0.849/0.923
$\mathrm{III}$	$0.368$	1.286/1.484/1.265/0.900	0.784/0.756/0.735/0.737	0.798/0.721/0.724/0.833	0.902/0.857/0.857/0.929
$\mathrm{III}$	$0.607$	1.286/1.541/1.282/0.994	0.784/0.808/0.763/0.797	0.798/0.754/0.759/0.861	0.902/0.872/0.873/0.940
$\mathrm{III}$	$1.000$	1.286/1.661/1.294/1.187	0.784/0.909/0.789/0.910	0.798/0.815/0.799/0.912	0.902/0.898/0.899/0.959
$\mathrm{III}$	$1.649$	1.286/1.933/1.290/1.600	0.784/1.109/0.786/1.115	0.798/0.920/0.799/0.996	0.902/0.941/0.903/0.990
$\mathrm{III}$	$2.718$	1.286/2.559/1.286/2.420	0.784/1.466/0.784/1.426	0.798/1.064/0.798/1.103	0.902/0.993/0.902/1.026

The Trans-RR-Ada safeguard nevertheless remains effective: its mean error stays close to min(Single-RR, Trans-RR) in every one of the 84 settings, just as under cross-validated penalties. The agreement between Trans-RR-Ada and min(Single-RR, Trans-RR) is therefore robust to a misspecified ridge penalty.

5. Real Data Analysis

We evaluate the proposed transfer procedure on the near-infrared (NIR) spectral dataset from the 2002 International Diffuse Reflectance Conference (IDRC) “Shootout” competition. The data consist of pharmaceutical tablet measurements collected by two spectrometers, with ASSAY as the response. For each instrument, the dataset contains a training sample of size 460 and an external test sample of size 155. Each spectrum is recorded at 650 wavelengths, yielding a moderate–dimensional regression problem.

Let X and $X_1$ denote the spectra from the two instruments. We consider two transfer directions. In Direction A, X is the target domain and $X_1$ is the source domain. In Direction B, the roles are reversed. In each repetition, we randomly split the training sample of the target instrument into two parts of sizes 160 and 300. The subset of size 160 is used as the target training sample. The remaining 300 tablets are matched with their measurements from the other instrument to form the source training sample. When X is the target domain, the target fit is constructed from 160 observations in X, and the source fit is constructed from the corresponding 300 observations in $X_1$. The same scheme is applied symmetrically in the reverse direction. The external test sample of the target instrument is used for evaluation. We repeat this procedure 20 times to reduce Monte Carlo variability.

The preprocessing used in both transfer directions has two stages. First, we thin the spectrum by retaining every fourth wavelength of the original 650, yielding $p=163$ predictors. Two considerations motivate this thinning. Adjacent NIR wavelengths record highly overlapping absorption signals, so a four-fold thinning preserves nearly all the spectral information. This is standard practice in NIR chemometrics. A step of four also gives $p=163\approx n=160$, the same $p/n\approx 1$ setting as in our simulations.

Second, for each domain, predictors are whitened using its own training sample,

$$\tilde{X}=(X-\widehat{\mu }){\displaystyle {\widehat{\sum }}^{-1/2}},$$

where $\widehat{\mu }$ and ${\displaystyle \widehat{\sum }}$ are estimated from that training sample, and the same transformation is applied to the associated test sample. Whitening decorrelates the highly collinear NIR predictors so that the sample second-moment matrix of $\tilde{X}$ approximates ${\mathit{I}}_p$, which is the design assumption underlying Theorem 1. The response is centered using the mean of the corresponding training response, and the same centering constant is used for the associated test response. All preprocessing parameters are thus estimated only on training data and carried over to the test set, avoiding information leakage.

We compare six methods: Single-RR, Trans-RR, the adaptive Trans-RR estimator ${\widehat{\mathit{\beta }}}_{\mathrm{ada}}$ from Section 3.3 (denoted Trans-RR-Ada in the table), Pooled-RR, Single-Lasso, and Trans-Lasso. For the ridge-type methods, we use the smoothed Huber loss in (6) with $(\delta ,\eta )=(1.35,0.1)$. All tuning parameters are selected by 5-fold cross-validation with mean absolute error as the validation criterion, from the common grid

$$\mathcal{G}=\{{10}^{-5},{10}^{-4.5},{10}^{-4},\dots ,{10}^1\}.$$

Performance is measured by the test-set RMSE,

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n_{\mathrm{test}}}}{\sum }_{i=1}^{n_{\mathrm{test}}}{(y_i-{\widehat{y}}_i)}^2},$$

where $y_i$ and ${\widehat{y}}_i$ are on the original ASSAY scale (predictions are uncentered before evaluation) and $n_{\mathrm{test}}=155$ in both transfer directions.

Table 7 reports the average RMSE and its standard deviation over the 20 repetitions, and Figure 5 displays the corresponding distributional information. Trans-RR achieves the smallest mean RMSE in both transfer directions, with relatively small variability across the 20 splits. The adaptive variant Trans-RR-Ada matches Trans-RR within $0.03$ in mean RMSE in both directions, indicating that the source and target are close enough for transfer to be uniformly beneficial on this dataset. Pooled-RR ranks third behind the two transfer-ridge methods and is uniformly dominated by Trans-RR, suggesting that naive pooling fails to account for cross-instrument differences. The lasso methods are less competitive overall, especially Single-Lasso. This is consistent with the non-sparse structure of NIR spectra, where relevant information is distributed over broad wavelength regions rather than concentrated on a small subset of predictors.

Table 7. Prediction performance on the NIR spectral dataset over 20 repeated random splits. Each entry reports the average RMSE, with the standard deviation in parentheses.

Method	Direction A	Direction B
Trans-RR	4.6230 (0.1732)	4.7933 (0.2736)
Trans-RR-Ada	4.6294 (0.1861)	4.8211 (0.3757)
Pooled-RR	5.0952 (0.0812)	5.4909 (0.1335)
Trans-Lasso	5.5668 (0.3650)	5.6803 (0.3131)
Single-RR	6.2666 (2.2628)	6.8272 (2.2674)
Single-Lasso	8.0672 (2.8904)	8.5607 (3.3739)

Figure 5. Distribution of RMSE across the 20 repeated random splits for each of the six methods, in the two transfer directions. Boxes show the inter-quartile range, whiskers extend to 1.5 × IQR, and circles mark splits beyond that.

We further checked the procedure for four starting offsets of the every-fourth-wavelength thinning and for the same procedure without whitening. Across all five resulting preprocessing variants and both transfer directions, Trans-RR and Trans-RR-Ada are the top two methods by mean RMSE, and their mean RMSE values differ by at most $0.05$. Mean RMSE for all six methods in each variant and direction is reported in Appendix C. The results, together with these robustness checks, therefore support the effectiveness of the proposed Trans-RR method.

6. Discussion

This paper introduces Trans-RR, a robust transfer-learning approach for moderate–dimensional linear regression. It extends transfer-learning ideas of [7,8] to a setting with non-sparse coefficients and heavy-tailed errors, without relying on sparsity assumptions or moment restrictions on the errors. The theory and the numerical results show that negative transfer can occur when the source study is not sufficiently informative for the target study. To guard against this, we also propose an adaptive aggregation of Trans-RR with the single-task estimator that selects the mixing weight by cross-validation. The theoretical results, simulation studies, and real-data analysis support the effectiveness of both procedures. A natural direction for further study is to identify the choice of loss function that minimizes the asymptotic estimation error in our framework.

Appendix E.1. Preliminaries

Lemma A1. 

Under Assumptions  P3–P4, for any $l=1,\dots ,p$, where $w(l)$ denotes the lth coordinate of a vector $\mathit{w}$,

$$|\widehat{w}(l)-w_0(l)|=O_{L_k}\left(\mathrm{polyLog}(n_1)n_1^{-1/2}+n_1^{-\mathrm{e}}\right)={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n_1)}{n_1^{1/2}\wedge n_1^{\mathrm{e}}}}\right).$$

In particular, if we further assume $n=\mathrm{O}(n_1)$, we have

$$|\widehat{w}(l)-w_0(l)|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{\sqrt{n}\wedge n^e}}\right).$$

Proof. 

By Proposition 3.12 of El Karoui [21],

$$\left|{\mathfrak{w}}_l-w_0(l)\right|=O_{L_k}\left(\mathrm{polyLog}(n_1)n_1^{-1/2}+{\parallel w_0\parallel }_{\infty }\right),$$

where ${\mathfrak{w}}_l$ denotes the analog of ${\mathfrak{b}}_p$ defined in Appendix 4 of El Karoui [21]. Its explicit construction is rather involved and is not needed here; we only use ${\mathfrak{w}}_l$ as an intermediate quantity in the argument. Under Assumption P3, $\parallel w_0{\parallel }_{\infty }=O(n_1^{-\mathrm{e}})$, hence

$$\left|{\mathfrak{w}}_l-w_0(l)\right|=O_{L_k}\left(\mathrm{polyLog}(n_1)n_1^{-1/2}+n_1^{-\mathrm{e}}\right)={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n_1)}{n_1^{1/2}\wedge n_1^{\mathrm{e}}}}\right).$$

(A2)

Moreover, by Theorem 3.20 of El Karoui [21],

$$\widehat{w}(l)-{\mathfrak{w}}_l=O_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n_1)n_1^{\alpha }}{{[n_1^{1/2}\wedge n_1^{\mathrm{e}}]}^2}}\right).$$

(A3)

Let $m:=\min (1/2,\mathrm{e})$. Then ${[n_1^{1/2}\wedge n_1^{\mathrm{e}}]}^2=n_1^{2m}$ and

$${\displaystyle \frac{\mathrm{polyLog}(n_1)n_1^{\alpha }}{{[n_1^{1/2}\wedge n_1^{\mathrm{e}}]}^2}}=\mathrm{polyLog}(n_1)n_1^{-m}{n}_1^{\alpha -m}.$$

Assumption P4 gives $m>\alpha$, thus

$${\displaystyle \frac{\mathrm{polyLog}(n_1)n_1^{\alpha }}{{[n_1^{1/2}\wedge n_1^{\mathrm{e}}]}^2}}=o\left(\mathrm{polyLog}(n_1)n_1^{-m}\right),$$

and hence (A3) yields

$$\widehat{w}(l)-{\mathfrak{w}}_l=o_{L_k}\left(\mathrm{polyLog}(n_1)n_1^{-m}\right).$$

(A4)

By the triangle inequality and Minkowski’s inequality in $L_k$,

$$\parallel \widehat{w}(l)-w_0{(l)\parallel }_{L_k}\le \parallel \widehat{w}(l)-{\mathfrak{w}}_l{\parallel }_{L_k}+{\parallel {\mathfrak{w}}_l-w_0(l)\parallel }_{L_k}.$$

Using (A2) and (A4), we obtain

$$\parallel \widehat{w}(l)-w_0{(l)\parallel }_{L_k}=\mathrm{O}\left({\displaystyle \frac{\mathrm{polyLog}(n_1)}{n_1^{1/2}\wedge n_1^{\mathrm{e}}}}\right),$$

which is equivalent to

$$|\widehat{w}(l)-w_0(l)|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n_1)}{n_1^{1/2}\wedge n_1^{\mathrm{e}}}}\right).$$

Assume $n=\mathrm{O}(n_1)$, i.e., $n_1\ge n/C$ for some $C>0$ and all large n. Since $\mathrm{polyLog}(t)$ denotes a power of $\log t$, write $\mathrm{polyLog}(t)={(\log t)}^q$ for some fixed $q\ge 0$ (up to a constant factor). Consider $f(t):={(\log t)}^q{t}^{-m}$. For $t\ge \exp (q/m)$,

$$f^{\prime }(t)={\displaystyle \frac{{(\log t)}^{q-1}}{t^{m+1}}}\left(q-m\log t\right)\le 0,$$

so f is decreasing for all sufficiently large t. Hence, for all sufficiently large n and all $n_1\ge n/C$,

$${\displaystyle \frac{\mathrm{polyLog}(n_1)}{\sqrt{n_1}\wedge n_1^e}}={\displaystyle \frac{{(\log n_1)}^q}{n_1^m}}=f(n_1)\le f(n/C)={\displaystyle \frac{{(\log (n/C))}^q}{{(n/C)}^m}}\le C^m{\displaystyle \frac{{(\log n)}^q}{n^m}}\asymp {\displaystyle \frac{\mathrm{polyLog}(n)}{\sqrt{n}\wedge n^e}},$$

where we use $a_n\asymp b_n$ to denote that $a_n=O(b_n)$ and $b_n=O(a_n)$. Therefore, we have

$$|\widehat{w}(l)-w_0(l)|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{\sqrt{n}\wedge n^e}}\right).$$

□

Proposition A1. 

Let ${\mathit{\delta }}_1$ and ${\mathit{\delta }}_2$ be the two vectors in ${\mathbb{R}}^p$. Then, when ρ’s are convex and twice-differentiable,

$$\parallel {\mathit{\delta }}_1-{\mathit{\delta }}_2\parallel \le {\displaystyle \frac{1}{\tau }}\parallel f({\mathit{\delta }}_1)-f({\mathit{\delta }}_2)\parallel .$$

(A5)

Proof. 

We have by definition

$$f({\mathit{\delta }}_1)-f({\mathit{\delta }}_2)=\tau ({\mathit{\delta }}_1-{\mathit{\delta }}_2)+{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\mathit{x}}_i[\psi \{{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }({\mathit{\delta }}_0-{\mathit{\delta }}_2)\}-\psi \{{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }({\mathit{\delta }}_0-{\mathit{\delta }}_1)\}].$$

By the mean value theorem, we have

$$\psi \{{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }({\mathit{\delta }}_0-{\mathit{\delta }}_2)\}-\psi \{{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }({\mathit{\delta }}_0-{\mathit{\delta }}_1)\}={\psi }^{\prime }({\gamma }_i^{🟉}){\mathit{x}}_i^{\top }({\mathit{\delta }}_0-{\mathit{\delta }}_1),$$

where ${\gamma }_i^{🟉}$ is in the interval $\left({\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }({\mathit{\delta }}_0-{\mathit{\delta }}_1),{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }({\mathit{\delta }}_0-{\mathit{\delta }}_2)\right)$.

Hence,

$$\begin{array}{ccc}f({\mathit{\delta }}_1)-f({\mathit{\delta }}_2)\hfill & =& \tau ({\mathit{\delta }}_1-{\mathit{\delta }}_2)+{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }({\gamma }_i^{🟉}){\mathit{x}}_i{\mathit{x}}_i^{\top }({\mathit{\delta }}_0-{\mathit{\delta }}_1)\hfill \\ & =& ({\mathit{S}}_{{\mathit{\delta }}_1,{\mathit{\delta }}_2}+\tau {\mathrm{I}}_p)({\mathit{\delta }}_1-{\mathit{\delta }}_2),\hfill \end{array}$$

where

$${\mathit{S}}_{{\mathit{\delta }}_1,{\mathit{\delta }}_2}={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }({\gamma }_i^{🟉}){\mathit{x}}_i{\mathit{x}}_i^{\top }.$$

This shows that

$${\mathit{\delta }}_1-{\mathit{\delta }}_2={({\mathit{S}}_{{\mathit{\delta }}_1,{\mathit{\delta }}_2}+\tau {\mathrm{I}}_p)}^{-1}\{f({\mathit{\delta }}_1)-f({\mathit{\delta }}_2)\}.$$

Since $\rho$ is convex, ${\psi }^{\prime }={\rho }^{″}$ is non-negative and ${\mathit{S}}_{{\mathit{\delta }}_1,{\mathit{\delta }}_2}$ is positive semi-definite. In the semi-definite order, we have ${\mathit{S}}_{{\mathit{\delta }}_1,{\mathit{\delta }}_2}+\tau {\mathrm{I}}_p⪰\tau {\mathrm{I}}_p$. In particular,

$$\parallel {\mathit{\delta }}_1-{\mathit{\delta }}_2\parallel \le {\displaystyle \frac{1}{\tau }}\parallel f({\mathit{\delta }}_1)-f({\mathit{\delta }}_2)\parallel .$$

□

Proposition A1 yields the following lemma.

Lemma A2. 

For any ${\mathit{\delta }}_1$,

$$\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_1\parallel \le {\displaystyle \frac{1}{\tau }}\parallel f({\mathit{\delta }}_1)\parallel .$$

The lemma is a simple consequence of Equation (A5) by definition $f(\widehat{\mathit{\delta }})=0$.

Appendix E.2. On $\parallel \widehat{\mathit{\delta }}\parallel$ and $\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0\parallel$

Lemma A3. 

Define ${\mathit{q}}_n(\mathit{b})=n^{-1}{\sum }_{i=1}^n{\mathit{x}}_i\psi \{{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }\mathit{b}\}$, ${\mathit{q}}_n\in {\mathbb{R}}^p$.

If ${\mathit{D}}_{\psi }$ is the $n\times n$ diagonal matrix with $(i,i)$-entry $\psi \{{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }{\mathit{\delta }}_0\}$,

$$\parallel \widehat{\mathit{\delta }}\parallel \le {\displaystyle \frac{1}{\tau }}\parallel {\mathit{q}}_n({\mathit{\delta }}_0)\parallel ={\displaystyle \frac{1}{\tau }}\sqrt{{\displaystyle \frac{1}{n^2}}{\mathbf{1}}^{\top }{\mathit{D}}_{\psi }\mathit{X}{\mathit{X}}^{\top }{\mathit{D}}_{\psi }\mathbf{1}},$$

and if ${\mathit{D}}_{\psi ({\xi }_i)}$ is the $n\times n$ diagonal matrix with $(i,i)$-entry $\psi ({\widetilde{ϵ}}_i)$,

$$\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0\parallel \le \parallel {\mathit{\delta }}_0\parallel +{\displaystyle \frac{1}{\tau }}\parallel {\mathit{q}}_n(\mathbf{0})\parallel =\parallel {\mathit{\delta }}_0\parallel +{\displaystyle \frac{1}{\tau }}\sqrt{{\displaystyle \frac{1}{n^2}}{\mathbf{1}}^{\top }{\mathit{D}}_{\psi ({\xi }_i)}\mathit{X}{\mathit{X}}^{\top }{\mathit{D}}_{\psi ({\xi }_i)}\mathbf{1}},$$

Also,

$$\parallel {\mathit{q}}_n({\mathit{\delta }}_0){\parallel }^2\le {\displaystyle \frac{{\mathbf{1}}^{\top }{\mathit{D}}_{\psi }^2\mathbf{1}}{n}}{\parallel \mathit{X}{\mathit{X}}^{\top }/n\parallel }_2,$$

where ${\parallel \mathit{A}\parallel }_2$ denotes the largest singular value of the matrix $\mathit{A}$.

Therefore, under Assumptions O1–O6,

$$\begin{array}{ccc}& & \mathrm{E}(\parallel \widehat{\mathit{\delta }}{\parallel }^2)\le {\displaystyle \frac{1}{{\tau }^2}}{\displaystyle \frac{p}{n}}{C}^2\mathrm{polyLog}(n),\hfill \\ & & \mathrm{E}(\parallel \widehat{\mathit{\delta }}{\parallel }^4)\le {\displaystyle \frac{1}{{\tau }^4}}C\mathrm{polyLog}(n).\hfill \end{array}$$

Similarly, for any finite k,

$$\mathrm{E}(\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0{\parallel }_2^k)\le C_k[\parallel {\mathit{\delta }}_0{\parallel }^k+\mathrm{polyLog}(n)/{\tau }^k].$$

In the case $k=2$, we have the more precise bound

$$\mathrm{E}(\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0{\parallel }^2)\le 2[\parallel {\mathit{\delta }}_0{\parallel }^2+{\displaystyle \frac{p/n}{{\tau }^2}}{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left\{{\psi }^2({\widetilde{ϵ}}_i)\right\}].$$

Noting that [21] has shown that $\mathrm{E}(\parallel \widehat{\mathit{w}}-{\mathit{w}}_0\parallel )=\mathrm{O}(1)$, we have $\mathrm{E}(\parallel \widehat{\mathit{\delta }}+\widehat{\mathit{w}}-{\mathit{\delta }}_0-{\mathit{w}}_0\parallel )$ is bounded by $K\mathrm{polyLog}(n)/{\tau }^k$.

Proof. 

Recall that $f(\mathit{\delta })={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n-{\mathit{x}}_i\psi \{{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }({\mathit{\delta }}_0-\mathit{\delta })\}+\tau \mathit{\delta }$. Applying Lemma A2 with ${\mathit{\delta }}_1=\mathbf{0}$ we have

$$\parallel \widehat{\mathit{\delta }}-\mathbf{0}\parallel \le {\displaystyle \frac{1}{\tau }}\parallel f(\mathbf{0})\parallel ={\displaystyle \frac{1}{\tau }}\parallel {\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}-{\mathit{x}}_i\psi \{{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }{\mathit{\delta }}_0\}\parallel ,$$

which gives the first inequality.

Using ${\mathit{\delta }}_1={\mathit{\delta }}_0$ we have

$$\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0\parallel \le {\displaystyle \frac{1}{\tau }}\parallel f({\mathit{\delta }}_0)\parallel ={\displaystyle \frac{1}{\tau }}\parallel {\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}-{\mathit{x}}_i\psi ({\widetilde{ϵ}}_i)+\tau {\mathit{\delta }}_0\parallel ,$$

which gives the second inequality.

We note that under our assumptions, according to Lemma 3.38 from [21], we have

$$\parallel \mathit{X}{\mathit{X}}^{\top }{/n\parallel }_2={\mathrm{O}}_{L_k}(\mathrm{polyLog}(n))$$

and

$${\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^2\{{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }{\mathit{\delta }}_0\}\le {\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\parallel {\psi }^2\parallel }_{\infty }=\mathrm{O}(1),$$

which gives all the results about $L_k$ bounds.

For the last result, we note that

$$\parallel {\mathit{q}}_n{(\mathbf{0})\parallel }^2={\mathit{q}}_n{(\mathbf{0})}^{\top }{\mathit{q}}_n(\mathbf{0})={\displaystyle \frac{1}{n^2}}\sum _{i,j}{\mathit{x}}_i^{\top }{\mathit{x}}_j\psi ({\widetilde{ϵ}}_i)\psi ({\widetilde{ϵ}}_j).$$

It implies that

$$\mathrm{E}\parallel {\mathit{q}}_n{(\mathbf{0})\parallel }^2={\displaystyle \frac{1}{n^2}}{\displaystyle \sum _{i=1}^n}\mathrm{E}(\parallel {\mathit{x}}_i{\parallel }^2)\mathrm{E}\left\{{\psi }^2({\widetilde{ϵ}}_i)\right\}.$$

Because $\mathrm{E}(\parallel {\mathit{x}}_i{\parallel }^2)=p$, we can conclude that

$$\mathrm{E}(\parallel {\mathit{q}}_n(\mathbf{0}){\parallel }^2)={\displaystyle \frac{p}{n}}{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left\{{\psi }^2({\widetilde{ϵ}}_i)\right\}.$$

Together with the bound

$$\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0{\parallel }^2\le 2\parallel {\mathit{\delta }}_0{\parallel }^2+{\displaystyle \frac{2}{{\tau }^2}}{\parallel {\mathit{q}}_n(\mathbf{0})\parallel }^2,$$

it implies the last result about $k=2$. □

Appendix E.3. Leave-One-Observation-Out

In this subsection, we approximate $\widehat{\mathit{\delta }}$ by ${\widehat{\mathit{\delta }}}_{(i)}$ via leave-one-observation-out method.

We consider the situation where we leave the i-th observation, $({\mathit{x}}_i,{ϵ}_i)$, out. By definition,

$${\widehat{\mathit{\delta }}}_{(i)}=\underset{\mathit{\delta }\in {\mathbb{R}}^p}{\mathrm{arg\; min}}{F}_i(\mathit{\delta }),\mathrm{where}{F}_i(\mathit{\delta })={\displaystyle \frac{1}{n}}\sum _{j\ne i}{\rho }_j\{{\widetilde{ϵ}}_j+{\mathit{x}}_j^{\top }{\mathit{\delta }}_0-{\mathit{x}}_j^{\top }\mathit{\delta }\}+{\displaystyle \frac{\tau }{2}}{\parallel \mathit{\delta }\parallel }^2.$$

We call

$$\begin{array}{ccc}{f}_i(\mathit{\delta })\hfill & =& -{\displaystyle \frac{1}{n}}\sum _{j\ne i}{\mathit{x}}_j{\psi }_j\{{\widetilde{ϵ}}_j+{\mathit{x}}_j^{\top }{\mathit{\delta }}_0-{\mathit{x}}_j^{\top }\mathit{\delta }\}+\tau \mathit{\delta }\hfill \\ & =& f(\mathit{\delta })+{\displaystyle \frac{1}{n}}{\mathit{x}}_i\psi \{{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }{\mathit{\delta }}_0-{\mathit{x}}_i^{\top }\mathit{\delta }\}.\hfill \end{array}$$

We have

$$f_i({\widehat{\mathit{\delta }}}_{(i)})=0.$$

We call

$${\tilde{r}}_{j,(i)}={\widetilde{ϵ}}_j-{\mathit{x}}_j^{\top }({\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0)\mathrm{and}{\mathit{S}}_i={\displaystyle \frac{1}{n}}\sum _{j\ne i}{\psi }_j^{\prime }({\tilde{r}}_{j,(i)}){\mathit{x}}_j{\mathit{x}}_j^{\top }.$$

Consider

$${\tilde{\mathit{\delta }}}_i={\widehat{\mathit{\delta }}}_{(i)}+{\displaystyle \frac{1}{n}}{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i\psi \left\{\mathrm{prox}(c_i\rho )({\tilde{r}}_{j,(i)})\right\}\triangleq {\widehat{\mathit{\delta }}}_{(i)}+{\mathit{\eta }}_i,$$

where

$$\begin{array}{ccc}& & c_i={\displaystyle \frac{1}{n}}{\mathit{x}}_i^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i,\hfill \\ & & {\mathit{\eta }}_i={\displaystyle \frac{1}{n}}{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i\psi \left\{\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})\right\}.\hfill \end{array}$$

(A6)

Appendix E.3.1. Deterministic Bounds

Proposition A2. 

We have

$$\parallel \widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i\parallel \le {\displaystyle \frac{1}{\tau }}\parallel {\mathcal{R}}_i\parallel ,$$

where

$${\mathcal{R}}_i={\displaystyle \frac{1}{n}}\sum _{j\ne i}[{\psi }_j^{\prime }\left\{{\gamma }^{★}({\mathit{x}}_j,{\widehat{\mathit{\delta }}}_{(i)},{\mathit{\eta }}_i)\right\}-{\psi }_j^{\prime }({\tilde{r}}_{j,(i)})]{\mathit{x}}_j{\mathit{x}}_j^{\top }{\mathit{\eta }}_i,$$

and ${\gamma }^{★}({\mathit{x}}_j,{\widehat{\mathit{\delta }}}_{(i)},{\mathit{\eta }}_i)$ is in the (“unordered”) interval $({\tilde{r}}_{j,(i)},{\tilde{r}}_{j,(i)}-{\mathit{x}}_j^{\top }{\mathit{\eta }}_i)$.

Proof. 

Recall that $y_i={ϵ}_i+{\mathit{x}}_i^{\top }{\mathit{w}}_0+{\mathit{x}}_i^{\top }{\mathit{\delta }}_0$.

Since $f_i({\widehat{\mathit{\delta }}}_{(i)})=0$, and ${\tilde{\mathit{\delta }}}_i={\widehat{\mathit{\delta }}}_{(i)}+{\mathit{\eta }}_i$, we have

$$\begin{array}{ccc}f({\tilde{\mathit{\delta }}}_i)\hfill & =& f({\tilde{\mathit{\delta }}}_i)-f_i({\widehat{\mathit{\delta }}}_{(i)})\hfill \\ & =& -{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\mathit{x}}_i\psi \{{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }{\mathit{\delta }}_0-{\mathit{x}}_i^{\top }{\tilde{\mathit{\delta }}}_i\}+\tau {\tilde{\mathit{\delta }}}_i+{\displaystyle \frac{1}{n}}\sum _{j\ne i}{\mathit{x}}_j\psi \{{\widetilde{ϵ}}_j+{\mathit{x}}_j^{\top }{\mathit{\delta }}_0-{\mathit{x}}_j^{\top }{\mathit{\delta }}_{(i)}\}-\tau {\mathit{\delta }}_{(i)}\hfill \\ & =& -{\displaystyle \frac{1}{n}}{\mathit{x}}_i\psi \{{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }{\mathit{\delta }}_0-{\mathit{x}}_i^{\top }{\tilde{\mathit{\delta }}}_i\}+\tau {\mathit{\eta }}_i\hfill \\ & & +{\displaystyle \frac{1}{n}}\sum _{j\ne i}{\mathit{x}}_j\left[{\psi }_j\{{\widetilde{ϵ}}_j+{\mathit{x}}_j^{\top }{\mathit{\delta }}_0-{\mathit{x}}_j^{\top }{\widehat{\mathit{\delta }}}_{(i)}\}-{\psi }_j\{{\widetilde{ϵ}}_j+{\mathit{x}}_j^{\top }{\mathit{\delta }}_0-{\mathit{x}}_j^{\top }({\widehat{\mathit{\delta }}}_{(i)}+{\mathit{\eta }}_i)\}\right].\hfill \end{array}$$

By the mean value theorem, we have

$$\begin{array}{ccc}& & {\psi }_j\{{\widetilde{ϵ}}_j+{\mathit{x}}_j^{\top }{\mathit{\delta }}_0-{\mathit{x}}_j^{\top }{\widehat{\mathit{\delta }}}_{(i)}\}-{\psi }_j\{{\widetilde{ϵ}}_j+{\mathit{x}}_j^{\top }{\mathit{\delta }}_0-{\mathit{x}}_j^{\top }({\widehat{\mathit{\delta }}}_{(i)}+{\mathit{\eta }}_i)\}\hfill \\ & & ={\psi }_j^{\prime }({\tilde{r}}_{j,(i)}){\mathit{x}}_j^{\top }{\mathit{\eta }}_i+[{\psi }_j^{\prime }\left\{{\gamma }^{★}({\mathit{x}}_j,{\widehat{\mathit{\delta }}}_{(i)},{\mathit{\eta }}_i)\right\}-{\psi }_j^{\prime }({\tilde{r}}_{j,(i)})]{\mathit{x}}_j^{\top }{\mathit{\eta }}_i,\hfill \end{array}$$

where ${\gamma }^{★}({\mathit{x}}_j,{\widehat{\mathit{\delta }}}_{(i)},{\mathit{\eta }}_i)$ is in the (“unordered”) interval $({\tilde{r}}_{j,(i)},{\tilde{r}}_{j,(i)}-{\mathit{x}}_j^{\top }{\mathit{\eta }}_i)$.

Therefore, if ${\mathcal{R}}_i$ is defined as above, we have

$$\begin{array}{cc}\hfill & {\displaystyle \frac{1}{n}}\sum _{j\ne i}{\mathit{x}}_j\left[{\psi }_j\{{\widetilde{ϵ}}_j+{\mathit{x}}_j^{\top }{\mathit{\delta }}_0-{\mathit{x}}_j^{\top }{\widehat{\mathit{\delta }}}_{(i)}\}-{\psi }_j\{{\widetilde{ϵ}}_j+{\mathit{x}}_j^{\top }{\mathit{\delta }}_0-{\mathit{x}}_j^{\top }({\widehat{\mathit{\delta }}}_{(i)}+{\mathit{\eta }}_i)\}\right]\hfill \\ =\hfill & {\displaystyle \frac{1}{n}}\sum _{j\ne i}{\psi }_j^{\prime }({\tilde{r}}_{j,(i)}){\mathit{x}}_j{\mathit{x}}_j^{\top }{\mathit{\eta }}_i+{\mathcal{R}}_i\hfill \\ =\hfill & {\mathit{S}}_i{\mathit{\eta }}_i+{\mathcal{R}}_i.\hfill \end{array}$$

The previous simplicities yield that

$$f({\tilde{\mathit{\delta }}}_i)=-{\displaystyle \frac{1}{n}}{\mathit{x}}_i\psi \{{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }{\mathit{\delta }}_0-{\mathit{x}}_i^{\top }{\tilde{\mathit{\delta }}}_i\}+({\mathit{S}}_i+\tau \mathrm{I}){\mathit{\eta }}_i+{\mathcal{R}}_i.$$

Since by definition, ${\mathit{\eta }}_i=n^{-1}{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i\psi \left\{\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})\right\}$, we have

$$({\mathit{S}}_i+\tau \mathrm{I}){\mathit{\eta }}_i={\displaystyle \frac{1}{n}}{\mathit{x}}_i\psi \left\{\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})\right\}.$$

Also, by definition we have

$${\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }{\mathit{\delta }}_0-{\mathit{x}}_i^{\top }{\tilde{\mathit{\delta }}}_i={\tilde{r}}_{i,(i)}-c_i\psi \left\{\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})\right\}.$$

When $\rho$ is differentiable, $x-c\psi \{\mathrm{prox}(c\rho )(x)\}=\mathrm{prox}(c\rho )(x)$. Therefore, ${\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }{\mathit{\delta }}_0-{\mathit{x}}_i^{\top }{\tilde{\mathit{\delta }}}_i=\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})$ and

$$-{\displaystyle \frac{1}{n}}{\mathit{x}}_i\psi \{{\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }{\mathit{\delta }}_0-{\mathit{x}}_i^{\top }{\tilde{\mathit{\delta }}}_i\}+({\mathit{S}}_i+\tau \mathrm{I}){\mathit{\eta }}_i=-{\displaystyle \frac{1}{n}}{\mathit{x}}_i\left[\psi \left\{\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})\right\}-\psi ({\tilde{r}}_{i,(i)})\right]=0.$$

Therefore, $f({\tilde{\mathit{\delta }}}_i)={\mathcal{R}}_i$. Applying Lemma A2 we have

$$\parallel \widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i\parallel \le {\displaystyle \frac{1}{\tau }}\parallel {\mathcal{R}}_i\parallel .$$

□

i.

On ${\mathcal{R}}_i$

Next, we provide a bound for ${\mathcal{R}}_i$.

Lemma A4. 

We have

$$\parallel {\mathit{\eta }}_i\parallel \le {\displaystyle \frac{1}{\sqrt{n}\tau }}{\displaystyle \frac{\parallel {\mathit{x}}_i\parallel }{\sqrt{n}}}|\psi ({\tilde{r}}_{i,(i)})|,$$

and

$$\parallel {\mathcal{R}}_i\parallel \le \parallel \widehat{\sum }{\parallel }_2\underset{j\ne i}{\sup }|{\psi }_j^{\prime }\left\{{\gamma }^{★}({\mathit{x}}_j,{\widehat{\mathit{\delta }}}_{(i)},{\mathit{\eta }}_i)\right\}-{\psi }_j^{\prime }({\tilde{r}}_{j,(i)})|{\displaystyle \frac{1}{\sqrt{n}\tau }}{\displaystyle \frac{\parallel {\mathit{x}}_i\parallel }{\sqrt{n}}}|\psi ({\tilde{r}}_{i,(i)})|.$$

Proof. 

We have

$${\mathcal{R}}_i={\displaystyle \frac{1}{n}}\sum _{j\ne i}[{\psi }_j^{\prime }\left\{{\gamma }^{★}({\mathit{x}}_j,{\widehat{\mathit{\delta }}}_{(i)},{\mathit{\eta }}_i)\right\}-{\psi }_j^{\prime }({\tilde{r}}_{j,(i)})]{\mathit{x}}_j{\mathit{x}}_j^{\top }{\mathit{\eta }}_i.$$

Note that $\mathcal{S}=n^{-1}{\sum }_{j\ne i}[{\psi }_j^{\prime }\left\{{\gamma }^{★}({\mathit{x}}_j,{\widehat{\mathit{\delta }}}_{(i)},{\mathit{\eta }}_i)\right\}-{\psi }_j^{\prime }({\tilde{r}}_{j,(i)})]{\mathit{x}}_j{\mathit{x}}_j^{\top }$ can be written as $\mathcal{S}=n^{-1}{\mathit{X}}^{\top }\mathit{D}\mathit{X}$, where $\mathit{D}$ is a diagonal matrix with $(j,j)$-entry $[{\psi }_j^{\prime }\left\{{\gamma }^{★}({\mathit{x}}_j,{\widehat{\mathit{\delta }}}_{(i)},{\mathit{\eta }}_i)\right\}-{\psi }_j^{\prime }({\tilde{r}}_{j,(i)})]$.

Using the property of matrix norm ${\parallel ·\parallel }_2$, we have ${\parallel \mathcal{S}\parallel }_2\le \parallel {\displaystyle \widehat{\sum }}{\parallel }_2{\parallel \mathit{D}\parallel }_2$, which implies that

$$\parallel {\mathcal{R}}_i\parallel \le \parallel \widehat{\sum }{\parallel }_2\underset{j\ne i}{\sup }|{\psi }_j^{\prime }\left\{{\gamma }^{★}({\mathit{x}}_j,{\widehat{\mathit{\delta }}}_{(i)},{\mathit{\eta }}_i)\right\}-{\psi }_j^{\prime }({\tilde{r}}_{j,(i)})|\parallel {\mathit{\eta }}_i\parallel ,$$

where ${\displaystyle \widehat{\sum }}=n^{-1}{\sum }_{j=1}^n{\mathit{x}}_j{\mathit{x}}_j^{\top }$ is the sample covariance matrix.

We now bound $\parallel {\mathit{\eta }}_i\parallel$. Note that

$$\parallel {\mathit{\eta }}_i\parallel \le {\displaystyle \frac{1}{\sqrt{n}\tau }}{\displaystyle \frac{\parallel {\mathit{x}}_i\parallel }{\sqrt{n}}}|\psi \left\{\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})\right\}|.$$

Using Lemma A-1 in [25], we see that

$$|\psi ({\mathrm{prox}}_{c_i}(\rho )({\tilde{r}}_{i,(i)}))|\le |\psi ({\tilde{r}}_{i,(i)})|.$$

□

Under our assumptions, we have

$$|\psi ({\tilde{r}}_{i,(i)}){|\le \parallel \psi \parallel }_{\infty }\le C\mathrm{polyLog}(n)$$

and later in the proof of Lemma A6, we will show that

$$\underset{i}{\sup }{\displaystyle \frac{\parallel {\mathit{x}}_i\parallel }{\sqrt{n}}}=\underset{i}{\sup }{\displaystyle \frac{|{\lambda }_i|\parallel {\mathcal{X}}_i\parallel }{\sqrt{n}}}={\mathrm{O}}_{L_k}(\underset{i}{\sup }|{\lambda }_i|).$$

ii.

On ${\gamma }^{★}({\mathit{x}}_j,{\widehat{\mathit{\delta }}}_{(i)},{\mathit{\eta }}_i)$ and related quantities

We now show how to control $n^{-1/2}{\sup }_{j\ne i}|{\psi }_j^{\prime }\left\{{\gamma }^{★}({\mathit{x}}_j,{\widehat{\mathit{\delta }}}_{(i)},{\mathit{\eta }}_i)\right\}-{\psi }_j^{\prime }({\tilde{r}}_{j,(i)})|$.

Lemma A5. 

Suppose that ${\psi }^{\prime }$ is $\mathrm{L}(n)$-Lipschitz. Then

$$\underset{j\ne i}{\sup }|{\psi }_j^{\prime }\left\{{\gamma }^{★}({\mathit{x}}_j,{\widehat{\mathit{\delta }}}_{(i)},{\mathit{\eta }}_i)\right\}-{\psi }_j^{\prime }({\tilde{r}}_{j,(i)})|\le \mathrm{L}(n)\underset{j\ne i}{\sup }|{\mathit{x}}_j^{\top }{\mathit{\eta }}_i|.$$

Proof. 

By definition, we have

$$|{\gamma }^{★}({\mathit{x}}_j,{\widehat{\mathit{\delta }}}_{(i)},{\mathit{\eta }}_i)-{\tilde{r}}_{j,(i)}|\le |{\mathit{x}}_j^{\top }{\mathit{\eta }}_i|.$$

Therefore, the bound follows, using the fact that ${\psi }_j^{\prime }$ is $\mathrm{L}(n)$-Lipschitz. □

Appendix E.3.2. Stochastic Aspects

Recall that

$${\mathit{x}}_j^{\top }{\mathit{\eta }}_i=\psi \left\{\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})\right\}{\displaystyle \frac{1}{n}}{\mathit{x}}_j^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i.$$

Therefore, we can bound $\parallel {\mathcal{R}}_i\parallel$ by

$$\parallel {\mathcal{R}}_i\parallel \le \left[\underset{j\ne i}{\sup }{\displaystyle \frac{|{\mathit{x}}_j^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i|}{n}}\right]{\displaystyle \frac{\mathrm{L}(n)}{\sqrt{n}\tau }}{\displaystyle \frac{\parallel {\mathit{x}}_i\parallel }{\sqrt{n}}}\parallel \widehat{\sum }{\parallel }_2{\parallel \psi \parallel }_{\infty }^2.$$

i.

On ${\sup }_{j\ne i}|{\mathit{x}}_j^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i|$

Now we control ${\mathit{x}}_j^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i/n$.

Lemma A6. 

Suppose ${\mathit{x}}_i$’s are independent and satisfy  O4; suppose that ${\lambda }_i$’s satisfy  O6. Then

$$\underset{j\ne i}{\sup }|{\mathit{x}}_j^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i/n|\le {\displaystyle \frac{1}{\sqrt{n}}}\underset{j\ne i}{\sup }{\displaystyle \frac{\parallel {\mathcal{X}}_j\parallel }{\tau \sqrt{n}}}\mathrm{polyLog}(n)$$

in $L_k$, for any finite k. Note that under AssumptionO4, for any finite k,

$$\underset{j\ne i}{\sup }|\parallel {\mathcal{X}}_j\parallel /\sqrt{n}|={\mathrm{O}}_{L_k}(1).$$

Proof. 

Note that

$$|{\mathit{x}}_j^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i/n|=|{\lambda }_i{\lambda }_j||{\mathcal{X}}_j^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathcal{X}}_i/n|.$$

Denote ${\mathcal{X}}_{(i)}=({\mathcal{X}}_1,\dots ,{\mathcal{X}}_{i-1},{\mathcal{X}}_{i+1},\dots ,{\mathcal{X}}_n)$, and ${\mathit{v}}_{j,(i)}={({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathcal{X}}_j$. Then we have the map $F_j({\mathcal{X}}_i)={\mathcal{X}}_j^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathcal{X}}_i={\mathcal{X}}_i^{\top }{\mathit{v}}_{j,(i)}$ is linear in ${\mathcal{X}}_i$ and it is Lipschitz with Lipschitz constant ${\left\{{\mathcal{X}}_j^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-2}{\mathcal{X}}_j\right\}}^{1/2}\le \parallel {\mathcal{X}}_j\parallel /\tau$. Therefore, using Lemma B-2 in [25] and the fact that ${\mathcal{X}}_i$ has mean 0, we have

$$\underset{j\ne i}{\sup }|{\mathcal{X}}_j^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathcal{X}}_i/n||{\mathcal{X}}_{(i)}\le {\displaystyle \frac{1}{\sqrt{n}}}\underset{j\ne i}{\sup }{\displaystyle \frac{\parallel {\mathcal{X}}_j\parallel }{\tau \sqrt{n}}}\mathrm{polyLog}(n)/{\mathrm{c}}_n^{1/2}$$

in $L_k$, when ${\sup }_{j\ne i}\parallel {\mathcal{X}}_j\parallel /(\tau n^{1/2})={\mathrm{O}}_{L_k}(1)$.

To prove it, Using the fact that ${\mathcal{X}}_j\to \parallel {\mathcal{X}}_j\parallel /n^{1/2}$ is $n^{-1/2}$-Lipschitz, we have

$$\underset{j\ne i}{\sup }|\parallel {\mathcal{X}}_j\parallel /\sqrt{n}-m_{\parallel {\mathcal{X}}_j\parallel /\sqrt{n}}|\le \mathrm{polyLog}(n)/\sqrt{n{\mathrm{c}}_n}\mathrm{in}\sqrt{L_{2k}}.$$

Note that $\mathrm{E}(\parallel {\mathcal{X}}_j\parallel )\le \{\mathrm{E}(\parallel {\mathcal{X}}_j{\parallel }^2{)\}}^{1/2}=p^{1/2}$, so $m_{\parallel {\mathcal{X}}_j\parallel /\sqrt{n}}$ is of order 1. Therefore, by Assumption O4 on ${\mathrm{c}}_n$ we have

$$\underset{j\ne i}{\sup }|\parallel {\mathcal{X}}_j\parallel /\sqrt{n}|={\mathrm{O}}_{L_k}(1).$$

Now our Assumption $\mathit{O}\mathbf{6}$ concerning ${\sup }_i|{\lambda }_i|={\mathrm{O}}_{L_k}(\mathrm{polyLog}(n))$ guarantee that the bounds we announced are valid. □

• Consequences

We have the following result.

Proposition A3. 

Under Assumptions  O1–O6, we have

$$\parallel {\mathcal{R}}_i\parallel ={\mathrm{O}}_{L_k}\left({\displaystyle \frac{{\left[\mathrm{L}(n)\right]\parallel \psi \parallel }_{\infty }^2}{n\tau }}\mathrm{polyLog}(n)\right).$$

Furthermore, the same bound hold for ${\sup }_{1\le i\le n}\parallel {\mathcal{R}}_i\parallel$.

Proof. 

By aggregating all the intermediate results, using Holder’s inequality and the fact that $\parallel {\displaystyle \widehat{\sum }}{\parallel }_2={\mathrm{O}}_{L_k}(\mathrm{polyLog}(n))$ shown in Lemma 3.38 from [21], we finish the proof. □

We can now prove and state the following result. Recall that

$${\tilde{\mathit{\delta }}}_i={\widehat{\mathit{\delta }}}_{(i)}+{\displaystyle \frac{1}{n}}{({\mathit{S}}_i+\tau {\mathrm{I}}_p)}^{-1}{\mathit{x}}_i\psi \left\{\mathrm{prox}(c_i\rho )({\tilde{r}}_{j,(i)})\right\}\triangleq {\widehat{\mathit{\delta }}}_{(i)}+{\mathit{\eta }}_i.$$

Theorem A1. 

Under Assumptions  O1–O7, we have, for any fixed k, when τ is held fixed and $\mathrm{L}(n)\le C{n}^{\alpha }$,

$$\underset{1\le i\le n}{\sup }\parallel \widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i\parallel ={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{n^{1-\alpha }}}\right).$$

In particular, we have

$$\forall 1\le i\le n,\mathrm{E}(\parallel \widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i{\parallel }^2)=\mathrm{O}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{n^{2-2\alpha }}}\right).$$

Also,

$$\underset{1\le i\le n}{\sup }\underset{j\ne i}{\sup }|{\tilde{r}}_{j,(i)}-R_j|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{n^{1/2-\alpha }}}\right).$$

Finally,

$$\underset{1\le i\le n}{\sup }|R_i-\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{n^{1/2-\alpha }}}\right).$$

(A7)

Proof. 

The first two results are direct consequences of the previous propositions.

The third result follows from that

$$\begin{array}{cc}\underset{j\ne i}{\sup }|{\tilde{r}}_{j,(i)}-R_j|\hfill & =\underset{j\ne i}{\sup }|{\mathit{x}}_j^{\top }(\widehat{\mathit{\delta }}-{\widehat{\mathit{\delta }}}_{(i)})|\le \underset{j\ne i}{\sup }|{\mathit{x}}_j^{\top }(\widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i)|+\underset{j\ne i}{\sup }|{\mathit{x}}_j^{\top }({\tilde{\mathit{\delta }}}_i-{\widehat{\mathit{\delta }}}_{(i)})|\hfill \\ \hfill & =\left(\underset{1\le j\le n}{\sup }{\displaystyle \frac{\parallel {\mathit{x}}_j\parallel }{\sqrt{n}}}\right)\sqrt{n}\parallel \widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i\parallel +\underset{j\ne i}{\sup }|{\mathit{x}}_j^{\top }{\mathit{\eta }}_i|,\hfill \end{array}$$

and the fact that ${\sup }_{1\le j\le n}\parallel {\mathit{x}}_j\parallel /n^{1/2}={\mathrm{O}}_{L_k}(\mathrm{polyLog}(n))$ under our assumptions. Control of the first term follows from the results on $\parallel \widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i\parallel$. Control of the second term follows from Lemma A6 and the assumption that $\psi$ is bounded by $C\mathrm{polyLog}(n)$.

For the last result, recall that

$$R_i={\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }({\mathit{\delta }}_0-\widehat{\mathit{\delta }})={\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }{\mathit{\delta }}_0-{\mathit{x}}_i^{\top }{\tilde{\mathit{\delta }}}_i-{\mathit{x}}_i^{\top }(\widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i).$$

Given the definition of ${\tilde{\mathit{\delta }}}_i$, we have

$${\mathit{x}}_i^{\top }{\tilde{\mathit{\delta }}}_i={\mathit{x}}_i^{\top }{\widehat{\mathit{\delta }}}_{(i)}+c_i\psi \left\{\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})\right\}.$$

By the property of the proximal operator, if $y=\mathrm{prox}(c\rho )(x)$, $y+c\psi (y)=x$, we have

$${\widetilde{ϵ}}_i+{\mathit{x}}_i^{\top }{\mathit{\delta }}_0-{\mathit{x}}_i^{\top }{\tilde{\mathit{\delta }}}_i={\tilde{r}}_{i,(i)}-c_i\psi \left[\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})\right]=\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)}).$$

Therefore, we have

$$\underset{i}{\sup }|R_i-\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})|=\underset{i}{\sup }|{\mathit{x}}_i^{\top }(\widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i)|,$$

which is controlled in the previous results. □

• On the limiting variance of $\parallel \widehat{\mathit{\delta }}{\parallel }^2$ and $\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0{\parallel }^2$

Proposition A4. 

Under Assumptions  O1–O7,

$$\mathrm{var}(\parallel \widehat{\mathit{\delta }}{\parallel }^2)\to 0\mathrm{as}n\to \infty .$$

Therefore, $\parallel \widehat{\mathit{\delta }}{\parallel }^2$ has a deterministic equivalent in probability and in $L_2$.

More precisely, we have

$$\mathrm{var}(\parallel \widehat{\mathit{\delta }}{\parallel }^2)=\mathrm{O}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{n^{1-2\alpha }}}\right).$$

The same type of results are true for $\mathrm{var}(\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0{\parallel }^2)$ and $\mathrm{var}(\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0{\parallel }^2)$ provided that $\parallel {\mathit{\delta }}_0\parallel =\mathrm{O}(\mathrm{polyLog}(n))$.

Proof. 

We use the Efron–Stein inequality [35]: if W is a function of n independent random variables, and $W_{(i)}$ is any function of all those random variables except the i-th,

$$\mathrm{var}(W)\le {\displaystyle \sum _{i=1}^n}\mathrm{var}(W-W_{(i)})\le {\displaystyle \sum _{i=1}^n}\mathrm{E}({(W-W_{(i)})}^2).$$

We apply this inequality to $W=\parallel \widehat{\mathit{\delta }}{\parallel }^2$ and $W_{(i)}={\parallel {\widehat{\mathit{\delta }}}_{(i)}\parallel }^2$. We first note that

$$\mathrm{E}(|\parallel \widehat{\mathit{\delta }}{\parallel }^2-\parallel {\widehat{\mathit{\delta }}}_{(i)}{\parallel }^2{|}^2)=2\left[\mathrm{E}\left(|\parallel \widehat{\mathit{\delta }}{\parallel }^2-\parallel {\tilde{\mathit{\delta }}}_i{{\parallel }^2|}^2\right)+\mathrm{E}\left(|\parallel {\tilde{\mathit{\delta }}}_i{\parallel }^2-\parallel {\widehat{\mathit{\delta }}}_{(i)}{{\parallel }^2|}^2\right)\right].$$

For the first term, we have $|\parallel \widehat{\mathit{\delta }}{\parallel }^2-\parallel {\tilde{\mathit{\delta }}}_i{{\parallel }^2|}^2={\left[{(\widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i)}^{\top }(\widehat{\mathit{\delta }}+{\tilde{\mathit{\delta }}}_i)\right]}^2$ and ${(\widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i)}^{\top }(\widehat{\mathit{\delta }}+{\tilde{\mathit{\delta }}}_i)=2{(\widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i)}^{\top }\widehat{\mathit{\delta }}-{\parallel \widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i\parallel }^2$. Therefore, by the Cauchy–Schwarz inequality, we have

$$|\parallel \widehat{\mathit{\delta }}{\parallel }^2-\parallel {\tilde{\mathit{\delta }}}_i{\parallel }^2{|}^2={\mathrm{O}}_{L_1}(\parallel \widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i{\parallel }^4)+\sqrt{{\mathrm{O}}_{L_1}(\mathrm{polyLog}(n)){\parallel \widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i\parallel }^4},$$

since $\mathrm{E}(\parallel \widehat{\mathit{\delta }}{\parallel }^k)$ exists and is bounded by $k\mathrm{polyLog}(n)/{\tau }^k$.

Using the results of Theorem A1 we have

$$\mathrm{E}\left(|\parallel \widehat{\mathit{\delta }}{\parallel }^2-\parallel {\tilde{\mathit{\delta }}}_i{{\parallel }^2|}^2\right)=\mathrm{O}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{n^{2-2\alpha }}}\right)=\mathrm{o}(n^{-1}),$$

provided that $\alpha <1/2$.

For the second term, by definition, we have

$$\begin{array}{cc}\parallel {\tilde{\mathit{\delta }}}_i{\parallel }^2-{\parallel {\widehat{\mathit{\delta }}}_{(i)}\parallel }^2=\hfill & {\displaystyle \frac{2}{n}}{\widehat{\mathit{\delta }}}_{(i)}^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i\psi (\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)}))\hfill \\ \hfill & +{\displaystyle \frac{1}{n^2}}{\mathit{x}}_i^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-2}{\mathit{x}}_i{\psi }^2(\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})).\hfill \end{array}$$

Since $\parallel {\widehat{\mathit{\delta }}}_{(i)}{\parallel }^2$ and ${\mathit{S}}_i$ are independent of ${\mathit{x}}_i$, and $\parallel {({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\parallel }_2\le {\tau }^{-1}$, we have ${\widehat{\mathit{\delta }}}_{(i)}^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i={\mathrm{O}}_{L_2}(|{\lambda }_i|\parallel {\widehat{\mathit{\delta }}}_{(i)}\parallel /{\mathrm{c}}_n^{1/2})$. Recall also that ${\sup }_i{\parallel \psi \parallel }_{\infty }=\mathrm{O}(\mathrm{polyLog}(n))$. Therefore, both terms are of order ${\mathrm{O}}_{L_2}(\mathrm{polyLog}(n)/n{\mathrm{c}}_n^{1/2})$.

We can now conclude that

$$\mathrm{E}\left(|\parallel {\tilde{\mathit{\delta }}}_i{\parallel }^2-\parallel {\widehat{\mathit{\delta }}}_{(i)}{{\parallel }^2|}^2\right)=\mathrm{O}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{n^2}}\right).$$

Therefore, we have

$$\mathrm{var}(\parallel \widehat{\mathit{\delta }}{\parallel }^2)=\mathrm{O}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{n^{1-2\alpha }}}\right)=\mathrm{o}(1).$$

This shows that $\parallel \widehat{\mathit{\delta }}{\parallel }^2$ has a deterministic equivalent in probability and in $L_2$.

For the second part of the result, similarly, we write that

$$\mathrm{E}(|\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0{\parallel }^2-\parallel {\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0{\parallel }^2{|}^2)=2\left[\mathrm{E}\left(|\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0{\parallel }^2-\parallel {\tilde{\mathit{\delta }}}_i-{\mathit{\delta }}_0{{\parallel }^2|}^2\right)+\mathrm{E}\left(|\parallel {\tilde{\mathit{\delta }}}_i-{\mathit{\delta }}_0{\parallel }^2-\parallel {\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0{{\parallel }^2|}^2\right)\right].$$

Using the fact that $|\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0{\parallel }^2-\parallel {\tilde{\mathit{\delta }}}_i-{\mathit{\delta }}_0{{\parallel }^2|}^2={\left[{(\widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i)}^{\top }(\widehat{\mathit{\delta }}+{\tilde{\mathit{\delta }}}_i-2{\mathit{\delta }}_0)\right]}^2$ and ${(\widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i)}^{\top }(\widehat{\mathit{\delta }}+{\tilde{\mathit{\delta }}}_i-2{\mathit{\delta }}_0)=2{(\widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i)}^{\top }(\widehat{\mathit{\delta }}-{\mathit{\delta }}_0)-{\parallel \widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i\parallel }^2$, by the Cauchy–Schwarz inequality we have

$$|\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0{\parallel }^2-\parallel {\tilde{\mathit{\delta }}}_i-{\mathit{\delta }}_0{\parallel }^2{|}^2={\mathrm{O}}_{L_1}(\parallel \widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i{\parallel }^4)+\sqrt{{\mathrm{O}}_{L_1}(\mathrm{polyLog}(n)){\parallel \widehat{\mathit{\delta }}-{\tilde{\mathit{\delta }}}_i\parallel }^4},$$

since $\mathrm{E}(\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0{\parallel }^k)$ exists and is bounded by $k\mathrm{polyLog}(n)/{\tau }^k$ following from Assumption O7 and Lemma A3.

Using the results of Theorem A1 we have

$$\mathrm{E}\left(|\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0{\parallel }^2-\parallel {\tilde{\mathit{\delta }}}_i-{\mathit{\delta }}_0{{\parallel }^2|}^2\right)=\mathrm{O}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{n^{2-2\alpha }}}\right)=\mathrm{o}(n^{-1}),$$

provided that $\alpha <1/2$.

Similarly, by definition we have

$$\begin{array}{cc}\parallel {\tilde{\mathit{\delta }}}_i-{\mathit{\delta }}_0{\parallel }^2-{\parallel {\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0\parallel }^2=\hfill & {\displaystyle \frac{2}{n}}{({\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0)}^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i\psi (\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)}))\hfill \\ \hfill & +{\displaystyle \frac{1}{n^2}}{\mathit{x}}_i^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-2}{\mathit{x}}_i{\psi }^2(\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})).\hfill \end{array}$$

Since ${({\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0)}^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i={\mathrm{O}}_{L_2}(|{\lambda }_i|\parallel {\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0\parallel /{\mathrm{c}}_n^{1/2})$, both terms are of order ${\mathrm{O}}_{L_2}(\mathrm{polyLog}(n)/n{\mathrm{c}}_n^{1/2})$.

Similarly, we have

$$\mathrm{var}(\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0{\parallel }^2)=\mathrm{O}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{n^{1-2\alpha }}}\right)=\mathrm{o}(1).$$

The results for $\mathrm{var}(\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0{\parallel }^2)$ are simply followed by the fact that $\mathrm{var}(\parallel \widehat{\mathit{w}}-{\mathit{w}}_0{\parallel }^2)=\mathrm{o}(1)$ in Proposition 3.10 of [21].

By assuming $\tilde{W}={\parallel \widehat{\mathit{\delta }}+\widehat{\mathit{w}}-{\mathit{\delta }}_0-{\mathit{w}}_0\parallel }^2$ and ${\tilde{W}}_{(i)}={\parallel {\widehat{\mathit{\delta }}}_{(i)}+\widehat{\mathit{w}}-{\mathit{\delta }}_0-{\mathit{w}}_0\parallel }^2$, we can similarly have

$$\mathrm{E}\left(|\parallel \widehat{\mathit{\delta }}+\widehat{\mathit{w}}-{\mathit{\delta }}_0-{\mathit{w}}_0{\parallel }^2-\parallel {\widehat{\mathit{\delta }}}_{(i)}+\widehat{\mathit{w}}-{\mathit{\delta }}_0-{\mathit{w}}_0{{\parallel }^2|}^2\right)=\mathrm{o}(1).$$

Note that $\mathrm{E}(\parallel \widehat{\mathit{w}}-{\mathit{w}}_0\parallel )=\mathrm{O}(1)$, we have $\mathrm{E}(\parallel \widehat{\mathit{\delta }}+\widehat{\mathit{w}}-{\mathit{\delta }}_0-{\mathit{w}}_0\parallel )$ is bounded by $K\mathrm{polyLog}(n)/{\tau }^k$. □

Appendix E.4. Leaving Out a Predictor

Let $\mathit{V}$ be the $n\times (p-1)$ matrix corresponding to the first $(p-1)$ columns of the design matrix $\mathit{X}$. We call ${\mathit{v}}_i$ in ${\mathbb{R}}^{p-1}$ the vector corresponding to the first $p-1$ entries of ${\mathit{x}}_i$, i.e ${\mathit{v}}_i^{\top }=(x_i(1),\dots ,x_i(p-1))$. Denote ${\mathcal{X}}_i={({\mathcal{X}}_i(1),\dots ,{\mathcal{X}}_i(p))}^{\top }$. We call $\mathit{X}(p)$ the vector in ${\mathbb{R}}^n$ with j-th entry $x_j(p)$, i.e the p-th entry of the vector ${\mathit{x}}_j$. When this does not create problems, we also use the standard notation $x_{j,p}$ for $x_j(p)$. We further denote ${\mathit{\delta }}_0={({\gamma }_0^{\top },{\mathit{\delta }}_0(p))}^{\top }$.

Call $\widehat{\gamma }$ the solution of

$$\widehat{\gamma }=\underset{\gamma \in {\mathbb{R}}^{p-1}}{\mathrm{arg\; min}}{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\rho \{{ϵ}_i+{\mathit{v}}_i^{\top }({\mathit{w}}_0-\widehat{\mathit{w}})-{\mathit{v}}_i^{\top }(\gamma -{\gamma }_0)\}+{\displaystyle \frac{\tau }{2}}{\parallel \gamma \parallel }^2.$$

(A8)

Note that $\left[\begin{array}{c}\widehat{\gamma }\hfill \\ 0\hfill \end{array}\right]$ is the solution of the original optimization problem (5) when $x_i(p)$ is replaced by 0.

• Approximation to $\widehat{\mathit{\delta }}$ via leave-one-predictor-out

We use the notations and partitions

$${\mathit{x}}_i=\left[\begin{array}{c}{\mathit{v}}_i\\ x_i(p)\end{array}\right],\widehat{\mathit{\delta }}=\left[\begin{array}{c}{\widehat{\mathit{\delta }}}_{-p}\\ \widehat{\mathit{\delta }}(p)\end{array}\right].$$

(A9)

Naturally, $\widehat{\gamma }$ satisfies

$$-{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\mathit{v}}_i\psi \{{ϵ}_i+{\mathit{v}}_i^{\top }({\mathit{w}}_{0,-p}-{\widehat{\mathit{w}}}_{-p})-{\mathit{v}}_i^{\top }(\widehat{\gamma }-{\gamma }_0)\}+\tau \widehat{\gamma }={\mathbf{0}}_{p-1}.$$

(A10)

Denote

$$r_{i,\left[p\right]}={ϵ}_i+{\mathit{v}}_i^{\top }({\mathit{w}}_{0,-p}-{\widehat{\mathit{w}}}_{-p})-{\mathit{v}}_i^{\top }(\widehat{\gamma }-{\gamma }_0).$$

i.e., the residuals based on $p-1$ predictors.

Recall that

$$-{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\mathit{x}}_i\psi \{{ϵ}_i+{\mathit{x}}_i^{\top }({\mathit{w}}_0-\widehat{\mathit{w}})-{\mathit{x}}_i^{\top }(\widehat{\mathit{\delta }}-{\mathit{\delta }}_0)\}+\tau \widehat{\mathit{\delta }}={\mathbf{0}}_p,$$

(A11)

and

$$R_i={ϵ}_i+{\mathit{x}}_i^{\top }({\mathit{w}}_0-\widehat{\mathit{w}})-{\mathit{x}}_i^{\top }(\widehat{\mathit{\delta }}-{\mathit{\delta }}_0).$$

Taking the difference between (A10) and (A11), we have

$${\displaystyle \frac{1}{n}}{\displaystyle \sum _i^n}\left\{{\mathit{x}}_i\psi (R_i)-\left[\begin{array}{c}{\mathit{v}}_i\\ 0\end{array}\right]\psi (r_{i,\left[p\right]})\right\}-\tau \left(\widehat{\mathit{\delta }}-\left[\begin{array}{c}\widehat{\gamma }\\ 0\end{array}\right]\right)={\mathbf{0}}_p.$$

Note that this p-dimensional equation can be separated into a scalar and a vector equation, namely,

$$\begin{array}{cc}\hfill & {\displaystyle \frac{1}{n}}{\displaystyle \sum _i^n}\left\{x_i(p)\psi (R_i)\right\}-\tau \widehat{\mathit{\delta }}(p)=0,\hfill \\ \hfill & {\displaystyle \frac{1}{n}}{\displaystyle \sum _i^n}{\mathit{v}}_i\left\{\psi (R_i)-\psi (r_{i,\left[p\right]})\right\}-\tau ({\widehat{\mathit{\delta }}}_{-p}-\widehat{\gamma })={\mathbf{0}}_{p-1}.\hfill \end{array}$$

Using a first-order Taylor expansion of $\psi (R_i)$ around $\psi (r_{i,\left[p\right]})$ and noting that $R_i-r_{i,\left[p\right]}={\mathit{v}}_i^{\top }(\widehat{\gamma }-{\widehat{\mathit{\delta }}}_{-p})+x_i(p)\{w_0(p)-\widehat{w}(p)+{\mathit{\delta }}_0(p)-\widehat{\mathit{\delta }}(p)\}$, we can transform the first equation above into

$${\displaystyle \frac{1}{n}}{\displaystyle \sum _i^n}{x}_i(p)\left(\psi (r_{i,\left[p\right]})+{\psi }^{\prime }(r_{i,\left[p\right]})\left[{\mathit{v}}_i^{\top }(\widehat{\gamma }-{\widehat{\mathit{\delta }}}_{-p})+x_i(p)\{w_0(p)-\widehat{w}(p)+{\mathit{\delta }}_0(p)-\widehat{\mathit{\delta }}(p)\}\right]\right)-\tau \widehat{\mathit{\delta }}(p)\simeq 0.$$

This gives the near equivalence

$$\widehat{\mathit{\delta }}(p)\simeq {\displaystyle \frac{\frac{1}{n}\sum x_i(p)\left(\psi (r_{i,\left[p\right]})+{\psi }^{\prime }(r_{i,\left[p\right]})\left[{\mathit{v}}_i^{\top }(\widehat{\gamma }-{\widehat{\mathit{\delta }}}_{-p})+x_i(p)\{w_0(p)-\widehat{w}(p)+{\mathit{\delta }}_0(p)\}\right]\right)}{\frac{1}{n}\sum x_i^2(p){\psi }^{\prime }(r_{i,\left[p\right]})+\tau }}.$$

Working similarly on the second equation involving ${\mathit{v}}_i$, we have

$${\displaystyle \frac{1}{n}}{\displaystyle \sum _i^n}{\psi }^{\prime }(r_{i,\left[p\right]}){\mathit{v}}_i(R_i-r_{i,\left[p\right]})-\tau ({\widehat{\mathit{\delta }}}_{-p}-\widehat{\gamma })\simeq {\mathbf{0}}_{p-1}.$$

Since $R_i-r_{i,\left[p\right]}={\mathit{v}}_i^{\top }(\widehat{\gamma }-{\widehat{\mathit{\delta }}}_{-p})+x_i(p)\{w_0(p)-\widehat{w}(p)+{\mathit{\delta }}_0(p)-\widehat{\mathit{\delta }}(p)\}$, the above equation can be transformed into

$$\left[{\displaystyle \frac{1}{n}}{\displaystyle \sum _i^n}{\psi }^{\prime }(r_{i,\left[p\right]}){\mathit{v}}_i{\mathit{v}}_i^{\top }\right](\widehat{\gamma }-{\widehat{\mathit{\delta }}}_{-p})+\{w_0(p)-\widehat{w}(p)+{\mathit{\delta }}_0(p)-\widehat{\mathit{\delta }}(p)\}{\displaystyle \frac{1}{n}}{\displaystyle \sum _i^n}{\psi }^{\prime }(r_{i,\left[p\right]}){\mathit{v}}_i{x}_i(p)-\tau ({\widehat{\mathit{\delta }}}_{-p}-\widehat{\gamma })\simeq {\mathbf{0}}_{p-1}.$$

Denote

$${\mathit{u}}_p={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }(r_{i,\left[p\right]}){\mathit{v}}_i{x}_i(p),\mathrm{and}{\mathfrak{S}}_p={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }(r_{i,\left[p\right]}){\mathit{v}}_i{\mathit{v}}_i^{\top },$$

we see that $\widehat{\gamma }-{\widehat{\mathit{\delta }}}_{-p}\simeq -\{w_0(p)-\widehat{w}(p)+{\mathit{\delta }}_0(p)-\widehat{\mathit{\delta }}(p)\}{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p$. Using the above approximation in the equation for $\widehat{\mathit{\delta }}(p)$, we can write

$$\widehat{\mathit{\delta }}(p)\simeq {\displaystyle \frac{\frac{1}{n}\sum x_i(p)\psi (r_{i,\left[p\right]})+\{w_0(p)-\widehat{w}(p)+{\mathit{\delta }}_0(p)\}\{\frac{1}{n}\sum x_i^2(p){\psi }^{\prime }(r_{i,\left[p\right]})-{\mathit{u}}_p^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}\}{\mathit{u}}_p}{\frac{1}{n}\sum x_i^2(p){\psi }^{\prime }(r_{i,\left[p\right]})-{\mathit{u}}_p^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p+\tau }}.$$

Denote

$${\xi }_n\triangleq {\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{x}_i^2(p){\psi }^{\prime }(r_{i,\left[p\right]})-{\mathit{u}}_p^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p,$$

and

$$N_p\triangleq {\displaystyle \frac{1}{\sqrt{n}}}{\displaystyle \sum _{i=1}^n}{x}_i(p)\psi (r_{i,\left[p\right]}).$$

We have

$$({\xi }_n+\tau )\widehat{\mathit{\delta }}(p)\simeq \sqrt{n}{N}_p+\{w_0(p)-\widehat{w}(p)+{\mathit{\delta }}_0(p)\}{\xi }_n.$$

Also we have

$${\widehat{\mathit{\delta }}}_{-p}\simeq \widehat{\gamma }+\{w_0(p)-\widehat{w}(p)+{\mathit{\delta }}_0(p)-\widehat{\mathit{\delta }}(p)\}{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p.$$

Thus, we construct an approximation to $\widehat{\mathit{\delta }}$. As a summary, we introduce the following definitions:

Definition A1. 

We call the residuals corresponding to this optimization problem ${\left\{r_{i,\left[p\right]}\right\}}_{i=1}^n$, in other words

$$r_{i,\left[p\right]}={ϵ}_i+{\mathit{v}}_i^{\top }({\mathit{w}}_{0,-p}-{\widehat{\mathit{w}}}_{-p})-{\mathit{v}}_i^{\top }(\widehat{\gamma }-{\gamma }_0).$$

We call

$${\mathit{u}}_p={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }(r_{i,\left[p\right]}){\mathit{v}}_i{x}_i(p),\mathrm{and}{\mathfrak{S}}_p={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }(r_{i,\left[p\right]}){\mathit{v}}_i{\mathit{v}}_i^{\top }.$$

Note that ${\mathit{u}}_p\in {\mathbb{R}}^{p-1}$ and ${\mathfrak{S}}_p$ is $(p-1)\times (p-1)$. We call

$${\xi }_n\triangleq {\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{x}_i^2(p){\psi }^{\prime }(r_{i,\left[p\right]})-{\mathit{u}}_p^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p,$$

and

$$N_p\triangleq {\displaystyle \frac{1}{\sqrt{n}}}{\displaystyle \sum _{i=1}^n}{x}_i(p)\psi (r_{i,\left[p\right]}).$$

We consider

$${\mathfrak{b}}_p\triangleq \{w_0(p)-\widehat{w}(p)+{\mathit{\delta }}_0(p)\}{\displaystyle \frac{{\xi }_n}{\tau +{\xi }_n}}+{\displaystyle \frac{1}{\sqrt{n}}}{\displaystyle \frac{N_p}{\tau +{\xi }_n}}.$$

Note that when ${\xi }_n>0$, we have

$${\mathfrak{b}}_p-{\mathit{\delta }}_0(p)=w_0(p)-\widehat{w}(p)+{\displaystyle \frac{n^{-1/2}{N}_p-\tau {\mathfrak{b}}_p}{{\xi }_n}}.$$

We call

$$\tilde{\mathit{b}}=\left[\begin{array}{c}\widehat{\gamma }\\ {\mathit{\delta }}_0(p)-\widehat{w}(p)+w_0(p)\end{array}\right]+\{{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)\}\left[\begin{array}{c}-{\left({\mathfrak{S}}_p+\tau \mathrm{I}\right)}^{-1}{\mathit{u}}_p\\ 1\end{array}\right].$$

Appendix E.4.1. Deterministic Aspects

Proposition A5. 

We have

$$\begin{array}{c}\parallel \widehat{\mathit{\delta }}-\tilde{\mathit{b}}\parallel \le {\displaystyle \frac{1}{\tau }}|{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)|\underset{1\le i\le n}{\sup }|{\mathrm{d}}_{i,p}|\parallel \widehat{\sum }{\parallel }_2\sqrt{\parallel {({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p{\parallel }^2+1}\hfill \end{array}$$

(A12)

where ${\mathrm{d}}_{i,p}=[{\psi }^{\prime }({\gamma }_{i,p}^{\ast })-{\psi }^{\prime }(r_{i,\left[p\right]})]$ and ${\gamma }_{i,p}^{\ast }$ is in the interval $({ϵ}_i+{\mathit{v}}_i^{\top }({\mathit{w}}_{0,-p}-{\widehat{\mathit{w}}}_{-p})-{\mathit{v}}_i^{\top }(\widehat{\gamma }-{\gamma }_0),{ϵ}_i+{\mathit{x}}_i^{\top }({\mathit{w}}_0-\widehat{\mathit{w}})-{\mathit{x}}_i^{\top }(\tilde{\mathit{b}}-{\mathit{\beta }}_0))$. Furthermore,

$$\parallel {({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{u}_p{\parallel }^2\le {\displaystyle \frac{1}{n\tau }}{\displaystyle \sum _{i=1}^n}{x}_i^2(p){\psi }^{\prime }(r_{i,\left[p\right]})={\displaystyle \frac{1}{n\tau }}{\displaystyle \sum _{i=1}^n}{\lambda }_i^2{\psi }^{\prime }(r_{i,\left[p\right]}){\mathcal{X}}_i^2(p).$$

(A13)

As in Lemma A2, we have

$$\parallel \widehat{\mathit{\delta }}-\tilde{\mathit{b}}\parallel \le {\displaystyle \frac{1}{\tau }}\parallel f(\tilde{\mathit{b}})\parallel ,$$

where

$$f(\tilde{\mathit{b}})=-{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\mathit{x}}_i\psi \{{ϵ}_i+{\mathit{x}}_i^{\top }({\mathit{w}}_0-\widehat{\mathit{w}})-{\mathit{x}}_i^{\top }(\tilde{\mathit{b}}-{\mathit{\delta }}_0)\}+\tau \tilde{\mathit{b}}.$$

(A14)

We note furthermore that, by definition of $\widehat{\gamma }$,

$$g(\widehat{\gamma })=-{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\mathit{v}}_i\psi \{{ϵ}_i+{\mathit{v}}_i^{\top }({\mathit{w}}_{0,-p}-{\widehat{\mathit{w}}}_{-p})-{\mathit{v}}_i^{\top }(\widehat{\gamma }-{\gamma }_0)\}+\tau \widehat{\gamma }={\mathbf{0}}_{p-1}.$$

Proof. 

i. Work on the first $\mathit{p}-\mathbf{1}$ coordinates of $\mathit{f}(\tilde{\mathit{b}})$

Denote $f_{p-1}(\mathit{\delta })$ the first $p-1$ coordinates of $f(\mathit{\delta })$. Denote ${\widehat{\gamma }}_{ext}$ the p-dimensional vector whose first $p-1$ coordinates are $\widehat{\gamma }$ and last coordinate is ${\mathit{\delta }}_0(p)$, i.e.,

$${\widehat{\gamma }}_{ext}=\left[\begin{array}{c}\widehat{\gamma }\\ {\mathit{\delta }}_0(p)-\widehat{w}(p)+w_0(p)\end{array}\right].$$

For a vector $\mathit{v}$, we use the notation ${\mathit{v}}_{-k}$ to denote the vector obtained by removing the k-th coordinate of $\mathit{v}$.

Note that

$$\begin{array}{ccc}{f}_{p-1}(\tilde{\mathit{b}})\hfill & =& f_{p-1}(\tilde{\mathit{b}})-g(\widehat{\gamma })\hfill \\ & =& -{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\mathit{v}}_i\left[\psi \{{ϵ}_i+{\mathit{x}}_i^{\top }({\mathit{w}}_0-\widehat{\mathit{w}})+{\mathit{x}}_i^{\top }({\mathit{\delta }}_0-\tilde{\mathit{b}})\}-\psi \{{ϵ}_i+{\mathit{v}}_i^{\top }({\mathit{w}}_{0,-p}-{\widehat{\mathit{w}}}_{-p})+{\mathit{v}}_i^{\top }({\gamma }_0-\widehat{\gamma })\}\right]\hfill \\ & & +\tau ({\tilde{\mathit{b}}}_{-p}-\widehat{\gamma }).\hfill \end{array}$$

By the mean value theorem, for ${\gamma }_{i,p}^{\ast }$ in the interval $({ϵ}_i+{\mathit{v}}_i^{\top }({\mathit{w}}_0-\widehat{\mathit{w}})-{\mathit{v}}_i^{\top }({\mathit{w}}_{0,-p}-{\widehat{\mathit{w}}}_{-p}),{ϵ}_i+{\mathit{x}}_i^{\top }({\mathit{w}}_0-\widehat{\mathit{w}})-{\mathit{x}}_i^{\top }(\tilde{\mathit{b}}-{\mathit{\delta }}_0))$, we have

$$\begin{array}{ccc}& & \psi \{{ϵ}_i+{\mathit{x}}_i^{\top }({\mathit{w}}_0-\widehat{\mathit{w}})+{\mathit{x}}_i^{\top }({\mathit{\delta }}_0-\tilde{\mathit{b}})\}-\psi \{{ϵ}_i+{\mathit{v}}_i^{\top }({\mathit{w}}_{0,-p}-{\widehat{\mathit{w}}}_{-p})+{\mathit{v}}_i^{\top }({\gamma }_0-\widehat{\gamma })\}\hfill \\ & =& {\psi }^{\prime }({\gamma }_{i,p}^{\ast }){\mathit{x}}_i^{\top }({\widehat{\gamma }}_{ext}-\tilde{\mathit{b}})\hfill \\ & =& {\psi }^{\prime }(r_{i,\left[p\right]}){\mathit{x}}_i^{\top }({\widehat{\gamma }}_{ext}-\tilde{\mathit{b}})+\{{\psi }^{\prime }({\gamma }_{i,p}^{\ast })-{\psi }^{\prime }(r_{i,\left[p\right]})\}{\mathit{x}}_i^{\top }({\widehat{\gamma }}_{ext}-\tilde{\mathit{b}}).\hfill \end{array}$$

Denote

$$\begin{array}{cc}{\mathrm{d}}_{i,p}\hfill & ={\psi }^{\prime }({\gamma }_{i,p}^{\ast })-{\psi }^{\prime }(r_{i,\left[p\right]}),\hfill \\ {\mathit{\delta }}_{i,p}\hfill & =\{{\psi }^{\prime }({\gamma }_{i,p}^{\ast })-{\psi }^{\prime }(r_{i,\left[p\right]})\}{\mathit{x}}_i^{\top }({\widehat{\gamma }}_{ext}-\tilde{\mathit{b}}),\hfill \\ {\mathrm{R}}_p\hfill & =-{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\mathit{v}}_i\left[\{{\psi }^{\prime }({\gamma }_{i,p}^{\ast })-{\psi }^{\prime }(r_{i,\left[p\right]})\}{\mathit{x}}_i^{\top }({\widehat{\gamma }}_{ext}-\tilde{\mathit{b}})\right].\hfill \end{array}$$

Therefore, we have

$$f_{p-1}(\tilde{\mathit{b}})=-{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }(r_{i,\left[p\right]}){\mathit{v}}_i{\mathit{x}}_i^{\top }({\widehat{\gamma }}_{ext}-\tilde{\mathit{b}})+\tau ({\tilde{\mathit{b}}}_{-p}-\widehat{\gamma })+{\mathrm{R}}_p\triangleq {\mathrm{A}}_p+{\mathrm{R}}_p.$$

Note that by definition,

$$\begin{array}{cc}{\widehat{\gamma }}_{ext}-\tilde{\mathit{b}}\hfill & =\{{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)\}\left[\begin{array}{c}{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p\\ -1\end{array}\right],\hfill \\ {\tilde{\mathit{b}}}_{-p}-\widehat{\gamma }\hfill & =-\{{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)\}{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p.\hfill \end{array}$$

Therefore, we have ${\mathit{x}}_i^{\top }({\widehat{\gamma }}_{ext}-\tilde{\mathit{b}})=\{{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)\}\{{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p-x_i(p)\}$, and

$${\mathrm{A}}_p=-\{{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)\}\left[{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }(r_{i,\left[p\right]}){\mathit{v}}_i\left\{{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p-x_i(p)\right\}+\tau {({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p\right].$$

By the definition of ${\mathfrak{S}}_p$ and ${\mathit{u}}_p$, we have

$${\mathrm{A}}_p=-\{{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)\}\{{\mathfrak{S}}_p{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p-{\mathit{u}}_p+\tau {({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p\}={\mathbf{0}}_{p-1},$$

since ${\mathfrak{S}}_p{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}+\tau {({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}=\mathrm{I}$.

Therefore, we conclude that

$$f_{p-1}(\tilde{\mathit{b}})={\mathrm{R}}_p.$$

ii. Work on the last coordinate of $\mathit{f}(\tilde{\mathit{b}})$

Denote ${\left[f(\tilde{\mathit{\delta }})\right]}_p$ the last coordinate of $f(\tilde{\mathit{\delta }})$. We have shown that

$$\begin{array}{ccc}& & \psi \{{ϵ}_i+{\mathit{x}}_i^{\top }({\mathit{w}}_0-\widehat{\mathit{w}})+{\mathit{x}}_i^{\top }({\mathit{\delta }}_0-\tilde{\mathit{b}})\}-\psi \{{ϵ}_i+{\mathit{v}}_i^{\top }({\mathit{w}}_{0,-p}-{\widehat{\mathit{w}}}_{-p})+{\mathit{v}}_i^{\top }({\gamma }_0-\widehat{\gamma })\}\hfill \\ & =& {\psi }^{\prime }(r_{i,\left[p\right]}){\mathit{x}}_i^{\top }({\widehat{\gamma }}_{ext}-\tilde{\mathit{b}})+\{{\psi }^{\prime }({\gamma }_{i,p}^{\ast })-{\psi }^{\prime }(r_{i,\left[p\right]})\}{\mathit{x}}_i^{\top }({\widehat{\gamma }}_{ext}-\tilde{\mathit{b}}).\hfill \end{array}$$

Recall that

$$\begin{array}{cc}{r}_{i,\left[p\right]}\hfill & ={ϵ}_i+{\mathit{v}}_i^{\top }({\mathit{w}}_{0,-p}-{\widehat{\mathit{w}}}_{-p})-{\mathit{v}}_i^{\top }(\widehat{\gamma }-{\gamma }_0),\hfill \\ {\mathit{\delta }}_{i,p}\hfill & =\{{\psi }^{\prime }({\gamma }_{i,p}^{\ast })-{\psi }^{\prime }(r_{i,\left[p\right]})\}{\mathit{x}}_i^{\top }({\widehat{\gamma }}_{ext}-\tilde{\mathit{b}}).\hfill \end{array}$$

We note that

$$\begin{array}{ccc}& & \psi \{{ϵ}_i+{\mathit{x}}_i^{\top }({\mathit{w}}_0-\widehat{\mathit{w}})+{\mathit{x}}_i^{\top }({\mathit{\delta }}_0-\tilde{\mathit{b}})\}\hfill \\ & =& \psi (r_{i,\left[p\right]})+{\psi }^{\prime }(r_{i,\left[p\right]}){\mathit{x}}_i^{\top }({\widehat{\gamma }}_{ext}-\tilde{\mathit{b}})+{\mathit{\delta }}_{i,p}\hfill \\ & =& \psi (r_{i,\left[p\right]})+{\psi }^{\prime }(r_{i,\left[p\right]})\{{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)\}\{{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p-x_i(p)\}+{\mathit{\delta }}_{i,p}.\hfill \end{array}$$

Therefore, we have

$$\begin{array}{ccc}& & {\left[f(\tilde{\mathit{\delta }})\right]}_p+{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{x}_i(p){\mathit{\delta }}_{i,p}\hfill \\ & =& -{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{x}_i(p)\left[\psi (r_{i,\left[p\right]})+{\psi }^{\prime }(r_{i,\left[p\right]})\{{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)\}\{{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p-x_i(p)\}\right]+\tau \tilde{b}(p),\hfill \\ & =& -{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{x}_i(p)\psi (r_{i,\left[p\right]})-\{{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)\}{\mathit{u}}_p^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p\hfill \\ & & +\{{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)\}{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }(r_{i,\left[p\right]})x_i^2(p)+\tau {\mathfrak{b}}_p,\hfill \\ & =& -\left\{{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{x}_i(p)\psi (r_{i,\left[p\right]})-\tau {\mathfrak{b}}_p\right\}\hfill \\ & & +\{{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)\}\left\{{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }(r_{i,\left[p\right]})x_i^2(p)-{\mathit{u}}_p^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p\right\},\hfill \\ & =& -\left({\displaystyle \frac{1}{\sqrt{n}}}{N}_p-\tau {\mathfrak{b}}_p\right)+\{{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)\}{\xi }_n,\hfill \\ & =& 0.\hfill \end{array}$$

We conclude that

$$\begin{array}{ccc}{\left[f(\tilde{\mathit{\delta }})\right]}_p\hfill & =& -{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{x}_i(p){\mathit{\delta }}_{i,p}\hfill \\ & =& -{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{x}_i(p)\{{\psi }^{\prime }({\gamma }_{i,p}^{\ast })-{\psi }^{\prime }(r_{i,\left[p\right]})\}{\mathit{x}}_i^{\top }({\widehat{\gamma }}_{ext}-\tilde{\mathit{b}}).\hfill \end{array}$$

iii. Representation of $\mathit{f}(\tilde{\mathit{b}})$

Aggregating all the results we have obtained so far, we see that

$$\begin{array}{cc}f(\tilde{\mathit{b}})=\hfill & -{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\mathrm{d}}_{i,p}{\mathit{x}}_i{\mathit{x}}_i^{\top }({\widehat{\gamma }}_{ext}-\tilde{\mathit{b}})\hfill \\ =\hfill & \{{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)\}\left\{{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\mathrm{d}}_{i,p}{\mathit{x}}_i{\mathit{x}}_i^{\top }\right\}\left[\begin{array}{c}{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p\\ -1\end{array}\right],\hfill \end{array}$$

(A15)

which implies (A12).

For $\parallel {({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p{\parallel }^2$, denote ${\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}$ the diagonal matrix with $(i,i)$ entry ${\psi }^{\prime }(r_{i,\left[p\right]})$. We have

$${\mathit{u}}_p={\displaystyle \frac{1}{n}}{\mathit{V}}^{\top }{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}\mathit{X}(p).$$

Therefore,

$$\begin{array}{ccc}& & \parallel {({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p{\parallel }^2\hfill \\ & =& {\displaystyle \frac{\mathit{X}(p)}{\sqrt{n}}}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}{\displaystyle \frac{{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{V}}{\sqrt{n}}}{\left({\displaystyle \frac{{\mathit{V}}^{\top }{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}\mathit{V}}{n}}+\tau \mathrm{I}\right)}^{-1}{\displaystyle \frac{{\mathit{V}}^{\top }{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}}{\sqrt{n}}}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}{\displaystyle \frac{\mathit{X}(p)}{\sqrt{n}}}.\hfill \end{array}$$

Note that

$${\displaystyle \frac{{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{V}}{\sqrt{n}}}{\left({\displaystyle \frac{{\mathit{V}}^{\top }{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}\mathit{V}}{n}}+\tau \mathrm{I}\right)}^{-1}{\displaystyle \frac{{\mathit{V}}^{\top }{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}}{\sqrt{n}}}⪯\mathrm{I}.$$

So we have

$$\parallel {({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p{\parallel }^2\le {\displaystyle \frac{1}{n}}\mathit{X}{(p)}^{\top }{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}\mathit{X}(p)={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{x}_i^2(p){\psi }^{\prime }(r_{i,\left[p\right]}).$$

□

Appendix E.4.2. Stochastic Aspects

Assume that $\mathcal{X}(p)={({\mathcal{X}}_1(p),\dots ,{\mathcal{X}}_n(p))}^{\top }$ is independent of ${\{{\mathcal{V}}_i,{ϵ}_i\}}_{i=1}^n$, where ${\mathcal{V}}_i={\mathit{V}}_i/{\lambda }_i$. This is consistent with Assumption P1.

To bound $\parallel \widehat{\mathit{\delta }}-\tilde{\mathit{b}}\parallel$, using Equation (A13) we have

$$\parallel {({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{u}_p{\parallel }^2\le {\displaystyle \frac{1}{n\tau }}{\displaystyle \sum _{i=1}^n}{\lambda }_i^2{\parallel {\psi }^{\prime }\parallel }_{\infty }{\mathcal{X}}_i^2(p),$$

and

$$\parallel {({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{u}_p{\parallel }^2\le {\displaystyle \frac{{\sup }_i{\parallel {\psi }^{\prime }\parallel }_{\infty }}{\tau }}{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\lambda }_i^2{\mathcal{X}}_i^2(p).$$

Therefore, under Assumptions O3, O4 and O6, we have for any fixed k and at fixed $\tau$,

$$\parallel {({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{u}_p{\parallel }^2={\mathrm{O}}_{L_k}(\mathrm{polyLog}(n)).$$

It guarantees that

$${∥\begin{array}{c}{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p\\ -1\end{array}∥}^2\le (1+\parallel {({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p{\parallel }^2)={\mathrm{O}}_{L_k}(\mathrm{polyLog}(n)).$$

Thus

$$\parallel \widehat{\mathit{\delta }}-\tilde{\mathit{b}}\parallel ={\mathrm{O}}_{L_k}\left({\displaystyle \frac{1}{\tau }}\mathrm{polyLog}(n)|{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)|\underset{1\le i\le n}{\sup }|{\mathrm{d}}_{i,p}|\parallel \widehat{\sum }{\parallel }_2\right).$$

Recall that Lemma 3.38 from [21] guarantees that $\parallel {\displaystyle \widehat{\sum }}\parallel ={\mathrm{O}}_{L_k}(\mathrm{polyLog}(n))$ under assumption O1–O7. Also we will show in Proposition A6 that $|{\mathfrak{b}}_p-{\mathit{\delta }}_0(p)+\widehat{w}(p)-w_0(p)|={\mathrm{O}}_{L_k}\left\{\mathrm{polyLog}(n)(n^{-1/2}\vee n^{-e})\right\}$ and in Proposition A8 that ${\sup }_{1\le i\le n}|{\mathrm{d}}_{i,p}|=\left\{\mathrm{polyLog}(n)(n^{\alpha -1/2}\vee n^{\alpha -e})\right\}$ to show that $M_1$ is small.

• On ${\mathfrak{b}}_p-{\mathit{\delta }}_0(p)$

Recall the notations

$$\begin{array}{cc}{N}_p\hfill & ={\displaystyle \frac{1}{\sqrt{n}}}{\displaystyle \sum _{i=1}^n}{x}_i(p)\psi (r_{i,\left[p\right]})={\displaystyle \frac{1}{\sqrt{n}}}{\displaystyle \sum _{i=1}^n}{\lambda }_i\psi (r_{i,\left[p\right]}){\mathcal{X}}_i(p),\hfill \\ {\xi }_n\hfill & ={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{x}_i^2(p){\psi }^{\prime }(r_{i,\left[p\right]})-{\mathit{u}}_p^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p.\hfill \end{array}$$

Under our assumptions, we have $\mathrm{E}({\mathcal{X}}_i)={\mathbf{0}}_p$ and $\mathrm{cov}({\mathcal{X}}_i)={\mathrm{I}}_p$, and $\mathcal{X}(p)$ is independent of ${\left\{r_{i,\left[p\right]}\right\}}_{i=1}^n$.

Proposition A6. 

We have

$$|{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)|\le {\displaystyle \frac{1}{\sqrt{n}\tau }}|N_p|+\parallel {\mathit{\delta }}_0{\parallel }_{\infty }+|\widehat{w}(p)-w_0(p)|.$$

Furthermore, under Assumptions  O1–O7   and  P1, $N_p={\mathrm{O}}_{L_k}(\mathrm{polyLog}(n))$ and therefore, when τ is fixed,

$$|{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)|={\mathrm{O}}_{L_k}(\mathrm{polyLog}(n)n^{-1/2}+\parallel {\mathit{\delta }}_0{\parallel }_{\infty }+|\widehat{w}(p)-w_0(p)|).$$

Note that Lemma A1 has shown that

$$|\widehat{w}(p)-w_0(p)|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{\sqrt{n}\wedge n^e}}\right).$$

(A16)

Under Assumption P3, $n\parallel {\mathit{\delta }}_0{\parallel }_{\infty }^2\mathrm{polyLog}(n)n^{2\alpha -1/2}\to 0$, and therefore we have

$$|{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{\sqrt{n}\wedge n^{\mathrm{e}}}}\right).$$

(A17)

Proof. 

From the definition of ${\mathfrak{b}}_p$, we have, when ${\xi }_n>0$,

$${\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)={\displaystyle \frac{1}{\sqrt{n}}}{\displaystyle \frac{N_p}{\tau +{\xi }_n}}-{\displaystyle \frac{\tau }{\tau +{\xi }_n}}\{{\delta }_0(p)+w_0(p)-\widehat{w}(p)\}.$$

We will see later that ${\xi }_n\ge 0$ in Lemma A7. It follows that

$$|{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)|\le {\displaystyle \frac{1}{\sqrt{n}\tau }}|N_p|+|{\delta }_0(p)+w_0(p)-\widehat{w}(p)|.$$

Using independence of $\mathcal{X}(p)$ and ${\{{\mathcal{V}}_i,{ϵ}_i\}}_{i=1}^n$, and the fact that $\mathrm{E}({\mathcal{X}}_i)={\mathbf{0}}_p$, we have

$$\mathrm{E}(N_p^2)={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left\{{\mathcal{X}}_i^2(p)\right\}\mathrm{E}\left\{{\lambda }_i^2{\psi }^2(r_{i,\left[p\right]})\right\},$$

whether the right-hand side is finite or not. Using the bounds on $\max {\lambda }_i^2$ and ${\sup }_i{\parallel \psi \parallel }_{\infty }$, we have

$$\mathrm{E}(N_p^2)\le {\displaystyle \frac{1}{\sqrt{n}}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left\{{\mathcal{X}}_i^2(p)\right\}{\parallel \psi \parallel }_{\infty }\mathrm{E}\left\{{\lambda }_i^2\right\}=\mathrm{O}(1)=\mathrm{O}(\mathrm{polyLog}(n)).$$

Similarly, it is clear that

$$N_p={\mathrm{O}}_{L_k}(\mathrm{polyLog}(n)).$$

Therefore, we have

$$|{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)|={\displaystyle \frac{1}{\sqrt{n}\tau }}{\mathrm{O}}_{L_k}(\mathrm{polyLog}(n))+\underset{1\le k\le p}{\sup }|{\delta }_0(k)|+|\widehat{w}(p)-w_0(p)|.$$

□

• On ${\xi }_n$

Write ${\xi }_n$ in matrix form. Let ${\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}$ be the diagonal matrix with $(i,i)$ entry ${\psi }^{\prime }(r_{i,\left[p\right]})$. Denote $\mathit{X}(p)$ the last column of the design matrix $\mathit{X}$. Then we have

$${\xi }_n={\displaystyle \frac{1}{n}}\mathit{X}{(p)}^{\top }{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{M}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{X}(p),$$

where

$$\mathit{M}={\mathrm{I}}_n-{\displaystyle \frac{{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{V}}{\sqrt{n}}}{\left({\displaystyle \frac{1}{n}}{\mathit{V}}^{\top }{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}\mathit{V}+\tau \mathrm{I}\right)}^{-1}{\displaystyle \frac{{\mathit{V}}^{\top }{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}}{\sqrt{n}}}.$$

(A18)

Lemma A7. 

We have

$${\xi }_n\ge 0.$$

Furthermore, under Assumptions  O1–O7   and  P1, if ${\mathit{D}}_{{\lambda }_i}$ is the diagonal matrix with $(i,i)$ entry ${\lambda }_i$,

$$|{\xi }_n-{\displaystyle \frac{1}{n}}\mathrm{tr}\left({\mathit{D}}_{{\lambda }_i}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{M}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}{\mathit{D}}_{{\lambda }_i}\right)|={\mathrm{O}}_{L_k}\left(\underset{1\le i\le n}{\sup }{\lambda }_i^2{\psi }^{\prime }(r_{i,\left[p\right]})/\sqrt{n{\mathrm{c}}_n}\right).$$

Proof. 

When $\tau >0$, all the eigenvalues of $\mathit{M}$ are positive. Indeed, if the singular values of $n^{1/2}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{V}$ are denoted by ${\sigma }_1$, the eigenvalues of $\mathit{M}$ are $\tau /({\sigma }_i^2+\tau )$.

Therefore, since ${\xi }_n=n^{-1}{\mathit{v}}^{\top }\mathit{M}\mathit{v}$ with $\mathit{v}={\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{X}(p)$, we have ${\xi }_n\ge 0$.

Since $\mathit{M}$ is symmetric and has eigenvalues between 0 and 1, using Lemma V.1.5 in [36], we have

$$0⪯{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{M}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}⪯{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}.$$

Under Assumption P1, the matrix $\mathit{M}$ is independent of $\mathcal{X}(p)$. ${\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}$ is also independent of $\mathcal{X}(p)$. By definition, $\mathit{X}(p)={\mathit{D}}_{{\lambda }_i}\mathcal{X}(p)$.

Under Assumption P1, ${\mathcal{X}}_p$ satisfy the necessary concentration assumptions. Using Lemma 3.37 in [21], we have

$$\begin{array}{cc}\hfill & |{\displaystyle \frac{1}{n}}\mathit{X}{(p)}^{\top }{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{M}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{X}(p)-{\displaystyle \frac{1}{n}}\mathrm{tr}\left({\mathit{D}}_{{\lambda }_i}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{M}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}{\mathit{D}}_{{\lambda }_i}\right)|\hfill \\ \hfill & ={\mathrm{O}}_{L_k}\left({\displaystyle \frac{1}{\sqrt{n{\mathrm{c}}_n}}}\underset{1\le i\le n}{\sup }{\lambda }_i^2{\psi }^{\prime }(r_{i,\left[p\right]})\right).\hfill \end{array}$$

□

• About ${\displaystyle \frac{1}{n}}\mathrm{tr}\left({\mathit{D}}_{{\lambda }_i}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{M}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}{\mathit{D}}_{{\lambda }_i}\right)$

Lemma A8. 

Denote

$${\mathfrak{S}}_p={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }(r_{i,\left[p\right]}){\mathit{v}}_i{\mathit{v}}_i^{\top },\mathrm{and}{\mathfrak{S}}_p(i)={\mathfrak{S}}_p-{\displaystyle \frac{1}{n}}{\psi }^{\prime }(r_{i,\left[p\right]}){\mathit{v}}_i{\mathit{v}}_i^{\top }.$$

Denote

$$\begin{array}{cc}{\mathrm{c}}_{\tau ,p}\hfill & ={\displaystyle \frac{1}{n}}\mathrm{tr}\left\{{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}\right\},\hfill \\ {\zeta }_i\hfill & ={\displaystyle \frac{1}{n}}{\mathit{v}}_i^{\top }{\{{\mathfrak{S}}_p(i)+\tau \mathrm{I}\}}^{-1}{\mathit{v}}_i-{\lambda }_i^2{\mathrm{c}}_{\tau ,p}.\hfill \end{array}$$

(A19)

Then we have under Assumptions  O1–O7andP1, if $\mathit{M}$ is the matrix defined in Lemma A7,

$$|{\displaystyle \frac{1}{n}}\mathrm{tr}({\mathrm{I}}_n-\mathit{M})-\left\{{\displaystyle \frac{1}{n}}\mathrm{tr}\left({\mathit{D}}_{{\lambda }_i}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{M}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}{\mathit{D}}_{{\lambda }_i}\right)\right\}{\mathrm{c}}_{\tau ,p}|\le \underset{i}{\sup }|{\zeta }_i|·{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }(r_{i,\left[p\right]}).$$

We also have

$${\displaystyle \frac{1}{n}}\mathrm{tr}({\mathrm{I}}_n-\mathit{M})={\displaystyle \frac{p-1}{n}}-\tau {\mathrm{c}}_{\tau ,p}.$$

Proof. 

Denote $d_{i,i}={\psi }^{\prime }(r_{i,\left[p\right]})/n$. By using the Sherman-MorrisonWoodbury formula (see, e.g., [37], p. 19),

$$\begin{array}{cc}{M}_{i,i}\hfill & =1-d_{i,i}{\mathit{v}}_i^{\top }{\left({\mathit{V}}^{\top }{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}\mathit{V}/n+\tau \mathrm{I}\right)}^{-1}{\mathit{v}}_i,\hfill \\ \hfill & =1-d_{i,i}{\displaystyle \frac{{\mathit{v}}_i^{\top }{\left({\mathfrak{S}}_p(i)+\tau \mathrm{I}\right)}^{-1}{\mathit{v}}_i}{1+d_{i,i}{\mathit{v}}_i^{\top }{\left({\mathfrak{S}}_p(i)+\tau \mathrm{I}\right)}^{-1}{\mathit{v}}_i}},\hfill \\ \hfill & ={\displaystyle \frac{1}{1+d_{i,i}{\mathit{v}}_i^{\top }{\left({\mathfrak{S}}_p(i)+\tau \mathrm{I}\right)}^{-1}{\mathit{v}}_i}}.\hfill \end{array}$$

Recall that we are interested in

$${\displaystyle \frac{1}{n}}\sum _i{\lambda }_i^2{\psi }^{\prime }(r_{i,\left[p\right]})M_{i,i}={\displaystyle \frac{1}{n}}\left({\mathit{D}}_{{\lambda }_i}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{M}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}{\mathit{D}}_{{\lambda }_i}\right).$$

By the property of trace, we have

$$\begin{array}{cc}\mathrm{tr}({\mathrm{I}}_n-\mathit{M})\hfill & =\mathrm{tr}\left\{{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathfrak{S}}_p\right\}\hfill \\ \hfill & =p-1-\tau \mathrm{tr}\left\{{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}\right\}=p-1-n\tau {\mathrm{c}}_{\tau ,p}.\hfill \end{array}$$

This shows the second result of the lemma.

The first result follows from the fact that

$$\begin{array}{cc}\mathrm{tr}({\mathrm{I}}_n-\mathit{M})\hfill & =\sum _i(1-M_{i,i})\hfill \\ \hfill & =\sum _i{d}_{i,i}{\displaystyle \frac{{\mathit{v}}_i^{\top }{\left({\mathfrak{S}}_p(i)+\tau \mathrm{I}\right)}^{-1}{\mathit{v}}_i}{1+d_{i,i}{\mathit{v}}_i^{\top }{\left({\mathfrak{S}}_p(i)+\tau \mathrm{I}\right)}^{-1}{\mathit{v}}_i}}.\hfill \end{array}$$

(A20)

With our definitions, we have, since ${\lambda }_i^2{\mathrm{c}}_{\tau ,p}+{\zeta }_i=n^{-1}{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}{\mathit{v}}_i$,

$$\begin{array}{cc}{\displaystyle \frac{1}{n}}\mathrm{tr}({\mathrm{I}}_n-\mathit{M})=\hfill & \left({\displaystyle \frac{1}{n}}\sum _i{\lambda }_i^2{\psi }^{\prime }(r_{i,\left[p\right]})M_{i,i}\right){\mathrm{c}}_{\tau ,p}+{\displaystyle \frac{1}{n}}\sum _i{\psi }^{\prime }(r_{i,\left[p\right]}){\displaystyle \frac{{\zeta }_i}{1+d_{i,i}{\mathit{v}}_i^{\top }{\left({\mathfrak{S}}_p(i)+\tau \mathrm{I}\right)}^{-1}{\mathit{v}}_i}}.\hfill \end{array}$$

It immediately follows that

$$|{\displaystyle \frac{1}{n}}\mathrm{tr}({\mathrm{I}}_n-\mathit{M})-\left({\displaystyle \frac{1}{n}}\sum _i{\lambda }_i^2{\psi }^{\prime }(r_{i,\left[p\right]})M_{i,i}\right){\mathrm{c}}_{\tau ,p}|\le \underset{i}{\sup }|{\zeta }_i|·{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }(r_{i,\left[p\right]}).$$

□

• Controlling ${\zeta }_i$

Lemma A9. 

Suppose we can find ${\left\{{\mathrm{r}}_{i,\left[p\right]}^{(i)}\right\}}_{j\ne i}$ independent of $({\lambda }_i,{\mathcal{V}}_i)$ and $K_n$ such that

$$\underset{i}{\sup }\underset{j\ne i}{\sup }|{\psi }_j^{\prime }({\mathrm{r}}_{j,\left[p\right]}^{(i)})-{\psi }_j^{\prime }(r_{j,\left[p\right]})|\le K_n.$$

Then

$$\underset{i}{\sup }|{\zeta }_i|={\mathrm{O}}_{L_k}\left[\left\{{\displaystyle \frac{1}{{\tau }^2}}{K}_n{\parallel \widehat{\sum }\parallel }_2+{\displaystyle \frac{\mathrm{polyLog}(n)}{\tau \sqrt{n{\mathrm{c}}_n}}}+{\displaystyle \frac{1}{n\tau }}\right\}\mathrm{polyLog}(n)\right],$$

provided that $K_n$ has $3k$ uniformly bounded moments.

Proof. 

Denote

$${\mathbf{AM}}_{i,p}={\displaystyle \frac{1}{n}}\sum _{j\ne i}{\psi }^{\prime }(r_{j,\left[p\right]}^{(i)}){\mathit{v}}_j{\mathit{v}}_j^{\top }.$$

Then, using the fact that ${\mathit{A}}^{-1}-{\mathit{B}}^{-1}={\mathit{A}}^{-1}(\mathit{B}-\mathit{A}){\mathit{B}}^{-1}$, we have

$$\parallel {({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}-{({\mathbf{AM}}_{i,p}+\tau \mathrm{I})}^{-1}\parallel \le {\displaystyle \frac{1}{{\tau }^2}}{K}_n{\parallel \widehat{\sum }\parallel }_2,$$

since $\parallel n^{-1}{\sum }_i{\mathit{v}}_i{\mathit{v}}_i^{\top }\parallel \le \parallel {\displaystyle \widehat{\sum }}{\parallel }_2$. In particular, we have

$$|{\displaystyle \frac{1}{n}}{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}{\mathit{v}}_i-{\displaystyle \frac{1}{n}}{\mathit{v}}_i^{\top }{({\mathbf{AM}}_{i,p}+\tau \mathrm{I})}^{-1}{\mathit{v}}_i|\le {\displaystyle \frac{\parallel {\mathit{v}}_i{\parallel }^2}{n}}{\displaystyle \frac{1}{{\tau }^2}}{K}_n{\parallel \widehat{\sum }\parallel }_2.$$

Since ${\mathbf{AM}}_{i,p}$ is independent of $({\lambda }_i,{\mathcal{V}}_i)$, we can use Lemma 3.37 in [21] and since $\mathit{v}={\lambda }_i{\mathcal{V}}_i$, we have

$$\underset{1\le i\le n}{\sup }|{\displaystyle \frac{1}{n}}{\mathit{v}}_i^{\top }{({\mathbf{AM}}_{i,p}+\tau \mathrm{I})}^{-1}{\mathit{v}}_i-{\displaystyle \frac{{\lambda }_i^2}{n}}\mathrm{tr}\left\{{({\mathbf{AM}}_{i,p}+\tau \mathrm{I})}^{-1}\right\}|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{\tau \sqrt{n{\mathrm{c}}_n}}}\underset{1\le i\le n}{\sup }{\lambda }_i^2\right),$$

by using the fact that ${\lambda }_{\max }({({\mathbf{AM}}_{i,p}+\tau \mathrm{I})}^{-1})\le {\tau }^{-1}$.

Using the operator norm bound we gave above, we also have

$$|{\displaystyle \frac{1}{n}}\mathrm{tr}\left\{{({\mathbf{AM}}_{i,p}+\tau \mathrm{I})}^{-1}\right\}-{\displaystyle \frac{1}{n}}\mathrm{tr}\left\{{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}\right\}|\le {\displaystyle \frac{1}{{\tau }^2}}{K}_n{\parallel \widehat{\sum }\parallel }_2{\displaystyle \frac{p}{n}}.$$

We can now conclude that

$$\begin{array}{cc}\hfill & \underset{1\le i\le n}{\sup }|{\displaystyle \frac{1}{n}}{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}{\mathit{v}}_i-{\displaystyle \frac{{\lambda }_i^2}{n}}\mathrm{tr}\left\{{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}\right\}|\hfill \\ \hfill & =\mathrm{O}\left[\left\{{\displaystyle \frac{1}{{\tau }^2}}{K}_n{\parallel \widehat{\sum }\parallel }_2\underset{1\le i\le n}{\sup }\left({\displaystyle \frac{p}{n}}+{\displaystyle \frac{\parallel {\mathit{v}}_i{\parallel }^2}{n}}\right)+{\displaystyle \frac{\mathrm{polyLog}(n)}{\tau \sqrt{n{\mathrm{c}}_n}}}\right\}\left(\underset{1\le i\le n}{\sup }{\lambda }_i^2\vee 1\right)\right].\hfill \end{array}$$

So under O1 and O4, ${\sup }_{1\le i\le n}{\parallel {\mathit{v}}_i\parallel }^2/n={\mathrm{O}}_{L_k}(1)$ and finally

$$\begin{array}{cc}\hfill & \underset{1\le i\le n}{\sup }|{\displaystyle \frac{1}{n}}{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}{\mathit{v}}_i-{\displaystyle \frac{{\lambda }_i^2}{n}}\mathrm{tr}\left\{{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}\right\}|\hfill \\ \hfill & ={\mathrm{O}}_{L_k}\left\{\left({\displaystyle \frac{1}{{\tau }^2}}{K}_n{\parallel \widehat{\sum }\parallel }_2+{\displaystyle \frac{\mathrm{polyLog}(n)}{\tau \sqrt{n{\mathrm{c}}_n}}}\right)\left(\underset{1\le i\le n}{\sup }{\lambda }_i^2\vee 1\right)\right\}.\hfill \end{array}$$

Control of $n^{-1}\mathrm{tr}\left\{{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}\right\}-n^{-1}\mathrm{tr}\left\{{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}\right\}$

Using the Sherman-Woodbury-Morrison formula, we have

$${({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}-{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}={\displaystyle \frac{{\psi }^{\prime }(r_{i,\left[p\right]})}{n}}{\displaystyle \frac{{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}{\mathit{v}}_i{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}}{1+\frac{{\psi }^{\prime }(r_{i,\left[p\right]})}{n}{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}{\mathit{v}}_i}}.$$

Take the trace of both sides, and we have

$$0\le \mathrm{tr}\left\{{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}\right\}-\mathrm{tr}\left\{{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}\right\}\le {\displaystyle \frac{1}{\tau }},$$

since ${\mathit{v}}_i^{\top }{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-2}{\mathit{v}}_i\le {\tau }^{-1}{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}{\mathit{v}}_i$.

Therefore,

$$0\le {\displaystyle \frac{1}{n}}\mathrm{tr}\left\{{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}\right\}-{\displaystyle \frac{1}{n}}\mathrm{tr}\left\{{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}\right\}\le {\displaystyle \frac{1}{n\tau }}.$$

We can now conclude that

$$\underset{i}{\sup }|{\zeta }_i|={\mathrm{O}}_{L_k}\left[\left\{{\displaystyle \frac{1}{{\tau }^2}}{K}_n{\parallel \widehat{\sum }\parallel }_2+{\displaystyle \frac{\mathrm{polyLog}(n)}{\tau \sqrt{n{\mathrm{c}}_n}}}+{\displaystyle \frac{1}{n\tau }}\right\}\left(\underset{1\le i\le n}{\sup }{\lambda }_i^2\vee 1\right)\right].$$

provided we can use Holder’s inequality. This, in turn, requires $K_n$ to have $3k$ moments. □

• Control of $K_n$

A natural choice for ${\left\{{\mathrm{r}}_{i,\left[p\right]}^{(i)}\right\}}_{j\ne i}$ defined in Lemma A9 is to use a leave-one-out estimator of $\widehat{\gamma }$, where the i-th observation (and hence ${\mathit{v}}_i$) is removed. Hence, all the work performed in Theorem A1 can be used here.

Lemma A10. 

Suppose we use for ${\mathrm{r}}_{i,\left[p\right]}^{(i)}$ the residuals we would obtain by using a leave-one-out estimator of $\widehat{\gamma }$, i.e., excluding $({\mathit{v}}_i,{ϵ}_i)$ from problem (A8).

With the notations of Lemma A9, we have under Assumptions  O1–O7   and  P1,

$$K_n={\mathrm{O}}_{L_k}(n^{2\alpha -1/2}\mathrm{polyLog}(n)).$$

In particular, for any fixed τ,

$$\underset{i}{\sup }|{\zeta }_i|={\mathrm{O}}_{L_k}(n^{2\alpha -1/2}\mathrm{polyLog}(n)).$$

Proof. 

Denote ${\mathit{\delta }}_n(i)$ random variables such that

$$\underset{j\ne i}{\sup }|{\mathrm{r}}_{j,\left[p\right]}^{(i)}-r_{j,\left[p\right]}|\le {\mathit{\delta }}_n(i).$$

Applying Theorem A1 with $R_j=r_{j,\left[p\right]}$ and ${\tilde{r}}_{j,(i)}={\mathrm{r}}_{j,\left[p\right]}^{(i)}$, we have

$$\underset{i}{\sup }{\mathit{\delta }}_n(i)={\mathrm{O}}_{L_k}(n^{2\alpha -1/2}\mathrm{polyLog}(n)).$$

The control of $K_n$ follows by using the fact that ${\psi }^{\prime }$ is $C{n}^{\alpha }$-Lipschitz. □

Corollary A1. 

Recall that in (A6) and (A1)

$$c_i={\displaystyle \frac{1}{n}}{\mathit{x}}_i^{\top }{({\mathit{S}}_i+\tau \mathrm{I})}^{-1}{\mathit{x}}_i,c_{\tau }={\displaystyle \frac{1}{n}}\mathrm{tr}{(\mathit{S}+\tau \mathrm{I})}^{-1}.$$

Then under Assumptions  O1–O7   and  P1, we have

$$\underset{i}{\sup }|c_i-{\lambda }_i^2{c}_{\tau }|={\mathrm{O}}_{L_k}(n^{2\alpha -1/2}\mathrm{polyLog}(n)).$$

Proof. 

We have shown that

$$\underset{i}{\sup }|{\displaystyle \frac{1}{n}}{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p(i)+\tau \mathrm{I})}^{-1}{\mathit{v}}_i-{\lambda }_i^2{\mathrm{c}}_{\tau ,p}|={\mathrm{O}}_{L_k}(n^{2\alpha -1/2}\mathrm{polyLog}(n)).$$

Recall that

$$c_{\tau }={\displaystyle \frac{1}{n}}\mathrm{tr}\left[{\left\{{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }(R_i){\mathit{x}}_i{\mathit{x}}_i^{\top }+\tau \mathrm{I}\right\}}^{-1}\right].$$

We see that $c_{\tau }$ is analogous to ${\mathrm{c}}_{\tau ,p}$ when we use all the data, rather than $(p-1)$ of them.

Indeed, $c_i$ in (A6) is defined, in the notation of the proof of Lemma A9 as an analog of $n^{-1}{\mathit{v}}_i^{\top }{({\mathbf{AM}}_{i,p}+\tau \mathrm{I})}^{-1}{\mathit{v}}_i$, with the role of ${\left\{{\mathrm{r}}_{i,\left[p\right]}^{(i)}\right\}}_{j\ne i}$ being played by the residuals obtained by the leave-one-out estimator of $\widehat{\gamma }$, excluding $({\mathit{x}}_i,y_i)$ from the problem. Lemma A9 in connection with Theorem A2 shows that ${\sup }_i|n^{-1}{\mathit{v}}_i^{\top }{({\mathbf{AM}}_{i,p}+\tau \mathrm{I})}^{-1}{\mathit{v}}_i-{\lambda }_i^2{\mathrm{c}}_{\tau ,p}|={\mathrm{O}}_{L_k}(n^{2\alpha -1/2}\mathrm{polyLog}(n))$. Passing from the $p-1$ dimensional version of this result, i.e., Lemma A9, to the p-dimensional version gives the approximation stated in the corollary.

We therefore have

$$\underset{i}{\sup }|c_i-{\lambda }_i^2{c}_{\tau }|={\mathrm{O}}_{L_k}(n^{2\alpha -1/2}\mathrm{polyLog}(n)).$$

□

• Further results on ${\xi }_n$ and ${\mathfrak{b}}_p$

Proposition A7. 

Under Assumptions  O1–O7   and  P1, we have

$$|{\mathrm{c}}_{\tau ,p}({\xi }_n+\tau )-{\displaystyle \frac{p-1}{n}}|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{n^{1/2-2\alpha }}}\right).$$

(A21)

Furthermore, under Assumptions  O1–O7   and  P1–P3, since $\parallel {\delta }_0{\parallel }_{\infty }={\mathrm{O}}_{L_k}(n^{-e})$,

$$\begin{array}{cc}{\left({\displaystyle \frac{p}{n}}\right)}^{2}n\mathrm{E}\left[{\{{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)\}}^2\right]=\hfill & {\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left[{\left\{{\mathrm{c}}_{\tau ,p}{\lambda }_i\psi (r_{i,\left[p\right]})\right\}}^2\right]\hfill \\ \hfill & +n{\tau }^2{\{{\delta }_0(p)-\widehat{w}(p)+w_0(p)\}}^2\mathrm{E}({\mathrm{c}}_{\tau ,p}^2)+\mathrm{o}(1).\hfill \end{array}$$

(A22)

Proof.  

For Equation (A21):

By Lemma A8, we have

$${\displaystyle \frac{p-1}{n}}-\tau {\mathrm{c}}_{\tau ,p}={\displaystyle \frac{1}{n}}\mathrm{tr}({\mathrm{I}}_n-\mathit{M})\ge 0.$$

The latter quantity was approximated in Lemma A8 by

$${\displaystyle \frac{1}{n}}\mathrm{tr}\left({\mathit{D}}_{{\lambda }_i}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}\mathit{M}{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}^{1/2}{\mathit{D}}_{{\lambda }_i}\right){\mathrm{c}}_{\tau ,p},$$

which approximate $\xi$ as in Lemma A7. This gives the result of Equation (A21), by simply keeping track of the approximation errors we make at each step.

For Equation (A22):

Recall that by definition:

$$\sqrt{n}\left[(\tau +{\xi }_n){\mathfrak{b}}_p-{\xi }_n{\delta }_0(p)+{\xi }_n\{\widehat{w}(p)-w_0(p)\}\right]=N_p={\displaystyle \frac{1}{\sqrt{n}}}{\displaystyle \sum _{i=1}^n}{\lambda }_i\psi (r_{i,\left[p\right]}){\mathcal{X}}_i(p).$$

Therefore, we have

$$\begin{array}{cc}{\mathrm{c}}_{\tau ,p}\sqrt{n}(\tau +{\xi }_n)\{{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)\}=\hfill & {\displaystyle \frac{1}{\sqrt{n}}}{\displaystyle \sum _{i=1}^n}{\mathrm{c}}_{\tau ,p}{\lambda }_i\psi (r_{i,\left[p\right]}){\mathcal{X}}_i(p)\hfill \\ \hfill & -{\mathrm{c}}_{\tau ,p}\sqrt{n}\tau \{{\delta }_0(p)-\widehat{w}(p)+w_0(p)\}.\hfill \end{array}$$

Note that ${\mathrm{c}}_{\tau ,p}{\lambda }_i\psi (r_{i,\left[p\right]})$ is independent of ${\mathcal{X}}_i(p)$, and ${\mathcal{X}}_i(p)$’s are independent with mean zero and variance one. We have

$$\begin{array}{cc}\mathrm{E}\left[{\mathrm{c}}_{\tau ,p}^{2}n{(\tau +{\xi }_n)}^2{\{{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)\}}^2\right]=\hfill & {\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left[{\left\{{\mathrm{c}}_{\tau ,p}{\lambda }_i\psi (r_{i,\left[p\right]})\right\}}^2\right]\hfill \\ \hfill & +n{\tau }^2{\{{\delta }_0(p)-\widehat{w}(p)+w_0(p)\}}^2\mathrm{E}({\mathrm{c}}_{\tau ,p}^2).\hfill \end{array}$$

(A23)

Recall that Proposition A6 gives that

$$|{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)|\le {\displaystyle \frac{1}{\sqrt{n}\tau }}|N_p|+\parallel {\mathit{\delta }}_0{\parallel }_{\infty }+|\widehat{w}(p)-w_0(p)|.$$

Under Assumption P3, $n\parallel {\mathit{\delta }}_0{\parallel }_{\infty }^2\mathrm{polyLog}(n)n^{2\alpha -1/2}\to 0$. Together with Equations (A21) and (A16), the LHS of (A23) can be written as

$$\begin{array}{cc}\hfill & \mathrm{E}\left[{\mathrm{c}}_{\tau ,p}^2{(\tau +{\xi }_n)}^{2}n{\{{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)\}}^2\right]\hfill \\ =\hfill & \mathrm{E}\left[{\left\{{\mathrm{c}}_{\tau ,p}(\tau +{\xi }_n)-{\displaystyle \frac{p-1}{n}}+{\displaystyle \frac{p-1}{n}}\right\}}^{2}n{\{{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)\}}^2\right]\hfill \\ =\hfill & {\left({\displaystyle \frac{p}{n}}\right)}^{2}n\mathrm{E}\left[{\{{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)\}}^2\right]+\mathrm{o}(1).\hfill \end{array}$$

This implies Equation (A22). □

• On ${\mathrm{d}}_{i,p}$

Recall that

$${\mathrm{d}}_{i,p}={\psi }^{\prime }({\gamma }_{i,p}^{\ast })-{\psi }^{\prime }(r_{i,\left[p\right]}),$$

where ${\gamma }_{i,p}^{\ast }$ in the interval $(r_{i,\left[p\right]},r_{i,\left[p\right]}+{\nu }_i)$, with

$$\begin{array}{cc}{\nu }_i\hfill & =\{{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)\}{\mathit{x}}_i^{\top }\left[\begin{array}{c}{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p\\ -1\end{array}\right]\hfill \\ \hfill & =\{{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)\}{\pi }_i.\hfill \end{array}$$

We have the following result.

Proposition A8. 

Under Assumptions  O1–O7   and  P1–P3, we have, at fixed τ,

$$\underset{i}{\sup }|{\mathrm{d}}_{i,p}|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)n^{\alpha }}{n^{1/2}\wedge n^{\mathrm{e}}}}\right).$$

Proof. 

Note that we can write

$${\pi }_i={\mathit{x}}_i^{\top }\left[\begin{array}{c}{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p\\ -1\end{array}\right]={\mathit{v}}_i^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p-x_i(p).$$

Recall that ${\mathit{u}}_p=n^{-1}{\mathit{V}}^{\top }{\mathit{D}}_{{\psi }^{\prime }(r_{·,\left[p\right]})}\mathit{X}(p)$. We can also write it as

$${\mathit{u}}_p={\displaystyle \frac{1}{n}}{\mathcal{V}}^{\top }{\mathit{D}}_{{\lambda }_i^2{\psi }^{\prime }(r_{·,\left[p\right]})}\mathcal{X}(p).$$

Using the independence of $\mathcal{X}(p)$ with ${\left\{({\mathcal{V}}_i,{ϵ}_i)\right\}}_{i=1}^n$, and the concentration assumptions on $\mathcal{X}(p)$ in Assumption P1, according to Lemma 3.36 in [21], we have

$$\underset{i}{\sup }|{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{{\mathrm{c}}_n^{1/2}}}\underset{i}{\sup }\parallel {\displaystyle \frac{1}{n}}{\mathit{D}}_{{\lambda }_i^2{\psi }^{\prime }(r_{·,\left[p\right]})}\mathcal{V}{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathcal{V}}_i\parallel \right),$$

where we view ${\mathit{v}}_i^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathit{u}}_p$ as a linear form in ${\mathcal{V}}_i$. Note that we have absorbed the ${\sup }_i|{\lambda }_i|$ in the $\mathrm{polyLog}(n)$ term.

We can write

$$\parallel {\displaystyle \frac{1}{n}}{\mathit{D}}_{{\lambda }_i^2{\psi }^{\prime }(r_{·,\left[p\right]})}\mathcal{V}{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathcal{V}}_i\parallel ={\displaystyle \frac{1}{n}}{\mathcal{V}}_i^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\displaystyle \frac{{\mathcal{V}}^{\top }{\mathit{D}}_{{\lambda }_i^2{\psi }^{\prime }(r_{·,\left[p\right]})}^2\mathcal{V}}{n}}{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathcal{V}}_i.$$

We notice that ${\mathfrak{S}}_p=n^{-1}{\mathcal{V}}^{\top }{\mathit{D}}_{{\lambda }_i^2{\psi }^{\prime }(r_{·,\left[p\right]})}\mathcal{V}$. Hence,

$${\displaystyle \frac{{\mathcal{V}}^{\top }{\mathit{D}}_{{\lambda }_i^2{\psi }^{\prime }(r_{·,\left[p\right]})}^2\mathcal{V}}{n}}⪯{\parallel {\mathit{D}}_{{\lambda }_i^2{\psi }^{\prime }(r_{·,\left[p\right]})}\parallel }_2{\mathfrak{S}}_p.$$

Therefore, we conclude that

$${\displaystyle \frac{1}{n}}{\mathcal{V}}_i^{\top }{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\displaystyle \frac{{\mathcal{V}}^{\top }{\mathit{D}}_{{\lambda }_i^2{\psi }^{\prime }(r_{·,\left[p\right]})}^2\mathcal{V}}{n}}{({\mathfrak{S}}_p+\tau \mathrm{I})}^{-1}{\mathcal{V}}_i\le {\displaystyle \frac{\parallel {\mathcal{V}}_i{\parallel }^2}{n\tau }}{\parallel {\mathit{D}}_{{\lambda }_i^2{\psi }^{\prime }(r_{·,\left[p\right]})}\parallel }_2={\displaystyle \frac{\parallel {\mathcal{V}}_i{\parallel }^2}{n\tau }}\underset{i}{\sup }{\lambda }_i^2{\psi }^{\prime }(r_{i,\left[p\right]}).$$

Note that ${\sup }_i|x_i(p)|={\mathrm{O}}_{L_k}(\mathrm{polyLog}(n)/\sqrt{{\mathrm{c}}_n})$ under Assumption O4, O6 and P1, according to Appendix 7 in [21]. Therefore, we have

$$\begin{array}{cc}\underset{i}{\sup }|{\pi }_i|\hfill & ={\mathrm{O}}_{L_k}\left[{\displaystyle \frac{\mathrm{polyLog}(n)}{{\mathrm{c}}_n^{1/2}}}\left\{1+\sqrt{{\sup }_i{\lambda }_i^2{\psi }^{\prime }(r_{i,\left[p\right]}){\sup }_i{\displaystyle \frac{\parallel {\mathcal{V}}_i{\parallel }^2}{n\tau }}}\right\}\right]\hfill \\ \hfill & ={\mathrm{O}}_{L_k}\left[{\displaystyle \frac{\mathrm{polyLog}(n)}{{\mathrm{c}}_n^{1/2}}}\left\{1+\sqrt{{\sup }_i{\lambda }_i^2{\psi }^{\prime }(r_{i,\left[p\right]})}\right\}\right]\hfill \\ \hfill & ={\mathrm{O}}_{L_k}\left\{\mathrm{polyLog}(n)\right\}.\hfill \end{array}$$

Recall that $|{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)|={\mathrm{O}}_{L_k}(\mathrm{polyLog}(n)n^{-1/2}+\parallel {\mathit{\delta }}_0{\parallel }_{\infty }+|\widehat{w}(p)-w_0(p)|)$ and by Lemma A1 we have

$$|\widehat{w}(p)-w_0(p)|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{\sqrt{n}\wedge n^e}}\right).$$

We have

$$\underset{i}{\sup }|{\nu }_i|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{\sqrt{n}\wedge n^{\mathrm{e}}}}\right).$$

Note that [21] has shown that Under our assumption that ${\psi }^{\prime }$ is $C{n}^{\alpha }$-Lipschitz, we see that

$$\underset{i}{\sup }|{\mathrm{d}}_{i,p}|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)n^{\alpha }}{n^{1/2}\wedge n^{\mathrm{e}}}}\right).$$

□

Appendix E.4.3. Final Conclusions

Gathering all the results, we have the following Theorem.

Theorem A2. 

Under Assumptions  O1–O7   and  P1–P3, we have, for any fixed τ,

$$\parallel \widehat{\mathit{\delta }}-\tilde{\mathit{b}}\parallel \le {\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)n^{\alpha }}{{(n^{1/2}\wedge n^{\mathrm{e}})}^2}}\right).$$

In particular,

$$\begin{array}{cc}\hfill \sqrt{n}({\widehat{\mathit{\delta }}}_p-{\mathfrak{b}}_p)& ={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)n^{\alpha +1/2}}{{(n^{1/2}\wedge n^{\mathrm{e}})}^2}}\right),\hfill \\ \hfill \underset{i}{\sup }|{\mathit{x}}_i^{\top }(\widehat{\mathit{\delta }}-\tilde{\mathit{b}})|& ={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)n^{\alpha +1/2}}{{(n^{1/2}\wedge n^{\mathrm{e}})}^2}}\right),\hfill \\ \underset{i}{\sup }|R_i-r_{i,\left[p\right]}|\hfill & ={\mathrm{O}}_{L_k}\left(\left\{{\displaystyle \frac{\mathrm{polyLog}(n)}{\sqrt{n}\wedge n^{\mathrm{e}}}}\right\}\vee \left\{{\displaystyle \frac{\mathrm{polyLog}(n)n^{\alpha +1/2}}{{(n^{1/2}\wedge n^{\mathrm{e}})}^2}}\right\}\right).\hfill \end{array}$$

Proof. 

The first two results are direct consequences of all our results, using the key bound on $\parallel \widehat{\mathit{\delta }}-\tilde{\mathit{b}}\parallel$ of Proposition A6.

The third result is a direct consequence of the fact that ${\sup }_i\parallel {\mathcal{X}}_i\parallel /n^{1/2}={\mathrm{O}}_{L_k}(1)$, which was shown in the proof of Lemma A6.

The last result follows from the fact that $R_i-r_{i,\left[p\right]}={\mathit{x}}_i^{\top }(\widehat{\mathit{\delta }}-\tilde{\mathit{b}})-{\nu }_i$. The result on ${\nu }_i$ is given in the proof of Proposition A8. □

Recall Equation (A22)

$$\begin{array}{cc}{\left({\displaystyle \frac{p}{n}}\right)}^{2}n\mathrm{E}\left[{\{{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)\}}^2\right]=\hfill & {\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left[{\left\{{\mathrm{c}}_{\tau ,p}{\lambda }_i\psi (r_{i,\left[p\right]})\right\}}^2\right]\hfill \\ \hfill & +n{\tau }^2{\{{\delta }_0(p)-\widehat{w}(p)+w_0(p)\}}^2\mathrm{E}({\mathrm{c}}_{\tau ,p}^2)+\mathrm{o}(1).\hfill \end{array}$$

and Equation (A17)

$$|{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{\sqrt{n}\wedge n^{\mathrm{e}}}}\right).$$

We have

$$\begin{array}{cc}\hfill & {\left({\displaystyle \frac{p}{n}}\right)}^{2}n\mathrm{E}\left[{\{{\widehat{\delta }}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)\}}^2\right]\hfill \\ \hfill & ={\left({\displaystyle \frac{p}{n}}\right)}^{2}n\mathrm{E}\left[{\{{\widehat{\delta }}_p-{\mathfrak{b}}_p+{\mathfrak{b}}_p-{\delta }_0(p)+\widehat{w}(p)-w_0(p)\}}^2\right]\hfill \\ \hfill & ={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left[{\left\{{\mathrm{c}}_{\tau ,p}{\lambda }_i\psi (r_{i,\left[p\right]})\right\}}^2\right]+n{\tau }^2{\{{\delta }_0(p)-\widehat{w}(p)+w_0(p)\}}^2\mathrm{E}({\mathrm{c}}_{\tau ,p}^2)+\mathrm{o}(A),\hfill \end{array}$$

where $A={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n\mathrm{E}\left[{\left\{{\mathrm{c}}_{\tau ,p}{\lambda }_i\psi (r_{i,\left[p\right]})\right\}}^2\right]+n{\tau }^2{\{{\delta }_0(p)-\widehat{w}(p)+w_0(p)\}}^2\mathrm{E}({\mathrm{c}}_{\tau ,p}^2)$.

We note that the index p in Equation (A22) and the previous theorem plays no particular role, and similar results hold when p is replaced by any k in the range $1\le k\le p$. Summing over all the coordinates, we have under Assumptions O1–O7 and P1–P4,

$$\begin{array}{cc}{\left({\displaystyle \frac{p}{n}}\right)}^2\mathrm{E}(\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0{\parallel }^2)=\hfill & {\displaystyle \frac{1}{n}}{\displaystyle \sum _{k=1}^p}\left({\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left[{\left\{{\mathrm{c}}_{\tau ,k}{\lambda }_i\psi (r_{i,\left[k\right]})\right\}}^2\right]\right)\hfill \\ \hfill & +{\tau }^2{\displaystyle \sum _{k=1}^p}{\{{\delta }_0(k)-\widehat{w}(k)+w_0(k)\}}^2\mathrm{E}({\mathrm{c}}_{\tau ,k}^2)+\mathrm{o}(B),\hfill \end{array}$$

where $B={\displaystyle \frac{1}{n}}{\sum }_{k=1}^p\left({\displaystyle \frac{1}{n}}{\sum }_{i=1}^n\mathrm{E}\left[{\left\{{\mathrm{c}}_{\tau ,k}{\lambda }_i\psi (r_{i,\left[k\right]})\right\}}^2\right]\right)+{\tau }^2{\sum }_{k=1}^p{\{{\delta }_0(k)-\widehat{w}(k)+w_0(k)\}}^2\mathrm{E}({\mathrm{c}}_{\tau ,k}^2)$.

Note that $0\le {\mathrm{c}}_{\tau ,k}\le \kappa /\tau$, $n^{-1}{\sum }_{i=1}^n{\parallel {\psi }^2\parallel }_{\infty }\le C$ from Assumption O3, and $\parallel {\mathit{\delta }}_0{\parallel }^2=\mathrm{O}(1)$. We have $B=\mathrm{O}(1)$. Therefore, we have

$$\begin{array}{cc}{\left({\displaystyle \frac{p}{n}}\right)}^2\mathrm{E}(\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0{\parallel }^2)=\hfill & {\displaystyle \frac{1}{n}}{\displaystyle \sum _{k=1}^p}\left({\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left[{\left\{{\mathrm{c}}_{\tau ,k}{\lambda }_i\psi (r_{i,\left[k\right]})\right\}}^2\right]\right)\hfill \\ \hfill & +{\tau }^2{\displaystyle \sum _{k=1}^p}{\{{\delta }_0(k)-\widehat{w}(k)+w_0(k)\}}^2\mathrm{E}({\mathrm{c}}_{\tau ,k}^2)+\mathrm{o}(1).\hfill \end{array}$$

• On ${\mathrm{c}}_{\tau ,k}$ and $c_{\tau }$

We now show that ${\mathrm{c}}_{\tau ,k}$ and $c_{\tau }$ are close to each other.

Proposition A9. 

We have, under Assumptions  O1–O7   and  P1–P3,

$$\underset{1\le k\le p}{\sup }|{\mathrm{c}}_{\tau ,k}-c_{\tau }|={\mathrm{O}}_{L_k}\left(\left[{\displaystyle \frac{\mathrm{polyLog}(n)n^{\alpha }}{\sqrt{n}\wedge n^{\mathrm{e}}}}\right]\vee \left[{\displaystyle \frac{\mathrm{polyLog}(n)n^{2\alpha +1/2}}{{(n^{1/2}\wedge n^{\mathrm{e}})}^2}}\right]\vee {\displaystyle \frac{\mathrm{polyLog}(n)}{n}}\right).$$

Of course, we also have $0\le c_{\tau }\le p/(n\tau )$ and $0\le {\mathrm{c}}_{\tau ,k}\le p/(n\tau )$.

Proof. 

Recall that

$$\mathit{S}={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\psi }^{\prime }(R_i){\mathit{x}}_i{\mathit{x}}_i^{\top }.$$

Denote $\mathbf{\Gamma }=n^{-1}{\sum }_{i=1}^n{\psi }^{\prime }(R_i)\mathit{v}{\mathit{v}}^{\top }$ and $a=n^{-1}{\sum }_{i=1}^n{\psi }^{\prime }(R_i)x_i^2(p)$, we have

$$\mathit{S}=\left(\begin{array}{cc}\mathbf{\Gamma }\hfill & \mathbf{v}\hfill \\ \mathbf{v}\hfill & a\hfill \end{array}\right).$$

According to Lemma 3.40 in [21], we have, since $c_{\tau }=n^{-1}\mathrm{tr}\left\{{(\mathit{S}+\tau {\mathrm{I}}_p)}^{-1}\right\}$,

$$|c_{\tau }-{\displaystyle \frac{1}{n}}\mathrm{tr}\left\{{(\mathbf{\Gamma }+\tau {\mathrm{I}}_{p-1})}^{-1}\right\}|\le {\displaystyle \frac{1}{n}}{\displaystyle \frac{1+a/\tau }{\tau }}.$$

We also have

$$a={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\lambda }_i^2{\mathcal{X}}_i^2(p){\psi }^{\prime }(R_i)\le \mathrm{polyLog}(n){\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\lambda }_i^2{\mathcal{X}}_i^2(p)={\mathrm{O}}_{L_k}(\mathrm{polyLog}(n)).$$

Since ${\psi }^{\prime }$ is $C{n}^{\alpha }$-Lipschitz and

$$\underset{i}{\sup }|R_i-r_{i,\left[p\right]}|={\mathrm{O}}_{L_k}\left(\left\{{\displaystyle \frac{\mathrm{polyLog}(n)}{\sqrt{n}\wedge n^{\mathrm{e}}}}\right\}\vee \left\{{\displaystyle \frac{\mathrm{polyLog}(n)n^{\alpha +1/2}}{{(n^{1/2}\wedge n^{\mathrm{e}})}^2}}\right\}\right),$$

we have

$$\underset{i}{\sup }|{\psi }^{\prime }(R_i)-{\psi }^{\prime }(r_{i,\left[p\right]})|={\mathrm{O}}_{L_k}\left(\left\{{\displaystyle \frac{\mathrm{polyLog}(n)n^{\alpha }}{\sqrt{n}\wedge n^{\mathrm{e}}}}\right\}\vee \left\{{\displaystyle \frac{\mathrm{polyLog}(n)n^{2\alpha +1/2}}{{(n^{1/2}\wedge n^{\mathrm{e}})}^2}}\right\}\right),$$

Hence, using arguments similar to those in the proof of Lemma A9, we have

$$|{\displaystyle \frac{1}{n}}\mathrm{tr}\left\{{(\mathrm{\Gamma }+\tau {\mathrm{I}}_{p-1})}^{-1}\right\}-{\displaystyle \frac{1}{n}}\mathrm{tr}\left\{{({\mathfrak{S}}_p+\tau {\mathrm{I}}_p)}^{-1}\right\}|={\mathrm{O}}_{L_k}\left(\left[{\displaystyle \frac{\mathrm{polyLog}(n)n^{\alpha }}{\sqrt{n}\wedge n^{\mathrm{e}}}}\right]\vee \left[{\displaystyle \frac{\mathrm{polyLog}(n)n^{2\alpha +1/2}}{{(n^{1/2}\wedge n^{\mathrm{e}})}^2}}\right]\right).$$

Since ${\mathrm{c}}_{\tau ,k}=n^{-1}\mathrm{tr}\left\{{({\mathfrak{S}}_p+\tau {\mathrm{I}}_p)}^{-1}\right\}$, the result follows immediately.

We note that p did not play a particular role in the proof and hence taking the sup over those indices only adds a $\mathrm{polyLog}(n)$ term. Hence the result holds for ${\sup }_{1\le k\le n}|{\mathrm{c}}_{\tau ,k}-c_{\tau }|$. □

We are now ready to prove the last proposition of this section, which will help us obtain the second equation of our System.

Proposition A10. 

$$\begin{array}{cc}{\left({\displaystyle \frac{p}{n}}\right)}^2\mathrm{E}(\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0{\parallel }^2)=\hfill & {\displaystyle \frac{p}{n}}{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left[{\left\{c_{\tau }{\lambda }_i\psi (\mathrm{prox}(c_{\tau }{\lambda }_i^2\rho )({\tilde{r}}_{i,(i)}))\right\}}^2\right]\hfill \\ \hfill & +{\tau }^2{\parallel {\mathit{\delta }}_0+{\mathit{w}}_0-\widehat{\mathit{w}}\parallel }^2\mathrm{E}(c_{\tau }^2)+\mathrm{o}(1).\hfill \end{array}$$

(A24)

Furthermore, when all ${\lambda }_i$’s are non-zero,

$${\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left[{\left\{c_{\tau }{\lambda }_i\psi (\mathrm{prox}(c_{\tau }{\lambda }_i^2\rho )({\tilde{r}}_{i,(i)}))\right\}}^2\right]={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left({\displaystyle \frac{{\{{\tilde{r}}_{i,(i)}-\mathrm{prox}(c_{\tau }{\lambda }_i^2\rho )({\tilde{r}}_{i,(i)})\}}^2}{{\lambda }_i^2}}\right).$$

Proof. 

In light of Proposition A9 and Assumption P3 which guarantees that $\parallel {\mathit{\delta }}_0{\parallel }^2$ is uniformly bounded in p and n, we have

$${\displaystyle \sum _{k=1}^p}{\{{\delta }_0(k)-\widehat{w}(k)+w_0(k)\}}^2\mathrm{E}({\mathrm{c}}_{\tau ,k}^2)={\parallel {\mathit{\delta }}_0+{\mathit{w}}_0-\widehat{\mathit{w}}\parallel }^2\mathrm{E}(c_{\tau }^2)+\mathrm{o}(1).$$

Using Theorem A2 and the bound on $\parallel {\psi }^{\prime }{\parallel }_{\infty }$ in Assumption O3, we have

$${\displaystyle \frac{1}{p}}{\displaystyle \sum _{k=1}^p}\mathrm{E}\left[{\left\{{\mathrm{c}}_{\tau ,k}{\lambda }_i\psi (r_{i,\left[k\right]})\right\}}^2\right]={\displaystyle \frac{1}{p}}{\displaystyle \sum _{k=1}^p}\mathrm{E}\left[{\left\{{\mathrm{c}}_{\tau ,k}{\lambda }_i\psi (R_i)\right\}}^2\right]+\mathrm{o}(1).$$

With the help of Proposition A9, we have

$${\displaystyle \frac{1}{p}}{\displaystyle \sum _{k=1}^p}\mathrm{E}\left[{\left\{{\mathrm{c}}_{\tau ,k}{\lambda }_i\psi (R_i)\right\}}^2\right]={\displaystyle \frac{1}{p}}{\displaystyle \sum _{k=1}^p}\mathrm{E}\left[{\left\{c_{\tau }{\lambda }_i\psi (R_i)\right\}}^2\right]+\mathrm{o}(1)=\mathrm{E}\left[{\left\{c_{\tau }{\lambda }_i\psi (R_i)\right\}}^2\right]+\mathrm{o}(1).$$

In light of Equation (A7), we have

$${\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left\{{(c_{\tau }{\lambda }_i\psi (R_i))}^2\right\}={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left({\left[c_{\tau }{\lambda }_i\psi \left\{\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})\right\}\right]}^2\right)+\mathrm{o}(1).$$

Lemma A11 gives the computation of the derivative of $\mathrm{prox}(c\rho )(x)$ with respect to c, which allows us to bound the error $|\psi \left\{\mathrm{prox}(c_i\rho )(x)\right\}-\psi \left\{\mathrm{prox}(c_{\tau }{\lambda }_i^2\rho )(x)\right\}|$. In light of this, by using the fact that ${\sup }_i|c_i-{\lambda }_i^2{c}_{\tau }|={\mathrm{O}}_{L_k}(n^{2\alpha -1/2}\mathrm{polyLog}(n))$ in Corollary A1, we can re-express the previous equation as

$${\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left\{{(c_{\tau }{\lambda }_i\psi (R_i))}^2\right\}={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left({\left[c_{\tau }{\lambda }_i\psi \left\{\mathrm{prox}(c_{\tau }{\lambda }_i^2\rho )({\tilde{r}}_{i,(i)})\right\}\right]}^2\right)+\mathrm{o}(1).$$

When all ${\lambda }_i$’s are non-zero, we have

$${\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left\{{(c_{\tau }{\lambda }_i\psi (R_i))}^2\right\}={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left({\displaystyle \frac{{\left[c_{\tau }{\lambda }_i^2\psi \left\{\mathrm{prox}(c_{\tau }{\lambda }_i^2\rho )({\tilde{r}}_{i,(i)})\right\}\right]}^2}{{\lambda }_i^2}}\right)+\mathrm{o}(1).$$

Finally, since by definition,

$$\forall x\in \mathbb{R},x=\mathrm{prox}(c\rho )(x)+c\psi (\mathrm{prox}(c\rho )(x)),$$

we have

$${\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left({\displaystyle \frac{{\left[c_{\tau }{\lambda }_i^2\psi \left\{\mathrm{prox}(c_{\tau }{\lambda }_i^2\rho )({\tilde{r}}_{i,(i)})\right\}\right]}^2}{{\lambda }_i^2}}\right)={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left({\displaystyle \frac{{[{\tilde{r}}_{i,(i)}-\mathrm{prox}(c_{\tau }{\lambda }_i^2\rho )({\tilde{r}}_{i,(i)})]}^2}{{\lambda }_i^2}}\right).$$

□

Lemma A11 

(El Karoui [21], Lemma 3.32). Suppose x is a real and ρ is twice differentiable and convex. Then, for $c>0$, we have

$${\displaystyle \frac{\partial }{\partial c}}\mathrm{prox}(c\rho )(x)=-{\displaystyle \frac{\psi (\mathrm{prox}(c\rho )(x))}{1+c{\psi }^{\prime }(\mathrm{prox}(c\rho )(x))}},$$

and

$${\displaystyle \frac{\partial }{\partial c}}\rho (\mathrm{prox}(c\rho )(x))=-{\displaystyle \frac{{\psi }^2(\mathrm{prox}(c\rho )(x))}{1+c{\psi }^{\prime }(\mathrm{prox}(c\rho )(x))}}.$$

In particular, at x given $c\to \rho (\mathrm{prox}(c\rho )(x))$ is decreasing in c.

Appendix E.5. Last Steps of the Proof

Appendix E.5.1. On the Asymptotic Behavior of r ˜ i,(i)

We have the following result.

Lemma A12. 

Under Assumptions  O1–O7  and  P1–P4, as n and p tend to infinity, ${\tilde{r}}_{i,(i)}={ϵ}_i+{\mathit{x}}_i^{\top }({\mathit{w}}_0-\widehat{\mathit{w}})-{\mathit{x}}_i^{\top }({\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0)$ behaves like ${ϵ}_i+{\lambda }_i\sqrt{\mathrm{E}(\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0{\parallel }^2)}{Z}_i$. where $Z_i\sim N(0,1)$ is independent of ${ϵ}_i$ and ${\lambda }_i$, in the sense of weak convergence.

Furthermore, if $i\ne j$, ${\tilde{r}}_{i,(i)}$ and ${\tilde{r}}_{j,(j)}$ are asymptotically (pairwise) independent. The same is true for $({\tilde{r}}_{i,(i)},{\lambda }_i)$ and $({\tilde{r}}_{j,(j)},{\lambda }_j)$.

Proof. 

First part.

In this part, we will show that ${\mathcal{X}}_i^{\top }({\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0)$ is asymptotically $N(0,\mathrm{E}(\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0{\parallel }^2))$. Recall that ${\widehat{\mathit{\delta }}}_{(i)}$ is independent of ${\mathcal{X}}_i$ and that $\mathrm{E}({\mathcal{X}}_i)=\mathbf{0}$, $\mathrm{cov}({\mathcal{X}}_i)=\mathrm{I}$ and that, for any finite k, the first k moments of its entries are bounded uniformly in n.

We have shown that in Proposition A4 that $\mathrm{var}(\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0{\parallel }^2)\to 0$. In light of Lemma A3, we also know that $\mathrm{E}(\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0{\parallel }^2)$ is uniformly bounded. Furthermore, in the proof of Proposition A4, we showed that $\mathrm{E}(\parallel \widehat{\mathit{\delta }}+\widehat{\mathit{w}}-{\mathit{\delta }}_0-{\mathit{w}}_0{\parallel }^2-\parallel {\widehat{\mathit{\delta }}}_{(i)}+\widehat{\mathit{w}}-{\mathit{\delta }}_0-{\mathit{w}}_0{\parallel }^2)\to 0$.

We use a simple generalization of the standard Lindeberg-Feller theorem (see e.g., [38]). Indeed, if $a_{n,p}(k)$ are random variables with $\sqrt{{\sum }_{k=1}^p{a}_{n,p}{(k)}^2}=A_n$, $\mathrm{E}(A_n^2)$ remains bounded in n, and $a_{n,p}(k)$’s are independent of ${\mathcal{X}}_i$, we see that: a) if $\mathit{z}\sim N(\mathbf{0},{\mathrm{I}}_p)$, independent of $a_{n,p}(k)$, then ${\mathit{a}}_{n,p}^{\top }\mathit{z}\sim A_n\mathrm{N}$ where $\mathrm{N}\sim N(0,1)$ and independent of $A_n$ (conditionally and unconditionally on ${\mathit{a}}_{n,p}$); b) Theorem 2.1.5 and its proof in [38] hold provided that ${\sum }_{i=1}^n\mathrm{E}(|a_{n,p}(k){|}^3)=\mathrm{o}(1)$. The proof simply needs to be started conditionally on $a_{n,p}(k)$, and the final moment bounds are then taken unconditionally. This very mild generalization gives, if $\varphi$ is a $C^3$ function, with bounded 2nd and 3rd derivatives,

$$\begin{array}{cc}\hfill & \forall ϵ>0,|\mathrm{E}\left\{\varphi ({\mathit{a}}_{n,p}^{\top }{\mathcal{X}}_i)\right\}-\mathrm{E}\left\{\varphi (A_n\mathrm{N})\right\}|\hfill \\ \hfill & \le K\left[ϵ\parallel {\varphi }^{(3)}{\parallel }_{\infty }\mathrm{E}\left\{{\displaystyle \sum _{k=1}^p}{a}_{n,p}{(k)}^2\right\}+{\displaystyle \frac{\parallel {\varphi }^{(2)}{\parallel }_{\infty }}{ϵ}}{\displaystyle \sum _{k=1}^p}\mathrm{E}(|a_{n,p}(k){|}^3)\right],\hfill \end{array}$$

where K is a constant depending on the second and third absolute moments of the entries of ${\mathcal{X}}_i$. It is therefore independent of n and p under our assumptions on ${\mathcal{X}}_i$.

We can apply this result to $a_{n,p}(k)={\widehat{\delta }}_{(i)}(k)-{\delta }_0(k)+\widehat{w}(k)-w_0(k)$. Recall that we have shown that

$$\widehat{\delta }(k)-{\delta }_0(k)+\widehat{w}(k)-w_0(k)={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{n^{1/2}\wedge n^{\mathrm{e}}}}\right)$$

for each coordinate k. The same arguments apply also to ${\widehat{\delta }}_{(i)}(k)$, the kth coordinate of ${\widehat{\mathit{\delta }}}_{(i)}$. We have

$$\mathrm{E}(|{\widehat{\delta }}_{(i)}(k)-{\delta }_0(k)+\widehat{w}(k)-w_0(k){|}^3)=\mathrm{O}\left\{{\displaystyle \frac{\mathrm{polyLog}(n)}{{(n^{1/2}\wedge n^{\mathrm{e}})}^3}}\right\}$$

Provided that $\mathrm{e}>1/3$, we have

$$\mathrm{E}\left({\displaystyle \sum _{k=1}^p}{|{\widehat{\mathit{\delta }}}_{(i)}(k)-{\mathit{\delta }}_0(k)+\widehat{w}(k)-w_0(k)|}^3\right)=\mathrm{O}\left\{{\displaystyle \frac{\mathrm{polyLog}(n)n}{{(n^{1/2}\wedge n^{\mathrm{e}})}^3}}\right\}=\mathrm{o}(1).$$

Therefore, in connection with Corollary 2.1.9 in [38], ${\mathcal{X}}_i^{\top }({\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0)$ behaves asymptotically like $\parallel {\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0\parallel \mathrm{N}$ in the sense of weak convergence.

Since $\parallel {\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0\parallel -\mathrm{E}\parallel {\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0\parallel \to 0$ in probability and the expectations are uniformly bounded, Slutsky’s lemma gives that

$${\mathcal{X}}_i^{\top }({\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0)\mathrm{behaves}\mathrm{like}\mathrm{E}(\parallel {\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0\parallel )\mathrm{N}$$

asymptotically in the sense of weak convergence. Using the fact that $\mathrm{E}(\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0{\parallel }^2-\parallel {\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0{\parallel }^2)\to 0$, another application of Slutsky’s lemma yields

$${\mathcal{X}}_i^{\top }({\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0)\mathrm{behaves}\mathrm{like}\mathrm{E}(\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0\parallel )\mathrm{N}$$

in the sense of weak convergence.

We note that the same reasoning applies when replacing $a_{n,p}(k)={\widehat{\mathit{\delta }}}_{(i)}(k)-{\mathit{\delta }}_0(k)+\widehat{w}(k)-w_0(k)$ by ${\tilde{a}}_{n,p}(k)={\lambda }_i\{{\widehat{\mathit{\delta }}}_{(i)}(k)-{\mathit{\delta }}_0(k)+\widehat{w}(k)-w_0(k)\}$, provided that ${\lambda }_i$ has 3 moments. It shows that

$${\mathcal{X}}_i^{\top }({\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0)\mathrm{behaves}\mathrm{like}{\lambda }_i\mathrm{E}(\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0\parallel )\mathrm{N}.$$

This shows the first part of the lemma, since $\mathrm{E}(\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0\parallel )=\sqrt{\mathrm{E}(\parallel \widehat{\mathit{\beta }}-{\mathit{\beta }}_0{\parallel }^2)}+\mathrm{o}(1)$ by Proposition A4.

Second part.

For the second part, we use a leave-two-out approach. More precisely, we use the approximation

$${\tilde{r}}_{i,(i)}={ϵ}_i+{\mathit{x}}_i^{\top }({\mathit{w}}_0-\widehat{\mathit{w}})-{\mathit{x}}_i^{\top }({\widehat{\mathit{\delta }}}_{(i)}-{\mathit{\delta }}_0)={ϵ}_i+{\mathit{x}}_i^{\top }({\mathit{w}}_0-\widehat{\mathit{w}})-{\mathit{x}}_i^{\top }({\widehat{\mathit{\delta }}}_{(ij)}-{\mathit{\delta }}_0)+{\mathrm{o}}_{L_k}(1),$$

and similarly for ${\tilde{r}}_{j,(j)}$, which follows from Theorem A1. Here ${\widehat{\mathit{\delta }}}_{(ij)}$ is computed by solving Problem (5) without $({\mathit{x}}_i,y_i)$ and $({\mathit{x}}_j,y_j)$. It is clear that ${\tilde{r}}_{i,(i)}$ and ${\tilde{r}}_{j,(j)}$ are asymptotically independent conditional on ${\mathit{X}}_{(ij)}$, i.e., the design matrix without the i-th and j-th rows.

Similarly, we have

$$\mathrm{E}\left({\displaystyle \sum _{k=1}^p}{|{\widehat{\delta }}_{(ij)}(k)-{\delta }_0(k)+\widehat{w}(k)-w_0(k)|}^3\right)=\mathrm{O}\left\{{\displaystyle \frac{\mathrm{polyLog}(n)n}{{(n^{1/2}\wedge n^{\mathrm{e}})}^3}}\right\}=\mathrm{o}(1).$$

Therefore, we also have

$${\displaystyle \sum _{k=1}^p}{|{\widehat{\delta }}_{(ij)}(k)-{\delta }_0(k)+\widehat{w}(k)-w_0(k)|}^3={\mathrm{o}}_p(1).$$

Note that ${\widehat{\mathit{\delta }}}_{(ij)}$ depends only on $\{{\mathit{X}}_{(ij)},{ϵ}_{(ij)}\}$. We call $P_{(ij)}$ the joint probability measure $P_{(ij)}={\prod }_{k\ne (i,j)}{P}_{{\mathit{x}}_k,{ϵ}_k}$, i.e., probability computed with respect to all our random variables except $({\mathit{x}}_i,{ϵ}_i)$ and $({\mathit{x}}_j,{ϵ}_j)$.

So we have found $E_{(ij)}^n$, which depends only on $({\mathit{X}}_{(ij)},{ϵ}_{(ij)})$, such that $P_{(ij)}(E_{(ij)}^n)\to 1$ and ${\sum }_{k=1}^p{|{\widehat{\mathit{\delta }}}_{(ij)}(k)-{\mathit{\delta }}_0(k)+\widehat{w}(k)-w_0(k)|}^3=\mathrm{o}(1)$ when $({\mathit{X}}_{(ij)},{\left\{{ϵ}_k\right\}}_{k\ne (i,j)})\in E_{(ij)}^n$. By treating $a_{n,p}$’s as deterministic quantities, the arguments we gave above then imply that, when $({\mathit{X}}_{(ij)},{ϵ}_{(ij)})\in E_{(ij)}^n$,

$${\mathcal{X}}_i^{\top }({\widehat{\mathit{\delta }}}_{(ij)}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0)|({\mathit{X}}_{(ij)},{ϵ}_{(ij)})\mathrm{behaves}\mathrm{like}(\parallel {\widehat{\mathit{\delta }}}_{(ij)}-{\mathit{\delta }}_0\parallel )\mathrm{N}.$$

We now use characteristic function arguments. Let ${\alpha }_i={\mathcal{X}}_i^{\top }({\widehat{\mathit{\delta }}}_{(ij)}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0)$ and ${\alpha }_j={\mathcal{X}}_j^{\top }({\widehat{\mathit{\delta }}}_{(ij)}-{\mathit{\delta }}_0+\widehat{\mathit{w}}-{\mathit{w}}_0)$.

Let $(w_i,w_j)\in {\mathbb{R}}^2$ be fixed and

$$\chi (w_i,w_j)=\mathrm{E}\left\{e^{i(w_1{\alpha }_i+w_2{\alpha }_j)}\right\}=\mathrm{E}\left\{e^{i(w_1{\alpha }_i+w_2{\alpha }_j)}\left(1_{E_{(ij)}^n}+1_{{\left[E_{(ij)}^n\right]}^c}\right)\right\}.$$

(A25)

Since $P(E_{(ij)}^n)=P_{(ij)}(E_{(ij)}^n)\to 1$, we can just focus on $\mathrm{E}\left\{e^{i(w_1{\alpha }_i+w_2{\alpha }_j)}{1}_{E_{(ij)}^n}\right\}$, since the modulus of the functions we are integrating is bounded by 1.

Now, we have

$$\mathrm{E}\left\{e^{i(w_1{\alpha }_i+w_2{\alpha }_j)}{1}_{E_{(ij)}^n}\right\}=\mathrm{E}\left[1_{E_{(ij)}^n}\mathrm{E}\left\{e^{i(w_1{\alpha }_i+w_2{\alpha }_j)}|{\mathit{X}}_{(ij)},{ϵ}_{(ij)}\right\}\right],$$

since $1_{E_{(ij)}^n}$ is a deterministic function of $({\mathit{X}}_{(ij)},{ϵ}_{(ij)})$. Independence of ${\mathcal{X}}_i$ and ${\mathcal{X}}_j$ gives

$$\mathrm{E}\left\{e^{i(w_1{\alpha }_i+w_2{\alpha }_j)}|{\mathit{X}}_{(ij)},{ϵ}_{(ij)}\right\}=\mathrm{E}\left(e^{i{w}_1{\alpha }_i}|{\mathit{X}}_{(ij)},{ϵ}_{(ij)}\right)\mathrm{E}\left(e^{i{w}_2{\alpha }_j}|{\mathit{X}}_{(ij)},{ϵ}_{(ij)}\right).$$

Also, the conditional Gaussian approximation established above implies that

$$1_{E_{(ij)}^n}\left\{\mathrm{E}\left(e^{i(w_1{\alpha }_i+w_2{\alpha }_j)}|{\mathit{X}}_{(ij)},{ϵ}_{(ij)}\right)-e^{-(w_1^2/2+w_2^2/2){\parallel {\widehat{\mathit{\delta }}}_{(ij)}-{\mathit{\delta }}_0\parallel }^2}\right\}\to 0$$

in $P_{(ij)}$-probability.

So we conclude that

$$\mathrm{E}\left\{1_{E_{(ij)}^n}{e}^{i(w_1{\alpha }_i+w_2{\alpha }_j)}\right\}-\mathrm{E}\left\{1_{E_{(ij)}^n}{e}^{-(w_1^2/2+w_2^2/2){\parallel {\widehat{\mathit{\delta }}}_{(ij)}-{\mathit{\delta }}_0\parallel }^2}\right\}\to 0.$$

Since $P(E_{(ij)}^n)\to 1$ and $\parallel {\widehat{\mathit{\delta }}}_{(ij)}-{\mathit{\delta }}_0{\parallel }^2$ is asymptotically deterministic by arguments similar to those in the proof of Proposition A4, we have

$$\mathrm{E}\left\{1_{E_{(ij)}^n}{e}^{-(w_1^2/2+w_2^2/2){\parallel {\widehat{\mathit{\delta }}}_{(ij)}-{\mathit{\delta }}_0\parallel }^2}\right\}-e^{-(w_1^2/2+w_2^2/2)\mathrm{E}(\parallel {\widehat{\mathit{\delta }}}_{(ij)}-{\mathit{\delta }}_0{\parallel }^2)}\to 0.$$

Therefore,

$$\mathrm{E}\left\{e^{i(w_1{\alpha }_i+w_2{\alpha }_j)}\right\}-\mathrm{E}\left(e^{i{w}_1{\alpha }_i}\right)\mathrm{E}\left(e^{i{w}_2{\alpha }_j}\right)\to 0.$$

This shows that ${\alpha }_i$ and ${\alpha }_j$ are asymptotically independent. It easily follows that ${\tilde{r}}_{i,(i)}$ and ${\tilde{r}}_{j,(j)}$ are asymptotically independent.

The same reasoning applies to $({\tilde{r}}_{i,(i)},{\lambda }_i)$ and $({\tilde{r}}_{j,(j)},{\lambda }_j)$, since ${\widehat{\mathit{\delta }}}_{(ij)}$ is independent of ${\lambda }_i$ and ${\lambda }_j$ under Assumption O6. □

Appendix E.5.2. On the Asymptotic Behavior of c τ

We now show that $c_{\tau }=n^{-1}\mathrm{tr}\left\{{(\mathit{S}+\tau {\mathrm{I}}_p)}^{-1}\right\}$ is asymptotically deterministic. This require several steps.

Lemma A13. 

We work under Assumptions  O1–O7  , P1–P4  and  F2–F4. Consider the random function

$$g_n(x)={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{1+x{\lambda }_i^2{\psi }^{\prime }\left\{\mathrm{prox}(x{\lambda }_i^2\rho )({\tilde{r}}_{i,(i)})\right\}}},\mathrm{defined}\mathrm{for}x\ge 0.$$

Let $B>0$ be in ${\mathbb{R}}_{+}$. We have, for any $(x,y)\in {\mathbb{R}}_{+}^2$, and $0\le x\le B$, $0\le y\le B$,

$$\underset{(x,y):|x-y|\le \mathit{\eta },0\le x\le B,0\le y\le B}{\sup }|g_n(x)-g_n(y)|\le \mathit{\eta }{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}({\lambda }_i^2\parallel {\psi }^{\prime }{\parallel }_{\infty }+B\mathrm{L}(n){\lambda }_i^4{\parallel \psi \parallel }_{\infty }).$$

In particular, under Assumption  P2   and  F3–F4, we have for $\mathrm{C}$ a constant independent of n and p,

$${\mathrm{Pr}}^{\ast }\left(\underset{(x,y):|x-y|\le \mathit{\eta },0\le x\le B,0\le y\le B}{\sup }|g_n(x)-g_n(y)|>\mathit{\delta }\right)\le {\displaystyle \frac{\mathit{\eta }}{\mathit{\delta }}}\mathrm{C}.$$

(A26)

Hence, $g_n$ is stochastic equicontinuous on $[0,B]$ for any $B>0$ given.

We used the notation ${\mathrm{Pr}}^{\ast }$ above to denote outer probability and avoid a discussion of potential measure theoretic issues associated with taking a supremum over a noncountable collection of random variables (see e.g., van der Vaart [39] (Sect. 18.2)).

Proof. 

Consider the function defined for $x\ge 0$,

$$h_u^{(i)}(x)={\displaystyle \frac{1}{1+x{\lambda }_i^2{\psi }^{\prime }\left\{\mathrm{prox}(x{\lambda }_i^2\rho )(u)\right\}}}={\displaystyle \frac{\partial }{\partial u}}\mathrm{prox}(x{\lambda }_i^2\rho )(u).$$

The last equality comes from Lemma 3.33 from El Karoui [21].

Since ${\psi }^{\prime }$ is non-negative,

$$\forall u,|h_u^{(i)}(x)-h_u^{(i)}(y)|\le |x{\lambda }_i^2{\psi }^{\prime }(\mathrm{prox}(x{\lambda }_i^2\rho )(u))-y{\lambda }_i^2{\psi }^{\prime }(\mathrm{prox}(y{\lambda }_i^2\rho )(u))|\wedge 1.$$

Thus, since $x,y\ge 0$, for all u,

$$\begin{array}{cc}|h_u^{(i)}(x)-h_u^{(i)}(y)|\le \hfill & {\lambda }_i^2|x-y|{\psi }^{\prime }(\mathrm{prox}(x{\lambda }_i^2\rho )(u))\hfill \\ \hfill & +{\lambda }_i^{2}y|{\psi }^{\prime }(\mathrm{prox}(x{\lambda }_i^2\rho )(u))-{\psi }^{\prime }(\mathrm{prox}(y{\lambda }_i^2\rho )(u))|.\hfill \end{array}$$

In particular, if $|x-y|\le \eta$, and $x\vee y\le B$, with $x,y\ge 0$, for all u,

$$\begin{array}{cc}\underset{y:|x-y|\le \eta ;x\vee y\le B}{\sup }|h_u^{(i)}(x)-h_u^{(i)}(y)|\le \hfill & {\lambda }_i^2\eta {\psi }^{\prime }\left\{\mathrm{prox}(x{\lambda }_i^2\rho )(u)\right\}\hfill \\ \hfill & +B{\lambda }_i^2\underset{y:|x-y|\le \eta ,x\vee y\le B}{\sup }|{\psi }^{\prime }\left\{\mathrm{prox}(x{\lambda }_i^2\rho )(u)\right\}-{\psi }^{\prime }\left\{\mathrm{prox}(y{\lambda }_i^2\rho )(u)\right\}|.\hfill \end{array}$$

Under Assumption O3, ${\psi }^{\prime }$ is $\mathrm{L}(n)$-Lipschitz. Therefore, for $x_i=x{\lambda }_i^2$, $y_i=y{\lambda }_i^2\ge 0$, we have

$$\forall u,|{\psi }^{\prime }\left\{\mathrm{prox}(x_i\rho )(u)\right\}-{\psi }^{\prime }\left\{\mathrm{prox}(y_i\rho )(u)\right\}|\le \mathrm{L}(n)|\mathrm{prox}(x_i\rho )(u)-\mathrm{prox}(y_i\rho )(u)|.$$

Recall that according to Lemma A11,

$${\displaystyle \frac{\partial }{\partial x}}\mathrm{prox}(x\rho )(u)=-{\displaystyle \frac{\psi \{\mathrm{prox}(x\rho )(u)\}}{1+x{\psi }^{\prime }\left\{\mathrm{prox}(x\rho )(u)\right\}}}.$$

Hence we have

$$\underset{x}{\sup }|{\displaystyle \frac{\partial }{\partial x}}\mathrm{prox}(x\rho )(u)|\le {\parallel \psi \parallel }_{\infty }.$$

We conclude that

$$\forall u,|{\psi }^{\prime }\left\{\mathrm{prox}(x_i\rho )(u)\right\}-{\psi }^{\prime }\left\{\mathrm{prox}(y_i\rho )(u)\right\}|\le \{\mathrm{L}(n){\parallel \psi \parallel }_{\infty }|x_i-y_i|\}\wedge {2\parallel \psi \parallel }_{\infty }.$$

We therefore have, for $x\vee y\le B$ and $x,y\ge 0$,

$$\forall u,\underset{y:|x-y|\le \eta }{\sup }|h_u^{(i)}(x)-h_u^{(i)}(y)|\le {\lambda }_i^2\eta {\psi }^{\prime }\left\{\mathrm{prox}(x{\lambda }_i^2\rho )(u)\right\}+B{\lambda }_i^4\mathrm{L}(n){\parallel \psi \parallel }_{\infty }\eta .$$

Thus, for $x,y\ge 0$,

$$\forall u,\underset{(x,y):|x-y|\le \eta ,x\vee y\le B}{\sup }|h_u^{(i)}(x)-h_u^{(i)}(y)|\le {\lambda }_i^2\eta \parallel {\psi }^{\prime }{\parallel }_{\infty }+B{\lambda }_i^4\mathrm{L}(n){\parallel \psi \parallel }_{\infty }\eta .$$

Since the right-hand side does not depend on u, we have

$$\underset{u}{\sup }\underset{(x,y):|x-y|\le \eta ,x\vee y\le B}{\sup }|h_u^{(i)}(x)-h_u^{(i)}(y)|\le {\lambda }_i^2\eta \parallel {\psi }^{\prime }{\parallel }_{\infty }+B{\lambda }_i^4\mathrm{L}(n){\parallel \psi \parallel }_{\infty }\eta .$$

Naturally, $g_n(x)$ can be written as

$$g_n(x)={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{h}_{{\tilde{r}}_{i,(i)}}^{(i)}(x).$$

Therefore, for any $x,y\ge 0$,

$$|g_n(x)-g_n(y)|\le {\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}|h_{{\tilde{r}}_{i,(i)}}^{(i)}(x)-h_{{\tilde{r}}_{i,(i)}}^{(i)}(y)|.$$

The bound we have obtained above on ${\sup }_u|h_u^{(i)}(x)-h_u^{(i)}(y)|$ when x and y are sufficiently close to one another can now be used. This shows that for x given, if $x,y\ge 0$, $|x-y|\le \eta$, and $x\vee y\le B$, we have

$$\underset{(x,y):|x-y|\le \eta ,0\le x\le B,0\le y\le B}{\sup }|g_n(x)-g_n(y)|\le \eta {\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}({\lambda }_i^2\parallel {\psi }^{\prime }{\parallel }_{\infty }+B\mathrm{L}(n){\lambda }_i^4{\parallel \psi \parallel }_{\infty }).$$

Under Assumptions P2 and F3–F4, all the terms on the right-hand side are bounded in $L_1$. We can now take expectations and obtain the result in $L_1$. □

Lemma A14. 

Let us call $G_n(x)=\mathrm{E}(g_n(x))$. Let $B>0$ be given. For any given $x_0\le B$,

$$g_n(x_0)-G_n(x_0)={\mathrm{o}}_{L_2}(1).$$

Under Assumptions  O1–O7, P1–P4  and  F1–F5, we have

$${\mathrm{E}}^{\ast }\left(\underset{0\le x\le B}{\sup }|g_n(x)-G_n(x)|\right)\to 0.$$

Proof. 

Under Assumptions F1 and F5, we can divide the index set $\{1,\dots ,n\}$ into finite K subsets $A_1,\dots ,A_K$, in which ${({\mathit{x}}_i,{ϵ}_i)}_{i\in A_j}$ play a symmetric role. Hence, $\mathrm{var}(g_n(x_0))$ can be expressed as a sum of variances and covariances of finitely many functions of finitely many random variables $({\lambda }_i,{\tilde{r}}_{i,(i)})$: for those random variables, we just need to pick a representative in each subset ${\left\{A_j\right\}}_{j=1}^K$.

Since ${\psi }^{\prime }$ is Lipschitz, $g_n$ is an average of bounded continuous functions of the random variables of interest to us.

Asymptotic pairwise independence of $({\lambda }_i,{\tilde{r}}_{i,(i)})$’s implies that

$$\mathrm{var}(g_n(x_0))=\mathrm{o}(1).$$

and therefore gives the first result.

Now we pick $ϵ>0$. By the stochastic equicontinuity of $g_n$ and the bound in (A26), we can find $x_1,\dots ,x_K$, independent of n, such that for all $x\in [0,B]$, there exists l such that when n is large enough,

$$\mathrm{E}(|g_n(x)-g_n(x_l)|)\le ϵ.$$

Notice that

$$|g_n(x)-G_n(x)|\le |g_n(x)-g_n(x_l)|+|g_n(x_l)-G_n(x_l)|+|G_n(x_l)-G_n(x)|.$$

We immediately get

$${\mathrm{E}}^{\ast }\left(\underset{0\le x\le B}{\sup }|g_n(x)-G_n(x)|\right)\le 2ϵ+\mathrm{E}\left(\underset{1\le l\le K}{\sup }|g_n(x_l)-G_n(x_l)|\right).$$

Because K is finite, the fact that for all l, $g_n(x_l)-G_n(x_l)={\mathrm{o}}_{L_2}(1)$ implies that ${\sup }_{1\le l\le K}|g_n(x_l)-G_n(x_l)|={\mathrm{o}}_{L_2}(1)$. In particular, if n is sufficiently large, we have

$$\mathrm{E}\left(\underset{1\le l\le K}{\sup }|g_n(x_l)-G_n(x_l)|\right)\le ϵ.$$

This gives the result. □

Lemma A15. 

Assume  O1–O7,  P1–P4   and  F1–F5. Recall that $c_{\tau }=n^{-1}\mathrm{tr}\left\{{(\mathit{S}+\tau {\mathrm{I}}_p)}^{-1}\right\}$. Call as before

$$g_n(x)={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{1+x{\lambda }_i^2{\psi }^{\prime }\left\{\mathrm{prox}(x{\lambda }_i^2\rho )({\tilde{r}}_{i,(i)})\right\}}}.$$

Then $c_{\tau }$ is a near solution of

$${\displaystyle \frac{p}{n}}-\tau x-1+g_n(x)=0,$$

i.e., $p/n-\tau c_{\tau }-1+g_n(c_{\tau })={\mathrm{o}}_{L_k}(1)$, when $3\alpha -1/2<0$.

Asymptotically, near solution of

$${\mathit{\delta }}_n(x)\triangleq {\displaystyle \frac{p}{n}}-\tau x-1+g_n(x)=0,$$

are close to solutions of

$${\mathrm{\Delta }}_n(x)={\displaystyle \frac{p}{n}}-\tau x-1+\mathrm{E}\left\{g_n(x)\right\}=0.$$

More precisely, call $T_{n,ϵ}=\{x:|{\mathrm{\Delta }}_n(x)|\le ϵ\}$. We note that $T_{n,ϵ}\subseteq (0,p/(n\tau )+ϵ/\tau )$. For any given ϵ, as $n\to \infty$, near solutions of ${\mathit{\delta }}_n(x)=0$ belong to $T_{n,ϵ}$ with high probability.

Our assumptions concerning the possible distributions of ${ϵ}_i$’s, specifically Assumption  F1, imply that as $n\to \infty$, there is a unique solution to ${\mathrm{\Delta }}_n(x)=0$.

Hence $c_{\tau }$ is asymptotically deterministic.

Proof. 

Note that $g_n(x)\le 1$.

Let ${\mathit{\delta }}_n(x)$ be the function

$${\mathit{\delta }}_n(x)={\displaystyle \frac{p}{n}}-\tau x-1+g_n(x),$$

and ${\mathrm{\Delta }}_n(x)=\mathrm{E}\left\{{\mathit{\delta }}_n(x)\right\}$. Call $x_n$ a solution of ${\mathit{\delta }}_n(x_n)=0$ and $x_{n,0}$ a solution of ${\mathrm{\Delta }}_n(x_{n,0})=0$.

Since $0\le g_n(x)\le 1$, we have $x_n\le p/(n\tau )$, for otherwise, ${\mathit{\delta }}_n(x)<0$. The same arguments shows that if $x>(p/n+ϵ)/\tau$, then ${\mathit{\delta }}_n(x)<-ϵ$ and $x\notin T_{n,ϵ}$. Similarly, near solutions of ${\mathit{\delta }}_n(x)=0$ must be less or equal to $(p/n+ϵ)/\tau$.

• Proof of the fact that ${\mathit{c}}_{\mathit{\tau }}$ is such that ${\mathit{\delta }}_{\mathit{n}}({\mathit{c}}_{\mathit{\tau }})=\mathrm{o}(\mathbf{1})$

Recall that in the notation of Lemma A8, we have

$${\displaystyle \frac{p-1}{n}}-\tau {\mathrm{c}}_{\tau ,p}={\displaystyle \frac{1}{n}}\mathrm{tr}({\mathrm{I}}_n-\mathit{M}).$$

According to (A20),

$${\displaystyle \frac{1}{n}}\mathrm{tr}({\mathrm{I}}_n-\mathit{M})=1-{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{1+{\psi }^{\prime }(r_{i,\left[p\right]})\frac{1}{n}{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p(i)+\tau {\mathrm{I}}_p)}^{-1}{\mathit{v}}_i}}.$$

According to Lemmas A8–A10, we have

$$\underset{i}{\sup }|{\displaystyle \frac{1}{n}}{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p(i)+\tau {\mathrm{I}}_p)}^{-1}{\mathit{v}}_i-{\lambda }_i^2{\mathrm{c}}_{\tau ,p}|={\mathrm{O}}_{L_k}\left({\displaystyle \frac{\mathrm{polyLog}(n)}{n^{1/2-2\alpha }}}\right).$$

When $x\ge 0$ and $y\ge 0$, $|1/(1+x)-1/(1+y)|\le |x-y|\wedge 1$. Hence, we have

$$\begin{array}{cc}\hfill & |{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{1+{\psi }^{\prime }(r_{i,\left[p\right]})\frac{1}{n}{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p(i)+\tau {\mathrm{I}}_p)}^{-1}{\mathit{v}}_i}}-{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{1+{\psi }^{\prime }(r_{i,\left[p\right]}){\lambda }_i^2{\mathrm{c}}_{\tau ,p}}}|\hfill \\ \hfill & \le \underset{1\le i\le n}{\sup }|{\displaystyle \frac{1}{n}}{\mathit{v}}_i^{\top }{({\mathfrak{S}}_p(i)+\tau {\mathrm{I}}_p)}^{-1}{\mathit{v}}_i-{\lambda }_i^2{\mathrm{c}}_{\tau ,p}|{\parallel {\psi }^{\prime }\parallel }_{\infty }.\hfill \end{array}$$

We conclude that

$$p/n-\tau {\mathrm{c}}_{\tau ,p}-1+{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{1+{\lambda }_i^2{\mathrm{c}}_{\tau ,p}{\psi }^{\prime }(r_{i,\left[p\right]})}}={\mathrm{O}}_{L_k}(n^{-1/2+2\alpha }\mathrm{polyLog}(n)).$$

Exactly the same computations can be performed for $c_{\tau }$. We have established that

$$p/n-\tau c_{\tau }-1+{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{1+{\lambda }_i^2{c}_{\tau }{\psi }^{\prime }(R_i)}}={\mathrm{O}}_{L_k}(n^{-1/2+2\alpha }\mathrm{polyLog}(n)).$$

Now we have seen in Theorem A1 that

$$\underset{i}{\sup }|R_i-\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})|={\mathrm{O}}_{L_k}(n^{-1/2+\alpha }\mathrm{polyLog}(n)).$$

By the assumptions on ${\psi }^{\prime }$, this implies that

$$\underset{i}{\sup }|{\psi }^{\prime }(R_i)-{\psi }^{\prime }\left\{\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})\right\}|={\mathrm{O}}_{L_k}(n^{-1/2+\alpha }\mathrm{polyLog}(n)).$$

We have furthermore noted that ${\sup }_i|c_i-{\lambda }_i^2{c}_{\tau }|={\mathrm{O}}_{L_k}(n^{-1/2+2\alpha }\mathrm{polyLog}(n))$ in Corollary A1. Using Lemma A11, we can write

$$|\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})-\mathrm{prox}({\lambda }_i^2{c}_{\tau }\rho )({\tilde{r}}_{i,(i)}){|\le \parallel \psi \parallel }_{\infty }|c_i-{\lambda }_i^2{c}_{\tau }|$$

and hence

$$|{\psi }^{\prime }\left\{\mathrm{prox}(c_i\rho )({\tilde{r}}_{i,(i)})\right\}-{\psi }^{\prime }\left\{\mathrm{prox}({\lambda }_i^2{c}_{\tau }\rho )({\tilde{r}}_{i,(i)})\right\}|={\mathrm{O}}_{L_k}{(\parallel \psi \parallel }_{\infty }{n}^{-1/2+3\alpha }\mathrm{polyLog}(n)).$$

Gathering all these results, we have

$$|{\psi }^{\prime }(R_i)-{\psi }^{\prime }(\mathrm{prox}({\lambda }_i^2{c}_{\tau }\rho )({\tilde{r}}_{i,(i)}))|={\mathrm{O}}_{L_k}{\{(\parallel \psi \parallel }_{\infty }+1)n^{-1/2+3\alpha }\mathrm{polyLog}(n)\}.$$

So we have shown that ${\mathit{\delta }}_n(c_{\tau })={\mathrm{O}}_{L_k}(n^{-1/2+3\alpha }\mathrm{polyLog}(n))$.

• Final details

By Lemma A14, we have ${\mathit{\delta }}_n(x)-{\mathrm{\Delta }}_n(x)={\mathrm{o}}_p(1)$ for any given x. In our case, using the notation of this lemma, $B=p/(n\tau )+\eta /\tau$, for $\eta >0$ given.

This implies that for any given $ϵ>0$,

$$\underset{x\in (0,p/(n\tau )+\eta /\tau )}{\sup }|{\mathit{\delta }}_n(x)-{\mathrm{\Delta }}_n(x)|<ϵ,$$

with high probability when n is large enough. Therefore, for any $ϵ>0$, if $x_n$ is a solution to ${\mathit{\delta }}_n(x)=0$,

$$|{\mathrm{\Delta }}_n(x_n)|<ϵ\mathrm{with}\mathrm{high}\mathrm{probability}.$$

This means that $x_n\in T_{n,ϵ}$ with high probability. The same reasoning applies for near solution of ${\mathit{\delta }}_n(x)=0$, which must belong to $T_{n,ϵ}$ as $n\to \infty$ with high probability for any given $ϵ>0$. Note that $T_{n,ϵ}$ is compact because it is bounded and closed, using the fact that $G_n=\mathrm{E}(g_n)$ is continuous.

If $T_{n,0}$ were reduced to a single point, we would have established the asymptotically deterministic character of $c_{\tau }$.

Given our work on the asymptotic behavior of ${\tilde{r}}_{i,(i)}$ and our assumptions on ${ϵ}_i$’s, we see that Lemma 3.39 from El Karoui [21] applies to ${\lim }_{n\to \infty }{\mathrm{\Delta }}_n(x)$ under Assumption F1. Therefore, $T_{n,0}$ is reduced to a single point as $n\to \infty$ and $c_{\tau }$ is asymptotically deterministic. □

Proof of Theorem 1. 

Notice that

$${\displaystyle \frac{\partial }{\partial t}}\mathrm{prox}(c\rho )(t)=\mathrm{prox}{(c\rho )}^{\prime }(t)={\displaystyle \frac{1}{1+c{\psi }^{\prime }(\mathrm{prox}(c\rho )(t))}}.$$

Therefore, ${\mathrm{\Delta }}_n$ can be written as

$${\mathrm{\Delta }}_n(x)={\displaystyle \frac{p}{n}}-\tau x-1+{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\mathrm{E}\left\{{\mathrm{prox}}^{\prime }(x{\lambda }_i^2\rho )({\tilde{r}}_{i,(i)})\right\}.$$

Hence, the limiting root of ${\mathrm{\Delta }}_n(x)=0$ satisfies the first fixed-point equation in Theorem 1. Since Lemma A15 shows that $c_{\tau }$ is asymptotically arbitrarily close to this root, the first equation follows. The second equation comes from (A24). Theorem 1 is proved, with $c_{\rho }(\kappa )$ being the limit of $c_{\tau }$. □