Robust Distributed High-Dimensional Regression: A Convoluted Rank Approach

Abstract

This paper investigates robust high-dimensional convoluted rank regression in distributed environments. We propose an estimation method suitable for sparse regimes, which remains effective under heavy-tailed errors and outliers, as it does not impose moment assumptions on the noise distribution. To facilitate scalable computation, we develop a local linear approximation algorithm, enabling fast and stable optimization in high-dimensional settings and across distributed systems. Our theoretical results provide non-asymptotic error bounds for both one-round and multi-round communication schemes, explicitly quantifying how estimation accuracy improves with additional communication rounds. Specifically, after a number of communication rounds (logarithmic in the number of machines), the proposed estimator achieves the minimax-optimal convergence rate, up to logarithmic factors. Extensive simulations further demonstrate stable performance across a wide range of error distributions, with accurate coefficient estimation and reliable support recovery.

1. Introduction

In the era of big data, the exponential growth in data volume and complexity has made traditional single-machine approaches insufficient for effective statistical modeling. To overcome this challenge, distributed algorithms break down data into manageable chunks and distribute them across multiple computing nodes for parallel processing. This approach greatly improves computational efficiency and scalability, proving essential for handling large-scale data analysis [1,2]. Distributed algorithms are widely used in various statistical learning tasks. For example, Wang et al. [3] studied distributed inference for the linear support vector machine (SVM) in the binary classification task and proposed a multi-round distributed linear-type estimator to achieve computational efficiency in conducting inference for the linear SVM. Chen et al. [4] established a new connection between quantile regression and ordinary linear regression and provided a distributed estimator that is both computationally and communicationally efficient. Chen and Peng [5] explored distributed statistical inference for a general class of statistics, including U-statistics and M-estimators, in the context of massive data. Additionally, concerns over privacy and regulatory compliance often result in data being stored across multiple institutions, where centralized aggregation is either impractical or prohibited. These challenges have further driven the development of distributed estimation methods that allow for statistical inference without requiring direct access to raw data [6].

A common class of distributed estimation methods follows the one-shot paradigm, where each machine independently computes a local estimator, and the final estimate is obtained through a single round of averaging [1,7,8]. While these methods are attractive due to their simplicity and low communication cost, they often require each machine to hold a sufficiently large number of samples to ensure accurate global estimation. When the local sample size is limited or the number of machines is large, their statistical performance can degrade significantly. To address the limitations of one-shot estimators under limited local sample sizes, Jordan et al. [2] proposed the Communication-Efficient Surrogate Likelihood (CSL) framework. For this approach, each machine computes the gradient of its local loss at a shared parameter value and sends it to a central node, which then constructs a surrogate loss function to approximate the global objective. The estimator is refined iteratively, leading to improved statistical accuracy with logarithmic communication rounds. Inspired by this, several works have extended the CSL method to high-dimensional settings, such as sparse linear models, generalized linear models, quantile regression and modal regression [9,10,11,12,13,14].

In the centralized setting, a substantial body of work has developed robust high-dimensional regression methods with strong theoretical guarantees in the presence of heavy-tailed errors and outliers. Among these methods, Wilcoxon rank regression (also referred to as rank regression) stands out. As introduced by Hettmansperger and McKean [15], the rank regression estimator achieves arbitrarily high relative efficiency under heavy-tailed error distributions while retaining at least 86.4% asymptotic relative efficiency under any symmetric error distribution with finite Fisher information. This combination of robustness and high estimation efficiency makes rank regression a powerful and attractive alternative to standard least squares. Consequently, rank regression has garnered increased attention in recent years. For instance, Wang et al. [16] proposed a tuning-free robust high-dimensional regression procedure based on a simulated tuning parameter, achieving near-oracle performance and strong robustness to heavy-tailed errors. Zhou et al. [17] developed sparse convoluted rank regression by smoothing the rank loss, establishing non-asymptotic properties and presenting an efficient algorithm for high-dimensional settings, in contrast to canonical rank regression. Cai et al. [18] explored statistical inference for high-dimensional convoluted rank regression, deriving estimation error bounds and constructing debiased estimators with valid simultaneous confidence intervals. Building on this line of work, we propose a distributed high-dimensional convoluted rank regression (DCR). Our main contributions are summarized as follows.

• (Methodology). In the CSL framework, we propose a robust, communication-efficient, penalized estimator for high-dimensional linear regression. The proposed procedure does not impose moment assumptions on the error distribution and therefore remains applicable in the presence of heavy-tailed noise and outliers. It is tailored for sparse high-dimensional regimes and delivers accurate coefficient estimation, along with reliable support recovery. Compared with the distributed Huber regression approach proposed by Luo et al. [14], our method does not require the noise to have a finite second moment and avoids the need to tune the two key hyperparameters in the Huber loss. As a result, it is computationally more efficient and easier to implement in practice. Extensive simulation studies in Section 5 demonstrate that the method is stable across a wide range of error distributions and achieves strong empirical performance in both estimation and support recovery.
• (Algorithm). To solve the resulting optimization problem efficiently, we develop a local linear approximation-based algorithm. The algorithm converts the original objective into a sequence of tractable subproblems and can be implemented efficiently in distributed environments, leading to fast and stable computation in high dimensions.
• (Theory). We establish non-asymptotic theoretical guarantees for the proposed estimator under both one-round and multiple-round communication schemes. Specifically, we derive non-asymptotic error bounds characterizing how the accuracy improves as the number of communication rounds increases. After a sufficient number of communication rounds (logarithmic in the number of machines), the distributed estimator attains the minimax convergence rate (up to logarithmic factors).

The rest of the paper is organized as follows. Section 2 introduces the DCR estimator. Section 3 presents an optimization algorithm for the DCR estimator. Section 4 provides a comprehensive theoretical study of the DCR estimator. Numerical results, including simulation studies and real data analysis, are given in Section 5. All technical proofs are provided in Appendix A, Appendix B and Appendix C.

Notation. For a positive integer m, let $[m]=\{1,\dots ,m\}$. Denote by ${\mathbb{S}}^{d-1}$, $I(·)$, ${\mathit{e}}_d$ and ${\mathbf{I}}_d$ the d-dimensional unit sphere, the indicator function, the identity vector and the $d\times d$ identity matrix, respectively. For any $m_1\times m_2$ matrix $\mathbf{H}={(h_{i,j})}_{m_1\times m_2}$, let ${\parallel \mathbf{H}\parallel }_2$ denote the spectral norm of matrix $\mathbf{H}$. Write ${\parallel \mathbf{H}\parallel }_{\infty }={max}_{i\in [m_1],j\in [m_2]}\left|h_{i,j}\right|$. Specifically, if $m_2=1$, we use ${|\mathbf{H}|}_1={\sum }_{i=1}^{m_1}|h_{i,1}|$ and ${|\mathbf{H}|}_2={({\sum }_{i=1}^{m_1}{h}_{i,1}^2)}^{1/2}$ to denote, respectively, the $L_1$-norm and $L_2$-norm of the $m_1$-dimensional vector $\mathbf{H}$. If $\mathbf{H}$ is positive semi-definite and $m_1=m_2$, for any vector $\mathit{u}\in {\mathbb{R}}^{m_1}$, let ${|\mathit{u}|}_{\mathbf{H}}={|{\mathbf{H}}^{1/2}\mathit{u}|}_2$. Define the ball ${\mathbb{B}}_{\mathbf{H}}(r)=\{\mathit{u}\in {\mathbb{R}}^{m_1}{:|\mathit{u}|}_{\mathbf{H}}\le r\}$ and its translation ${\mathbb{B}}_{\mathbf{H}}(\mathit{v},r)=\mathit{v}+{\mathbb{B}}_{\mathbf{H}}(r)$ for any $r>0$ and $\mathit{v}\in {\mathbb{R}}^{m_1}$. For a countable set S, we use $|S|$ to denote its cardinality. For any two real numbers a and b, we write $a\wedge b=min(a,b)$ and $a\vee b=max(a,b)$. For two sequences of positive numbers $\{a_n\}$ and $\{b_n\}$, we write $a_n\lesssim b_n$ or $b_n\gtrsim a_n$ if there exist a positive constant c and a large enough integer $n_0$ such that $a_n/b_n\le c$ for all $n\ge n_0$. Write $a_n\asymp b_n$ if and only if $a_n\lesssim b_n$ and $b_n\lesssim a_n$ hold simultaneously. We use $C,C_1,\dots$ to denote some generic positive constants, which may be different in different uses. Let $\mathbb{B}(r)=\{\mathit{u}\in {\mathbb{R}}^k:|u{|}_2\le r\}$ be the Euclidean ball with radius r. Denote the zero-mean operator $(1-\mathbb{E})(\mathit{\zeta })=\mathit{\zeta }-\mathbb{E}(\mathit{\zeta })$ for any random matrix $\mathit{\zeta }$.

2. Distributed High-Dimensional Convoluted Rank Regression

Suppose that the sample of i.i.d. observations ${\{(Y_i,{\mathbf{X}}_i)\}}_{i=1}^N$ are generated from the following linear regression model,

$$Y_i={\mathbf{X}}_i^{\top }{\mathit{\beta }}^{*}+{\epsilon }_i,$$

where ${\{Y_i\}}_{i=1}^N$ are one-dimensional responses, ${\{{\epsilon }_i\}}_{i=1}^N$ are random errors, ${\{{\mathbf{X}}_i\}}_{i=1}^N$ are p-dimensional random vectors with mean zero and covariance matrix $\Sigma$ and ${\mathit{\beta }}^{*}={({\beta }_1^{*},\dots ,{\beta }_p^{*})}^{\top }\in {\mathbb{R}}^p$ is the unknown parameter of interest.

We begin by introducing the global convoluted rank regression, which forms the basis for our distributed extension. Given a sample of observations ${\{(Y_i,{\mathbf{X}}_i)\}}_{i=1}^N$, the objective is to estimate the regression coefficient vector ${\mathit{\beta }}^{*}\in {\mathbb{R}}^p$ by minimizing a smoothed version of the canonical rank loss. Following the formulation in Zhou et al. [17], we define the convolution-based rank loss as

$${\widehat{\mathcal{L}}}_h(\mathit{\beta })={\displaystyle \frac{1}{N(N-1)}}\sum _{i=1}^N\sum _{j\ne i}{L}_h((Y_i-{\mathbf{X}}_i^{\top }\mathit{\beta })-(Y_j-{\mathbf{X}}_j^{\top }\mathit{\beta })),$$

where $L_h(·)$ is a kernel-smoothed convex loss function defined as

$$L_h(u)={\int }_{-\infty }^{\infty }|u-v|K_h(v)\mathrm{d}v,$$

where $K_h(·)=h^{-1}K(·/h)$ with $K(·)$ being the kernel function and $h>0$ being the bandwidth. We define the population parameter as

$${\mathit{\beta }}_h^{*}\in \underset{\beta \in {\mathbb{R}}^p}{\mathrm{argmin}}\mathbb{E}\{{\widehat{\mathcal{L}}}_h(\mathit{\beta })\}.$$

By Theorem 1 of Zhou et al. [17], we have ${\mathit{\beta }}_h^{*}={\mathit{\beta }}^{*}$ for any $h>0$. Thus, in high-dimensional settings ($p\gg n$), we introduce the global penalized convoluted rank regression estimator for ${\mathit{\beta }}^{*}$ with entrywise regularizations as follows:

$$\widehat{\mathit{\beta }}\in \underset{\mathit{\beta }\in {\mathbb{R}}^p}{\mathrm{argmin}}\left\{{\widehat{\mathcal{L}}}_h(\mathit{\beta })+\sum _{j=1}^p{p}_{\lambda }(|{\beta }_j|)\right\},$$

where ${\beta }_j$ is the j-th element of $\mathit{\beta }\in {\mathbb{R}}^p$ and $p_{\lambda }(·)$ is a penalty function with $\lambda >0$ being a tuning parameter.

In the distributed setting, assume that the entire dataset ${\{(Y_i,{\mathbf{X}}_i)\}}_{i=1}^N$ is stored across m node machines: one central machine and $m-1$ local machines connected to the central machine. For $j\in [m]$, the j-th machine stores a subsample of $n_j$ observations, denoted by ${\{(Y_i,{\mathbf{X}}_i)\}}_{i\in {\mathcal{I}}_j}$, where ${\mathcal{I}}_j$ are disjoint index sets such that ${\cup }_{j=1}^m{\mathcal{I}}_j=[N]$ and $N={\sum }_{j=1}^m|{\mathcal{I}}_j|={\sum }_{j=1}^m{n}_j$. Without loss of generality, we assume that $n_1=\cdots =n_m=n$, so that $N=nm$. We refer to n as the local sample size.

Given bandwidth $h>0$, we define the local convoluted rank loss as

$$\begin{array}{c}{\widehat{\mathcal{L}}}_{j,h}(\mathit{\beta })={\displaystyle \frac{1}{n(n-1)}}\sum _{i\in {\mathcal{I}}_j}\sum _{k\in {\mathcal{I}}_j:k\ne i}{L}_h((Y_i-{\mathbf{X}}_i^{\top }\mathit{\beta })-(Y_k-{\mathbf{X}}_k^{\top }\mathit{\beta })).\end{array}$$

(1)

Starting with an initial estimator ${\widehat{\mathit{\beta }}}^{(0)}$ for ${\mathit{\beta }}^{*}$, we define the surrogate loss as

$${\widehat{\mathcal{L}}}_h^{(t)}(\mathit{\beta })={\widehat{\mathcal{L}}}_{1,h}(\mathit{\beta })-⟨\nabla {\widehat{\mathcal{L}}}_{1,h}({\widehat{\mathit{\beta }}}^{(t-1)})-\nabla {\widehat{\mathcal{L}}}_h({\widehat{\mathit{\beta }}}^{(t-1)}),\mathit{\beta }⟩.$$

(2)

We allow the parametric dimension p to be much larger than the local sample size n. Denote by $\mathcal{S}=\mathrm{supp}({\mathit{\beta }}^{*})=\{j\in [p]:{\beta }_j^{*}\ne 0\}$ the true active set. Hence, ${\mathit{\beta }}^{*}$ is s-sparse with $s=|\mathcal{S}|$. At iteration $t\in [T]$ for integer $T>0$, the first machine solves the following minimization problem:

$$\begin{array}{c}\hfill {\widehat{\mathit{\beta }}}^{(t)}\in \underset{\beta \in {\mathbb{R}}^p}{\mathrm{argmin}}\left\{{\widehat{\mathcal{L}}}_h^{(t)}(\mathit{\beta })+\sum _{j=1}^p{p}_{{\lambda }_t}(|{\beta }_j|)\right\},\end{array}$$

(3)

where ${\lambda }_t>0$ is a tuning parameter. We choose ${\widehat{\mathit{\beta }}}^{(0)}$ to be the local penalized convoluted rank regression estimator such that

$$\begin{array}{c}\hfill {\widehat{\mathit{\beta }}}^{(0)}\in \underset{\beta \in {\mathbb{R}}^p}{\mathrm{argmin}}\left\{{\widehat{\mathcal{L}}}_{1,h}(\mathit{\beta })+\sum _{j=1}^p{p}_{{\lambda }_0}(|{\beta }_j|)\right\}.\end{array}$$

In the next section, we will propose an optimization algorithm for solving (3).

3. Optimization Algorithm

In this section, we focus on the iterations after the $(t-1)$-th communication round. At the $(k+1)$-th iteration, we use the local linear approximation (LLA) algorithm to update the estimate by solving (3) as follows:

$$\begin{array}{c}\hfill {\widehat{\mathit{\beta }}}_{k+1}^{(t)}\in \underset{\beta \in {\mathbb{R}}^p}{\mathrm{argmin}}\left\{{\widehat{\mathcal{L}}}_h^{(t)}(\mathit{\beta })+\sum _{j=1}^p{p}_{{\lambda }_t}^{\prime }(|{\beta }_{k,j}^{(t)}|)|{\beta }_j|\right\},\end{array}$$

where ${\mathit{\beta }}_k^{(t)}={({\beta }_{k,1}^{(t)},\dots ,{\beta }_{k,p}^{(t)})}^{\top }$ is the previous estimate in the k-th iteration. Specifically, in a coordinate-wise manner, we update the l-th coordinate ${\widehat{\beta }}_{l,k+1}^{(t)}$ by minimizing

$$\begin{array}{cc}\hfill F({\widehat{\beta }}_{l,k+1}^{(t)}|{\beta }_k^{(t)})=& {\displaystyle \frac{1}{n(n-1)}}\sum _{i\in {\mathcal{I}}_1}\sum _{j\in {\mathcal{I}}_1:j\ne i}{L}_h({\omega }_{i,j}-(X_{i,l}-X_{j,l})({\widehat{\beta }}_{l,k+1}^{(t)}-{\beta }_{l,k}^{(t)}))\hfill \\ \hfill & -\left\{{\nabla }_l{\widehat{\mathcal{L}}}_{1,h}({\widehat{\mathit{\beta }}}^{(t-1)})-{\nabla }_l{\widehat{\mathcal{L}}}_h({\widehat{\mathit{\beta }}}^{(t-1)})\right\}{\widehat{\beta }}_{l,k+1}^{(t)}+p_{{\lambda }_t}^{\prime }(|{\beta }_{l,k}^{(t)}|)|{\widehat{\beta }}_{l,k+1}^{(t)}|,\hfill \end{array}$$

where ${\widehat{\mathit{\beta }}}^{(t-1)}$ is defined in (3) and ${\omega }_{i,j}=Y_i-Y_j-{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }{\widehat{\mathit{\beta }}}_k^{(t)}$. We let

$$\begin{array}{c}{\upsilon }_{l,1}=-{\displaystyle \frac{1}{n(n-1)}}\sum _{i\in {\mathcal{I}}_1}\sum _{j\in {\mathcal{I}}_1:i\ne j}{L}_h^{\prime }({\omega }_{i,j})(X_{i,l}-X_{j,l}),\end{array}$$

(4)

$$\begin{array}{c}{\upsilon }_{l,2}={\displaystyle \frac{1}{n(n-1)}}\sum _{i\in {\mathcal{I}}_1}\sum _{j\in {\mathcal{I}}_1:i\ne j}{(X_{i,l}-X_{j,l})}^2,\end{array}$$

(5)

$$\begin{array}{c}{\upsilon }_{l,3}={\nabla }_l{\widehat{\mathcal{L}}}_{1,h}({\widehat{\mathit{\beta }}}^{(t-1)})-{\nabla }_l{\widehat{\mathcal{L}}}_h({\widehat{\mathit{\beta }}}^{(t-1)}).\end{array}$$

(6)

By Lemma 1 of Zhou et al. [17], we can construct a quadratic majorization function for $F({\widehat{\beta }}_{l,k+1}^{(t)}|{\beta }_k^{(t)})$:

$$\begin{array}{cc}\hfill G({\widehat{\beta }}_{l,k+1}^{(t)}|{\beta }_k^{(t)})=& {\displaystyle \frac{1}{n(n-1)}}\sum _{i\in {\mathcal{I}}_1}\sum _{j\ne i\in {\mathcal{I}}_1}{L}_h({\omega }_{i,j})+{\upsilon }_{l,1}({\widehat{\beta }}_{l,k+1}^{(t)}-{\beta }_{l,k}^{(t)})\hfill \\ \hfill & +{\displaystyle \frac{k_6{\upsilon }_{l,2}}{h}}{({\widehat{\beta }}_{l,k+1}^{(t)}-{\beta }_{l,k}^{(t)})}^2-{\upsilon }_{l,3}{\widehat{\beta }}_{l,k+1}^{(t)}+p_{{\lambda }_t}^{\prime }(|{\beta }_{l,k}^{(t)}|)|{\widehat{\beta }}_{l,k+1}^{(t)}|.\hfill \end{array}$$

Hence, we can update ${\widehat{\beta }}_{l,k+1}^{(t)}$ using the minimizer of $G({\widehat{\beta }}_{l,k+1}^{(t)}|{\beta }_k^{(t)})$:

$$\begin{array}{c}\hfill {\widehat{\beta }}_{l,k+1}^{(t)}=\mathrm{sgn}\left({\beta }_{l,k}^{(t)}-{\displaystyle \frac{h({\upsilon }_{l,1}-{\upsilon }_{l,3})}{2{\upsilon }_{l,2}{k}_6}}\right)max\left\{|{\beta }_{l,k}^{(t)}-{\displaystyle \frac{h({\upsilon }_{l,1}-{\upsilon }_{l,3})}{2{\upsilon }_{l,2}{k}_6}}|-{\displaystyle \frac{h{p}_{{\lambda }_t}^{\prime }(|{\beta }_{l,k}^{(t)}|)}{2{\upsilon }_{l,2}{k}_6}},0\right\}.\end{array}$$

For tuning parameter ${\lambda }_t$, we set ${\lambda }_t={\lambda }_1$ for all $t\in [T]$. To select an appropriate value of ${\lambda }_1$, we propose a high-dimensional Bayesian information criterion (HBIC) in the distributed setting, which is defined as

$$\begin{array}{c}\hfill \mathrm{HBIC}(\lambda )=\log {\widehat{\mathcal{L}}}_h^{(1)}({\widehat{\mathit{\beta }}}^{(1)})+{\displaystyle \frac{|M_{\lambda }|(\log \log n)\log p}{n}},\end{array}$$

where $M_{\lambda }=\{j\in [p]:{\widehat{\beta }}_j^{(1)}\ne 0\}$ with ${\widehat{\beta }}_j^{(1)}$ being the j-th element of ${\widehat{\mathit{\beta }}}^{(1)}$. Compared to cross-validation, selecting ${\lambda }_1$ by minimizing the HBIC eliminates the need to split the data or repeatedly evaluate test error across multiple sub-datasets, thereby reducing computational cost while maintaining theoretical interpretability. Specifically, we compute the solution path over a decreasing grid ${\{{\lambda }_{1,\ell }\}}_{\ell =1}^L$, evaluating the HBIC for each ${\lambda }_{1,\ell }$ and choosing the minimizer. In implementation, we first compute an initial penalty level ${\lambda }_{1,1}$ similarly to Zhou et al. [17], using only the local data on the first machine to avoid extra communication, which is represented as follows:

$$\begin{array}{c}\hfill {\lambda }_{1,1}=|{\displaystyle \frac{1}{n(n-1)}}\sum _{i\in I_1}\sum _{j\in I_1,j\ne i}{L}_h^{\prime }(Y_i-Y_j)({\mathbf{X}}_i-{\mathbf{X}}_j){|}_{\infty }.\end{array}$$

We then form a decreasing geometric grid ${\lambda }_{\ell }={0.1}^{1/(L-1)}{\lambda }_{\ell -1}$ for $\ell >1$. In our experiments, we set $L=50$. We summarize this procedure in Algorithm 1, which is based on Algorithm 2.

Algorithm 1 Estimation procedure for DCR

Require:  (i) data batches ${\{(Y_i,{\mathbf{X}}_i)\}}_{i\in {\mathcal{I}}_j}$, $j\in [m]$, stored on m local machines; (ii) initialization ${\widehat{\mathit{\beta }}}^{(0)}$; (iii) bandwidth h, regularization parameter $\lambda >0$, and communication times T.  1: for $t\in [T]$ do  2:   Compute local gradient $\nabla {\widehat{\mathcal{L}}}_{1,h}({\widehat{\mathit{\beta }}}^{(t-1)})$ and global gradient $\nabla {\widehat{\mathcal{L}}}_h({\widehat{\mathit{\beta }}}^{(t-1)})$;  3:   Use Algorithm 2 to compute ${\widehat{\mathit{\beta }}}^{(t)}$.  4: end for Ensure:  ${\widehat{\mathit{\beta }}}^{(T)}$.

Algorithm 2 Local linear approximation (LLA) algorithm for solving (3)

Require:  (i) local data ${\{(Y_i,{\mathbf{X}}_i)\}}_{i\in {\mathcal{I}}_1}$; (ii) initialization ${\widehat{\mathit{\beta }}}^{(t-1)}$; (iii) local gradient $\nabla {\widehat{\mathcal{L}}}_{1,h}({\widehat{\mathit{\beta }}}^{(t-1)})$; (iv) global gradient $\nabla {\widehat{\mathcal{L}}}_h({\widehat{\mathit{\beta }}}^{(t-1)})$; (v) bandwidth h and (vi) regularization parameter $\lambda >0$.  1: Initial $\overline{\mathit{\beta }}\leftarrow {\widehat{\mathit{\beta }}}^{(t-1)}$.  2: repeat  3:   for $l\in [p]$ do  4:     Compute ${\upsilon }_{l,1}$, ${\upsilon }_{l,2}$ and ${\upsilon }_{l,3}$ for $l\in [p]$ by (4), (5) and (6), respectively.  5:     Compute $$\begin{array}{c}\hfill {\tilde{\beta }}_l=\mathrm{sign}\left({\overline{\beta }}_l-{\displaystyle \frac{h({\upsilon }_{l,1}-{\upsilon }_{l,3})}{2{\upsilon }_{l,2}{k}_6}}\right)max\left\{|{\overline{\beta }}_l-{\displaystyle \frac{h({\upsilon }_{l,1}-{\upsilon }_{l,3})}{2{\upsilon }_{l,2}{k}_6}}|-{\displaystyle \frac{h{p}^{\prime }(|{\overline{\beta }}_l|)}{2{\upsilon }_{l,2}{k}_6}},0\right\}.\end{array}$$  6:     Let $d_l={\tilde{\beta }}_l-{\overline{\beta }}_l$ and set ${\overline{\beta }}_l\leftarrow {\tilde{\beta }}_l$.  7:   end for  8: until ${max}_{l\in [p]}{d}_l^2<{10}^{-7}$. Ensure:   ${\widehat{\mathit{\beta }}}^{(t)}={({\tilde{\beta }}_1,\dots ,{\tilde{\beta }}_p)}^{\top }$.

4. Theoretical Results

In this section, we investigate the theoretical properties for the DCR estimator proposed in Section 2. Let $\mathbf{\Sigma }=\mathbb{E}(\mathbf{X}{\mathbf{X}}^{\top })$ and $\mathbf{Z}={\mathbf{\Sigma }}^{-1/2}\mathbf{X}$. Denote by $g(·)$ the probability density function of ${\epsilon }_i-{\epsilon }_j$ conditional on ${\mathbf{X}}_i$ and ${\mathbf{X}}_j$. To establish the theoretical guarantees, we need the following regularity assumptions.

Assumption 1.

There exist universal constants $k_1,k_2,k_3>0$ such that (i) ${max}_{i\in [N],j\in [p]}|X_{i,j}|\le k_1$; (ii) ${sup}_{\mathit{u}\in {\mathbb{S}}^{p-1}}\mathbb{E}\{{({\mathit{u}}^{\top }\mathbf{Z})}^4\}\le k_2$; and (iii) the eigenvalues of matrix Σ are bounded above by $k_3$ and below by $k_3^{-1}$.

Assumption 2.

There exist universal constants $k_4,k_5>0$ such that (i) $|g(x)-g(y)|\le k_4|x-y|$ for all $x,y\in \mathbb{R}$ and (ii) $k_5^{-1}\le g(0)\le k_5$.

Assumption 3.

The kernel function $K(·)$ satisfies that $K(v)\ge 0$ and $K(-v)=K(v)$ for any $v\in \mathbb{R}$, ${sup}_{v\in \mathbb{R}}{K}^{\prime }(v)<\infty$, and ${\int }_{\mathbb{R}}K(v)\mathrm{d}v=1$. There exist universal constants $k_6,k_7>0$ such that ${sup}_{v\in \mathbb{R}}K(v)<k_6$ and ${inf}_{v\in [-1,1]}K(v)=k_7$.

Assumption 4.

Suppose that (i) the penalty function $p_{\lambda }(·)$ satisfies $p_{\lambda }(0)=0$ and $p_{\lambda }(t)=p_{\lambda }(-t)$ for all $t\in \mathbb{R}$; (ii) $p_{\lambda }(·)$ is nondecreasing on the nonnegative real line; (iii) $p_{\lambda }(t)/t$ is nonincreasing for $t>0$; (iv) $p_{\lambda }(·)$ is differentiable for all $t\ne 0$ and has a right derivative at $t=0$ with ${lim}_{t\to 0^{+}}{p}_{\lambda }^{\prime }(t)=k_8\lambda$ for universal constant $k_8>0$; and (v) there exists $0<\mu <7/(5k_3)$ such that $p_{\lambda ,\mu }(t)=p_{\lambda }(t)+\mu t^2/2$ is convex.

Assumptions 1–3 are standard in high-dimensional convoluted rank regression, as discussed in, e.g., Zhou et al. [17]. Assumption 4 is standard in the literature on nonconvex penalized M-estimation and has been used in prior works such as Loh and Wainwright [19], Loh [20], Loh and Wainwright [21]. These assumptions ensure basic regularity properties (e.g., symmetry, monotonicity, local differentiability and a localized convexification) and are satisfied by a broad class of nonconvex penalties, including SCAD [22] and MCP [23].

Define the event ${\mathcal{E}}_0(r_0)=\{{\widehat{\mathit{\beta }}}^{(0)}\in {\mathbb{B}}_{\mathsf{\Sigma }}({\mathit{\beta }}^{*},r_0)\cap \mathsf{\Lambda }\}$ for some $r_0\ge 0$, where $\mathsf{\Lambda }=\{\mathit{\beta }\in {\mathbb{R}}^p:|\mathit{\beta }-{\mathit{\beta }}^{*}{|}_1\le 4s^{1/2}|\mathit{\beta }-{\mathit{\beta }}^{*}{|}_{\mathsf{\Sigma }}\}$ is an ${\ell }_1$-cone. Since we are interested in cases where the penalty may be nonconvex, we include the side condition ${|\mathit{\beta }|}_1\le R$ for $R>0$ in (3) in order to guarantee the existence of local/global optima. Theorem 1 gives the convergence rate of the estimator after the first iteration under the norm ${|·|}_{\mathsf{\Sigma }}$, whose proof is given in Appendix A.

Theorem 1.

Let Assumptions 1–4 hold. Select ${\lambda }_1\asymp r_0{(s\log p/n)}^{1/2}+{(\log p/N)}^{1/2}$, $h=O(1)$ and $R\asymp {(n/\log p)}^{1/2}$. Restricted on the event ${\mathcal{E}}_0(r_0)$, it holds with probability at least $1-3p^{-1}$ that $\{{\widehat{\mathit{\beta }}}^{(1)}\in \mathsf{\Lambda }\}$ and

$$\begin{array}{c}\hfill |{\widehat{\mathit{\beta }}}^{(1)}-{\mathit{\beta }}^{*}{|}_{\mathsf{\Sigma }}\lesssim {\displaystyle \frac{r_{0}s{(\log p)}^{1/2}}{n^{1/2}}}+{\displaystyle \frac{{(s\log p)}^{1/2}}{N^{1/2}}}\end{array}$$

(7)

provided that $n\gg \log p$.

Remark 1.

Theorem 1 illustrates that a single round of communication among machines can substantially improve the accuracy of the initial estimator ${\widehat{\mathit{\beta }}}^{(0)}$, which is obtained from local computation based on only n samples. Specifically, ${\widehat{\mathit{\beta }}}^{(1)}$ is computed by solving a regularized surrogate problem (3) with $t=1$, which incorporates both the local loss function ${\widehat{\mathcal{L}}}_{1,h}(\mathit{\beta })$ and the global gradient $\nabla {\widehat{\mathcal{L}}}_h({\widehat{\mathit{\beta }}}^{(0)})$, effectively integrating information from the entire dataset of size $N=mn$. The error bound on the right hand side of (7) demonstrates the gain from global information sharing: the first term reduces the estimation error related to limited local sample size n, while the second term decays with the total sample size N, enabling ${\widehat{\mathit{\beta }}}^{(1)}$ to reach the efficiency of a global estimator.

Theorem 2 gives the convergence rate of the estimator after the t-th iteration under the norm ${|·|}_{\mathsf{\Sigma }}$, whose proof is given in Appendix B.

Theorem 2.

Let Assumptions 1–4 hold. Select $h=O(1)$, $R\asymp {(n/\log p)}^{1/2}$ and ${\lambda }_t\asymp {(\log p/N)}^{1/2}+s^{(2t+1)/2}{(\log p/n)}^{(t+1)/2}$. Then, it holds with probability at least $1-3(t+1)p^{-1}$ that

$$\begin{array}{c}\hfill |{\widehat{\mathit{\beta }}}^{(t)}-{\mathit{\beta }}^{*}{|}_{\mathsf{\Sigma }}\lesssim {\displaystyle \frac{s^{(2t+1)/2}{(\log p)}^{(t+1)/2}}{n^{(t+1)/2}}}+{\displaystyle \frac{{(s\log p)}^{1/2}}{N^{1/2}}}\end{array}$$

(8)

provided that $n\gg s^2\log p$.

Remark 2.

Theorem 2 highlights the effectiveness of multiple rounds of communication in distributed learning with penalized convoluted rank loss. On the right hand side of (8), the first term decays exponentially with the number of communication rounds t, while the second term matches the optimal rate of global convoluted rank regression with access to the full dataset of size N. Notably, when $t\asymp \log m$, the influence of the initial estimator ${\widehat{\mathit{\beta }}}^{(0)}$, which only uses local data, is effectively eliminated. As a result, the final estimator ${\widehat{\mathit{\beta }}}^{(t)}$ achieves the same efficiency as one that directly solves the full-data problem, but without the need to collect all N samples on a single machine. This demonstrates that the proposed multi-round distributed procedure enables communication-efficient optimality, and only a logarithmic number of machines are needed to match the convergence rate of global learning.

5. Numerical Studies

5.1. Monte Carlo Examples

We consider three methods for comparison with our proposed method: (i) the global convoluted rank regression (Global CR) estimator [17] that uses all the available $N=mn$ observations; (ii) the divide-and-conquer CR (DC-CR) estimator, which is based on averaging m local CR estimators; and (iii) the local convoluted rank regression (Local CR). A regularization parameter $\lambda$ is included in all methods, and its value is selected by minimizing the corresponding HBIC. To measure the sparsity recovery performance, we calculated the F1 score:   

$$\mathrm{F}1={\displaystyle \frac{2(\mathrm{TP})}{2(\mathrm{TP})+\mathrm{FP}+\mathrm{FN}}},$$

where TP, FP and FN denote the numbers of true positives, false positives and false negatives, respectively. To measure estimation accuracy, we calculate the square root of the mean squared error (RMSE).

We generate data vectors ${\{(Y_i,{\mathbf{X}}_i)\}}_{i=1}^N$ from a heteroscedastic model $Y={\mathbf{X}}^{\top }{\mathit{\beta }}^{*}+(\sqrt{3}|{\mathit{\beta }}^{*}{{|}_2^2)}^{-1}{({\mathbf{X}}^{\top }{\mathit{\beta }}^{*})}^2\epsilon$, where $c=\sqrt{3}{|{\mathit{\beta }}^{*}|}_2^2$, ${\mathit{\beta }}^{*}={(\sqrt{3},\sqrt{3},\sqrt{3},\sqrt{3},\sqrt{3},0,0,\dots ,0)}^{\top }\in {\mathbb{R}}^p$, and $\mathbf{X}$ is independently generated from $\mathcal{N}(0,\mathsf{\Sigma })$ with ${\mathsf{\Sigma }}_{i,j}={({0.5}^{|i-j|})}_{p\times p}$. We consider that $\epsilon$ is independently generated from (1) standard normal distribution $\epsilon \sim \mathcal{N}(0,1)$; (2) t distribution $\epsilon \sim t(2)$; (3) standard Cauchy distribution $\epsilon \sim \mathrm{Cauchy}(0,1)$; and (4) mixture normal distribution $\epsilon \sim 0.75\mathcal{N}(0,1)+0.25\mathcal{N}(0,100)$. We set $p=400$, $n=200$, $m\in \{10,20,30,40,50\}$ and the number of communication rounds $T=\lceil \log m\rceil$. In practice, we recommend an early-stopping strategy for T similar to that proposed by Zhang et al. [1]: communication iterations can be terminated when the gradient norm of the loss Function (2) drops below a predefined threshold.

We adopt the Epanechnikov kernel $K(u)=3/4·(1-u^2)I(-1\le u\le 1)$ and construct the loss function as

$$\begin{array}{c}\hfill L_h(u)=\left\{\begin{array}{cc}u,\hfill & u\ge h,\hfill \\ {\displaystyle \frac{3u^2}{4h}}-{\displaystyle \frac{u^4}{8h^3}}+{\displaystyle \frac{3h}{8}},\hfill & -h<u<h,\hfill \\ u,\hfill & u\le -h.\hfill \end{array}\right.\end{array}$$

In Algorithm 1, the bandwidth h is used only to smooth the rank loss. Importantly, this smoothing does not change the target parameter: by Theorem 1 of Zhou et al. [17], the population minimizer satisfies ${\mathit{\beta }}_h^{*}={\mathit{\beta }}^{*}$ for any $h>0$. Therefore, h mainly affects numerical smoothness and finite-sample constants rather than the estimator. Our theory only requires h to be of constant order, i.e., $h=O(1)$. In practice, we recommend a simple default choice of $h=1$. For regularization, we employ the SCAD penalty, which is given by

$$\begin{array}{c}\hfill p_{\lambda }(|t|)=\left\{\begin{array}{cc}\lambda |t|,\hfill & 0\le |t|<\lambda ,\hfill \\ {\displaystyle \frac{a\lambda |t|-(t^2+{\lambda }^2)/2}{a-1}},\hfill & \lambda \le |t|\le a\lambda ,\hfill \\ {\displaystyle \frac{(a+1){\lambda }^2}{2}},\hfill & |t|>a\lambda ,\hfill \end{array}\right.\end{array}$$

for some $a>2$. The default choice of parameter a is $3.7$.

The simulation results, summarized in Table 1, show that the proposed DCR method achieves the best overall performance across all settings. It consistently yields the lowest RMSE and the highest F1 scores, indicating both accurate estimation and reliable support recovery. The DC-CR estimator, while comparable to DCR in terms of RMSE, suffers from significantly lower F1 scores, particularly under Cauchy and contaminated distributions, due to unstable variable selection. These results highlight the advantage of DCR in combining robustness, sparsity and distributed efficiency.

Table 1. Comparison of Local CR, DC-CR, DCR and Global CR.

	$\mathcal{N}(0,1)$
	$m=10$		$m=20$		$m=30$		$m=40$		$m=50$
Model	RMSE	F1	RMSE	F1	RMSE	F1	RMSE	F1	RMSE	F1
Local CR	0.010	1.000	0.010	1.000	0.010	1.000	0.010	1.000	0.010	1.000
DC-CR	0.003	1.000	0.002	0.997	0.002	0.997	0.001	0.995	0.001	0.995
DCR	0.003	1.000	0.002	1.000	0.002	1.000	0.002	1.000	0.001	1.000
Global CR	0.004	1.000	0.003	1.000	0.002	1.000	0.002	1.000	0.002	1.000
	$t(2)$
	$m=10$		$m=20$		$m=30$		$m=40$		$m=50$
Model	RMSE	F1	RMSE	F1	RMSE	F1	RMSE	F1	RMSE	F1
Local CR	0.012	0.999	0.012	0.999	0.012	0.999	0.012	0.999	0.012	0.999
DC-CR	0.004	0.977	0.003	0.967	0.003	0.953	0.002	0.940	0.002	0.927
DCR	0.004	0.999	0.003	0.999	0.002	0.999	0.002	0.999	0.002	0.999
Global CR	0.005	1.000	0.004	1.000	0.003	1.000	0.002	1.000	0.002	1.000
	$\mathrm{Cauchy}(0,1)$
	$m=10$		$m=20$		$m=30$		$m=40$		$m=50$
Model	RMSE	F1	RMSE	F1	RMSE	F1	RMSE	F1	RMSE	F1
Local DCR	0.016	0.983	0.016	0.983	0.016	0.983	0.016	0.983	0.016	0.983
DC-CR	0.009	0.872	0.007	0.760	0.006	0.683	0.005	0.610	0.005	0.556
DCR	0.005	0.985	0.004	0.985	0.004	0.985	0.003	0.985	0.003	0.985
Global CR	0.007	1.000	0.004	1.000	0.004	1.000	0.003	1.000	0.003	1.000
	$0.75\mathcal{N}(0,1)+0.25\mathcal{N}(0,100)$
	$m=10$		$m=20$		$m=30$		$m=40$		$m=50$
Model	RMSE	F1	RMSE	F1	RMSE	F1	RMSE	F1	RMSE	F1
Local CR	0.018	0.967	0.018	0.967	0.018	0.967	0.018	0.967	0.018	0.967
DC-CR	0.009	0.799	0.006	0.690	0.005	0.591	0.004	0.521	0.004	0.460
DCR	0.006	0.977	0.004	0.977	0.004	0.977	0.004	0.977	0.004	0.977
Global CR	0.007	1.000	0.004	1.000	0.003	1.000	0.003	1.000	0.003	1.000

5.2. Real Data Example: LGBT Tweets

We illustrate our proposed method on a real-world dataset of tweets related to LGBTQ topics, collected using the keyword “LGBT” over the period from 21 August to 26 August 2022 (the dataset is publicly available at https://www.kaggle.com/datasets/vencerlanz09/lgbt-tweets (accessed on 20 April 2025)). The goal of this analysis is to identify linguistically salient words that have a significant impact on the like count of each tweet.

Each observation in our study represents a single post. Our initial sample consists of a total of $N=\mathrm{32,456}$ tweets, each containing the keyword “LGBT”. For each post, we record the number of likes it received, denoted by $Y_i$, which serves as the outcome variable. The original like counts $Y_i$ exhibit substantial variation, with most tweets receiving only a few likes and a small number garnering extremely high engagement. To address this scale disparity and stabilize the variance, we apply the transformation $\log (1+Y_i)$. As shown in Figure 1, the transformed variable retains a heavy-tailed distribution, indicating that the imbalance in user attention persists even after logarithmic scaling.

Figure 1. Probability density of $\log (1+\mathrm{likes}\mathrm{count})$.

To construct the covariates, we preprocess the tweet text using Python 3.11.10 libraries, including spaCy for tokenization, lemmatization and part-of-speech filtering, and CountVectorizer for feature extraction. During preprocessing, we remove punctuation, stop words, auxiliary verbs, adverbs and informal abbreviations. After cleaning, we apply a binary bag-of-words encoding based on the 500 most frequently used informative words in the corpus. Specifically, the j-th coordinate $X_{ij}$ of the covariate vector ${\mathbf{X}}_i$ equals 1 if the j-th word in the vocabulary appears in the i-th tweet, and 0 otherwise. Posts that do not contain any of these 500 words are excluded from the analysis. After filtering, we retain a final sample of N = 31,644 observations. The data is randomly distributed across 100 machines, with an average of 316 data points per machine. The number of iterations is set to $T=5$.

From Table 2, we observe that several keywords show strong positive associations with the number of likes, such as “character”, “queer” and “let”. These terms frequently appear in contexts emphasizing representation, identity affirmation or emotional solidarity (e.g., “queer character in media” and “let people love who they love”). Such expressions typically convey inclusive and positive sentiments, which tend to resonate with audiences and encourage affirmative responses in the form of likes.

Table 2. Estimation result for LGBT tweets data.

Word	Count	${\widehat{\mathit{\beta }}}_{\mathit{j}}$
loom	139	−0.443
character	735	0.335
queer	680	0.214
groom	130	−0.117
ban	689	−0.109
let	400	0.088
write	200	0.070
school	806	−0.051
right	2323	−0.027
force	460	−0.027
future	117	−0.022

In contrast, negatively weighted terms such as “groom”, “ban” and “loom” are often embedded in narratives involving controversy, fear or stigmatization. For instance, tweets referencing “ban LGBT education” or “danger looms” may attract attention but are less likely to elicit endorsement via likes, as users may refrain from visibly supporting content that is perceived as divisive or politically sensitive. Additionally, high-frequency yet semantically broad words such as “right” and “school” demonstrate weak or even negative associations with like counts. In particular, “school” frequently appears in tweets related to educational debates (e.g., “LGBT in schools” and “teaching gender identity”), which are often contentious. The polarizing nature of such discussions may lead users to engage cautiously or avoid expressing approval through likes.

Overall, the keywords selected by our proposed method highlight the crucial role of emotional valence and semantic framing in shaping engagement behavior. While positively framed language that conveys support, identity and collective belonging tends to promote user interaction, negatively charged or controversial expressions are less likely to generate positive engagement, despite their potential visibility.

6. Discussion

In this paper, we study high-dimensional linear regression in distributed settings and develop a DCR framework to mitigate the severe performance degradation of least-squares-based distributed procedures under heavy-tailed errors and outliers. Methodologically, the proposed approach integrates a convolution-smoothed rank loss into a communication-efficient surrogate-likelihood paradigm: at each communication round, a tractable surrogate objective is constructed by combining local empirical losses with a global gradient correction, and the central node solves a sparsity-regularized problem with nonconvex penalties (e.g., SCAD/MCP) to aggregate information without transferring raw data. To enable scalable computation, we further devise an optimization strategy based on the local linear approximation (LLA) method, reducing the nonconvex regularized estimation to a sequence of efficiently solvable subproblems. Additionally, an HBIC-based tuning procedure is provided to alleviate the computational burden of cross-validation in large-scale distributed regimes.

On the theoretical side, we establish non-asymptotic error bounds for both one-shot and multi-round communication schemes, explicitly characterizing how additional communication improves statistical accuracy. In particular, a single communication round already yields a substantial correction over purely local estimators, while the multi-round procedure exhibits a rapid contraction of the “local-sample-limitation” term in the error bound. When the number of communication rounds reaches a logarithmic order in the number of machines, the resulting estimator attains (up to logarithmic factors) the optimal convergence rate comparable to its centralized counterpart, thereby achieving communication-efficient statistical optimality.

Extensive experiments corroborate the robustness and effectiveness of the proposed method. Across a broad collection of noise distributions, DCR delivers stable performance in terms of estimation accuracy and support recovery. In a real-data application, DCR identifies interpretable keywords and yields effect estimates with coherent directions, illustrating its practical utility for large-scale high-dimensional problems with non-Gaussian noise.

Several directions merit further investigation. First, it would be valuable to extend the proposed framework to broader model classes (e.g., generalized linear or semiparametric models) and to formally analyze robustness and computational behavior under more complex data structures such as dependence, nonstationarity or distributional drift. Second, building on the current estimation theory, developing feasible distributed inference procedures along with a principled characterization of the trade-offs among inferential accuracy, communication cost and computation would substantially broaden the applicability of method. Third, for partially trusted distributed environments, integrating DCR with Byzantine-resilient aggregation or fault-tolerant mechanisms and establishing statistical consistency and worst-case error guarantees in the presence of adversarially corrupted worker messages would enhance both the security and reliability of the method in real-world deployments. We leave these considerations for future research.

Appendix A.1. Proof of (A2)

Proof of

(A2). Case 1: ${|\mathit{\mu }|}_1/{|\mathit{\mu }|}_{\mathsf{\Sigma }}\ge {\{7n/(8{\tau }_1\log p)\}}^{1/2}$. Due to $\widehat{\mathcal{L}}(·)$ being convex, it holds that

$$\begin{array}{c}\hfill ⟨\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}+\mathit{\mu })-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}),\mathit{\mu }⟩\ge 0\ge {\displaystyle \frac{{7|\mathit{\mu }|}_{\mathsf{\Sigma }}^2}{8}}-{\displaystyle \frac{{\tau }_1\log p{|\mathit{\mu }|}_1^2}{n}}.\end{array}$$

(A16)

Case 2: ${|\mathit{\mu }|}_1/{|\mathit{\mu }|}_{\mathsf{\Sigma }}<{\{7n/(8{\tau }_1\log p)\}}^{1/2}$. Recall $n\gg \log p$. For any ${|\mathit{\mu }|}_{\mathsf{\Sigma }}\le r_1$, by Assumption 1, we have ${|\mathit{\mu }|}_{\mathsf{\Sigma }}\le k_3^{1/2}$. By Lemma A3 with $l={\{7n/(8{\tau }_1\log p)\}}^{1/2}$, we have

$$\begin{array}{c}\hfill ⟨\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}+\mathit{\mu })-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}),\mathit{\mu }⟩\ge {\displaystyle \frac{{7|\mathit{\mu }|}_{\mathsf{\Sigma }}^2}{8}}\end{array}$$

(A17)

provided that $16k_2^{1/2}{r}_1\le h\le {(2k_4{k}_5)}^{-1}$, ${\tau }_1{r}_1^2\ge 7{(55k_1{k}_5)}^2/8$ and ${(18h^2\log p/n)}^{1/2}\le r_1\le {(32k_2^{1/2}{k}_4{k}_5)}^{-1}$, with probability at least $1-p^{-1}$. Combining (A16) and (A17), (A2) holds. □

Appendix A.2. Proof of (A3)

Proof of (A3).

Let $f(t)={\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}+t\mathit{\mu })$. Since ${\widehat{\mathcal{L}}}_h(·)$ is convex, $f(t)$ is also convex, and $f^{\prime }(1)-f^{\prime }(0)\ge f^{\prime }(t)-f^{\prime }(0)$ for $t\in [0,1]$. Then, we obtain that

$$\begin{array}{c}\hfill ⟨\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}+\mathit{\mu })-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}),\mathit{\mu }⟩\ge {\displaystyle \frac{1}{t}}\times ⟨\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}+t\mathit{\mu })-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}),t\mathit{\mu }⟩.\end{array}$$

Selecting $t=r_1/{|\mathit{\mu }|}_{\mathsf{\Sigma }}\in (0,1)$, by (A2), we have

$$\begin{array}{cc}\hfill ⟨\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}+\mathit{\mu })-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}),\mathit{\mu }⟩\ge & {\displaystyle \frac{{|\mathit{\mu }|}_{\mathsf{\Sigma }}}{r_1}}\left({\displaystyle \frac{7}{8}}|{\displaystyle \frac{\mathit{\mu }}{{|\mathit{\mu }|}_{\mathsf{\Sigma }}}}{|}_{\mathsf{\Sigma }}^2-{\displaystyle \frac{{\tau }_1{r}_1^2\log p}{n}}\times |{\displaystyle \frac{\mathit{\mu }}{{|\mathit{\mu }|}_{\mathsf{\Sigma }}}}{|}_1^2\right)\hfill \\ \hfill =& {\displaystyle \frac{{|\mathit{\mu }|}_{\mathsf{\Sigma }}}{r_1}}\left({\displaystyle \frac{7r_1^2}{8}}-{\displaystyle \frac{{\tau }_1{r}_1^2\log p}{n}}\times {\displaystyle \frac{{|\mathit{\mu }|}_1^2}{{|\mathit{\mu }|}_{\mathsf{\Sigma }}^2}}\right)\hfill \\ \hfill \ge & {\displaystyle \frac{{|\mathit{\mu }|}_{\mathsf{\Sigma }}}{r_1}}\left({\displaystyle \frac{7r_1^2}{8}}-{\displaystyle \frac{2R{\tau }_1{r}_1^2\log p}{n}}\times {\displaystyle \frac{{|\mathit{\mu }|}_1}{{|\mathit{\mu }|}_{\mathsf{\Sigma }}^2}}\right)\hfill \\ \hfill \ge & {\displaystyle \frac{{|\mathit{\mu }|}_{\mathsf{\Sigma }}}{r_1}}\left({\displaystyle \frac{7r_1^2}{8}}-{\displaystyle \frac{2R{\tau }_1{r}_1\log p}{n}}\times {\displaystyle \frac{{|\mathit{\mu }|}_1}{{|\mathit{\mu }|}_{\mathsf{\Sigma }}}}\right)\hfill \\ \hfill =& {\displaystyle \frac{7r_1{|\mathit{\mu }|}_{\mathsf{\Sigma }}}{8}}-{\displaystyle \frac{2R{\tau }_1\log p{|\mathit{\mu }|}_1}{n}}\hfill \end{array}$$

provided that $16k_2^{1/2}{r}_1\le h\le {(2k_4{k}_5)}^{-1}$, ${\tau }_1{r}_1^2\ge 7{(55k_1{k}_5)}^2/8$ and ${(18h^2\log p/n)}^{1/2}\le r_1\le {(32k_2^{1/2}{k}_4{k}_5)}^{-1}$, with probability at least $1-p^{-1}$. Hence, (A3) holds. □

Appendix A.3. Proof of (A4)

Proof of

(A4). If ${|\mathit{\mu }|}_{\mathsf{\Sigma }}>r_1$, by $\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}+\mathit{\mu })=-\nabla p_{{\lambda }_1}({\widehat{\mathit{\beta }}}^{(1)})$ and (A3), we have

$$\begin{array}{c}\hfill ⟨-\nabla p_{{\lambda }_1}({\widehat{\mathit{\beta }}}^{(1)})-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}),\mathit{\mu }⟩\ge {\displaystyle \frac{7r_1{|\mathit{\mu }|}_{\mathsf{\Sigma }}}{8}}-{\displaystyle \frac{2R{\tau }_1\log p{|\mathit{\mu }|}_1}{n}}\end{array}$$

(A18)

provided that $16k_2^{1/2}{r}_1\le h\le {(2k_4{k}_5)}^{-1}$, ${\tau }_1{r}_1^2\ge 7{(55k_1{k}_5)}^2/8$ and ${(18h^2\log p/n)}^{1/2}\ge r_1\ge {(32k_2^{1/2}{k}_4{k}_5)}^{-1}$, with probability at least $1-2p^{-1}$. Moreover by Assumption 4, the Hölder inequality and Lemma 4 of Loh and Wainwright [21], restricted on ${\mathcal{E}}_0(r_0)$, we have

$$\begin{array}{cc}\hfill & ⟨-\nabla p_{{\lambda }_1}({\widehat{\mathit{\beta }}}^{(1)})-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}),\mathit{\mu }⟩\hfill \\ \hfill & \le [|\nabla p_{{\lambda }_1}({\widehat{\mathit{\beta }}}^{(1)}){|}_{\infty }+\underset{\mathit{\beta }-{\mathit{\beta }}^{*}\in {\mathbb{B}}_{\mathsf{\Sigma }}({\mathit{\beta }}^{*},r_0)\cap \mathsf{\Lambda }}{sup}{|(1-\mathbb{E})\{\nabla {\widehat{\mathcal{L}}}_h(\mathit{\beta })-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*})\}|}_{\infty }\hfill \\ \hfill & +\underset{\mathit{\beta }\in {\mathbb{B}}_{\mathsf{\Sigma }}({\mathit{\beta }}^{*},r_0)\cap \mathsf{\Lambda }}{sup}|(1-\mathbb{E})\{\nabla {\widehat{\mathcal{L}}}_{1,h}(\mathit{\beta })-\nabla {\widehat{\mathcal{L}}}_{1,h}({\mathit{\beta }}^{*})\}{|}_{\infty }+|\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}){|}_{\infty }]·{|\mathit{\mu }|}_1\hfill \\ \hfill & \le k_8{\lambda }_1{|\mathit{\mu }|}_1+\{{\mathsf{\Pi }}_1(r_0)+{\mathsf{\Pi }}_2(r_0)+{\mathsf{\Pi }}_3\}{|\mathit{\mu }|}_1.\hfill \end{array}$$

(A19)

Recall ${|\mathit{\mu }|}_1\le 2R$. Together with (A18), selecting $u=2\log p$, by Lemmas A1 and A2, we have

$$\begin{array}{cc}\hfill {\displaystyle \frac{7r_1{|\mathit{\mu }|}_{\mathsf{\Sigma }}}{8}}\le & \left(k_8{\lambda }_1+{\displaystyle \frac{2R{\tau }_1\log p}{n}}\right){|\mathit{\mu }|}_1+\left\{{\mathsf{\Pi }}_1(r_0)+{\mathsf{\Pi }}_2(r_0)+{\mathsf{\Pi }}_3\right\}{|\mathit{\mu }|}_1\hfill \\ \hfill \le & 2\left(k_8{\lambda }_1+{\displaystyle \frac{2R{\tau }_1\log p}{n}}\right)R+2\left\{{\mathsf{\Pi }}_1(r_0)+{\mathsf{\Pi }}_2(r_0)+{\mathsf{\Pi }}_3\right\}R\hfill \\ \hfill \le & 2\left(k_8{\lambda }_1+{\displaystyle \frac{2R{\tau }_1\log p}{n}}\right)R+C_1\left\{s^{1/2}{r}_0\left({\displaystyle \frac{{(\log p)}^{1/2}}{b{N}^{1/2}}}+{\displaystyle \frac{{(\log p)}^{1/2}}{b{n}^{1/2}}}\right)+{\displaystyle \frac{{(\log p)}^{1/2}}{N^{1/2}}}\right\}R\hfill \\ \hfill \le & 2\left(k_8{\lambda }_1+{\displaystyle \frac{2R{\tau }_1\log p}{n}}\right)R+C_2\left\{{\displaystyle \frac{r_0{s}^{1/2}{(\log p)}^{1/2}}{n^{1/2}}}+{\displaystyle \frac{{(\log p)}^{1/2}}{N^{1/2}}}\right\}R\hfill \end{array}$$

which implies that

$$\begin{array}{c}\hfill {|\mathit{\mu }|}_{\mathsf{\Sigma }}\le {\displaystyle \frac{16}{7r_1}}\left(k_8{\lambda }_1+{\displaystyle \frac{2R{\tau }_1\log p}{n}}\right)R+{\displaystyle \frac{C_3}{r_1}}\left\{{\displaystyle \frac{r_0{s}^{1/2}{(\log p)}^{1/2}}{n^{1/2}}}+{\displaystyle \frac{{(\log p)}^{1/2}}{N^{1/2}}}\right\}R\le r_1\end{array}$$

provided that $16k_2^{1/2}{r}_1\le h\le {(2k_4{k}_5)}^{-1}$, ${\tau }_1{r}_1^2\ge 7{(55k_1{k}_5)}^2/8$ and ${(18h^2\log p/n)}^{1/2}\ge r_1\le {(32k_2^{1/2}{k}_4{k}_5)}^{-1}$ and

$$\begin{array}{c}\hfill R\le min\left\{{\displaystyle \frac{7r_1^2}{4k_8{\lambda }_1}},{\left({\displaystyle \frac{7r_1^{2}n}{8{\tau }_1\log p}}\right)}^{1/2},{\tilde{C}}_1{r}_1^2\left({\displaystyle \frac{n^{1/2}}{r_0{(s\log p)}^{1/2}}}+{\displaystyle \frac{N^{1/2}}{{(\log p)}^{1/2}}}\right)\right\},\end{array}$$

where ${\tilde{C}}_1>0$ is a sufficiently small constant. Hence, (A4) holds. □

Appendix C.1. Proof of Lemma A1

Proof of Lemma A1.

Given $r_1,r_2>0$, define the local cone-neighborhood of ${\mathit{\beta }}^{*}$

$$\begin{array}{c}\hfill {\mathsf{\Theta }}_0(r_1,r_2)=\{\mathit{\upsilon }\in {\mathbb{R}}^p{:|\mathit{\upsilon }|}_{\mathsf{\Sigma }}\le r_1{,|\mathit{\upsilon }|}_1\le r_2\}.\end{array}$$

Write $\mathit{\upsilon }=\mathit{\beta }-{\mathit{\beta }}^{*}$ and

$$\begin{array}{c}\hfill f_k({\mathit{\xi }}_i,{\mathit{\xi }}_j,\mathit{\upsilon })=\{L_h^{\prime }({\epsilon }_i-{\epsilon }_j-{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\mathit{\upsilon })-L_h^{\prime }({\epsilon }_i-{\epsilon }_j)\}(X_{i,k}-X_{j,k})\end{array}$$

for $k\in [p]$ and ${\mathit{\xi }}_i=({\mathbf{X}}_i,{\epsilon }_i)$. Then,

$$\begin{array}{cc}\hfill & \underset{\mathit{\beta }-{\mathit{\beta }}^{*}\in {\mathbb{B}}_{\mathsf{\Sigma }}(r)\cap \mathsf{\Lambda }}{sup}{|(1-\mathbb{E})\{\nabla {\widehat{\mathcal{L}}}_h(\mathit{\beta })-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*})\}|}_{\infty }\hfill \\ \hfill & =\underset{k\in [p]}{max}\underset{\mathit{\upsilon }\in {\mathsf{\Theta }}_0(r,4s^{1/2}r)}{sup}|{\displaystyle \frac{1}{N(N-1)}}\sum _{i=1}^N\sum _{j\ne i}(1-\mathbb{E})\{f_k({\mathit{\xi }}_i,{\mathit{\xi }}_j,\mathit{\upsilon })\}|.\hfill \end{array}$$

(A21)

Recall $L_h^{\prime }(u)=2{\int }_0^u{K}_h(v)\mathrm{d}v$ with $K_h(v)=k(v/h)/h$. By Assumptions 1 and 3, for any $\mathit{\upsilon }\in {\mathsf{\Theta }}_0(r,l)$, we have

$$\begin{array}{cc}\hfill |f_k({\mathit{\xi }}_i,{\mathit{\xi }}_j,\mathit{\upsilon })|\le & C_1\underset{i,j\in [n],k\in [d]}{max}|X_{i,k}-X_{j,k}|·|{({\mathbf{X}}_i^{\top }-{\mathbf{X}}_j^{\top })}^{\top }\mathit{\upsilon }|/h\hfill \\ \hfill \le & C_2(|{\mathbf{X}}_i{|}_{\infty }+|{\mathbf{X}}_j{|}_{\infty }{)·|\mathit{\upsilon }|}_1/h\hfill \\ \hfill \le & C_3{s}^{1/2}r/h.\hfill \end{array}$$

By Assumption 3 again, for any $\mathit{\upsilon }\in {\mathsf{\Theta }}_0(r,l)$, we have

$$\begin{array}{cc}\hfill & \mathbb{E}\{L_h^{\prime }({\epsilon }_i-{\epsilon }_j-{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\mathit{\upsilon })-L_h^{\prime }({\epsilon }_i-{\epsilon }_j)|{\mathbf{X}}_i,{\mathbf{X}}_j\}\hfill \\ \hfill & =2\mathbb{E}\left\{{\int }_{{\epsilon }_i-{\epsilon }_j}^{{\epsilon }_i-{\epsilon }_j-{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\upsilon }{K}_h(v)\mathrm{d}v|{\mathbf{X}}_i,{\mathbf{X}}_j\right\}\hfill \\ \hfill & =2\mathbb{E}\left\{{\int }_0^{-{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\mathit{\upsilon }}K(u+({\epsilon }_i+{\epsilon }_j)/h)\mathrm{d}u|{\mathbf{X}}_i,{\mathbf{X}}_j\right\}\hfill \\ \hfill & \le C_1|{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\mathit{\upsilon }|,\hfill \end{array}$$

which implies that

$$\begin{array}{cc}\hfill \mathbb{E}\{f_k^2({\mathit{\xi }}_i,{\mathit{\xi }}_j,\mathit{\upsilon })\}\le & C_1\mathbb{E}\{|{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\mathit{\upsilon }{|}^2(X_{i,k}-X_{j,k})\}\hfill \\ \hfill \le & C_1{[\mathbb{E}\{{(X_{i,k}-X_{j,k})}^4\}]}^{1/2}[\mathbb{E}\{|{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }{\mathit{\upsilon }|}^4{\}]}^{1/2}\hfill \\ \hfill \le & C_2{[\mathbb{E}\{{({\mathbf{X}}_i^{\top }\mathit{\upsilon })}^4\}]}^{1/2}+C_2{[\mathbb{E}\{{({\mathbf{X}}_j^{\top }\mathit{\upsilon })}^4\}]}^{1/2}\lesssim 1.\hfill \end{array}$$

By Theorem 2 of Rejchel [24], we have

$$\begin{array}{cc}\hfill & \underset{\mathit{\upsilon }\in {\mathsf{\Theta }}_0(r,l)}{sup}\left|{\displaystyle \frac{1}{N(N-1)}}\sum _{i=1}^N\sum _{j\ne i}(1-\mathbb{E})\{f_k({\mathit{\xi }}_i,{\mathit{\xi }}_j,\mathit{\upsilon })\}\right|\hfill \\ \hfill & \lesssim \mathbb{E}\left\{\underset{\upsilon \in {\mathsf{\Theta }}_0(r,l)}{sup}\left|{\displaystyle \frac{1}{\lfloor N/2\rfloor }}\sum _{i=1}^{\lfloor N/2\rfloor }{\sigma }_i{f}_k({\mathit{\xi }}_i,{\mathit{\xi }}_{\lfloor N/2\rfloor +i},\mathit{\upsilon })\right|\right\}+{\displaystyle \frac{r{u}^{1/2}}{N^{1/2}}}+{\displaystyle \frac{s^{1/2}ru}{hN}}\hfill \end{array}$$

(A22)

for any $u>0$ with probability at least $1-e^{-u}$, where ${\{{\sigma }_i\}}_{i=1}^{\lfloor N/2\rfloor }$ is a sequence of Rademacher random variables. For each i, write $f_k({\mathit{\xi }}_i,{\mathit{\xi }}_{\lfloor N/2\rfloor +i},\mathit{\upsilon })={{\rm Y}}_i({({\mathbf{X}}_i-{\mathbf{X}}_{\lfloor N/2\rfloor +i})}^{\top }\mathit{\upsilon })$, where ${{\rm Y}}_i(·)$ is such that ${{\rm Y}}_i(0)=0$ and $|{{\rm Y}}_i(u)-{{\rm Y}}_i(v)|\lesssim h^{-1}|X_{i,k}-X_{\lfloor N/2\rfloor +i}|·|u-v|$. Then, by the Talagrand contraction principle,

$$\begin{array}{cc}\hfill & \mathbb{E}\left\{\underset{\upsilon \in {\mathsf{\Theta }}_0(r,l)}{sup}\left|{\displaystyle \frac{1}{\lfloor N/2\rfloor }}\sum _{i=1}^{\lfloor N/2\rfloor }{\sigma }_i{f}_k({\mathit{\xi }}_i,{\mathit{\xi }}_{\lfloor N/2\rfloor +i},\mathit{\upsilon })\right|\right\}\hfill \\ \hfill & \le C_3\mathbb{E}\left\{\underset{\mathit{\upsilon }\in {\mathsf{\Theta }}_0(r,4s^{1/2}r)}{sup}|{\displaystyle \frac{1}{\lfloor N/2\rfloor }}\sum _{i=1}^{\lfloor N/2\rfloor }{\sigma }_i{h}^{-1}|X_{i,k}-X_{\lfloor N/2\rfloor +i}|·{({\mathbf{X}}_i-{\mathbf{X}}_{\lfloor N/2\rfloor +i})}^{\top }\mathit{\upsilon }|\right\}\hfill \\ \hfill & \le {\displaystyle \frac{C_4{s}^{1/2}r}{h}}\mathbb{E}\left\{|{\displaystyle \frac{1}{\lfloor N/2\rfloor }}\sum _{i=1}^{\lfloor N/2\rfloor }{\sigma }_i|X_{i,k}-X_{\lfloor N/2\rfloor +i}|({\mathbf{X}}_i-{\mathbf{X}}_{\lfloor N/2\rfloor +i}){|}_{\infty }\right\}\hfill \\ \hfill & \lesssim {\displaystyle \frac{s^{1/2}r{(\log p)}^{1/2}}{h{N}^{1/2}}}.\hfill \end{array}$$

Together with (A22), by (A21) and Bonferroni inequality, it holds that

$$\begin{array}{c}\hfill \underset{\mathit{\beta }-{\mathit{\beta }}^{*}\in {\mathbb{B}}_{\mathsf{\Sigma }}(r)\cap \mathsf{\Lambda }}{sup}{|(1-\mathbb{E})\{\nabla {\widehat{\mathcal{L}}}_h(\mathit{\beta })-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*})\}|}_{\infty }\lesssim {\displaystyle \frac{s^{1/2}r{(\log p)}^{1/2}}{h{N}^{1/2}}}+{\displaystyle \frac{r{u}^{1/2}}{N^{1/2}}}+{\displaystyle \frac{s^{1/2}ru}{hN}}\end{array}$$

for any $u>0$ with probability at least $1-p{e}^{-u}$. Hence, we complete the proof of Lemma A1. □

Appendix C.2. Proof of Lemma A2

Proof of Lemma A2.

Recall $L_h^{\prime }(u)=2{\int }_0^u{K}_h(v)\mathrm{d}v$ with $K_h(v)=K(v/h)/h$. By Assumptions 1 and 3, we have

$$\begin{array}{c}\hfill \underset{i,j\in [N],k\in [p]}{max}|L_h^{\prime }({\epsilon }_i-{\epsilon }_j)(X_{i,k}-X_{j,k})|\le C_1.\end{array}$$

Moreover, by Lemma 1 of Zhou et al. [17] and the fact that the distribution of ${\epsilon }_i-{\epsilon }_j$ is symmetric about zero, we have $\mathbb{E}\{\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*})\}=0$. Then, by Bonferroni inequality and Equation (A.1) of Clémençon et al. [25], it holds that

$$\begin{array}{cc}\hfill \mathbb{P}(|\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}){|}_{\infty }>x)\le & p\underset{k\in [p]}{max}\mathbb{P}\left\{|{\displaystyle \frac{1}{N(N-1)}}\sum _{i=1}^N\sum _{j\ne i}{L}_h^{\prime }({\epsilon }_i-{\epsilon }_j)(X_{i,k}-X_{j,k})|>x\right\}\hfill \\ \hfill \le & 2p\exp (-C_{1}n{x}^2)\hfill \end{array}$$

for any $x>0$, which implies that

$$\begin{array}{c}\hfill |\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}){|}_{\infty }\lesssim {\displaystyle \frac{{(\log p+u)}^{1/2}}{N^{1/2}}}\end{array}$$

for any $u>0$, with probability at least $1-e^{-u}$. Then, we complete the proof of Lemma A2. □

Appendix C.3. Proof of Lemma A3

Proof of Lemma A3.

For any $\mathit{\beta }\in {\mathbb{R}}^d$, write $\check{\mathit{\beta }}=(\mathit{\beta }-{\mathit{\beta }}^{*})/{|\mathit{\beta }-{\mathit{\beta }}^{*}|}_{\mathsf{\Sigma }}$. Consider the event

$$\begin{array}{c}\hfill {\mathcal{A}}_{i,j}(\mathit{\beta })=\{|{\epsilon }_i-{\epsilon }_j|\le b/2\}\cap \{|{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}|\le b/(2r)\}.\end{array}$$

Restricted on ${\mathcal{A}}_{i,j}(\mathit{\beta })$, for any $\mathit{\beta }\in {\mathbb{B}}_{\mathsf{\Sigma }}({\mathit{\beta }}^{*},r)$, we have

$$\begin{array}{cc}\hfill |(Y_i-{\mathbf{X}}_i^{\top }\mathit{\beta })-(Y_j-{\mathbf{X}}_j^{\top }\mathit{\beta })|=& |(Y_i-{\mathbf{X}}_i^{\top }{\mathit{\beta }}^{*})+{\mathbf{X}}_i^{\top }({\mathit{\beta }}^{*}-\mathit{\beta })-(Y_j-{\mathbf{X}}_j^{\top }{\mathit{\beta }}^{*})+{\mathbf{X}}_j^{\top }(\mathit{\beta }-{\mathit{\beta }}^{*})|\hfill \\ \hfill \le & |{\epsilon }_i-{\epsilon }_j|+|{\mathbf{X}}_i^{\top }(\mathit{\beta }-{\mathit{\beta }}^{*})|+|{\mathbf{X}}_j^{\top }(\mathit{\beta }-{\mathit{\beta }}^{*})|\hfill \\ \hfill \le & b/2+b/2=b.\hfill \end{array}$$

By Assumption 3,

$$\begin{array}{cc}\hfill & ⟨\nabla {\widehat{\mathcal{L}}}_h(\mathit{\beta })-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}),\mathit{\beta }-{\mathit{\beta }}^{*}⟩\hfill \\ \hfill & =⟨\nabla {\widehat{\mathcal{L}}}_{1,h}(\mathit{\beta })-\nabla {\widehat{\mathcal{L}}}_{1,h}({\mathit{\beta }}^{*}),\mathit{\beta }-{\mathit{\beta }}^{*}⟩\hfill \\ \hfill & ={\displaystyle \frac{1}{n(n-1)}}\sum _{i\in {\mathcal{I}}_1}\sum _{j\ne i\in {\mathcal{I}}_1}\{L_{1,h}^{\prime }({\epsilon }_i(\mathit{\beta })-{\epsilon }_j(\mathit{\beta }))-L_{1,h}^{\prime }({\epsilon }_i-{\epsilon }_j)\}⟨{\mathbf{X}}_j-{\mathbf{X}}_i,\mathit{\beta }-{\mathit{\beta }}^{*}⟩\hfill \\ \hfill & \ge {\displaystyle \frac{1}{n(n-1)}}\sum _{i\in {\mathcal{I}}_1}\sum _{j\ne i\in {\mathcal{I}}_1}\{L_{1,h}^{\prime }({\epsilon }_i(\mathit{\beta })-{\epsilon }_j(\mathit{\beta }))-L_{1,h}^{\prime }({\epsilon }_i-{\epsilon }_j)\}⟨{\mathbf{X}}_j-{\mathbf{X}}_i,\mathit{\beta }-{\mathit{\beta }}^{*}⟩I\{{\mathcal{A}}_{i,j}(\mathit{\beta })\}\hfill \\ \hfill & \ge {\displaystyle \frac{2k_5}{n(n-1)b}}\sum _{i\in {\mathcal{I}}_1}\sum _{j\ne i\in {\mathcal{I}}_1}{(⟨{\mathbf{X}}_j-{\mathbf{X}}_i,\mathit{\beta }-{\mathit{\beta }}^{*}⟩)}^{2}I\{{\mathcal{A}}_{i,j}(\mathit{\beta })\}\hfill \end{array}$$

(A23)

with ${\epsilon }_i(\mathit{\beta })=Y_i-{\mathbf{X}}_i^{\top }\mathit{\beta }$. For any $R>0$, define the functions

$$\begin{array}{cc}\hfill {\phi }_R(x)& =x^{2}I(|x|\le R/2)+{\{x-\mathrm{sign}(x)·R\}}^{2}I(R/2<|x|\le R),\hfill \end{array}$$

(A24)

which are smoothed proxies of $x\to x^{2}I(|x|\le R)$. Moreover, ${\phi }_R(·)$ is R-Lipschitz continuous and satisfies (i) $x^{2}I(|x|\le R/2)\le {\phi }_R(x)\le x^{2}I(|x|\le R)\le R|x|$ and (ii) ${\phi }_{cR}(cx)=c^2{\phi }_R(x)$ for any $c>0$. Combining (A23), we have

$$\begin{array}{cc}\hfill & ⟨\nabla {\widehat{\mathcal{L}}}_h(\mathit{\beta })-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}),\mathit{\beta }-{\mathit{\beta }}^{*}⟩\hfill \\ \hfill & \ge {\displaystyle \frac{2k_5}{n(n-1)b}}\sum _{i\in {\mathcal{I}}_1}\sum _{j\ne i\in {\mathcal{I}}_1}{\phi }_{b|\mathit{\beta }-{\mathit{\beta }}^{*}{|}_{\mathsf{\Sigma }}/(2r)}({({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }(\mathit{\beta }-{\mathit{\beta }}^{*}))I(|{\epsilon }_i-{\epsilon }_j|\le b/2)\hfill \\ \hfill & \ge 2k_5{|\mathit{\beta }-{\mathit{\beta }}^{*}|}_{\mathsf{\Sigma }}\times \underset{\mathrm{U}(\check{\mathit{\beta }})}{\underbrace{{\displaystyle \frac{1}{n(n-1)b}}{\sum }_{i\in {\mathcal{I}}_1}{\sum }_{j\ne i\in {\mathcal{I}}_1}{\phi }_{b/(2r)}({({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }})I(|{\epsilon }_i-{\epsilon }_j|\le b/2)}}\hfill \end{array}$$

(A25)

with $\check{\mathit{\beta }}=(\mathit{\beta }-{\mathit{\beta }}^{*})/{|\mathit{\beta }-{\mathit{\beta }}^{*}|}_{\mathsf{\Sigma }}$. Recall $\mathsf{\Theta }(r,l)=\{\mathit{\beta }\in {\mathbb{R}}^p:\mathit{\beta }-{\mathit{\beta }}^{*}\in {\mathbb{B}}_{\mathsf{\Sigma }}(r)\cap {\mathbb{C}}_{\mathsf{\Sigma }}(l)\}$. As we will show in Appendix C.4 and Appendix C.5,

$$\begin{array}{c}\hfill \underset{\mathit{\beta }\in \mathsf{\Theta }(r,l)}{inf}\mathbb{E}\{\mathrm{U}(\check{\mathit{\beta }})\}\ge {\displaystyle \frac{7}{8k_5}}\end{array}$$

(A26)

provided that $b\le {(2k_4{k}_5)}^{-1}$ and $r\le {(256k_2)}^{-1/2}h$, and

$$\begin{array}{c}\hfill \underset{\mathit{\beta }-{\mathit{\beta }}^{*}\in {\mathbb{C}}_{\mathsf{\Sigma }}(l)}{sup}|(1-\mathbb{E})\{\mathrm{U}(\check{\mathit{\beta }})\}|\le {\displaystyle \frac{8k_{1}l{(\log p)}^{1/2}}{r{n}^{1/2}}}+{\displaystyle \frac{7h}{16k_{5}r}}\times {\left({\displaystyle \frac{2u}{n}}\right)}^{1/2}+{\displaystyle \frac{5hu}{6n}}\end{array}$$

(A27)

with probability at least $1-e^{-u}$ for any $u>0$. Taking $u=\log p$, by (A25), we have

$$\begin{array}{cc}\hfill {\displaystyle \frac{⟨\nabla {\widehat{\mathcal{L}}}_h(\mathit{\beta })-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}),\mathit{\beta }-{\mathit{\beta }}^{*}⟩}{|\mathit{\beta }-{\mathit{\beta }}^{*}{|}_{\mathsf{\Sigma }}}}\ge & 2k_5\mathrm{U}(\check{\mathit{\beta }})\hfill \\ \hfill \ge & 2k_5\underset{\mathit{\beta }\in \mathsf{\Theta }(r,l)}{inf}\mathbb{E}\{\mathrm{U}(\check{\mathit{\beta }})\}-\underset{\mathit{\beta }-{\mathit{\beta }}^{*}\in {\mathbb{C}}_{\mathsf{\Sigma }}(l)}{sup}|(1-\mathbb{E})\{\mathrm{U}(\check{\mathit{\beta }})\}|\hfill \\ \hfill \ge & {\displaystyle \frac{7}{4}}-{\displaystyle \frac{16k_1{k}_{5}l{(\log p)}^{1/2}}{r{n}^{1/2}}}-{\displaystyle \frac{7h}{8r}}\times {\left({\displaystyle \frac{2\log p}{n}}\right)}^{1/2}-{\displaystyle \frac{10k_{5}h\log p}{6n}}\hfill \\ \hfill \ge & {\displaystyle \frac{7}{8}}\hfill \end{array}$$

provided that $h\le {(2k_4{k}_5)}^{-1}$ and $r\le {(256k_2)}^{-1/2}h$ and

$$\begin{array}{c}\hfill n\ge max\left\{{\displaystyle \frac{{(55k_1{k}_{5}l)}^2\log p}{r^2}},{\displaystyle \frac{18h^2\log p}{r^2}},{\displaystyle \frac{40k_{5}h\log p}{7}}\right\},\end{array}$$

with probability at least $1-p^{-1}$. Then, we complete the proof of Lemma A3. □

Appendix C.4. Proof of (A26)

Proof of

(A26). By Assumption 2, we have

$$\begin{array}{cc}\hfill \mathbb{E}\{I(|{\epsilon }_i-{\epsilon }_j|\le h/2)|{\mathbf{X}}_i,{\mathbf{X}}_j\}=& {\int }_{-h/2}^{h/2}g(u)\mathrm{d}u=hg(0)+{\int }_{-h/2}^{h/2}\{g(u)-g(0)\}\mathrm{d}u\hfill \\ \hfill \ge & hg(0)-{\int }_{-h/2}^{h/2}|g(u)-g(0)|\mathrm{d}u\ge h/k_5-k_4{\int }_{-h/2}^{h/2}|u|\mathrm{d}u\hfill \\ \hfill \ge & h/k_5-k_4{h}^2/4\ge {\displaystyle \frac{7h}{8k_5}}\hfill \end{array}$$

provided that $h\le {(2k_4{k}_5)}^{-1}$. Hence, by $x^{2}I(|x|\le R/2)\le {\phi }_R(x)$, we have

$$\begin{array}{cc}\hfill \mathbb{E}\{\mathrm{U}(\check{\mathit{\beta }})\}\ge & {\displaystyle \frac{1}{n(n-1)h}}\sum _{i\in {\mathcal{I}}_1}\sum _{j\ne i\in {\mathcal{I}}_1}\mathbb{E}[{\{{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}\}}^2\hfill \\ \hfill & \times I\{|{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}|\le h/(4r)\}I(|{\epsilon }_i-{\epsilon }_j|\le h/2)]\hfill \\ \hfill \ge & {\displaystyle \frac{7}{8k_{5}n(n-1)}}\sum _{i\in {\mathcal{I}}_1}\sum _{j\ne i\in {\mathcal{I}}_1}\mathbb{E}[{\{{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}\}}^{2}I\{|{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}|\le h/(4r)\}]\hfill \end{array}$$

(A28)

provided that $h\le {(2k_4{k}_5)}^{-1}$. Notice that

$$\begin{array}{cc}\hfill & \mathbb{E}[{\{{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}\}}^{2}I(|{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}|\le h/(4r))]\hfill \\ \hfill & =\mathbb{E}[{\{{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}\}}^2]-\mathbb{E}[{\{{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}\}}^{2}I(|{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}|>h/(4r))].\hfill \end{array}$$

(A29)

We have

$$\begin{array}{cc}\hfill \mathbb{E}[{\{{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}\}}^2]=& \mathbb{E}[{\{{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }(\mathit{\beta }-{\mathit{\beta }}^{*})\}}^2]/{|\mathit{\beta }-{\mathit{\beta }}^{*}|}_{\mathsf{\Sigma }}^2\hfill \\ \hfill =& {(\mathit{\beta }-{\mathit{\beta }}^{*})}^{\top }\mathbb{E}({\mathbf{X}}_i{\mathbf{X}}_i^{\top }-{\mathbf{X}}_i{\mathbf{X}}_j^{\top }-{\mathbf{X}}_j{\mathbf{X}}_i^{\top }+{\mathbf{X}}_j{\mathbf{X}}_j^{\top })(\mathit{\beta }-{\mathit{\beta }}^{*})/{|\mathit{\beta }-{\mathit{\beta }}^{*}|}_{\mathsf{\Sigma }}^2\hfill \\ \hfill =& {(\mathit{\beta }-{\mathit{\beta }}^{*})}^{\top }\{\mathbb{E}({\mathbf{X}}_i{\mathbf{X}}_i^{\top })+\mathbb{E}({\mathbf{X}}_j{\mathbf{X}}_j^{\top })\}{(\mathit{\beta }-{\mathit{\beta }}^{*})}^{\top }/{|\mathit{\beta }-{\mathit{\beta }}^{*}|}_{\mathsf{\Sigma }}^2=2.\hfill \end{array}$$

(A30)

By Assumption 1 and Markov inequality, it holds that

$$\begin{array}{cc}\hfill & \mathbb{E}[{\{{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}\}}^{2}I(|{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}|>h/(4r))]\hfill \\ \hfill & \le {(\mathbb{E}[{\{{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}\}}^4])}^{1/2}[\mathbb{P}\{|{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}|>h/(4r){\}]}^{1/2}\hfill \\ \hfill & \le 16{(r/h)}^2\mathbb{E}[{\{{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\check{\mathit{\beta }}\}}^4]\hfill \\ \hfill & \le 16{(r/h)}^2\underset{\mathbf{u}\in {\mathbb{S}}^{p-1}}{sup}\mathbb{E}[{\{{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\mathit{u}\}}^4]\hfill \\ \hfill & \le 256{(r/h)}^2\underset{\mathbf{u}\in {\mathbb{S}}^{p-1}}{sup}\mathbb{E}\{{({\mathbf{X}}^{\top }\mathit{u})}^4\}\hfill \\ \hfill & \le 256k_2{(r/h)}^2.\hfill \end{array}$$

Together with (A29) and (A30), by (A28), we have

$$\begin{array}{c}\hfill \mathbb{E}\{\mathrm{U}(\check{\mathit{\beta }})\}\ge {\displaystyle \frac{7\{2-256k_2{(r/h)}^2\}}{8k_5}}\ge {\displaystyle \frac{7}{8k_5}}\end{array}$$

provided that $b\le {(2k_4{k}_5)}^{-1}$ and $r\le {(256k_2)}^{-1/2}h$. Then, (A26) holds. □

Appendix C.5. Proof of (A27)

Proof of

(A27). Write $\varpi ({\mathbf{X}}_i,{\mathbf{X}}_j,\mathit{\delta })={\phi }_{h/(2r)}({({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\mathit{\delta }/{|\mathit{\delta }|}_{\mathsf{\Sigma }})I(|{\epsilon }_i-{\epsilon }_j|\le h/2)/h$. Since $|{\phi }_R(x)|\le R|x|I(|x|\le R)\le R^2$, we have $|\varpi ({\mathbf{X}}_i,{\mathbf{X}}_j,\mathit{\delta })|\le h/4$ and

$$\begin{array}{cc}\hfill \mathbb{E}\{{\varpi }^2({\mathbf{X}}_i,{\mathbf{X}}_j,\mathit{\delta })\}\le & {\displaystyle \frac{1}{4}}\mathbb{E}[\{{({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\mathit{\delta }/{|\mathit{\delta }|}_{\mathsf{\Sigma }}{\}}^2\mathbb{E}\{I(|{\epsilon }_i-{\epsilon }_j|\le h/2)|{\mathbf{X}}_i,{\mathbf{X}}_j\}]\hfill \\ \hfill \le & {\displaystyle \frac{7h}{32k_5}}\mathit{\delta }\mathbb{E}\{({\mathbf{X}}_i-{\mathbf{X}}_j){({\mathbf{X}}_i-{\mathbf{X}}_j)}^{\top }\}\mathit{\delta }/{|\mathit{\delta }|}_{\mathsf{\Sigma }}^2\hfill \\ \hfill \le & {\displaystyle \frac{7h}{16k_{5}r}}.\hfill \end{array}$$

By Theorem 2 of Rejchel [24], we have

$$\begin{array}{cc}\hfill \underset{\mathit{\beta }-{\mathit{\beta }}^{*}\in {\mathbb{C}}_{\mathsf{\Sigma }}(l)}{sup}|(1-\mathbb{E})\{\mathrm{U}(\check{\mathit{\beta }})\}|=& \underset{\mathit{\delta }\in {\mathbb{C}}_{\mathsf{\Sigma }}(l)}{sup}\left|{\displaystyle \frac{1}{n(n-1)}}\sum _{i\in {\mathcal{I}}_1}\sum _{j\ne i\in {\mathcal{I}}_1}(1-\mathbb{E})\{\varpi ({\mathbf{X}}_i,{\mathbf{X}}_j,\mathit{\delta })\}\right|\hfill \\ \hfill \le & 4\underset{{\mathrm{U}}_1}{\underbrace{\mathbb{E}\left\{{sup}_{\mathit{\delta }\in {\mathbb{C}}_{\mathsf{\Sigma }}(l)}\left|{\displaystyle \frac{1}{\lfloor n/2\rfloor }}\sum _{i=1}^{\lfloor n/2\rfloor }{\sigma }_i\varpi ({\mathbf{X}}_i,{\mathbf{X}}_{\lfloor n/2\rfloor +i},\mathit{\delta })\right|\right\}}}\hfill \\ \hfill & +{\displaystyle \frac{7b}{16k_{5}r}}\times {\left({\displaystyle \frac{2u}{n}}\right)}^{1/2}+{\displaystyle \frac{5bu}{6n}}\hfill \end{array}$$

(A31)

with probability at least $1-e^{-u}$ for any $u>0$, where ${\{{\sigma }_i\}}_{i=1}^{\lfloor n/2\rfloor }$ is a sequence of Rademacher random variables. Write ${\mathbf{X}}_i={(X_{i,1},\dots ,X_{i,p})}^{\top }$. Due to ${\psi }_{b/(2r)}(x)/b$ being a $1/(2r)$-Lipschitz function, by Talagrand’s contraction principle, we have

$$\begin{array}{cc}\hfill {\mathrm{U}}_1\le & {\displaystyle \frac{1}{r}}\mathbb{E}\left\{\underset{\mathit{\delta }\in {\mathbb{C}}_{\mathsf{\Sigma }}(l)}{sup}|{\displaystyle \frac{1}{\lfloor n/2\rfloor }}\sum _{i=1}^{\lfloor n/2\rfloor }{\sigma }_i{({\mathbf{X}}_i-{\mathbf{X}}_{\lfloor n/2\rfloor +i})}^{\top }\mathit{\delta }I(|{\epsilon }_i-{\epsilon }_{\lfloor n/2\rfloor +i}{|\le b/2)/|\mathit{\delta }|}_{\mathsf{\Sigma }}|\right\}\hfill \\ \hfill \le & {\displaystyle \frac{l}{r}}\mathbb{E}\left\{|{\displaystyle \frac{1}{\lfloor n/2\rfloor }}\sum _{i=1}^{\lfloor n/2\rfloor }{\sigma }_i({\mathbf{X}}_i-{\mathbf{X}}_{\lfloor n/2\rfloor +i})I(|{\epsilon }_i-{\epsilon }_{\lfloor n/2\rfloor +i}|\le b/2){|}_{\infty }\right\}\hfill \\ \hfill =& {\displaystyle \frac{l}{r}}\mathbb{E}\left\{\underset{k\in [p]}{max}|{\displaystyle \frac{1}{\lfloor n/2\rfloor }}\sum _{i=1}^{\lfloor n/2\rfloor }{\sigma }_i(X_{i,k}-X_{\lfloor n/2\rfloor +i,k})I(|{\epsilon }_i-{\epsilon }_{\lfloor n/2\rfloor +i}|\le b/2)|\right\}\hfill \\ \hfill \le & {\displaystyle \frac{2k_{1}l{(\log p)}^{1/2}}{r{(\lfloor n/2\rfloor )}^{1/2}}}\le {\displaystyle \frac{8k_{1}l{(\log p)}^{1/2}}{r{n}^{1/2}}},\hfill \end{array}$$

where the last inequality is due to ${\sigma }_i({\mathbf{X}}_{i,k}-{\mathbf{X}}_{\lfloor n/2\rfloor +i,k})I\{|{\epsilon }_{\pi (i)}-{\epsilon }_{\pi (\lfloor n/2\rfloor +i)}|\le b/2\}$ being a sub-Gaussian random variable for any $i\in [\lfloor n/2\rfloor ]$ and $k\in [p]$. By (A31), it holds that

$$\begin{array}{c}\hfill \underset{\mathit{\beta }-{\mathit{\beta }}^{*}\in {\mathbb{C}}_{\mathsf{\Sigma }}(l)}{sup}|(1-\mathbb{E})\{\mathrm{U}(\check{\mathit{\beta }})\}|\le {\displaystyle \frac{8k_{1}l{(\log p)}^{1/2}}{r{n}^{1/2}}}+{\displaystyle \frac{7b}{16k_{5}r}}\times {\left({\displaystyle \frac{2u}{n}}\right)}^{1/2}+{\displaystyle \frac{5bu}{6n}}\end{array}$$

with probability at least $1-e^{-u}$ for any $u>0$. Hence, (A27) holds. □

Appendix C.6. Proof of Lemma A4

Proof of Lemma A4.

Let $\tilde{\mathit{\mu }}={\widehat{\mathit{\beta }}}^{(0)}-{\mathit{\beta }}^{*}$. As we show in Appendix C.7 that

$$\begin{array}{c}\hfill |\tilde{\mathit{\mu }}{|}_{\mathsf{\Sigma }}\le r_0\end{array}$$

(A32)

with probability at least $1-p^{-1}$, provided that $16k_2^{1/2}{r}_1\le h\le {(2k_4{k}_5)}^{-1}$, ${\tau }_1{r}_1^2\ge 7{(55k_1{k}_5)}^2/8$ and ${(18h^2\log p/n)}^{1/2}\le r_1\le {(32k_2^{1/2}{k}_4{k}_5)}^{-1}$ and

$$\begin{array}{c}\hfill R\le min\left\{{\displaystyle \frac{7r_1^2}{8k_8{\lambda }_0}},{\displaystyle \frac{{\tilde{C}}_1{r}_1^2{n}^{1/2}}{{(\log p)}^{1/2}}},{\left({\displaystyle \frac{7r_1^{2}n}{8{\tau }_1\log p}}\right)}^{1/2}\right\}.\end{array}$$

(A33)

Then, by (A2) and (A32), we have

$$\begin{array}{c}\hfill ⟨\nabla {\widehat{\mathcal{L}}}_{1,h}({\widehat{\mathit{\beta }}}^{(0)})-\nabla {\widehat{\mathcal{L}}}_{1,h}({\mathit{\beta }}^{*}),\tilde{\mathit{\mu }}⟩=⟨\nabla {\widehat{\mathcal{L}}}_h({\widehat{\mathit{\beta }}}^{(0)})-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}),\tilde{\mathit{\mu }}⟩\ge {\displaystyle \frac{7|\tilde{\mathit{\mu }}{|}_{\mathsf{\Sigma }}^2}{8}}-{\displaystyle \frac{{\tau }_1\log p{|\tilde{\mathit{\mu }}|}_1^2}{n}}\end{array}$$

with probability at least $1-2p^{-1}$, provided that $16k_2^{1/2}{r}_1\le h\le {(2k_4{k}_5)}^{-1}$, ${\tau }_1{r}_1^2\ge 7{(55k_1{k}_5)}^2/8$ and ${(18h^2\log p/n)}^{1/2}\le r_1\le {(32k_2^{1/2}{k}_4{k}_5)}^{-1}$ and R satisfies (A33). Write

$$\begin{array}{c}\hfill {\tilde{\mathsf{\Pi }}}_3={|\nabla {\widehat{\mathcal{L}}}_{1,h}({\mathit{\beta }}^{*})|}_{\infty }.\end{array}$$

Analogously to the proof of (A10), we have

$$\begin{array}{cc}\hfill {\displaystyle \frac{7-4k_3\mu }{8}}·{|\tilde{\mathit{\mu }}|}_{\mathsf{\Sigma }}^2\le & p_{{\lambda }_0}({\mathit{\beta }}^{*})-p_{{\lambda }_0}({\widehat{\mathit{\beta }}}^{(0)})-⟨\nabla {\widehat{\mathcal{L}}}_{1,h}({\mathit{\beta }}^{*}),\tilde{\mathit{\mu }}⟩+{\displaystyle \frac{{\tau }_1\log p{|\tilde{\mathit{\mu }}|}_1^2}{n}}\hfill \\ \hfill \le & p_{{\lambda }_0}({\mathit{\beta }}^{*})-p_{{\lambda }_0}({\widehat{\mathit{\beta }}}^{(0)})+{\tilde{\mathsf{\Pi }}}_3·{|\tilde{\mathit{\mu }}|}_1+{\displaystyle \frac{{\tau }_1\log p{|\tilde{\mathit{\mu }}|}_1^2}{n}}\hfill \\ \hfill \le & p_{{\lambda }_0}({\mathit{\beta }}^{*})-p_{{\lambda }_0}({\widehat{\mathit{\beta }}}^{(0)})+\left\{{\tilde{\mathsf{\Pi }}}_3+{\displaystyle \frac{2R{\tau }_1\log p}{n}}\right\}·{|\tilde{\mathit{\mu }}|}_1\hfill \end{array}$$

with probability at least $1-2p^{-1}$, provided that $16k_2^{1/2}{r}_1\le h\le {(2k_4{k}_5)}^{-1}$, ${\tau }_1{r}_1^2\ge 7{(55k_1{k}_5)}^2/8$ and ${(18h^2\log p/n)}^{1/2}\le r_1\le {(32k_2^{1/2}{k}_4{k}_5)}^{-1}$ and R satisfies (A33). Let ${\lambda }_0$ satisfy that

$$\begin{array}{c}\hfill {\lambda }_0\ge 4k_8^{-1}\left\{{\tilde{\mathsf{\Pi }}}_3+{\displaystyle \frac{2R{\tau }_1\log p}{n}}\right\}.\end{array}$$

Recall $0<\mu <7/(5k_3)$. Analogously to the deviation (A11), it holds that

$$\begin{array}{c}\hfill 0\le {\displaystyle \frac{7-5k_3\mu }{8}}·{|\tilde{\mathit{\mu }}|}_{\mathsf{\Sigma }}^2\le {\displaystyle \frac{3p_{{\lambda }_0}({\mathit{\beta }}^{*})}{2}}-{\displaystyle \frac{p_{{\lambda }_0}({\widehat{\mathit{\beta }}}^{(0)})}{2}}\end{array}$$

with probability at least $1-2p^{-1}$, provided that $16k_2^{1/2}{r}_1\le h\le {(2k_4{k}_5)}^{-1}$, ${\tau }_1{r}_1^2\ge 7{(55k_1{k}_5)}^2/8$ and ${(18h^2\log p/n)}^{1/2}\le r_1\le {(32k_2^{1/2}{k}_4{k}_5)}^{-1}$ and R satisfies (A33). Recall $\mathcal{S}=\mathrm{supp}({\mathit{\beta }}^{*})\subseteq [p]$ can be the true active set with cardinality $|\mathcal{S}|\le s$. By Lemma A.5 of Loh and Wainwright [21], we have

$$\begin{array}{c}\hfill 0\le 3p_{{\lambda }_0}({\mathit{\beta }}^{*})-p_{{\lambda }_0}({\widehat{\mathit{\beta }}}^{(0)})\le 3k_8{\lambda }_0|{\tilde{\mathit{\mu }}}_{\mathcal{S}}{|}_1-k_8{\lambda }_0{|{\tilde{\mathit{\mu }}}_{{\mathcal{S}}^{\mathrm{c}}}|}_1,\end{array}$$

which implies that $|{\tilde{\mathit{\mu }}}_{{\mathcal{S}}^{\mathrm{c}}}{|}_1\le 3{|{\tilde{\mathit{\mu }}}_{\mathcal{S}}|}_1$, with probability at least $1-2p^{-1}$, provided that $16k_2^{1/2}{r}_1\le h\le {(2k_4{k}_5)}^{-1}$, ${\tau }_1{r}_1^2\ge 7{(55k_1{k}_5)}^2/8$ and ${(18h^2\log p/n)}^{1/2}\le r_1\le {(32k_2^{1/2}{k}_4{k}_5)}^{-1}$ and R satisfies (A33). Hence,

$$\begin{array}{c}\hfill |\tilde{\mathit{\mu }}{|}_1=|{\tilde{\mathit{\mu }}}_{\mathcal{S}}{|}_1+|{\tilde{\mathit{\mu }}}_{{\mathcal{S}}^{\mathrm{c}}}{|}_1\le 4s^{1/2}{|{\tilde{\mathit{\mu }}}_{\mathcal{S}}|}_2,\end{array}$$

with probability at least $1-2p^{-1}$, provided that $16k_2^{1/2}{r}_1\le h\le {(2k_4{k}_5)}^{-1}$, ${\tau }_1{r}_1^2\ge 7{(55k_1{k}_5)}^2/8$ and ${(18h^2\log p/n)}^{1/2}\le r_1\le {(32k_2^{1/2}{k}_4{k}_5)}^{-1}$ and R satisfies (A33). Then, we obtain that $\{{\widehat{\mathit{\beta }}}^{(0)}\in \mathsf{\Lambda }\}$ with probability at least $1-2p^{-1}$, provided that $16k_2^{1/2}{r}_1\le h\le {(2k_4{k}_5)}^{-1}$, ${\tau }_1{r}_1^2\ge 7{(55k_1{k}_5)}^2/8$ and ${(18h^2\log p/n)}^{1/2}\le r_1\le {(32k_2^{1/2}{k}_4{k}_5)}^{-1}$ and R satisfies (A33). Analogous to the proof of (A13), we have

$$\begin{array}{c}\hfill |\tilde{\mathit{\mu }}{|}_{\mathsf{\Sigma }}\lesssim s^{1/2}{\lambda }_0\end{array}$$

(A34)

with probability at least $1-2p^{-1}$, provided that $16k_2^{1/2}{r}_1\le h\le {(2k_4{k}_5)}^{-1}$, ${\tau }_1{r}_1^2\ge 7{(55k_1{k}_5)}^2/8$ and ${(18h^2\log p/n)}^{1/2}\le r_1\le {(32k_2^{1/2}{k}_4{k}_5)}^{-1}$ and R satisfies (A33). Analogous to the proof of Lemma A2, we can show that

$$\begin{array}{c}\hfill {\tilde{\mathsf{\Pi }}}_3\lesssim {\displaystyle \frac{{(\log p+u)}^{1/2}}{n^{1/2}}}\end{array}$$

(A35)

for any $u>0$, with probability at least $1-e^{-u}$. To ensure that $|{\widehat{\mathit{\beta }}}^{(0)}-{\mathit{\beta }}^{*}{|}_{\mathsf{\Sigma }}$ converges as fast as possible, selecting $r_1=O(1)$, ${\tau }_1=O(1)$, $h=O(1)$, $R\asymp n^{1/2}{(\log p)}^{-1/2}$ and

$$\begin{array}{c}\hfill {\lambda }_0\asymp {\displaystyle \frac{{(\log p)}^{1/2}}{n^{1/2}}}.\end{array}$$

Together with (A34), it holds that

$$\begin{array}{c}\hfill |{\widehat{\mathit{\beta }}}^{(0)}-{\mathit{\beta }}^{*}{|}_{\mathsf{\Sigma }}\lesssim {\displaystyle \frac{{(s\log p)}^{1/2}}{n^{1/2}}},\end{array}$$

provided that $h=O(1)$, $R\asymp {(n/\log p)}^{1/2}$ and ${\lambda }_0\asymp {(\log p/n)}^{1/2}$. Then, we complete the proof of Lemma A4. □

Appendix C.7. Proof of (A32)

Proof of

(A32). Notice that If $|\tilde{\mathit{\mu }}{|}_{\mathsf{\Sigma }}>r_0$, by $\nabla {\widehat{\mathcal{L}}}_{1,h}({\mathit{\beta }}^{*}+\tilde{\mathit{\mu }})=-\nabla p_{{\lambda }_0}({\widehat{\mathit{\beta }}}^{(0)})$ and (A3), we have

$$\begin{array}{cc}\hfill ⟨-\nabla p_{{\lambda }_0}({\widehat{\mathit{\beta }}}^{(0)})-\nabla {\widehat{\mathcal{L}}}_{1,h}({\mathit{\beta }}^{*}),\mathit{\mu }⟩=& ⟨\nabla {\widehat{\mathcal{L}}}_{1,h}({\mathit{\beta }}^{*}+\tilde{\mathit{\mu }})-\nabla {\widehat{\mathcal{L}}}_{1,h}({\mathit{\beta }}^{*}),\tilde{\mathit{\mu }}⟩\hfill \\ \hfill =& ⟨\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}+\tilde{\mathit{\mu }})-\nabla {\widehat{\mathcal{L}}}_h({\mathit{\beta }}^{*}),\tilde{\mathit{\mu }}⟩\hfill \\ \hfill \ge & {\displaystyle \frac{7r_1{|\tilde{\mathit{\mu }}|}_{\mathsf{\Sigma }}}{8}}-{\displaystyle \frac{2R{\tau }_1\log p{|\tilde{\mathit{\mu }}|}_1}{n}}\hfill \end{array}$$

(A36)

provided that $16k_2^{1/2}{r}_1\le h\le {(2k_4{k}_5)}^{-1}$, ${\tau }_1{r}_1^2\ge 7{(55k_1{k}_5)}^2/8$ and ${(18h^2\log p/n)}^{1/2}\le r_1\le {(32k_2^{1/2}{k}_4{k}_5)}^{-1}$, with probability at least $1-p^{-1}$. By Assumption 4, the Hölder inequality and Lemma 4 of Loh and Wainwright [21], we have

$$\begin{array}{cc}\hfill ⟨-\nabla p_{{\lambda }_0}({\widehat{\mathit{\beta }}}^{(0)})-\nabla {\widehat{\mathcal{L}}}_{1,h}({\mathit{\beta }}^{*}),\tilde{\mathit{\mu }}⟩\le & \{|\nabla p_{{\lambda }_0}({\widehat{\mathit{\beta }}}^{(0)}){|}_{\infty }+|\nabla {\widehat{\mathcal{L}}}_{1,h}({\mathit{\beta }}^{*}){|}_{\infty }\}·|\tilde{\mathit{\mu }}{|}_1\hfill \\ \hfill \le & (k_8{\lambda }_0+{\tilde{\mathsf{\Pi }}}_3){|\tilde{\mathit{\mu }}|}_1.\hfill \end{array}$$

Recall ${\tilde{\mathsf{\Pi }}}_3={|\nabla {\widehat{\mathcal{L}}}_{1,h}({\mathit{\beta }}^{*})|}_{\infty }$. and $|\tilde{\mathit{\mu }}{|}_1\le 2R$. Together with (A35) and (A36), selecting $u=2\log p$, we have

$$\begin{array}{c}\hfill {\displaystyle \frac{7r_1{|\tilde{\mathit{\mu }}|}_{\mathsf{\Sigma }}}{8}}\le \left\{k_8{\lambda }_0+{\displaystyle \frac{C_1{(\log p)}^{1/2}}{n^{1/2}}}+{\displaystyle \frac{2R{\tau }_1\log p}{n}}\right\}R\end{array}$$

which implies that

$$\begin{array}{c}\hfill |\tilde{\mathit{\mu }}{|}_{\mathsf{\Sigma }}\le {\displaystyle \frac{8}{7r_1}}\left\{k_8{\lambda }_0+{\displaystyle \frac{C_1{(\log p)}^{1/2}}{n^{1/2}}}+{\displaystyle \frac{2R{\tau }_1\log p}{n}}\right\}R\le r_1\end{array}$$

with probability at least $1-2p^{-1}$, provided that $16k_2^{1/2}{r}_1\le h\le {(2k_4{k}_5)}^{-1}$, ${\tau }_1{r}_1^2\ge 7{(55k_1{k}_5)}^2/8$ and ${(18h^2\log p/n)}^{1/2}\le r_1\le {(32k_2^{1/2}{k}_4{k}_5)}^{-1}$ and

$$\begin{array}{c}\hfill R\le min\left\{{\displaystyle \frac{7r_1^2}{8k_8{\lambda }_0}},{\displaystyle \frac{{\tilde{C}}_1{r}_1^2{n}^{1/2}}{{(\log p)}^{1/2}}},{\left({\displaystyle \frac{7r_1^{2}n}{8{\tau }_1\log p}}\right)}^{1/2}\right\},\end{array}$$

where ${\tilde{C}}_1>0$ is a sufficiently small constant. Hence, (A32) holds. □