Distributed Estimation for ℓ₀-Constrained Quantile Regression Using Iterative Hard Thresholding

Abstract

Distributed frameworks for statistical estimation and inference have become a critical toolkit for analyzing massive data efficiently. In this paper, we present distributed estimation for high-dimensional quantile regression with ${\mathsf{\ell }}_0$ constraint using iterative hard thresholding (IHT). We propose a communication-efficient distributed estimator which is linearly convergent to the true parameter up to the statistical precision of the model, despite the fact that the check loss minimization problem with an ${\mathsf{\ell }}_0$ constraint is neither strongly smooth nor convex. The distributed estimator we develop can achieve the same convergence rate as the estimator based on the whole data set under suitable assumptions. In our simulations, we illustrate the convergence of the estimators under different settings and also demonstrate the accuracy of nonzero parameter identification.

1. Introduction

In statistical modeling, it is often the case that the response is only associated with a subset of predictors. Researchers tend to exclude irrelevant variables, referred to as variable selection, for better prediction accuracy and model interpretability [1]. A common approach is to constrain or regularize the coefficient estimates, or equivalently shrink the coefficient estimates towards zero. The shrinkage also has the effect of reducing variance. Depending on the type of shrinkage performed, some of the coefficients may be estimated to be exactly zero. The best-known techniques include ridge regression and the lasso [2]. In addition to these methods, we can also add an ${\mathsf{\ell }}_0$ constraint to yield sparse coefficient estimates [3].

In high-dimensional statistics, ${\mathsf{\ell }}_0$ regularization is widely used in feature selection. It removes noninformative features by penalizing nonzero coefficients and can ultimately result in a sparse model with a subset of the most significant features, serving as an efficient tool for model interpretation and prediction. Compared to other popular approaches such as lasso, ${\mathsf{\ell }}_0$ constraint has the advantage of directly specifying the sparsity level, which may be useful in some scientific investigations (for example, biological researchers may be interested in extracting the top 50 genes for further investigation), but at the same time, the nonconvex nature of the constraint causes some algorithmic and theoretical challenges. Among the known methods for optimization problems with ${\mathsf{\ell }}_0$ constraint, iterative hard thresholding (IHT), which combines gradient descent with a projection operation, offers a fast and scalable solution. Ref. [4] applied this method to models in high-dimensional statistical settings and provided convergence guarantees that hold for differentiable functions, which do not directly apply to non-smooth losses. Ref. [5] extended his work and established the linear convergence of IHT under certain statistical precision for the quantile regression model where the check loss function is non-smooth.

Quantile regression proposed by [6], a natural extension to linear regression, estimates the conditional median or other quantiles rather than the conditional mean of the response variables given the predictors. It can thus provide more information about the response distribution. Furthermore, the quantile regression estimates are more robust against outliers in the response measurements [7]. It also has a number of excellent statistical properties, such as invariance to monotone transformation [8]. The empirical applications of quantile regression have appeared in many fields, such as economics, survival analysis, and ecology [9,10,11].

Literature Review on Distributed Estimation and Our Contribution

With the rapid development of information technology, there has been a staggering increase in the scale and scope of data collection. Meanwhile, the network bandwidth and privacy or security concerns set limitations on processing a large amount of data on one single machine. Inspired by the idea of divide-and-conquer, distributed frameworks for statistical estimation and inference have become a critical toolkit for researchers to understand complicated large data. Various distributed methods have been proposed [12,13,14], allowing us to store data separately and utilize the computing power of all machines by analyzing data simultaneously. Carefully designed algorithms could improve the performance in a distributed system. Ref. [15] proposed a refinement of the simple averaging estimator involving Hessian matrices to be computed and transferred, which leads to a heavy communication cost of the order $O\left(d^2\right)$ when the parameter dimension d is high. To tackle this problem, ref. [16] conducted Newton-type iteration distributedly instead, without transferring the Hessian matrices. With a similar strategy, ref. [17] suggested a well-designed distributed framework and introduced an approximate likelihood approach. It can dramatically reduce the communication cost by replacing the transmission of higher-order derivatives with local first-order derivatives. Following the existing key ideas, in this paper, our research question is as follows:

How could distributed estimation methods with ${\mathsf{\ell }}_0$-constraints be designed to achieve convergence rates comparable to centralized estimators in sparse quantile regression models?

The main contribution of the current work is to present distributed estimation for the above-stated problem with nonconvex constraint and non-smooth loss, distinguishing our work from the existing ones. We adapt the framework of [17] to develop a communication-efficient distributed method and provide the convergence rate theoretically. The conclusions of [17] on the convergence of the distributed system that are valid for loss functions with continuous second derivatives cannot be directly applied to the non-smooth check loss function for quantile regression. We show that under suitable assumptions, the distributed estimator has the same convergence rate as the estimator utilizing the whole data set. Existing distributed learning with strong theoretical guarantees is mostly concerned with smooth convex models, and thus, our work fills in an important gap in the literature.

In the next section, we present the distributed estimator for quantile regression with ${\mathsf{\ell }}_0$ constraint. In Section 3, we provide the convergence rate of the distributed estimator. Some numerical experiments are presented in Section 4 to investigate the finite-sample performance of the estimator. We conclude the paper with some discussions in Section 5. The proofs are contained in the Appendix A.

2. Background and Methodology

We begin by giving a description of quantile regression with an ${\mathsf{\ell }}_0$ constraint. We then turn to developing a communication-efficient distributed estimator.

2.1. Quantile Regression with ${\mathsf{\ell }}_0$ Constraint

Let the sample $\left(y_i,{\mathbf{x}}_i={(x_{i1},\dots ,x_{id})}^{\mathrm{T}}\right)$, $i=1,\dots ,n$ be independent and identically distributed (i.i.d.) and satisfy

$$y_i={\mu }_0+{\mathbf{x}}_i^{\mathrm{T}}{\mathit{\theta }}_0+{ϵ}_i,$$

(1)

with $P({ϵ}_i\le 0|{\mathbf{x}}_i)=\tau$, where $\mathbf{x}\in {\mathbb{R}}^d$ and $({\mu }_0,{\mathit{\theta }}_0)$ denotes the true parameter values. The sparsity of ${\mathit{\theta }}_0$ (that is, number of nonzero components) is denoted as $s_0$. We assume s is a known upper bound to $s_0$. The estimator for quantile regression with ${\mathsf{\ell }}_0$ constraint is defined as

$$\widehat{\mathit{\theta }}=\underset{\mu ,\mathit{\theta }\in {\mathbb{R}}^d:{\parallel \mathit{\theta }\parallel }_0\le s}{arg\; min}{\displaystyle \frac{1}{n}}\sum _i{\rho }_{\tau }(y_i-\mu -{\mathbf{x}}_i^{\mathrm{T}}\mathit{\theta }),$$

where ${\rho }_{\tau }\left(x\right)=x(\tau -I\{x\le 0\})$ is the piecewise linear check loss and $\tau \in (0,1)$ represents the quantile level. In Figure 1, we show the loss function ${\rho }_{\tau }(.)$ for different values of $\tau$. When $\tau =0.5$, the loss function becomes ${\rho }_{0.5}\left(x\right)={\displaystyle \frac{1}{2}}\left|x\right|$. To see why this makes sense, we note that it is well-known that given a sample $\{y_1,\dots ,y_n\}$, the minimizer of ${\min }_a{\sum }_{i=1}^n{\displaystyle \frac{1}{2}}|y_i-a|$ is the sample median (the minimizer of the least squares loss is the sample mean). For other values of $\tau$, the loss function has a similar shape to absolute value function but is skewed so that it can recover other quantiles. For details, see [6]. The intercept $\mu$ would be omitted in the following for the simplicity of notation.

A subgradient of the check loss function is $(1/n){\sum }_i{\mathbf{x}}_i(I\{y_i-{\mathbf{x}}_i^{\mathrm{T}}\mathit{\theta }\le 0\}-\tau )$. With an initial value ${\mathit{\theta }}^0$, we can apply the iterative hard thresholding algorithm, which is a projected gradient descent method for the nonconvex case, to compute the estimator $\widehat{\mathit{\theta }}$ iteratively as

$${\mathit{\theta }}^{t+1}=P_s\left({\mathit{\theta }}^t-\eta {\displaystyle \frac{1}{n}}\sum _i{\mathbf{x}}_i(I\{y_i-{\mathbf{x}}_i^{\mathrm{T}}{\mathit{\theta }}^t\le 0\}-\tau )\right),t=0,1,\dots ,$$

where $\eta >0$ denotes the step size. After a sufficiently large number of iterations T, we set $\widehat{\mathit{\theta }}={\mathit{\theta }}^T$. In the above, the projection operator $P_s\left(\mathit{\theta }\right)$ enforces the ${\mathsf{\ell }}_0$ constraint by retaining s elements with the largest absolute values (in magnitude) and setting the other entries of $\mathit{\theta }$ to be zero.

2.2. Distributed Estimation

In this subsection, we apply the communication-efficient surrogate likelihood (CSL) framework, proposed by [17], to solve ${\mathsf{\ell }}_0$-constraint quantile regression.

We first review the definitions and notations required in the distributed framework. In a distributed setting, N observations are randomly and evenly distributed to m machines with n observations stored on each one. Let $\mathbf{Z}:=\{Z_{ij},i=1,\dots ,n,j=1,\dots ,m\}$ be a set of $N=mn$ observations independently and identically distributed with the distribution ${\mathbb{P}}_{{\mathit{\theta }}_0}$, where $\{{\mathbb{P}}_{\mathit{\theta }}:\mathit{\theta }\in \Theta$} is a family of statistical models parameterized by $\mathit{\theta }\in \mathbf{\Theta }\subset {\mathbb{R}}^d$. Here, $\mathbf{\Theta }$ is the parameter space and ${\mathit{\theta }}_0$ is the true parameter generating the data set. In a regression problem, we have $Z_{ij}=({\mathbf{x}}_{ij},y_{ij})$. We use ${\mathbf{Z}}_j:=\{Z_{ij},i=1,\dots ,n\}$ to denote the n observations stored on the jth machine. In particular, ${\mathbf{Z}}_1$ stands for the data on the first machine, which is regarded as the central machine (server) that can directly communicate with all others, while other machines are worker machines that can only communicate with the central machine.

Ref. [17] presented a surrogate loss function to approximate the global loss function, from which we can derive the surrogate loss function $\tilde{\mathcal{L}}\left(\mathit{\theta }\right)$ for the quantile regression model as

$$\tilde{\mathcal{L}}\left(\mathit{\theta }\right)={\mathcal{L}}_1\left(\mathit{\theta }\right)-〈\mathit{\theta },\nabla {\mathcal{L}}_1\left(\tilde{\mathit{\theta }}\right)-\nabla {\mathcal{L}}_N\left(\tilde{\mathit{\theta }}\right)〉,$$

(2)

where $\tilde{\mathit{\theta }}$ is a suitable initial estimator. For example, $\tilde{\mathit{\theta }}$ can be the minimizer of the empirical loss function on the first machine. ${\mathcal{L}}_1\left(\mathit{\theta }\right)$ is the local loss function on the first machine and $\nabla {\mathcal{L}}_1\left(\mathit{\theta }\right)$ is the subgradient of ${\mathcal{L}}_1\left(\mathit{\theta }\right)$, that is, ${\mathcal{L}}_1\left(\mathit{\theta }\right)=(1/n){\displaystyle \sum _{i=1}^n}{\rho }_{\tau }(y_{i1}-{\mathbf{x}}_{i1}^{\mathrm{T}}\mathit{\theta })$, $\nabla {\mathcal{L}}_1\left(\mathit{\theta }\right)=(1/n)){\displaystyle \sum _{i=1}^n}{\mathbf{x}}_{i1}(I\left\{y_{i1}-{\mathbf{x}}_{i1}^{\mathrm{T}}\mathit{\theta }\le 0\right\}-\tau )$. ${\mathcal{L}}_N\left(\mathit{\theta }\right)$ is the global loss function based on all observations, and $\nabla {\mathcal{L}}_N\left(\mathit{\theta }\right)$ is the subgradient of ${\mathcal{L}}_N\left(\mathit{\theta }\right)$. To understand the meaning of $\tilde{\mathcal{L}}$ in (2), we note that the first term ${\mathcal{L}}_1\left(\mathit{\theta }\right)$ is the loss based on data on machine 1. The desired global loss is ${\mathcal{L}}_N\left(\mathit{\theta }\right)$. Thus, to correct for the difference between ${\mathcal{L}}_1\left(\mathit{\theta }\right)$ and ${\mathcal{L}}_N\left(\mathit{\theta }\right)$, the second term $〈\mathit{\theta },\nabla {\mathcal{L}}_1\left(\tilde{\mathit{\theta }}\right)-\nabla {\mathcal{L}}_N\left(\tilde{\mathit{\theta }}\right)〉$ in (2) adjusts the gradient of the loss at the current estimate $\tilde{\mathit{\theta }}$, to make $\tilde{\mathcal{L}}\left(\mathit{\theta }\right)$ approximate ${\mathcal{L}}_N\left(\mathit{\theta }\right)$ better.

Using the surrogate loss function, the communication-efficient distributed estimator is

$$\stackrel{\vee }{\mathit{\theta }}\in \underset{{\mathit{\theta }:\parallel \mathit{\theta }\parallel }_0\le s}{arg\; min}\tilde{\mathcal{L}}\left(\mathit{\theta }\right).$$

In practice, we apply IHT to compute the distributed estimator $\stackrel{\vee }{\mathit{\theta }}$, noting that the subgradient of the surrogate loss function is $\nabla \tilde{\mathcal{L}}\left(\mathit{\theta }\right)=\nabla {\mathcal{L}}_1\left(\mathit{\theta }\right)-\left(\nabla {\mathcal{L}}_1\left(\tilde{\mathit{\theta }}\right)-\nabla {\mathcal{L}}_N\left(\tilde{\mathit{\theta }}\right)\right)$. Note that to obtain $\nabla {\mathcal{L}}_N\left(\tilde{\mathit{\theta }}\right)$ in the distributed setting, $\tilde{\mathit{\theta }}$ needs to be broadcast to all local machines and the local subgradients are computed on all machines and sent to the central machine to form the global gradient $\nabla {\mathcal{L}}_N\left(\tilde{\mathit{\theta }}\right)$.

We can repeat the above procedure for multiple stages. In stage t, we set the starting value ${\mathit{\theta }}_0^{\left(t\right)}$ to be the estimator obtained in the previous stage, and then form the surrogate loss function

$${\tilde{\mathcal{L}}}^{\left(t\right)}\left(\mathit{\theta }\right)={\mathcal{L}}_1\left(\mathit{\theta }\right)-〈\mathit{\theta },\nabla {\mathcal{L}}_1\left({\mathit{\theta }}_0^{\left(t\right)}\right)-\nabla {\mathcal{L}}_N\left({\mathit{\theta }}_0^{\left(t\right)}\right)〉,$$

and then obtain ${\mathit{\theta }}_0^{(t+1)}$ by minimizing the surrogate loss function above (using IHT)

$${\mathit{\theta }}_0^{(t+1)}\in \underset{{\mathit{\theta }:\parallel \mathit{\theta }\parallel }_0\le s}{arg\; min}{\tilde{\mathcal{L}}}^{\left(t\right)}\left(\mathit{\theta }\right),$$

which serves as the initial value for the next stage (see Algorithm 1 for pseudocode).

Remark 1.

For convenience of presentation only, we assume data are evenly distributed on all machines. However, when the total sample size N is fixed, the only important quantity is the sample size n on machine 1, which determines the accuracy of the initial estimator, as well as the surrogate function, while sample sizes for other machines play no role since other machines only contribute the gradients and all gradients are aggregated by machine 1.

To see this point more clearly, in (2), the only term that requires the participation of other machines is the term ${\mathcal{L}}_N\left(\tilde{\mathit{\theta }}\right)$. Whatever the local sample sizes for machines $2,\dots ,m$, denoted by $n_2,\dots ,n_m$, each machine simply sends $\nabla {\mathcal{L}}_j\left(\tilde{\mathit{\theta }}\right)={\displaystyle \frac{1}{n_j}}{\sum }_{i=1}^{n_j}{\mathbf{x}}_{ij}(I\{y_{ij}-{\mathbf{x}}_{ij}^{\mathrm{T}}\tilde{\mathit{\theta }}\le 0\}-\tau )$, and machine 1 can aggregate them by an weighted average to obtain exactly $\nabla {\mathcal{L}}_N\left(\tilde{\mathit{\theta }}\right)={\displaystyle \frac{n_j}{N}}{\sum }_j\nabla {\mathcal{L}}_j\left(\tilde{\mathit{\theta }}\right)$. Alternatively, the machines $2,\dots ,m$ can simply send the sum of gradients on $n_j$ observations (instead of the mean of gradients on $n_j$ observations) and then machine 1 can sum up these quantities and divide it by N, to obtain exactly $\nabla {\mathcal{L}}_N\left(\tilde{\mathit{\theta }}\right)$.

Algorithm 1 distributed estimation for quantile regression using IHT

1: Input: sparsity level s, number of stages T, number of iterations of IHT Q, step size $\eta$  2: Initialize ${\mathit{\theta }}_0^{\left(0\right)}$  3: for $t=0,\dots ,T-1$ do  4:    for $j=1,\dots ,m$ do  5:      (in parallel)  6:      Receive ${\mathit{\theta }}_0^{\left(t\right)}$ from machine 1  7:      Compute the local subgradient $\nabla {\mathcal{L}}_j\left({\mathit{\theta }}_0^{\left(t\right)}\right)$ on machine j  8:      Transmit the local subgradient $\nabla {\mathcal{L}}_j\left({\mathit{\theta }}_0^{\left(t\right)}\right)$ to machine 1 (central machine)  9:    end for 10:    Calculate $\nabla {\mathcal{L}}_N\left({\mathit{\theta }}_0^{\left(t\right)}\right)={\displaystyle \frac{1}{m}}{\sum }_{j=1}^m\nabla {\mathcal{L}}_j\left({\mathit{\theta }}_0^{\left(t\right)}\right)$ on machine 1 11:    for $q=0,\dots ,Q-1$ do 12:      ${\mathit{\theta }}_{q+1}^{\left(t\right)}=P_s\left({\mathit{\theta }}_q^{\left(t\right)}-\eta \nabla {\tilde{\mathcal{L}}}^{\left(t\right)}\left({\mathit{\theta }}_q^{\left(t\right)}\right)\right)$ where $\nabla {\tilde{\mathcal{L}}}^{\left(t\right)}\left(\mathit{\theta }\right)=\nabla {\mathcal{L}}_1\left(\mathit{\theta }\right)-\left(\nabla {\mathcal{L}}_1\left({\mathit{\theta }}_0^{\left(t\right)}\right)-\nabla {\mathcal{L}}_N\left({\mathit{\theta }}_0^{\left(t\right)}\right)\right)$ 13:    end for 14:    Set ${\mathit{\theta }}_0^{(t+1)}={\mathit{\theta }}_Q^{\left(t\right)}$; 15: end for 16: Output ${\mathit{\theta }}_0^{\left(T\right)}$

3. Main Results

In this section, we provide the theoretical result showing the convergence rate of the distributed estimator. We will see that the distributed estimator can achieve the same convergence rate as the global estimator based on the whole data set under suitable assumptions. We note that the check loss function for quantile regression is neither strongly convex nor second-order differentiable, and ${\mathsf{\ell }}_0$ constraint is nonconvex, which renders the theory in [17] not directly applicable to our current problem.

Let $f(.|\mathbf{x})$ be the conditional density of ${ϵ}_i$ (as defined in (1)). The following assumptions are imposed.

(A1)

Define $\mathbf{\Theta }=\{\mathit{\theta }:\parallel \mathit{\theta }\parallel \le \parallel {\mathit{\theta }}_0\parallel \vee 1\}$ and $H\left(\mathit{\theta }\right)=E\left[f\left({\mathbf{x}}^{\mathrm{T}}\mathit{\theta }\right|\mathbf{x})\mathbf{x}{\mathbf{x}}^{\mathrm{T}}\right]$ (it can be shown that $H\left(\mathit{\theta }\right)$ actually contains the second-order partially derivatives of $E\left[{\rho }_{\tau }(y-{\mathbf{x}}^{\mathrm{T}}\mathit{\theta })\right]$ and is thus referred to as the population Hessian matrix). We assume that ${\sup }_{\mathit{\theta }\in \mathbf{\Theta }}{\mathbf{a}}^{\mathrm{T}}H\left(\mathit{\theta }\right)\mathbf{a}$ is bounded from above by a constant $C_2>0$ and ${\inf }_{\mathit{\theta }\in \mathbf{\Theta }}{\mathbf{a}}^{\mathrm{T}}H\left(\mathit{\theta }\right)\mathbf{a}$ is bounded from below by a constant $C_1>0$, for all $(2s+s_0)$-sparse unit vectors $\mathbf{a}$ (that is, $\parallel \mathbf{a}\parallel =1$).

(A2)

Components of $\mathbf{x}={(x_1,\dots ,x_d)}^{\mathrm{T}}$ are sub-Gaussian random variables in the sense that $E\left(e^{a{x}_j}\right)\le e^{C_3{a}^2}$ for any $a\in \mathbb{R}$ and some positive constants $C_3$.

Assumption (A1) imposes a constraint on the population Hessian matrix. To see that $E\left[f\left({\mathbf{x}}^{\mathrm{T}}\mathit{\theta }\right|\mathbf{x})\mathbf{x}{\mathbf{x}}^{\mathrm{T}}\right]$ is the Hessian matrix (in other words, it contains the second partial derivatives of $E\left[{\rho }_{\tau }(y-{\mathbf{x}}^{\mathrm{T}}\mathit{\theta })\right]$), we note that it is easy to see the first-order partial derivatives (gradient) are given by $E\left[\mathbf{x}(I\{y-{\mathbf{x}}^{\mathrm{T}}\mathit{\theta }\le 0\}-\tau )\right]=E\left[\mathbf{x}(F\left({\mathbf{x}}^{\mathrm{T}}\mathit{\theta }\right|\mathbf{x})-\tau )\right]$, where $F(.|\mathbf{x})$ denotes the conditional distribution function of y. Thus, taking the derivative with respect to $\mathbf{x}$ gives us $E\left[f\left({\mathbf{x}}^{\mathrm{T}}\mathit{\theta }\right|\mathbf{x})\mathbf{x}{\mathbf{x}}^{\mathrm{T}}\right]$. This is the quantile regression counterpart of the standard assumption in least squares regression that usually requires that eigenvalues of $E\left[\mathbf{x}{\mathbf{x}}^{\mathrm{T}}\right]$ are bounded from above and below by some positive constants. This assumption is also adapted to deal with the sparse regression case where we only need to deal with a sparse unit vector $\mathbf{a}$. Assumption (A2) requires the predictors to be sub-Gaussian, which is necessary for our technical analysis using empirical processes theory.

In the following, C will denote a generic positive constant whose value may vary in different places. In the statement of the theorem, $C_1$ and $C_2$ are constants bounding the eigenvalues of the population Hessian matrix as in (A1). ${\mathit{\theta }}_0^{\left(0\right)}$ and ${\mathit{\theta }}_0^{\left(T\right)}$ are the initial estimator and the final estimator output by our algorithm, respectively.

Theorem 1.

Under the assumptions (A1) and (A2), let $\eta ={\displaystyle \frac{2}{C_1+C_2}}$, $s\ge c{s}_0$ with $c\ge 1$ large enough such that $(1+\sqrt{s_0/s}){\displaystyle \frac{C_2-C_1}{C_2+C_1}}=:1-\delta <1$; when the initial estimator ${\mathit{\theta }}_0^{\left(0\right)}$ satisfies $\parallel {\mathit{\theta }}_0^{\left(0\right)}-{\mathit{\theta }}_0\parallel \le C\left(s^{{\displaystyle \frac{5}{2}}}{\displaystyle \frac{{\mathit{log}}^{\frac{3}{2}}(d\vee n)}{n}}+\sqrt{{\displaystyle \frac{s\mathit{log}(d\vee n)}{n}}}\right)$, after $T\ge C\mathit{log}\left({\displaystyle \frac{N\parallel {\mathit{\theta }}_0^{\left(0\right)}-{\mathit{\theta }}_0\parallel }{s\mathit{log}(d\vee n)}}\right)$ stages with $Q\ge C\mathit{log}\left({\displaystyle \frac{N\parallel {\mathit{\theta }}_0^{\left(0\right)}-{\mathit{\theta }}_0\parallel }{s\mathit{log}(d\vee n)}}\right)$ iterations for each stage using IHT, with a probability of at least $1-{(d\vee n)}^{-C}$, it holds that

$$\parallel {\mathit{\theta }}_0^{\left(T\right)}-{\mathit{\theta }}_0\parallel \le C\left({\displaystyle \frac{s^{\frac{5}{2}}{\mathit{log}}^{\frac{3}{2}}(d\vee n)}{n}}+\sqrt{{\displaystyle \frac{s\mathit{log}(d\vee n)}{N}}}\right),$$

where $d\vee n$ denotes $\max \{d,n\}$.

Remark 2.

We require the initial estimator to be sufficiently accurate, with $\parallel {\mathit{\theta }}_0^{\left(0\right)}-{\mathit{\theta }}_0\parallel \le C\left(s^{{\displaystyle \frac{5}{2}}}{\displaystyle \frac{{\mathit{log}}^{\frac{3}{2}}(d\vee n)}{n}}+\sqrt{{\displaystyle \frac{s\mathit{log}(d\vee n)}{n}}}\right)$. This can be guaranteed when ${\mathit{\theta }}_0^{\left(0\right)}$ is computed on the first machine using its local data of size n if n is not too small, as shown in [5]. Intuitively, the initial estimator should be sufficiently accurate for the surrogate loss to be a good approximation of the global loss. If the initial estimator is not accurate enough (for example in our simulation later, this is the case when the local sample size n is small), the method might have bad performance.

Remark 3.

The iterative bound for the estimation error is shown in (A11) in the proof, which indicates that the distributed estimator can be linearly convergent to the true parameter ${\mathit{\theta }}_0$. Compared to the iterative bound for the non-distributed global estimator provided by [5], which is $e_{t+1}=(1-\delta /2)e_t+O\left({\displaystyle \frac{s^{5/2}{\mathit{log}}^{3/2}(d\vee N)}{N}}+\sqrt{{\displaystyle \frac{s\mathit{log}(d\vee N)}{N}}}\right)$, we can conclude that the distributed estimator can achieve the same convergence rate as the estimator with all data stored on a single machine, when the term $\sqrt{{\displaystyle \frac{s\mathit{log}(d\vee n)}{N}}}$ is the dominating term. Here, $\sqrt{{\displaystyle \frac{s\mathit{log}(d\vee n)}{N}}}$ is the statistical precision since it is the error bound when using the entire sample with size N. In other words, if all data with sample size N are stored on a single machine, which performs IHT to obtain an estimator $\widehat{\mathit{\theta }}$, then we have $\parallel \widehat{\mathit{\theta }}-{\mathit{\theta }}_0\parallel \le \sqrt{{\displaystyle \frac{s\mathit{log}(d\vee n)}{N}}}$, according to the result in [5]. Thus, in this paper, we call this term the statistical precision, which is the benchmark error for the global estimator. In particular, this holds when ${\displaystyle \frac{s^{\frac{5}{2}}{\mathit{log}}^{\frac{3}{2}}(d\vee n)}{n}}\le \sqrt{{\displaystyle \frac{s\mathit{log}(d\vee n)}{N}}}$, or equivalently, $m\le {\displaystyle \frac{\sqrt{N}}{s^2\mathit{log}(d\vee n)}}$. In other words, when the number of machines is not too large, the distributed estimation has the same convergence rate as the global estimator.

4. Simulations

In this section, we numerically illustrate the convergence of the distributed estimator to verify the theoretical conclusions we obtain in Section 3, comparing it to the local estimator $\tilde{\mathit{\theta }}$ (only using the data stored on the first machine) and the global estimator ${\mathit{\theta }}^G$ (assuming all data are stored on a single machine). We then turn to demonstrating the robustness to different parameter setups and the accuracy of nonzero parameter identification.

4.1. Convergence Illustration

We first generate N i.i.d observations by $y_i={\mathbf{x}}_i^{\mathrm{T}}{\mathit{\theta }}_0+{ϵ}_i,i=1,\dots ,N$, where ${\mathbf{x}}_i$ is from the multivariate normal distribution with mean $\mathbf{0}$ and the covariance matrix ${0.2}^{|j-k|},j,$$k=1,\dots ,d$. The noise $ϵ$ is generated from the normal distribution with mean 0 and variance $0.16$. We set ${\mathit{\theta }}_0={(1,2,-1,0.5,-0.3,0.1,0.2,-0.1,0.05,-0.03,0,\dots ,0)}^{\mathrm{T}}$, with the dimension $d=100$, $s=15$. Strictly speaking, as a tuning parameter, we could use cross-validation to fix the proper value of s. Here, we solely fix s to discuss the convergence of the estimators.

We will report the estimation error (EE) $\parallel \mathit{\theta }-{\mathit{\theta }}_0{\parallel }^2$ of the estimators using 100 repetitions in each setting, and the prediction error (PE) based on additional independently generated 5000 observations.

For the first simulation, we fix the whole sample size $N=\mathrm{10,000}$ and vary the number of machines m, with the results shown in Figure 2. We can observe that the errors decrease with the number of stages used, and with a five-stage estimator, the errors are usually close to those of the global estimator. We see that the one-stage estimation is typically insufficient, with much larger errors. For $m=5$, the estimator quickly converges with a very small number of stages, while for a larger number of machines, more stages are probably required. We also note that the estimation errors increase with the number of machines m.

In our second simulation, we keep the local sample size $n=400$ fixed and increase the number of machines m. According to Figure 3, the estimation errors of all estimators decrease with m as expected, since the total sample size N is proportional to m. Again, we see one-stage estimators perform much worse than multiple-stage estimators.

In the third simulation, we fix $m=5$ and increase the local sample size n. Again, we see in Figure 4 that the errors decrease with n. When the local sample size n is sufficiently large, two or three stages suffice for the distributed estimators to gain good performances comparable to the global estimators. We also note that when n is small ($n=400$), there is still a significant gap between the distributed estimator and the global estimator after five stages.

We also ran experiments to verify the performance of the distributed estimator we developed in some more extreme cases. We explored the scenarios in which the number of machines m and the dimension d became large with the entire sample size $N=\mathrm{120,000}$. We use $m=10$ and a much larger $m=100$. With $m=100$, the local data size is $n=120$, which is much smaller than $d=1500$. As shown in

Table 1. The estimation and prediction errors $(\%)$ of the local estimator $\tilde{\mathit{\theta }}$, the two-stage distributed estimator ${\mathit{\theta }}_0^{\left(2\right)}$, the three-stage distributed estimator ${\mathit{\theta }}_0^{\left(3\right)}$, and the global estimator ${\mathit{\theta }}^G$ with $N=\mathrm{12,000}$, $s=15, d\in \{100,1500\}, m\in \{10,100\}$ at quantile level $\tau =0.25$.

N	d	m	$\tilde{\mathit{\theta }}$			${\mathit{\theta }}_0^{\left(2\right)}$			${\mathit{\theta }}_0^{\left(3\right)}$			${\mathit{\theta }}^{\mathit{G}}$ *
			PE $(\%)$	EE $(\%)$		PE $(\%)$	EE $(\%)$		PE $(\%)$	EE $(\%)$		PE $(\%)$	EE $(\%)$
12,000	100	10	$\begin{array}{c}\hfill 23.131 \left(0.484\right)\end{array}$	$\begin{array}{c}\hfill 0.811 \left(0.043\right)\end{array}$		$\begin{array}{c}\hfill 23.125 \left(0.486\right)\end{array}$	$\begin{array}{c}\hfill 0.328 \left(0.025\right)\end{array}$		$\begin{array}{c}\hfill 23.122 \left(0.486\right)\end{array}$	$\begin{array}{c}\hfill 0.238 \left(0.023\right)\end{array}$		$\begin{array}{c}\hfill 23.219 \left(0.313\right)\end{array}$	$\begin{array}{c}\hfill 0.091 \left(0.025\right)\end{array}$
		100	$\begin{array}{c}\hfill 26.264 \left(0.553\right)\end{array}$	$\begin{array}{c}\hfill 21.373 \left(6.840\right)\end{array}$		$\begin{array}{c}\hfill 24.229 \left(0.508\right)\end{array}$	$\begin{array}{c}\hfill 9.491 \left(5.547\right)\end{array}$		$\begin{array}{c}\hfill 23.278 \left(0.491\right)\end{array}$	$\begin{array}{c}\hfill 6.931 \left(5.265\right)\end{array}$
	1500	10	$\begin{array}{c}\hfill 23.701 \left(0.495\right)\end{array}$	$\begin{array}{c}\hfill 1.188 \left(0.714\right)\end{array}$		$\begin{array}{c}\hfill 23.519 \left(0.490\right)\end{array}$	$\begin{array}{c}\hfill 0.351 \left(0.024\right)\end{array}$		$\begin{array}{c}\hfill 23.420 \left(0.489\right)\end{array}$	$\begin{array}{c}\hfill 0.289 \left(0.047\right)\end{array}$		$\begin{array}{c}\hfill 23.332 \left(0.353\right)\end{array}$	$\begin{array}{c}\hfill 0.166 \left(0.052\right)\end{array}$
		100	$\begin{array}{c}\hfill 26.771 \left(0.560\right)\end{array}$	$\begin{array}{c}\hfill 26.874 \left(2.656\right)\end{array}$		$\begin{array}{c}\hfill 24.853 \left(0.497\right)\end{array}$	$\begin{array}{c}\hfill 16.396 \left(2.307\right)\end{array}$		$\begin{array}{c}\hfill 23.763 \left(0.495\right)\end{array}$	$\begin{array}{c}\hfill 13.187 \left(2.083\right)\end{array}$

* The results for the global estimator ${\mathit{\theta }}^G$ are the same when $m=10$ and $m=100$; thus, we do not present the latter.

, the errors grow with the dimension d, as expected. The distributed estimator largely reduces the estimation errors of the local estimator and can decrease the errors stage by stage in high-dimensional settings. The estimation errors exhibit a surge when the number of machines is 100, showing the method would fail if m is too big.

We now illustrate what happens if n is very small. We set $N=5000$, and $n=\{10,100,500\}$, with the other settings unchanged as in the first simulation. We can observe from Figure 5 that when the first machine has significantly less data, the initial estimator would not be accurate enough. Correspondingly, the distributed estimators perform much worse, and, even after five stages, the errors are still large; increasing the number of stages further does not help.

Finally, we illustrate the convergence of the proposed distributed estimator in the case where the noise distribution is chi-squared with 5 degrees of freedom. The other settings remain unchanged as in the first simulation. The boxplots displaying the estimation errors are shown in Figure 6. We see that qualitatively, the results are similar to those of Figure 2. When m is small, the two-stage estimator is almost as good as the global estimator. When m becomes larger, more stages are required and there is a perceivable performance gap between the distributed estimator and the global estimator.

4.2. Variable Identification Performance

In this simulation, to examine whether the magnitude of the true parameter and the sparsity affect the estimation and variable selection, we generate $N=5000$ i.i.d observations with ${\mathit{\theta }}_0={(1,0.98,0.96,\dots ,0.02,0,\dots ,0)}^{\mathrm{T}}$, where there are 50 nonzero parameters following a decreasing pattern and the dimension $d=100$. Using 100 repetitions, we record the number of times each of the 100 parameters is identified as nonzero. With this, we can obtain the corresponding identification frequency. We also report the overall misclassification rate (MCR), which is the proportion of the sum of false positives and false negatives in identifying nonzero parameters.

In our simulations, we set $s\in \{10,30\}, m\in \{10,20\}$. As shown in Figure 7 and

Table 2. The misclassification rate $(\%)$, estimation and prediction errors of the local estimator $\tilde{\mathit{\theta }}$, the three-stage distributed estimator ${\mathit{\theta }}_0^{\left(3\right)}$, and the global estimator ${\mathit{\theta }}^G$ with $N=5000, d=100$, $s\in \{10,30\}, m\in \{10,20\}$ at quantile level $\tau =0.25$.

N	s	m	$\tilde{\mathit{\theta }}$				${\mathit{\theta }}_0^{\left(3\right)}$				${\mathit{\theta }}^{\mathit{G}}$ *
			MCR (%)	PE	EE		MCR (%)	PE	EE		MCR (%)	PE	EE
5000	30	10 1	$\begin{array}{c}\hfill 22.320 \left(0.202\right)\end{array}$	$\begin{array}{c}\hfill 2.425 \left(0.053\right)\end{array}$	$\begin{array}{c}\hfill 2.523 \left(0.052\right)\end{array}$		$\begin{array}{c}\hfill 21.320 \left(0.171\right)\end{array}$	$\begin{array}{c}\hfill 2.249 \left(0.049\right)\end{array}$	$\begin{array}{c}\hfill 2.093 \left(0.047\right)\end{array}$		$\begin{array}{c}\hfill 20.000 \left(0.000\right)\end{array}$	$\begin{array}{c}\hfill 1.813 \left(0.026\right)\end{array}$	$\begin{array}{c}\hfill 1.191 \left(0.009\right)\end{array}$
		20 2	$\begin{array}{c}\hfill 26.420 \left(0.303\right)\end{array}$	$\begin{array}{c}\hfill 4.940 \left(0.114\right)\end{array}$	$\begin{array}{c}\hfill 4.102 \left(0.102\right)\end{array}$		$\begin{array}{c}\hfill 21.400 \left(0.194\right)\end{array}$	$\begin{array}{c}\hfill 2.619 \left(0.060\right)\end{array}$	$\begin{array}{c}\hfill 2.304 \left(0.069\right)\end{array}$
	10	10 3	$\begin{array}{c}\hfill 40.000 \left(0.020\right)\end{array}$	$\begin{array}{c}\hfill 13.313 \left(0.308\right)\end{array}$	$\begin{array}{c}\hfill 10.316 \left(0.057\right)\end{array}$		$\begin{array}{c}\hfill 40.000 \left(0.000\right)\end{array}$	$\begin{array}{c}\hfill 12.956 \left(0.300\right)\end{array}$	$\begin{array}{c}\hfill 10.100 \left(0.059\right)\end{array}$		$\begin{array}{c}\hfill 40.000 \left(0.000\right)\end{array}$	$\begin{array}{c}\hfill 12.196 \left(0.173\right)\end{array}$	$\begin{array}{c}\hfill 8.829 \left(0.023\right)\end{array}$
		20 4	$\begin{array}{c}\hfill 40.280 \left(0.081\right)\end{array}$	$\begin{array}{c}\hfill 13.324 \left(0.298\right)\end{array}$	$\begin{array}{c}\hfill 11.398 \left(0.103\right)\end{array}$		$\begin{array}{c}\hfill 40.160 \left(0.055\right)\end{array}$	$\begin{array}{c}\hfill 12.958 \left(0.294\right)\end{array}$	$\begin{array}{c}\hfill 10.871 \left(0.082\right)\end{array}$

* The results for the global estimator ${\mathit{\theta }}^G$ are the same when $m=10$ and $m=20$; thus, we do not present the latter. ¹ The computational time is $20.777$ min when $(s,m)=(30,10)$. ² The computational time is $25.502$ min when $(s,m)=(30,20)$. ³ The computational time is $22.565$ min when $(s,m)=(10,10)$. ⁴ The computational time is $21.565$ min when $(s,m)=(10,20)$.

, with the special setting of ${\mathit{\theta }}_0$, the identification rates decrease as the parameter index grows. At $s=30$, the three-stage estimator slightly outperforms the local estimator, with higher identification rate for the parameters with large magnitudes and a lower misclassification rate. At $s=10$, the local estimator and the three-stage estimator have almost identical performance. We can also observe that the misclassification rate of the three-stage distributed estimator increases slightly with the number of machines. In addition, we report the computational time of the four cases, each using 100 repetitions, as shown in the footnotes of Table 2. Our algorithm is implemented in R (R-4.4.2) on our desktop computer with 12th Gen Intel(R) Core(TM) i5-12400F 2.50 GHz and 16 GB RAM.

5. Conclusions

In this article, we theoretically establish the convergence rate of the distributed estimator for quantile regression with ${\mathsf{\ell }}_0$ constraint. We extend the conclusions of [17] to the model with a non-smooth loss function and a nonconvex constraint. The numerical experiments show that the distributed estimators have satisfactory performance. The distributed estimator can decrease the estimation and prediction errors stage by stage. It also largely reduces the errors of the local estimator and two or three stages often suffice for the distributed framework to gain almost the same accuracy as the global estimator. The errors increase with the number of machines m when N is fixed, decrease with m with n fixed, and also decrease as n grows with m fixed.

Our work has its limitations; for example, in our simulations, we notice that only with each machine having sufficient observations can we have satisfactory performance. This is probably due to the initial estimator being important for performance, and when the local sample size is too small, the initial estimator is not good enough. This makes the proposed method sometimes unreliable in practice.

The proposed method contributes theoretically and businesswise. Existing distributed frameworks with high-dimensional statistical settings have convergence guarantees held only for smooth convex models. We bridge the gap and extend it to the non-smooth and nonconvex situation. In addition, it is effective for addressing large-scale statistical optimization problems. It divides a substantial task into smaller components that can be processed in parallel across multiple machines. This approach can be utilized by commercial companies to analyze vast amounts of browsing data and identify the most influential factors for personalized recommendations. Additionally, biological researchers may resort to this method to pinpoint the most significant genes to diagnose certain diseases.

Some extensions can be considered. For example, one can consider composite quantile regression (CQR) proposed by [18], which is regarded as a robust and efficient alternative to mean regression. One can also consider the quantile matrix regression problem, where the predictors and parameters are in the form of matrices. Also, it is well-known that standard quantile regression based on the check loss does not work for extreme quantile levels such as $\tau =0.99$, for which more complicated methods such as [19] should be applied. Furthermore, we did not consider the variance estimation problem, which would be useful if we want to perform statistical inference. Furthermore, we chose one particular approach of distributed estimation for its simplicity and feasibility to demonstrate its extension to the particular non-smooth nonconvex problem that interests us; other approaches such as using DGD (distributed gradient descent) or ADMM (alternating direction method of multipliers) could also be investigated [20,21]. Finally, federated learning emphasizes the case of heterogeneous data distribution and asynchronous update [22] and is more challenging to deal with. These interesting problems are left for future work to solve.