Error-Function-Based Penalized Quantile Regression in the Linear Mixed Model

Abstract

We study a novel Doubly penalized ERror Function regularized Quantile Regression (DERF-QR) in this paper. This is a method of variable selection by ERror Function (ERF) regularization in the linear effects model. We introduce a two-stage iterative algorithm combining the iterative reweighted $L_1$ approach and the alternating direction method of multipliers (ADMM) to estimate the model parameters. Numerical simulations show that by using this method we remove redundant variables effectively and obtain accurate coefficient estimations. Our method outperforms two existing penalized quantile regression methods in various error conditions by comparison. Finally, we apply the methodology in a financial dataset and showcase its practicality.

1. Introduction

Quantile regression has become a powerful tool in statistical theory, with extensive methodological developments. It has also found wide application in engineering and related fields [1]. In recent years, statistical inference and variable selection in high-dimensional quantile regression have become an important research focus [2,3]. Tan et al. proposed a convolutional smooth quantile regression method with an iteratively reweighted $L_1$ algorithm to enhance robustness [4]. Wang et al. combined frequentist model averaging with high-dimensional quantile regression to improve model construction and selection [5]. Research on high-dimensional quantile regression has also been extended to a wider range of models [6,7].

Compared to mean regression, quantile regression offers greater robustness to outliers and skewed distributions [8], and is particularly useful for capturing heterogeneous effects in panel data models [9]. Traditional panel data models often neglect individual heterogeneity, whereas linear mixed-effects models effectively address this by capturing both data heterogeneity and potential correlations among individuals. They incorporate random effects to effectively model unobservable individual-level variations, such as behavioral preferences and baseline differences. The combination of linear mixed-effects models and quantile regression has attracted growing attention, and the variable selection for these models emerges as a critical research area.

Variable selection and dimensionality reduction have attracted extensive theoretical and practical attention in the past decade. Traditional methods often fail to perform effectively in high-dimensional settings. Several regularization techniques have been developed to overcome this difficulty in existing work. Tibshirani introduced the Lasso method in [10] and simultaneously obtained the coefficient estimation and variable selection by $L_1$ regularization. Fan et al. proposed the non-convex SCAD penalty with Oracle properties in [11]. Based on these foundations, many regularization methods and extensions have been developed to handle more complex models.

In quantile regression for the linear mixed-effects model, neglecting random effects can lead to biased estimates by failing to capture individual heterogeneity. Including too many random-effect covariates may cause overfitting, resulting in unstable estimates and reduced interpretability and generalizability. Thus, effective variable selection is crucial for simplifying models, eliminating redundancy, and improving computational efficiency. However, it is more challenging to estimate the coefficients and select the variables due to the interaction between fixed and random effects in the linear mixed-effects model. Koenker proposed a quantile regression approach in random-effects models and applied the Lasso penalty in random effects to eliminate redundant effects, but this method does not allow for selection among fixed-effect variables [12]. Li et al. proposed using $L_1$ penalties to simultaneously select fixed and random effects in quantile regression for linear mixed-effects models, but the estimates obtained with the Lasso penalty are biased [13]. Bondell et al. used improved Cholesky decomposition for fixed- and random-effects estimation and variable selection in linear mixed-effects models [14]. Qi et al. proposed a double-penalized likelihood method for finite mixture regression models, imposing penalties on both the mixing proportions and regression coefficients. This approach enables simultaneous parameter estimation and variable selection [15]. Inspired by the above proposition, we propose a Doubly penalized ERror Function regularized Quantile Regression (DERF-QR) method in a linear mixed-effects model in this paper.

The innovations of this paper are as follows. First, we improve the existing doubly penalized quantile regression method by incorporating a novel regularization term based on the error function in a linear mixed-effects model [16]. This penalty function approximates the $L_0$ penalty under certain conditions and substantially enhances the performance of coefficient estimation and variable selection. Second, we propose an efficient algorithm for this method by combining a two-step iterative method with the iterative reweighted $L_1$ proximal to the alternating direction method of multipliers algorithm (IRW-pADMM) [17].

The structure of this paper is as follows. In Section 2, we introduce the properties of penalty functions and present an estimation method for the doubly penalized method. We give the Monte Carlo simulations to validate the performance of this method in coefficient estimation and variable selection in Section 3. Section 4 shows that the method proposed in this paper is effective in financial data from multiple companies.

2. Model and Algorithm

Consider a linear mixed-effects model with random-effect coefficients as follows:

$$\begin{array}{cc}\hfill Y_{ij}& =x_{ij}^{\mathrm{T}}\beta +z_{ij}^{\mathrm{T}}{\alpha }_i+{\epsilon }_{ij},i=1,2,\dots ,n,j=1,2,\dots ,m,\hfill \end{array}$$

(1)

where $Y_{ij}$ and ${\epsilon }_{ij}$ denote the response variable and the error at the j-th observation of the i-th individuals, respectively. The vector $x_{ij}={(x_{ij1},x_{ij2},\dots ,x_{ijp})}^{\mathrm{T}}\in {\mathbb{R}}^p$ is the fixed-effect variables, and $\beta ={({\beta }_1,\dots ,{\beta }_p)}^{\mathrm{T}}$ denotes the corresponding p-dimensional coefficients. Similarly, $z_{ij}={(z_{ij1},z_{ij2},\dots ,z_{ijq})}^{\mathrm{T}}\in {\mathbb{R}}^q$ and ${\alpha }_i={({\alpha }_{i1},\dots ,{\alpha }_{iq})}^{\mathrm{T}}$ are the random-effects covariates and their corresponding q-dimensional coefficients, respectively.

In this linear mixed-effects model, an excessive number of random-effect covariates may lead to overfitting and hinder accurate parameter estimation. To address this, we study a doubly penalized quantile regression method for the linear mixed-effects model, with regularization terms based on the error function applied to both fixed- and random-effects coefficients. For a given response variable $Y_{ij}$ and quantile $\tau \in \left(0,1\right)$, the conditional quantile regression function is expressed as follows:

$$\begin{array}{cc}\hfill Q\left(\tau \mid x_{ij},z_{ij},{\alpha }_i\right)& =x_{ij}^{\mathrm{T}}\beta +z_{ij}^{\mathrm{T}}{\alpha }_i,\hfill \end{array}$$

(2)

where $Q\left(\tau |x_{ij},z_{ij},{\alpha }_i\right)=\inf \left\{y:F\left(y|x_{ij},z_{ij},{\alpha }_i\right)\ge \tau \right\}$ is the $\tau$-quantile of $Y_{ij}$. The model can be transformed into the following minimization problem:

$$\begin{array}{cc}\hfill & \underset{\beta ,\alpha }{\min }\left\{\sum _{i=1}^n\sum _{j=1}^m{\rho }_{\tau }\left(Y_{ij}-x_{ij}^{\mathrm{T}}\beta -z_{ij}^{\mathrm{T}}{\alpha }_i\right)+{\lambda }_{\beta }{J}_{\sigma }\left(\beta \right)+\sum _{i=1}^n{\lambda }_{\alpha }{J}_{\sigma }\left({\alpha }_i\right)\right\}\hfill \\ \hfill & J_{\sigma }\left(\beta \right)=\sum _{l=1}^p\mathrm{\Phi }\left(|{\beta }_l|\right),\mathrm{\Phi }\left(\right|{\beta }_l\left|\right)={\int }_0^{|{\beta }_l|}{e}^{-{\displaystyle \frac{t^2}{{\sigma }^2}}}dt,\hfill \end{array}$$

(3)

where ${\rho }_{\tau }\left(u\right)=u(\tau -I(u\le 0))$ is the quantile loss function, and ${\lambda }_{\beta }\ge 0$, ${\lambda }_{\alpha }\ge 0$ are the penalty parameters associated with the fixed-effects and random-effects coefficients, respectively. $\mathrm{\Phi }\left(x\right)$ is the error function and define $J_{\sigma }\left(\beta \right)$ is the ERF(ERorr Function) regularization term [16]. The parameter $\sigma >0$ controls the intensity of the penalty. For clarity of discussion, we reformulate Equations (3) as follows:

$$\begin{array}{cc}\hfill \underset{\beta ,\alpha }{\min }\{& \sum _{i=1}^n\sum _{j=1}^m{\rho }_{\tau }\left(Y_{ij}-x_{ij}^{\mathrm{T}}\beta -z_{ij}^{\mathrm{T}}{\alpha }_i\right)\hfill \\ \hfill & +{\lambda }_{\beta }\sum _{l=1}^p\mathrm{\Phi }\left(\right|{\beta }_l\left|\right)+{\lambda }_{\alpha }\sum _{i=1}^n\sum _{k=1}^q\mathrm{\Phi }\left(\right|{\alpha }_{ik}\left|\right)\}.\hfill \end{array}$$

(4)

By imposing penalties on both the fixed- and random-effects coefficients in the linear mixed model, one can not only perform variable selection for the fixed effects but also eliminate redundant information in the random effects. Li et al. proved that, in doubly penalized quantile regression for linear mixed-effects models, the estimator under double Lasso penalties converges in distribution asymptotically, and the bias in estimating non-zero coefficients is controlled by the penalty parameters [13]. We implement doubly penalized regression for linear mixed-effects models by applying ERF regularization to both the fixed- and random-effects coefficients.

The $L_1$ is the most commonly used penalty function in quantile regression. However, it introduces estimation bias, particularly for large coefficients. To address this issue, several non-convex penalties such as SCAD and MCP have been proposed [11,18]. These penalties adaptively apply varying degrees of shrinkage to different coefficient magnitudes, thereby reducing the bias introduced by $L_1$. Nevertheless, their non-convex and non-smooth nature poses significant challenges in solving the associated optimization problems. ERF regularization is a non-convex penalty function with smoothness, which increases flexibility in the optimization process and improves computational efficiency. Existing work has shown that this function reduces bias more effectively than most other non-convex penalties [16]. With appropriate parameters $\sigma$, the ERF regularization term can closely approximate the $L_0$ penalty, which directly controls the sparsity of the model and yields an optimal sparse solution. For any $x\in {\mathbb{R}}^n$, we have

$$\begin{array}{cc}\hfill J_{\sigma }\left(x\right)& \to {\parallel x\parallel }_1,\mathrm{if}\sigma \to +\infty ,\hfill \\ \hfill {\displaystyle \frac{J_{\sigma }\left(x\right)}{\sigma }}& \to {\displaystyle \frac{\sqrt{\pi }}{2}}{\parallel x\parallel }_0,\mathrm{if}\sigma \to 0^{+}.\hfill \end{array}$$

Moreover, Guo provided the following four properties and their proofs [16].

(1) The error function $\mathrm{\Phi }\left(x\right)$ is smooth, and the following is its derivative:

$$\begin{array}{c}\hfill {\displaystyle \frac{d}{dx}}{\mathrm{\Phi }}_{\sigma }\left(x\right)=\exp \left(-{\left({\displaystyle \frac{x}{\sigma }}\right)}^2\right).\end{array}$$

(5)

(2) For any $x\in \mathbb{R}$, we have

$$\begin{array}{c}\hfill c\sqrt{1-e^{-a{x}^2}}\le {\mathrm{\Phi }}_{\sigma }\left(x\right)\le c\sqrt{1-e^{-b{x}^2}},\end{array}$$

(6)

where $a={\displaystyle \frac{1}{{\sigma }^2}},b={\displaystyle \frac{\pi }{{\left(4\sigma \right)}^2}}$ and $c={\displaystyle \frac{\sigma \sqrt{\pi }}{2}}.$

(3) $J_{\sigma }\in {\mathbb{R}}_{+}^n$ is concave. For any $t\in \left[0,1\right]$ and $x,y\in {\mathbb{R}}_{+}^n$, we have

$$\begin{array}{c}\hfill J_{\sigma }\left(tx+(1-t)y\right)\ge t{J}_{\sigma }\left(x\right)+(1-t)J_{\sigma }\left(y\right).\end{array}$$

(7)

(4) For any $x,y\in {\mathbb{R}}^n,$

$$\begin{array}{c}\hfill J_{\sigma }(x+y)\le J_{\sigma }\left(x\right)+J_{\sigma }\left(y\right).\end{array}$$

(8)

These properties show that the error function is smooth and bounded. Equation (8) further confirms that the ERF regularization satisfies the triangle inequality.

We employ the proximal operator to measure the bias of the penalty function [19]. For a function $f:{\mathbb{R}}^n\to {\mathbb{R}}^n\cup \{+\infty \}$, the proximal operator $\mathrm{prox}\left(v\right):{\mathbb{R}}^n⟶{\mathbb{R}}^n$ of the function f is defined as follows:

$$\mathrm{prox}\left(v\right)={\arg \min }_x\left(\mu f\left(x\right)+{\displaystyle \frac{1}{2}}\left|\right|x-{v\left|\right|}_2^2\right).$$

In Figure 1, we plot the proximal operators for $L_1$, $L_2$, SCAD, and ERF with different $\sigma$ at $\mu =1$. The proximity of each proximal operator to the line $y=x$ reflects the bias level, that is, a smaller deviation indicates lower bias. The bias of the ERF penalty increases as $\sigma$ increases, while for all values $\sigma$, it remains smaller than that of $L_1$.

Considering the non-smoothness of the quantile loss function and the non-convexity of the error function, we can transform Equation (4) into the following form by the iterative reweighted $L_1$ algorithm [17]:

$$\begin{array}{cc}\hfill \underset{\beta ,\alpha }{\min }\{& \sum _{i=1}^n\sum _{j=1}^m{\rho }_{\tau }\left(Y_{ij}-x_{ij}^{\mathrm{T}}\beta -z_{ij}^{\mathrm{T}}{\alpha }_i\right)\hfill \\ \hfill & +{\lambda }_{\beta }\sum _{l=1}^p{\omega }_{{\beta }_l}|{\beta }_l|+{\lambda }_{\alpha }\sum _{i=1}^n\sum _{k=1}^q{\omega }_{\alpha ik}\left|{\alpha }_{ik}\right|\},\hfill \end{array}$$

(9)

where the corresponding weights ${\omega }_{{\beta }_l}$ are computed by differentiating ${\mathrm{\Phi }}_{\sigma }\left(|{\beta }_l|\right)$ for each ${\beta }_l$. Specially, we have

$$\begin{array}{ccc}& & {\omega }_{{\beta }_l}={\mathrm{\Phi }}_{\sigma }^{\prime }\left(|{\beta }_l|\right)=\exp \left(-{\left({\displaystyle \frac{{\beta }_l}{\sigma }}\right)}^2\right),\hfill \\ & & {\omega }_{\alpha ik}=\exp \left(-{\left({\displaystyle \frac{{\alpha }_{ik}}{\sigma }}\right)}^2\right).\hfill \end{array}$$

We iteratively compute the weights and solve the optimization problem of Equation (9) until convergence is achieved. Gu et al. proposed the proximal ADMM and sparse coordinate descent ADMM algorithms, and proved their global convergence [20]. These two methods are particularly effective for solving high-dimensional sparse quantile regression problems, and show high estimation accuracy and computational efficiency. In this paper, we utilize the IRW-pADMM algorithm, which combines the proximal ADMM algorithm with the iterative reweighted $L_1$ algorithm. Consider the univariate quantile regression method with ERF regularization:

$$\begin{array}{c}\hfill \min \left\{\sum _{i=1}^n{\rho }_{\tau }\left(Y_i-{x_i}^{\mathrm{T}}\beta \right)+\lambda \sum _{j=1}^p\mathrm{\Phi }\left(|{\beta }_j|\right)\right\},\end{array}$$

(10)

where $\beta ={({\beta }_1,\dots ,{\beta }_p)}^{\mathrm{T}}$, $x_i^{\mathrm{T}}\in {\mathbb{R}}^p$ is the i-th row of the design matrix $X={(x_1,x_2,\dots ,x_n)}^{\mathrm{T}}$, and $Y={(Y_1,\dots ,Y_n)}^{\mathrm{T}}\in {\mathbb{R}}^n$ is the response vector. The IRW-pADMM algorithm is summarized in Algorithm 1. Here, $\mathrm{shrink}\left(x,\alpha \right)=\mathrm{sgn}\left(x\right)\max \left(\left|x\right|-\alpha ,0\right)$ is the soft thresholding operator and we define

$${\mathrm{Prox}}_{x_{{\rho }_{\tau }}}(x,\alpha )=\left\{\begin{array}{cc}x-{\displaystyle \frac{\tau }{a}},& x>{\displaystyle \frac{\tau }{a}},\\ 0,& {\displaystyle \frac{\tau -1}{a}}<x<{\displaystyle \frac{\tau }{a}},\\ x-{\displaystyle \frac{\tau -1}{a}},& x<{\displaystyle \frac{\tau -1}{a}},\end{array}\right.$$

where the parameter $\delta >0$ is the penalty parameter in the augmented Lagrangian of the ADMM algorithm, which affects the convergence speed of the model. $\gamma >0$ is the step size parameter that regulates the update of $\theta$. Based on the research conducted by Gu et al., $\eta \ge {\lambda }_{\max }\left(X^{\mathrm{T}}X\right)$ is used in the update of $\beta$ to ensure that the matrix in the proximal term

$$S=\sigma \left(\eta I_p-X^{\mathrm{T}}X\right)$$

is positive semi-definite, and ${\lambda }_{\max }\left(X^{\mathrm{T}}X\right)$ denotes the largest eigenvalue of the matrix $X^{\mathrm{T}}X$ [20].

Algorithm 1 IRW-pADMM algorithm.
1:	Input: $X\in {\mathbb{R}}^{n\times p}$, $Y\in {\mathbb{R}}^n$, $\sigma$, $\lambda$, $\gamma$, $\tau$, $\delta$, maxouter, maxinner
2:	Initialize: ${\widehat{\beta }}^{\left(0\right)}$, ${\widehat{z}}^{\left(0\right)}$, ${\widehat{\theta }}^{\left(0\right)}$, $k\leftarrow 0$
3:	while  $k<\mathrm{maxouter}$ and $\parallel {\widehat{\beta }}^{(k+1)}-{\widehat{\beta }}^{\left(k\right)}{\parallel }_2>{\epsilon }_1$  do
4:	${\omega }^{\left(k\right)}\leftarrow \exp \left(-{\left({\displaystyle \frac{{\widehat{\beta }}^{\left(k\right)}}{\sigma }}\right)}^2\right)$
5:	$l\leftarrow 0$, ${\beta }^{\left(l\right)}\leftarrow {\widehat{\beta }}^{\left(k\right)}$
6:	while $l<\mathrm{maxinner}$ and $\parallel {\beta }^{(l+1)}-{\beta }^{\left(l\right)}{\parallel }_2>{\epsilon }_2$ do
7:	${\beta }^{(l+1)}\leftarrow {\left(\mathrm{shrink}\left({\beta }_j^{\left(l\right)}+{\displaystyle \frac{1}{\delta \eta }}{X}_j^{\mathrm{T}}\left({\theta }^{\left(l\right)}+\delta Y-\delta X{\beta }^{\left(l\right)}-\delta z^{\left(l\right)}\right),{\displaystyle \frac{\lambda {\omega }_j^{\left(l\right)}}{\delta \eta }}\right)\right)}_{1\le j\le p}$
8:	$z^{(l+1)}\leftarrow {\left({\mathrm{Prox}}_{{\rho }_{\tau }}\left(Y_i-x_i^{\mathrm{T}}{\beta }^{(l+1)}+{\displaystyle \frac{1}{\delta }}{\theta }_i^{\left(l\right)},\delta \right)\right)}_{1\le i\le n}$
9:	${\theta }^{(l+1)}\leftarrow {\theta }^{\left(l\right)}-\gamma \delta \left(X{\beta }^{(l+1)}+z^{(l+1)}-Y\right)$
10:	$l\leftarrow l+1$
11:	end while
12:	${\widehat{\beta }}^{(k+1)}\leftarrow {\beta }^{\left(l\right)}$
13:	$k\leftarrow k+1$
14:	end while
15:	return  ${\widehat{\beta }}^{\left(k\right)}$

We present the two-step DERF-QR Algorithm in Algorithm 2 for the doubly penalized quantile regression method Equation (4). In Algorithm 2, we first initialize the fixed-effect coefficients as ${\widehat{\beta }}_i^{\left(0\right)}=0$, and then compute the initial estimates of the random-effect coefficients ${\widehat{\alpha }}^{\left(0\right)}$ using the IRW-pADMM algorithm. Subsequently, we alternately update the fixed- and random-effect coefficients by fixing one set and optimizing the other using the IRW-pADMM algorithm. Specifically, at each iteration k, the fixed-effect coefficients are updated by solving ${\widehat{\beta }}^{(k+1)}={\arg \min }_{\beta }L(\beta ,{\widehat{\alpha }}^{\left(k\right)})$ where the response variable is adjusted as ${Y_{ij}}^{\left(k\right)}=Y_{ij}-{z_{ij}}^{\mathrm{T}}{{\widehat{\alpha }}_i}^{\left(k\right)}$. The updated ${\widehat{\beta }}^{(k+1)}$ is then substituted into the model to update the random-effects coefficients by solving ${\widehat{\alpha }}^{(k+1)}={\arg \min }_{\alpha }L({\widehat{\beta }}^{(k+1)},\alpha )$. The residuals are recalculated as ${r_{ij}}^{\left(k+1\right)}=Y_{ij}-{x_{ij}}^{\mathrm{T}}{\widehat{\beta }}^{\left(k+1\right)}$. This alternating optimization procedure is repeated until convergence, which is determined when $\parallel {\widehat{\beta }}^{(k+1)}-{\widehat{\beta }}^{\left(k\right)}{\parallel }_2\le \epsilon$.

Algorithm 2 Two-step DERF-QR algorithm.

1: Input: $X\in {\mathbb{R}}^{nm\times p},Y\in {\mathbb{R}}^n,Z\in {\mathbb{R}}^{nm\times q},\tau ,\epsilon >0$, maxiter 2: Initialize:  ${\widehat{\beta }}_i^{\left(0\right)}=0,k\leftarrow 0$ 3: ${\widehat{\alpha }}^{\left(0\right)}={\arg \min }_{\alpha }L\left({\widehat{\beta }}^{\left(0\right)},\alpha \right)$ 4: while  $k<\mathrm{maxiter}$  and  $\parallel {\widehat{\beta }}^{(k+1)}-{\widehat{\beta }}^{\left(k\right)}{\parallel }_2>\epsilon$  do 5:     ${\widehat{\beta }}^{(k+1)}={\arg \min }_{\beta }L\left(\beta ,{\widehat{\alpha }}^{\left(k\right)}\right)$ 6:     ${\widehat{\alpha }}^{(k+1)}={\arg \min }_{\alpha }L\left({\widehat{\beta }}^{(k+1)},\alpha \right)$ 7:     $k\leftarrow k+1$ 8: end while 9: return  ${\widehat{\beta }}^{\left(k\right)}$

To proceed further, it is critical to select appropriate penalty parameters. The Schwarz information criterion (SIC) and the generalized approximate cross-validation (GACV) are common selection criteria for quantile regression [21,22,23]. In this paper, we utilize the extended Bayesian information criterion (EBIC) to select penalty parameters [24]:

$$\begin{array}{c}\hfill \mathrm{EBIC}=\ln \left(S_M\right)+\left|M\right|+{\gamma }_p\ln \left(p\right),\end{array}$$

(11)

where $S_M={\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j=1}^m}{\rho }_{\tau }\left(Y_{ij}-{x_{ij}}^{\mathrm{T}}\beta -{z_{ij}}^{\mathrm{T}}{\alpha }_i\right)$, and $\left|M\right|$ are the nonzero parameters in the model. p denotes the dimensionality of the variables. The criteria are extended from BIC to EBIC by adding a penalty term to the dimensionality of the variables [24]. The regularization term ${\gamma }_p\ln \left(p\right)$ with parameter ${\gamma }_p>0$, which is imposed on the model by restricting the dimensionality of the variables, improves EBIC in the high-dimensional cases. Specifically, each iteration in the two-step DERF-QR algorithm can be viewed as a univariate penalized quantile regression problem, in which only one parameter is updated at a time. In this paper, we take $\epsilon ={10}^{-4}$ and ${\epsilon }_1,{\epsilon }_2={10}^{-4}$ in Algorithms 1 and 2.

3. Monte Carlo Simulation

The simulated data are generated from the following model:

$$\begin{array}{cc}\hfill Y_{ij}& ={\beta }_0+x_{ij}^{\mathrm{T}}\beta +z_{ij}^{\mathrm{T}}{\alpha }_i+\upsilon {\epsilon }_{ij},i=1,2,\dots ,n,j=1,2,\dots ,m\hfill \end{array}$$

(12)

Here, let $x_{ij}={\left(x_{ij1}, \dots ,x_{ij8}\right)}^{\mathrm{T}}$, ${\beta }_0=0$, $\beta ={\left(3,1,6,0,0,0,0,0\right)}^{\mathrm{T}}$, $n=30$, and $m=10$. $X_k={(x_{11k},\dots ,x_{nmk})}^{\mathrm{T}}\sim N_{nm}(0,I)$ is the covariate vector, and ${\rho }^{|l-k|}$ is the correlation between $X_l$ and $X_k$. For random effects, define $z_{ij}={\left(1,x_{ij1},x_{ij2},x_{ij3}\right)}^{\mathrm{T}}$ for the first three variables, and let ${\alpha }_i={\left({\alpha }_{i1},{\alpha }_{i2},{\alpha }_{i3},{\alpha }_{i4}\right)}^{\mathrm{T}}\sim N\left(0,D\right)$, where $D=diag\left(1,1,0,0\right)$. $\upsilon$ denotes the signal-to-noise ratio (SNR). In the following, we introduce PE, TP, and TN to evaluate the variable selection and parameter estimation in the model.

(1) Prediction Error (PE):

$$\mathrm{PE}={\left(\widehat{\beta }-\beta \right)}^{\mathrm{T}}\Sigma \left(\widehat{\beta }-\beta \right),$$

(13)

where $\widehat{\beta }$ is the estimate of $\beta$, $\sum ={\left({\rho }^{|l-k|}\right)}_{8\times 8}.$

(2) True positives (TP): The number of variables selected correctly.

(3) True negatives (TN): The number of variables excluded correctly.

We show the performance of the DERF-QR method for different correlation coefficients and different sample sizes, that is, there are four scenarios, as follows: (1) $n=30$, $\rho =0.5$, (2) $n=30$, $\rho =0.8$, (3) $n=50$, $\rho =0.5$, (4) $n=50$, $\rho =0.8$ at quantiles $\tau =0.25$, $0.5$, and $0.75$, respectively. In addition, we discuss the cases $\sigma =0.2$ and 2, respectively. Here, ${\epsilon }_{ij}\sim N(0,1)$ and $\upsilon =1$. We select the penalty parameters by EBIC and take ${\gamma }_p=1$. We take the coefficient ${\beta }_l=0$ if $|{\beta }_l|<{10}^{-4}$, and the same for ${\alpha }_i$.

Table 1. PE, TP, and TN for different correlation coefficients and sample sizes.

$\mathit{\tau }$	Sample Size	Correlation Coefficient	$\mathit{\sigma }$	PE (SD)	TP	TN
0.25	n = 30	$\rho =0.5$	$0.2$	0.110 (0.101)	3	4.36
			2	0.290 (0.201)	3	4.36
		$\rho =0.8$	$0.2$	0.123 (0.101)	3	3.98
			2	0.223 (0.138)	3	3.72
	n = 50	$\rho =0.5$	$0.2$	0.067 (0.071)	3	3.89
			2	0.120 (0.092)	3	3.68
		$\rho =0.8$	$0.2$	0.073 (0.066)	3	3.55
			2	0.103 (0.085)	3	3.33
0.5	n = 30	$\rho =0.5$	$0.2$	0.097 (0.088)	3	3.97
			2	0.261 (0.185)	3	4.48
		$\rho =0.8$	0.2	0.088 (0.081)	3	3.87
			2	0.191 (0.123)	3	3.81
	n = 50	$\rho =0.5$	$0.2$	0.059 (0.046)	3	4.01
			2	0.119 (0.080)	3	4.03
		$\rho =0.8$	$0.2$	0.074 (0.064)	3	3.27
			2	0.096 (0.079)	3	3.33
0.75	n = 30	$\rho =0.5$	$0.2$	0.130 (0.105)	3	4.01
			2	0.321 (0.229)	3	4.11
		$\rho =0.8$	$0.2$	0.110 (0.094)	3	4.10
			2	0.204 (0.129)	3	3.81
	n = 50	$\rho =0.5$	$0.2$	0.064 (0.055)	3	3.85
			2	0.122 (0.091)	3	3.70
		$\rho =0.8$	$0.2$	0.080 (0.068)	3	3.47
			2	0.117 (0.091)	3	3.53

reports the mean values of PE, standard deviation (SD), and the mean values of TP and TN based on 100 replications.

The simulation shows that we could select the right variable by DERF-QR in all experimental conditions in Table 1. Also, PE decreases and TN decreases slightly when sample size n increases. It is observed that stronger correlation $\rho =0.8$ yields smaller PE values, compared with $\rho =0.5$. The DERF-QR method in strong-correlation scenarios shows robust performance due to its adaptive penalty weights. This improves coefficient estimation accuracy under strong variable correlations. Parameter $\sigma$ significantly influences the result. The PE value of $\sigma =2$ is greater than that of $\sigma =0.2$. Additionally, we observe that the model is less sensitive to variations of the parameter $\sigma$ at the median quantile. The median quantile regression also yields a smaller PE value, compared with other quantiles. The simulation results are presented in Figure 2.

Consider different SNR $\upsilon =1$ and $\upsilon =2$, and define three different random-effect matrices, $D_1=diag\left(1,1,0,0\right),$ $D_2=diag\left(2,2,0,0\right)$ and $D_3=diag\left(3,3,0,0\right)$, to investigate the effects of different SNRs and random effects on the model. Take $\tau =0.5$, $n=30$ and $\rho =0.5$.

Table 2. Average values of PE, TP, and TN in different conditions.

$\mathit{\upsilon }$	Random-Effect Matrix	PE (SD)	TP	TN
1	$D_1$	0.097 (0.088)	3	3.97
	$D_2$	0.186 (0.166)	3	4.63
	$D_3$	0.274 (0.264)	3	4.39
2	$D_1$	0.286 (0.234)	3	4.35
	$D_2$	0.343 (0.288)	3	4.40
	$D_3$	0.486 (0.394)	2.99	4.25

indicates the average values of PE, TP, and TN in 100 simulations.

In Table 2, as the SNR becomes greater and the influence of random effects increases, PE and SD increase gradually, while TN decreases slightly. If the SNR becomes greater, the model is less accurate and loses stability. In this case, it is more likely to select incorrect variables. However, we observe that even in the case $\upsilon =2$ and $D=D_3$, there are still relatively low errors for the model, and most incorrect variables are excluded. In general, an increase in SNR and random effects significantly impacts the coefficient estimation and variable selection. Meanwhile, the model is robust and effective in the process.

To investigate the performance of DERF-QR and other methods, we discuss the following four scenarios with different error distributions. ${\epsilon }_{ij}\sim N\left(0,1\right),$ ${\epsilon }_{ij}\sim t\left(3\right),$ ${\epsilon }_{ij}\sim Cauchy\left(1,0\right)$ and ${\epsilon }_{ij}\sim 0.8N\left(0,1\right)+0.2N\left(0,5^2\right),$ which denotes ${\epsilon }_{ij}\sim \mathrm{mixnorm}.$

We would make comparisons among the following four methods: Quantile Regression without penalization (QR), Doubly penalized Lasso regularized Quantile Regression (DLQR), Doubly penalized SCAD regularized Quantile Regression (DSQR), and the DERF-QR model with the parameter $\sigma =0.2$ proposed in this paper. Consider two distinct scenarios as follows:

$$\begin{array}{cc}\hfill & sparse model\beta ={\left(3,1,6,0,0,0,0,0\right)}^{\mathrm{T}},\hfill \\ \hfill & dense model\beta ={\left(0.75,0.75,0.75,0.75,0.75,0.75,0.75,0.75\right)}^{\mathrm{T}}.\hfill \end{array}$$

Repeat 100 times, and take $\tau =0.5$, $n=30$ and $\rho =0.5$.

Table 3. Comparison among different methods in the sparse model.

Error Distribution	Method	PE (SD)	TP	TN
${\epsilon }_{ij}\sim N(0,1)$	QR	0.182 (0.100)	3	0
	DLQR	0.455 (0.324)	3	3.95
	DSQR	0.101 (0.094)	3	2.35
	DERF-QR	0.085 (0.071)	3	4.38
${\epsilon }_{ij}\sim \mathrm{Cauchy}(1,0)$	QR	0.063 (0.061)	3	0
	DLQR	0.526 (0.386)	3	3.90
	DSQR	0.088 (0.105)	3	3.57
	DERF-QR	0.128 (0.138)	3	4.56
${\epsilon }_{ij}\sim t\left(3\right)$	QR	0.264 (0.139)	3	0
	DLQR	0.565 (0.328)	3	3.99
	DSQR	0.145 (0.113)	3	1.79
	DERF-QR	0.078 (0.069)	3	4.51
${\epsilon }_{ij}\sim \mathrm{mixnorm}$	QR	0.242 (0.144)	3	0
	DLQR	0.582 (0.371)	3	3.94
	DSQR	0.146 (0.102)	3	2.51
	DERF-QR	0.110 (0.098)	3	4.44

and

Table 4. Comparison among different methods in the dense model.

Error Distribution	Method	PE (SD)	TP	TN
${\epsilon }_{ij}\sim N(0,1)$	QR	0.165 (0.081)	8	0
	DLQR	1.02 (0.366)	8	0
	DSQR	0.173 (0.133)	8	0
	DERF-QR	0.169 (0.113)	7.99	0
${\epsilon }_{ij}\sim \mathrm{Cauchy}(1,0)$	QR	0.078 (0.046)	8	0
	DLQR	1.153 (0.488)	8	0
	DSQR	0.091 (0.108)	8	0
	DERF-QR	0.186 (0.115)	7.99	0
${\epsilon }_{ij}\sim t\left(3\right)$	QR	0.245 (0.116)	8	0
	DLQR	1.243 (0.505)	8	0
	DSQR	0.193 (0.113)	7.98	0
	DERF-QR	0.178 (0.093)	8	0
${\epsilon }_{ij}\sim \mathrm{mixnorm}$	QR	0.263 (0.146)	8	0
	DLQR	1.34 (0.493)	7.98	0
	DSQR	0.252 (0.184)	8	0
	DERF-QR	0.22 (0.143)	7.96	0

depict the results, and Figure 3 and Figure 4 show the boxplots of PE for these two models.

It can be observed from Table 3 that we could select the right variables and derive lower PE by using DLQR, DSQR, and DERF-QR in the sparse model. The DERF-QR method shows robust coefficient estimation performance. Notably, it excludes up to 80% of incorrect variables across all cases. There are four different error distributions in Table 3, where there are a significant number of outliers in the Cauchy distribution with heavy tails. DERF-QR exhibits relatively greater PE in the Cauchy distribution but maintains the lowest PE among all methods across other distributions. It is unstable to exclude incorrect variables by DSQR, even though DSQR performs well in coefficient estimation and its PE is close to that of DERF-QR. The PE in DLQR is relatively greater, even though it could exclude incorrect variables clearly. Generally, DERF-QR is the optimal one of these four methods.

In the dense model, DERF-QR still performs the best among the four methods. We could select the correct variables from all by these four methods. In most cases, the Cauchy distribution shows smaller errors, while the mixnorm distribution results in relatively greater errors. For Cauchy distribution, we obtain lower PE by DQR and DSQR, compared with DERF-QR. In other distributions, the PE of DSQR is similar to that of DERF-QR. The PE of DLQR is slightly greater than the others in all cases.

4. Financial Data Analysis

The DERF-QR method is developed in the financial statements and balance sheets of 12 companies from 2009 to 2023 [25]. Due to missing data, we select 10 companies from 2009 to 2022 for simulation. The revenue is regarded as the response variable. We aim to find the factors influencing the revenue. Consider 20 variables as explanatory variables as in

Table 5. Explanatory variables.

Variable	Name	Variable	Name
X1	CFF Activities	X11	Market Cap
X2	CFI Activities	X12	Net Income
X3	CFO Activities	X13	Net Profit Margin
X4	Current Ratio	X14	Number of Employees
X5	Debt/Equity Ratio	X15	ROA
X6	EBITDA	X16	ROE
X7	Earnings/Share	X17	ROI
X8	Free Cash Flow/Share	X18	Return on Tangible Equity
X9	Gross Profit	X19	Shareholder Equation
X10	Inflation Rate	X20	Bank
X21	ELEC	X22	FOOD
X23	IT	X24	LOGI
X25	Manufacturing

.

In Table 5, X₂₀, X₂₁, X₂₂, X₂₃, X₂₄, and X₂₅ denote the numerical variables for the company’s six categories. Specifically, each category is represented using binary indicators, where applicable categories are marked with a 1 and non-applicable ones with a 0. The other explanatory variables are continuous. Consider the following linear mixed-effects model:

$$\begin{array}{cc}\hfill Y_{ij}& =x_{ij}^{\mathrm{T}}\beta +z_{ij}^{\mathrm{T}}{\alpha }_i+{\epsilon }_{ij},i=1,2,\dots ,10,j=1,2,\dots ,14,\hfill \end{array}$$

(14)

where $Y_{ij}$ denotes the revenue of company i in year j. For the random-effects coefficients ${\alpha }_i$, we consider that random effects exist in all variables, i.e., $z_{ij}^{\mathrm{T}}=x_{ij}^{\mathrm{T}}.$

We empirically evaluated DERF-QR under $\sigma =0.2$ and $\sigma =2$, at quantile levels $\tau =0.25$, $0.5$, and $0.75$. For comparison, we also included the DLQR method. The mean squared error (MSE) is defined as

$$\mathrm{MSE}={\displaystyle \frac{1}{nm}}\sum _{i=1}^n\sum _{j=1}^m{\left(Y_{ij}-{\widehat{Y}}_{ij}\right)}^2$$

and the number of non-zero coefficients (model size, MS) is used to assess variable selection performance.

Figure 5 presents the estimated coefficients across quantiles. Both DERF-QR and DLQR successfully eliminate most irrelevant variables. Notably, all methods consistently selected X₆ and X₉, while the other variables are very small or shrink to zero. This indicates that gross profit and EBITDA have a significant impact on revenue. We find that variable selection differs across quantile levels. At $\tau =0.75$, all three methods tend to select X₆, X₉, and X₁₄.

Table 6. MSE and MS for different methods at various quantile levels.

Method	$\mathit{\tau }=0.25$		$\mathit{\tau }=0.5$		$\mathit{\tau }=0.75$
	MSE	MS	MSE	MS	MSE	MS
DERFQR ($\sigma =0.2$)	0.230	11	0.048	11	0.060	13
DERFQR ($\sigma =2$)	0.256	11	0.032	11	0.061	11
DLQR	0.266	10	0.046	15	0.063	12

shows the MSE and MS for each method. When $\sigma =0.2$, DERF-QR achieves the lowest MSE at $\tau =0.25$ and $\tau =0.75$. With $\sigma =2$, DERF-QR attains the lowest MSE at $\tau =0.5$ and also yields the smallest MS at both $\tau =0.5$ and $\tau =0.75$. DERF-QR consistently yields the lowest MSE across quantile levels. Overall, DERF-QR consistently achieves superior estimation accuracy and variable selection across quantile levels.

5. Discussion

We conducted Monte Carlo simulations of the proposed method under various conditions, including different SNRs, sample sizes n, correlation coefficients $\rho$, and sparsity levels. The results demonstrate that DERF-QR consistently achieves the lowest PE and highest TN in most scenarios, indicating its strong performance in both variable selection and parameter estimation.

Despite DERF-QR’s strong overall performance, its estimation accuracy declines under Cauchy-distributed errors. This can be attributed to the heavy-tailed nature of the Cauchy distribution, which introduces more extreme outliers and may compromise the robustness of coefficient estimation. Therefore, evaluating the robustness of DERF-QR and related methods under heavy-tailed error distributions remains an important direction for future research.

The simulation results show that $\sigma$ significantly affects DERF-QR’s estimation performance. We conducted simulations under the setting to examine how different values of $\sigma$ affect the PE at $\tau =0.25$, $0.5$, and $0.75$, respectively. Taking $n=30$, $\rho =30$, ${\epsilon }_{ij}\sim N\left(0,1\right)$, Figure 6 shows the trend of how the PE varies with increasing $\sigma$ at different quantile levels. As $\sigma$ increases, the PE increases across all three quantile levels. When $\tau =0.5$, the impact of $\sigma$ on PE is relatively smaller compared to the other quantiles. Based on these observations, we recommend selecting the $\sigma$ within a relatively small range through cross-validation in practical applications.

Additionally, we focus exclusively on EBIC performance for ${\lambda }_{\beta }$ and ${\lambda }_{\alpha }$ selection. Other parameter selection approaches, including the Schwarz information criterion (SIC), generalized approximate cross-validation (GACV), and cross-validation methods, were not evaluated in our comparative analysis. Future comparisons of these methods would strengthen tuning strategy evaluation.

6. Conclusions

In this paper, we established the DERF-QR method for both parameter estimation and variable selection in the linear mixed-effects model. We studied a non-convex and smooth error function regularization, which is quite different from traditional regularization methods for variable selection in quantile regression for this model. We derived the asymptotic behavior of the penalty, and viewed the penalty as a close approximation to $L_0$ regularization as $\sigma \to 0^{+}$.

We conducted simulations under various scenarios. DERF-QR yielded lower PE across various signal-to-noise ratios and random-effect structures. Even under the challenging setting of $\upsilon =2$ and $D=D_3$, it retained low PE and successfully eliminated 80% irrelevant variables. When $n=50$ and $\rho =0.8$, PE remained lower than under $n=30$ and $\rho =0.5$, indicating strong stability in the presence of larger sample sizes and high correlation. Compared with other penalized quantile regression methods, DERF-QR consistently achieved the lowest PE and the highest TN in both sparse and dense settings, demonstrating robust performance in coefficient estimation and variable selection. We also developed this method for financial data and showed that it is practically effective.

Our proposed DERF-QR method effectively extends penalized quantile regression for linear mixed-effects models, demonstrating excellent performance in both parameter estimation and variable selection. There are still some open issues for future research in this field. For instance, the IRW-pADMM algorithm involves parameter selection, and selecting inappropriate parameters could result in slow convergence during the iteration process. Therefore, improving the efficiency of the algorithm remains an important topic for future work. Additionally, this approach can be further applied and studied in other models, such as spatial mixed-effects models or generalized linear mixed models.