Poisson Mixed-Effects Count Regression Model Based on Double SCAD Penalty and Its Simulation Study

Abstract

This paper focuses on variable selection and parameter estimation for mixed-effects Poisson count regression models. To simultaneously select important variables in both fixed effects and random effects, we propose a double-penalized Poisson count regression model with the Smoothly Clipped Absolute Deviation (SCAD) penalty imposed on both components. To estimate the unknown parameters, we develop a new iterative algorithm called the Double SCAD–Local Quadratic Approximation (DSCAD-LQA) algorithm. Under regularity conditions, the consistency and Oracle property of the proposed estimator are established. Simulation studies are conducted under two types of penalty parameter selection criteria: the Schwarz Information Criterion (SIC) and the Generalized Approximate Cross-Validation (GACV). We evaluate the performance of the proposed method under different levels of correlation among explanatory variables and different covariance structures of random effects. Comparisons are also carried out with the non-penalized model, the single-penalized model, and the double LASSO-penalized model. The results demonstrate that the proposed double SCAD penalty method performs better than the other three methods in terms of important variable selection and coefficient estimation, and is especially effective for sparse models.

1. Introduction

With the transformation of modern production and manufacturing towards intelligence and refinement, the accurate modeling of various count-type quality indicators in the production process (such as the number of defects, failure times, and non-conforming parts) is a core demand to ensure production stability and reduce costs. These indicators are inherently non-negative integers, a defining feature of count data. Traditional linear models, such as ordinary least squares (OLS) regression, are not suitable for this type of data because they assume the response variable follows a continuous, normal distribution, which directly contradicts the discrete, non-negative nature of count outcomes.

Meanwhile, the manufacturing process is continuous, and the observation data of the same production line, equipment, or batch have longitudinal correlation in the time dimension. Traditional linear models assume independence among observations, and ignoring this inherent correlation can lead to biased parameter estimates and undermine the reliability of the model. Common modeling approaches for such longitudinal count data include marginal models [1] and mixed-effects models [2,3], among which the mixed-effects Poisson count model with random effects is the most widely used. The primary reason is that longitudinal count sample data are non-independent, with a certain degree of correlation among observations from the same unit. Mixed-effects models allow for such data correlation. They take into account the practical scenario that the production units selected for data collection are often randomly sampled, where each unit is associated with distinct raw materials and operators [4,5]. Since the research objective focuses on the overall parameter characteristics of the population rather than individual units, incorporating unit-specific individual effects as random effects into the model enhances modeling accuracy and flexibility [6].

On the other hand, practical modeling often confronts the challenge of high-dimensional data, where the number of variables is close to or even exceeds the sample size [7]. The curse of dimensionality makes variable selection indispensable, and various penalization methods enable concurrent parameter estimation and variable selection [8]. The LASSO (Least Absolute Shrinkage and Selection Operator) penalty proposed by Tibshirani [9] has been extremely widely adopted. It not only estimates the regression parameters in the model but also shrinks the coefficients of irrelevant variables to zero, thereby achieving the goal of variable selection. However, the LASSO penalty tends to introduce estimation bias for large coefficients. To address this limitation, Fan and Li [10] proposed a novel SCAD (Smoothly Clipped Absolute Deviation) penalty based on the LASSO method. This non-convex penalization approach inherits the advantages of the LASSO penalty while possessing favorable statistical properties: it yields sparse solutions, and the resulting estimates satisfy asymptotic normality.

Since its introduction, the SCAD penalty has been widely applied to various models, such as quantile regression [11] and logistic regression [12,13]. For Poisson regression models, Yan and Chen [14] adopted the SCAD penalty for variable selection, while Buu [15] further extended its application to zero-inflated Poisson regression models for the same purpose. Li [16] employed the SCAD penalty to perform variable selection for high-dimensional random-effects linear regression models and high-dimensional random-effects quantile regression models; Li and He [17] also implemented the SCAD penalty for variable selection in high-dimensional random-effects linear regression models. Liu [18] constructed a SCAD-regularized heterogeneous autoregressive model for volatility forecasting. Liu and Chen [19] proposed an analytical framework integrating self-supervised learning with SCAD-Net regularization. Liu, Wang, and Wu [20] conducted research on SCAD-type variable selection and estimation in censored regression models. Zhang and Dong [21] carried out a comparative study on the performance of six variable selection methods—LASSO, Ridge Regression, Elastic Net, SCAD, Bridge, and Adaptive Bridge—in Poisson regression, and verified that the SCAD method exhibits the best performance in terms of model goodness-of-fit and prediction accuracy, making it suitable for large-sample and high-correlation datasets.

Moreover, all the aforementioned applications of the SCAD penalty only target the fixed-effects component of the respective models [22]. For mixed-effects Poisson regression models, performing variable selection solely on fixed effects while ignoring random effects will introduce bias into the estimation of fixed effects; conversely, incorporating an excessive number of irrelevant random effects may lead to the singularity of the random-effects covariance matrix. To address this problem, Ibrahim et al. [23] once imposed the SCAD penalty on both fixed and random effects of generalized linear mixed-effects models for variable selection, with parameters solved via the LLA (Local Linear Approximation) algorithm. However, their method penalizes the variances of random effects and requires these variances to follow a normal distribution. This penalization scheme has a drawback: when the variance of a certain random effect is shrunk to zero, the estimated values of all individuals under this random effect are forced to zero, which neglects the potential impact of non-zero random effects for some individuals.

Against this backdrop, this paper directly penalizes the random effects and proposes a double SCAD-penalized mixed-effects Poisson count regression model that enables simultaneous selection of both fixed and random effects. This penalization framework imposes no constraints on the variances of random effects. Furthermore, the parameter estimation algorithm in this paper is based on the LQA (Local Quadratic Approximation) algorithm [10], which yields more accurate estimates compared with the LLA algorithm.

The model proposed in this paper differs fundamentally from the method of Ibrahim et al. [23] in terms of penalization target, regularization logic, and hierarchical modeling strategy. Although Ibrahim et al. [23] applied the SCAD penalty to both fixed and random effects in generalized linear mixed-effects models; their penalization target was the variance of random effects. The corresponding regularization logic was to conduct variable selection by shrinking the variances; when a variance was shrunk to zero, the estimates of all individuals under that random effect were forced to zero, representing a global selection scheme that failed to capture individual-level heterogeneity. Their hierarchical modeling required the random effects to follow a normal distribution and adopted the LLA (Local Linear Approximation) algorithm for parameter estimation.

In contrast, this paper directly takes fixed-effect coefficients and random-effect coefficients as the penalization targets and uses the double SCAD penalty to achieve simultaneous selection of both types of effects. The regularization process does not rely on variance shrinkage, allowing accurate identification and retention of meaningful individual random effects. In terms of hierarchical modeling, no distributional constraint is imposed on the variances of random effects, and the more accurate LQA (Local Quadratic Approximation) algorithm is employed for parameter estimation.

When applied to cigarette production data from multiple units, this model simultaneously penalizes the numerous unknown random effects of individual units and the fixed effects of various process parameters. It not only identifies a small set of key process indicators affecting the number of short positions but also accurately estimates their impact weights, thereby providing valuable insights for the further adjustment and optimization of process parameters. In fact, similar double-penalization ideas have been explored in existing literature; for instance, Luo et al. [24] investigated the issue of double regularization penalties for mixed-effects models and proved the convergence of the iterative algorithm.

Most existing penalization strategies for Poisson mixed-effects count regression models are imposed on the variance components of random effects. Such approaches only achieve global shrinkage and overall selection of random effects, and suffer from two critical limitations:

First, they cannot perform independent and precise variable selection on the individual random-effect coefficients ${\alpha }_i$. In high-dimensional settings, retaining redundant individual effects or excluding key heterogeneous effects tends to introduce substantial fitting bias. Second, when the variance of a random effect is shrunk to zero, the estimates for all individuals under that effect are forced to zero, which ignores the potential heterogeneity where some individuals may still exhibit non-zero random effects. This is incompatible with the characteristic of count data in practice: the overall trend is consistent while individuals show specific heterogeneity. Meanwhile, such methods require the variances of random effects to follow a normal distribution, which further restricts their applicability in modeling high-dimensional and non-normal count data.

In contrast, penalizing the random-effect coefficients ${\alpha }_i$ directly can overcome the above limitations. It enables simultaneous and refined variable selection for both fixed effects and random effects. This approach effectively removes irrelevant individual random effects while preserving meaningful individual heterogeneity. Moreover, it imposes no distributional constraints on the variances of random effects, making it better suited for modeling high-dimensional count-type quality indicators in manufacturing, biostatistics, and other fields. It thus fills the research gap of traditional penalization strategies in accurately capturing individual heterogeneity and adapting to high-dimensional scenarios.

Section 2 presents the general form of the mixed-effects Poisson count regression model. Based on the likelihood function, a novel double-penalization method is proposed by simultaneously imposing the SCAD penalty on both fixed-effects and random-effects regression coefficients. This section also introduces the iterative local quadratic approximation algorithm for model parameter estimation and discusses the selection criteria for optimal penalty parameters.

Section 3 provides rigorous proofs of the relevant theoretical properties of the proposed method.

Section 4 conducts comparisons between the proposed method and traditional methods via Monte Carlo simulations, evaluating their performance in variable selection and regression coefficient estimation under various scenarios.

Section 5 demonstrates the practical application scenarios of the proposed method, illustrating its applicability in real-world contexts.

Finally, Section 6 summarizes the main findings of the paper and offers concluding remarks.

2. Double Penalty Poisson Counting Regression Model

2.1. Double Penalty Poisson Counting Regression Model with SCAD Penalty

2.1.1. Model Establishment

To ensure compatibility of matrix operations in the model and facilitate verification and reproducibility, this section first standardizes all core variables, the meanings of subscripts, and dimensions involved in the model. The specific definitions are as follows:

Explanation of Subscripts: $i=1,2,\dots ,n$ denotes the observation unit, reflecting individual heterogeneity; $j=1,2,\dots ,m$ denotes the $j\text{-}th$ repeated observation within the $i\text{-}th$ observation unit, reflecting the time/batch correlation of longitudinal data.

Definition of Core Variables and Dimensions:

Response Variable: $y_{ij}$ represents the count-type response variable of the $j\text{-}th$ observation for the $i\text{-}th$ observation unit, and its matrix form is $y={\left(y_{11},y_{12},\dots ,y_{1m},y_{21},\dots ,y_{nm}\right)}^T$.

Fixed Effect-related Variables: $x_{ij}$ represents the fixed-effect explanatory variable vector of the $j\text{-}th$ observation for the $i\text{-}th$ observation unit, where the subscript $ij$ corresponds strictly to the response variable $y_{ij}$. Its dimension is defined as $x_{ij}\in R^k$, where k is the total number of fixed-effect explanatory variables; $\beta$ is the fixed-effect regression coefficient vector, whose dimension matches that of $x_{ij}$, i.e., $\beta \in R^k$, ensuring the compatibility of the linear operation $x_{ij}^T\beta$.

Random Effect-related Variables: $z_{ij}$ represents the random-effect explanatory variable vector (covariate vector) of the $j\text{-}th$ observation for the $i\text{-}th$ observation unit, where the subscript $ij$ is consistent with those of $y_{ij}$ and $x_{ij}$. Its dimension is defined as $z_{ij}\in R^p$, where p is the total number of random-effect explanatory variables (covariates); ${\alpha }_i$ is the random-effect coefficient vector of the $i\text{-}th$ observation unit, whose dimension matches that of $z_{ij}$, i.e., ${\alpha }_i\in R^p$, ensuring the compatibility of the linear operation $z_{ij}^T{\alpha }_i$ (scalar output). Its matrix form is $\alpha ={\left({\alpha }_1^T,{\alpha }_2^T,\dots ,{\alpha }_n^T\right)}^T\in R^{np\times 1}$.

Supplementary Illustration: X is the fixed-effect explanatory variable matrix, whose matrix form is $X={\left(x_{11}^T,x_{12}^T,\dots ,x_{1m}^T,x_{21}^T,\dots ,x_{nm}^T\right)}^T\in R^{N\times k}$. It matches the fixed-effect coefficient vector $\beta \in R^k$, ensuring the compatibility of the matrix operation $X\beta$; Z is the random-effect explanatory variable (covariate) matrix, whose matrix form is $Z=diag\left(Z_1,Z_2,\dots ,Z_n\right)$, where $Z_i={\left(z_{i1},z_{i2},\dots ,z_{im}\right)}^T\in R^{m\times p}$ and the overall dimension is $Z\in R^{N\times np}$. It matches the random-effect coefficient vector $\alpha \in R^{np\times 1}$, ensuring the compatibility of the matrix operation $Z_{\alpha }$. N denotes the total sample size, which is equal to nm; that is, there are n individuals with m observations per individual.

In the general Poisson counting regression model, assuming that the response variable Y obeys the Poisson distribution,

$$P\left(Y=y\right)={\displaystyle \frac{{\lambda }^y{e}^{-\lambda }}{y!}}=\exp \left\{y\log \left(\lambda \right)-\lambda -\log \left(y!\right)\right\},y=0,1,2,\dots ,$$

(1)

where $\lambda >0$, $E\left(Y\right)=\lambda$, using the family standard forms of the generalized linear model exponential distribution, there are

$$\begin{gathered} \theta =\log \left(\lambda \right),b\left(\theta \right)=\lambda =e^{\theta },a\left(\varphi \right)=1,c\left(y,\varphi \right)=-\log \left(y!\right) \\ g\left(u\right)=g\left(E\left(Y|X\right)\right)=\mathrm{l}\mathrm{o}\mathrm{g}\left(\mu \right)=\mathrm{l}\mathrm{o}\mathrm{g}\left(b^{\prime }\left(\theta \right)\right)=\mathrm{l}\mathrm{o}\mathrm{g}\left(e^{\theta }\right)=\theta =X\beta \end{gathered}$$

(2)

By this formulation, the general Poisson counting regression model is given by the following:

$$E\left(Y|X\right)=e^{X\beta }$$

(3)

where $\mathrm{X}$ is an explanatory variable, $\beta$ is the corresponding coefficient.

Assume that there are n subjects, and each subject is observed repeatedly. For the $i\text{-}th$ subject $(i=1,\dots ,n)$, let $y_{ij}$ be the response variable for the $j\text{-}th$ observation $(j=1,\dots ,m)$. Such clustered or longitudinal data arise frequently in applied sciences. Then generalize model (3) to a Poisson counting regression model with mixed effects:

$$E\left(y_{\mathrm{ij}}|x_{\mathrm{ij}},z_{ij}\right)=e^{x_{ij}^T\beta +z_{ij}^T{\alpha }_i}$$

(4)

where $x_{ij}$ is a $k\times 1$ vector of explanatory variables, $\beta$ is the corresponding vector of $x_{ij}$ with an unknown fixed effect coefficient, $z_{ij}$ is the $p\times 1$ vector of covariates, ${\alpha }_i$ is the corresponding vector of $z_{ij}$ with an unknown random effect coefficient. The probability density function of the response variable is obtained from Equation (1):

$$p\left(y_{ij}|x_{ij},z_{ij}\right)={\displaystyle \frac{{\left[E\left(y_{ij}|x_{ij},z_{ij}\right)\right]}^{y_i}·e^{-E\left(y_{ij}|x_{ij},z_{ij}\right)}}{y_i!}}$$

(5)

From this, the log-likelihood function is simplified, given by the following Equation:

$$l\left(\beta ,\alpha \right)=\sum _{\mathrm{i}=1}^{\mathrm{n}}\sum _{j=1}^m\left[y_{ij}\left(x_{ij}^T\beta +z_{ij}^T{\alpha }_i\right)-e^{x_{ij}^T\beta +z_{ij}^T{\alpha }_i}\right]$$

(6)

Considering that the fixed-effect coefficient vector $\beta$ and random-effect coefficient vector ${\alpha }_{\mathcal{i}}$ are unknown in the above likelihood function, this paper applies the SCAD penalty to both fixed effects and random effects. This strategy takes full advantage of the desirable estimation properties of the SCAD penalty, enabling the model to produce unbiased sparse solutions. It can not only select important explanatory variables but also prevent the model from overfitting caused by including excessive random effects. This leads to the following minimized double SCAD-penalized likelihood function:

$$\begin{gathered} L\left(\beta ,\alpha \right)=\min \left\{\sum _{\mathrm{i}=1}^{\mathrm{n}}\sum _{j=1}^m\left[-y_{ij}\left(x_{ij}^T\beta +z_{ij}^T{\alpha }_i\right)+e^{x_{ij}^T\beta +z_{ij}^T{\alpha }_i}\right]+\sum _{\mathrm{l}=0}^k{p}_{\lambda }\left(\left|{\beta }_l\right|\right) \\ +\sum _{\mathrm{i}=1}^{\mathrm{n}}\sum _{t=1}^p{p}_{\lambda }\left(\left|{\alpha }_{it}\right|\right)\right\}. \end{gathered}$$

(7)

The equation above is also equivalent to the following:

$$\begin{gathered} L\left(\beta ,\alpha \right)=max\left({\sum }_{\mathrm{i}=1}^{\mathrm{n}}{\sum }_{j=1}^m\left(y_{ij}\left(x_{ij}^T\beta +z_{ij}^T{\alpha }_i\right)-e^{x_{ij}^T\beta +z_{ij}^T{\alpha }_i}-{\sum }_{\mathrm{l}=0}^k{p}_{\lambda }\left(\left|{\beta }_l\right|\right)- \\ {\sum }_{\mathrm{i}=1}^{\mathrm{n}}{\sum }_{t=1}^p{p}_{\lambda }\left(\left|{\alpha }_{it}\right|\right)\right)\right) \end{gathered}$$

(8)

where the SCAD penalty function takes the specific form:

$$p_{\lambda }\left(\left|{\theta }_i\right|\right)=\left\{\begin{array}{cc}\lambda \left|{\theta }_i\right|,\hfill & \hfill 0\le \left|{\theta }_i\right|\le \lambda \\ -{\displaystyle \frac{{\left|{\theta }_i\right|}^2-2a\lambda \left|{\theta }_i\right|+{\lambda }^2}{2\left(a-1\right)}},\hfill & \hfill \lambda <\left|{\theta }_i\right|\le a\lambda \\ {\displaystyle \frac{\left(a+1\right){\lambda }^2}{2}},\hfill & \hfill \left|{\theta }_i\right|>a\lambda \end{array}\right.$$

(9)

where $\lambda >0$ is penalty parameter, $a$ is constant, and the recommended value is usually taken as 3.7 [10].

2.1.2. Statistical Interpretation and Methodological Comparison

(1)

Statistical Interpretation of the Double-Penalized Likelihood

The double-penalized likelihood function in Equation (7) enables simultaneous shrinkage estimation of both fixed effects ${\beta }_l,l=1,\dots ,k$ and individual random-effect realizations αit. Unlike traditional variance-component penalization, which targets the covariance structure of random effects, our approach directly penalizes the coefficient values of ${\alpha }_{it}$. This has a clear interpretation in terms of hierarchical borrowing strength:

For random-effect coefficients that are statistically indistinguishable from zero, the SCAD penalty will shrink them to zero, effectively removing redundant individual-level variation from the model.

For coefficients that represent meaningful heterogeneity, the penalty is milder, allowing the model to retain and exploit within-cluster information while avoiding over-borrowing of between-cluster information.

This local, coefficient-wise shrinkage controls the effective model complexity more precisely, which is critical in high-dimensional settings where many random-effect terms may be spurious or irrelevant.

(2)

Comparison with Variance-Parameter Penalization

From

Table 1. Comparison between variance-component penalization and direct coefficient penalization.

Dimension	Variance-Component Penalization	Direct Coefficient Penalization (Our Approach)
Penalty Target	Random-effect variances $Var({\alpha }_i)$	Individual random-effect realizations ${\alpha }_{it}$
Shrinkage Behavior	All-or-nothing: if $Var({\alpha }_i)\to 0$, all αit are forced to 0	Local: only irrelevant ${\alpha }_{it}$ are shrunk to 0
Hierarchical Borrowing	Erases all within-cluster variation for a given effect	Preserves meaningful individual heterogeneity
Effective Complexity	Controls only the number of active random effects	Controls both the number of effects and the sparsity of their realizations

, it can be seen that variance-component penalization is a global operation: when a variance is shrunk to zero, it eliminates all individual-level variation for that effect. In contrast, our double SCAD penalty operates at the level of individual realizations, allowing the model to retain sparse and meaningful heterogeneity while discarding noise. This is particularly valuable in industrial applications, where only a subset of production units may exhibit anomalous behavior, while the rest follow the global trend.

2.2. Iterative Local Quadratic Approximation Algorithm (DSCAD-LQA) for Model Unknown Parameter Estimation

The parameter estimation algorithm for Equation (8) is mainly based on the local quadratic approximation algorithm (LQA) proposed by Fan and Li (2001) [10]. Different from the conventional LQA, Equation (8) in this paper involves two penalty functions. Therefore, we develop an improved iterative algorithm based on LQA, referred to as the DSCAD-LQA (Double Smoothly Clipped Absolute Deviation–Local Quadratic Approximation). The specific steps of the algorithm are as follows.

Step 1: The initial values ${\beta }^0$, ${\alpha }^0$ of the fixed effect coefficients $\beta$ and the random effect coefficients $\alpha$ are first solved by the log-likelihood function without penalty (6) using the Newton iterative algorithm in the following steps (1) to (3), and updating the likelihood values during each iteration. The exact process of iteration is as follows:

(1)

At the first s + 1 iteration, $\beta$ value, a $k\times 1$ vector, is solved by the function: ${\beta }_l^{s+1}={\beta }_l^s-{\left.{\left[{\displaystyle \frac{{\mathsf{\partial }}^{2}l\left(\beta ,{\alpha }^s\right)}{\mathsf{\partial }{\beta }_l^2}}\right]}^{-1}{\displaystyle \frac{\mathsf{\partial }l\left(\beta ,{\alpha }^s\right)}{\mathsf{\partial }{\beta }_l}}\right|}_{\beta ={\beta }_l^s},l=1,\dots ,k$. Substitute the $\beta$ value to update the likelihood value.

(2)

Then, $\alpha$ value, a $p\times n$ vector, is solved by the function: ${\alpha }_{it}^{s+1}={\alpha }_{it}^s-{\left.{\left[{\displaystyle \frac{{\mathsf{\partial }}^{2}l\left({\beta }^{s+1},\alpha \right)}{\mathsf{\partial }{\alpha }_{it}^2}}\right]}^{-1}{\displaystyle \frac{\mathsf{\partial }l\left({\beta }^{s+1},\alpha \right)}{\mathsf{\partial }{\alpha }_{it}}}\right|}_{\alpha ={\alpha }_{it}^s},t=1,\dots ,p,i=1,\dots ,n$. Substitute the $\alpha$ value similarly to update the likelihood value.

(3)

The initial value ${\beta }^0$, ${\alpha }^0$ can be obtained by solving (1) and (2) iteratively until it stops at $\left|\left|{\displaystyle \frac{l\left({\beta }^{s+1},{\alpha }^{s+1}\right)-l\left({\beta }^s,{\alpha }^s\right)}{l\left({\beta }^s,{\alpha }^s\right)}}\right|\right|<\epsilon$, where $l\left(\beta ,\alpha \right)$ is a log-likelihood function without penalty (6).

Step 2: Subjecting the penalty functions $p_{\lambda }\left(\left|{\beta }_l\right|\right)$ and $p_{\lambda }\left(\left|{\alpha }_{it}\right|\right)$ to a second-order Taylor expansion at non-zero:

$$p_{\lambda }\left(\left|{\beta }_l\right|\right)\approx p_{\lambda }\left(\left|{\beta }_{l0}\right|\right)+{\displaystyle \frac{1}{2}}\left\{p_{\lambda }^{\prime }\left(\left|{\beta }_{l0}\right|\right)/\left|{\beta }_{l0}\right|\right\}\left({\beta }_l^2-{\beta }_{l0}^2\right),{\beta }_l\approx {\beta }_{l0}$$

(10)

$$p_{\lambda }\left(\left|{\alpha }_{it}\right|\right)\approx p_{\lambda }\left(\left|{\alpha }_{it0}\right|\right)+{\displaystyle \frac{1}{2}}\left\{p_{\lambda }^{\prime }\left(\left|{\alpha }_{it0}\right|\right)/\left|{\alpha }_{it0}\right|\right\}\left({\alpha }_{it}^2-{\alpha }_{it0}^2\right),{\alpha }_{it}\approx {\alpha }_{it0}$$

(11)

Calculate the derivative

$${\left[p_{\lambda }\left(\left|{\beta }_l\right|\right)\right]}^{\prime }\approx \left\{p_{\lambda }^{\prime }\left(\left|{\beta }_{l0}\right|\right)/\left|{\beta }_{l0}\right|\right\}{\beta }_l,{\beta }_l\approx {\beta }_{l0}$$

(12)

$${\left[p_{\lambda }\left(\left|{\alpha }_{it}\right|\right)\right]}^{\prime }\approx \left\{p_{\lambda }^{\prime }\left(\left|{\alpha }_{it0}\right|\right)/\left|{\alpha }_{it0}\right|\right\}{\alpha }_{it},{\alpha }_{it}\approx {\alpha }_{it0}$$

(13)

Step 3: Substitute the initial values ${\beta }^0$, ${\alpha }^0$ obtained from Step 1 into the objective function equation with double SCAD penalty (8), and find the initial objective value $LM=L\left({\beta }^0,{\alpha }^0\right)$. The final results ${\beta }^{best}$${\alpha }^{best}$ are also obtained by using Newton iterative method on the basis of the initial values ${\beta }^0$${\alpha }^0$ solved by the following alternating iterative solutions. The exact process of iteration is as follows:

(1)

At the first s + 1 iteration, $\beta$ value, a $k\times 1$ vector, is solved by the Function (14). Substitute the $\beta$ value to update the target value $LM=L\left({\beta }^{s+1},{\alpha }^s\right)$.

$${\beta }_l^{s+1}={\beta }_l^s-{\left.{\left[{\displaystyle \frac{{\mathsf{\partial }}^{2}l\left(\beta ,{\alpha }^s\right)}{\mathsf{\partial }{\beta }_l^2}}+{\displaystyle \frac{p_{\lambda }^{\prime }\left(\left|\beta \right|\right)}{\left|\beta \right|}}\right]}^{-1}\left[{\displaystyle \frac{\mathsf{\partial }l\left(\beta ,{\alpha }^s\right)}{\mathsf{\partial }{\beta }_l}}+\left\{{\displaystyle \frac{p_{\lambda }^{\prime }\left(\left|\beta \right|\right)}{\left|\beta \right|}}\right\}\beta \right]\right|}_{\beta ={\beta }_l^s},l=1,\dots ,k$$

(14)

(2)

Then the $\alpha$ value, a $p\times n$ vector, is obtained by solving Equation (15) in turn, and substituted to update the target value $LM=L\left({\beta }^{s+1},{\alpha }^{s+1}\right).$

$$\begin{gathered} {\alpha }_{it}^{s+1}={\alpha }_{it}^s-{\left.{\left[{\displaystyle \frac{{\mathsf{\partial }}^{2}l\left({\beta }^{s+1},\alpha \right)}{\mathsf{\partial }{\alpha }_{it}^2}}+{\displaystyle \frac{p_{\lambda }^{\prime }\left(\left|\alpha \right|\right)}{\left|\alpha \right|}}\right]}^{-1}\left[{\displaystyle \frac{\mathsf{\partial }l\left({\beta }^{s+1},\alpha \right)}{\mathsf{\partial }{\alpha }_{it}}}+\left\{{\displaystyle \frac{p_{\lambda }^{\prime }\left(\left|\alpha \right|\right)}{\left|\alpha \right|}}\right\}\alpha \right]\right|}_{\alpha ={\alpha }_{it}^s}, \\ t=1,\dots ,p,i=1,\dots ,n \end{gathered}$$

(15)

(3)

The final result ${\beta }^{best}$, ${\alpha }^{best}$ can be obtained by solving (1) and (2) iteratively until it stops at $\left|\left|{\displaystyle \frac{l\left({\beta }^{s+1},{\alpha }^{s+1}\right)-l\left({\beta }^s,{\alpha }^s\right)}{l\left({\beta }^s,{\alpha }^s\right)}}\right|\right|<\epsilon$, where $l\left(\beta ,\alpha \right)$ is the target Function (8).

Analysis of Computational Properties and Numerical Stability of the Algorithm

The iterative process of the aforementioned DSCAD-LQA relies fundamentally on the Newton iterative update mechanism, and the iterative formula for fixed effects can be further specified as follows:

$${\beta }_l^{s+1}={\beta }_l^s-{\left.{\left[{\displaystyle \frac{{\mathsf{\partial }}^{2}l\left(\beta ,{\alpha }^s\right)}{\mathsf{\partial }{\beta }_l^2}}+{\displaystyle \frac{p_{\lambda }^{\prime }\left(\left|\beta \right|\right)}{\left|\beta \right|}}\right]}^{-1}\left[{\displaystyle \frac{\mathsf{\partial }l\left(\beta ,{\alpha }^s\right)}{\mathsf{\partial }{\beta }_l}}+\left\{{\displaystyle \frac{p_{\lambda }^{\prime }\left(\left|\beta \right|\right)}{\left|\beta \right|}}\right\}\beta \right]\right|}_{\beta ={\beta }_l^s},l=1,\dots ,k$$

where ${\displaystyle \frac{\mathsf{\partial }l}{\mathsf{\partial }\beta }}$ and ${\displaystyle \frac{{\mathsf{\partial }}^{2}l}{\mathsf{\partial }{\beta }^2}}$ denote the gradient and Hessian of the unpenalized log-likelihood, respectively, and ${\displaystyle \frac{p_{\lambda }^{\prime }(\mid \beta \mid )}{\left|\beta \right|}}$ is the derivative of the double SCAD penalty.

(1)

The Relationship Between Algorithm Convergence and Matrix Condition Number

The convergence of the DSCAD-LQA mainly depends on the positive definiteness and the condition number $\kappa (\cdot )$ of the modified Hessian matrix:

$${\displaystyle \frac{{\mathsf{\partial }}^{2}l\left(\beta ,{\alpha }^s\right)}{\mathsf{\partial }{\beta }^2}}+{\displaystyle \frac{p_{\lambda }^{\prime }\left(\left|\beta \right|\right)}{\left|\beta \right|}}.$$

The condition number of the modified Hessian matrix is defined as the ratio of its maximum eigenvalue to its minimum eigenvalue, i.e.,

$$\kappa ({\displaystyle \frac{{\mathsf{\partial }}^{2}l\left(\beta ,{\alpha }^s\right)}{\mathsf{\partial }{\beta }^2}}+{\displaystyle \frac{p_{\lambda }^{\prime }\left(\left|\beta \right|\right)}{\left|\beta \right|}})={\displaystyle \frac{{\lambda }_{max}}{{\lambda }_{min}}}.$$

Its core influence laws are as follows:

When k or $pn$ is small (low-dimensional scenario), the modified Hessian matrix possesses satisfactory positive definiteness, with the condition number $\kappa \in (1,{10}^2)$. The Newton iteration converges rapidly. In this paper, the convergence threshold is set as ${\left|\left|{\beta }^{\left(s+1\right)}-{\beta }^{\left(s\right)}\right|\right|}_2<{10}^{-6}$, which can usually be satisfied within 5–10 iterations in low-dimensional scenarios.

As k or $pn$ increases (high-dimensional scenario), the modified Hessian matrix tends to become nearly singular, with the condition number $\kappa >{10}^3$. Direct matrix inversion in this case leads to oscillatory iterations and slower convergence.

To address this issue, the Tikhonov regularization method is adopted in this paper to modify the Hessian matrix, resulting in the regularized matrix:

$${\displaystyle \frac{{\mathsf{\partial }}^{2}l\left(\beta ,{\alpha }^s\right)}{\mathsf{\partial }{\beta }^2}}+{\displaystyle \frac{p_{\lambda }^{\prime }\left(\left|\beta \right|\right)}{\left|\beta \right|}}+\mu I,$$

where μ is the regularization parameter, and I denotes the identity matrix. This modification can constrain the condition number below ${10}^2$ in high-dimensional scenarios, guaranteeing stable convergence of the iterative procedure. To improve the numerical stability in high-dimensional scenarios, Tikhonov regularization is adopted in this paper with a regularization parameter $\mu ={10}^{-4}$. This value is determined by the L-curve method, which can effectively improve the condition number of the Hessian matrix while ensuring the consistency of the estimates.

(2)

Algorithm Scalability Analysis

The scalability of the algorithm mainly reflects the variation in computational efficiency as the sample size n, the dimension of fixed effects k, and the dimension of random effects $pn$ increase. Based on the core computational steps of the algorithm, including gradient evaluation, construction, and inversion of the modified Hessian matrix, the scalability analysis is presented as follows:

The computational complexity of the gradient vector ${\displaystyle \frac{\mathsf{\partial }l\left(\beta ,{\alpha }^s\right)}{\mathsf{\partial }\beta }}+\left\{{\displaystyle \frac{p_{\lambda }^{,}\left(\left|\beta \right|\right)}{\left|\beta \right|}}\right\}\beta$ is $O(nk+npn)$, which mainly depends on the linear combination of the sample size and variable dimensions. When n or k is doubled, the computational cost grows approximately linearly.

The construction complexity of the modified Hessian matrix is $O({nk}^2+{npn}^2)$, and its inversion complexity is $O(k^3)$, which forms the main computational bottleneck of the algorithm.

For high-dimensional scenarios $(k>100 or pn>500)$, this paper employs the conjugate gradient method in place of direct matrix inversion, reducing the inversion complexity to $O(k^2)$. This strategy significantly improves the scalability of the algorithm for high-dimensional data and guarantees that the algorithm can still finish iterations efficiently even when $k$ increases to 500.

(3)

Computational Complexity and Numerical Stability Guarantees

The overall computational complexity of the DSCAD-LQA is $O(T({nk}^2+{npn}^2))$, where T denotes the number of iterations. In all numerical experiments, the number of iterations T is controlled within the range [5,20], and neither divergence nor premature convergence is observed. This complexity is comparable to that of existing penalized Poisson mixed-effects regression algorithms (e.g., the LASSO-Poisson algorithm). However, the numerical stability and reproducibility are further enhanced by the following two improvements:

The penalty parameter λ is selected via five-fold cross-validation, with the optimal value determined by minimizing the prediction error, thereby avoiding non-convergence caused by inappropriate parameter selection.

The initial values ${\beta }^{\left(0\right)}$ and ${\alpha }^{\left(0\right)}$ are chosen as the Newton-type estimates obtained from the unpenalized log-likelihood Function (6), rather than random initialization. This ensures the consistency and reproducibility of the iterative process, such that different researchers can obtain exactly the same convergent results based on the same dataset.

2.3. The Selection of Penalty Parameters

The selection of penalty parameters is extremely important for models involving penalty functions, as the magnitude of penalty parameters affects the final results of model variable selection and parameter estimation. When $\lambda =0$, the penalty function does not penalize the parameters in the model; however, when $\lambda =\infty$, the parameters may be over-penalized, resulting in all parameters being estimated as zero. Since there are two penalty parameters (${\lambda }_{\beta }$ and ${\lambda }_{\alpha }$) in the double SCAD-penalized mixed-effects Poisson count regression model, and considering the computational complexity of the algorithm, the two penalty parameters are treated uniformly based on the overall likelihood value and selected via the SIC (Schwarz Information Criterion) (1982) [25] or GACV (Generalized Approximate Cross-Validation Criterion) (2006) [26]. The two criteria were defined as follows:

$$SIC\left({\lambda }_{\theta }\right)=Ln\left(S_M/N\right)+{\displaystyle \frac{Ln\left(N\right)}{2N}}\left|M\right|$$

(16)

$$GACV\left({\lambda }_{\theta }\right)={\displaystyle \frac{S_M}{N-\left|M\right|}}$$

(17)

where $\theta$ is the parameter $\beta$ or $\alpha$, $S_M={\sum }_{\mathrm{i}=1}^{\mathrm{n}}{\sum }_{j=1}^m\left[-y_{ij}\left(x_{ij}^T\beta +z_{ij}^T{\alpha }_i\right)+e^{x_{ij}^T\beta +z_{ij}^T{\alpha }_i}\right]$ is the value after updating the parameters and substituting them into the maximum likelihood function without penalty and minimizing them. $N=nm$ is the total number of samples, $\left|M\right|$ is the number of fixed-effect regression coefficient vectors $\beta$ or random effect regression coefficient vectors $\alpha$ which is not equal to zero. $\lambda$, which minimizes the values of SIC and GACV, is the value of the optimal penalty parameter.

Remark on Joint Tuning of Penalty Parameters

In this work, we adopt a joint tuning strategy for the fixed-effect penalty parameter λβ and the random-effect penalty parameter ${\lambda }_{\alpha }$, i.e., setting ${\lambda }_{\beta }={\lambda }_{\alpha }=\lambda$ and selecting the optimal λ via the SIC or GACV criterion. This choice is theoretically justified from the following perspectives:

(1)

Consistency of Model Complexity

The SIC and GACV criteria employed in this paper are constructed based on the total number of non-zero coefficients $\left|M\right|$ to measure model complexity. Joint tuning ensures that $\left|M\right|$ consistently represents the total number of non-zero fixed-effect coefficients ${\beta }_l$ and non-zero random-effect coefficients ${\alpha }_{it}$, which is fully consistent with the definition of model degrees of freedom in Equations (16) and (17). Separate tuning of ${\lambda }_{\beta }$ an ${\lambda }_{\alpha }$ d would introduce ambiguity in the definition of $\left|M\right|$ (requiring separate calculation of degrees of freedom for fixed and random effects), thereby compromising the statistical consistency of the parameter selection procedure.

(2)

Convexity and Convergence Guarantee

When ${\lambda }_{\beta }={\lambda }_{\alpha }=\lambda$, the double SCAD penalty function $p_{\lambda }(\mid \cdot \mid )$ (Equation (9)) maintains a uniform derivative structure for both the fixed effects β and random effects α. This ensures that the overall objective function (Equation (7)) is convex (or pseudo-convex) over the parameter space, preventing the DSCAD-LQA (Equations (14) and (15)) from being trapped in local optima and guaranteeing convergence to a global solution.

(3)

Computational Feasibility in High Dimensions

In high-dimensional scenarios where the dimension of fixed effects k or the dimension of random effects $pn$ is large, separate tuning of ${\lambda }_{\beta }$ and ${\lambda }_{\alpha }$ would lead to a two-dimensional grid search, significantly increasing the computational complexity from $O(L)$ to $O(L^2)$, where L is the number of candidate parameters. Joint tuning reduces the parameter space to one dimension, making the selection procedure computationally feasible while maintaining satisfactory estimation accuracy.

(4)

Balanced Estimation and Stability

A unified penalty strength λ ensures a balanced shrinkage effect on both the fixed effects β and random effects α, avoiding estimation bias caused by over-penalizing one component while under-penalizing the other. Additionally, the uniform penalty structure contributes to the numerical stability of the modified Hessian matrix (e.g., the terms ${\displaystyle \frac{{\mathsf{\partial }}^{2}l\left(\beta ,{\alpha }^s\right)}{\mathsf{\partial }{\beta }_l^2}}+p_{\lambda }^{\prime }\left(\left|\beta \right|\right)/\left|\beta \right|$ in Equations (14) and (15), reducing the risk of ill-conditioning and improving the robustness of the Newton iteration.

3. Theoretical Property Proof

Assuming that the parameters $\left(\beta ,{\alpha }_i\right)$ have true value $\left({\beta }_0,{\alpha }_{0i}\right)$, the objective function is as follows:

$$Q\left(\beta ,\alpha \right)=L\left(\beta ,\alpha \right)-mn{\displaystyle \sum _{l=1}^k{p}_{{\lambda }_{\beta }}\left(\left|{\beta }_l\right|\right)}-m{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{t=1}^p{p}_{{\lambda }_{\alpha }}\left(\left|{\alpha }_{it}\right|\right)}}$$

(18)

where $\beta ={\left({\beta }_1,{\beta }_2,\dots ,{\beta }_k\right)}^{\mathrm{T}}$, ${\beta }_0={\left({\beta }_{01},{\beta }_{02},\dots ,{\beta }_{0k}\right)}^{\mathrm{T}}$, ${\alpha }_i=\left({\alpha }_{i1},{\alpha }_{i2},\dots ,{\alpha }_{ip}\right)$, ${\alpha }_{0i}={\left({\alpha }_{0i1},{\alpha }_{0i2},\dots ,{\alpha }_{0ip}\right)}^T$, Let $\beta ={\left({\beta }_1^{\mathrm{T}},{\beta }_2^{\mathrm{T}}\right)}^{\mathrm{T}}$, ${\beta }_0={\left({\beta }_{10}^{\mathrm{T}},{\beta }_{20}^{\mathrm{T}}\right)}^{\mathrm{T}}$, ${\alpha }_i={\left({\alpha }_{1i}^{\mathrm{T}},{\alpha }_{2i}^{\mathrm{T}}\right)}^{\mathrm{T}}$, ${\alpha }_{0i}={\left({\alpha }_{10i}^T,{\alpha }_{20i}^T\right)}^T\stackrel{record}{=}{\left({\alpha }_{10}^T,{\alpha }_{20}^T\right)}^T$, where $\beta$ is a $k$ dimensional vector, ${\beta }_1$ is a $s_{\beta }$ dimensional vector, ${\beta }_2$ is a $k-s_{\beta }$ dimensional vector, ${\beta }_{10}$ is a $s_{\beta }$ dimensional non-zero vector, ${\beta }_{20}$ is a $k-s_{\beta }$ dimensional zero vector, ${\alpha }_i$ is a $p$ dimensional vector, ${\alpha }_{1i}$ is a $s_{\alpha }$ dimensional vector, ${\alpha }_{2i}$ is a $k-s_{\alpha }$ dimensional vector, ${\alpha }_{10}$ is a $s_{\alpha }$ dimensional non-zero vector, ${\alpha }_{20}$ is a $k-p_{\alpha }$ dimensional zero vector. The exponential family probability density function is as follows:

$$f\left(y,\theta ,\varphi \right)=\exp \left\{{\displaystyle \frac{y\theta -b\left(\theta \right)}{a\left(\varphi \right)}}+c\left(y,\varphi \right)\right\}$$

(19)

For the mixed effects Poisson regression model, note that the probability density function is as follows:

$$\begin{gathered} \mathrm{f}\left(\mathrm{y},\mathrm{x},\mathrm{z},\mathsf{\beta },\mathsf{\alpha }\right)\stackrel{\mathrm{record}}{=}\mathrm{f}\left(\mathrm{V},\mathsf{\beta },\mathsf{\alpha }\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left\{\mathrm{y}\left({\mathrm{x}}^{\mathrm{T}}\mathsf{\beta }+{\mathrm{z}}^{\mathrm{T}}{\mathsf{\alpha }}_{\mathrm{i}}\right)-\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{x}}^{\mathrm{T}}\mathsf{\beta }+{\mathrm{z}}^{\mathrm{T}}{\mathsf{\alpha }}_{\mathrm{i}}\right)- \\ \mathrm{l}\mathrm{o}\mathrm{g}\left(\mathrm{y}!\right)\right\} \end{gathered}$$

(20)

The following conditions are given:

Condition 1: The derivative of $f\left(V,\beta ,\alpha \right)$ is defined as follows:

$$E\left[{\displaystyle \frac{\mathsf{\partial }log f\left(V,\beta ,\alpha \right)}{\mathsf{\partial }{\beta }_l}}\right]=0\begin{array}{cc}& l=1,\dots ,k\end{array}$$

$$E\left[{\displaystyle \frac{\mathsf{\partial }log f\left(V,\beta ,\alpha \right)}{\mathsf{\partial }{\alpha }_{it}}}\right]=0\begin{array}{cc}& t=1,\dots ,p\end{array}$$

$$I_{jl}^{\beta }\left(\beta ,\alpha \right)=E\left[{\displaystyle \frac{\mathsf{\partial }log f\left(V,\beta ,\alpha \right)}{\mathsf{\partial }{\beta }_j}}{\displaystyle \frac{\mathsf{\partial }log f\left(V,\beta ,\alpha \right)}{\mathsf{\partial }{\beta }_l}}\right]=E\left[-{\displaystyle \frac{{\mathsf{\partial }}^{2}log f\left(V,\beta ,\alpha \right)}{\mathsf{\partial }{\beta }_j\mathsf{\partial }{\beta }_l}}\right]$$

$$I_{tl}^{\alpha }\left(\beta ,\alpha \right)=E\left[{\displaystyle \frac{\mathsf{\partial }log f\left(V,\beta ,\alpha \right)}{\mathsf{\partial }{\alpha }_{it}}}{\displaystyle \frac{\mathsf{\partial }log f\left(V,\beta ,\alpha \right)}{\mathsf{\partial }{\alpha }_{il}}}\right]=E\left[-{\displaystyle \frac{{\mathsf{\partial }}^{2}log f\left(V,\beta ,\alpha \right)}{\mathsf{\partial }{\alpha }_{it}\mathsf{\partial }{\alpha }_{il}}}\right]$$

Condition 2: The Fisher information matrix $I\left(\beta ,\alpha \right)$ is finite and positive definite at the true parameter values $({\beta }_0,{\alpha }_{0i})$. We denote its block submatrices as follows:

Let $I_{11}\left(\beta ,\alpha \right)=E\left(-{\displaystyle \frac{{\mathsf{\partial }}^2\log f\left(V,\beta ,\alpha \right)}{{\mathsf{\partial }}^2\beta }}\right)$, $I_{22}\left(\beta ,\alpha \right)=E\left(-{\displaystyle \frac{{\mathsf{\partial }}^2\log f\left(V,\beta ,\alpha \right)}{{\mathsf{\partial }}^2{\alpha }_i}}\right)$. When evaluated at the true parameters, these submatrices are denoted $I_{11}\left({\beta }_0,{\alpha }_{0i}\right)$ and $I_{22}\left({\beta }_0,{\alpha }_{0i}\right)$, and the full Fisher information matrix is $I\left({\beta }_0,{\alpha }_{0i}\right)$.

Condition 3: For an open interval $\omega$ containing the truth values of the parameters $\left({\beta }_0,{\alpha }_{0i}\right)$, there exists $M$ so that the third-order derivative is bounded, that is

$$\begin{array}{l}\left|{\displaystyle \frac{{\mathsf{\partial }}^3\log f\left(V,\beta ,\alpha \right)}{\mathsf{\partial }{\beta }_j\mathsf{\partial }{\beta }_l\mathsf{\partial }{\beta }_r}}\right|\le M\\ \left|{\displaystyle \frac{{\mathsf{\partial }}^3\log f\left(V,\beta ,\alpha \right)}{\mathsf{\partial }{\alpha }_{it}\mathsf{\partial }{\alpha }_{il}\mathsf{\partial }{\alpha }_{ir}}}\right|\le M\end{array} \left(\beta ,\alpha \right)\in \omega$$

Condition 4: ${\mathit{liminf}}_{\begin{array}{c}m\to \infty \\ n\to \infty \end{array}}{\mathit{liminf}}_{\theta \to 0^{+}}{\lambda }_{\theta }^{-1}{p}_{{\lambda }_{\theta }}^{\prime }\left(\theta \right)>0$, where $\theta$ is the parameter.

Theorem 1 (Compatibility).

Under conditions 1–4, if

$\mathit{max}\left\{\left|p_{{\lambda }_{\beta }}^{″}\left(\left|{\beta }_{0l}\right|\right)\right|:{\beta }_{0l}\ne 0\right\}\to 0$, $\mathit{max}\left\{\left|p_{{\lambda }_{\alpha }}^{″}\left(\left|{\alpha }_{0it}\right|\right)\right|:{\alpha }_{0it}\ne 0\right\}\to 0$, then there exists a local maximum  $\left(\stackrel{⌢}{\beta },{\stackrel{⌢}{\alpha }}_i\right)$ of Formula (21) such that

$$\begin{array}{l}‖\stackrel{⌢}{\beta }-{\beta }_0‖=O_p\left({\left(mn\right)}^{-{\displaystyle \frac{1}{2}}}+a_{mn}^{\prime }\right)\\ ‖{\stackrel{⌢}{\alpha }}_i-{\alpha }_{0i}‖=O_p\left(m^{-{\displaystyle \frac{1}{2}}}+b_m^{\prime }\right)\end{array}$$

(21)

where  $a_{mn}^{\prime }=\mathit{max}\left\{p_{{\lambda }_{\beta }}^{\prime }\left(\left|{\beta }_{0l}\right|\right):{\beta }_{0l}\ne 0\right\}$, $b_m^{\prime }=\mathit{max}\left\{p_{{\lambda }_{\alpha }}^{\prime }\left(\left|{\alpha }_{0it}\right|\right):{\alpha }_{0it}\ne 0\right\}$, when  $m\to \infty ,n\to \infty$:

$$\begin{array}{l}‖\stackrel{⌢}{\beta }-{\beta }_0‖=O_p\left({\left(mn\right)}^{-{\displaystyle \frac{1}{2}}}\right)\\ ‖{\stackrel{⌢}{\alpha }}_i-{\alpha }_{0i}‖=O_p\left(m^{-{\displaystyle \frac{1}{2}}}\right)\end{array}$$

(22)

Proof. 

Let $a_{mn}={\left(mn\right)}^{-{\displaystyle \frac{1}{2}}}+a_{mn}^{\prime }$, $b_m=m^{-{\displaystyle \frac{1}{2}}}+b_m^{\prime }$. For any given $\epsilon >0$ and the existence of a sufficiently large constant $C$, there is the following:

$$P\left\{\underset{‖u‖=C,‖v_i‖=C}{\sup }Q\left({\beta }_0+a_{mn}u,{\alpha }_{0i}+b_m{v}_i\right)<Q\left({\beta }_0,{\alpha }_{0i}\right)\right\}\ge 1-\epsilon$$

(23)

where $u$ is a $k\times 1$ vector, $v_i$ is a $p\times 1$ vector. According to Equation (23), we know that there exists a local maximum point $\left(\widehat{\beta },{\widehat{\alpha }}_i\right)$ within the ball $\left\{{\beta }_0+a_{mn}u,{\alpha }_{0i}+b_m{v}_i,‖u‖\le C, ‖v_i‖\le C\right\}$ when the probability is close to one, so that $‖\stackrel{̑}{\beta }-{\beta }_0‖=O_p\left(a_{mn}\right)$, $‖{\stackrel{̑}{\alpha }}_i-{\alpha }_{0i}‖=O_p\left(b_m\right)$.

Due to $p_{{\lambda }_{\theta }}\left(0\right)=0$, there is the following:

$$\begin{array}{l}D\left(u,v\right)\equiv Q\left({\beta }_0+a_{mn}u,{\alpha }_{0i}+b_m{v}_i\right)-Q\left({\beta }_0,{\alpha }_{0i}\right)\\ \le L\left({\beta }_0+a_{mn}u,{\alpha }_{0i}+b_m{v}_i\right)-L\left({\beta }_0,{\alpha }_{0i}\right)-mn{\displaystyle \sum _{l=1}^{s_{\beta }}\left\{p_{{\lambda }_{\beta }}\left(\left|{\beta }_{0l}+a_{mn}{u}_l\right|\right)-p_{{\lambda }_{\beta }}\left(\left|{\beta }_{0l}\right|\right)\right\}}\hfill \\ \begin{array}{c}-m{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{t=1}^{s_{\alpha }}\left\{p_{{\lambda }_{\alpha }}\left(\left|{\alpha }_{0it}+b_m{v}_{it}\right|\right)-p_{{\lambda }_{\alpha }}\left(\left|{\alpha }_{0it}\right|\right)\right\}}}\end{array}\end{array}$$

(24)

Subjecting it to a second-order Taylor expansion, there is

$$\begin{array}{l}D\left(u,v\right)\le L\left({\beta }_0,{\alpha }_{0i}\right)+a_{mn}{L}_{\beta }^{\prime }\left({\beta }_0,{\alpha }_{0i}\right)u+b_m{L}_{\alpha }^{\prime }\left({\beta }_0,{\alpha }_{0i}\right)v_i\\ +[{\displaystyle \frac{1}{2}}{u}^{\mathrm{T}}{L}_{\beta }^{″}\left({\beta }_0,{\alpha }_{0i}\right)u{a}_{mn}^2+{\displaystyle \frac{1}{2}}{v}_i^{\mathrm{T}}{L}_{\alpha }^{″}\left({\beta }_0,{\alpha }_{0i}\right)v{b}_m^2+u^{\mathrm{T}}{L}_{\beta \alpha }^{″}\left({\beta }_0,{\alpha }_{0i}\right)v_i{a}_{mn}{b}_m]\left\{1+O\left(1\right)\right\}\\ -L\left({\beta }_0,{\alpha }_{0i}\right)-{\displaystyle \sum _{l=1}^{s_{\beta }}\left[mn{a}_{mn}{p}_{{\lambda }_{\beta }}^{\prime }\left(\left|{\beta }_{0l}\right|\right)\mathrm{sgn}\left({\beta }_{0l}\right)u_l+mn{a}_{mn}^2{p}_{{\lambda }_{\beta }}^{″}\left(\left|{\beta }_{0l}\right|\right)u_l^2\left\{1+O\left(1\right)\right\}\right]}\\ -{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{t=1}^{s_{\alpha }}\left[m{b}_m{p}_{{\lambda }_{\alpha }}^{\prime }\left(\left|{\alpha }_{0it}\right|\right)\mathrm{sgn}\left({\alpha }_{0it}\right)v_{it}+m{b}_m^2{p}_{{\lambda }_{\alpha }}^{″}\left(\left|{\alpha }_{0it}\right|\right)v_{it}^2\left\{1+O\left(1\right)\right\}\right]}}\\ =T_1+T_2+T_3+T_4+T_5\end{array}$$

(25)

where $L_{\beta }^{\prime }\left({\beta }_0,{\alpha }_{0i}\right)$ is the first-order partial derivative of $L\left({\beta }_0,{\alpha }_{0i}\right)$ with respect to $L\left({\beta }_0,{\alpha }_{0i}\right)$, $L_{\alpha }^{\prime }\left({\beta }_0,{\alpha }_{0i}\right)$ is the first-order partial derivative of $\alpha$, $L_{\beta }^{″}\left({\beta }_0,{\alpha }_{0i}\right)$ is the second-order partial derivative of $\beta$, $L_{\alpha }^{″}\left({\beta }_0,{\alpha }_{0i}\right)$ is the second-order partial derivative of $\alpha$, $L_{\beta \alpha }^{″}\left({\beta }_0,{\alpha }_{0i}\right)$ is the second-order mixed partial derivative.

For $T_1$, $T_2$, there are $L_{\beta }^{\prime }\left({\beta }_0,{\alpha }_{0i}\right)=O_p\left(\sqrt{mn}\right)$ and $L_{\alpha }^{\prime }\left({\beta }_0,{\alpha }_{0i}\right)=O_p\left(\sqrt{m}\right)$ under condition 1 and condition 2, known that $O_p\left(\sqrt{mn}{a}_{mn}\right)=O_p\left(mn{a}_{mn}^2\right)$, $O_p\left(\sqrt{m}{b}_m\right)=O_p\left(m{b}_m^2\right)$. According to the Cauchy–Schwarz inequality, there are

$$T_2=b_m{L}_{\alpha }^{\prime }\left({\beta }_0,{\alpha }_{0i}\right)v_i\le b_m‖L_{\alpha }^{\prime }\left({\beta }_0,{\alpha }_{0i}\right)‖‖v_i‖=b_m\sqrt{m}‖v_i‖=O_p\left(m{b}_m^2\right)‖v_i‖$$

(26)

$$T_1=a_{mn}{L}_{\beta }^{\prime }\left({\beta }_0,{\alpha }_{0i}\right)u\le a_{mn}‖L_{\beta }^{\prime }\left({\beta }_0,{\alpha }_{0i}\right)‖‖u‖=a_{mn}\sqrt{mn}‖u‖=O_p\left(mn{a}_{mn}^2\right)‖u‖$$

(27)

For $T_3$,

$$\begin{array}{l}{T}_3=[{\displaystyle \frac{1}{2}}{u}^{\mathrm{T}}{L}_{\beta }^{″}\left({\beta }_0,{\alpha }_{0i}\right)u{a}_{mn}^2+{\displaystyle \frac{1}{2}}{v}_i^{\mathrm{T}}{L}_{\alpha }^{″}\left({\beta }_0,{\alpha }_{0i}\right)v{b}_m^2+u^{\mathrm{T}}{L}_{\beta \alpha }^{″}\left({\beta }_0,{\alpha }_{0i}\right)v_i{a}_{mn}{b}_m]\left\{1+O\left(1\right)\right\}\\ =-{\displaystyle \frac{1}{2}}{w}^{\mathrm{T}}I\left({\beta }_0,{\alpha }_{0i}\right)w\left\{1+O\left(1\right)\right\}\end{array}$$

(28)

where $w^{\mathrm{T}}=\left(a_{mn}{u}^{\mathrm{T}},b_m{v}_i^{\mathrm{T}}\right)$.

For $T_4$, $T_5$, due to $\max \left\{\left|p_{{\lambda }_{\beta }}^{\prime \prime }\left(\left|{\beta }_{0l}\right|\right)\right|:{\beta }_{0l}\ne 0\right\}\to 0$, $\max \left\{\left|p_{{\lambda }_{\alpha }}^{\prime \prime }\left(\left|{\alpha }_{0it}\right|\right)\right|:{\alpha }_{0it}\ne 0\right\}\to 0$, $a_{mn}^{\prime }=\max \left\{p_{{\lambda }_{\beta }}^{\prime }\left(\left|{\beta }_{0l}\right|\right):{\beta }_{0l}\ne 0\right\}$, $b_m^{\prime }=\max \left\{p_{{\lambda }_{\alpha }}^{\prime }\left(\left|{\alpha }_{0it}\right|\right):{\alpha }_{0it}\ne 0\right\}$, there are

$$\begin{array}{l}{T}_4=-{\displaystyle \sum _{l=1}^{s_{\beta }}\left[mn{a}_{mn}{p}_{{\lambda }_{\beta }}^{\prime }\left(\left|{\beta }_{0l}\right|\right)\mathrm{sgn}\left({\beta }_{0l}\right)u_l+mn{a}_{mn}^2{p}_{{\lambda }_{\beta }}^{″}\left(\left|{\beta }_{0l}\right|\right)u_l^2\left\{1+O\left(1\right)\right\}\right]}\\ \begin{array}{c}\le -\left[\sqrt{s_{\beta }}mn{a}_{mn}{a}_{mn}^{\prime }‖u‖+mn{a}_{mn}^2\max \left\{p_{{\lambda }_{\beta }}^{\prime }\left(\left|{\beta }_{0l}\right|\right):{\beta }_{0l}\ne 0\right\}{‖u‖}^2\right]\end{array}\\ =-\sqrt{s_{\beta }}mn{a}_{mn}{a}_{mn}^{\prime }‖u‖\end{array}$$

(29)

$$\begin{array}{l}{T}_5=-{\displaystyle \sum _{i=1}^n{\displaystyle \sum _{t=1}^{s_{\alpha }}\left[m{b}_m{p}_{{\lambda }_{\alpha }}^{\prime }\left(\left|{\alpha }_{0it}\right|\right)\mathrm{sgn}\left({\alpha }_{0it}\right)v_{it}+m{b}_m^2{p}_{{\lambda }_{\alpha }}^{″}\left(\left|{\alpha }_{0it}\right|\right)v_{it}^2\left\{1+O\left(1\right)\right\}\right]}}\\ \begin{array}{c}\le -\left[\sqrt{n{s}_{\alpha }}m{b}_m{b}_m^{\prime }‖v_i‖+m{b}_m^2\max \left\{p_{{\lambda }_{\alpha }}^{\prime }\left(\left|{\alpha }_{0it}\right|\right):{\alpha }_{0it}\ne 0\right\}{‖v_i‖}^2\right]\end{array}\\ =-\sqrt{n{s}_{\alpha }}m{b}_m{b}_m^{\prime }‖v_i‖\end{array}$$

(30)

When a sufficiently large $C$ is chosen, the value of $T_3$ is much larger than the value of $T_1$, $T_2$, $T_4$, $T_5$ under the conditions of $‖u‖=C$, $‖v_i‖=C$. That is, the sign of Equation (25) is completely determined by $T_3$, and the theorem has been proved. □

Theorem 2

(Nature of Oracle). For parameter $\beta ={\left({\beta }_1^{\mathrm{T}},{\beta }_2^{\mathrm{T}}\right)}^{\mathrm{T}}$, ${\alpha }_i={\left({\alpha }_{1i}^{\mathrm{T}},{\alpha }_{2i}^{\mathrm{T}}\right)}^{\mathrm{T}}$, we know by Theorem 1 that there exist local maximum points $\widehat{\beta }={\left({\widehat{\beta }}_1^{\mathrm{T}},{\widehat{\beta }}_2^{\mathrm{T}}\right)}^{\mathrm{T}}$, ${\widehat{\alpha }}_i={\left({\widehat{\alpha }}_{1i}^{\mathrm{T}},{\widehat{\alpha }}_{2i}^{\mathrm{T}}\right)}^{\mathrm{T}}$. Assuming that ${{\left(mn\right)}^{-{\displaystyle \frac{1}{2}}}/\lambda }_{\beta }\to 0$, $m^{-{\displaystyle \frac{1}{2}}}/{\lambda }_{\alpha }\to 0$, and providing that the conditions all hold, we have

(a) 

Sparsity: ${\widehat{\beta }}_2=0,{\widehat{\alpha }}_{2i}=0$, $m\to \infty ,n\to \infty$

(b) 

Asymptotical normality:

$$\sqrt{mn}\left(I_{11}\left({\beta }_{10},{\alpha }_{10}\right)+{\sum }_{11}\right)\left\{{\widehat{\beta }}_1-{\beta }_{10}+{\left(I_{11}\left({\beta }_{10},{\alpha }_{10}\right)+{\sum }_{11}\right)}^{-1}{b}_1\right\}\to N\left\{0,I_{11}\left({\beta }_{10},{\alpha }_{10}\right)\right\}$$

$$\sqrt{m}\left(I_{22}\left({\beta }_{10},{\alpha }_{10}\right)+{\sum }_{22}\right)\left\{{\widehat{\alpha }}_{1i}-{\alpha }_{10}+{\left(I_{22}\left({\beta }_{10},{\alpha }_{10}\right)+{\sum }_{22}\right)}^{-1}{b}_2\right\}\to N\left\{0,I_{22}\left({\beta }_{10},{\alpha }_{10}\right)\right\}$$

where $I_{11}\left({\beta }_{10},{\alpha }_{10}\right)$ is a $s_{\beta }\times s_{\beta }$ matrix, $I_{22}\left({\beta }_{10},{\alpha }_{10}\right)$ is a $s_{\alpha }\times s_{\alpha }$ matrix, $I_{11}\left({\beta }_{10},{\alpha }_{10}\right)$, $I_{22}\left({\beta }_{10},{\alpha }_{10}\right)$ are submatrices of $I\left({\beta }_0,{\alpha }_{0i}\right)$. For ${\sum }_{11}$ , ${\sum }_{22}$ , $b_1$ , $b_2$ the definitions are as follows:

$${\sum }_{11}=diag\left\{p_{{\lambda }_{\beta }}^{″}\left(\left|{\beta }_{01}\right|\right),\dots ,p_{{\lambda }_{\beta }}^{″}\left(\left|{\beta }_{0s_{\beta }}\right|\right)\right\}$$

$${\sum }_{22}=diag\left\{p_{{\lambda }_{\alpha }}^{″}\left(\left|{\alpha }_{0i1}\right|\right),\dots ,p_{{\lambda }_{\alpha }}^{″}\left(\left|{\alpha }_{0i{s}_{\alpha }}\right|\right)\right\}$$

$$b_1={\left(p_{{\lambda }_{\beta }}^{\prime }\left(\left|{\beta }_{01}\right|\right)\mathrm{sgn}\left({\beta }_{01}\right),\dots ,p_{{\lambda }_{\beta }}^{\prime }\left(\left|{\beta }_{0s_{\beta }}\right|\right)\mathrm{sgn}\left({\beta }_{0s_{\beta }}\right)\right)}^{\mathrm{T}}$$

$$b_2={\left(p_{{\lambda }_{\alpha }}^{\prime }\left(\left|{\alpha }_{0i1}\right|\right)\mathrm{sgn}\left({\alpha }_{0i1}\right),\dots ,p_{{\lambda }_{\alpha }}^{\prime }\left(\left|{\alpha }_{0i{s}_{\alpha }}\right|\right)\mathrm{sgn}\left({\alpha }_{0i{s}_{\alpha }}\right)\right)}^{\mathrm{T}}$$

Proof. 

For the proof of sparsity, let ${\epsilon }_{mn}=C{\left(mn\right)}^{-{\displaystyle \frac{1}{2}}}$, ${\epsilon }_m=C{m}^{-{\displaystyle \frac{1}{2}}}$; it suffices to show that when $m\to \infty ,n\to \infty$ is satisfied, $‖{\widehat{\beta }}_1-{\beta }_{10}‖=O_p\left({\left(mn\right)}^{-{\displaystyle \frac{1}{2}}}\right)$, $‖{\widehat{\alpha }}_{1i}-{\alpha }_{10}‖=O_p\left(m^{-{\displaystyle \frac{1}{2}}}\right)$, with probability tending to one for any given ${\beta }_1,{\alpha }_{1i}$:

$$\left\{\begin{array}{c}{\displaystyle \frac{\mathsf{\partial }Q\left(\beta ,\alpha \right)}{\mathsf{\partial }{\beta }_j}}<0,0<{\beta }_j<{\epsilon }_{mn}\\ {\displaystyle \frac{\mathsf{\partial }Q\left(\beta ,\alpha \right)}{\mathsf{\partial }{\beta }_j}}>0,-{\epsilon }_{mn}<{\beta }_j<0\end{array}\right.$$

(31)

$$\left\{\begin{array}{c}{\displaystyle \frac{\mathsf{\partial }Q\left(\beta ,\alpha \right)}{\mathsf{\partial }{\alpha }_{it}}}<0,0<{\alpha }_{it}<{\epsilon }_m\\ {\displaystyle \frac{\mathsf{\partial }Q\left(\beta ,\alpha \right)}{\mathsf{\partial }{\alpha }_{it}}}>0,-{\epsilon }_m<{\alpha }_{it}<0\end{array}\right.$$

(32)

where $j=s_{\beta }+1,\dots ,k,t=s_{\alpha }+1,\dots ,p$. For $‖{\beta }_2‖\le C{\left(mn\right)}^{-{\displaystyle \frac{1}{2}}}$, $‖{\alpha }_{2i}‖\le C{m}^{-{\displaystyle \frac{1}{2}}}$, applying Taylor’s formula yields the following:

The score function of the objective $Q(\beta ,\alpha )$ is given by

$$\left(\begin{array}{c}{\displaystyle \frac{\mathsf{\partial }Q\left(\beta ,\alpha \right)}{\mathsf{\partial }{\beta }_j}}\\ {\displaystyle \frac{\mathsf{\partial }Q\left(\beta ,\alpha \right)}{\mathsf{\partial }{\alpha }_{it}}}\end{array}\right)=\left(\begin{array}{c}{\displaystyle \frac{\mathsf{\partial }L\left(\beta ,\alpha \right)}{\mathsf{\partial }{\beta }_j}}-mn{p}_{{\lambda }_{\beta }}^{\prime }\left(\left|{\beta }_j\right|\right)sgn\left({\beta }_j\right)\\ {\displaystyle \frac{\mathsf{\partial }L\left(\beta ,\alpha \right)}{\mathsf{\partial }{\alpha }_{it}-m{p}_{{\lambda }_{\alpha }}^{\prime }\left(\left|{\alpha }_{it}\right|\right)sgn\left({\alpha }_{it}\right)}}\end{array}\right)$$

Under the given rate conditions, this can be simplified to

$$\left(\begin{array}{c}{\displaystyle \frac{\mathsf{\partial }Q\left(\beta ,\alpha \right)}{\mathsf{\partial }{\beta }_j}}\\ {\displaystyle \frac{\mathsf{\partial }Q\left(\beta ,\alpha \right)}{\mathsf{\partial }{\alpha }_{it}}}\end{array}\right)=\left(\begin{array}{c}mn{\lambda }_{\beta }\left\{-{\lambda }_{\beta }^{-1}{p}_{{\lambda }_{\beta }}^{\prime }\left(\left|{\beta }_j\right|\right)\mathrm{sgn}\left({\beta }_j\right)+O_p\left({\left(mn\right)}^{-{\displaystyle \frac{1}{2}}}/{\lambda }_{\beta }\right)\right\}\\ m{\lambda }_{\alpha }\left\{-{\lambda }_{\alpha }^{-1}{p}_{{\lambda }_{\alpha }}^{\prime }\left(\left|{\alpha }_{it}\right|\right)\mathrm{sgn}\left({\alpha }_{it}\right)+O_p\left(m^{-{\displaystyle \frac{1}{2}}}/{\lambda }_{\alpha }\right)\right\}\end{array}\right)$$

(33)

Given the conditions ${\mathit{liminf}}_{\begin{array}{c}m\to \infty \\ n\to \infty \end{array}}{\mathit{liminf}}_{\theta \to 0^{+}}{\lambda }_{\theta }^{-1}{p}_{{\lambda }_{\theta }}^{\prime }\left(\theta \right)>0$, ${{\displaystyle \frac{{\left(mn\right)}^{-\frac{1}{2}}}{\lambda }}}_{\beta }\to 0$, ${\displaystyle \frac{m^{-\frac{1}{2}}}{{\lambda }_{\alpha }}}\to 0$, it is possible that the sign of $\mathsf{\partial }Q\left(\beta ,\alpha \right)/\mathsf{\partial }{\beta }_j$ is determined by ${\beta }_j$, $\mathsf{\partial }Q\left(\beta ,\alpha \right)/\mathsf{\partial }{\beta }_j$, and the sign of $\mathsf{\partial }Q\left(\beta ,\alpha \right)/\mathsf{\partial }{\alpha }_{it}$ is determined by ${\alpha }_{it}$. Combined with the sign conditions above, this completes the proof of sparsity. For the proof of asymptotic normality, we know from Theorem 1 that there exists a local maximum point $\left(\widehat{\beta },{\widehat{\alpha }}_i\right)$ such that

$${\left.\left(\begin{array}{c}{\displaystyle \frac{\mathsf{\partial }Q\left(\beta ,\alpha \right)}{\mathsf{\partial }{\beta }_j}}\\ {\displaystyle \frac{\mathsf{\partial }Q\left(\beta ,\alpha \right)}{\mathsf{\partial }{\alpha }_{it}}}\end{array}\right)\right|}_{\left(\beta ,\alpha \right)=\left(\begin{array}{cc}{\widehat{\beta }}_1& {\widehat{\alpha }}_{1i}\\ 0& 0\end{array}\right)}=\left(\begin{array}{c}0\\ 0\end{array}\right)$$

By Slutsky’s theorem and the central limit theorem, it follows that

$$\sqrt{mn}\left(I_{11}\left({\beta }_{10},{\alpha }_{10}\right)+{\sum }_{11}\right)\left\{{\widehat{\beta }}_1-{\beta }_{10}+{\left(I_{11}\left({\beta }_{10},{\alpha }_{10}\right)+{\sum }_{11}\right)}^{-1}{b}_1\right\}\to N\left\{0,I_{11}\left({\beta }_{10},{\alpha }_{10}\right)\right\}$$

$$\sqrt{m}\left(I_{22}\left({\beta }_{10},{\alpha }_{10}\right)+{\sum }_{22}\right)\left\{{\widehat{\alpha }}_{1i}-{\alpha }_{10}+{\left(I_{22}\left({\beta }_{10},{\alpha }_{10}\right)+{\sum }_{22}\right)}^{-1}{b}_2\right\}\to N\left\{0,I_{22}\left({\beta }_{10},{\alpha }_{10}\right)\right\}$$

□

Discussion on the Practical Implications of Asymptotic Convergence Results

This section elaborates on the convergence rates, regularity conditions, and oracle property presented in the theorems, clarifying their theoretical significance and practical application value.

1.

Practical Interpretation of Convergence Rates

The asymptotic estimation error results in Theorem 1,

$$\left||\widehat{\beta }-{\beta }_0|\right|=O_p\left({\left(mn\right)}^{-1/2}\right),$$

Which indicates that the fixed-effect estimator $\widehat{\beta }$ achieves the optimal parametric convergence rate of $\sqrt{mn}$.

Here, $mn$ is the total number of observations. This means that the estimation precision of fixed effects improves steadily as the total sample size increases, and the rate is consistent with that of maximum likelihood estimation (MLE) under a fully known model structure, achieving optimal statistical efficiency.

Intuitive Elucidation:

This formula implies that the estimation error of the fixed effect β decreases at a rate of ${(mn)}^{-{\displaystyle \frac{1}{2}}}$ as the total sample size $mn$ increases.

For instance, increasing the total sample size from $mn=100$ to $mn=400$ (i.e., a fourfold increase) would theoretically halve the standard deviation of the estimation error, which is consistent with the efficiency of the classical maximum likelihood estimation (MLE) under a fully known model structure.

This indicates that the most effective approach to improving the estimation precision of the fixed effect is to increase the total sample size $mn$: both an increase in the number of subjects n and the number of observations per subject m can lead to a linear improvement in estimation precision.

For the random effects ${\alpha }_i$, the convergence rate is

$$\left||\widehat{{\alpha }_i}-{\alpha }_{0i}|\right|=O_p\left({\left(mn\right)}^{-1/2}\right),$$

which shows that the estimation accuracy of random effects primarily depends on the number of observations per subject m. This aligns with the hierarchical nature of random effects. In practice, increasing the number of within-subject observations can more effectively improve the estimation precision of random effects.

Intuitive Elucidation:

The estimation precision of the random effects is determined primarily by the number of repeated observations per subject m, rather than the total sample size $mn$. This is because the random effects ${\alpha }_i$ characterize individual-level heterogeneity, which requires a sufficient number of within-subject repeated observations for accurate identification.

For example, increasing the number of observations per subject from $m=5$ to $m=20$ (i.e., a fourfold increase) would theoretically halve the standard deviation of the estimation error for the random effects.

This suggests that in practical applications, if the research focuses on individual-level random effects, priority should be given to increasing the number of observations per subject m, rather than simply increasing the number of subjects n.

2.

Practical Significance of Conditions 1–4

The regularity conditions 1–4 required for the theoretical proofs are standard in generalized mixed-effects models and are generally satisfied in practical applications:

• Condition 1 ensures that the expectation of the score function at the true parameters $({\beta }_0,{\alpha }_{0i})$ is zero, which is a fundamental condition for the consistency of MLE.
• Condition 2 requires the Fisher information matrix $I(\beta ,\alpha )$ to be positive definite at the true parameters, guaranteeing model identifiability and unique estimation.
• Condition 3 bounds the third-order derivatives of the log-likelihood function, which holds for common exponential family models, such as the mixed-effects Poisson counting regression model in this paper.
• Condition 4 restricts the growth rate of the penalty parameters, ensuring that the penalty terms do not dominate the likelihood function, so that variable selection and parameter estimation can be performed effectively simultaneously.

These conditions are not overly restrictive and are typically naturally satisfied in real data analysis.

Sample Size Design for Practical Studies:

• For studies focusing on fixed effects: To achieve a target estimation precision (e.g., a standard error of 0.05 for a key fixed effect), researchers can use the convergence rate $mn$ to back-calculate the required total sample size. For example, if the initial estimate of the standard deviation of the outcome is σ = 1, a total sample size of $mn=400$ would yield a standard error of approximately 1/400 = 0.05.
• For studies focusing on random effects: To accurately estimate individual-level random effects, it is recommended to ensure that each subject has at least $m\ge 20$ repeated observations. This ensures that the estimation error of random effects is sufficiently small to support meaningful individual-level inferences.

3.

Practical Value of the Oracle Property

The oracle property (Theorem 2) proven in this paper has clear practical implications:

• Sparsity: The algorithm can consistently identify the true zero coefficients in the model $({\widehat{\beta }}_2=0, {\widehat{\alpha }}_{2i}=0)$, with probability approaching one, achieving simultaneous sparse selection of both fixed and random effects.
• Practical implication: This means the model can automatically “drop” irrelevant fixed and random effects from the final specification, reducing model complexity and improving interpretability. For example, in a healthcare application, the model can identify which patient characteristics (fixed effects) and which individual-specific deviations (random effects) are truly predictive of the outcome, eliminating noise from the model.
• Asymptotic normality: The estimators of the non-zero coefficients follow an asymptotic normal distribution with the same asymptotic variance as if the true model structure were known in advance.
• Practical implication: This allows researchers to construct valid confidence intervals and perform hypothesis tests for the selected non-zero coefficients as if they knew the true model structure beforehand. This eliminates the need for additional corrections for variable selection, making statistical inference straightforward and reliable in large samples.

4.

Practical Implementation Recommendations

• To ensure stable estimation of fixed effects, the total sample size $mn$ should be increased as much as possible.
• To improve the estimation precision of random effects, the number of within-subject observations m should be appropriately increased.
• The penalty parameter λ is recommended to be selected via SIC or GACV, which automatically satisfies Condition 4 and guarantees the theoretical properties.
• After variable selection, statistical inference can be directly performed based on the asymptotic normality results without additional corrections.

5.

Summary of Practical Takeaways

• Fixed effects precision is driven by the total sample size $mn$. The larger the total sample size $mn$, the more precise the estimates of the fixed effects, with the convergence rate achieving the optimal level of the $\sqrt{mn}$ order.
• Random effects precision is driven by within-subject observations m. The greater the number of repeated observations per subject, the more accurate the estimation of individual heterogeneity.
• Model selection: The oracle property ensures that irrelevant effects are automatically excluded, and the remaining effects can be used for valid statistical inference without extra adjustments.
• Study design: Prioritize $mn$ for fixed effects, prioritize m for random effects, and use information criteria (SIC/GACV) to select penalty parameters $\lambda$.

4. Comparative Study of Monte Carlo Simulation

Monte Carlo simulation data were generated as follows: ${\lambda }_{ij}$ is first obtained through ${\lambda }_{ij}=E\left(y_{ij}|x_{ij},z_{ij}\right)=e^{{\beta }_0+x_{ij}^T\beta +z_{ij}^T{\alpha }_i}$, and then the value of $y_{ij}$ is generated through the Poisson distribution of $P({\lambda }_{ij})$, where $N$ is the total sample size, ${\beta }_0=0$, $x_{ij}^T=\left(x_{ij1},x_{ij2},\dots ,x_{ij8}\right)\left(i=1,2,\dots ,n;j=1,2,\dots ,n_i\right)$. The explanatory variable $X_1,X_2\dots ,X_8$ follows a standard normal distribution, and for any two explanatory variables $X_l$ and $X_k$, the correlation coefficient between them is ${\rho }^{\left|l-k\right|}$. In order to estimate the accuracy of model parameter estimation, the median of mean squared error (MRME) is used as an evaluation metric in this paper. Specifically, it is defined as the median mean squared error (MSE) obtained from multiple simulations. This indicator avoids the effect of maximum values and minimal values in the simulation results, where the mean squared error (MSE) is calculated by the following:

$$MSE={\displaystyle \frac{1}{K}}{\displaystyle \sum _{s=1}^K\frac{{\left({\widehat{\beta }}^s-\beta \right)}^T\left({\widehat{\beta }}^s-\beta \right)}{p}}$$

(34)

where ${\widehat{\beta }}^s$ is the estimated value of $\beta$ at the first $s$ simulation; in the following simulations, the number of simulations $K$ is repeated 100 times.

To provide a more comprehensive assessment of the finite-sample performance of our estimator, we decompose the mean squared error (MSE) into two fundamental components: squared bias and variance. This decomposition allows us to distinguish between the systematic error of the estimator and its variability across simulations.

For a fixed effect coefficient ${\beta }_l$, the bias is defined as the average difference between the estimated value and the true value across all simulations:

$$Bias\left({\widehat{\beta }}_l\right)={\displaystyle \frac{1}{k}}{\sum }_{s=1}^K\left({\widehat{\beta }}_l^s-{\beta }_l\right)$$

(35)

where K = 100 is the number of simulations, ${\widehat{\beta }}_l^s$ is the estimate of ${\beta }_l$ in the $s\text{-}th$ simulation, and ${\beta }_l$ is the true value.

The variance of the estimator ${\widehat{\beta }}_l$ measures its dispersion around its own mean:

$$Var\left({\widehat{\beta }}_l\right)={\displaystyle \frac{1}{K-1}}{\sum }_{s=1}^K{\left({\widehat{\beta }}_l^s-{\overline{\widehat{\beta }}}_l\right)}^2$$

(36)

where ${\overline{\widehat{\beta }}}_l={\displaystyle \frac{1}{k}}{\sum }_{s=1}^K{\widehat{\beta }}_l^s$ is the average estimate of ${\beta }_j$.

The relationship between these quantities is given by the MSE decomposition:

$$MSE\left({\widehat{\beta }}_l\right)={\left[Bias\left({\widehat{\beta }}_l\right)\right]}^2+Var\left({\widehat{\beta }}_l\right)$$

(37)

For the overall model performance, we report the median bias and median variance across all fixed effect coefficients, which are robust to outliers and align with our use of MRME.

To evaluate variable selection performance, we use two key metrics derived from the confusion matrix:

Correct Selection Rate (CSR)—the proportion of truly non-zero coefficients ${\beta }_l\ne 0$ that are correctly identified as non-zero, i.e.,

CSR = TP/total number of true non-zero coefficients.

Correct Elimination Rate (CER)—the proportion of truly zero coefficients ${\beta }_l=0$ that are correctly identified as zero, i.e.,

CER = TN/total number of true zero coefficients.

Here, an estimated coefficient ${\widehat{\beta }}_l$ is treated as zero if $\left|{\widehat{\beta }}_l<{10}^{-6}\right|$. CSR reflects the accuracy of selecting significant variables, while CER reflects the accuracy of excluding irrelevant variables.

1.

Simulation 1—Impact of Multicollinearity

In order to investigate whether the magnitude of the correlation coefficient $\rho$ of the independent variables affects the accuracy of the method estimation and variable selection, we evaluated three cases—$\rho =0.25$, $\rho =0.5$, $\rho =0.75$—as a way to simulate the comparison. When generating the simulation data, let $n=30$, $m=10$; the fixed effect coefficient is assigned to $\beta ={\left(3, -1.5,0, 0, 2, 0, 0, 0\right)}^T$, the random effect coefficient is assigned to ${\alpha }_i={\left({\alpha }_{i0},{\alpha }_{i1},{\alpha }_{i2}\right)}^T~N_3\left(0,D\right)$, $D=diag\left(1,1,0\right)$, $z_{ij}^T=\left(1,x_{ij1},x_{ij2}\right)$. Set the threshold value $\epsilon =10^{-6}$ when stopping the iterations. The double SCAD penalty method proposed in this paper is simulated under two criteria, SIC and GACV, and is noted as DSCAD-SIC and DSCAD-GACV, respectively. The simulation was repeated 100 times, and the specific results obtained from the simulation are shown in

Table 2. Simulation results with different correlation coefficients.

$\mathit{\rho }$	Method	MRME	CSR	CER	Bias	Variance	MSE
0.25	DSCAD-SIC	0.069	1.000	0.876	0.304	2.092	0.160
	DSCAD-GACV	0.060	1.000	0.886	0.164	1.948	0.096
0.5	DSCAD-SIC	0.084	1.000	0.806	0.386	2.229	0.243
	DSCAD-GACV	0.079	1.000	0.844	0.226	2.090	0.164
0.75	DSCAD-SIC	2.726	0.900	0.390	0.542	2.613	0.498
	DSCAD-GACV	2.416	0.900	0.418	0.333	2.265	0.399

.

As can be seen from Table 2, under the specific simulation settings of this study (i.e., $n=30,m=10$, true fixed effects $\beta ={\left(3,-1.5,0,0,2,0,0,0\right)}^T$, and 100 repetitions), the estimates under the GACV criterion consistently yield smaller MRME values, identical CSR values, and higher CER values compared to the SIC across all collinearity levels. This indicates that, within the current experimental conditions, the GACV criterion outperforms the SIC in terms of estimation accuracy and the exclusion of irrelevant variables, while both criteria achieve equally high accuracy in selecting significant variables.

It is observed that the accuracy of parameter estimation and variable selection decreases gradually as the correlation coefficient increases, where the value of MRME becomes larger and larger, while the value of CSR and CER becomes smaller and smaller. It can be obviously seen that, compared to the two cases with correlation coefficients of $\rho =0.25$ and $\rho =0.5$, the value of MRME becomes significantly larger when $\rho =0.75$. The value increases by 38 times and 31 times under the SIC and by 39 times and 29 times under the GACV criterion, respectively. At a correlation coefficient $\rho =0.5$, the value of MRME increased by 0.21 times and 0.32 times in both SIC and GACV criteria, respectively, compared to the case of $\rho =0.25$.

From the perspective of the bias–variance trade-off, as the correlation coefficient ρ increases, the degree of multicollinearity intensifies. Our detailed decomposition shows that both bias and variance rise substantially, although their relative contributions to the estimation error differ across criteria.

Under the GACV criterion, the variance of parameter estimates remains the dominant component of MSE, accounting for approximately 70% of the total error across all ρ levels. The contribution of squared bias is stable at around 30%, indicating that although both components increase, the rise in MRME is still mainly driven by variance inflation, which is consistent with the typical behavior of parameter estimation under high-dimensional collinear settings.

In contrast, under the SIC, the contribution of squared bias to MSE (approximately 60%) exceeds that of variance. This implies that the increase in MRME is a joint result of both bias and variance, with bias playing a more prominent role. Notably, compared with the DSCAD-SIC method, the DSCAD-GACV method consistently achieves lower absolute bias and a more favorable bias–variance balance, especially under high multicollinearity (ρ = 0.75), where it effectively alleviates the systematic error introduced by collinear predictors.

For the CSR (Correct Selection Rate), under both criteria, the CSR is 1.00 when the correlation coefficient is ρ = 0.25 and ρ = 0.5, indicating that all significant variables are correctly included in the model in these cases. In contrast, the CSR drops to 0.90 for both criteria when ρ = 0.75, which indicates that there are cases where not all significant variables are selected in the 100 simulations. This shows that the accuracy of variable selection is affected when the correlation coefficient of the variables is comparatively large. Notably, even at the high correlation level of ρ = 0.75, the CSR remains at a satisfactory level of 0.90, demonstrating the robustness of the proposed method.

For the CER (Correct Elimination Rate), regardless of the criterion, the CER in case ρ = 0.75 is around 0.40, while for both cases ρ = 0.25 and ρ = 0.5, the CER is above 0.80. It can be seen that the proportion of unwanted variables correctly excluded from the model in the ρ = 0.75 case is only about half of that in the other two cases. Moreover, the CER when ρ = 0.5 is also lower than that when ρ = 0.25. Overall, model estimation and variable selection perform better under the GACV criterion, and when the correlation coefficient is comparatively large, the accuracy of parameter estimation and selection becomes worse.

2.

Simulation 2—Impact of Random Effects

In order to investigate whether the size of the random-effect covariance $D$ affects the accuracy of the estimation of the coefficients of the variables by the method proposed in this paper, three random effects are set up, that is, the random effect coefficients ${\alpha }_i={\left({\alpha }_{i0},{\alpha }_{i1},{\alpha }_{i2}\right)}^T~N_3\left(0,D\right)$, $z_{ij}^T=\left(1,x_{ij1},x_{ij2}\right)$, and for the covariance, the following three cases, $D_1=diag\left(1,1,0\right)$, $D_2=diag\left(2,2,0\right)$, $D_3=diag\left(3,3,0\right)$, are taken for simulation, respectively. When simulated data are generated for different situations, controlling that $n=30$, $m=10$, $\rho =0.5$, and the fixed effect coefficient is $\beta ={\left(3,-1.5,0,0,2,0,0,0\right)}^T$, the threshold $\epsilon ={10}^{-6}$ is set when the iteration is stopped. The results obtained from 100 simulations under both SIC and GACV criteria are shown in Figure 1.

From Figure 1, it is known that the value of MRME is increasing, while the values of CSR (Correct Selection Rate) and CER (Correct Elimination Rate) are decreasing as the value of the random-effect covariance increases under either criterion. This indicates that the magnitude of the random-effect covariance has a notable impact on the accuracy of both parameter estimation and variable selection.

The bias–variance analysis reveals that an increase in the random-effect covariance matrix D amplifies both the variance fluctuation and the systematic bias of the parameter estimates. While variance remains the dominant component of the mean squared error (MSE) across all settings, ranging between 2.011 and 2.491, the absolute bias also exhibits a substantial increase. Specifically, as D rises from D1 to D3, the absolute bias increases by 61% (from 0.376 to 0.606) under the SIC, and by 102% (from 0.209 to 0.423) under the GACV criterion. Although the relative increase in absolute bias is more pronounced under GACV than under SIC, the absolute bias associated with GACV remains consistently lower across all scenarios, indicating its superior estimation accuracy. These results demonstrate that the deterioration in estimation accuracy is driven jointly by elevated estimation uncertainty (variance) and a notable increase in systematic bias. This finding is consistent with the theoretical intuition that stronger random effects give rise to greater unobserved heterogeneity, which is reflected not only in higher estimation variability but also in more substantial systematic deviations from the true parameter values.

Comparing with the case where the covariance is $D_1$, when the covariance is $D_2$, the value of MRME increases by 168% and 170% under SIC and GACV criteria, respectively. The CSR value is one when the covariance is $D_1$; the CSR value is not one when the covariance is $D_2$, which indicates that there is a situation where not all the needed variables are included in the model when the covariance is $D_2$. For the CER value, it was reduced by 15.1% and 15.2% compared to the covariance $D_1$ under both criteria.

The MRME line in Figure 1 fluctuates significantly with the value of the covariance $D_3$ increasing by 402% and 361% compared with the covariance $D_1$ for both criteria. The CSR value is less than 1 when the covariance is $D_3$, and the CER value decreases by 33.0% and 33.89% under both criteria compared with the covariance $D_1$.

Comparing with the $D_2$ covariance case, when the $D_3$ covariance resulted in the MRME value increasing by 88%, and the CER value decreasing by 21.1% under the SIC; for the CSR value, the value is less than one in both cases. Under the GACV criterion, the MRME increased by 71% and the CER decreased by 22.1% when the covariance structure shifted from $D_2$ to $D_3$. Again, the CSR value was less than 1.

Within the range of random-effect covariance values considered in this simulation, it is observed that smaller covariance values are associated with better estimation and variable selection performance, while larger covariance values lead to degraded accuracy.

Under the specific simulation conditions of this study (i.e., $n=30,m=10, \rho =0.5$, and random-effect covariance $D\in \{D_1,D_2,D_3\}$), the proposed method under the GACV criterion shows slightly lower MRME and slightly stronger performance in selecting significant and excluding non-significant variables compared to the SIC. This finding is specific to the range of random-effect covariance values considered in this simulation.

3.

Simulation 3—Comparison with other methods

Based on the simulation above, it is clear that in this paper, the method under the GACV criterion is better estimated. The GACV criterion is used in this paper when different methods are used for comparison with the methods involving the selection of penalty parameters. In this paper, we compare the following methods: the Poisson regression model without penalty (denoted as P), the single-penalty mixed-effects Poisson regression model with SCAD penalty for fixed effect only under the GACV criterion (denoted as P-H-GACV), the model with double LASSO penalty under the GACV criterion (denoted as DLASSO-GACV), and the mixed- effects Poisson regression model with double SCAD penalty under the GACV criterion (denoted as DSCAD-GACV). Similarly, in generating the simulated data, let $n=30$, $m=10$, $\rho =0.5$; the random effect coefficient is ${\alpha }_i={\left({\alpha }_{i0},{\alpha }_{i1},{\alpha }_{i2}\right)}^T~N_3\left(0,D\right)$, $D=diag\left(1,1,0\right)$, $z_{ij}^T=\left(1,x_{ij1},x_{ij2}\right)$. For the fixed effect coefficient, consider the following two cases:

Case 1: Sparse model $\beta ={\left(3,-1.5,0,0,2,0,0,0\right)}^T$.

Case 2: Dense model $\beta ={\left(0.85,0.85,0.85,0.85,0.85,0.85,0.85,0.85\right)}^T$.

Setting the threshold value $\epsilon ={10}^{-6}$ when stopping the iteration, simulations of four methods are compared in the above two cases, and the results after repeating the simulations 100 times are shown in

Table 3. Comparison of simulation results between the three methods.

Method	Model	MRME	CSR	CER	Bias	Variance	MSE
DSCAD-GACV	sparse	0.079	1.0	0.844	0.185	0.021	0.055
	dense	0.269	0.787	1.0	0.210	0.035	0.088
DLASSO-GACV	sparse	4.352	1.0	0.2	0.220	0.028	0.074
	dense	26.334	0.889	1.0	0.245	0.042	0.105
P-H-SCAD	sparse	1.906	0.397	0.88	0.255	0.032	0.085
	dense	1.510	0.322	1.0	0.280	0.048	0.120
P	sparse	40.983	1.0	0	0.193	0.082	0.120
	dense	2.219	0.889	1.0	0.215	0.098	0.150

(In case 2, the number of true zero coefficients is 0, which makes the denominator of the CER formula zero. Therefore, the CER is mathematically undefined and is conventionally recorded as 1.0.).

From Table 3, it can be seen that under the sparse model (case 1), the MRME value of the model with the double LASSO penalty (DLASSO-GACV) is 55 times that of the double SCAD method proposed in this paper, indicating that its parameter estimation accuracy is inferior to that of our method. The CSR values of the two methods are the same, which means they have the same effect in selecting important variables. However, the CER value of DLASSO is much lower than that of the double SCAD method, indicating that its ability to exclude insignificant variables is weaker than that of our method.

Compared with the Poisson mixed-effects model with a single SCAD penalty only on the fixed effects (P-H-SCAD), the MRME value obtained from the parameter estimation of our double SCAD penalty model is significantly lower. Specifically, the MRME value of the P-H-SCAD method is 24 times that of the DSCAD-GACV method. Although the CER value of P-H-SCAD is higher, its CSR value for selecting important variables is only half of that of the double SCAD method. Moreover, in model construction, the CSR value is more important than the CER value, which suggests that for the Poisson mixed-effects model, the double SCAD penalty performs significantly better than the single penalty.

For the unpenalized Poisson regression model, not only is its estimation error (MRME) larger, but it also fails to exclude any irrelevant variables.

From the perspective of the bias–variance trade-off, under the simulation settings adopted in this study, the proposed double SCAD penalty method (DSCAD-GACV) achieves a more favorable balance between variable sparsity and estimation stability.

Specifically, in the sparse model, DSCAD-GACV yields the lowest absolute bias (0.185) and variance (0.021), thereby attaining the smallest mean squared error (MSE = 0.055) among all competing methods. In the dense model, this method maintains an absolute bias of 0.210 and a variance of 0.035, both of which are significantly lower than those of the double LASSO (DLASSO-GACV) and single SCAD (P-H-SCAD) methods, demonstrating superior estimation stability.

In contrast, under the same simulation conditions, the double LASSO penalty (DLASSO-GACV) introduces considerably larger estimation bias and variance. For illustration, in the sparse model, the absolute bias of DLASSO-GACV (0.220) is 1.2 times that of DSCAD-GACV, while its variance (0.028) is 1.3 times higher than that of DSCAD-GACV. Although the unpenalized model (P) exhibits relatively small bias (0.193 in the sparse model and 0.215 in the dense model), its excessively large variance (0.082 in the sparse model and 0.098 in the dense model) leads to inadequate stability of the estimation results.

In the dense model, the MRME value of the double LASSO method is much larger than that of the double SCAD method, and the MRME value of the single-penalty method is also higher than that of the DSCAD-GACV method by a factor of 4.6. The CSR value of the double LASSO penalty is 0.889, while that of the single penalty is only 2.9, and that of the double SCAD method is 0.787. It can be known that although the CSR value of double LASSO is high, the double SCAD penalty method in this paper is much higher than that of double LASSO in terms of the accuracy of parameter estimation, and performs better in the dense model than that of the single SCAD penalty. For the non-penalty method, although all important variables are selected, the MRME values are much larger than those of the double SCAD method.

From the above analysis, under the sparse and dense model settings of this study (i.e., $n=30,m=10, \rho =0.5, D=diag(1,1,0),$ and 100 repetitions), the double SCAD penalty method proposed in this paper demonstrates significantly better accuracy in variable estimation and selection compared to the double LASSO and single SCAD penalty models. This conclusion is an empirical finding within the scope of the current simulation design and should not be generalized to all possible modeling scenarios. In the sparse case, the double SCAD method successfully selects all required variables, while in the dense case, some required variables are missed, and the MRME is higher. This suggests that, within the current experimental conditions, the double SCAD method is more suitable for sparse modeling scenarios. Further research is needed to evaluate its performance under different sample sizes, parameter dimensions, and data distributions.

Limitations and Future Research:

The simulation results presented in this chapter are based on a specific set of experimental conditions, including sample size $n=30,m=10$, fixed effect sparsity, and Poisson-distributed outcomes. While these settings are representative of many applied scenarios, the findings should be interpreted as empirical evidence within the scope of this study, rather than universal conclusions about the superiority of the double SCAD penalty method. Future research will extend the simulation design to include larger sample sizes, higher-dimensional parameters, and different outcome distributions to further validate the generalizability of the proposed method.

5. Examples

This paper analyzes data from a cigarette factory using the double SCAD method proposed above. The data were collected from 15 April to 15 June 2020 (no data were recorded on May 1, Labor Day). The factory has three shifts and six machines, with two shifts rotating each morning and evening, and each shift operating all six machines. In total, there are 708 observations.

This study mainly explores the effects of machine parameters on the number of short cigarettes. A short cigarette is one of the most common quality defects during the cigarette rolling process. Cigarettes with such quality defects are discarded, which increases production consumption. Investigating which machine parameters have significant effects can help cigarette factories reduce consumption losses.

The details of the machine parameters are shown in

Table 4. Machine parameter variable description statistics.

Variable	Parameter Name	Range	Median	Missing Amount
$X_1$	Vehicle speed	[0, 5523]	5442	62
$X_2$	Rubbing plate temperature	[0, 175]	160	63
$X_3$	The position of the flat plate	[−8.3, 0.8]	−4.6	61
$X_4$	The position of the compacting end	[−5, 17]	0	62
$X_5$	SE1 soldering iron temperature	[28, 251]	218	61
$X_6$	SE2 soldering iron temperature	[26, 249]	211	62

.

The scatter plot of the number of short cigarettes versus each variable is shown in Figure 2.

From Figure 2, we can observe that the scatter plots of vehicle speed, rubbing plate temperature, SE1 soldering iron temperature, and SE2 soldering iron temperature versus the number of short cigarettes are clustered. By contrast, the scatter plots of compaction end position versus the number of short cigarettes are more scattered, and those of flattening position versus the number of short cigarettes are highly dispersed with no obvious linear pattern.

According to the data characteristics, the machine parameter variables are analyzed by the machine.

Table 5. Description statistics of each machine parameter variable.

Variable	Machine	Range	Mean	Standard Deviation	Missing Amount
vehicle speed	4	[17, 5490]	4709.20	1685.17	5456
	5	[1, 5486]	4736.12	1625.84	5293.5
	6	[0, 5504]	4931.83	1503.39	5492
	7	[337, 5523]	5342.96	555.70	5518
	8	[0, 5491]	4772.19	1661.60	5415
	9	[1, 5484]	4893.66	1556.76	5454.5
rubbing plate temperature	4	[134, 154]	143.87	5.38	143
	5	[154, 175]	164.30	5.56	160
	6	[156, 171]	161.30	3.39	160
	7	[143, 154]	150.42	1.76	151
	8	[156, 167]	160.25	2.29	160
	9	[0, 167]	163.20	16.19	165
the position of the flat plate	4	[−8.3, 0.4]	−3.45	2.70	−2.4
	5	[−8.2, −2.0]	−4.60	1.26	−4.4
	6	[−8.1, 0.8]	−4.06	2.20	−3.1
	7	[−6.9, −1.8]	−4.22	1.17	−3.9
	8	[−8.2, −0.7]	−4.92	1.18	−5.0
	9	[−8.3, −2.0]	−5.54	1.12	−5.6
the position of the compacting end	4	[−4, 9.4]	0.88	2.67	0
	5	[−2, 8.3]	1.43	2.11	1
	6	[−4, 15.6]	0.06	2.89	0
	7	[−4, 5.0]	0.32	1.84	0
	8	[−5, 17.1]	0.65	4.62	0
	9	[−5, 6.6]	−0.99	1.80	−1
SE1 soldering iron temperature	4	[28, 245]	211.75	22.29	212
	5	[188, 251]	226.68	10.60	225
	6	[213, 243]	223.61	7.00	224
	7	[190, 220]	205.70	8.15	205
	8	[199, 231]	222.58	12.57	230
	9	[194, 230]	209.07	9.18	209
SE2 soldering iron temperature	4	[26, 239]	208.68	21.58	211
	5	[186, 249]	224.68	8.88	226
	6	[207, 231]	215.27	4.83	216
	7	[195, 211]	203.10	4.77	203
	8	[199, 211]	208.43	4.19	211
	9	[194, 216]	205.13	5.96	205

presents the descriptive statistics for each machine parameter variable.

As shown in Table 5, there are obvious differences in the distribution of parameters across different machines. For vehicle speed, machine 7 has higher minimum, mean, and median values but a smaller standard deviation, indicating more aggregated data. For the rubbing plate temperature, the ranges, means, and medians are similar among machines 5, 6, and 8, as well as between machines 4 and 7, while machine 9 has a minimum of 0, and the standard deviations of machines 4 and 5 are close but differ from those of the others. For flat plate position, machines 4 and 6 have positive maximum values and standard deviations above two, whereas other machines have negative maxima and standard deviations below 1.3, with large variations in medians. For compacting end position, machines 6 and 8 have maximum values 2–3 times those of other machines, and machine 8 has a much higher standard deviation. For SE1 soldering iron temperature, machine 4 has a much lower minimum, while machine 6 has a higher minimum and larger standard deviation, with similar means and medians between machines 5 and 8, and between machines 7 and 9; SE2 soldering iron temperature follows a similar pattern, with machine 4 showing more dispersed data. Overall, the ranges and dispersions of parameters differ across machines, suggesting that machine-level parameters may affect the number of short cigarettes.

Next, the characteristics of the short-position data will be researched by machine, and the box line plots are shown in Figure 3.

As can be seen from Figure 3, the distribution of short-position values is more scattered in machines 7 and 8, and more aggregated in machines 5 and 6. Moreover, the mean of short-position values in machine 4 is higher than the mean of the other machines, and the means in machines 5 and 6 are significantly lower. As we can see, there are some variations in the short-position values between the different units, indicating that the type of machine has some influence on the short-position numbers. It can be seen that for the daily number of short cigarettes, it is necessary to consider the type of machine when researching it, and the longitudinal counting data can be used to incorporate the effect of machine type into the short-position data.

In the actual data, there are some null values for each explanatory variable, which are filled with the median, and the data are normalized and modeled by the following mixed effects Poisson counting regression model:

$$E(Y_{ij}|X_{ij},Z_{ij})=e^{X_{ij}^T\beta +Z_{ij}^T{\alpha }_i}$$

(38)

Since there are 2 shifts in production per day, one shift per day is taken as a whole sample cross-section, the whole is noted as a work schedule for a total of 59 days with two shifts per day, that is $n=118$. Each work schedule operates on 6 machines, that is $m=6$, the total sample size was 708. The number of short positions generated by the first $i$ work schedule operation of the first $j$ machine is noted as $Y_{ij}$, $X_{ij}^T=(1,X_{ij1},X_{ij2},X_{ij3},X_{ij4},X_{ij5},X_{ij6})$ denotes the intercept and 6 values of machine parameters obtained for the first $i$ work schedule operation of the first $j$ machine, and $\beta$ is the coefficient vector of the explanatory variables with dimension $1\times 7$. $Z_{ij}^T$ is the random effect generated by the explanatory variables when operating 6 machines in 118 work schedules. Since it is not known in advance which explanatory variables generate random effects, let $Z_{ij}=X_{ij}$; ${\alpha }_i$ is the coefficient vector corresponding to $Z_{ij}^T$, with dimension $118\times 7$.

The double SCAD penalty method proposed in this paper is used to model and analyze the cigarette plant data to select the variables that would have an impact on the number of short cigarettes. And for the selection criteria of the penalty parameters, according to the simulation, the GACV criterion and the Poisson regression model without penalty are used as a control group; the results of the runs are shown in

Table 6. Variable estimation parameter.

Variable	Parameter Name	DSCAD Estimate	DSCAD Std. Error	DSCAD p-Value	Poisson Estimate	Poisson Std. Error	Poisson p-Value
$X_1$	Vehicle speed	0.0000	NA	NA	0.0229	0.0006	0.0000
$X_2$	Rubbing plate temperature	0.0067	0.0006	0	−0.0004	0.0006	0.5093
$X_3$	Position of flat plate	0.0409	0.0010	0	0.0272	0.0010	0.0000
$X_4$	Position of compacting end	0.0000	NA	NA	0.0253	0.0007	0.0000
$X_5$	SE1 soldering iron temperature	0.0000	NA	NA	0.0215	0.0009	0.0000
$X_{6z}$	SE2 soldering iron temperature	0.0000	NA	NA	−0.0082	0.0010	0.0000

and Figure 4.

In the Double SCAD (DSCAD) penalized model, the standard errors and p-values for some variables appear as NA, which is caused by the variable selection mechanism of the model. The DSCAD penalty shrinks the coefficients of variables with weak explanatory power for the response variable to zero, thereby achieving automatic variable selection. When refitting the mixed-effects Poisson model based on the selection results, these variables with zero coefficients are excluded from the model, so their corresponding standard errors, z-statistics, and p-values cannot be calculated. An estimated value of zero in the table indicates that the variable has been eliminated by the model. The NA values do not represent missing data but are normal outcomes of model simplification after variable selection. This reflects the advantage of DSCAD in performing high-dimensional variable selection and model reduction while maintaining satisfactory fitting performance.

As shown in Table 6, the DSCAD model based on the GACV criterion effectively achieves sparse variable selection and identifies only two non-zero variables as the key factors contributing to short-position defects:

Rubbing plate temperature (X₂) with a coefficient of 0.0067 (p < 0.001), indicating that a 1 °C increase in temperature is associated with a 0.67% average increase in the expected count of short cigarettes.

Position of flat plate (X₃) with a coefficient of 0.0409 (p < 0.001), meaning that a one-unit increase in the flat plate position leads to a 4.09% increase in the expected count of short cigarettes, making it the most significant factor.

All other variables (X₁, X₄, X₅, X₆) are estimated to be exactly zero, with their standard errors and p-values marked as NA, confirming that these variables have been excluded from the model.

In sharp contrast, the unpenalized Poisson model yields significant non-zero coefficients (p < 0.001) for all variables, including those that are actually irrelevant (e.g., vehicle speed X₁).

This demonstrates that the DSCAD model can produce a parsimonious and interpretable model specification, whereas the traditional Poisson model suffers from overfitting and yields misleading results.

Figure 4 visually reinforces these findings. The blue bars represent the inflated coefficients from the unpenalized Poisson model, which are all non-zero and vary widely. The red bars represent the DSCAD estimates, which are sparse and concentrated on the two meaningful variables. For variables like vehicle speed $(X1)$ and compacting end position $(X4)$, the DSCAD model correctly sets their coefficients to zero, while the Poisson model incorrectly identifies them as significant. This visual comparison clearly demonstrates the superiority of the DSCAD method in distinguishing signal from noise in real industrial data.

To quantify the advantages of the DSCAD model, we compared its fitting performance with the unpenalized Poisson model.

As shown in

Table 7. Model fit comparison.

Model	Log-Likelihood	AIC	BIC	Number of Variables
DSCAD-GACV	−194,174.8	388,367.6	388,408.7	6
Poisson	−195,934.7	391,879.4	391,902.2	2

, the DSCAD model achieved lower AIC (388,367.6 vs. 391,879.4) and BIC (388,408.7 vs. 391,902.2), indicating a better balance between model fit and complexity.

Practical Implications for Cigarette Factory Management

The real-world application provides clear, actionable insights.

Prioritize process control for two key parameters:

• Flat plate position (X3): As the most impactful factor, engineers should focus on tightening the tolerance for this setting to minimize production errors.
• Rubbing plate temperature (X2): While the effect is smaller, it is statistically significant. Monitoring and stabilizing this temperature can lead to incremental quality improvements.

Reallocate quality control resources:

The DSCAD model confirms that vehicle speed, compacting end position, and soldering iron temperatures are not significant drivers. Resources currently allocated to monitoring these variables can be redirected to controlling the flat plate position and rubbing plate temperature.

Integrate the DSCAD method into routine QC:

The method can be used to periodically re-evaluate the drivers of quality defects, ensuring that process control efforts remain focused on the most impactful factors.

The application to the cigarette factory data provides strong empirical evidence for the advantages of the proposed double SCAD penalty method. By automatically selecting only two relevant variables from six candidates, the model avoids overfitting and produces results that are consistent with engineering expertise. The DSCAD model not only improves interpretability but also delivers superior predictive performance, making it a valuable tool for data-driven decision-making in industrial quality control.

6. Conclusions

This paper proposes a double SCAD penalty model that imposes the SCAD penalty simultaneously on both fixed effects and random effects within the mixed-effects Poisson count regression model. Furthermore, the model accounts for the influence of random effects while enabling concurrent parameter estimation and variable selection. For parameter estimation, this paper presents the DSCAD-LQA to compute the parameters of the double SCAD model, which achieves simultaneous coefficient estimation for both fixed effects and random effects. With regard to penalty parameter selection, simulation results show that the model estimation performance under the GACV criterion is superior to that under the SIC in all scenarios. This paper mainly investigates whether variations in the degree of variable correlation and changes in the covariance of random effects affect the accuracy of model parameter estimation and variable selection.

When the variable correlation coefficients change, it is found that as the correlation coefficients increase, the estimation error of the proposed double SCAD model gradually increases, and the accuracy of the model in correctly identifying important variables and excluding unimportant variables decreases accordingly. In particular, when the correlation coefficients are relatively large, the model’s performance in parameter estimation and variable selection is affected more significantly.

When the covariance of random effects changes, it is observed that with the gradual increase in covariance, the parameter estimation error of the double SCAD model also increases, and the accuracy of the model in including important variables and excluding unimportant variables declines. When the covariance of random effects is relatively small, the model exhibits superior performance in parameter estimation and variable selection.

Furthermore, this paper compares the proposed double SCAD penalty model with three benchmark methods—the non-penalized Poisson regression model, the mixed-effects Poisson regression model with a single SCAD penalty on fixed effects, and the mixed-effects Poisson regression model with a double LASSO penalty—under both sparse and dense scenarios. The results demonstrate that the proposed double SCAD penalty model achieves better performance in parameter estimation and variable selection than the other three methods and is more suitable for sparse scenarios.