Heterogeneous Overdispersed Count Data Regressions via Double-Penalized Estimations

Abstract

Recently, the high-dimensional negative binomial regression (NBR) for count data has been widely used in many scientific fields. However, most studies assumed the dispersion parameter as a constant, which may not be satisfied in practice. This paper studies the variable selection and dispersion estimation for the heterogeneous NBR models, which model the dispersion parameter as a function. Specifically, we proposed a double regression and applied a double ${\ell }_1$-penalty to both regressions. Under the restricted eigenvalue conditions, we prove the oracle inequalities for the lasso estimators of two partial regression coefficients for the first time, using concentration inequalities of empirical processes. Furthermore, derived from the oracle inequalities, the consistency and convergence rate for the estimators are the theoretical guarantees for further statistical inference. Finally, both simulations and a real data analysis demonstrate that the new methods are effective.

1. Introduction

In many scientific fields, such as biomedical science, ecology, and economics, experimental and observational studies often yield count data, a type of data in which the observations can take only the non-negative integer values. The Poisson regression models are commonly used for count data. However, it needs a restrictive assumption that the variance equals the mean. For many count data, the variance is often larger than the mean [1], which is called overdispersion. Because the Poisson regression model is invalid under the overdispersion case, a more general and flexible regression model, the negative binomial regression, has attracted lots of research attention and become popular in analyzing count data [2,3,4].

With the advance of modern data collection techniques, high-dimensional data are becoming increasingly common in scientific studies. The widely used estimations for the high-dimensional parameter include the lasso [5], the scad [6], the elastic net [7], the adaptive lasso [8], and so on. Recently, there has been much research on the high-dimensional NBR model, such as [9,10,11,12,13,14]. All of these works assumed the dispersion parameter as a constant. In practice, however, not all models satisfy the assumption. If the dispersion parameter is wrongly assumed to be a constant, the estimation of the mean regression will perform poorly as shown in the simulation in Section 4.1, thus the need to model the dispersion parameter as a function of some covariates. The heterogeneous negative binomial regression (HNBR) extends the NBR by observation-specific parameterization of the dispersion parameter [3]. The HNBR is a valuable tool for assessing the source of overdispersion. It belongs to the double-generalized linear models (DGLMs) or vector-generalized linear models (VGLMs), which are very useful in fitting more complex and potentially realistic models [15,16,17,18]. However, it appears that there is no study on selecting the dispersion explanation variables in the HNBR model.

In this paper, we study the variable selection and dispersion estimation for the heterogeneous NBR models. To the best of our knowledge and based on the literature, this study is the first. Specifically, we propose a double regression to estimate the coefficients of NB dispersion and NBR simultaneously. Because of the high dimension of the covariates, we apply a double ${\ell }_1$ penalty to both regressions. The two adjustment parameters we set are different because the first-order conditions for estimating the regression coefficients are entirely different from those for estimating the dispersion parameters. We construct an algorithm to perform variable selection and dispersion estimation simultaneously. Similar studies on high-dimensional NBR models include [19], which assumed the dispersion parameter as a constant. Their method requires an iterative algorithm to estimate the mean regression and dispersion alternatively and implement a lasso in each iteration. If there are many iterations, such an algorithm is a waste of computing resources.

The rest of the paper is organized as follows. Section 2 introduces the heterogeneous overdispersed count data model and defines the double ${\ell }_1$-penalized estimators for the mean and dispersion regressions. Then we use a technique called the stochastic Lipschitz condition to derive the asymptotic results in Section 3. Simulation studies and a real data application are given in Section 4. Finally, Section 5 concludes the article with a discussion. All proofs and technical details are provided in Appendix A.

2. Double ${\ell }_{\mathbf{1}}$-Penalized NBR

2.1. Heterogeneous Overdispersed Count Data Regressions

Suppose we have n count responses $Y_i$ and p-dimensional covariates $X_i=(x_{i1},\cdots ,x_{ip})$, $i\in \left[n\right]:=\{1,2,\dots ,n\}$. For the Poisson regression models, the response obeys the Poisson distribution

$$\mathrm{P}\left(Y_i=y_i{\left|\lambda \right.}_i\right)=\frac{{\lambda }_i^{y_i}}{y_i!}{e}^{-{\lambda }_i},i\in \left[n\right].$$

with ${\lambda }_i=\mathrm{E}\left(Y_i\right)$, we require that the positive parameter ${\lambda }_i$ is related to a linear combination of p covariates. A plausible assumption for the link function is $\eta \left({\lambda }_i\right)=log\left({\lambda }_i\right)=X_i^{\top }\beta$. It is worth noting that $\mathrm{E}\left(Y_i\right|X_i)=\mathrm{var}\left(Y_i\right|X_i)=exp\left(X_i^{\top }\beta \right)>0.$

For the traditional negative binomial regression, it assumes that the count data response obeys the NB distribution with overdispersion:

$$\mathrm{P}(Y_{\mathrm{i}}=y_i|X_i)=:f(y_i;k,{\mu }_i)=\frac{\mathsf{\Gamma }(k+y_i)}{\mathsf{\Gamma }\left(k\right)y_i!}{\left(\frac{{\mu }_i}{k+{\mu }_i}\right)}^{y_i}{\left(\frac{k}{k+{\mu }_i}\right)}^k,i\in \left[n\right],$$

(1)

with $\mathrm{E}\left(Y_i\right|X_i)={\mu }_i=exp\left({\beta }^{\top }{X}_i\right)$ and k is an unknown qualification of the overdispersion level. When $k\to \infty$, we have $\mathrm{var}\left(Y_i\right|X_i)={\mu }_i+\frac{{\mu }_i^2}{k}\to {\mu }_i=\mathrm{E}(Y_i\left|X_i\right)$, the Poisson regression for the mean parameter ${\mu }_i$. Thus, the Poisson regression is a limiting case of negative binomial regression when the dispersion parameter k tends to infinite.

In the heterogeneous negative binomial regression, k is proposed as a specific parameterization, i.e., $k=k\left(X_i\right)$. More specifically, we assume in this paper that

$$\mu \left(x\right)=exp\left\{{\theta }^{\left(1\right)\top }x\right\},k\left(x\right)=exp\left\{{\theta }^{\left(2\right)\top }x\right\}.$$

For notation simplicity, we denote

$$Pf:=\mathrm{E}f(X_i,Y_i),{\mathbb{P}}_{n}f:=\frac{1}{n}\sum _{i=1}^{n}f(X_i,Y_i),{\mathbb{G}}_{n}f:=\sqrt{n}({\mathbb{P}}_n-P)f,$$

for any measurable and integrable function f.

Let $\theta ={({\theta }^{\left(1\right)\top },{\theta }^{\left(2\right)\top })}^{\top }\in {\mathbb{R}}^{2p}$, the log-likelihood is

$$\begin{array}{cc}\hfill n\ell \left(\theta \right)& =log\prod _{i=1}^{n}f(y_i,k_i,{\mu }_i)=\sum _{i=1}^{n}log\left\{\frac{\mathsf{\Gamma }(k_i+Y_i)}{\mathsf{\Gamma }\left(k_i\right)Y_i!}{\left(\frac{{\mu }_i}{k_i+{\mu }_i}\right)}^{Y_i}{\left(\frac{k_i}{k_i+{\mu }_i}\right)}^{k_i}\right\}\hfill \\ \hfill & =\sum _{i=1}^n[-log\frac{\mathsf{\Gamma }\left(exp\left\{X_i^{\top }{\theta }^{\left(2\right)}\right\}\right)}{\mathsf{\Gamma }\left(Y_i+exp\left\{X_i^{\top }{\theta }^{\left(2\right)}\right\}\right)}+Y_i{X}_i^{\top }\left({\theta }^{\left(1\right)}-{\theta }^{\left(2\right)}\right)\hfill \\ \hfill & -\left[Y_i+exp\left\{X_i^{\top }{\theta }^{\left(2\right)}\right\}\right]log\left(1+exp\left\{X_i^{\top }\left({\theta }^{\left(1\right)}-{\theta }^{\left(2\right)}\right)\right\}\right)-log{Y}_i!]\hfill \end{array}$$

We use the negative log-likelihood as the loss function $\gamma$, and define

$$\begin{array}{cc}\hfill \gamma \left(\theta \right)& :=-logf\left(y\right|x,\theta )+logy!.\hfill \end{array}$$

Denote ${\partial }_j:=\frac{\partial }{\partial {\theta }^{\left(j\right)}}$, $j=1,2$, the score function for ${\theta }^{\left(1\right)}$ is

$${\partial }_1\ell \left(\theta \right)=-{\mathbb{P}}_n{\partial }_1\gamma \left(\theta \right)=\frac{1}{n}\sum _{i=1}^n(Y_i-e^{X_i^{\top }{\theta }^{\left(1\right)}})\frac{e^{X_i^{\top }{\theta }^{\left(2\right)}}{X}_i}{e^{X_i^{\top }{\theta }^{\left(1\right)}}+e^{X_i^{\top }{\theta }^{\left(2\right)}}}.$$

Furthermore, fix ${\theta }^{\left(1\right)}$, the score function for ${\theta }^{\left(2\right)}$ is

$${\partial }_2\ell \left(\theta \right)={\mathbb{P}}_n{\partial }_2\gamma \left(\theta \right)=\frac{1}{n}\sum _{i=1}^n\left\{\left[log\left(1+e^{X_i^{\top }({\theta }^{\left(1\right)}-{\theta }^{\left(2\right)})}\right)-\sum _{j=0}^{Y_i-1}\frac{1}{j+e^{X_i^{\top }{\theta }^{\left(2\right)}}}\right]+\frac{Y_i-e^{X_i^{\top }{\theta }^{\left(1\right)}}}{e^{X_i^{\top }{\theta }^{\left(1\right)}}+e^{X_i^{\top }{\theta }^{\left(2\right)}}}\right\}{e}^{X_i^{\top }{\theta }^{\left(2\right)}}{X}_i.$$

It is easy to verify that

$$P{\partial }_1\ell \left(\theta \right)=P{\partial }_2\ell \left(\theta \right)=0.$$

Thus, from now, we will suppose the true value of parameter $\theta$ is ${\theta }^{*}$.

2.2. Heterogeneous Overdispersed NBR via Double ${\ell }_1$ Penalty

The weighted lasso estimator under our circumstance is defined as

$${\widehat{\theta }}_n=\underset{\theta \in \mathsf{\Theta }}{argmin}\left({\mathbb{P}}_n\gamma \left(\theta \right)+\lambda {\parallel \theta \parallel }_{\omega ,1}\right),$$

(2)

where $\lambda >0$ is the tuning parameter and the weighted norm is defined by

$${\lambda \parallel \theta \parallel }_{\omega ,1}={\lambda }_1\parallel {\theta }^{\left(1\right)}{\parallel }_1+{\lambda }_2{\parallel {\theta }^{\left(2\right)}\parallel }_1=\lambda \left({\omega }_1\parallel {\theta }^{\left(1\right)}{\parallel }_1+{\omega }_2{\parallel {\theta }^{\left(2\right)}\parallel }_1\right),$$

and $\omega ={({\omega }_1,{\omega }_2)}^{\top }={({\lambda }_1/\lambda ,{\lambda }_2/\lambda )}^{\top }\in [0,1]\times [0,1]$ is the weight, ${\parallel ·\parallel }_1$ means the ${\ell }_1$-norm. This technique is also used in [20]. Equation (2) is a weighted double ${\ell }_1$-penalized problem, which is a kind of convex penalty optimization, and when ${\lambda }_1={\lambda }_2$, it becomes a single-penalized problem. In this paper, we use different ${\lambda }_1$ and ${\lambda }_2$, as the first-order conditions for estimating the regression coefficients are entirely different from those for estimating the dispersion parameters, and take $\lambda ={\lambda }_1\vee {\lambda }_2$.

Because the weighted group lasso estimator ${\widehat{\theta }}_n$ has no closed-form solution, we need to use iterative methods such as quasi-Newton or coordinate descent methods. We use BIC to choose the parameter ${\lambda }_1$ and ${\lambda }_2$.

$$BIC({\lambda }_1,{\lambda }_2)=-2\ell \left({\widehat{\theta }}_n\right)+{\displaystyle \frac{logn}{n}}k,$$

where k is the number of nonzero estimated coefficients. To illustrate the algorithm explicitly, we rewrite $\gamma \left(\theta \right)$ as $\gamma ({\theta }^{\left(1\right)\top }x,{\theta }^{\left(2\right)\top }x)$ and define ${\theta }^{\left(3\right)}={\lambda }_2/{\lambda }_1{\theta }^{\left(2\right)}$, ${\theta }^{†}={({\theta }^{\left(1\right)\top },{\theta }^{\left(3\right)\top })}^{\top }$. Converting ${\theta }^{\left(2\right)}$ into ${\theta }^{\left(3\right)}$ turns the double ${\ell }_1$-penalized problem into a single penalized one, which can be solved through some R packages, such as “lbfgs”. The algorithm is formally given in Algorithm 1.

Algorithm 1 Double ${\ell }_1$-Penalized Optimization

Input: the set of tuning parameters $\mathsf{\Lambda }={\left\{({\lambda }_{1,i},{\lambda }_{2,i})\right\}}_{i=1}^m$ Output: the estimate ${\widehat{\theta }}_n$   for $i=1,\dots ,m$, do     let $x^{*}=\frac{{\lambda }_{1,i}}{{\lambda }_{2,i}}x$;     solve ${\widehat{\theta }}^{†}={({\widehat{\theta }}^{\left(1\right)\top },{\widehat{\theta }}^{\left(3\right)\top })}^{\top }={argmin}_{{\theta }^{†}\in \mathsf{\Theta }}\left({\mathbb{P}}_n\gamma ({\theta }^{\left(1\right)\top }x,{\theta }^{\left(3\right)\top }{x}^{*}))+{\lambda }_{1,i}{\parallel {\theta }^{†}\parallel }_1\right)$;     obtain the estimate ${\widehat{\theta }}_{n,i}={({\widehat{\theta }}^{\left(1\right)\top },\frac{{\lambda }_{2,i}}{{\lambda }_{1,i}}{\widehat{\theta }}^{\left(3\right)\top })}^{\top }$;     compute $BIC({\lambda }_{1,i},{\lambda }_{2,i})=-2\ell \left({\widehat{\theta }}_{n,i}\right)+{\displaystyle \frac{logn}{n}}{k}_i$;   end for   find $i_{opt}={argmin}_{i=1,\dots ,m}BIC({\lambda }_{1,i},{\lambda }_{2,i})$;   return ${\widehat{\theta }}_{n,i_{opt}}$

The proposed algorithm can perform variable selection and dispersion estimation simultaneously. Similar studies on high-dimensional NBR models include [19], which assumed the dispersion parameter as a constant. However, their method requires an iterative algorithm to estimate the mean regression and dispersion alternatively and implement lasso in each iteration. If there are many iterations, such an algorithm is a waste of computing resources.

3. Main Results

3.1. Stochastic Lipschitz Conditions

We write the maximum of $Y_i$ from the sample of size n as $M_{Y,n}$, then the sample space for ${\left\{Y_i\right\}}_{i=1}^n$ is $\mathcal{Y}:=\{y\in \mathbb{N},y\le M_{y,n}\}$., i.e., $M_{y,n}={max}_{i\in \left[n\right]}{Y}_i$. Note that ${lim}_{n\to \infty }\mathrm{P}(M_{y,n}=\infty )=1$; what we need to tackle is actually an unbounded empirical process. However, for $z:=\left(\begin{array}{cc}{x}^{\top }& \\ & x^{\top }\end{array}\right)\in {\mathbb{R}}^{2\times 2p}$, we can assume the value space $\mathcal{S}$ for $s:=z\theta$ is bounded and satisfies

$$\mathcal{S}:=\left\{s={(s_1,s_2)}^{\top }\in {\mathbb{R}}^2,-\infty <m_{s,n}\le s_j\le \left|s_j\right|\le M_{s,n}<\infty ,j=1,2\right\}.$$

As we can see, the most significant difference between this article and other conventional literature about lasso estimators is that we use $s=z\theta$ rather than $\theta$ as the explanatory variable to analyze the properties of the loss function $\gamma$. This is not a traditional way. At first glimpse, the combination may complicate the analysis in the next step because the KKT condition requires the story about $\frac{\partial }{\partial \theta }\gamma$, which is critical for the traditional convex penalty problem. However, this article will try a different approach, the stochastic Lipschitz conditions introduced in the event $\mathcal{A}$ of Proposition 1 in [14], to solve the ${\ell }_1$-penalization problem. Define the local stochastic Lipschitz constant by

$$Lip(f;{\theta }^{*}):=\underset{\theta \in \mathsf{\Theta }/\left\{{\theta }^{*}\right\}}{sup}\left|\frac{\sqrt{n}{\mathbb{G}}_n\left(f\left(\theta \right)-f\left({\theta }^{*}\right)\right)}{\parallel \theta -{\theta }^{*}{\parallel }_1}\right|.$$

The most apparent advantage of the stochastic Lipschitz conditions over the KKT condition is that it can easily deal with the several parameters involved in different locations of the model that need to impose the same penalty on them, which is why we do not need to derive the KKT condition in this paper.

To establish the stochastic Lipschitz conditions for this unbounded counting process, another assumption, called the strongly midpoint log-convex, for some positive $\gamma$ should be satisfied, which states for the joint density from the sample $\mathbb{Y}:={(Y_1,\dots ,Y_n)}^{\top }\in {\mathbb{Z}}^n$’s negative log-density of n independent NB responses $\psi \left(y\right):=-log{p}_{\mathbb{Y}}\left(y\right)$ satisfies

$$\psi \left(x\right)+\psi \left(y\right)-\psi \left(⌈\frac{1}{2}x+\frac{1}{2}y⌉\right)-\psi \left(⌊\frac{1}{2}x+\frac{1}{2}y⌋\right)\ge \frac{\gamma }{4}{\parallel x-y\parallel }_2^2,\forall x,y\in {\mathbb{Z}}^n.$$

This assumption is a condition that ensures that the suprema of the multiplier empirical processes of n independent responses have sub-exponential concentration phenomena, which can be alternatively checked by the tail inequality for the suprema of the empirical processes corresponding to classes of unbounded functions ([21]).

Theorem 1.

Suppose ${max}_{i\in \left[n\right],1\le k\le p}\left|X_{ik}\right|\le M_x<\infty$, the parameter space Θ is convex and its diameter $D_{\mathsf{\Theta }}<\infty$. If ${\left\{Y_i\right\}}_{i=1}^n$ and ${\left\{Z_i\theta \right\}}_{i\in \left[n\right],\theta \in \mathsf{\Theta }}$ are both in the value space $\mathcal{Y}$ and $\mathcal{S}$ defined as previous, then for any $\theta \in \mathsf{\Theta }$,

$$\begin{array}{cc}\hfill Lip(\gamma ;{\theta }^{*})& =\underset{\theta \in \mathsf{\Theta }/\{{\theta }^T*\}}{sup}\left|\frac{\sqrt{n}{\mathbb{G}}_n\left(\gamma \left(\theta \right)-\gamma \left({\theta }^{*}\right)\right)}{\parallel \theta -{\theta }^{*}{\parallel }_1}\right|\hfill \\ \hfill & \le \sqrt{n}{M}_q:=\left(A_1\sqrt{log(2p/q_2)}+A_2\sqrt{logp}+A_3\sqrt{log(p/q_3)}\right)\sqrt{\underset{1\le k\le p}{max}\sum _{i=1}^n{X}_{ik}^2}\hfill \\ \hfill & +B\sqrt{log(2p/q_1)}\sqrt{{\left(\underset{1\le k\le p}{max}\sum _{i=1}^n{X}_{ik}^4\right)}^{1/2}}\vee Clog(2p/q_1)+Dlog(p/q_3),\hfill \end{array}$$

with probability at least $1-q_0$, where $q_1,q_2,q_3\in (0,1)$ satisfy $q_1+q_2+q_3=q_0$, and the constants are as follows:

$$\begin{array}{c}{A}_1=\sqrt{2}{F}_1,A_2=32\sqrt{2}{M}_x{F}_2{D}_{\mathsf{\Theta }},A_3=\sqrt{2}\left(2(F_1+M_{y,n})\vee F_2{M}_x{D}_{\mathsf{\Theta }}\right),\\ B=6\sqrt{2\left(w^{\left(1\right)}\vee w^{\left(2\right)}\right){\left(\sum _{i=1}^{n}a{({\mu }_i,k_i)}^4\right)}^{1/2}},C=12M_x\left(w^{\left(1\right)}\vee w^{\left(2\right)}\right)\underset{1\le i\le n}{max}a({\mu }_i,k_i),\\ D=8\left(2(F_1+M_{y,n})\vee F_2{M}_x{D}_{\mathsf{\Theta }}\right)M_x,w^{\left(1\right)}=\frac{e^{M_{s,n}}}{e^{m_{s,n}}+e^{M_{s,n}}},\\ w^{\left(2\right)}=\frac{e+e^{M_{s,n}-m_{s,n}}}{1+e^{m_{s,n}-M_{s,n}}}+\frac{1}{1+e^{m_{s,n}-M_{s,n}}},\end{array}$$

where $M_{y,n}={max}_{i\in \left[n\right]}{Y}_i$ is the suprema empirical process.

It is worthy to note that the $M_{y,n}$ in Theorem 1 is a random process; hence, the bound above is not deterministic. Fortunately, $M_{y,n}$ can use the strongly midpoint log-convex condition to be bounded, which we state in Lemma A3. Theorem 1 combined with Lemma A3 will give the following result as a step more.

Theorem 2.

Assume the conditions are the same as that in Theorem 1, then the stochastic Lipschitz constant has a nonrandom upper bound:

$$\begin{array}{cc}\hfill Lip(\gamma ;{\theta }^{*})& \le \sqrt{n}{M}_q^{\prime }:=(A_1\sqrt{log(2p/q_2)}+A_2\sqrt{logp}+2A_3^{\prime }\left(log(2n/q_4)+\sqrt{log(np/q_3))}\right)\sqrt{\underset{1\le k\le p}{max}\sum _{i=1}^n{X}_{ik}^2}\hfill \\ \hfill & +B\sqrt{log(2p/q_1)}\sqrt{{\left(\underset{1\le k\le p}{max}\sum _{i=1}^n{X}_{ik}^4\right)}^{1/2}}\vee Clog(2p/q_1)+Dlog(p/q_3),\hfill \end{array}$$

with probability at least $1-q_0$, where $q_1,q_2,q_3,q_4\in (0,1)$ satisfy $q_1+q_2+q_3+q_4=q_0$, and

$$A_3^{\prime }=2\sqrt{2}\left(F_1+\left(2\gamma \underset{i\in \left[n\right]}{max}\left[a({\mu }_i,k_i)-\frac{{\mu }_i}{log2}\right]\right)\vee F_2{M}_x{D}_{\mathsf{\Theta }}\right).$$

Theorem 1 gives us a different sight of the loss function far more than KKT conditions. However, the stochastic Lipschitz condition above does not compare the estimated and true values directly. We can resolve this issue by using an eigenvalue condition on the design matrix consisting of $X_i$. Because the design matrix X is fixed, the eigenvalue condition in the next section is reasonable. It is worthy to note that this inequality is an oracle because it involves an unknown empirical process on the right side.

3.2. ${\ell }_2$-Estimation Error Oracle Inequalities RE Conditions

As we said previously, although we use stochastic Lipschitz conditions instead of KKT conditions, the restricted eigenvalue conditions (RE conditions) are still required. We denote by ${\delta }_J$ the vector in ${\mathbb{R}}^p$ with the same coordinates as v on J and zero coordinates on the complement $J^c$ of J, and $spt\left(v\right)=\{j:v_j\ne 0\}$. We will assume that the minima in (2) can always be obtained in the following setting, but it may not be unique. In general, to bound $\widehat{\theta }-{\theta }^{*}$, some conditions on the design matrix $\mathrm{X}\in {\mathbb{R}}^{n\times p}$ are needed for obtaining abound in terms of the ${\ell }_2$ norm of $\theta -{\theta }^{*}$. Here, we will utilize the restricted eigenvalue condition introduced in [22], which says that for some $1\le s\le p$ and $K>0$,

$$\kappa (s,K)=min\left\{\frac{{\parallel \mathrm{X}v\parallel }_2}{\sqrt{n}{\parallel v_J\parallel }_2}:1\le \left|J\right|\le s,v\in {\mathbb{R}}^p/\left\{0\right\},\parallel v_{J^c}{\parallel }_1\le K{\parallel v_J\parallel }_1\right\}>0.$$

(3)

It should be noted that omitting the weight $\omega$ and the sparse restricted set $\parallel v_{J^c}{\parallel }_1\le K{\parallel v_J\parallel }_1$ leads to $v^{\top }\left[\frac{1}{n}{\mathrm{X}}^{\top }\mathrm{X}\right]v/v^{\top }v\ge {\kappa }^2(s,K)$. Thus, it means that the smallest eigenvalue of the sample covariance matrix $\frac{1}{n}{\mathrm{X}}^{\top }\mathrm{X}$ is positive, which is impossible when $p>n$ because $\frac{1}{n}{\mathrm{X}}^{\top }\mathrm{X}$ is not full rank. To avoid this problem, ref. [22] consider the restricted eigenvalue condition under the sparse restricted set $\parallel v_{J^c}{\parallel }_1\le K{\parallel v_J\parallel }_1$ as a considerable relation in sparse high-dimensional estimation. The restricted eigenvalue is from the restricted strong convexity, which enforces a strong convexity condition for the negative log-likelihood function of linear models under a certain sparse restrict set.

Due to the double penalty, besides the RE condition, we also require another condition similar to the RE condition, the so-called l-restricted isometry constant defined in [23], as follows

$${\sigma }_{\mathrm{X},l}^2=max\left\{{\parallel \mathrm{X}v\parallel }_2^2/{\parallel v\parallel }_2^2:v\in {\mathbb{R}}^p,1\le spt\left(v\right)\le l\right\}\in (0,\infty ),$$

which essentially requires the eigenvalue of the sample covariance matrix under every vector with cardinality less than l (l should be no more than n) approximately behaves normally like the low-dimensional case.

With the RE condition and l-restricted isometry constant, and the two theorems we established before, the lasso estimator in (2) can guarantee a good consistent property.

Lemma 1

(see Lemma 3.1 in [23]). Suppose $T_0$ is a set of cardinality S. For a vector $h\in {\mathbb{R}}^p$, we let $T_1$ be the S largest positions of h outside of $T_0$. Put $T_{01}=T_0\cup T_1$, then

$${\parallel h\parallel }_2^2\le \parallel h_{T_{01}}{\parallel }_2^2+S^{-1}{\parallel h_{T_0^c}\parallel }_1^2.$$

Theorem 3.

Suppose the condition is the same as that in Theorem A1. Furthermore, assume $p_1=spt\left({\theta }^{(*1)}\right)\vee spt\left({\theta }^{(*2)}\right)\le p/2$, and there exists some $K>1$, $\kappa :=\kappa (2p_1,K)>0$. Let $\lambda =\frac{(K+1)M_q}{n(K-1)}$, then using this λ in (2), with probability at least $1-q$,

$${\parallel \widehat{\theta }-{\theta }^{*}\parallel }_2^2\le \frac{8p_1{M}_q^{2\prime }{K}^2}{{\kappa }^4{n}^2{C}_{\gamma }^2{(K-1)}^2}\left[2+K^2+\frac{2(1+2p_1{K}^2)(n{\kappa }^2+2{\sigma }_{\mathrm{X},p_1}^2)}{n{\kappa }^2}\right],$$

where $M_q$, $C_{\gamma }$ are defined in Theorems 1 and A1, respectively.

Remark 1.

Compared to the single lasso problem, in which we only have one unknown vectorized parameter, the oracle inequality in Theorem 3 has an extra term $\frac{2(1+2p_1{K}^2)(n{\kappa }^2+2{\sigma }_{\mathrm{X},p_1}^2)}{n{\kappa }^2}$.

Remark 2.

From Theorem 3, we know that the ${\ell }_2$ convergence rate is minimax optimal, as studied in [14].

Remark 3.

In this study, we use the lasso estimators of two partial regression coefficients because it is one of the most popular techniques for high-dimensional data. It is worth mentioning that the algorithms and theoretical results could be similarly generalized to other shrinkage estimators, such as the elastic net [7], the adaptive lasso [8], and so on.

4. Numerical Studies

4.1. Simulations

In this section, we evaluate the finite sample performance of the proposed method. The response is generated from the negative binomial regression model (1) with

$$\mu \left(x\right)=exp\left\{{\theta }^{\left(1\right)\top }x\right\},\mathrm{and}k\left(x\right)=exp\left\{{\theta }^{\left(2\right)\top }x\right\},$$

where ${\theta }^{\left(1\right)}$ and ${\theta }^{\left(2\right)}$ are two p-dimensional parameters. The explanatory variables are generated from the multivariate normal distributions with mean vector 0 and $Cov(x_i,x_j)={\rho }^{|i-j|}$, where $\rho =0,0.5$. The following two examples show the performance of the proposed estimator for the low-dimensional heterogeneous negative binomial regression and the variable selection in the high-dimensional case, respectively. The R package “lbfgs” is required to solve the optimization problem.

Example 1

(Low dimension). We set $p=3$ and $n=100,200,400$. The true parameters are ${\theta }^{\left(1\right)}=(1,2,-1)$ and ${\theta }^{\left(2\right)}=(-1,0.5,1)$, and their maximum likelihood estimators are denoted as ${\widehat{\theta }}^{\left(1\right)}$ and ${\widehat{\theta }}^{\left(2\right)}$, respectively. We compare the estimator ${\widehat{\theta }}^{\left(1\right)}$ with ${\widehat{\theta }}^{\left(1\right)*}$, which ignores the heterogeneity of the overdispersion and treats $k\left(x\right)$ as a constant. Table 1 displays the average squared estimation errors ${\parallel \widehat{\theta }-\theta \parallel }_2^2$ based on 200 repetitions.

Table 1. The average squared estimation errors of the estimators.

n	$\mathit{\rho }=0$			$\mathit{\rho }=0.5$
	${\widehat{\mathbf{\theta }}}^{\left(\mathbf{1}\right)*}$	${\widehat{\mathbf{\theta }}}^{\left(\mathbf{1}\right)}$	${\widehat{\mathbf{\theta }}}^{\left(\mathbf{2}\right)}$	${\widehat{\mathbf{\theta }}}^{\left(\mathbf{1}\right)*}$	${\widehat{\mathbf{\theta }}}^{\left(\mathbf{1}\right)}$	${\widehat{\mathbf{\theta }}}^{\left(\mathbf{2}\right)}$
100	0.1597	0.0335	0.72414	0.1809	0.0397	0.68904
200	0.0862	0.01	0.22149	0.0837	0.0169	0.33048
400	0.05	0.0047	0.08847	0.0619	0.0067	0.15066

We can make the following observations from the table. Firstly, the performances of the three estimators become better and better as n increases. Secondly, the estimator ${\widehat{\theta }}^{\left(1\right)}$, which estimates the parameter in the mean function $\mu \left(x\right)$, performs better than ${\widehat{\theta }}^{\left(2\right)}$, which estimates the parameter in the overdispersion function $k\left(x\right)$. Last, but the most important, ${\widehat{\theta }}^{\left(1\right)*}$ performs much worse than ${\widehat{\theta }}^{\left(1\right)}$. For example, the average squared estimation error of ${\widehat{\theta }}^{\left(1\right)*}$ is about 5 times of ${\widehat{\theta }}^{\left(1\right)}$’s when $n=100$, and 10 times of ${\widehat{\theta }}^{\left(1\right)}$’s when $n=400$. The comparison between ${\widehat{\theta }}^{\left(1\right)}$ and ${\widehat{\theta }}^{\left(1\right)*}$ indicates the necessity of considering the heterogeneity of the overdispersion.

Example 2

(High dimension). The sample sizes are chosen to be $n=100,200,400$, with dimension $p\in (25,50,150)$, $(50,100,250)$ and $(100,200,500)$, respectively. We set ${\theta }^{\left(1\right)}=(1,2,-1,0,\cdots ,0)$ and ${\theta }^{\left(2\right)}=(-1,0.5,1,0,\cdots ,0)$. The unknown tuning parameters $({\lambda }_1,{\lambda }_2)$ for the penalty functions are chosen by BIC criterion in the simulation. Results over 200 repetitions are reported. We compared the variable selection performance of the proposed method to the previous method, which ignores the heterogeneity of the overdispersion and treats $k\left(x\right)$ as a constant. For each case, Table 2 reports the number of repetitions that each important explanatory variable is selected in the final model and also the average number of unimportant explanatory variables being selected.

Table 2. The results of variable selection.

		Previous Method				Proposed Method
		$\mathbf{\mu }\left(\mathit{x}\right)$				$\mathbf{\mu }\left(\mathit{x}\right)$				$\mathit{k}\left(\mathit{x}\right)$
n	p	${\mathbf{\theta }}_{\mathbf{1}}^{\left(\mathbf{1}\right)}$	${\mathbf{\theta }}_{\mathbf{2}}^{\left(\mathbf{1}\right)}$	${\mathbf{\theta }}_{\mathbf{3}}^{\left(\mathbf{1}\right)}$	Other ${\mathbf{\theta }}^{\left(\mathbf{1}\right)}$s	${\mathbf{\theta }}_{\mathbf{1}}^{\left(\mathbf{1}\right)}$	${\mathbf{\theta }}_{\mathbf{2}}^{\left(\mathbf{1}\right)}$	${\mathbf{\theta }}_{\mathbf{3}}^{\left(\mathbf{1}\right)}$	Other ${\mathbf{\theta }}^{\left(\mathbf{1}\right)}$s	${\mathbf{\theta }}_{\mathbf{1}}^{\left(\mathbf{2}\right)}$	${\mathbf{\theta }}_{\mathbf{2}}^{\left(\mathbf{2}\right)}$	${\mathbf{\theta }}_{\mathbf{3}}^{\left(\mathbf{2}\right)}$	Other ${\mathbf{\theta }}^{\left(\mathbf{2}\right)}$s
$\rho =0$
100	25	173	198	171	2.33	192	200	190	0.37	180	184	180	0.32
	50	164	197	147	2.885	196	200	193	0.52	182	180	188	0.41
	150	136	182	111	2.725	194	194	192	1.02	188	182	186	0.41
200	50	196	200	192	1.435	200	200	200	0.59	200	190	198	0.53
	100	193	200	193	2.05	200	200	200	0.91	196	186	196	0.69
	250	162	198	155	1.5	199	199	198	1.18	198	198	198	0.69
400	100	200	200	200	0.605	200	200	200	0.4	200	198	200	0.55
	200	200	200	199	0.88	200	200	200	0.6	200	200	200	0.51
	500	197	200	198	1.29	200	200	200	1.21	200	200	200	0.61
$\rho =0.5$
100	25	183	199	179	2.3	194	198	194	0.41	179	184	180	0.35
	50	172	197	150	2.66	196	196	190	0.63	178	182	180	0.42
	150	134	191	99	2.32	194	196	192	1.01	180	184	182	0.43
200	50	195	200	197	1.48	200	200	198	0.38	196	183	190	0.32
	100	189	200	179	1.52	199	200	198	0.53	194	186	194	0.44
	250	178	200	154	1.39	196	198	196	1.1	196	196	194	0.55
400	100	200	200	200	0.435	200	200	200	0.28	200	199	194	0.34
	200	200	200	199	0.675	200	200	198	0.47	200	198	196	0.36
	500	199	200	194	1.12	200	200	198	1.07	200	198	196	0.56

We see from the table that our method performs much better than the previous method that treats $k\left(x\right)$ as a constant. Specifically, our method correctly selects important variables more times than the previous method, and it is less likely to select unimportant variables. Furthermore, the variable selection procedure performs better and better as the sample size n increases. When $n=400$, the important explanatory variables in $\mu \left(x\right)$ and $k\left(x\right)$ are correctly selected in almost every repetitions. When the dimension p increases, the procedure may select more unimportant explanatory variables, but the average numbers are less than $1.3$. The important variables in $k\left(x\right)$ are less likely to be selected than the important variables in $\mu \left(x\right)$ especially when the sample size is small, as well as the unimportant variables.

4.2. A Real Data Example

In this section, we apply the proposed method to the dataset of German health care demand. The data were employed in [24] and could be downloaded on http://qed.econ.queensu.ca/jae/2003-v18.4/riphahn-wambach-million/, accessed on 1 January 2022. The data contain 27,326 observations on 25 variables, including 2 dependent variables, Docvis (number of doctor visits in the last three months) and Hospvis (number of hospital visits in the last calendar year). For conciseness, we focus on Docvis in this study. We build the HNBR model based on the proposed variable selection procedure and make the standard NBR model a comparison. Define the fitting errors (FE) as $n^{-1}{\sum }_{i=1}^n(y_i-{\widehat{y}}_i)$, where $y_i$ denotes the raw data of Hospvis, ${\widehat{y}}_i$ is the predicted value, and n is the sample size. As the data are observed during 1984–1988, 1991, and 1994, we make the analysis for each observed year. Table 3 displays the variable selection results and fitting errors.

Table 3. The variable selection results and the fitting errors (FE) of NBR and HNBR models. The variable Others = {Married, Haupts, Reals, Fachhs, Abitur, Univ, Working, Bluec, Whitec, Self, Beamt, Public, Addon}. Because these variables are not selected in any year, we put them in “Others” for brevity.

Variables	1984			1985			1986			1987
	NBR	HNBR		NBR	HNBR		NBR	HNBR		NBR	HNBR
		$\mathbf{\mu }\left(\mathit{x}\right)$	$\mathit{k}\left(\mathit{x}\right)$		$\mathbf{\mu }\left(\mathit{x}\right)$	$\mathit{k}\left(\mathit{x}\right)$		$\mathbf{\mu }\left(\mathit{x}\right)$	$\mathit{k}\left(\mathit{x}\right)$		$\mathbf{\mu }\left(\mathit{x}\right)$	$\mathit{k}\left(\mathit{x}\right)$
Female	0	0	0	0	0	0	0	0	0	0	0	0
Age	−0.013	−0.013	−0.012	−0.009	−0.01	−0.007	−0.006	−0.006	−0.013	−0.002	−0.001	−0.018
Hsat	−0.205	−0.2	−0.025	−0.244	−0.237	0	−0.188	−0.195	−0.045	−0.158	−0.153	−0.043
Handdum	0	0	0	0	0	0	0	0	0	0	0	0
Handper	0.005	0.005	0.004	0.007	0.006	0.007	0.007	0.007	0	0.007	0.007	0.01
Hhninc	0	0	0	0	0	0	0	0	0	0	0	0
Hhkids	0	0	0	0	0	0	0	0	0	0	0	0
Educ	0	0	−0.027	0	0	−0.064	−0.035	−0.038	0	−0.095	−0.106	−0.003
Others	0	0	0	0	0	0	0	0	0	0	0	0
FE	0.798	0.602		2.203	1.874		0.735	0.581		1.314	1.027
Variables	1988			1991			1994
	NBR	HNBR		NBR	HNBR		NBR	HNBR
		$\mathbf{\mu }\left(\mathit{x}\right)$	$\mathit{k}\left(\mathit{x}\right)$		$\mathbf{\mu }\left(\mathit{x}\right)$	$\mathit{k}\left(\mathit{x}\right)$		$\mathbf{\mu }\left(\mathit{x}\right)$	$\mathit{k}\left(\mathit{x}\right)$
Female	0	0	0	0	0	0	0	0	0
Age	−0.015	−0.014	−0.012	−0.022	−0.019	−0.003	−0.005	−0.004	−0.011
Hsat	−0.191	−0.187	−0.015	−0.112	−0.132	−0.049	−0.226	−0.224	−0.06
Handdum	0	0	0	0	0	0	0	0	0
Handper	0.011	0.009	0.006	0.014	0.013	0	0.007	0.008	0.004
Hhninc	0	0	0	0	0	0	0	0	0
Hhkids	0	0	0	0	0	0	0	0	0
Educ	−0.016	−0.023	−0.002	−0.074	−0.068	0	−0.064	−0.069	0
Others	0	0	0	0	0	0	0	0	0
FE	1.144	0.912		1.007	0.787		0.713	0.58

We have the following findings from the table. First, the important variables in the NBR are the same as HNBR models in each year, and the estimates are close. Second, the selected variables in $\mu \left(x\right)$ are almost the same every year, namely Age, Hsat (health satisfaction), Handper (degree of handicap), and Educ (years of schooling). Moreover, some of these variables still play an essential role in $k\left(x\right)$, and $k\left(x\right)$ contains no variables other than these. Moreover, we can see that the fitting errors of the HNBR is less than that of the NBR. All of these illustrate the advantage of our method.

5. Conclusions and Future Study

We study the high-dimensional heterogeneous overdispersed count data via negative binomial regression models and propose a double ${\ell }_1$-regularized method for simultaneous variable selection and dispersion estimation. Under the restricted eigenvalue conditions, we prove the oracle inequalities with lasso estimators of two partial regression coefficients for the first time, using concentration inequalities of empirical processes. Furthermore, we derive the consistency and convergence rate for the estimators, which are the theoretical guarantees for further statistical inference. Simulation studies and a real example from the German health care demand data indicate that the proposed method works satisfactorily.

There are some limitations of this study. First, we assume that the responses are independent in this work. However, the NB responses are temporal dependent in the time-series data [25]. Thus, weak dependence conditions, including $\rho$-mixing, m-dependent types, could be considered in the future. Second, this study focuses little on the statistical inference, such as testing heterogeneous

$$H_0:{\theta }^{\left(2\right)}=0\mathrm{vs}.H_1:{\theta }^{\left(2\right)}\ne 0.$$

The issues concerning the hypothesis testing are via the debiased lasso estimator; see [26] and references therein. This will comprise our future research work. Another possible study is the false discovery rate (FDR) control, which aims to identify some small number of statistically significantly nonzero results after obtaining the sparse penalized estimation of the HNBR; see [27,28].