The Ridge-Hurdle Negative Binomial Regression Model: A Novel Solution for Zero-Inflated Counts in the Presence of Multicollinearity

Abstract

Datasets with many zero outcomes are common in real-world studies and often exhibit overdispersion and strong correlations among predictors, creating challenges for standard count models. Traditional approaches such as the Zero-Inflated Poisson (ZIP), Zero-Inflated Negative Binomial (ZINB), and Hurdle models can handle extra zeros and overdispersion but struggle when multicollinearity is present. This study introduces the Ridge-Hurdle Negative Binomial model, which incorporates L₂ regularization into the truncated count component of the hurdle framework to jointly address zero inflation, overdispersion, and multicollinearity. Monte Carlo simulations under varying sample sizes, predictor correlations, and levels of overdispersion and zero inflation show that Ridge-Hurdle NB consistently achieves the lowest mean squared error (MSE) compared to ZIP, ZINB, Hurdle Poisson, Hurdle Negative Binomial, Ridge ZIP, and Ridge ZINB models. Applications to the Wildlife Fish and Medical Care datasets further confirm its superior predictive performance, highlighting RHNB as a robust and efficient solution for complex count data modeling.

1. Introduction

Count data, representing event frequencies across domains such as transportation safety, epidemiology, and insurance, present persistent methodological challenges due to zero-inflation and the presence of multicollinearity. Negative Binomial (NB) regression model serves as the primary framework for count data analysis, incorporating a dispersion parameter that accommodates variance exceeding the mean, a phenomenon known as overdispersion [1,2]. Despite this flexibility, NB models encounter significant limitations when datasets exhibit zero inflation, characterized by observed zero counts substantially exceeding theoretical expectations under standard distributions [3].

Zero inflation manifests through dual mechanisms: structural zeros emerge when events become inherently impossible for specific population subsets, while sampling zeros result from stochastic processes [4,5,6]. This complexity necessitates sophisticated modeling approaches that can simultaneously address overdispersion and excess zeros.

Two-component models address zero inflation effectively, such as the Zero-Inflated Poisson (ZIP) and Zero-Inflated Negative Binomial (ZINB) models [7]. They combine a binary mechanism for classifying structural zeros with a count process that generates observations, including potential zeros. These frameworks have demonstrated substantial utility across highway safety [8], health sciences [9,10], and ecological modeling [11].

Hurdle models provide distinct two-part architectures where all zeros come from the binary component, while the count component uses truncated distributions to model only positive values, in contrast to ZINB approaches that allow zeros from both components [12,13,14]. This structure is valuable when factors influencing activity initiation differ from those affecting intensity, as shown in transportation research distinguishing crash occurrence from frequency [15].

Unlike ZIP models that assume Poisson-distributed positive counts and suffer from equidispersion constraints, Hurdle NB accommodates overdispersion in the positive count component through the gamma-distributed heterogeneity parameter [16]. Compared to ZINB models, Hurdle NB provides more precise conceptual interpretation by completely separating the zero-generating process from positive count generation, eliminating potential confusion about zero sources [17,18]. The additional dispersion parameter in Hurdle NB effectively captures unobserved heterogeneity that Hurdle Poisson cannot accommodate, resulting in improved model fit and more accurate predictions [19,20]. Likelihood ratio tests often favor Hurdle NB over Hurdle Poisson in real datasets, highlighting the importance of modeling overdispersion in the positive count component [21,22].

Multicollinearity among predictors introduces additional analytical complications, inflating coefficient standard errors and compromising inferential stability, especially problematic in high-dimensional datasets where covariate control remains challenging [23,24]. Traditional solutions like variable selection or principal component regression often sacrifice interpretability or exclude crucial covariates. Regularization techniques have emerged as powerful solutions for multicollinearity and enhanced predictive performance. Ridge regression incorporates L₂ penalties that shrink coefficients toward zero without elimination, stabilizing estimates amid correlated predictors [25].

Recent developments have extended regularization to count data through penalized generalized linear models, including penalized NB regression [26,27,28]. Akram et al. (2024) developed a ridge-type estimator for zero-inflated negative binomial [29]. Zeeshan et al. (2024) proposed a new ridge-type estimator for zero-inflated Poisson regression [30]. Penalized models demonstrate superior predictive accuracy, interpretability, and coefficient stability in high-dimensional or multicollinear contexts [31]. These advances have extended to zero-inflated models with penalization applied to both components [32].

Despite substantial progress, Hurdle models remain underexplored in penalized regression literature. While numerous studies have introduced regularized ZIP and ZINB models [33,34,35,36,37,38], limited research addresses penalized Hurdle frameworks, particularly regarding simultaneous zero inflation and multicollinearity management.

This study addresses a critical methodological gap by proposing an innovative Ridge-Hurdle Negative Binomial model that integrates L₂ regularization into the truncated NB count component. The approach stabilizes coefficient estimates under highly correlated predictors while maintaining the interpretability and flexibility inherent in Hurdle architectures for zero-inflated scenarios.

This study systematically compares performance across multiple count regression models: ZIP, ZINB, Hurdle Ridge, Hurdle Negative Binomial, Ridge ZIP, Ridge ZINB, and the proposed Ridge-Hurdle Negative Binomial under diverse data conditions. Through comprehensive simulation frameworks, the study evaluates model performance across varying zero inflation levels, predictor multicollinearity degrees, and sample sizes, using Mean Squared Error (MSE) as the primary assessment criterion.

The remainder of this paper is structured as follows. Section 2 presents the models under comparison, introduces the proposed estimator, and discusses its key statistical properties. Section 3 describes the design of the Monte Carlo simulations and reports the results across multiple scenarios based on the mean squared error criterion. Section 4 demonstrates the application of two real-world datasets, providing comparative analyses and discussions that highlight the reinforcement of theoretical findings. Finally, Section 5 concludes the paper with a summary of key insights and potential directions for future research.

2. Materials and Methods

This study employs a rigorous analytical framework to assess the effectiveness of competing count data models, contrasting traditional approaches (ZIP, ZINB, Hurdle Ridge, Hurdle Negative Binomial) against regularized alternatives (Ridge ZIP and Ridge ZINB) and the novel Ridge-Hurdle Negative Binomial (RHNB) methodology. The experimental design unfolds through two strategic phases: executing extensive simulation studies that explore model behavior under challenging conditions of excessive zeros and correlated predictors, and demonstrating practical applicability through empirical data analysis. The entire computational workflow leverages R statistical software (Version 4.5.1) to guarantee methodological transparency and replicable results.

2.1. Zero-Inflated Poisson (ZIP) Model

Let $Y_i\in {\mathbb{N}}_0$ be a count variable with excess zeros. The ZIP model assumes: each observation is zero with probability ${\pi }_i$ or comes from a Poisson distribution with mean ${\mu }_i>0$ with probability $1-{\pi }_i$ [4]. The Poisson PMF is:

$$\mathsf{\pi }\left({\mathrm{Y}}_{\mathrm{i}}=\mathrm{y}\right)={\displaystyle \frac{{\mathrm{e}}^{-{\mathsf{\mu }}_{\mathrm{i}}}{\mathsf{\mu }}_{\mathrm{i}}^{\mathrm{y}}}{\mathrm{y}!}},\mathrm{y}\in {\mathbb{N}}_0$$

In ZIP, ${\mathsf{\mu }}_{\mathrm{i}}=\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{\top }}\mathsf{\beta }\right)$ and ${\mathsf{\pi }}_{\mathrm{i}}={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{z}}_{\mathrm{i}}^{\mathrm{\top }}\mathsf{\gamma }\right)}{1+\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{z}}_{\mathrm{i}}^{\mathrm{\top }}\mathsf{\gamma }\right)}}$, giving:

$$\mathsf{\pi }\left({\mathrm{Y}}_{\mathrm{i}}=\mathrm{y}\right)=\left\{\begin{array}{ll}{\mathsf{\pi }}_{\mathrm{i}}+\left(1-{\mathsf{\pi }}_{\mathrm{i}}\right){\mathrm{e}}^{-{\mathsf{\mu }}_{\mathrm{i}}},& \mathrm{y}=0\\ \left(1-{\mathsf{\pi }}_{\mathrm{i}}\right){\displaystyle \frac{{\mathrm{e}}^{-{\mathsf{\mu }}_{\mathrm{i}}}{\mathsf{\mu }}_{\mathrm{i}}^{\mathrm{y}}}{\mathrm{y}!}},& \mathrm{y}>0\end{array}\right.$$

Let $\mathrm{X}\in {\mathbb{R}}^{\mathrm{n}\times \mathrm{p}}$ and $\mathrm{Z}\in {\mathbb{R}}^{\mathrm{n}\times \mathrm{q}}$ be covariate matrices, and $\mathsf{\theta }=(\mathsf{\beta },\mathsf{\gamma })$. The log-likelihood for independent observations is:

$$\mathcal{l}(\mathsf{\theta })=\sum _{{\mathrm{y}}_{\mathrm{i}}=0} \mathrm{l}\mathrm{o}\mathrm{g}\left[{\mathsf{\pi }}_{\mathrm{i}}+\left(1-{\mathsf{\pi }}_{\mathrm{i}}\right){\mathrm{e}}^{-{\mathsf{\mu }}_{\mathrm{i}}}\right]+\sum _{{\mathrm{y}}_{\mathrm{i}}>0} \left[\mathrm{l}\mathrm{o}\mathrm{g}\left(1-{\mathsf{\pi }}_{\mathrm{i}}\right)-{\mathsf{\mu }}_{\mathrm{i}}+{\mathrm{y}}_{\mathrm{i}}\mathrm{l}\mathrm{o}\mathrm{g}{\mathsf{\mu }}_{\mathrm{i}}-\mathrm{l}\mathrm{o}\mathrm{g}{\mathrm{y}}_{\mathrm{i}}!\right]$$

Define ${\mathrm{V}}_{\mathsf{\mu }}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\mathsf{\mu }}_1,\dots ,{\mathsf{\mu }}_{\mathrm{n}}\right)$. For the Poisson part (non-zero counts), the score and information matrix are approximated as:

$${\mathrm{D}}_{\mathsf{\beta }}={\mathrm{X}}^{\mathrm{\top }}(\mathrm{y}-\mathsf{\mu }),{\mathrm{I}}_{\mathsf{\beta }}={\mathrm{X}}^{\mathrm{\top }}{\mathrm{V}}_{\mathsf{\mu }}\mathrm{X}$$

The approximate estimator is:

$$\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}\approx {\left({\mathrm{X}}^{\mathrm{\top }}{\mathrm{V}}_{\mathsf{\mu }}\mathrm{X}\right)}^{-1}{\mathrm{X}}^{\mathrm{\top }}{\mathrm{V}}_{\mathsf{\mu }}\mathrm{y}$$

2.2. Zero-Inflated Negative Binomial (ZINB) Model

The ZINB model extends the ZIP by assuming a Negative Binomial distribution for the count component [4,7]. Let ${\mathrm{Y}}_{\mathrm{i}}\in {\mathbb{N}}_0$ denote a count response with overdispersion and excess zeros. ${\mathrm{Y}}_{\mathrm{i}}\sim \mathrm{N}\mathrm{B}\left({\mathsf{\mu }}_{\mathrm{i}},\mathsf{\gamma }\right),{\mathsf{\mu }}_{\mathrm{i}}=\mathbb{E}\left[{\mathrm{Y}}_{\mathrm{i}}\right],{\mathsf{\gamma }}^{-1}\mathrm{is}\mathrm{the}\mathrm{dispersion}.$

The NB PMF is [2]

$$\mathrm{P}\left({\mathrm{Y}}_{\mathrm{i}}={\mathrm{y}}_{\mathrm{i}}\right)={\displaystyle \frac{\mathsf{\Gamma }\left({\mathrm{y}}_{\mathrm{i}}+\mathsf{\gamma }\right)}{\mathsf{\Gamma }\left({\mathrm{y}}_{\mathrm{i}}+1\right)\mathsf{\Gamma }\left(\mathsf{\gamma }\right)}}{\left({\displaystyle \frac{{\mathsf{\mu }}_{\mathrm{i}}}{{\mathsf{\mu }}_{\mathrm{i}}+\mathsf{\gamma }}}\right)}^{{\mathrm{y}}_{\mathrm{i}}}{\left({\displaystyle \frac{\mathsf{\gamma }}{{\mathsf{\mu }}_{\mathrm{i}}+\mathsf{\gamma }}}\right)}^{\mathsf{\gamma }},{\mathrm{y}}_{\mathrm{i}}\in {\mathbb{N}}_0.$$

In ZINB, zeros arise either from a structural process with probability ${\mathsf{\pi }}_{\mathrm{i}}$, or from the NB distribution with probability $1-{\mathsf{\pi }}_{\mathrm{i}}$. Thus,

$$\mathrm{P}\left({\mathrm{Y}}_{\mathrm{i}}={\mathrm{y}}_{\mathrm{i}}\right)=\left\{\begin{array}{ll}{\mathsf{\pi }}_{\mathrm{i}}+\left(1-{\mathsf{\pi }}_{\mathrm{i}}\right){\left({\displaystyle \frac{\mathsf{\gamma }}{{\mathsf{\mu }}_{\mathrm{i}}+\mathsf{\gamma }}}\right)}^{\mathsf{\gamma }},& {\mathrm{y}}_{\mathrm{i}}=0\\ \left(1-{\mathsf{\pi }}_{\mathrm{i}}\right){\displaystyle \frac{\mathsf{\Gamma }\left({\mathrm{y}}_{\mathrm{i}}+\mathsf{\gamma }\right)}{\mathsf{\Gamma }\left({\mathrm{y}}_{\mathrm{i}}+1\right)\mathsf{\Gamma }\left(\mathsf{\gamma }\right)}}{\left({\displaystyle \frac{{\mathsf{\mu }}_{\mathrm{i}}}{{\mathsf{\mu }}_{\mathrm{i}}+\mathsf{\gamma }}}\right)}^{{\mathrm{y}}_{\mathrm{i}}}{\left({\displaystyle \frac{\mathsf{\gamma }}{{\mathsf{\mu }}_{\mathrm{i}}+\mathsf{\gamma }}}\right)}^{\mathsf{\gamma }},& {\mathrm{y}}_{\mathrm{i}}>0\end{array}\right.$$

with

$${\mathsf{\mu }}_{\mathrm{i}}=\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{\top }}\mathsf{\beta }\right),{\mathsf{\pi }}_{\mathrm{i}}={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{z}}_{\mathrm{i}}^{\mathrm{\top }}\mathsf{\gamma }\right)}{1+\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{z}}_{\mathrm{i}}^{\mathrm{\top }}\mathsf{\gamma }\right)}}$$

The observed information (negative Hessian) for $\mathsf{\beta }$ is

$${\mathrm{I}}_{\mathsf{\beta }}={\mathrm{X}}^{\mathrm{\top }}\mathrm{D}\mathrm{X},$$

with $\mathrm{D}$ the weight matrix from second derivatives. The estimator is approximated by

$$\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}\approx {\left({\mathrm{X}}^{\mathrm{\top }}\mathrm{D}\mathrm{X}\right)}^{-1}{\mathrm{X}}^{\mathrm{\top }}\mathrm{D}\tilde{\mathrm{y}}$$

2.3. Hurdle Poisson Model

Hurdle models, introduced by Mullahy (1986) and refined by Cameron and Trivedi (1998), address excess zeros in count data by modeling the probability of crossing a hurdle before generating positive counts from a zero-truncated distribution [1,13].

Let ${\mathrm{f}}_1\left(\mathrm{y}\right)$ denote the binary (hurdle) process and ${\mathrm{f}}_2\left(\mathrm{y}\right)$ the count model. Then the Hurdle PMF is defined as [11]:

$$\mathrm{P}(\mathrm{Y}=\mathrm{y})=\left\{\begin{array}{ll}{\mathrm{f}}_1\left(0\right)=\mathsf{\pi },& \mathrm{y}=0\\ (1-\mathsf{\pi }){\displaystyle \frac{{\mathrm{f}}_2\left(\mathrm{y}\right)}{1-{\mathrm{f}}_2\left(0\right)}}=(1-\mathsf{\pi }){\mathrm{f}}_2^{\ast }(\mathrm{y}),& \mathrm{y}>0\end{array}\right.$$

where ${\mathrm{f}}_2^{\ast }\left(\mathrm{y}\right)$ is the zero-truncated version of ${\mathrm{f}}_2\left(\mathrm{y}\right)$, and $\mathsf{\pi }=\mathrm{P}(\mathrm{Y}=0)$. If ${\mathrm{f}}_2\left(\mathrm{y}\right)$ is Poisson with mean $\mathsf{\mu }$, and the hurdle is Bernoulli-logistic with ${\mathsf{\pi }}_{\mathrm{i}}=\mathrm{P}\mathrm{r}\left({\mathrm{Y}}_{\mathrm{i}}=0\right)$, the Hurdle Poisson (HP) model is

$$\mathrm{P}\left({\mathrm{Y}}_{\mathrm{i}}={\mathrm{y}}_{\mathrm{i}}\right)=\left\{\begin{array}{ll}{\mathsf{\pi }}_{\mathrm{i}},& {\mathrm{y}}_{\mathrm{i}}=0\\ \left(1-{\mathsf{\pi }}_{\mathrm{i}}\right){\displaystyle \frac{{\mathrm{e}}^{-{\mathsf{\mu }}_{\mathrm{i}}}{\mathsf{\mu }}_{\mathrm{i}}^{{\mathrm{y}}_{\mathrm{i}}}}{\left(1-{\mathrm{e}}^{-{\mathsf{\mu }}_{\mathrm{i}}}\right){\mathrm{y}}_{\mathrm{i}}!}},& {\mathrm{y}}_{\mathrm{i}}>0\end{array}\right.$$

with

$$\mathrm{l}\mathrm{o}\mathrm{g}\left({\displaystyle \frac{{\mathsf{\pi }}_{\mathrm{i}}}{1-{\mathsf{\pi }}_{\mathrm{i}}}}\right)={\mathrm{z}}_{\mathrm{i}}^{\mathrm{\top }}\mathsf{\gamma },{\mathsf{\mu }}_{\mathrm{i}}=\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{\top }}\mathsf{\beta }\right)$$

where ${\mathrm{x}}_{\mathrm{i}}\in {\mathbb{R}}^{\mathrm{p}}$ (count covariates) and ${\mathrm{z}}_{\mathrm{i}}\in {\mathbb{R}}^{\mathrm{q}}$ (hurdle covariates). The mean and variance are

$$\begin{array}{c}\mathbb{E}\left[{\mathrm{Y}}_{\mathrm{i}}\right]={\displaystyle \frac{\left(1-{\mathsf{\pi }}_{\mathrm{i}}\right){\mathsf{\mu }}_{\mathrm{i}}}{1-{\mathrm{e}}^{-{\mathsf{\mu }}_{\mathrm{i}}}}}\\ \mathrm{V}\mathrm{a}\mathrm{r}\left({\mathrm{Y}}_{\mathrm{i}}\right)=\left(1-{\mathsf{\pi }}_{\mathrm{i}}\right)\left[{\displaystyle \frac{{\mathsf{\mu }}_{\mathrm{i}}^2+{\mathsf{\mu }}_{\mathrm{i}}}{1-{\mathrm{e}}^{-{\mathsf{\mu }}_{\mathrm{i}}}}}-{\left({\displaystyle \frac{{\mathsf{\mu }}_{\mathrm{i}}}{1-{\mathrm{e}}^{-{\mathsf{\mu }}_{\mathrm{i}}}}}\right)}^2\right]+{\mathsf{\pi }}_{\mathrm{i}}{\left({\displaystyle \frac{{\mathsf{\mu }}_{\mathrm{i}}}{1-{\mathrm{e}}^{-{\mathsf{\mu }}_{\mathrm{i}}}}}\right)}^2\end{array}$$

2.4. Hurdle Negative Binomial (Hurdle NB) Model

The Hurdle NB model extends hurdle regression to accommodate overdispersed count data using a zero-truncated negative binomial (ZTNB) distribution for positive counts [39]. The model consists of two parts:

Binary component (zero vs. positive count), modeled via logistic regression:

$${\mathrm{f}}_1(0)=\mathrm{P}(\mathrm{Y}=0\mid \mathrm{V})={\displaystyle \frac{1}{1+\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{V}}^{\mathrm{\top }}\mathsf{\delta }\right)}}={\displaystyle \frac{1}{1+{\mathrm{H}}_{\mathrm{i}}}}$$

where $\mathrm{V}$ is the design matrix for the binary part, $\mathsf{\delta }$ the associated parameters, and ${\mathrm{H}}_{\mathrm{i}}=\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{V}}^{\mathrm{\top }}\mathsf{\delta }\right)$. The odds ratio is $\mathrm{e}\mathrm{x}\mathrm{p}\left(\mathsf{\delta }\right)$.

Thus,

$$1-{\mathrm{f}}_1(0)={\displaystyle \frac{{\mathrm{H}}_{\mathrm{i}}}{1+{\mathrm{H}}_{\mathrm{i}}}}$$

Positive count component (for $\mathrm{Y}\ge 1$), modeled via a zero-truncated negative binomial:

$${\mathrm{f}}_2(\mathrm{y})={\displaystyle \frac{\mathsf{\Gamma }(\mathrm{y}+\mathsf{\gamma })}{\mathsf{\Gamma }(\mathrm{y}+1)\mathsf{\Gamma }\left(\mathsf{\gamma }\right)}}{\left({\displaystyle \frac{{\mathrm{H}}_{\mathrm{j}}}{{\mathrm{H}}_{\mathrm{j}}+\mathsf{\gamma }}}\right)}^{\mathrm{y}}{\left({\displaystyle \frac{\mathsf{\gamma }}{{\mathrm{H}}_{\mathrm{j}}+\mathsf{\gamma }}}\right)}^{\mathsf{\gamma }}{\left[1-{\left({\displaystyle \frac{\mathsf{\gamma }}{{\mathrm{H}}_{\mathrm{j}}+\mathsf{\gamma }}}\right)}^{\mathsf{\gamma }}\right]}^{-1},\mathrm{y}=\mathrm{1,2},\dots$$

where ${\mathrm{H}}_{\mathrm{j}}=\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{X}}^{\mathrm{\top }}\mathsf{\beta }\right),\mathrm{X}$ is the design matrix for the count part, and $\mathsf{\beta }$ are the parameters. Now, the Hurdle NB probability mass function

$$\mathrm{P}(\mathrm{Y}=\mathrm{y})=\left\{\begin{array}{ll}{\displaystyle \frac{1}{1+{\mathrm{H}}_{\mathrm{i}}}},& \mathrm{y}=0\\ {\displaystyle \frac{{\mathrm{H}}_{\mathrm{i}}}{1+{\mathrm{H}}_{\mathrm{i}}}}{\mathrm{f}}_2(\mathrm{y}),& \mathrm{y}=\mathrm{1,2},\dots \end{array}\right.$$

The MLE ${\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{H}\mathrm{N}\mathrm{B}}$ is obtained by solving $\mathrm{U}\left(\mathsf{\beta }\right)=0$ using Newton-Raphson. In matrix form:

$${\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{H}\mathrm{N}\mathrm{B}}={\left({\mathrm{X}}^{\mathrm{\top }}\mathrm{D}\mathrm{X}\right)}^{-1}{\mathrm{X}}^{\mathrm{\top }}\mathrm{D}\tilde{\mathrm{y}}$$

where $\mathrm{D}$ is a diagonal weight matrix depending on ${\mathsf{\mu }}_{\mathrm{i}}$ and $\mathsf{\gamma }$, and $\tilde{\mathrm{y}}$ is the pseudo-response vector from the score expansion.

2.5. Ridge Zero-Inflated Poisson (Ridge ZIP) Model

Let $\mathrm{Y}\in {\mathbb{R}}^{\mathrm{n}}$ be a vector of count responses (non-negative integers), $\mathrm{X}\in {\mathbb{R}}^{\mathrm{n}\times \mathrm{p}}$ be the design matrix of predictors, and $\mathsf{\beta }\in {\mathbb{R}}^{\mathrm{p}}$ be the coefficient vector. The Poisson regression model, which assumes that each observation ${\mathrm{y}}_{\mathrm{i}}\sim \mathrm{P}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{n}\left({\mathsf{\mu }}_{\mathrm{i}}\right)$, where the mean ${\mathsf{\mu }}_{\mathrm{i}}$ is related to the predictors through a log link function:

$$\mathrm{l}\mathrm{o}\mathrm{g}\left({\mathsf{\mu }}_{\mathrm{i}}\right)={\mathrm{x}}_{\mathrm{i}}^{\mathrm{T}}\mathsf{\beta }\mathrm{or}\mathrm{equivalently}{\mathsf{\mu }}_{\mathrm{i}}=\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{T}}\mathsf{\beta }\right)$$

$$\mathcal{l}(\mathsf{\beta })=\sum _{\mathrm{i}=1}^{\mathrm{n}} \left({\mathrm{y}}_{\mathrm{i}}{\mathrm{x}}_{\mathrm{i}}^{\mathrm{T}}\mathsf{\beta }-\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{T}}\mathsf{\beta }\right)-\mathrm{l}\mathrm{o}\mathrm{g}\left({\mathrm{y}}_{\mathrm{i}}!\right)\right)$$

However, when the predictors in $\mathrm{X}$ are highly correlated (i.e., multicollinearity), the MLE can be unstable and lead to overfitting [40].

Hoerl and Kennard (1970) introduced ridge regression to handle multicollinearity by minimizing squared residuals with a constraint on the sum of squared coefficients [25]. This concept extends to count data through GLMs, particularly the Poisson model, where ${\mathrm{Y}}_{\mathrm{i}}\sim \mathrm{P}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{o}\mathrm{n}\left({\mathsf{\mu }}_{\mathrm{i}}\right)$ with ${\mathsf{\mu }}_{\mathrm{i}}=\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{\top }}\mathsf{\beta }\right)$. Multicollinearity can make the MLE $\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}$ unstable. The Ridge ZIP model extends the standard ZIP model to handle excessive zeros and multicollinearity. The ZIP PMF is [41]:

$$\mathrm{P}\left({\mathrm{Y}}_{\mathrm{i}}={\mathrm{y}}_{\mathrm{i}}\right)=\left\{\begin{array}{ll}{\mathsf{\pi }}_{\mathrm{i}}+\left(1-{\mathsf{\pi }}_{\mathrm{i}}\right){\mathrm{e}}^{-{\mathsf{\mu }}_{\mathrm{i}}},& {\mathrm{y}}_{\mathrm{i}}=0\\ \left(1-{\mathsf{\pi }}_{\mathrm{i}}\right){\displaystyle \frac{{\mathrm{e}}^{-{\mathsf{\mu }}_{\mathrm{i}}}{\mathsf{\mu }}_{\mathrm{i}}^{{\mathrm{y}}_{\mathrm{i}}}}{{\mathrm{y}}_{\mathrm{i}}!}},& {\mathrm{y}}_{\mathrm{i}}>0\end{array}\right.$$

with ${\mathsf{\pi }}_{\mathrm{i}}={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{z}}_{\mathrm{i}}^{\mathrm{\top }}\mathsf{\gamma }\right)}{1+\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{z}}_{\mathrm{i}}^{\mathrm{\top }}\mathsf{\gamma }\right)}}$ and ${\mathsf{\mu }}_{\mathrm{i}}=\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{\top }}\mathsf{\beta }\right)$. The ridge estimator maximizes the penalized log-likelihood:

$${\mathcal{l}}_{\mathrm{p}\mathrm{e}\mathrm{n}}(\mathsf{\beta },\mathsf{\gamma })=\mathcal{l}(\mathsf{\beta },\mathsf{\gamma })-\mathsf{\lambda }{\mathsf{\beta }}^{\mathrm{\top }}\mathsf{\beta },\mathsf{\lambda }>0.$$

In matrix form, the Ridge ZIP estimator for $\mathsf{\beta }$ is:

$${\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{Ridge}\mathrm{ZIP}}={\left({\mathrm{X}}^{\mathrm{\top }}\mathrm{D}\mathrm{X}+\mathsf{\lambda }{\mathrm{I}}_{\mathrm{p}}\right)}^{-1}{\mathrm{X}}^{\mathrm{\top }}\mathrm{D}{\mathrm{Y}}^{\ast },$$

where $\mathrm{X}\in {\mathbb{R}}^{\mathrm{n}\times \mathrm{p}}$ is the count design matrix, $\mathrm{D}$ the weight matrix from the Fisher scoring, ${\mathrm{Y}}^{\ast }$ the adjusted response, $\mathsf{\lambda }$ the ridge penalty, and ${\mathrm{I}}_{\mathrm{p}}$ the $\mathrm{p}\times \mathrm{p}$ identity.

2.6. Ridge Zero-Inflated Negative Binomial (Ridge ZINB) Model

Multicollinearity among explanatory variables can destabilize regression estimates, making the standard ZINB estimator unreliable when the eigenvalues of ${\mathrm{X}}^{\mathrm{\top }}\mathrm{X}$ are small. To address this, the ridge estimator adds a positive constant to the diagonal of ${\mathrm{X}}^{\mathrm{\top }}\mathrm{X}$, producing the Ridge ZINB estimator [42]:

$${\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{Ridge}\mathrm{ZINB}}={\left({\mathrm{X}}^{\mathrm{\top }}\mathrm{D}\mathrm{X}+\mathsf{\lambda }\mathrm{I}\right)}^{-1}{\mathrm{X}}^{\mathrm{\top }}\mathrm{D}{\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{Z}\mathrm{I}\mathrm{N}\mathrm{B}}$$

where $\mathrm{X}\in {\mathbb{R}}^{\mathrm{n}\times \mathrm{p}}$ is the design matrix for the count component, $\mathrm{D}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\stackrel{\mathrm{ˆ}}{\mathsf{\mu }}}_{\mathrm{i}}\right)$ is the weight matrix from ZINB, with ${\stackrel{\mathrm{ˆ}}{\mathsf{\mu }}}_{\mathrm{i}}={\mathrm{x}}_{\mathrm{i}}^{\mathrm{\top }}{\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{ZINB}}$, ${\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{Z}\mathrm{I}\mathrm{N}\mathrm{B}}\approx {\left({\mathrm{X}}^{\mathrm{\top }}\mathrm{D}\mathrm{X}\right)}^{-1}{\mathrm{X}}^{\mathrm{\top }}\mathrm{D}\tilde{\mathrm{y}}$ is the standard ZINB estimate, $\mathsf{\lambda }\ge 0$ is the ridge parameter, and $\mathrm{I}$ is the $\mathrm{p}\times \mathrm{p}$ identity matrix.

Properties:

• $\mathsf{\lambda }=0\Rightarrow {\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{Ridge}\mathrm{ZINB}}={\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{ZINB}}$
• $\mathsf{\lambda }>0\Rightarrow ‖{\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{Ridge}\mathrm{ZINB}}‖<‖{\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{ZINB}}‖,$ reducing variance and improving stability under multicollinearity

2.7. Proposed Ridge-Hurdle Negative Binomial (RHNB) Model

In Section 2.4, ${\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{H}\mathrm{N}\mathrm{B}}$ denoted the MLE obtained from the positive-count component of the Hurdle NB model. The proposed RHNB estimator introduces a ridge penalty to control for multicollinearity or overfitting in the count component of the Hurdle NB model. The following is the estimator of the RHNB, adding a weight matrix.

$${\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{R}\mathrm{H}\mathrm{N}\mathrm{B}}={\left({\mathrm{X}}^{\mathrm{\top }}\widehat{\mathrm{W}}\mathrm{X}+\mathsf{\lambda }{\mathrm{I}}_{\mathrm{p}}\right)}^{-1}{\mathrm{X}}^{\mathrm{\top }}\widehat{\mathrm{W}}\mathrm{X}{\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{H}\mathrm{N}\mathrm{B}}$$

where $\mathrm{X}\in {\mathbb{R}}^{\mathrm{n}\times \mathrm{p}}$: design matrix for the count component (excluding zero counts), $\widehat{\mathrm{W}}=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\stackrel{\mathrm{ˆ}}{\mathsf{\mu }}}_1,\dots ,{\stackrel{\mathrm{ˆ}}{\mathsf{\mu }}}_{\mathrm{n}}\right)$: diagonal weight matrix with elements ${\stackrel{\mathrm{ˆ}}{\mathsf{\mu }}}_{\mathrm{i}}=\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{x}}_{\mathrm{i}}^{\mathrm{\top }}{\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{H}\mathrm{N}\mathrm{B}}\right)$, $\mathsf{\lambda }\ge 0$: ridge penalty parameter, ${\mathrm{I}}_{\mathrm{p}}:\mathrm{p}\times \mathrm{p}$ identity matrix, and ${\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{R}\mathrm{H}\mathrm{N}\mathrm{B}}={\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{H}\mathrm{N}\mathrm{B}}$ when $\mathsf{\lambda }=0$. Here, $\widehat{\mathrm{W}}$ plays an analogous role to the diagonal weight matrix $\mathrm{D}$ in Section 2.4, Section 2.5, Section 2.6.

Let the eigen-decomposition of ${\mathrm{X}}^{\mathrm{\top }}\widehat{\mathrm{W}}\mathrm{X}$ be:

$${\mathrm{X}}^{\mathrm{\top }}\widehat{\mathrm{W}}\mathrm{X}=\sum _{\mathrm{j}=1}^{\mathrm{p}} {\mathsf{\lambda }}_{\mathrm{j}}{\mathsf{\psi }}_{\mathrm{j}}{\mathsf{\psi }}_{\mathrm{j}}^{\mathrm{\top }}$$

Let the true coefficient vector $\mathsf{\beta }$ be expressed in terms of this eigenbasis:

$$\mathsf{\beta }=\sum _{\mathrm{j}=1}^{\mathrm{p}} {\mathsf{\alpha }}_{\mathrm{j}}{\mathsf{\psi }}_{\mathrm{j}},\mathrm{where}{\mathsf{\alpha }}_{\mathrm{j}}={\mathsf{\psi }}_{\mathrm{j}}^{\mathrm{\top }}\mathsf{\beta }$$

Then the MSE of ${\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{R}\mathrm{H}\mathrm{N}\mathrm{B}}$ is:

$$\mathrm{M}\mathrm{S}\mathrm{E}\left({\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{R}\mathrm{H}\mathrm{N}\mathrm{B}}\right)=\stackrel{\mathrm{ˆ}}{\mathsf{\tau }}\sum _{\mathrm{j}=1}^{\mathrm{p}} {\displaystyle \frac{{\mathsf{\lambda }}_{\mathrm{j}}}{{\left({\mathsf{\lambda }}_{\mathrm{j}}+\mathrm{k}\right)}^2}}+{\mathsf{\lambda }}^2\sum _{\mathrm{j}=1}^{\mathrm{p}} {\displaystyle \frac{{\mathsf{\alpha }}_{\mathrm{j}}^2}{{\left({\mathsf{\lambda }}_{\mathrm{j}}+\mathrm{k}\right)}^2}}$$

where ${\mathsf{\lambda }}_{\mathrm{j}}:\mathrm{j}$-th eigenvalue of ${\mathrm{X}}^{\mathrm{\top }}\widehat{\mathrm{W}}\mathrm{X}$, ${\mathsf{\psi }}_{\mathrm{j}}$: corresponding orthonormal eigenvector, ${\mathsf{\alpha }}_{\mathrm{j}}={\mathsf{\psi }}_{\mathrm{j}}^{\mathrm{\top }}\mathsf{\beta }$: projection of the true coefficient vector onto the eigenvector ${\mathsf{\psi }}_{\mathrm{j}}$, and $\stackrel{\mathrm{ˆ}}{\mathsf{\tau }}={\displaystyle \frac{1}{\mathrm{n}-\mathrm{p}-1}}\sum _{\mathrm{i}=1}^{\mathrm{n}} {\left({\mathrm{y}}_{\mathrm{i}}-{\stackrel{\mathrm{ˆ}}{\mathsf{\mu }}}_{\mathrm{i}}\right)}^2$: estimated residual dispersion.

A conceptual flowchart of the models compared in this study is shown in Figure 1.

Figure 1. Conceptual flowchart of the models compared in this study.

The proposed RHNB estimator theoretically surpasses the HNB, Ridge ZIP, and Ridge ZINB models by integrating the ridge penalty $\mathrm{k}>0$ directly into the zero-truncated NB component, thereby reducing estimator variance and enhancing stability when ${\mathrm{X}}^{\mathrm{\top }}\mathrm{X}$ exhibits small eigenvalues (multicollinearity). Unlike Ridge ZIP and Ridge ZINB, which conflate zero and count processes, RHNB separates structural zeros from positive counts, ensuring efficient estimation under overdispersion and zero inflation. Its MSE expression confirms a favorable bias–variance trade-off, yielding lower total error and improved robustness in high-dimensional or correlated count data contexts.

3. Simulation Study

This section outlines the simulation design, varying sample sizes, correlation structures, and levels of zero inflation, and presents results that reveal the relative strengths, limitations, and robustness of each method under diverse and challenging data-generating processes.

3.1. Simulation Design

The data generation process followed a well-established methodology [43,44,45,46]. Correlated predictors were generated as

$${\mathrm{X}}_{\mathrm{i}\mathrm{j}}=\sqrt{1-{\mathsf{\rho }}^2}{\mathrm{h}}_{\mathrm{i}\mathrm{j}}+\mathsf{\rho }{\mathrm{h}}_{\mathrm{i},\mathrm{p}+1},\mathrm{i}=1,\dots ,\mathrm{n},\mathrm{j}=1,\dots ,\mathrm{p},$$

where ${\mathrm{h}}_{\mathrm{i}\mathrm{j}}\sim \mathrm{N}\left(\mathrm{0,1}\right),\mathrm{p}$ is the number of predictors, and $\mathsf{\rho }$ controls intercorrelation. The response variable ${\mathrm{Y}}_{\mathrm{i}}$ followed a zero-inflated negative binomial mechanism:

$${\mathrm{Y}}_{\mathrm{i}}=\left\{\begin{array}{ll}0,& \mathrm{with}\mathrm{probability}{\mathsf{\pi }}_{\mathrm{i}}\\ \mathrm{N}\mathrm{B}\left({\mathsf{\mu }}_{\mathrm{i}},\mathsf{\theta }\right),& \mathrm{with}\mathrm{probability}1-{\mathsf{\pi }}_{\mathrm{i}}\end{array}\right.$$

with linear predictor ${\mathsf{\eta }}_{\mathrm{i}}={\mathsf{\beta }}_0+{\mathsf{\beta }}_1{\mathrm{Z}}_{\mathrm{i}1}+\cdots +{\mathsf{\beta }}_{\mathrm{p}}{\mathrm{Z}}_{\mathrm{i}\mathrm{p}}$, zero-inflation probability ${\mathsf{\pi }}_{\mathrm{i}}={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathsf{\gamma }}_0\right)}{1+\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathsf{\gamma }}_0\right)}}$, and dispersion $\mathsf{\theta }\in \left\{\mathrm{1,5}\right\}$. Simulation scenarios varied by the number of predictors ($\mathrm{p}=\mathrm{10,20}$), high to severe correlation levels ($\mathsf{\rho }=\mathrm{0.80,0.90,0.95,0.99}$), sample sizes ($\mathrm{n}=\mathrm{100,200,500}$), and zero-inflation intercepts (${\mathsf{\gamma }}_0=\mathrm{1,2}$), corresponding to approximately $73\%$ and $83\%$ structural zeros [47,48]. Overdispersion was incorporated through $\mathsf{\theta }$, with higher values inducing greater variability [49]. Each configuration was replicated $\mathrm{N}=5000$ times for robust evaluation [50].

Model performance was assessed using mean squared error (MSE), computed as [51]

$$\mathrm{M}\mathrm{S}\mathrm{E}\left({\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}^{\ast }\right)={\displaystyle \frac{1}{\mathrm{N}}}\sum _{\mathrm{i}=1}^{\mathrm{N}} {\left({\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{i}}-\mathsf{\beta }\right)}^{\mathrm{\top }}\left({\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}_{\mathrm{i}}-\mathsf{\beta }\right),$$

where ${\stackrel{\mathrm{ˆ}}{\mathsf{\beta }}}^{\ast }$ is an estimator and $\mathsf{\beta }$ the true coefficient vector, aligned with the normalized eigenvector of ${\mathrm{X}}^{\mathrm{\top }}\mathrm{X}$ associated with its largest eigenvalue. Lower MSE values indicate superior estimator performance.

The ridge penalty parameter $\mathsf{\lambda }$ in this study for Ridge ZIP, Ridge ZINB, and RHNB models was optimized through cross-validation to ensure a fair balance between model bias and variance. Specifically, for both the Poisson and logistic components of the regularized models, $\mathsf{\lambda }$ was selected using $\mathrm{k}$-fold cross-validation (with $\mathrm{k}=5$) across a logarithmic grid of candidate values $\mathsf{\lambda }\in \left\{{10}^{-4},{10}^{-3},\dots ,{10}^2\right\}$ [52].

At each fold, the model was refitted on the training data and evaluated on the validation subset using appropriate loss functions-mean squared error (MSE) for Poisson models and log-loss for logistic models. The optimal penalty ${\mathsf{\lambda }}^{\ast }$ was chosen as [53]

$${\mathsf{\lambda }}^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\mathsf{\lambda }}{\mathrm{m}\mathrm{i}\mathrm{n}} {\displaystyle \frac{1}{\mathrm{k}}}\sum _{\mathrm{i}=1}^{\mathrm{k}} {\mathrm{L}}_{\mathrm{i}}(\mathsf{\lambda }),$$

where ${\mathrm{L}}_{\mathrm{i}}\left(\mathsf{\lambda }\right)$ denotes the fold-specific validation loss. This data-driven approach ensures that the ridge term $\mathsf{\lambda }\Vert \mathsf{\beta }{\Vert }_2^2$ effectively stabilizes parameter estimation under high multicollinearity and overdispersion, providing a sensitivity-controlled, empirically tuned penalty rather than an arbitrarily fixed one.

3.2. Results Discussion

This simulation study evaluates the performance of various count data models for zero-inflated and overdispersed count outcomes in roadway safety analysis. The study examines how model performance, measured by Mean Squared Error (MSE), is influenced by sample size, number of predictors, predictor correlation, intercept logit, and overdispersion levels. Models evaluated include traditional approaches (ZIP, ZINB, Hurdle Poisson, Hurdle NB) and regularized variants (Ridge ZIP, Ridge ZINB, RHNB). Table A1, Table A2, Table A3, Table A4, Table A5, Table A6, Table A7 and Table A8 in Appendix A present the MSE of the models under various scenarios.

3.2.1. Effectiveness Relative to Sample Size

Across all compared models, larger sample sizes consistently improved estimation accuracy, as reflected by declining MSE values in Table A1 and Table A2. Yet, the RHNB model stood out for its remarkable stability and reliability, even when data were limited. Unlike traditional zero-inflated and hurdle models that showed volatility in small samples, RHNB maintained low variability and robust predictive precision. The results in Table A1 illustrate that its accuracy improved steadily with increasing n, while Table A2 confirms that this pattern held even under stronger correlation conditions where competing models suffered from overfitting and inflated error. Overall, these findings demonstrate that RHNB effectively balances bias and variance, ensuring dependable performance across varying sample sizes.

3.2.2. Effectiveness Relative to the Number of Predictors

As the number of predictors increased, most models exhibited noticeable performance deterioration, underscoring their sensitivity to dimensionality (Table A5 and Table A8). Traditional ZIP and ZINB models, in particular, showed severe instability as multicollinearity intensified. In contrast, the RHNB model demonstrated strong resistance to variance inflation, with only a slight rise in error even when the predictor set doubled. This stability highlights the regularization effect of the ridge penalty, which effectively mitigates overfitting and preserves accuracy in high-dimensional and highly correlated environments. Overall, the results confirm that RHNB maintains reliable estimation performance as model complexity increases, outperforming conventional alternatives across all predictor settings.

3.2.3. Effectiveness Relative to Correlation Coefficients

Increasing correlation among predictors substantially impaired the performance of traditional models, as reflected in Table A1 to Table A2, Table A2 to Table A3, and Table A3 to Table A4. However, the RHNB model maintained strong resilience, showing only modest increases in error even under extreme multicollinearity. While all models experienced some degradation as correlation approached 0.99, RHNB consistently preserved estimation stability and predictive accuracy, unlike ZIP and ZINB, which deteriorated sharply. These results emphasize the model’s ability to counteract multicollinearity through ridge regularization, ensuring reliable inference and minimizing overfitting across varying correlation strengths.

3.2.4. Effectiveness Relative to Intercept Logit

Higher intercept logits, which correspond to stronger zero-inflation, posed major challenges for traditional models, as reflected in Table A1 and Table A7. In these scenarios, RHNB consistently exhibited exceptional robustness, maintaining stable and low error levels even when competing models failed. While ZIP and ZINB suffered from extreme error inflation under heavy zero-inflation, RHNB preserved accuracy across different correlations and dimensionalities. This demonstrates the model’s capacity to handle severe zero-inflated conditions through its ridge penalty, which effectively stabilizes estimation and curbs variance amplification when structural zeros dominate the data.

3.2.5. Effectiveness Relative to Overdispersion

Overdispersion posed significant difficulties for traditional count models, especially those without explicit mechanisms to handle extra-Poisson variation (Table A5, Table A6 and Table A8). While ZINB and Hurdle NB offered moderate resilience, the RHNB model consistently demonstrated superior adaptability. Its performance remained stable and accurate even as overdispersion intensified, reflecting the combined benefits of the hurdle framework and ridge regularization. Unlike ZIP and ZINB, which showed escalating error under high variance, RHNB effectively controlled instability and preserved predictive precision. These findings underscore its robustness in managing overdispersed data, a common feature of real-world count processes.

4. Application

To validate the robustness and practical applicability of the proposed count data models under various complex scenarios, two real-world datasets were analyzed. The analysis aimed to determine if the performance trends from the simulation study apply in real-world settings with multicollinearity, overdispersion, and excess zeros. The real data findings validate and strengthen the simulation results, enhancing the reliability of the comparison among the modeling methods.

In real-world applications, lower MSE values indicate greater predictive reliability and closer alignment between true and estimated coefficients, underscoring the model’s practical interpretability and usefulness for decision-making.

4.1. Wildlife Fish Data

This dataset contains 250 observations with five predictors: X1 (nofish) indicates whether the trip was not solely for fishing, X2 (livebait) indicates whether live bait was used, X3 (camper) indicates whether a camper was brought, X4 (persons) indicates the total number of participants, and X5 (child) indicates the number of children present. The response variable y is the number of fish caught [54]. The histogram in Figure 2 reveals clear zero inflation along with a few extreme values of interest for truncation. Additionally, the condition number of 10.25 suggests moderate multicollinearity in the dataset.

Figure 2. Zero-Inflated response of wildlife fish data.

The study assessed overdispersion in the fish count data by fitting a Poisson regression model. Although the residual deviance-to-degrees-of-freedom ratio was close to 1 (1.03), a formal score-based test using the R function dispersiontest() under the AER package indicated significant overdispersion (dispersion = 1.36, p = 0.025), showing that the variance of the counts exceeds the Poisson assumption. Table A9 in Appendix B showed that the coefficient and standard error vary from model to model, especially the standard error was smallest for the regularized models.

Figure 3 illustrates the MSE values across competing models, where the proposed RHNB model achieves the lowest error (0.718), clearly outperforming both traditional (ZIP, ZINB, HP, Hurdle NB) and regularized (Ridge ZIP, Ridge ZINB) alternatives. This superior performance reinforces the findings from the simulation study, highlighting RHNB’s robustness in handling zero inflation, overdispersion, and multicollinearity in the fish catch data.

Figure 3. MSE of the models for the wildlife fish data.

4.2. Medical Care Data

The Medical Care dataset (NMES1988) consists of 4406 Medicare-covered individuals aged 66 and older, drawn from the U.S. National Medical Expenditure Survey of 1987–1988 [9,55]. The response variable, ovisits (number of physician outpatient visits), exhibits clear evidence of zero inflation, as shown in Figure 4. Alongside measures of health-care utilization such as emergency visits and hospital stays, the dataset includes demographic, socioeconomic, and health-status indicators (e.g., age, gender, income, chronic conditions, activity limitations, and insurance coverage). Notably, the condition number of 212.22 signals severe multicollinearity among covariates, underscoring the need for robust modeling approaches.

Figure 4. Zero-inflated response of the data.

The analysis also evaluated overdispersion in the medical care visit data by fitting a Poisson regression model. The residual deviance relative to the degrees of freedom was substantially greater than 1 (2.77), and a formal score-based test using the R function dispersiontest() function from the AER package confirmed significant overdispersion (dispersion = 16.67, p = 0.001), indicating that the variance of the count outcome exceeds the assumptions of the Poisson model. Table A10 in Appendix B for this dataset showed that the coefficient and standard error vary from model to model, especially the standard error was the smallest for the proposed RHNB model.

Residual analysis further highlights the superiority of the proposed RHNB model, which demonstrates more stable variance and improved fit compared to its ridge counterparts, such as Ridge ZIP and Ridge ZINB. The comprehensive residual diagnostic plots supporting these findings are presented in Figure A1, Figure A2, and Figure A3 respectively in Appendix C.

Figure 5 presents the MSE values for the Medicare data, highlighting the clear superiority of the proposed RHNB model, which achieves the lowest error (0.068). Traditional models such as ZIP and ZINB, as well as their ridge counterparts, show notably higher errors, indicating their limitations in handling the complexities of this dataset. The results further demonstrate that RHNB effectively addresses both zero inflation and multicollinearity, leading to substantial gains in predictive accuracy. These findings are consistent with and strongly validate the insights obtained from the simulation study.

Figure 5. MSE of the Medical Care Data.

5. Conclusions

This study introduces the Ridge-Hurdle NB model, a novel framework that integrates the hurdle structure with ridge regularization to effectively address zero inflation, overdispersion, and multicollinearity in count data. Unlike earlier work on penalized Poisson and negative binomial models, the incorporation of ridge penalization within a hurdle-based design marks a unique methodological advancement.

Simulation experiments, along with applications to the Wildlife Fish Catch dataset and the Medicare dataset, consistently showed that the RHNB outperforms both traditional and regularized alternatives, validating its robustness and practical utility. While RHNB offers strong performance, its effectiveness depends on the careful selection of the ridge tuning parameter and may require a high configuration computer. Beyond its immediate contributions, this work lays the foundation for future research on regularized mixture models that jointly accommodate structural zeros, overdispersion, and predictor dependencies. Potential directions include developing open-source software for broader adoption, extending the framework to Bayesian inference and nonlinear effects, and adapting the model to longitudinal or spatially correlated count processes. Taken together, RHNB offers a powerful and flexible tool for applied domains such as health sciences, ecology, and transportation safety, where complex count data challenges are the norm.