Zero-Inflated Binary Classification Model with Elastic Net Regularization

Abstract

Zero inflation and overfitting can reduce the accuracy rate of using machine learning models for characterizing binary data sets. A zero-inflated Bernoulli (ZIBer) model can be the right model to characterize zero-inflated binary data sets. When the ZIBer model is used to characterize zero-inflated binary data sets, overcoming the overfitting problem is still an open question. To improve the overfitting problem for using the ZIBer model, the minus log-likelihood function of the ZIBer model with the elastic net regularization rule for an overfitting penalty is proposed as the loss function. An estimation procedure to minimize the loss function is developed in this study using the gradient descent method (GDM) with the momentum term as the learning rate. The proposed estimation method has two advantages. First, the proposed estimation method can be a general method that simultaneously uses $L_1$- and $L_2$-norm terms for penalty and includes the ridge and least absolute shrinkage and selection operator methods as special cases. Second, the momentum learning rate can accelerate the convergence of the GDM and enhance the computation efficiency of the proposed estimation procedure. The parameter selection strategy is studied, and the performance of the proposed method is evaluated using Monte Carlo simulations. A diabetes example is used as an illustration.

1. Introduction

Zero-inflated models help characterize binary data sets with excess zero counts. Structural and sampling zeros can be found in a zero-inflated data set. When the response variable cannot take positive values due to inherent constraints or conditions, the zero is called structural zero. In a medicate setting, Diop et al. [1] investigated the infection response related to some diseases where $Y=1$ indicates the individual was infected; otherwise, $Y=0$. Assume that an immunity agency controls whether an individual is infected. The individual is immune and labeled by $Y=0$ if the individual cannot undergo the interesting outcome. In both examples, $Y=0$ is not a random zero and can be a structural zero. Ridout et al. [2] has also commented on the distinction between structural zero and random zero. In contrast to the structural zero, the zeros raised from random chance are the sampling zero. Using a mixture mechanism, the zero-inflated model can account for structural and sampling zero sources and performs better than the traditional logistic, Poisson, and non-negative binomial regression models.

Occasionally, the highly disproportionate distribution of categories also needs to be improved for data analysis. Such data sets are also referred to as imbalanced data. In binary classifications, the size of one category significantly dominates the other. Those binary data sets could be inherently imbalanced in real-world scenarios for various reasons.

The initial zero-inflated model was the zero-inflated Poisson (ZIP), proposed by Lambert [3], where a zero-inflated Poisson (ZIP) regression model was investigated via maximum likelihood estimation method to make statistical inferences. Moreover, a data set about the soldering defect on the printed wiring board was used to illustrate the proposed ZIP regression model. To overcome the drawback of over-dispersion from the Poisson model, the zero-inflated negative Binomial (ZINB) model was another competitive model. Reducing the impacts of random effects on the zero-inflated data sets, Hall [4] conducted case studies for the ZIP and zero-inflated binomial (ZIB) regression models. Hall [4] successfully showed that random effects help account for the correlation between observations within clusters. Cheung [5] explored the theoretical foundations of zero-inflated models, illustrated the practical applications with examples, and analyzed the implications of using these models for precise inference in the context of growth and development studies. Gelfand and Citron-Pousty [6] demonstrated the utilization of zero-inflated models for spatial count data in environmental and ecological statistics. Additionally, they investigated how these models address both spatial correlation and the frequent occurrence of excess zero counts in environmental and ecological applications. Rodrigues [7] and Ghosh et al. [8] studied the applications of using Bayesian techniques to estimate the parameters of zero-inflated models.

Harri and Zhao [9] introduced a zero-inflated-ordered probit model to process data sets with ordered response variables and excess zeros. They also utilized a tobacco consumption data set for modeling illustration. Loeys et al. [10] considered ZIP, ZINB, Poisson logit hurdle model (PLH), and Binomial logit hurdle model (NBLH). Diop et al. [11] investigated the maximum likelihood estimation method for the logistic regression model with a cure fraction. Staub and Winkelmann [12] conducted inferences to study the consistency of the parameter estimation of the zero-inflated count model. He et al. [13] investigated the relevance of the structural zeroes to a zero-inflated model. Diop et al. [1] conducted simulation-based inference for the zero-inflated binary regression model and proposed the maximum likelihood estimation procedure. Zuur and Ieno [14] gave comprehensive guidance for users to use R codes to apply zero-inflated models.

2. Motivation and Organization

Zero inflation and overfitting can reduce the accuracy rate of using machine learning models for characterizing binary data sets. A zero-inflated Bernoulli (ZIBer) model can be the right model to characterize zero-inflated binary data sets. To our best knowledge, few works have studied the ZIBer model for binary classification, and overcoming the overfitting problem of the ZIBer model is an open question. Recent works about the applications of the ZIBer model can be summarized as follows: When covariates missing at random is found in the ZIBer regression model, Lee et al. [15] proposed a validation likelihood method to estimate the model parameters. Pho [16] proposed a goodness-of-fit test for the ZIBer model besides revisiting the goodness-of-fit test of the logit model. Li and Lu [17] proposed an implementing procedure and consistent estimation method to obtain the variances of the regression parameter estimators. Then, Monte Carlo simulation methods are used to evaluate their proposed method. Pho [18] proposed a new zero-inflated probit Bernoulli model and constructed the parameter estimation process. Lu et al. [19] proposed a penalized estimation method for partially linear additive models to ZIBer outcome data. The B-spline method is employed to approximate unknown nonparametric components. Moreover, they proposed a two-stage iterative EM algorithm to obtain the penalized spline estimates. When the response variables in a zero-inflated data set follow a Bernoulli distribution, Chiang et al. [20] suggested an expectation-maximum (EM) maximum likelihood estimation process for the ZIBer model to obtain reliable estimates of the model parameters. We denote the EM maximum likelihood estimation method proposed by Chiang et al. [20] as the EM-ZIBer method.

The EM-ZIBer method of Chiang et al. [20] can be an easy method to obtain more reliable maximum likelihood estimates (MLEs) of the ZIBer model than the typical maximum likelihood estimation method. However, the EM-ZIBer method loses feature selection efficiency when the zero-inflated data set size is vast and has an imbalanced structure. This drawback reduces the feasibility of applying the EM-ZIBer method to an extensive zero-inflated data set. In this study, the gradient descent method (GDM) with momentum term as learning rate is used to minimize the loss function, which combines the minus log-likelihood function and using elastic net regularization rule for an overfitting penalty. The proposed estimation procedure has two advantages: First, the proposed estimation method simultaneously uses $L_1$- and $L_2$-norm terms for penalty and includes the ridge and least absolute shrinkage and selection operator (lasso) methods as special cases. Hence, the proposed estimation procedure is a general method. Second, the momentum learning rate can accelerate the convergence of the GDM and enhance the computation efficiency of the proposed estimation procedure. For simplicity, we named the new parameter estimation method ENR-ZIBer method. Because the lasso regulation uses $L_1$-norm for a penalty. The target function for optimization is not a convex function. Hence, the GDM replaces the Newton method for the loss function minimization in this study.

Steps to implement the proposed ENR-ZIBer method will be analytically studied in Section 3.3. Second, to accelerate the convergence of the proposed ENR-ZIBer method, we suggest using the momentum term to the learning rate parameter to implement the gradient descent method. The implementation of using the momentum term in the ENR-ZIBer method will be analytically studied in Section 3.4.

The rest of this article is organized as follows. The ZIBer model is addressed in Section 3. The maximum likelihood estimation is discussed in Section 3.2. Moreover, the proposed elastic net penalty regularization that includes the $L_1$- and $L_2$-norm rules to the maximum likelihood estimation for the ZIBer model, are studied in Section 3.3. To accelerate the convergence, the GDM with the momentum term as the learning rate is used to the GDM for the elastic net regularization rule is examined in Section 3.4. In Section 4, the quality of the proposed ENR-ZIB method is evaluated using Monte Carlo simulations. One example is used in Section 5 for illustration. Finally, some conclusions are given in Section 6.

3. The Statistical Model and Optimization

The ZIBer model will be established and the inference procedure of model parameters will be addressed in this section.

3.1. The Statistical Model

Regarding the binary classification modeling, the response variable Y can be labeled as 0 or 1, where $Y=1$ indicates an individual infected by a disease; otherwise, $Y=0$. Let the probability of $Y=1$ depend upon the covariate ${\mathit{x}}^T=(x_1,x_2,\cdots ,x_p)$ with $x_1=1$; that is, $P(Y=1|\mathit{x})=p$, where p is a function of $\mathit{x}$. Therefore, $Y|\mathit{x}\sim B(1,p)$, where $B(1,p)$ denotes the Bernoulli distribution with the conditional success probability $P(Y=1|\mathit{x})$. To construct the ZIBer model, the logit (or named logistic regression) function is used as the link and defined by

$$\begin{array}{c}\hfill log\left({\displaystyle \frac{p}{1-p}}\right)={\mathsf{\beta }}^T\mathit{x}={\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_p{x}_p,\end{array}$$

(1)

where ${\mathsf{\beta }}^T=({\beta }_1,{\beta }_2,\cdots ,{\beta }_p)$. It can be shown that

$$p={\displaystyle \frac{e^{{\mathsf{\beta }}^T\mathit{x}}}{1+e^{{\mathsf{\beta }}^T\mathit{x}}}}.$$

(2)

In a zero-inflated model, the zero inflation occurs in the analysis of counts if the data-generating process results in too many zeros. Failure to account for extra zeros could produce unreasonable estimates and inferences. To remove the drawback, two possible random resources of $Y=0$ are considered here. The structural zero occurs with probability $\delta$, and the non-structural zero happens with the probability $1-\delta$. Hence, the unconditional probability of $Y=0$ can be obtained by

$$\begin{array}{ccc}\hfill P(Y=0)& =& \delta +(1-\delta )\times (1-p)\hfill \end{array}$$

(3)

$$\begin{array}{ccc}& =& 1-(1-\delta )p.\hfill \end{array}$$

(4)

It is easy to obtain the probability

$$\begin{array}{c}\hfill P(Y=1)=1-P(Y=0)=(1-\delta )p.\end{array}$$

(5)

Assume that $\delta$ depends on the latent covariate ${\mathit{z}}^T=(z_1,z_2,\cdots ,z_m)$, where $z_1=1$. Using the logit model again by

$$\begin{array}{c}\hfill log\left({\displaystyle \frac{\delta }{1-\delta }}\right)={\mathsf{\theta }}^T\mathit{z}={\theta }_1{z}_1+{\theta }_2{z}_2+\cdots +{\theta }_m{z}_m,\end{array}$$

(6)

where ${\mathsf{\theta }}^T=({\theta }_1,{\theta }_2,\cdots ,{\theta }_m)$, it can be shown that

$$\delta ={\displaystyle \frac{e^{{\mathsf{\theta }}^T\mathit{z}}}{1+e^{{\mathsf{\theta }}^T\mathit{z}}}}.$$

(7)

Given the covariate information $\mathit{x},\mathit{z}$, and let $\pi :=\pi (\mathsf{\beta },\mathsf{\theta }|\mathit{x},\mathit{z})$ denote the probability of $P(Y=1|\mathit{x},\mathit{z})$. We can represent

$$\begin{array}{ccc}\hfill \pi & =& P(Y=1|\mathit{x},\mathit{z})=(1-\delta )p\hfill \\ & =& {\displaystyle \frac{1}{1+e^{{\mathsf{\theta }}^T\mathit{z}}}}\times {\displaystyle \frac{e^{{\mathsf{\beta }}^T\mathit{x}}}{1+e^{{\mathsf{\beta }}^T\mathit{x}}}}\hfill \\ & =& {\displaystyle \frac{e^{{\mathsf{\beta }}^T\mathit{x}}}{(1+e^{{\mathsf{\theta }}^T\mathit{z}})(1+e^{{\mathsf{\beta }}^{\mathbf{T}}\mathit{x}})}}.\hfill \end{array}$$

(8)

The odd can be presented by

$$\begin{array}{c}\hfill {\displaystyle \frac{\pi }{1-\pi }}={\displaystyle \frac{e^{{\mathsf{\beta }}^T\mathit{x}}}{1+e^{{\mathsf{\theta }}^T\mathit{z}}+e^{{\mathsf{\theta }}^T\mathit{z}+{\mathsf{\beta }}^T\mathit{x}}}}.\end{array}$$

(9)

Therefore, the ZIBer model can be defined by the following: the response variable $Y|\mathit{x},\mathit{z}\sim B(1,\pi )$, where $\pi$ is defined by Equation (8).

3.2. Maximum Likelihood Estimation

Let $\Theta =(\mathsf{\beta },\mathsf{\theta })$. Given independent responses, $y_1,y_2,\cdots ,y_n$, of Y with the corresponding observed and latent covariates, $\mathit{D}=({\mathit{x}}_1,{\mathit{x}}_2,\cdots ,{\mathit{x}}_n,{\mathit{z}}_1,{\mathit{z}}_2,\cdots ,{\mathit{z}}_n)$, where ${\mathit{x}}_i=(x_{i1},x_{i2},\cdots ,x_{ip})$ and ${\mathit{z}}_i=(z_{i1},z_{i2},\cdots ,z_{im})$ for $i=1,2,\cdots ,n$. The likelihood function can be presented by

$$\begin{array}{ccc}\hfill L(\Theta |\mathit{D})& =& \prod _{i=1}^n{\pi }_i^{y_i}{(1-{\pi }_i)}^{1-y_i}\hfill \\ & =& \prod _{i=1}^n\{e^{y_i{\mathsf{\beta }}^T{\mathit{x}}_i}{(1+e^{{\mathsf{\theta }}^T{\mathit{z}}_i}+e^{{\mathsf{\beta }}^T{\mathit{x}}_i+{\mathsf{\theta }}^T{\mathit{z}}_i})}^{1-y_i}\hfill \\ & & \times \left({\displaystyle \frac{1}{1+e^{{\mathsf{\beta }}^T{\mathit{x}}_i}}}\right)\left({\displaystyle \frac{1}{1+e^{{\mathsf{\theta }}^T{\mathit{z}}_i}}}\right)\}.\hfill \end{array}$$

(10)

The log-likelihood function of the ZIBer model can be presented by

$$\begin{array}{ccc}\hfill \ell (\Theta |\mathit{D})& =& \sum _{i=1}^n{y}_i{\mathsf{\beta }}^T{\mathit{x}}_i+\sum _{i=1}^n(1-y_i)log(1+e^{{\mathsf{\theta }}^T{\mathit{z}}_i}+e^{{\mathsf{\beta }}^T{\mathit{x}}_i+{\mathsf{\theta }}^T{\mathit{z}}_i})\hfill \\ & & -\sum _{i=1}^{n}log(1+e^{{\mathsf{\beta }}^T{\mathit{x}}_i})-\sum _{i=1}^{n}log(1+e^{{\mathsf{\theta }}^T{\mathit{z}}_i}).\hfill \end{array}$$

(11)

The first derivative of $\ell (\Theta |\mathit{D})$ with respect to $\mathsf{\beta }$ and $\mathsf{\theta }$ are

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial \ell (\Theta |\mathit{D})}{\partial \mathsf{\beta }}}=\sum _{i=1}^n\left(y_i+{\displaystyle \frac{(1-y_i)e^{{\mathsf{\beta }}^T{\mathit{x}}_i+{\mathsf{\theta }}^T{\mathit{z}}_i}}{1+e^{{\mathsf{\theta }}^T{\mathit{z}}_i}+e^{{\mathsf{\beta }}^T{\mathit{x}}_i+{\mathsf{\theta }}^T{\mathit{z}}_i}}}-{\displaystyle \frac{e^{{\mathsf{\beta }}^T{\mathit{x}}_i}}{1+e^{{\mathsf{\beta }}^T{\mathit{x}}_i}}}\right){\mathit{x}}_i\end{array}$$

(12)

and

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial \ell (\Theta |\mathit{D})}{\partial \mathsf{\theta }}}=\sum _{i=1}^n\left((1-y_i){\displaystyle \frac{e^{{\mathsf{\theta }}^T{\mathit{z}}_i}+e^{{\mathsf{\beta }}^T{\mathit{x}}_i+{\mathsf{\theta }}^T{\mathit{z}}_i}}{1+e^{{\mathsf{\theta }}^T{\mathit{z}}_i}+e^{{\mathsf{\beta }}^T{\mathit{x}}_i+{\mathsf{\theta }}^T{\mathit{z}}_i}}}-{\displaystyle \frac{e^{{\mathsf{\theta }}^T{\mathit{z}}_i}}{1+e^{{\mathsf{\theta }}^T{\mathit{z}}_i}}}\right){\mathit{z}}_i,\end{array}$$

(13)

respectively. The typical maximum likelihood estimates of $\mathsf{\beta }$ and $\mathsf{\theta }$ can be obtained by

$$(\widehat{\mathsf{\beta }},\widehat{\mathsf{\theta }})=\arg \underset{\Theta \in \Omega }{max}\ell (\Theta |\mathit{D}),$$

(14)

where $\Omega$ is the solution space of $\Theta =(\mathsf{\beta },\mathsf{\theta })$.

3.3. Optimization with Regularization

The $L_1$-norm regularization uses lasso rule to penalize the model overfitting. The $L_1$ norm is defined by ${\left|\right|\mathsf{\beta }\left|\right|}_1:={\sum }_{i=1}^p\left|{\beta }_j\right|$. Assume that we want to minimize $-\ell (\Theta |\mathit{D})$ with the $L_1$-norm penalty, the lasso target function can be modified as

$$\begin{array}{c}\hfill {\ell }^L(\Theta |\mathit{D})=-\ell (\Theta |\mathit{D})+{\lambda }_L{\left|\right|\mathsf{\beta }\left|\right|}_1,\end{array}$$

(15)

where ${\lambda }_L$ is a constant, which is used to control the regularization strength.

The $L_2$-norm regularization uses ridge regression to prevent the model overfitting. The $L_2$ norm is defined by ${\left|\right|\mathsf{\beta }\left|\right|}_2:=\sqrt{{\sum }_{i=1}^p{\beta }_j^2}$. Without loss of generality, we use the term ${\sum }_{i=1}^p{\beta }_j^2$ to develop the ridge regularization for the zero-inflated binary model. Assume that we want to minimize $-\ell (\Theta |\mathit{D})$ with the $L_2$-norm penalty, the ridge target function can be modified as

$$\begin{array}{c}\hfill {\ell }^R(\Theta |\mathit{D})=-\ell (\Theta |\mathit{D})+{\displaystyle \frac{{\lambda }_R}{2}}{\left({\left|\right|\mathsf{\beta }\left|\right|}_2\right)}^2.\end{array}$$

(16)

where ${\lambda }_R$ is a constant for controlling the regularization strength, and the constant ${\displaystyle \frac{1}{2}}$ is used to reduce the computation loading.

Lasso regularization using $L_1$-norm can result in a parsimonious model. Large ${\lambda }_L$ can help to result in a sparse model by setting some coefficients to zero, reducing the number of effective parameters. However, lasso regularization rule is sensitive to outliers because of taking the absolute value for the penalty.

The $L_2$-norm penalty in ridge regularization discourages large coefficients. The $L_2$-norm penalty pushes the model’s coefficients toward zero but never exactly reaches zero. Compared with the $L_1$-norm regularization rule, the ridge regularization rule is less sensitive to outliers because it takes a square value for the penalty.

It is a good idea to use a regularization method that covers the strength of the lasso and ridge regularization rules. Elastic net regularization uses the penalties from the lasso and ridge techniques to regularize the target model. Taking advantage of the $L_1$- and $L_2$-norms, the elastic net regularization rule combines the lasso and ridge regularization rules and learns from their shortcomings to improve the regularization of statistical models. The loss function based on the log-likelihood function and elastic net regulation can be defined by

$$\begin{array}{c}\hfill {\ell }^E(\Theta |\mathit{D})=-\ell (\Theta |\mathit{D})+{\displaystyle \frac{{\lambda }_1}{2}}{\left({\left|\right|\mathsf{\beta }\left|\right|}_2\right)}^2+{\lambda }_2{\left|\right|\mathsf{\beta }\left|\right|}_1,\end{array}$$

(17)

where ${\lambda }_1$ and ${\lambda }_2$ are constants to control the regularization strength. It is obvious that ${\ell }^E(\Theta |\mathit{D})$ reduces to ${\ell }^L(\Theta |\mathit{D})$ if ${\lambda }_1=0$, and ${\ell }^E(\Theta |\mathit{D})$ reduces to ${\ell }^R(\Theta |\mathit{D})$ if ${\lambda }_2=0$. When ${\lambda }_1=0$, the elastic net regularization rule reduces to the lasso regularization rule. When ${\lambda }_2=0$, the elastic net regularization rule reduces to the ridge regularization rule. Therefore, the elastic net regularization rule covers the lasso and ridge regularization rules as a special case. The elastic regularization rule is a generalized rule for practical applications.

The first derivative of ${\ell }^E(\Theta |\mathit{D})$ with respect to $\mathsf{\beta }$ can be obtained by

$$\begin{array}{ccc}\hfill {\displaystyle \frac{\partial {\ell }^E(\Theta |\mathit{D})}{\partial \mathsf{\beta }}}& =& -\sum _{i=1}^n\left(y_i+{\displaystyle \frac{(1-y_i)e^{{\mathsf{\beta }}^T{\mathit{x}}_i+{\mathsf{\theta }}^T{\mathit{z}}_i}}{1+e^{{\mathsf{\theta }}^T{\mathit{z}}_i}+e^{{\mathsf{\beta }}^T{\mathit{x}}_i+{\mathsf{\theta }}^T{\mathit{z}}_i}}}-{\displaystyle \frac{e^{{\mathsf{\beta }}^T{\mathit{x}}_i}}{1+e^{{\mathsf{\beta }}^T{\mathit{x}}_i}}}\right){\mathit{x}}_i\hfill \\ & +& \left({\lambda }_1\sum _{j=1}^p{\beta }_j\right)\mathbf{1}+{\mathsf{\lambda }}_2,\hfill \end{array}$$

(18)

where ${\mathsf{\lambda }}_2^T=({\lambda }_{21},{\lambda }_{22},\cdots ,{\lambda }_{2p})$. If ${\beta }_j>0$, then ${\lambda }_{2j}={\lambda }_2$; otherwise, ${\lambda }_{2j}=-{\lambda }_2$. The first derivative of ${\ell }^E(\Theta |\mathit{D})$ with respect to $\mathsf{\theta }$ can be obtained by

$$\begin{array}{c}\hfill {\displaystyle \frac{\partial {\ell }^E(\Theta |\mathit{D})}{\partial \mathsf{\theta }}}=-\sum _{i=1}^n\left((1-y_i){\displaystyle \frac{e^{{\mathsf{\theta }}^T{\mathit{z}}_i}+e^{{\mathsf{\beta }}^T{\mathit{x}}_i+{\mathsf{\theta }}^T{\mathit{z}}_i}}{1+e^{{\mathsf{\theta }}^T{\mathit{z}}_i}+e^{{\mathsf{\beta }}^T{\mathit{x}}_i+{\mathsf{\theta }}^T{\mathit{z}}_i}}}-{\displaystyle \frac{e^{{\mathsf{\theta }}^T{\mathit{z}}_i}}{1+e^{{\mathsf{\theta }}^T{\mathit{z}}_i}}}\right){\mathit{z}}_i\end{array}$$

(19)

The gradient descent method is used to obtain the estimates of $\mathsf{\beta }$ and $\mathsf{\theta }$. Let ${\mathsf{\beta }}^{\left(t\right)}$ and ${\mathsf{\theta }}^{\left(t\right)}$ denote the solutions of $\mathsf{\beta }$ and $\mathsf{\theta }$ at the tth iteration. Let

$$\begin{array}{c}\hfill ▿{\ell }_{\mathsf{\beta }}^E\left({\Theta }^{\left(t\right)}\right|\mathit{D})={\displaystyle \frac{{\ell }^E(\Theta |\mathit{D})}{\partial \mathsf{\beta }}}{|}_{\mathsf{\beta }={\mathsf{\beta }}^{\left(t\right)},\mathsf{\theta }={\mathsf{\theta }}^{\left(t\right)}}.\end{array}$$

(20)

and

$$\begin{array}{c}\hfill ▿{\ell }_{\mathsf{\theta }}^E\left({\Theta }^{\left(t\right)}\right|\mathit{D})={\displaystyle \frac{{\ell }^E(\Theta |\mathit{D})}{\partial \mathsf{\theta }}}{|}_{\beta ={\beta }^{\left(t\right)},\theta ={\theta }^{\left(t\right)}}.\end{array}$$

(21)

The values of $\beta$ and $\theta$ at the $(t+1)$th iteration can be updated by

$$\begin{array}{c}\hfill {\beta }^{(t+1)}={\beta }^{\left(t\right)}-{\gamma }_1▿{\ell }_{\beta }^E\left({\Theta }^{\left(t\right)}\right|\mathit{D})\end{array}$$

(22)

and

$$\begin{array}{c}\hfill {\theta }^{(t+1)}={\theta }^{\left(t\right)}-{\gamma }_2▿{\ell }_{\theta }^E\left({\Theta }^{\left(t\right)}\right|\mathit{D}),\end{array}$$

(23)

respectively, for $t=1,2,\cdots$; ${\gamma }_1$ and ${\gamma }_2$ are learning rates. The update process stops if the condition

$$\begin{array}{c}\hfill |{\ell }^E\left({\Theta }^{\left(t\right)}\right|\mathit{D})-{\ell }^E\left({\Theta }^{(t-1)}\right|\mathit{D})|\le ϵ\end{array}$$

(24)

is satisfied, where $ϵ$ is a small positive number. Then, ${\beta }^{\left(t\right)}$ and ${\theta }^{\left(t\right)}$ are the required solution of $\beta$ and $\theta$, respectively.

3.4. Convergence Acceleration for the Gradient Descent Method

Constant learning rates have been commonly used to implement the gradient descent method. However, implementing a gradient descent method with a constant learning rate for optimization could take more computation time to reach convergence. Improving the learning rate based on the one-step ahead estimation results to the next step can be a good idea to accelerate the convergence of the gradient descent method. In this study, the momentum learning rate method is adapted to accelerate the convergence speed of the gradient descent method. More comprehensive discussions for using the momentum term to the learning rates can be found in Yu and Chen [21], Attoh-Okine [22], Qian [23], Wang et al. [24], Liu et al. [25], Friedman et al. [26], and Simon et al. [27].

At the initial step of iteration $t=0$, let

$$\begin{array}{c}\hfill {\nu }_{\beta }^{\left(0\right)}={\gamma }_1▿{\ell }_{\beta }^E\left({\Theta }^{\left(0\right)}\right|\mathit{D})\end{array}$$

(25)

and

$$\begin{array}{c}\hfill {\nu }_{\theta }^{\left(0\right)}={\gamma }_2▿{\ell }_{\theta }^E\left({\Theta }^{\left(0\right)}\right|\mathit{D}),\end{array}$$

(26)

where ${\Theta }^{\left(0\right)}=({\beta }^{\left(0\right)},{\theta }^{\left(0\right)})$ is the initial solution vector of $\Theta$. ${\beta }^{\left(0\right)}$ and ${\theta }^{\left(0\right)}$ are the initial solutions of $\beta$ and $\theta$, respectively. For iteration $t\ge 1$, the values of ${\nu }_{\beta }^{\left(t\right)}$ and ${\nu }_{\theta }^{\left(t\right)}$ are updated by

$$\begin{array}{c}\hfill {\nu }_{\beta }^{(t+1)}=m\times {\nu }_{\beta }^{\left(t\right)}+{\gamma }_1▿{\ell }_{\beta }^E\left({\Theta }^{\left(t\right)}\right|\mathit{D})\end{array}$$

(27)

and

$$\begin{array}{c}\hfill {\nu }_{\theta }^{(t+1)}=m\times {\theta }^{\left(t\right)}+{\gamma }_2▿{\ell }_{\theta }^E\left({\Theta }^{\left(t\right)}\right|\mathit{D}),\end{array}$$

(28)

where m is the momentum term that has is a constant in [0,1). Various existing studies have verified the appropriate value of the momentum term can be $m=0.9$; see Liu et al. [25]. Then $\beta$ and $\theta$ can be updated at the iteration t by

$$\begin{array}{c}\hfill {\beta }^{(t+1)}={\beta }^{\left(t\right)}-{\nu }_{\beta }^{\left(t\right)}\end{array}$$

(29)

and

$$\begin{array}{c}\hfill {\theta }^{(t+1)}={\theta }^{\left(t\right)}-{\nu }_{\theta }^{\left(t\right)},\end{array}$$

(30)

respectively.

The optimization to minimize ${\ell }^E(\Theta |\mathit{D})$ depends on the value of ${\lambda }_1$ and ${\lambda }_2$. We use the same setting as the R package glmnet. Let ${\lambda }_1=\lambda \left({\displaystyle \frac{1-\alpha }{2}}\right)$ and ${\lambda }_2=\lambda \times \alpha$; see [26,27]. We can see that ${\ell }^E(\Theta |\mathit{D})$ reduces ${\ell }^L(\Theta |\mathit{D})$ if $\alpha =1$ and reduces to ${\ell }^R(\Theta |\mathit{D})$ if $\alpha =0$. In this study, we suggest to set $\alpha =0.5$. Then, search the value of $\lambda$ over the interval [0,1] for the optimal values of ${\lambda }_1$ and ${\lambda }_2$. We will study the performance of the ENR-ZIB method in Section 4 using the Monte Carlo simulation method.

4. Monte Carlo Simulations

To assess the quality of the proposed ENR-ZIBer estimation method, we generated zero-inflated data sets from the ZIBer model. Let $Y_i|{\mathit{x}}_i,{\mathit{z}}_1\sim B(1,{\pi }_i)$ for $i=1,2,\cdots ,n$, ${\beta }^T=({\beta }_1,{\beta }_2,\cdots ,{\beta }_8)$ and ${\mathit{x}}_i^T=(x_{i1},x_{i2},\cdots ,x_{i8})$. The logit model based on $p_i$ and ${\mathit{x}}_i$ is defined by

$$\begin{array}{c}\hfill log\left({\displaystyle \frac{p_i}{1-p_i}}\right)={\beta }^T{\mathit{x}}_i,i=1,2,...,n.\end{array}$$

where $x_{i1}=1$, $x_{i2}\sim N(0,1)$, $x_{i3}\sim B(1,0.3)$, $x_{i4}\sim N(10,1)$, and $x_{i5}\sim N(2,1)$, $x_{i6}\sim N(20,2^2)$, $x_{i7}\sim N(0,1)$, $x_{i8}\sim B(5,0.3)$, for $i=1,2,...,n$, and $N(\mu ,{\sigma }^2)$ denotes the normal distribution with $(\mathrm{mean},\mathrm{variance})=(\mu ,{\sigma }^2)$. Let ${\theta }^T=({\theta }_1,{\theta }_2,{\theta }_3)$ and ${\mathit{z}}_i^T=(z_{1i},z_{2i},z_{3i})$ The logit model based on ${\delta }_i$ and ${\mathit{z}}_i$ is defined by

$$\begin{array}{c}\hfill log\left({\displaystyle \frac{{\delta }_i}{1-{\delta }_i}}\right)={\theta }^T{\mathit{z}}_i,i=1,2,...n.\end{array}$$

where $z_{i1}=1$, $z_{i2}\sim B(1,0.6)$, and $z_{i3}\sim N(30,7^2)$, for $i=1,2,...,n$.

The values of ${\beta }^T=(2,2,3,0.5,1,0.1,4,1.5)$ and ${\theta }^T=(1,1,2)$ are used for the simulations. Before generating $Y_i$ from $B(1,{\pi }_i)$, $\{x_{ij},i=1,2,\cdots ,n\}$ and $\{z_{ih},i=1,2,\cdots ,n\}$ are standardized for each $j=2,3,4,5$ and $h=2,3$, respectively. Moreover, the coefficient vectors of $\beta$ and $\theta$ are also normalized by ${\displaystyle \frac{\beta }{{\left|\right|\beta \left|\right|}_2}}$ and ${\displaystyle \frac{\theta }{{\left|\right|\theta \left|\right|}_2}}$, respectively. The learning rates ${\gamma }_1=0.01$ and ${\gamma }_2=0.01$ are used for the gradient descent method. Moreover, the momentum term is used to update the values of ${\nu }_{\beta }$ and ${\nu }_{\theta }$ in Equations (27) and (28), respectively, at each iteration run to accelerate the convergence speed of the numerical computation.

To make the proposed ENR-ZIBer method a general model that can cover the results of the lasso and ridge regularization penalty model, k values of $\lambda \in (0,1)$ are first generated, in which $k/3$ values of $\lambda$ are used for ${\lambda }_1=0$ ($\alpha =1$), ${\lambda }_1=1$ ($\alpha =0$), and ${\lambda }_1\ne 0$ and ${\lambda }_2\ne 0$ ($\alpha =0.5$), respectively. Hence, the lasso, ridge, and elastic net regularization penalty models are included in the pool to compete with the best model. The cross-validation rule is used in this study to make the competition fair. Seventy percent of the sample was used as the training sample for establishing the target model. Then, the other third percent of the sample was used as the testing sample for the error rate evaluation. The best model with the lowest error rate can be screened out based on the testing sample among the model competition.

First, we determine how many repetitions are required to obtain a reliable error rate for the convergence of the mean error rate (MER) based on the simulation study with 5000 repetitions. The obtained MERs are reported in Figure A1. From Figure A1, we find that the MER converges at 1000 or more repetitions for the sample size 300 and 500. Hence, 1000 repetitions should be enough for the performance comparison of competitive models.

The ZIBer model is a weak classifier because it uses an analytical nonlinear function for classification. It is expected to enhance the accuracy rate of the ZIBer model by using regularization rules to penalize the model overfitting. Because the GDM works stable for large samples. Training a weak learner does not need a huge sample. However, the cross-validation rule asks to cut the original sample into training and testing samples. To achieve these two goals, at least 150 cases and up to 500 cases are considered for implementing the proposed method in simulations.

Second, the performance of the proposed ENR-ZIBer method is compared with the EM-ZIBer method and logit model for the sample sizes $n=150$, 300, and 500. 70% observations in the sample were used as the training sample to establish the models, and the other 30% observations were used as the testing sample for evaluation. The mean error rates (MERs) of the EM-ZIBer method, the logit model, and the proposed ENR-ZIBer method were evaluated based on the testing sample using the models established with the training sample. Then, the model performance is evaluated based on 1000 repetitions. All simulation results are reported in Table 1.

Table 1. The MER of the ENR-ZIBer, EM-ZIBer, and logistic regression methods for $n=150,300,500$ and $k=90,150$ in 1000 repetitions.

Method	n	k	MRE	SD *
ENR-ZIBer	150	90	0.201	0.0553
		150	0.197	0.0555
EM-ZIBer		–	0.254	0.0774
Logit		–	0.268	0.0718
ENR-ZIBer	300	90	0.209	0.0397
		150	0.207	0.0398
EM-ZIBer		–	0.248	0.0590
Logit		–	0.251	0.0456
ENR-ZIBer	500	90	0.212	0.0318
		150	0.209	0.0304
EM-ZIBer		–	0.238	0.0383
Logit		–	0.244	0.0351

* “SD” is the acronym of the standard deviation.

Table 1 shows that the MER is stable under each sample size for $k=90$ and 150. The best ENR-ZIBer model is selected from 30 (for $k=90$) and 50 (for $k=150$) lasso, ridge, and elastic net penalty regularization models, respectively. Because the cross-evaluation rule is used. The model is established based on the training sample, which contains 70% observations. The error rate is evaluated based on the testing sample, which is composed of the other 30% observation. The sample size cannot be small. A small testing sample will make the MER estimation unstable. We also find that the proposed ENR-ZIBer method works to generate a stable MER and smaller standard deviation if $k=150$. This finding will be used for the data analysis in Section 5.

The logit model in Table 1 has the largest MER compared with its competitors of the ENR-ZIBer and EM-ZIBer methods. The proposed ENR-ZIBer method with $k=150$ can well characterize zero-inflated data and outperform the EM-ZIBer method and logit model regarding the MER and its standard deviation in most cells based on 1000 repetitions. Hence, $k=150$ is recommended for the implementation of the ENR-ZIBer method. The ENR-ZIBer method can include 50 lasso (${\lambda }_1=0$ or $\alpha =1$), 50 ridge (${\lambda }_2=0$ or $\alpha =0$), and 50 elastic net (${\lambda }_1\ne 0$ and ${\lambda }_2\ne 0$; or $\alpha =1/2$) penalty regulations sets with 50 random generated values of $\lambda$, respectively, in which ${\lambda }_1=(1-\alpha )/2$ and ${\lambda }_2=\alpha /2$, to compete the best model.

5. An Example

In this section, the proposed ENR-ZIBer method is illustrated by a diabetes data set. A total of 768 cases with eight covariate variables are included in this data set. The purpose is to predict whether each case will have a positive or negative response to diabetes. Hence, the response variable is Diabetes, labeled by y, and can be either a positive ($y=1$) or negative ($y=0$) response. The covariate variables are

(1)

Pregnant ($x_2$): pregnancy numbers;

(2)

Glucose($x_3$): plasma glucose concentration based on the 2-h oral glucose tolerance test;

(3)

Pressure($x_4$): diastolic blood pressure (mm Hg);

(4)

Triceps($x_5$): Triceps skin fold thickness (mm);

(5)

Insulin($x_6$): Two-hour serum insulin ($\mu$ U/mL);

(6)

Mass($x_7$): Body mass index;

(7)

Pedigree($x_8$): Diabetes pedigree function;

(8)

Age($x_9$): The age in years.

This data set can be obtained from the R package “mlbench”. The R codes for this sample analysis are given in Appendix B.

First, we cleaned the data set to remove 376 cases that contained incomplete covariate information. After data cleaning, 392 cases are kept as the final data set for modeling. In summary for the response variable in the cleaned sample, the diabetes rate is 0.332. Following the suggestion of Chiang et al. [20], the “pressure”, “mass”, and “Age” are selected as the covariate variables of the second logit model, labeled by $z_2,z_3$, and $z_4$. The $x_1$ and $z_1$ are a m-dimensional vector with entry 1. Using the EM algorithm proposed by Chiang et al. [20], the EM-ZIBer model was established and reported in Table 2.

Table 2. The estimates of the model coefficients for the diabetes data set.

Coefficients	EM-ZIBer	Logit	ENR-ZIBer
${\beta }_1$	−7.562	−5.771	−6.615
${\beta }_2$	2.665	1.397	2.273
${\beta }_3$	6.931	5.434	6.571
${\beta }_4$	1.99	−0.122	1.79
${\beta }_5$	−0.19	0.628	−0.255
${\beta }_6$	3.34	−0.687	2.335
${\beta }_7$	5.246	3.449	4.437
${\beta }_8$	5.313	2.664	4.355
${\beta }_9$	−2.863	2.037	−2.856
${\theta }_1$	0.005	−	0.041
${\theta }_2$	2.232	−	2.236
${\theta }_3$	−1.94	−	−2.048
${\theta }_4$	−10.491	−	−10.781
${\lambda }_1$	−	−	0.049
${\lambda }_2$	−	−	0.098
Accuracy	80.6%	79.8%	81.1%

Chiang et al. [20] suggest the best cutting probability of the ZIBer is $p_i\ge 0.53$ for $i=1,2,\cdots ,392$, which results in an accuracy of 80.6% for this data set. The logit model was also established for the diabetes data set and reported at the third column of Table 2. The best cutting probability of the typical logit model $p_i\ge 0.39$, $i=1,2,\cdots ,392$, which results in the accuracy is 79.8%.

The ENR-ZIBer method with $k=150$ was established to compete with the logit model and the EM-ZIBer methods for the best model. The model coefficients of the ENR-ZIBer model are reported at the forth column of Table 2. As discussed in Section 4, the ENR-ZIBer method with $k=150$ includes 50 lasso, 50 ridge, and 50 elastic net penalty regularization models with ${\lambda }_1\ne 0$ and ${\lambda }_2\ne 0$ for the model selection to compete with the best model. The MLEs of the EM-ZIBer method were used as the initial solutions of the model parameters to implement the ENR-ZIBer method. Finally, we obtained the best model with the lowest error rate, which has $({\lambda }_1,{\lambda }_2)=(0.049,0.098)$. Moreover, the best cutting probability based on the ENR-ZIBer method is 0.45, which results in an error rate of 0.188 or accuracy 1 − 0.188 = 81.1%. The accuracy of the ENR-ZIBer model beats the logit model and the EM-ZIBer method. We also found that the Triceps($x5$) contributes less to the binary classification using the ENR-ZIB model than other covariates of $x^{\prime }s$.

6. Concluding Remarks

In this study, we established the loss function that is composed of the log-likelihood function of the ZIBer model and the $L_1$- and $L_2$-norm terms for penalty. The computation procedure to obtain reliable estimates of the proposed ENR-ZIBer method was also developed using the GDM to overcome the computation difficulties caused by the complicated target function for minimization. To accelerate the convergence of the GDM for zero-inflated data, the momentum term is used as the learning rate for the numerical computation to implement the proposed ENR-ZIBer method. We also find the ENR-ZIBer method with $k=150$ works well. This model can include the lasso and ridge regularization rules for a penalty in the pool to compete with the best model.

Monte Carlo simulations were conducted to evaluate the performance of the proposed method. Moreover, the performance of the proposed ENR-ZIBer method is compared with the EM-ZIBer method and the logit model. Simulation results show that the proposed ENR-ZIBer method with $k=150$ outperforms its competitors with the lowest MRE. We recommend using the ENR-ZIBer with $k=150$ for characterizing zero-inflated data with a feature selection consideration.

The proposed method fills the gap of the EM-ZIBer method proposed by Chiang et al. [20] to reach a compromise of high accuracy and feature selection for the zero-inflated data set. Other feasible learning rates could have similar performance as the momentum term. Moreover, using the elastic net regulation rule to other zero-inflated models can be another important topic. These two topics will be studied in the near future.