A Hybrid Algorithm with a Data Augmentation Method to Enhance the Performance of the Zero-Inflated Bernoulli Model

Abstract

The zero-inflated Bernoulli model, enhanced with elastic net regularization, effectively handles binary classification for zero-inflated datasets. This zero-inflated structure significantly contributes to data imbalance. To improve the ZIBer model’s ability to accurately identify minority classes, we explore the use of momentum and Nesterov’s gradient descent methods, particle swarm optimization, and a novel hybrid algorithm combining particle swarm optimization with Nesterov’s accelerated gradient techniques. Additionally, the synthesized minority oversampling technique is employed for data augmentation and training the model. Extensive simulations using holdout cross-validation reveal that the proposed hybrid algorithm with data augmentation excels in identifying true positive cases. Conversely, the hybrid algorithm without data augmentation is preferable when aiming for a balance between the metrics of recall and precision. Two case studies about diabetes and biopsy are provided to demonstrate the model’s effectiveness, with performance assessed through K-fold cross-validation.

1. Introduction

If a binary dataset with the responses of $Y=0$ and 1 has excess zeros, the dataset is zero-inflated. The zero-inflated Bernoulli, denoted by ZIBer, model can be used to characterize binary zero-inflated datasets. The excess zeros can be structural or chance zeros. A structural zero means that the zeros are caused by inherent constraints or conditions. Other than structural zeros, the response variable Y follows a Bernoulli distribution, in which $Y=1$ with a probability $0\le p\le 1$ and $Y=0$ with the probability $1-p$. Diop et al. [1] provided a medical example. Assume that the infection response is related to some diseases. $Y=1$ means an infected individual and $Y=0$ otherwise. An immunity agent can control if the individual is infected; that is, the individual is immune and labeled by $Y=0$ if the individual cannot be infected. See Ridout et al. [2] for more discussions about structural and chance zeros. The ZIBer model has been suggested for modeling zero-inflated datasets, with its superiority over the logistic model. To save pages, abbreviations in

Table 1. A summary of abbreviations.

ZIBer	zero-inflated Bernoulli
ZIP	zero-inflated Poisson
EM	expectation–maximization method
EM-ZIBer	ZIBer model using the EM method
ENR	elastic net regularization
ENR-ZIBer	ZIBer model using the ENR rule
GDM	gradient descent method
GDM-Mom	GDM using the momentum learning rate
SMOTE	synthesized minority oversampling technique
MK-FCM	multi-kernel Fuzzy C-Means
PSO	particle swarm optimization
G-SMOTENC	geometric SMOTE method for Nominal and Continuous features
GDM-NAG	GDM using Nesterov’s accelerated gradient factor
PSO-NAG	A new algorithm that integrates the PSO and GDM-NAG methods
PSO-NAG-SMOTE	PSO-NAG algorithm using SMOTE for data augmented sampling
TP	true positive
TN	true negative
FP	flase positive
FN	false negarive

are used in this study so that the readers can follow the arguments.

Today, datasets with a highly disproportionate distribution of categories can be seen occasionally. Such datasets are called imbalanced data in machine learning analysis. The ZIBer model can be an alternative to characterize imbalanced datasets, see Chiang et al. [3] and Xin et al. [4].

Lambert [5] was a pioneer in using the zero-inflated model with the Poisson distribution, named the ZIP model. He used the logit function to link the probability of response $Y=1$ and covariates. Moreover, the maximum likelihood estimation method was used for making statistical inferences. The ZIP model addresses random events that include an abundance of zero-count data within a given time frame. The ZIP model combines two processes that produce zeros. The first process is responsible for generating zeros, while the second process, guided by a Poisson distribution, produces counts. After that, the ZIP model has earned more attention. However, Lambert [5] mentioned that the ZIP model has the drawback of over-dispersion inherited from the Poisson model. To overcome this drawback, he also proposed the zero-inflated negative Binomial model. The zero-inflated model was also extended to the Binomial distribution, see [6,7,8,9,10,11] for more comprehensive discussions. Recent studies about using zero-inflated models for characterizing binary responses have been conducted by [12,13,14,15,16].

Compared with the ZIP, zero-inflated negative Binomial, and zero-inflated Binomial models, fewer existing studies have paid attention to the binary classification for zero-inflated datasets. Dio et al. [1] proposed a simulation-based inference method for binary responses with zero-inflated datasets. Chiang et al. [3] proposed an expectation–maximization (EM) algorithm to obtain reliable maximum likelihood estimates of the ZIBer parameters and named it EM-ZIBer. Considering an overfitting penalty for the ZIBer model, Xin et al. [4] developed a modeling process by using elastic net regularization (ENR) to prohibit model overfitting and named the model the ENR-ZIBer model. The sensitivity of the method proposed by Xin et al. [4] is still lower and unsatisfactory. There is room to improve the performance of the estimation method provided by Xin et al. [4] to that of the ZIBer model for zero-inflated datasets.

The zero-inflated structure is a crucial factor in making the data imbalanced. The ENR-ZIBer model outperforms the logistic model in characterizing zero-inflated datasets. However, there is room to enhance the performance of the ENR-ZIBer model for characterizing imbalanced datasets or keeping a high performance if the model contains many explanatory variables. Using the regularization rule can force the ENR-ZIBer model to screen out less important explanatory variables. However, the regularization rule also increases the possibility of not obtaining reliable model estimates. For more information, readers are suggested to refer to Xin et al. [4].

For zero-inflated datasets, Chiang et al. [3] showed that the EM-ZIBer model is competitive with other weak learners, for example, the light gradient boosting machine and artificial neural network methods. Xin et al. [4] showed the superiority of the ENR-ZIBer model over the EM-ZIBer model proposed by Chiang et al. [3] and the logistic model. However, the ENR-ZIBer model in Xin et al. [4] failed due to the covariates of the latent variable being difficult to identify. Moreover, Xin et al. [4] did not study the performance of the ENR-ZIBer model with data augmentation. To enhance the classification performance of the ENR-ZIBer model for zero-inflated data, machine learning algorithms could have the potential to beat the gradient descent method using the momentum learning rate, labeled as GDM-Mom, proposed by Xin et al. [4]. By the way, if the data are zero-inflated, it could make the data imbalanced. The performance of the ENR-ZIBer model needs to be verified.

Kennedy and Eberhart [17] first proposed the particle swarm optimization (PSO) method in 1995. Inspired by the collective behavior of fish schooling and birds flocking, the PSO is a population-based metaheuristic algorithm for optimization problems by iteratively updating a swarm of particles or candidate solutions. The PSO algorithm is effective for continuous optimization problems and can be implemented with a few hyperparameters. Wang [18] used the multi-kernel functions to improve Fuzzy C-Means, named MK-FCM, and then improve the performance of the PSO method.

In machine learning applications for binary-response data, the synthesized minority oversampling technique, denoted by SMOTE, is a popular method for data augmentation to deal with imbalanced data. The augmented data are then used to train the model for binary classification. When the positive class is under-represented, standard classifiers could perform biased classification on the majority class and lead to poor detection of the minority class. SMOTE is particularly effective when used in combination with ensemble methods or resampling techniques. Dablain et al. [19] proposed a novel oversampling algorithm, named DeepSMOTE, for deep learning models. Integrating the geometric SMOTE method and the SMOTE method for Nominal and Continuous features, Fonseca et al. [20] proposed a geometric SMOTE method for Nominal and Continuous features, named the G-SMOTENC method. G-SMOTENC is an oversampling method to reach a significant improvement in classification performance. Elreedy et al. [21] derived the mathematical formulation for the probability distribution of generated samples using the SMOTE method. More applications using different SMOTE methods can be found in [22,23,24,25,26,27,28,29]. To the best of the authors’ knowledge, using the SMOTE method to enhance the performance of the ENR-ZIBer model is an open issue and must be verified.

The methods of Xin et al. [4] have three weaknesses:

1. 

The ENR-ZIBer model used by Xin et al. [4] will fail if the covariates of the latent variable cannot be clearly identified. We suggest using an unknown proportion parameter to replace the second logit model used by Xin et al. [4] to enhance the performance of the ENR-ZIBer model.

2. 

The GDM-Mom method proposed by Xin et al. [4] has room to be improved to obtain reliable parameter estimates.

3. 

Xin et al. [4] did not evaluate the effect of data augmentation for the ENR-ZIBer model. SMOTE is a popular technique for data augmentation in machine learning applications. It is important to check if SMOTE can enhance the predictive performance of the ENR-ZIBer model.

To validate the quality of the competitive models used in this study, three popular methods for the cross-validation of machine learning models are considered. They are the holdout cross-validation, leave-one-out cross-validation, and K-fold cross-validation methods. In the holdout cross-validation method, a sample is randomly split into a training and test sample. It is common to keep a large proportion of the observations in the sample for the training sample and use the other observations for the test sample. Leave-one-out cross-validation keeps the hth observation for testing and uses the other for the training sample to train the machine learning model, $h=1,2,\dots ,n$. Then, the mean metric value is used to evaluate the quality of the machine learning method. The leave-one-out cross-validation method is good but time-consuming. K-fold cross-validation can be an alternative to overcome this drawback of the leave-one-out cross-validation method. K-fold cross-validation splits the sample into K equal parts. K − 1 folds are chosen for the training sample to train the machine learning model, and the quality of the machine learning model is validated based on the test sample composed of the remainder part. In this paper, the holdout cross-validation and K-fold cross-validation methods are used to assess the performance of the models used in this study.

Based on the aforementioned reasons, three tasks are finished in this study:

Task 1. 

We establish the GDM update process using the NAG factor, denoted by GDM-NAG. Monte Carlo simulations are conducted to screen out the best algorithmic methods for the implementation of the ENR-ZIBer model, from among GDM-Mom, GDM-NAG, PSO, and the proposed hybrid PSO-NAG algorithm that integrates the PSO and GDM-NAG algorithms.

Task 2. 

The selected best performance algorithms from Task 1 are used for training the ENR-ZIBer model via SMOTE data augmented sampling. Monte Carlo simulations are conducted to verify the performance of the GDM-Mom, GDM-NAG, PSO, PSO-NAG, and PSO-NAG-SMOTE algorithms in terms of proper metrics.

Task 3. 

Holdout cross-validation is used for performance assessment in the Monte Carlo simulation study. Moreover, K-fold cross-validation is used to verify the model’s quality for the real example of diabetes.

The rest of this article is organized as follows. Section 2 introduces zero-inflated data and their modeling. The ZIBer model is established in Section 2.1. The loss function with the ENR rule is defined. Moreover, the implementation of the proposed optimization process using the GDM-Mom, GDM-NAG, PSO, PSO-NAG, and PSO-NAG-SMOTE methods for the implementation of the ENR-ZIBer model is presented in Section 2.2. In Section 3, the performance of using the aforementioned algorithmic methods for the optimization process of the ENR-ZIBer model is evaluated using intensive Monte Carlo simulations. Two examples are used in Section 4 to demonstrate the applications of the proposed methods. Some concluding remarks are given in Section 5 to comment on the proposed methods. Moreover, the topics for future study are discussed.

2. The SMOTE-ZIBer Model

Let $Y_i(=0,1)$ follow a Bernoulli distribution with a vector of covariates defined by ${\mathit{x}}_i^T=(x_{1i},x_{2i},\dots ,x_{mi})$, $i=1,2,\dots ,n$ with $x_{1i}=1$. Without considering the structural zero assumption, denote the probability of $Y_i=1$ by $p_i$, and $p_i\in (0,1)$ can be linked to ${\mathit{x}}_i$ by

$$log\left({\displaystyle \frac{p_i}{1-p_i}}\right)={\mathsf{\beta }}^T{\mathit{x}}_i,i=1,2,\dots ,n,$$

(1)

where ${\mathsf{\beta }}^T=({\beta }_1,{\beta }_2,\dots ,{\beta }_m)$.

2.1. The Zero-Inflated Bernoulli Model

Assuming that some zeros in responses are structural zeros, which are determined through an unobserved or latent factor Z, $Z_i=1$ indicates that the zero in the response is a structural zero. Considering the fact that it is not easy to precisely identify the covariates correlated with the logarithm of the odds of $P(Z_i=0)$ and $P(Z_i=1)$ in practical applications, an unknown constant $\delta$ is used to denote the probability of structural zero. Then, we can have

$$\begin{array}{cc}\hfill P(Y_i=0)& =P(Z_i=1)P(Y_i=0|Z_i=1)+P(Z_i=0)P(Y_i=0|Z_i=0)\hfill \\ \hfill & =\delta \times 1+(1-\delta )\times (1-p_i)\hfill \\ \hfill & =1-(1-\delta )p_i,i=1,2,\dots n.\hfill \end{array}$$

(2)

The probability $P(Y_i=1)$ can be modified as

$$P(Y_i=1)=(1-\delta )p_i,i=1,2,\dots n,$$

(3)

after considering the structural zero assumption.

For simplicity, denote

$$\begin{array}{cc}\hfill {\pi }_i\equiv {\pi }_i\left(\mathrm{\Theta }\right)& =P(Y_i=1)=(1-\delta )p_i,i=1,2,\dots n,\hfill \end{array}$$

where $\mathrm{\Theta }=(\mathsf{\beta },\delta )$.

The loss function based on the data $\mathcal{D}=\{({\mathbf{x}}_i,y_i)$, $i=1,2,\dots ,n.\}$ can be defined by

$$\mathcal{L}\left(\mathrm{\Theta }\right)\equiv \mathcal{L}\left(\mathrm{\Theta }\right|\mathcal{D})=-\sum _{i=1}^n\left(y_{i}ln\left({\pi }_i\right)+(1-y_i)ln\left(1-{\pi }_i\right)\right).$$

(4)

Let ${\lambda }_1$ and ${\lambda }_2$ be constants for the regularization strength of $L_2$ and $L_1$ norms, respectively. The loss function with the ENR regularization rule is presented by

$$\begin{array}{cc}\hfill \mathcal{L}\left(\mathrm{\Theta }\right)& =-\sum _{i=1}^n\left(y_{i}ln\left({\pi }_i\right)+(1-y_i)ln\left(1-{\pi }_i\right)\right)+{\displaystyle \frac{{\lambda }_1}{2}}\sum _{i=1}^n{\beta }_i^2+{\lambda }_2\sum _{i=1}^n\left|{\beta }_i\right|.\hfill \end{array}$$

(5)

Because the domain of $\delta$, (0, 1), is bounded at two sides, a good strategy to obtain the minimal loss $\mathcal{L}\left(\mathrm{\Theta }\right)$ for the estimates of βs, given $\delta ={\delta }_h$, where ${\delta }_h\in {\mathrm{\Omega }}_{\delta }=\{{\delta }_1,{\delta }_2,\dots ,{\delta }_H\}\subset (0,1)$, is denoted by

$${\widehat{\mathsf{\beta }}}^{\left(h\right)}=arg\underset{\mathsf{\beta }\in {\mathrm{\Omega }}_{\beta }}{min}\mathcal{L}(\mathsf{\beta },{\delta }_h),$$

(6)

and the approximate optimal solution can be obtained as

$$(\widehat{\mathsf{\beta }},\widehat{\delta })=arg\underset{h=1,2,\dots ,H}{min}\mathcal{L}({\widehat{\mathsf{\beta }}}^{\left(h\right)},{\delta }_h).$$

(7)

Following the same setting in the R package glmnet for the values of ${\lambda }_1$ and ${\lambda }_2$, let ${\lambda }_1=\lambda \times {\displaystyle \frac{1-\alpha }{2}}$ and ${\lambda }_2=\lambda \times \alpha$. The value of $\alpha =0.5$ can be a good reference to balance the $L_1$-norm and $L_2$-norm penalties. If H is large and the domain ${\mathrm{\Omega }}_{\delta }$ is uniformly spread over the interval (0, 1), the obtained optimal solution is close to the global optimal solution.

The target function of ${\mathrm{CE}}_{\mathrm{ENR}}$ is complicated. Xin et al. [4] used the gradient descent method with the momentum learning rate to obtain the estimate of the ENR-ZIBer model. The gradient descent method is a popular method to obtain local optimal solutions for a target function. The gradient of $\mathcal{L}\left(\mathrm{\Theta }\right)$ with respect to $\mathsf{\beta }$ can be obtained by

$$\begin{array}{cc}\hfill {▿}_{\beta }\mathcal{L}\left(\mathrm{\Theta }\right)& =-\sum _{i=1}^n\left\{{\displaystyle \frac{y_i}{\pi }}-{\displaystyle \frac{1-y_i}{1-\pi }}\right\}{\displaystyle \frac{\partial {\pi }_i}{\partial \mathsf{\beta }}}+{\lambda }_1{\mathsf{\beta }}^T\mathbf{1}+{\mathit{\lambda }}_2^T\mathbf{1}\hfill \\ \hfill & =-(1-\delta )\sum _{i=1}^n\left\{{\displaystyle \frac{y_i}{\pi }}-{\displaystyle \frac{1-y_i}{1-\pi }}\right\}{p}_i(1-p_i){\mathit{x}}_i+{\lambda }_1{\mathsf{\beta }}^T\mathbf{1}+{\mathit{\lambda }}_2^T\mathbf{1}\hfill \end{array}$$

(8)

where $\overline{y}$ is the sample mean of ys, $\mathbf{1}$ is an n-dimensional vector with entry 1, and ${\mathit{\lambda }}_{\mathbf{2}}$ is an n-dimensional vector with ith entry ${\lambda }_2$ if ${\beta }_i>0$ and $-{\lambda }_2$ if ${\beta }_i<0$.

The GDM-NAG method can be obtained as follows. Let $\gamma$ be the learning rate. For iteration $(t+1)$, update ${\nu }^{\left(t\right)}$ by

$$\begin{array}{c}\hfill {\nu }^{(t+1)}=m\times {\nu }^{\left(t\right)}-\gamma {▿}_{\beta }\mathcal{L}({\mathrm{\Theta }}^{\left(t\right)}+m\times {\nu }^{\left(t\right)}),\mathrm{if}t\ge 0;\end{array}$$

(9)

where m is 0.9 and ${\nu }^{\left(0\right)}=0$. Update ${\mathsf{\beta }}^{\left(t\right)}$ by

$${\mathsf{\beta }}^{(t+1)}={\mathsf{\beta }}^{\left(t\right)}+{\nu }^{(t+1)},t=1,2,\dots ,t_{max},$$

(10)

where $t_{max}$ is the claimed maximum number of iterations for updating parameters. Replace the update formula in Equation (9) by

$$\begin{array}{c}\hfill {\nu }^{(t+1)}=m\times {\nu }^{\left(t\right)}-\gamma {▿}_{\beta }\mathcal{L}\left({\mathrm{\Theta }}^{\left(t\right)}\right),\mathrm{if}t\ge 0.\end{array}$$

(11)

The update process converts to the GDM-Mom method. Normally, GDM-NAD can perform the convergence of the minimization for the loss function quicker than the GDM-Mom method.

PSO is another competitive algorithm with the potential to have better computational efficiency than the gradient descent method. In this study, the PSO algorithm is used to obtain the optimal estimates, $\widehat{\mathsf{\beta }}$ and $\widehat{\delta }$, to minimize $\mathcal{L}(\mathsf{\beta },\delta )$, which is defined by

$$\mathcal{L}(\mathsf{\beta },\delta )=-\sum _{i=1}^n\left(y_i\times ln\left({\displaystyle \frac{{\pi }_i}{1-{\pi }_i}}\right)+ln\left(1-{\pi }_i\right)\right)+{\displaystyle \frac{{\lambda }_1}{2}}\sum _{i=1}^n{\beta }_i^2+{\lambda }_2\sum _{i=1}^n\left|{\beta }_i\right|,$$

(12)

where ${\pi }_i=(1-\delta )p_i$. The steps to implement the PSO algorithm are given below:

• The implementation of the PSO algorithm:

Initialization Step.    

Using $\mathcal{L}(\mathsf{\beta },\delta )$ in Equation (12) as the target function, initialize a population of particles as candidate solutions in the solution space and assign each particle a random velocity.

Evaluation Step.      

Evaluate the fitness of each particle using the target function. Update the personal best for each particle if the current position is better. Update the global best if any particle’s personal best is better than the current one.

Update Velocity Step.  

Update the particle’s velocity based on its current velocity, the distance to its personal best, and the global best. The velocity updating formula typically includes inertia, cognitive (individual), and social (global) components.

Update Position Step.  

Move each particle to a new position based on its updated velocity. Ensure the new positions are in the solution space.

Iteration Step.      

Repeat the evaluation, velocity, and position updating steps for a predetermined number of iterations or until a stopping criterion is met.

Let N denote the number of particles; generate ${\beta }_{i,j}\sim U({\beta }_{j,min},{\beta }_{j,max})$, $j=1,3,\dots ,m$, and ${\delta }_i\sim U(0,1)$, $i=1,2,\dots ,N$, where $U(a,b)$ is the uniform distribution over the range from a to b, and ${\beta }_{j,min}$ and ${\beta }_{j,max}$ are the minimum and maximum of ${\beta }_j$, respectively. Let $\mathit{pb}$ and $\mathit{gb}$ denote the personal best and group best positions. At the initial step, $t=0$, with ${\mathrm{\Theta }}_i^{\left(0\right)}=({\mathsf{\beta }}_i^{\left(0\right)},{\delta }_i^{\left(0\right)})$, where ${\mathsf{\beta }}_i^{\left(0\right)}={({\beta }_{i,1}^{\left(0\right)},{\beta }_{i,2}^{\left(0\right)},\dots ,{\beta }_{i,m}^{\left(0\right)})}^T$, $i=1,2,\dots ,N$.

$${\mathit{pb}}_i^{\left(0\right)}={\mathrm{\Theta }}_i^{\left(0\right)},i=1,2,\dots ,N.$$

(13)

Evaluate the value of $\mathcal{L}\left({\mathrm{\Theta }}_i^{\left(0\right)}\right)$. Let

$${\mathit{gb}}^{\left(0\right)}=\underset{i=1,2,\dots ,N}{min}\mathcal{L}\left({\mathrm{\Theta }}_i^{\left(0\right)}\right).$$

(14)

Update the velocity vector ${\mathit{\upsilon }}_i={({\upsilon }_{i,1},{\upsilon }_{i,2},\dots ,{\upsilon }_{i,m+1})}^T$ and $\mathrm{\Theta }$ at iteration $t+1$ by

$$\begin{array}{}\hfill \mathrm{(15)}& \begin{array}{cc}\hfill {\mathit{\upsilon }}_i^{(t+1)}& =w^{\left(t\right)}{\mathit{\upsilon }}_i^{\left(t\right)}+c_1{R}_1({\mathit{pb}}_i^{\left(t\right)}-{\mathrm{\Theta }}_i^{\left(t\right)})+c_2{R}_2({\mathit{gb}}^{\left(t\right)}-{\mathrm{\Theta }}_i^{\left(t\right)}),i=1,2,\dots ,N,\hfill \end{array}\hfill \hfill \mathrm{(16)}& \begin{array}{cc}\hfill {\mathrm{\Theta }}_i^{(t+1)}& ={\mathrm{\Theta }}_i^{\left(t\right)}+{\mathit{\upsilon }}_i^{(t+1)},i=1,2,\dots ,N,\hfill \end{array}\hfill \end{array}$$

where $c_1>0$ and $c_2>0$ are the cognitive and social coefficients, respectively; $R_1$ and $R_2$ are generated from $U(0,1)$; and $w^{\left(t\right)}$ is the weight used to reduce ${\upsilon }_i^{\left(t\right)}$. The initial velocity ${\upsilon }_i^{\left(0\right)}$ can be zero or generated from a uniform distribution. In practical applications, $w^{\left(t\right)}$ can be evaluated by

$$w^{\left(t\right)}=w_{max}-{\displaystyle \frac{w_{max}-w_{min}}{t_{max}}}\times t,$$

(17)

where $w_{min}$ and $w_{max}$ are the predetermined minimum and maximum of the weights. Moreover, $t_{max}$ is the maximal iteration to run PSO. While searching for the optimal solution, we can give upper bounds, ${\upsilon }_j^{max}$, $j=1,2,\dots ,m$, to the velocity by the following:

• If ${\upsilon }_{i,j}^{(t+1)}>v_j^{max}$, then ${\upsilon }_{i,j}^{(t+1)}={\upsilon }_j^{max}$, $i=1,2,\dots ,N$;
• If ${\upsilon }_{i,j}^{(t+1)}<-{\upsilon }_j^{max}$, then ${\upsilon }_{i,j}^{(t+1)}=-{\upsilon }_j^{max}$, $i=1,2,\dots ,N$.

• The ${\upsilon }_j^{max}$ can be considered by

$${\upsilon }_j^{max}=\ell \times {\displaystyle \frac{{\mathrm{\Theta }}_{j,max}-{\mathrm{\Theta }}_{j,min}}{2}},\ell \in (0,1].$$

(18)

Update $\mathit{pb}$ and $\mathit{gb}$ at iteration $t+1$ by the following:

• If $\mathcal{L}\left({\mathrm{\Theta }}_i^{(t+1)}\right)\le \mathcal{L}\left({\mathit{pb}}_i^{\left(t\right)}\right)$, then ${\mathit{pb}}_i^{(t+1)}={\mathrm{\Theta }}_i^{(t+1)}$, $i=1,2,\dots ,N$.
• If $\mathcal{L}\left({\mathrm{\Theta }}_i^{(t+1)}\right)\le \mathcal{L}\left({\mathit{gb}}^{\left(t\right)}\right)$, then ${\mathit{gb}}^{(t+1)}={\mathrm{\Theta }}_i^{(t+1)}$, $i=1,2,\dots ,N$.

2.2. The PSO-NAG and PSO-NAG-SMOTE Algorithms

For an imbalanced dataset with majority and minority classes, SMOTE can be a data augmentation technique for the minority class. First, SMOTE selects a minority class instance, say A, at random. Its k nearest neighbors are found in the minority class. The synthetic instance can be created by choosing one of the k nearest neighbors, say B, at random. Then, A and B are connected to form a line segment in the feature space. The synthetic instances are generated as a convex combination of instances A and B.

At the beginning, partition the dataset into one training and one test dataset, respectively. Use SMOTE with the training dataset for data augmentation for the minority class, and then use the augmented dataset to train the model; that is, minimize the target loss function. There are two integrated algorithms that will be introduced to implement the ENR-ZIBer model. The first is the PSO-NAG algorithm, which can be implemented using the following steps.

• The implementation of the PSO-NAG algorithm:

Step 1. 

Use the PSO algorithm described in Section 2.1 to obtain 100 estimates $({\widehat{\mathsf{\beta }}}^{\left(j\right)},{\widehat{\delta }}^{\left(j\right)})$, $j=1,2,\dots ,100$.

Step 2. 

Use the median of 100 estimates of ${\widehat{\delta }}^{\left(j\right)}$, $j=1,2,\dots ,100$ as the optimal estimate of $\delta$, and denote it by $\widehat{\delta }$.

Step 3. 

Use the GDM-NAG updating method mentioned in Section 2.1 to obtain the estimate $\widehat{\mathsf{\beta }}$, which minimizes the loss function defined by

$$\begin{array}{cc}\hfill \mathcal{L}(\mathrm{\Theta },\widehat{\delta })& =-\sum _{i=1}^n\left(y_{i}ln\left({\pi }_i^{*}\right)+(1-y_i)ln(1-{\pi }_i^{*})\right)+{\displaystyle \frac{{\lambda }_1}{2}}\sum _{i=1}^n{\beta }_i^2+{\lambda }_2\sum _{i=1}^n\left|{\beta }_i\right|,\hfill \end{array}$$

(19)

where ${\pi }_i^{*}=(1-\widehat{\delta })\times p_i$, and $p_i={\displaystyle \frac{e^{{\mathsf{\beta }}^T{\mathit{x}}_i}}{1+e^{{\mathsf{\beta }}^T{\mathit{x}}_i}}}$, $i=1,2,\dots ,n$.

The PSO-NAG-SMOTE algorithm is similar to the PSO-NAG algorithm. The only difference is adding one more step for data augmentation before using the PSO-NAG algorithm. The PSO-NAG-SMOTE algorithm can be implemented with the following steps.

Data Augmentation Step. 

Use the SMOTE method to augment the data.

Following Steps.     

Use the augmented data to train the ENR-ZIBer model based on Step 1 to Step 3 of the PSO-NAG algorithm.

Cut the data $\mathcal{D}$ into training and test samples. Using the training sample to train the ENR-ZIBer model. The contingency table of the true and predicted results is given in

Table 2. The confusion matrix.

		True
		Positive	Negative
Predicted	Positive	TP	FP	$Precision={\displaystyle \frac{TP}{TP+FP}}$
	Negative	FN	TN
		$Recall={\displaystyle \frac{TP}{TP+FN}}$		$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$

. In Table 2, $TP$, $TN$, $FP$, and $FN$ denote the number of true positive cases, true negative cases, false positive cases, and false negative cases, respectively. Then, we can define the metrics of accuracy, recall (sensitivity), precision, and $F_1$-score, respectively. After the ENR-ZIBer model is established using the training sample, the test sample is used to evaluate the classification performance based on the following metrics:

$$\begin{array}{cc}\hfill Accuracy& ={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}},\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill Recall\left(Sensitivity\right)& ={\displaystyle \frac{TP}{TP+FN}},\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill Precision& ={\displaystyle \frac{TP}{TP+FP}},\hfill \end{array}$$

(22)

and

$$\begin{array}{c}\hfill F_1-\mathrm{score}={\displaystyle \frac{2}{\frac{1}{Recall}+\frac{1}{Precision}}}=2\times {\displaystyle \frac{Recall\times Precision}{Recall+Precision}}.\end{array}$$

(23)

Accuracy measures the rate of correct classification of the whole sample. Accuracy can work well for balanced classes; however, it can be misleading if the dataset is imbalanced. Recall is used to measure how well the model identifies TP cases. A high recall indicates fewer FN cases. Recall can be an important metric for medical diagnoses or fraud detection. Precision emphasizes the rate of positive predictions based on the group with cases that are predicted as positive. A high precision indicates fewer FP cases, which is crucial in spam detection or legal cases. The $F_1$-score is the harmonic mean of the precision and recall metrics. $F_1$-score can well balance the FP and FN cases.

3. Performance Evaluation for the ENR-ZIBer Model

In this section, an intensive Monte Carlo simulation study is conducted to verify the classification performance of the proposed methods based on zero-inflated data with a medium and high probability of structural zeros. The parameters for the Monte Carlo simulation are addressed as follows.

• The coefficient parameter ${\mathsf{\beta }}^T=(2, 2, 3, 0.5, 1, 0.1, 4, 1.5)$ is used as a logit link to generate the random variable of the positive responses and the chance zeros.
• The values of $\delta =0.5, 0.7$, and 0.85 are used to generate zero-inflated data with different levels of imbalance. The proportion of the minority group is about 31%, 19%, and 9% in a sample for $\delta =0.5, 0.7$, and 0.85, respectively.
• The explanatory variables are generated based on the following distributions: $x_{1,j}=1$, $x_{2,j}\sim N(0,1)$, $x_{3,j}\sim B(1,0.3)$, $x_{4,j}\sim N(10,1)$, $x_{5,j}\sim N(2,1)$, $x_{6,j}\sim N(20,2)$, $x_{7,j}\sim N(0,1)$, and $x_{8,j}\sim B(5,0,3)$, $j=1,2,\dots ,n$. $N(\mu ,\sigma )$ denotes the normal distribution with the mean $\mu$ and standard deviation $\sigma$; $B(s,p_s)$ denotes the Binomial distribution with the number of trials s and success probability $p_s$.
• The sample sizes $n=300$, 500, and 800 for $\delta =0.5$ and 0.7; and $n=500$ and 800 for $\delta =0.85$. For an imbalanced sample, we need a bigger sample to establish the model such that the test sample can contain more positive cases for performance evaluation. The proportion of the minority group is smaller than 10%. Hence, we keep the sample size at at least $n=500$ for the case of $\delta =0.85$.
• The number of iterations to evaluate metrics is 1000.
• The learning rate $\gamma =0.1$ is used to implement the GDM-Mom and GDM-NAG methods.
• The penalty coefficients are given by $\alpha =0.5$ and $\lambda =0.3$ for regularization.
• The searching range of ${\beta }_i$ is from 1 to 10, and the searching range of $\delta$ is from 0 to 1.
• Random samples are generated from the Bernoulli distribution with a successful probability of

  $${\pi }_i=(1-\delta )\times {\displaystyle \frac{e^{{\mathsf{\beta }}^T{\mathit{x}}_i}}{1+e^{{\mathsf{\beta }}^T{\mathit{x}}_i}}},i=1,2,\dots ,n.$$
• The maximum number of iterations to implement the PSO algorithm is 1000.
• The weight at iteration t is set to be $w^{\left(t\right)}={\displaystyle \frac{1}{2log\left(2\right)}}=0.721$ and $c_1=c_2={\displaystyle \frac{1}{2}}+log\left(2\right)=0,1.193$.
• The swarm size s is the number of particles in the swarm. In this study, it is given by $s=floor(10+2\sqrt{thelengthofparameters})$, where floor(x) returns a value that is the largest integer not greater than x. More particles in the swarm result in a larger initial diversity of the swarm to allow larger parts of the search space to be covered in each iteration. However, more particles also increase the computational complexity in each iteration. The empirical heuristic value of s is in the interval of [10, 30].
• The holdout cross-validation method is used for evaluating the model’s performance. For each generated sample, 70% of observations are randomly selected from the sample for the training sample to train models, and the other 30 observations are for the test sample.

• The metrics of accuracy, recall, precision, and $F_1$-score defined in Equations (20)–(23) are used to compare the performance of the methods considered. R codes are prepared to implement the GDM-Mom and GDM-NAG methods to update the model parameters. The package psooptim in the library pso is used to implement the PSO algorithm. First, we focus on the metric of recall, which is the probability of correctly identifying if the positive case is truly positive. The violin plots of recall based on 1000 simulation runs for various methods are displayed in Figure 1. The violin plots are a hybrid plots of the box plots and kernel density plots that are used to indicate summary statistics and the density of variables.

From Figure 1, we can see that the GDM-Mom, GDM-NAG, and PSO-NAG methods have similar shapes and are competitive in terms of the recall metric. The PSO method can compute more quickly than all the GDMs. However, its performance in recall is slightly worse than the GDM-Mom, GDM-NAG, and PSO-NAG methods with a narrower spread range. To overcome the time-consuming drawback of gradient descent methods, we suggest obtaining the estimate of $\delta$ based on the PSO method and then running the GDM-NAG method by replacing $\delta$ with the mean of 100 PSO estimates of $\delta$. The method is named the PSO-NAG method. The PSO-NAG method can run more quickly than the GDM-NAG method.

In view of Figure 1, we also find that data augmentation can help the ZIBer model to enhance the value of recall; that is, the ability of the ZIBer model to correctly identify if the positive case is truly positive can be enhanced after using the SMOTE method for data augmentation. Figure 1 shows that the PSO-NAG-SMOTE method outperforms all the competitors in terms of a high value of recall and the narrowest spread range.

Figure 1. The violins of recall based on 1000 simulation runs for various methods.

Figure 1. The violins of recall based on 1000 simulation runs for various methods.

Recall is not the only metric to evaluate the ability to correctly identify the TP cases. We carefully study the ability to correctly identify the TP cases using the metrics of recall, precision, and their integrated metric, the $F_1$-score metric. All simulation results are reported in

Table 3. The mean and standard deviation of the metrics of accuracy, recall, precision, and $F_1$-score based on 1000 simulation runs with $\delta =0.5$.

		Mean				Standard Deviation
$\mathit{n}$	Method	Accuracy	Recall	Precision	${\mathit{F}}_{\mathbf{1}}$	Accuracy	Recall	Precision	${\mathit{F}}_{\mathbf{1}}$
300	GDM-Mom	0.6737	0.8836	0.4892	0.6241	0.0640	0.0833	0.0747	0.0614
300	GDM-NAG	0.6744	0.8851	0.4894	0.6249	0.0631	0.0817	0.0741	0.0617
300	PSO	0.5954	0.8152	0.4282	0.5571	0.0684	0.0894	0.0732	0.0701
300	PSO-NAG	0.6626	0.8950	0.4828	0.6221	0.0645	0.0740	0.0744	0.0618
300	PSO-NAG-SMOTE	0.6442	0.9141	0.4674	0.6140	0.0692	0.0624	0.0752	0.0670
500	GDM-Mom	0.6697	0.8923	0.4848	0.6248	0.0494	0.0679	0.0582	0.0486
500	GDM-NAG	0.6701	0.8945	0.4848	0.6256	0.0476	0.0643	0.0575	0.0486
500	PSO	0.5906	0.8121	0.4203	0.5509	0.0600	0.0787	0.0586	0.0570
500	PSO-NAG	0.6643	0.8983	0.4793	0.6221	0.0473	0.0633	0.0551	0.0472
500	PSO-NAG-SMOTE	0.6429	0.9240	0.4654	0.6164	0.0519	0.0484	0.0562	0.0500
800	GDM-Mom	0.6682	0.8955	0.4839	0.6260	0.0396	0.0574	0.0459	0.0381
800	GDM-NAG	0.6669	0.8994	0.4829	0.6262	0.0402	0.0547	0.0462	0.0381
800	PSO	0.5882	0.8038	0.4181	0.5477	0.0523	0.0699	0.0473	0.0454
800	PSO-NAG	0.6645	0.9029	0.4820	0.6266	0.0385	0.0504	0.0434	0.0369
800	PSO-NAG-SMOTE	0.6396	0.9282	0.4601	0.6145	0.0390	0.0373	0.0444	0.0406

,

Table 4. The mean and standard deviation of the metrics of accuracy, recall, precision, and $F_1$-score based on 1000 simulation runs s with $\delta =0.7$.

		Mean				Standard Deviation
$\mathit{n}$	Method	Accuracy	Recall	Precision	${\mathit{F}}_{\mathbf{1}}$	Accuracy	Recall	Precision	${\mathit{F}}_{\mathbf{1}}$
300	GDM-Mom	0.6260	0.8076	0.3182	0.4471	0.0968	0.1377	0.0720	0.0704
300	GDM-NAG	0.6249	0.8080	0.3176	0.4463	0.0956	0.1377	0.0737	0.0709
300	PSO	0.5306	0.8119	0.2639	0.3940	0.0759	0.1078	0.0635	0.0765
300	PSO-NAG	0.6170	0.8058	0.3150	0.4417	0.1056	0.1433	0.0788	0.0742
300	PSO-NAG-SMOTE	0.5351	0.8972	0.2781	0.4194	0.0936	0.0888	0.0664	0.0771
500	GDM-Mom	0.6074	0.8338	0.3049	0.4407	0.0788	0.1120	0.0570	0.0587
500	GDM-NAG	0.6038	0.8398	0.3030	0.4403	0.0774	0.1061	0.0556	0.0592
500	PSO	0.5265	0.8031	0.2562	0.3855	0.0659	0.0919	0.0488	0.0589
500	PSO-NAG	0.6083	0.8238	0.3037	0.4370	0.0834	0.1177	0.0585	0.0589
500	PSO-NAG-SMOTE	0.5424	0.9124	0.2811	0.4272	0.0597	0.0612	0.0472	0.0557
800	GDM-Mom	0.5994	0.8424	0.2998	0.4387	0.0626	0.0948	0.0439	0.0468
800	GDM-NAG	0.5946	0.8543	0.2982	0.4392	0.0590	0.0847	0.0427	0.0467
800	PSO	0.5271	0.7926	0.2559	0.3847	0.0619	0.0806	0.0395	0.0471
800	PSO-NAG	0.5997	0.8336	0.3011	0.4381	0.0682	0.1021	0.0445	0.0459
800	PSO-NAG-SMOTE	0.5333	0.9248	0.2763	0.4239	0.0471	0.0477	0.0370	0.0442

and

Table 5. The mean and standard deviation of the metrics of accuracy, recall, precision, and $F_1$-score based on 1000 simulation runs s with $\delta =0.85$.

		Mean				Standard Deviation
$\mathit{n}$	Method	Accuracy	Recall	Precision	${\mathit{F}}_{\mathbf{1}}$	Accuracy	Recall	Precision	${\mathit{F}}_{\mathbf{1}}$
500	GDM-Mom	0.6114	0.7349	0.1681	0.2618	0.1366	0.1800	0.0645	0.0629
500	GDM-NAG	0.6119	0.7341	0.1680	0.2619	0.1363	0.1787	0.0632	0.0634
500	PSO	0.4819	0.8154	0.1311	0.2237	0.0763	0.1159	0.0369	0.0557
500	PSO-NAG	0.6596	0.6399	0.1865	0.2646	0.1599	0.2164	0.0798	0.0606
500	PSO-NAG-SMOTE	0.4744	0.8761	0.1377	0.2357	0.0834	0.1008	0.0370	0.0556
800	GDM-Mom	0.6025	0.7489	0.1625	0.2616	0.1060	0.1523	0.0382	0.0465
800	GDM-NAG	0.5963	0.7561	0.1611	0.2605	0.1062	0.1495	0.0369	0.0461
800	PSO	0.4857	0.8001	0.1324	0.2259	0.0707	0.0980	0.0286	0.0428
800	PSO-NAG	0.6387	0.6655	0.1765	0.2628	0.1417	0.2003	0.0576	0.0502
800	PSO-NAG-SMOTE	0.4597	0.9033	0.1380	0.2381	0.0649	0.0710	0.0294	0.0443

. As we have mentioned in the parameter settings, only $n=500$ and $n=800$ are used for the case of $\delta =0.85$ for Table 5 to allow the test sample to have more cases with $Y=1$ for model evaluation. Different from the screening process suggested by Xin et al. [4], we use $F_1$-score instead of accuracy to determine the cut point. If we focus on the proportion of TP, high recall and precision mean that most TP cases have been successfully identified and most predicted positive cases are truly positive, respectively. A method with a high recall and precision is expected; otherwise, a balance between recall and precision is needed. Hence, the metric $F_1$-score is considered in this study to replace recall for assessing classification performance.

From Table 3, Table 4 and Table 5, we find the following results.

1.

The classification quality of the ENR-ZIBer model depends on the level of imbalance. The mean recall columns in Table 3, Table 4 and Table 5 report high values, which indicates that the ENR-ZIBer model have a good ability to correctly identify the TP cases. However, we also find that the value of precision is significantly lower than the value of tecall.

2.

The violin plots of precision based on 1000 simulation runs for various methods shown in Figure 2 also indicate that the values of precision are lower than the values of recall shown in Figure 1. Figure 2 shows that the PSO-NAG method slightly outperforms the other competitors with a higher precision metric. The findings indicate that enhancing the ability to identify the TP cases through using the SMOTE method also incurs more FP cases in classification. Table 5 shows that when the proportion of the minority group is lower than 10%, the precision significantly drops from the corresponding figures in Table 3 and Table 4. This is a trade-off to enhance the recall of the ENR-ZIBer model by using the SMOTE method for data augmentation.

3.

The rule of using a high $F_1$-score to screen out the best model hurts the accuracy. It is also important to have a compromise between correctly identifying the TP cases and keeping a high rate of TP prediction. Considering a higher value of the $F_1$-score as the metric to screen out the best model, we can find that the GDM-NAG algorithm outperforms the GDM-Mom and PSO algorithms for most of the cells in Table 3, Table 4 and Table 5. Moreover, we can take advantage of the PSO algorithm, which can save computation time to obtain a reliable estimate of $\delta$. The PSO-NAG method is recommended due to the fact that it can save more computation time than the GDM-Mom and GDM-NAG methods.

4

Table 3, Table 4 and Table 5 show that the GDM-NAG method produces a similar estimation quality to that obtained by using the GDM-Mom method. The PSO algorithm works most efficiently to save computation time. However, its performance is inferior to the GDM-NAG and GDM-Mom methods. The PSO-NAG algorithm uses the PSO algorithm to obtain the estimate of $\delta$ based on 100 iterations, and then to obtain the estimate of $\mathsf{\beta }$ using the GDM-NAG method and the PSO estimate of $\delta$. Table 3, Table 4 and Table 5 show that the PSO-NAG algorithm can be a good method to have a compromise between having a high $F_1$-score and saving computation time (Figure 3).

5.

The SMOTE method significantly enhances the value of recall but also decreases the value of precision. After balancing using $F_1$-score, we find that the proposed PSO-NAG method beats the PSO-NAG-SMOTE method. SMOTE is a popular technique to promote the performance of machine learning tools for imbalanced data. However, we find that the trade-off of using the SMOTE method for the ENR-ZIBer model is to enhance the value of recall but decrease the value of precision. Moreover, the PSO-NAG-SMOTE method consumes more computation time to work. The benefit of using SMOTE for the PSO-NAG algorithm in the ENR-ZIBer model is insignificant if the payoff for precision is considered. If dealing with FP cases is not an issue for practical applications, the PSO-NAG-SMOTE method is recommended.

6.

Table 3, Table 4 and Table 5 show that increasing the sample size helps to decrease the standard deviation of the metrics from the simulation study. Hence, reliable estimation results depend on the sample size. Basically, at least 300 observations in a sample are recommended to implement the ZIBer model if the level of zero-inflated data and imbalances is moderate. If the level of zero-inflated data and imbalances is extremely high, then at least 500 observations in a sample are recommended to implement the ZIBer model.

In summary, if dealing with FP cases is not an issue for practical applications, the PSO-NAG-SMOTE method is recommended. If we consider a balance between the metrics of recall and precision to implement the ENR-ZIBer model, the PSO-NAG method is recommended based on the $F_1$-score. The recall value from PSO-NAG is significantly high overall in the simulation results. The findings indicate that the proposed hybrid algorithm with SMOTE for data augmentation can significantly enhance the ability to correctly identify the TP cases.

4. Examples

In this section, two examples are given for illustration. The first example is about Pima Indian diabetes, and the second example is about the biopsy of breast tumors.

4.1. Example 1

In Example 1, a Pima Indian diabetes dataset with 768 cases is used to demonstrate the applications of the proposed methods. This dataset, which contains eight explanatory variables for the response of Pima Indians, has a positive response for diabetes or not. The explanatory variables are defined as follows:

$x_2$:

The number of pregnancies.

$x_3$:

The plasma glucose concentration based on the 2 h oral glucose tolerance test numbers.

$x_4$:

The diastolic blood pressure in mm Hg.

$x_5$:

The triceps skin fold thickness in mm.

$x_6$:

The two-hour serum insulin in $\mu$ U/mL.

$x_7$:

The body mass index.

$x_8$:

The diabetes pedigree function.

$x_9$:

The age in years.

y:

The response variable: $y=1$ or 0 denotes that the Pima Indian has a positive or negative response for diabetes, respectively.

The readers can retrieve the dataset from the R package “mlbench” from R 4.5.0 with the code data(PimaIndiansDiabetes, package = “mlbench”).

The dataset does not have missing cells, and the diabetes rate is about 34.9% in this dataset. Please note that the dataset in this study is different from the one in Xin et al. [4]. They used the data “PimaIndiansDiabetes2” in the package mlbench, and we use the data “PimaIndiansDiabetes” in the package mlbench. Moreover, we do not use the second logistic model in this study. Hence, the estimation results cannot be compared with the results that were obtained by Xin et al. [4]. In this section, the 5-fold cross-validation method (K = 5) is used to assess the quality of the models. In Section 3, we already mentioned that PSO-NAG could help to save computation time with good performance. In this section, we use the PSO, PSO-Mom, PSO-NAG, and PSO-NAG-SMOTE methods for the modeling of the Pima Indian diabetes dataset. PSO-Mom means that the $\delta$ is estimated based on the median of 100 PSO estimates of the $\delta$, and then GDM-Mon is used to update parameter $\beta$ by replacing $\delta$ with the PSO estimate. For simplicity, we denote the PSO estimate of $\delta$ by $\delta$-PSO. The parameter settings for the PSO and SMOTE methods are the same as those given in Section 3.

The estimates of $\delta$ for 5-fold estimation are 0.5384, 0.5124, 0.46715, 0.62145, and 0.31255. Their mean value is 0.49, which is the PSO estimate of $\delta$. The mean values of the accuracy, recall, precision, and $F_1$-score of the PSO, PSO-Mom, PSO-NAG, and PSO-NAG-SMOTE methods are given in

Table 6. The mean of accuracy, recall, precision, and $F_1$-score based on the Pima Indian diabetes dataset using 5-fold estimation results.

Method	Accuracy	Recall	Precision	${\mathit{F}}_1$-Score
PSO	0.669	0.761	0.519	0.616
PSO-Mom	0.710	0.863	0.558	0.675
PSO-NAG	0.707	0.851	0.563	0.672
PSO-NAG-SMOTE	0.682	0.885	0.531	0.661

. The hybrid algorithms of PSO-Mom and PSO-NAG are recommended based on the $F_1$-score. Moreover, the PSO-NAG-SMOTE method is recommended based on the metric of recall. The ENR-ZIBer model with the PSO-NAG-SMOTE method has a good ability to correctly predict the true positive cases.

Based on the full dataset with $\delta =0.49$ and using the PSO-NAG method, the estimated ENR-ZIBer model can be obtained by

$$\begin{array}{cc}\hfill log\left({\displaystyle \frac{p_i}{1-p_i}}\right)& =3.3587+0.8707x_{i,2}+2.4353x_{i,3}-1.0721x_{i,4}+0.6627x_{i,5}-0.9416x_{i,6}\hfill \\ \hfill & +1.701x_{i,7}+0.747x_{i,8}+3.943x_{i,9},i=1,2,\dots ,768.\hfill \end{array}$$

(24)

4.2. Example 2

A biopsy dataset about breast cancer is used for illustration. This dataset was released by Dr. William H. Wolberg from the University of Wisconsin Hospitals, Madison. All biopsies of breast tumors were assessed for 699 patients up to 15 July 1992. In this dataset, nine attributes, except for the ID and response variable, have been scored on a scale of 1 to 10 and given as follows:

ID: 

sample code number (not unique).

x₂: 

clump thickness.

x₃: 

uniformity of cell size.

x₄: 

uniformity of cell shape.

x₅: 

marginal adhesion.

x₆: 

single epithelial cell size.

x₇: 

bare nuclei (16 values are missing).

x₈: 

bland chromatin.

x₉: 

normal nucleoli.

x₁₀: 

mitoses

y: 

“benign” (y = 0) or “malignant” (y = 1).

This dataset can be retrieved from the R package “MASS” from R 4.5.0 using the code data(biopsy, package = “MASS”). There are 16 missing values in bare nuclei. After removing the missing value cases, this dataset has 683 cases, each of which has 9 attributes and the response variable for data analysis.

The estimates of $\delta$ for 5-fold estimation are 0.05495, 0.0537, 0.0484, 0.05875, and 0.085. Their mean value is 0.06, which is the PSO estimate of $\delta$. Because $\delta$ is close to zero, this means the structural zero effect is insignificant. In the biopsy dataset, we can find that the performance of the ENR-ZIBer model is excellent in terms of all metrics (

Table 7. The mean of accuracy, recall, precision, and $F_1$-score based on the biopsy dataset using 5-fold estimation results.

Method	Accuracy	Recall	Precision	${\mathit{F}}_1$-Score
PSO	0.975	0.992	0.941	0.966
PSO-Mom	0.976	0.988	0.949	0.967
PSO-NAG	0.976	0.998	0.942	0.968
PSO-NAG-SMOTE	0.976	0.992	0.945	0.967

).

Using $\delta =0.06$ and the PSO-NAG method, the estimated ENR-ZIBer model based on the full dataset can be obtained by

$$\begin{array}{cc}\hfill log\left({\displaystyle \frac{p_i}{1-p_i}}\right)& =-0.3971+2.0635x_{i,2}+1.8843x_{i,3}+0.2503x_{i,4}+0.9914x_{i,5}-0.024x_{i,6}\hfill \\ \hfill & +2.6169x_{i,7}+0.6191x_{i,8}+0.9361x_{i,9}+0.7805x_{i,10},i=1,2,\dots ,683.\hfill \end{array}$$

(25)

We can find that the coefficient of $x_6$ is small and indicates that the attribute $x_6$ (single epithelial cell size) is insignificant in this example.

5. Concluding Remarks

In this study, we investigated the estimation quality of using the machine learning methods of PSO, GDM-Mom, GDM-NAG, PSO-NAG, and PSO-NAG-SMOTE to establish the ENR-ZIBer model when the dataset is zero-inflated or imbalanced. Monte Carlo simulations were conducted to verify the classification performance of the ENR-ZIBer model based on the holdout cross-validation method. The purpose is to find the best competitive machine learning method for the prediction of the ENR-ZIBer model and save computation time. The simulation results have shown that the hybrid algorithm, by integrating the PSO and GDM-NAG algorithms with two steps, is the most competitive in balancing the metrics of recall and precision.

The performance of the SMOTE method for data augmentation for the zero-inflated data was also studied in this manuscript. The PSO-NAG-SMOTE method enhances the ENR-ZIBer model’s ability to identify TP cases. However, the augmented data based on the SMOTE method also result in more FP cases in the prediction results using the ENR-ZIBer model. This is a trade-off in using the SMOTE method for the hybrid algorithm of PSO-NAG. In practical applications, the PSO-NAG-SMOTE method can be used to establish the ENR-ZIBer model if the issue of FP cases is not a concern. Only the PSO-NAD-SMOTE algorithm for simulations with 1000 iterations is time-consuming. In practical applications for small to moderate sample sizes, the computation time is acceptable.

Two datasets about Pima Indian diabetes and biopsy were used to illustrate the applications of the proposed method. In this example, the PSO, PSO-Mom, PSO-NAG, and PSO-NAG-SMOTE methods are used to establish the ENR-ZIBer model, and the performance is evaluated via utilizing the 5-fold cross-validation method. As we found in the simulation study, the hybrid algorithm outperforms the other competitors based on the $F_1$-score metric. We also find the good ability of the ENR-ZIBer model to correctly predict the true positive cases.

Simulation results show that using the proposed PSO-NAG-SMOTE hybrid algorithms can enhance the recall of the ENR-ZIBer model. The findings indicate that the proposed hybrid algorithm with SMOTE for data augmentation can significantly enhance the ability to correctly identify the TP cases. How to reduce the possibility of FP cases in the prediction based on the data augmentation method is important. The ZIBer model uses a sigmoid function as a link function. It is easy to implement, but its performance has room to improve. The proposed method asks for using the median of 100 PSO estimates as the estimate of $\delta$. It will spend more time on parameter estimation. To enhance the classification performance of the ENR-ZIBer model, how to combine the ENR-ZIBer model with other competitive machine learning methods is also interesting. These two topics are still open and will be studied in the future.