K-L Estimator: Dealing with Multicollinearity in the Logistic Regression Model

Abstract

Multicollinearity negatively affects the efficiency of the maximum likelihood estimator (MLE) in both the linear and generalized linear models. The Kibria and Lukman estimator (KLE) was developed as an alternative to the MLE to handle multicollinearity for the linear regression model. In this study, we proposed the Logistic Kibria-Lukman estimator (LKLE) to handle multicollinearity for the logistic regression model. We theoretically established the superiority condition of this new estimator over the MLE, the logistic ridge estimator (LRE), the logistic Liu estimator (LLE), the logistic Liu-type estimator (LLTE) and the logistic two-parameter estimator (LTPE) using the mean squared error criteria. The theoretical conditions were validated using a real-life dataset, and the results showed that the conditions were satisfied. Finally, a simulation and the real-life results showed that the new estimator outperformed the other considered estimators. However, the performance of the estimators was contingent on the adopted shrinkage parameter estimators.

1. Introduction

Frisch [1] coined the term “multicollinearity” to describe the problem that occurs when the explanatory variables in a model are linearly related. This problem posed a severe threat to different regression models, e.g., the linear regression model (LRM), logistic regression model, Poisson regression model and gamma regression model. The parameters in the linear and logistic regression models are popularly estimated using the ordinary least squares (OLS) estimator and the maximum likelihood estimator (MLE), respectively. However, both estimators with multicollinearity possess high standard error, and occasionally, the estimated regression coefficients exhibit the wrong coefficient signs, making the conclusion doubtful [2,3]. The ridge regression estimator (RRE) and the logistic ridge estimator are notable alternatives to the OLS estimator in the LRM and the logistic regression model [4,5]. The Liu estimator is an alternative to the ridge estimator which accounts for multicollinearity in the LRM and the logistic regression model [6,7]. The modified ridge-type estimator is a two-parameter estimator that competes favorably with the ridge and Liu estimators [8,9]. Recently the K-L estimator emerged as another estimator in the ridge class and the Liu estimator with a single biasing parameter class [10]. The K-L estimator is a form of the Liu-type estimator with one parameter that minimizes the residual sum of squares with respect to the L2 norm with a prior information. The K-L estimator outperforms the RRE and Liu estimators based on the theoretical conditions. In this study, we developed the K-L estimator for parameter estimation with the logistic regression model, derived its statistical properties, performed a theoretical comparison with other estimators, and validated its performance by a simulation and a real-life application.

The organization of this paper is as follows. The proposed estimator is discussed in Section 2. A theoretical comparison of various estimators is presented in Section 3. A simulation study is conducted in Section 4. Real-life data are analyzed in Section 5. Finally, some concluding remarks are given in Section 6.

2. Proposed Estimator

Given that $y_i$ is a binary response variable, then the logistic regression model is defined as a Bernoulli distribution: $y_i~Be\left({\pi }_i\right)$.

$${p(y}_i)={{\pi }_i}^{y_i}{{(1-\pi }_i)}^{(1-y_i)}$$

(1)

where ${\pi }_i=\frac{e^{x_i^T\beta }}{1+e^{x_i^T\beta }}=\frac{1}{1+e^{-\left(x_i^T\beta \right)}},i=\mathrm{1,2},\dots ,n$ and $x_i$ is the $i^{th}$ row of $X$, which is an $n\times (p+1)$ matrix of explanatory variables, $\beta$ is a $(p+1)\times 1$ vector of regression coefficients and $y_i~Be\left({\pi }_i\right).$ The parameters in the logistic regression model are estimated by the method of MLE. The MLE of $\beta$ is

$${\widehat{\beta }}_{MLE}={\left(X^T{\widehat{G}}_{n}X\right)}^{-1}{X}^T{\widehat{G}}_n\widehat{z}$$

(2)

where ${\widehat{G}}_n=diag\left({\widehat{\pi }}_i(1-{\widehat{\pi }}_i)\right)$ and ${\widehat{z}}_i=\mathit{log}\left({\widehat{\pi }}_i\right)+\frac{y_i-{\widehat{\pi }}_i}{{\widehat{\pi }}_i\left(1-{\widehat{\pi }}_i\right)}.$

Multicollinearity among the explanatory variables affects the MLE. The variance of the regression parameter is often influenced by the presence of multicollinearity [11,12]. The RRE is an alternative to the MLE in linear and logistic regression models [4,5]. The logistic ridge estimator (LRE) is defined as:

$${\widehat{\beta }}_{LRE}={\left(X^T{\widehat{G}}_{n}X+k{I}_p\right)}^{-1}{X}^T{\widehat{G}}_{n}X{\widehat{\beta }}_{MLE,}$$

(3)

where I is an identity matrix, k (k > 0) is the ridge parameter, and ${\widehat{G}}_n$ is the estimate of G using ${\widehat{\beta }}_{MLE}$. The ridge parameter [13] is defined as

$$k=\frac{(p{+1)\sigma }^2}{\sum _{j=1}^{p+1}{\alpha }_j^2}$$

(4)

while the logistic version [14] is as follows:

$$k=\frac{(p+1)}{\sum _{j=1}^p{\alpha }_j^2}$$

(5)

The Liu estimator [6] is an alternative to the ridge estimator in the linear regression model, while the logistic Liu estimator (LLE) [7] is expressed as follows:

$${\widehat{\beta }}_{LLE}={\left(X^T{\widehat{G}}_{n}X+I_p\right)}^{-1}\left(X^T{\widehat{G}}_{n}X+d{I}_p\right){\widehat{\beta }}_{MLE},$$

(6)

where d $\left(0<d<1\right)$ is the Liu parameter. Further, we adopted the following method to compute Liu parameter d [15]:

$$d=min\left(\frac{{\alpha }_j^2}{\frac{1}{{\lambda }_j}+{\alpha }_j^2}\right)$$

(7)

where max and min represent the maximum and minimum operators, respectively. Further, ${\lambda }_j$ represents the jth eigenvalue of the $X^T{\widehat{G}}_{n}X$ and $\alpha =Q^T{\widehat{\beta }}_{MLE}$, where Q is the eigenvector of $X^T{\widehat{G}}_{n}X$.

Liu [16] proposed a two-parameter estimator called the Liu-type estimator. Inan and Erdogan [17] extended this work to the logistic regression model. The logistic Liu-type estimator (LLTE) is as follows:

$${\widehat{\beta }}_{LLTE}={\left(X^T{\widehat{G}}_{n}X+k{I}_p\right)}^{-1}\left(X^T{\widehat{G}}_{n}X-d{I}_p\right){\widehat{\beta }}_{MLE}$$

(8)

where k (k > 0) and d $\left(-\infty <d<\infty \right)$ are the biasing parameters of the LLTE.

Ozkale and Kaciranlar [15] developed the two-parameter estimator (TPE) to mitigate multicollinearity in the LRM. Huang [18] developed the logistic TPE estimator (LTPE), defined as follows:

$${\widehat{\beta }}_{LTPE}={\left(X^T{\widehat{G}}_{n}X+k{I}_p\right)}^{-1}\left(X^T{\widehat{G}}_{n}X+kd{I}_p\right){\widehat{\beta }}_{MLE}$$

(9)

where k (k > 0) and d $\left(-\infty <d<\infty \right)$ are the biasing parameters. The biasing parameters are defined in Equations (5) and (7), respectively.

Recently, the K-L estimator (KLE) [10] has shown better performance than the ordinary least squares, the RRE and the LE for parameter estimation in the LRM. The KLE is defined as

$${\widehat{\beta }}_{KLE}=(X^{T}X+k{I}_p{)}^{-1}(X^{T}X-k{I}_p){\widehat{\beta }}_{MLE}$$

(10)

where k (k > 0) is the KLE biasing parameter, which, as will be discussed in Section 3.6, was obtained by minimizing the mean squared error (MSE). However, in this study, we propose the logistic K-L estimator (LKLE) as

$${\widehat{\beta }}_{LKLE}=(X^T{\widehat{G}}_{n}X+k{I}_p{)}^{-1}\left(X^T{\widehat{G}}_{n}X-k{I}_p\right){\widehat{\beta }}_{MLE}$$

(11)

The bias and the matrix mean squared error (MMSE) of the LKLE is obtained as follows:

The bias of the LKLE is as follows:

$$B\left({\widehat{\beta }}_{LKLE}\right)=-2kQ{\Lambda }^k\alpha$$

(12)

where ${\Lambda }^k={\left(\Lambda +k{I}_p\right)}^{-1}$.

The variance of the LKLE is defined as follows:

$$Cov\left({\widehat{\beta }}_{LKLE}\right)=Q(\Lambda -k{I}_p){{\Lambda }^k\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p)Q^T$$

(13)

where ${\Lambda }^k={\left(\Lambda +k{I}_p\right)}^{-1}$.

Therefore, the MMSE and the scalar mean squared error (MSE) are, respectively, defined by

$$MMSE\left({\widehat{\beta }}_{LKLE}\right)=Q(\Lambda -k{I}_p){{\Lambda }^k\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p)Q^T+4k^{2}Q{\Lambda }^k\alpha {\alpha }^T{\Lambda }^k{Q}^T$$

(14)

and

$$MSE\left({\widehat{\beta }}_{LKLE}\right)=\sum _{j=1}^p\left(\frac{{\left({\lambda }_j-k\right)}^2}{{\lambda }_j{\left({\lambda }_j+k\right)}^2}\right)+4k^2\sum _{j=1}^p\left(\frac{{\alpha }_j^2}{{\left({\lambda }_j+k\right)}^2}\right)$$

(15)

The MMSE and MSE of the MLE, LRE, LLE, LLTE and LTPE are given, respectively, as follows:

$$MMSE\left({\widehat{\beta }}_{MLE}\right)=Q{\Lambda }^{-1}{Q}^T$$

(16)

$$MSE\left({\widehat{\beta }}_{MLE}\right)=\sum _{i=1}^p\frac{1}{{\lambda }_j}$$

(17)

$$MMSE\left({\widehat{\beta }}_{LRE}\right)=Q{\Lambda }^k\Lambda {\Lambda }^k{Q}^T+k^{2}Q{\Lambda }^k\alpha {\alpha }^T{\Lambda }^k{Q}^T$$

(18)

$$MSE\left({\widehat{\beta }}_{LRE}\right)=\sum _{j=1}^p\left(\frac{{\lambda }_j}{{\left({\lambda }_j+k\right)}^2}\right)+k^2\sum _{j=1}^p\left(\frac{{\alpha }_j^2}{{\left({\lambda }_j+k\right)}^2}\right)$$

(19)

$$MMSE\left({\widehat{\beta }}_{LLE}\right)=Q{\Lambda }_d{\Lambda }^{-1}{\Lambda }_d^T{Q}^T+Q\left({\Lambda }_d-I_p\right)\alpha {\alpha }^T{\left({\Lambda }_d-I_p\right)}^T{Q}^T,$$

(20)

where ${\Lambda }_d={\left(\Lambda +I_p\right)}^{-1}\left(\Lambda +d{I}_p\right).$

$$MSE\left({\widehat{\beta }}_{LLE}\right)=\sum _{j=1}^p\left(\frac{{\left({\lambda }_j+d\right)}^2}{{\lambda }_j{\left({\lambda }_j+1\right)}^2}+\frac{{\left(d-1\right)}^2{\alpha }_j^2}{{\left({\lambda }_j+1\right)}^2}\right)$$

(21)

$$MMSE\left({\widehat{\beta }}_{LLTE}\right)=Q{\Lambda }_{kd}{\Lambda }^{-1}{\Lambda }_{kd}^T{Q}^T+Q{\left(d+k\right)}^2{\Lambda }^k\alpha {\alpha }^T{\Lambda }^k{Q}^T$$

(22)

where ${\Lambda }_{kd}={\left(\Lambda +k{I}_p\right)}^{-1}\left(\Lambda -d{I}_p\right).$

$$MSE\left({\widehat{\beta }}_{LLTE}\right)=\sum _{j=1}^p\left(\frac{{\left({\lambda }_j-d\right)}^2}{{\lambda }_j{\left({\lambda }_j+k\right)}^2}+\frac{{\left(d+k\right)}^2{\alpha }_j^2}{{\left({\lambda }_j+k\right)}^2}\right)$$

(23)

$$MMSE\left({\widehat{\beta }}_{LTPE}\right)=Q{\Lambda }_{k-d}{\Lambda }^{-1}{\Lambda }_{k-d}^T{Q}^T+{{Qk}^2\left(1-d\right)}^2{\Lambda }^k\alpha {\alpha }^T{\Lambda }^k{Q}^T$$

(24)

where ${\Lambda }_{k-d}={\left(\Lambda +k{I}_p\right)}^{-1}\left(\Lambda +kd{I}_p\right).$

$$MSE\left({\widehat{\beta }}_{LTPE}\right)=\sum _{j=1}^p\left(\frac{{\left({\lambda }_j+kd\right)}^2}{{\lambda }_j{\left({\lambda }_j+k\right)}^2}+\frac{k^2{\left(1-d\right)}^2{\alpha }_j^2}{{\left({\lambda }_j+k\right)}^2}\right)$$

(25)

The following lemmas are needful to prove the statistical properties of the proposed estimator.

Lemma 1.

Let M be a positive definite matrix, that is, M > 0, and$\alpha$be some vector. Then$M-\alpha {\alpha }^T\ge 0$if and only if${\alpha }^T{M}^{-1}\alpha \le 1$[19].

Lemma 2.

Let${\widehat{\beta }}_j=A_{j}y,\hspace{0.33em}j=1,\hspace{0.33em}2$be two linear estimators of β [20]. Suppose that$D=Cov\left({\widehat{\beta }}_1\right)-Cov\left({\widehat{\beta }}_2\right)>0,$where$Cov\left({\widehat{\beta }}_j\right),⥂j=1,\hspace{0.33em}2$denotes the covariance matrix of${\widehat{\beta }}_j$and$b_j=Bias\left({\widehat{\beta }}_j\right)=\left(A_{j}X-I\right)\beta ,\hspace{0.33em}j=1,\hspace{0.33em}2.$Consequently,

$$\Delta \left({\widehat{\beta }}_1-{\widehat{\beta }}_2\right)=MMSE\left({\widehat{\beta }}_1\right)-MMSE\left({\widehat{\beta }}_2\right)={\sigma }^{2}D+b_1{b}_1^T-b_2{b}_2^T>0$$

(26)

if and only if$b_2^T{\left[{\sigma }^{2}D+b_1{b}_1^T\right]}^{-1}{b}_2<1,$where$MSE\left({\widehat{\beta }}_j\right)=Cov\left({\widehat{\beta }}_j\right)+b_j^T{b}_j.$

3. Comparison among the Estimators

In this section, we will perform a theoretical comparison of the proposed estimator with the available estimators in terms of MMSEs.

3.1. Comparison between ${\widehat{\beta }}_{MLE}$ and ${\widehat{\beta }}_{LKLE}$

Theorem 1.

If k > 0, the estimator${\widehat{\beta }}_{LKLE}$is preferable to the estimator${\widehat{\beta }}_{MLE}$in the MMSE sense, if and only if,$b^T{\left[\right({\Lambda }^{-1}-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p)\left)\right]}^{-1}b<1$, where$b=-2k{\Lambda }^k\alpha$.

Proof. 

$$MMSE\left({\widehat{\beta }}_{MLE}\right)-MMSE\left({\widehat{\beta }}_{LKLE}\right)=\left[{\Lambda }^{-1}-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p)\right]-4k^2{\Lambda }^k{\Lambda }^k\alpha {\alpha }^T$$

$${Cov\left({\widehat{\beta }}_{MLE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)=\Lambda }^{-1}-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p)$$

can be written in scalar form as follows:

$$Cov\left({\widehat{\beta }}_{MLE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)=Q diag{\left\{\frac{1}{{\lambda }_j}-\frac{{\left({\lambda }_j-k\right)}^2}{{\lambda }_j{\left({\lambda }_j+k\right)}^2}\right\}}_{j=1}^p{Q}^T$$

(27)

$$=Q diag{\left\{\frac{\left[\left({\lambda }_j+k\right)-\left({\lambda }_j-k\right)\right]\left[\left({\lambda }_j+k\right)+\left({\lambda }_j-k\right)\right]}{{{\lambda }_j\left({\lambda }_j+k\right)}^2}\right\}}_{j=1}^p{Q}^T$$

$Cov\left({\widehat{\beta }}_{MLE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)$ is positive definite since $4k{\lambda }_j>0$. Hence, using Lemma 2, $MMSE\left({\widehat{\beta }}_{MLE}\right)-MMSE\left({\widehat{\beta }}_{LKLE}\right)$ > 0 if and only if

$${4k^2{\alpha }^T{\Lambda }^k\left[\right({\Lambda }^{-1}-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p)\left)\right]}^{-1}{\Lambda }^k\alpha <1$$

(28)

Simplifying Equation (27), we have ${4k^2{\alpha }^T\Lambda [{(\Lambda +k)}^2-{(\Lambda -k)}^2]}^{-1}\alpha <1$. This was practically illustrated in Section 3.5 (Proof completed). □

3.2. Comparison between ${\widehat{\beta }}_{LRE}$ and ${\widehat{\beta }}_{LKLE}$

Theorem 2.

If k > 0, the estimator${\widehat{\beta }}_{LKLE}$is preferable to the estimator${\widehat{\beta }}_{LRE}$in the MMSE sense, if and only if,$b^T{\left[\right({\Lambda }^k\Lambda {\Lambda }^k-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p))+b_1^T{b}_1]}^{-1}b<1$, where$b=-2k{\Lambda }^k\alpha$and$b_1=-k{\Lambda }^k\alpha .$

Proof. 

$$MMSE\left({\widehat{\beta }}_{LRE}\right)-MMSE\left({\widehat{\beta }}_{LKLE}\right)=\left[{\Lambda }^k\Lambda {\Lambda }^k-{\Lambda }^k\left(\Lambda -k{I}_p\right){\Lambda }^{-1}{\Lambda }^k\left(\Lambda -k{I}_p\right)\right]+b_1{b}_1^T-b{b}^T$$

$$Cov\left({\widehat{\beta }}_{LRE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)={\Lambda }^k\Lambda {\Lambda }^k-{\Lambda }^k\left(\Lambda -k{I}_p\right){\Lambda }^{-1}{\Lambda }^k\left(\Lambda -k{I}_p\right)$$

can be written in scalar form as follows:

$$Cov\left({\widehat{\beta }}_{LRE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)=Q diag{\left\{\frac{{\lambda }_j}{{\left({\lambda }_j+k\right)}^2}-\frac{{\left({\lambda }_j-k\right)}^2}{{\lambda }_j{\left({\lambda }_j+k\right)}^2}\right\}}_{j=1}^p{Q}^T$$

(29)

$$=Q diag{\left\{\frac{{\lambda }_j^2{\left({\lambda }_j+k\right)}^2-{\left({\lambda }_j-k\right)}^2{\left({\lambda }_j+k\right)}^2}{{{\lambda }_j\left({\lambda }_j+k\right)}^2}\right\}}_{j=1}^p{Q}^T$$

$Cov\left({\widehat{\beta }}_{LRE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)$ is positive definite since ${\lambda }_j^2{\left({\lambda }_j+k\right)}^2>{\left({\lambda }_j-k\right)}^2{\left({\lambda }_j+k\right)}^2$ for k $>0$.

$$B\left({\widehat{\beta }}_{LRE}\right)-B\left({\widehat{\beta }}_{LKLE}\right)=-k{\left({\lambda }_j+k\right)}^{-1}\alpha +2k{\left({\lambda }_j+k\right)}^{-1}\alpha$$

(30)

$$=k{\left({\lambda }_j+k\right)}^{-1}\alpha$$

$$B\left({\widehat{\beta }}_{LRE}\right)-B\left({\widehat{\beta }}_{LKLE}\right)>0$$

Hence, using Lemma 2, $MMSE\left({\widehat{\beta }}_{LRE}\right)-MMSE\left({\widehat{\beta }}_{LKLE}\right)$ > 0 if and only if

$$b^T{\left[\right({\Lambda }^k\Lambda {\Lambda }^k-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p))+b_1^T{b}_1]}^{-1}b<1$$

(31)

Simplifying Equation (29), we have ${4k^2{\alpha }^T[\Lambda +k^2\alpha {\alpha }^T-{{\Lambda }^{-1}(\Lambda -k{I}_p)}^2]}^{-1}\alpha <1$. This was practically illustrated in Section 5 (Proof completed). □

3.3. Comparison between ${\widehat{\beta }}_{LLE}$ and ${\widehat{\beta }}_{LKLE}$

Theorem 3.

If k > 0 and 0 < d < 1, the estimator${\widehat{\beta }}_{LKLE}$is preferable to the estimator${\widehat{\beta }}_{LLE}$in MMSE sense if and only if,$b^T{\left[\right({\Lambda }_d{\Lambda }^{-1}{\Lambda }_d^T-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p))+b_2^T{b}_2]}^{-1}b<1,$where$b=-2k{\Lambda }^k\alpha$and$b_2=-\left(1-d\right){\left(\Lambda +I\right)}^{-1}\alpha .$

Proof. 

$$MMSE\left({\widehat{\beta }}_{LLE}\right)-MMSE\left({\widehat{\beta }}_{LKLE}\right)=\left[{\Lambda }_d{\Lambda }^{-1}{\Lambda }_d^T-{\Lambda }^k\left(\Lambda -k{I}_p\right){\Lambda }^{-1}{\Lambda }^k\left(\Lambda -k{I}_p\right)\right]+b_2{b}_2^T-\mathrm{b}{b}^T$$

$$Cov\left({\widehat{\beta }}_{LLE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)={\Lambda }_d{\Lambda }^{-1}{\Lambda }_d^T-{\Lambda }^k\left(\Lambda -k{I}_p\right){\Lambda }^{-1}{\Lambda }^k\left(\Lambda -k{I}_p\right)$$

can be written in scalar form as follows:

$$Cov\left({\widehat{\beta }}_{LLE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)=Q diag{\left\{\frac{{\left({\lambda }_j+d\right)}^2}{{\lambda }_j{\left({\lambda }_j+1\right)}^2}-\frac{{\left({\lambda }_j-k\right)}^2}{{\lambda }_j{\left({\lambda }_j+k\right)}^2}\right\}}_{j=1}^p{Q}^T=Q diag{\left\{\frac{{\left({\lambda }_j+d\right)}^2{\left({\lambda }_j+K\right)}^2-{\left({\lambda }_j-K\right)}^2{\left({\lambda }_j+1\right)}^2}{{\lambda }_j{\left({\lambda }_j+K\right)}^2{\left({\lambda }_j+1\right)}^2}\right\}}_{j=1}^p{Q}^T$$

(32)

$Cov\left({\widehat{\beta }}_{LLE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)$ since ${\left({\lambda }_j+k\right)}^2{\left({\lambda }_j+d\right)}^2-{\left({\lambda }_j-k\right)}^2{\left({\lambda }_j+1\right)}^2>0$ for $k>\frac{{\lambda }_j\left(1-d\right)}{d+{\lambda }_j+1}$ and $d>\frac{{\lambda }_j\left(1-k\right)-k}{{\lambda }_j+k}$.

$$B\left({\widehat{\beta }}_{LLE}\right)-B\left({\widehat{\beta }}_{LKLE}\right)=-(1-d){\left({\lambda }_j+1\right)}^{-1}\alpha +2k{\left({\lambda }_j+k\right)}^{-1}\alpha$$

(33)

$$=-\alpha diag{\left\{\frac{(1-d)}{{\left({\lambda }_j+1\right)}^2}-\frac{2k}{{\lambda }_j+k}\right\}}_{j=1}^p$$

$$B\left({\widehat{\beta }}_{LLE}\right)-B\left({\widehat{\beta }}_{LKLE}\right)=(k+d{\lambda }_j+kd+2k{\lambda }_j-{\lambda }_j)\alpha >0$$

Hence, using Lemma 2.2, $MMSE\left({\widehat{\beta }}_{LLE}\right)-MMSE\left({\widehat{\beta }}_{LKLE}\right)$ > 0 if and only if

$$b^T{\left[\right({\Lambda }_d{\Lambda }^{-1}{\Lambda }_d^T-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p))+b_2^T{b}_2]}^{-1}b<1$$

(34)

This was practically illustrated in Section 5 (Proof completed). □

3.4. Comparison between ${\widehat{\beta }}_{LLTE}$ and ${\widehat{\beta }}_{LKLE}$

Theorem 4.

If k > 0 and$-\infty <d<\infty$, the estimator${\widehat{\beta }}_{LKLE}$is preferable to the estimator${\widehat{\beta }}_{LLTE}$in MMSE sense if and only if,$b^T{\left[\right({\Lambda }_{kd}{\Lambda }^{-1}{\Lambda }_{kd}^T-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p))+b_2^T{b}_2]}^{-1}b<1,$where${\Lambda }_{kd}={\left(\Lambda +k\right)}^{-1}\left(\Lambda -dI\right)$,$b=-2k{\Lambda }^k\alpha$and$b_2=-\left(d+k\right){\left(\Lambda +k\right)}^{-1}\alpha .$

Proof. 

$$MMSE\left({\widehat{\beta }}_{LLTE}\right)-MMSE\left({\widehat{\beta }}_{LKLE}\right)=\left[{\Lambda }_{kd}{\Lambda }^{-1}{\Lambda }_{kd}^T-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p)\right]+b_2{b}_2^T-\mathrm{b}{b}^T$$

$$Cov\left({\widehat{\beta }}_{LLTE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)={\Lambda }_{kd}{\Lambda }^{-1}{\Lambda }_{kd}^T-{\Lambda }^k\left(\Lambda -k{I}_p\right){\Lambda }^{-1}{\Lambda }^k\left(\Lambda -k{I}_p\right)$$

can be written in scalar form as follows:

$$\begin{gathered} Cov\left({\widehat{\beta }}_{LLTE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)=Q diag{\left\{\frac{{\left({\lambda }_j-d\right)}^2}{{\lambda }_j{\left({\lambda }_j+k\right)}^2}-\frac{{\left({\lambda }_j-k\right)}^2}{{\lambda }_j{\left({\lambda }_j+k\right)}^2}\right\}}_{j=1}^p{Q}^T \\ =Q diag{\left\{\frac{{\left({\lambda }_j-d\right)}^2-{\left({\lambda }_j-k\right)}^2}{{\lambda }_j{\left({\lambda }_j+k\right)}^2}\right\}}_{j=1}^p{Q}^T+b_2^T{b}_2-b^{T}b. \end{gathered}$$

(35)

$Cov\left({\widehat{\beta }}_{LLTE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)$ is non-negative (nn), since ${\left({\lambda }_j-d\right)}^2>{\left({\lambda }_j-k\right)}^2$.

$$B\left({\widehat{\beta }}_{LLTE}\right)-B\left({\widehat{\beta }}_{LKLE}\right)=-(\mathrm{d}+k){\left({\lambda }_j+k\right)}^{-1}\alpha +2k{\left({\lambda }_j+k\right)}^{-1}\alpha$$

(36)

$$=(k-d){\left({\lambda }_j+k\right)}^{-1}\alpha$$

$$B\left({\widehat{\beta }}_{LLTE}\right)-B\left({\widehat{\beta }}_{LKLE}\right)>0\mathrm{for}\mathrm{k}>\mathrm{d}.$$

Hence, using Lemma 2.2, $MMSE\left({\widehat{\beta }}_{LLTE}\right)-MMSE\left({\widehat{\beta }}_{LKLE}\right)>0$ if and only if

$$b^T{\left[\right({\Lambda }_{kd}{\Lambda }^{-1}{\Lambda }_{kd}^T-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p))+b_2^T{b}_2]}^{-1}b<1$$

(37)

This was practically illustrated in Section 5 (Proof completed). □

3.5. Comparison between ${\widehat{\beta }}_{LTPE}$ and ${\widehat{\beta }}_{LKLE}$

Theorem 5.

If k > 0 and$-\infty <d<\infty$, the estimator${\widehat{\beta }}_{LKLE}$is preferable to the estimator${\widehat{\beta }}_{LTPE}$in MMSE sense if and only if,$b^T{\left[\right({\Lambda }_{k-d}{\Lambda }^{-1}{\Lambda }_{k-d}^T-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p))+b_2^T{b}_2]}^{-1}b<1,$where${\Lambda }_{k-d}={\left(\Lambda +k\right)}^{-1}\left(\Lambda +kdI\right)$,$b=-2k{\Lambda }^k\alpha$and$b_2=-k\left(1-d\right){\left(\Lambda +k\right)}^{-1}\alpha .$

Proof. 

$$MMSE\left({\widehat{\beta }}_{LTPE}\right)-MMSE\left({\widehat{\beta }}_{LKLE}\right)=\left[{\Lambda }_{k-d}{\Lambda }^{-1}{\Lambda }_{k-d}^T-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p)\right]+b_2{b}_2^T-\mathrm{b}{b}^T$$

$$Cov\left({\widehat{\beta }}_{LTPE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)={\Lambda }_{k-d}{\Lambda }^{-1}{\Lambda }_{k-d}^T-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p)$$

can be written in scalar form as follows:

$$\begin{gathered} Cov\left({\widehat{\beta }}_{LTPE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)=Q diag{\left\{\frac{{\left({\lambda }_j+kd\right)}^2}{{\lambda }_j{\left({\lambda }_j+k\right)}^2}-\frac{{\left({\lambda }_j-k\right)}^2}{{\lambda }_j{\left({\lambda }_j+k\right)}^2}\right\}}_{j=1}^p{Q}^T \\ =Q diag{\left\{\frac{{\left({\lambda }_j+kd\right)}^2-{\left({\lambda }_j-k\right)}^2}{{\lambda }_j{\left({\lambda }_j+k\right)}^2}\right\}}_{j=1}^p{Q}^T. \end{gathered}$$

(38)

$Cov\left({\widehat{\beta }}_{LTPE}\right)-Cov\left({\widehat{\beta }}_{LKLE}\right)$ is non-negative (nn) since ${\left({\lambda }_j+kd\right)}^2>{\left({\lambda }_j-k\right)}^2$.

$$B\left({\widehat{\beta }}_{LTPE}\right)-B\left({\widehat{\beta }}_{LKLE}\right)=-k(1-\mathrm{d}){\left({\lambda }_j+k\right)}^{-1}\alpha +2k{\left({\lambda }_j+k\right)}^{-1}\alpha$$

(39)

$$=k(1+d){\left({\lambda }_j+k\right)}^{-1}\alpha$$

$$B\left({\widehat{\beta }}_{LTPE}\right)-B\left({\widehat{\beta }}_{LKLE}\right)>0$$

Hence, using Lemma 2, $MMSE\left({\widehat{\beta }}_{LTPE}\right)-MMSE\left({\widehat{\beta }}_{LKLE}\right)$ > 0 if and only if

$$b^T{\left[\right({\Lambda }_{k-d}{\Lambda }^{-1}{\Lambda }_{k-d}^T-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p))+b_2^T{b}_2]}^{-1}b<1$$

(40)

This was practically illustrated in Section 5 (Proof completed). □

3.6. Selection of k

Since the shrinkage parameter plays a significant role in estimating biased estimators such as the LRE, LLE and LKLE, several researchers have introduced various shrinkage parameter estimation methods for the different regression models [21,22,23,24,25,26,27,28,29]. Based on these studies, we propose some shrinkage estimators of the parameter k for the LKLE.

To estimate parameter k, following [4], we will consider the generalized version of KL- estimator, which is given as follows:

$${\widehat{\beta }}_{LKLE}=(X^T{\widehat{G}}_{n}X+K{I}_p{)}^{-1}\left(X^T{\widehat{G}}_{n}X-K{I}_p\right){\widehat{\beta }}_{MLE},$$

(41)

where K = diag(k₁, k₂, …, k_p).

The MSE of KL estimator in (44) would be

$$MSE\left({\widehat{\beta }}_{LKLE}\right)=\sum _{j=1}^p\left(\frac{{\left({\lambda }_j-k_j\right)}^2}{{\lambda }_j{\left({\lambda }_j+k_j\right)}^2}\right)+4\sum _{j=1}^p\left(\frac{{\alpha }_j^2{k}_j^2}{{\left({\lambda }_j+k_j\right)}^2}\right)$$

(42)

Differentiating Equation (42) with respect to $k_j$, (all terms except k_j will be 0) and equating to 0, we have

$$-2\left(\frac{{\lambda }_j-k_j}{{\lambda }_j{\left({\lambda }_j+k_j\right)}^2}\right)-2\left(\frac{{\left({\lambda }_j-k_j\right)}^2}{{\lambda }_j{\left({\lambda }_j+k_j\right)}^3}\right)+8\left(\frac{{k_j\alpha }_j^2}{{\left({\lambda }_j+k_j\right)}^2}\right)-8\left(\frac{{\alpha }_j^2{k}_j^2}{{\left({\lambda }_j+k_j\right)}^3}\right)=0$$

$$\frac{-2\left({\lambda }_j-k_j\right)\left({\lambda }_j+k_j\right)-2{\left({\lambda }_j-k_j\right)}^2+8{\lambda }_j{k}_j{\alpha }_j^2\left({\lambda }_j+k_j\right)-8{\lambda }_j{\alpha }_j^2{k}_j^2}{{\lambda }_j{\left({\lambda }_j+k_j\right)}^3}=0$$

$$-2\left({\lambda }_j-k_j\right)\left({\lambda }_j+k_j\right)-2{\left({\lambda }_j-k_j\right)}^2+8{\lambda }_j{k}_j{\alpha }_j^2\left({\lambda }_j+k_j\right)-8{\lambda }_j{\alpha }_j^2{k}_j^2=0$$

$$\left({\lambda }_j-k_j\right)\left({\lambda }_j+k_j\right)+{\left({\lambda }_j-k_j\right)}^2-4{\lambda }_j{k}_j{\alpha }_j^2\left({\lambda }_j+k_j\right)+4{\lambda }_j{\alpha }_j^2{k}_j^2=0$$

(43)

Simplifying further Equation (43), we have

$$2{\lambda }_j^2-2{\lambda }_j{k}_j-4k_j{\lambda }_j^2{\alpha }_j^2=0$$

(44)

Dividing both sides of Equation (44) by 2${\lambda }_j$, we obtain

$$k_j=\frac{{\lambda }_j}{1+2{\lambda }_j{\alpha }_j^2}$$

(45)

By replacing $\alpha \mathrm{a}\mathrm{n}\mathrm{d} \lambda$ with its unbiased estimates, Equation (45) becomes

$${\widehat{k}}_j=\frac{\widehat{{\lambda }_j}}{1+2{\widehat{\lambda }}_j{\widehat{\alpha }}_j^2} (\mathrm{j}=1,2,\dots ,\mathrm{p})$$

(46)

Following Hoerl et al. [13], and based on the study of Mansson et al. [7], Lukman and Ayinde [3] and Qasim et al. [22,30], we suggest the following biasing parameter estimators for the logistic regression model:

• LKLE 1: $k=min\left(\frac{1}{{\widehat{\alpha }}_j^2}\right)$
• LKLE 2: k = $\mathit{min}\left({\widehat{k}}_j\right)$
• LRE 1: $k=min\left(\frac{1}{{\widehat{\alpha }}_j^2}\right)$
• LRE 2: $k=\frac{p}{\sum _{j=1}^p{\widehat{\alpha }}_j^2}$
• LLE: $d=\mathit{min}\left(\frac{{\alpha }_j^2}{\frac{1}{{\lambda }_j}+{\alpha }_j^2}\right)$
• LLTE: $k=\frac{p}{\sum _{j=1}^p{\widehat{\alpha }}_j^2}$, $d=\mathit{min}\left(\frac{{\alpha }_j^2}{\frac{1}{{\lambda }_j}+{\alpha }_j^2}\right)$
• LTPE: $k=\frac{p}{\sum _{j=1}^p{\widehat{\alpha }}_j^2}$, $d=\mathit{min}\left(\frac{{\alpha }_j^2}{\frac{1}{{\lambda }_j}+{\alpha }_j^2}\right)$

4. Monte Carlo Simulation

In this section, we compare the performance of the logistic regression estimators using a simulation study. A significant number of simulation studies have been conducted to compare the performance of estimators for both linear and logistic regression models [24,25,26,27,28,29,30,31,32,33,34,35]. The MSE is a function of $\beta ,$${\sigma }^2,$ p and is minimized subject to constraint β′β = 1 [36,37]. Schaefer [14] showed that the logistic regression model can be designed employing a similar approach to that of the linear regression model. The correlated explanatory variables can be obtained using the simulation procedure given in [38,39].

$$x_{ij}={\left(1-{\rho }^2\right)}^{1/2}{w}_{ij}+\rho w_{i(j+1)},\hspace{0.33em}i=\mathrm{1,2},\dots ,n;\hspace{0.33em}j=\mathrm{1,2},\dots ,p,$$

(47)

where $w_{ij}$ are independent standard normal pseudo-random numbers and $\rho$ is the correlation between the explanatory variables. The values of $\rho$ are chosen to be 0.9, 0.95, 0.99 and 0.999. The response variable is generated from the Bernoulli distribution, i.e., $y_i~Be\left({\pi }_i\right),$ where ${\pi }_i=\frac{e^{x_i^T\beta }}{1+e^{x_i^T\beta }}$. Sample size, n, is varied, i.e., 50, 100, 250 or 300. The estimated MSE is calculated as

$$MSE\left(\widehat{\beta }\right)=\frac{1}{2000}\sum _{i=1}^{2000}{\left({\widehat{\beta }}_i-\beta \right)}^T\left({\widehat{\beta }}_i-\beta \right)$$

(48)

where ${\widehat{\beta }}_i$ denotes the vector of the estimated regression coefficient in ith replication and β is the vector of the true parameter values, chosen such that ${\beta }^{’}\beta =1$. The experiment was replicated 2000 times. We present the estimated MSEs and the bias of each of the estimators for p = 3 in Table 1 and Table 2, respectively. For p = 7, the results are provided in Table 3 and Table 4, respectively. We observed that increasing the sample size resulted in a decrease in the MSE values for each case. The following observations were obtained from the simulation result. The MSE values of the estimators increased as the degree of correlation and the number of explanatory variables increased. The simulation results show that the LKLE performed best at most levels of multicollinearity, sample sizes and the number of explanatory variables with few exceptions. The LTPE competed favorably in most cases, except on a few occasions. Upon comparing the performance of the shrinkage parameters in the LKLE, we found that LKLE 1 performed well except in a few cases. The MLE performed least well when there was multicollinearity in the data. Of the two-parameter estimators (LTPE and LLTE), LTPE performed better. Additionally, it is obvious that the bias of the proposed estimator was the lowest in most cases. Generally, the LKLE estimator is preferred over the two-parameter estimator.

Table 1. Estimated MSEs and Bias for p = 3.

n		50	50	50	50	100	100	100	100
ρ		0.9	0.95	0.99	0.999	0.9	0.95	0.99	0.999
MLE	MSE	1.4837	2.5668	11.6012	112.132	0.8648	1.3979	5.6404	53.8
LLE	MSE	1.0597	1.4893	5.0029	47.674	0.7492	1.0409	2.6449	21.6
	BIAS	−0.7965	−0.8000	−0.8029	−0.8034	−0.8014	−0.8030	−0.8080	−0.8126
LRE 1	MSE	0.9867	1.4492	5.4291	49.507	0.6677	0.9090	2.7611	23.9
	BIAS	−0.8349	−0.8190	−0.7934	−0.7802	−0.8323	−0.8234	−0.8098	−0.8039
LRE 2	MSE	0.8332	1.1187	3.7215	32.719	0.6040	0.7538	1.9624	15.9
	BIAS	−0.8967	−0.8651	−0.8150	−0.7906	−0.8695	−0.8517	−0.8225	−0.8089
LLTE	MSE	0.6927	0.8063	7.3947	50.125	0.5983	0.6394	2.6679	48
	BIAS	−0.9207	−0.9011	−0.9448	−1.9698	−0.8751	−0.8595	−0.8447	−1.0067
LTPE	MSE	0.7238	0.8732	2.3002	18.585	0.5911	0.6384	1.3563	9.82
	BIAS	−0.9322	−0.8974	−0.8384	−0.8075	−0.8897	−0.8697	−0.8342	−0.8164
LKLE 1	MSE	0.7458	0.8043	1.6510	11.563	0.5918	0.6303	1.0533	6.21
	BIAS	−1.0436	−0.9752	−0.8734	−0.8256	−0.9502	−0.9131	−0.8536	−0.8251
LKLE 2	MSE	0.9598	1.4017	5.2142	47.268	0.6582	0.8916	2.6610	22.7
	BIAS	−0.8186	−0.8032	−0.7848	−0.7790	−0.8224	−0.8136	−0.8038	−0.8029

Table 2. Estimated MSEs and Bias for p = 3.

n		250	250	250	250	300	300	300	300
ρ		0.9	0.95	0.99	0.999	0.9	0.95	0.99	0.999
MLE	MSE	0.5112	0.7083	2.2958	20.4	0.4834	0.6532	1.9918	16.931
LLE	MSE	0.4997	0.6530	1.4819	8.36	0.4757	0.6119	1.3344	6.708
	BIAS	−0.8273	−0.8256	−0.8239	−0.8271	−0.8189	−0.8180	−0.8187	−0.8198
LRE 1	MSE	0.4847	0.5844	1.2803	9.07	0.4659	0.5550	1.1523	7.558
	BIAS	−0.8497	−0.8417	−0.8290	−0.8244	−0.8389	−0.8329	−0.8247	−0.8182
LRE 2	MSE	0.4825	0.5421	0.9858	6.14	0.4656	0.5192	0.9011	5.118
	BIAS	−0.8746	−0.8612	−0.8381	−0.8275	−0.8603	−0.8501	−0.8331	−0.8207
LLTE	MSE	0.4794	0.5239	0.6785	10.23	0.4634	0.5059	0.6580	7.865
	BIAS	−0.8766	−0.8637	−0.8438	−0.8609	−0.8618	−0.8519	−0.8370	−0.8414
LTPE	MSE	0.4785	0.5083	0.7690	3.98	0.4631	0.4894	0.7201	3.398
	BIAS	−0.8873	−0.8723	−0.8456	−0.8314	−0.8726	−0.8608	−0.8404	−0.8241
LKLE 1	MSE	0.2344	0.3576	0.6250	2.81	0.3087	0.3285	0.5771	2.364
	BIAS	−0.9275	−0.9024	−0.8594	−0.8367	−0.9053	−0.8856	−0.8515	−0.8281
LKLE 2	MSE	0.4344	0.5027	1.2531	8.66	0.4625	0.5533	1.1303	7.200
	BIAS	−0.8469	−0.8385	−0.8261	−0.8237	−0.8358	−0.8291	−0.8211	−0.8170

Table 3. Estimated MSEs and Bias for p = 7.

n		50	50	50	50	100	100	100	100
ρ		0.9	0.95	0.99	0.999	0.9	0.95	0.99	0.999
MLE	MSE	6.2754	11.5251	61.5	585	2.5865	4.9251	23.922	236
LLE	MSE	1.9336	2.3581	10.2	117	1.5505	2.0802	4.720	40.9
	BIAS	−1.4187	−1.4699	−1.5091	−1.5071	−1.3487	−1.3548	−1.4039	−1.4231
LRE 1	MSE	3.5013	6.0056	31.8	297	1.6169	2.8887	13.218	128
	BIAS	−1.3122	−1.3329	−1.3042	−1.3101	−1.3431	−1.3225	−1.3280	−1.3272
LRE 2	MSE	1.6420	2.6788	13.2	122	0.9322	1.5026	6.273	59.3
	BIAS	−1.4628	−1.4578	−1.4198	−1.4181	−1.4314	−1.3944	−1.3825	−1.3761
LLTE	MSE	1.2682	1.8409	1200	477	0.8472	1.2163	15.024	188
	BIAS	−1.5043	−1.5429	−2.2742	−10.5093	−1.4396	−1.4161	−1.5587	−3.0291
LTPE	MSE	1.3488	2.0536	8.96	78	0.8076	1.2358	4.747	43.2
	BIAS	−1.4979	−1.4971	−1.4701	−1.4802	−1.4483	−1.4162	−1.4102	−1.4045
LKLE 1	MSE	0.7149	0.9988	3.43	31	0.6939	0.7074	1.710	14.5
	BIAS	−1.7501	−1.7118	−1.6656	−1.6547	−1.5857	−1.5257	−1.4894	−1.4755
LKLE 2	MSE	3.3646	5.7296	30.1	281	1.5856	2.8236	12.878	125
	BIAS	−1.3135	−1.3373	−1.3132	−1.3202	−1.3416	−1.3222	−1.3293	−1.3294

Table 4. Estimated MSEs and Bias for p = 7.

n		250	250	250	250	300	300	300	300
ρ		0.9	0.95	0.99	0.999	0.9	0.95	0.99	0.999
MLE	MSE	1.0284	1.7662	7.58	72.9	0.8894	1.4841	6.2274	60.1
LLE	MSE	0.8894	1.3001	2.64	13.5	0.7944	1.1611	2.4346	9.43
	BIAS	−1.4056	−1.4033	−1.4287	−1.4415	−1.3846	−1.3865	−1.3990	−1.4195
LRE 1	MSE	0.7632	1.1834	4.40	40.5	0.6815	1.0195	3.6523	33.6
	BIAS	−1.4242	−1.4107	−1.4121	−1.4083	−1.4047	−1.3981	−1.3904	−1.3917
LRE 2	MSE	0.5474	0.7325	2.23	19.2	0.5088	0.6572	1.8915	16.2
	BIAS	−1.4755	−1.4483	−1.4333	−1.4245	−1.4518	−1.4331	−1.4089	−1.4063
LLTE	MSE	0.5392	0.7005	1.62	115	0.5033	0.6341	1.4637	47
	BIAS	−1.4768	−1.4502	−1.4422	−1.5435	−1.4529	−1.4353	−1.4163	−1.4908
LTPE	MSE	0.5117	0.6530	1.78	14.2	0.4808	0.5892	1.5361	12.6
	BIAS	−1.4849	−1.4559	−1.4403	−1.4332	−1.4619	−1.4433	−1.4169	−1.4139
LKLE 1	MSE	0.5344	0.5576	0.7250	2.81	0.5087	0.5285	0.6771	2.364
	BIAS	−1.5577	−1.5105	−1.4735	−1.4582	−1.5285	−1.4925	−1.4471	−1.4375
LKLE 2	MSE	0.4844	0.5827	1.2531	8.66	0.4655	0.5533	1.1303	7.200
	BIAS	−1.4217	−1.4089	−1.4117	−1.4085	−1.4026	−1.3968	−1.3900	−1.3919

5. Application: Cancer Data

The performance of LKLE and the other estimators was evaluated using a cancer remission dataset [34,40]. In the dataset, the binary response variable y_i is 1 if the patient experiences complete cancer remission and 0 otherwise. There are five explanatory variables. These explanatory variables include cell index (x₁), smear index (x₂), infıl index (x₃), blast index (x₄) and temperature (x₅). There were 27 patients, of which nine experienced complete remission. The eigenvalues of the $X^{’}{\widehat{G}}_{n}X$ matrix were found to be λ₁ = 9.2979, λ₂ = 3.8070, λ₃ = 3.0692, λ₄ = 2.2713 and λ₅ = 0.0314. To test the multicollinearity among the explanatory variables, we use condition index (CI), computed as $CI=\sqrt{\frac{\max \left({\lambda }_j\right)}{\min \left({\lambda }_j\right)}}$ = 17.2. There was moderate collinearity when CI was between 10 and 30 and severe multicollinearity when CI exceeded 30 [41]. Thus, the results provide evidence of moderate multicollinearity among the explanatory variables. Next, we compared the performance of the estimators using the previously described dataset. The estimated regression coefficients and the corresponding scalar MSE values are given in Table 5. The scalar MSEs of each of the estimators under study were obtained using Equations (17), (19), (21) and (23)–(25), respectively. The proposed LKLE estimator surpassed the other estimators in this study in terms of MSE.

Table 5. Validation of the theoretical conditions for the cancer data.

Theorems	Conditions	Value
1	$b^T{\left[\right({\Lambda }^{-1}-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p)\left)\right]}^{-1}b<1$	0.2413
2	$b^T{\left[\right({\Lambda }^k\Lambda {\Lambda }^k-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p))+b_1^T{b}_1]}^{-1}b<1$	0.8866
3	$b^T{\left[\right({\Lambda }_d{\Lambda }^{-1}{\Lambda }_d^T-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p))+b_2^T{b}_2]}^{-1}b<1$	0.6443
4	$b^T{\left[\right({\Lambda }_{kd}{\Lambda }^{-1}{\Lambda }_{kd}^T-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p))+b_2^T{b}_2]}^{-1}b<1$	0.0958
5	$b^T{\left[\right({\Lambda }_{k-d}{\Lambda }^{-1}{\Lambda }_{k-d}^T-{\Lambda }^k(\Lambda -k{I}_p){\Lambda }^{-1}{\Lambda }^k(\Lambda -k{I}_p))+b_2^T{b}_2]}^{-1}b<1$	0.7540

Moreover, we also evaluated the theoretical conditions as stated in Theorems 1 to 5 for the actual dataset. The validation results of these conditions are given in Table 6. As shown, all the theorem conditions hold for the cancer data, because all the inequalities in the theorems were less than one, as expected.

Table 6. Regression coefficients and MSEs of the logistic regression estimators for the cancer dataset *.

Estimators	${\widehat{\mathit{\beta }}}_1$	${\widehat{\mathit{\beta }}}_2$	${\widehat{\mathit{\beta }}}_3$	${\widehat{\mathit{\beta }}}_4$	${\widehat{\mathit{\beta }}}_5$	Estimated MSEs
MLE	−0.1966	−1.5957	1.8139	1.3073	−0.4208	32.9393
	(0.328)	(0.513)	(0.571)	(0.664)	(5.639)
LLE	0.1940	−0.4751	0.5808	1.0259	−0.3078	5.0989
	(0.305)	(0.498)	(0.571)	(0.592)	(1.645)
LRE 1	0.3706	−0.2218	0.2266	1.1366	−0.3457	1.4278
	(0.318)	(0.489)	(0.544)	(0.605)	(0.652)
LRE 2	0.3503	−0.0999	0.1472	0.9838	−0.2883	1.2544
	(0.303)	(0.504)	(0.584)	(0.596)	(0.461)
LLTE	0.5373	0.4116	−0.4227	0.8732	−0.2430	4.1372
	(0.295)	(0.536)	(0.643)	(0.624)	(1.720)
LTPE	0.4949	0.2955	−0.2933	0.8983	−0.2533	3.8161
	(0.310)	(0.491)	(0.556)	(0.592)	(1.679)
LKLE 1	0.8972	1.3960	−1.5195	0.6604	−0.1558	28.9236
	(0.278)	(0.400)	(0.446)	(0.418)	(5.208)
LKLE 2	0.4696	−0.1212	0.0576	1.2298	−0.3815	1.1350
	(0.326)	(0.505)	(0.560)	(0.646)	(0.084)

* Standard error for each of the estimators is in parenthesis.

The logistic ridge estimator competed favorably in the simulation and the real-life application. The real-life application result agreed with the simulation study. However, the performance of the estimators in both the simulation and real life was a function of the biasing parameter. For instance, LKLE 1 performed best in the simulation study, while in the real-life analysis, LKLE 2 outperformed LKLE 1. Among the two-parameter estimators, the logistic two-parameter estimator (LTPE) performed best. Of the one-parameter estimators, LKLE outperformed the ridge and the Liu estimator. Generally, LKLE dominated among both the one- and two-parameter estimator. The performance of these estimators is a function of biasing parameters k and d. Additionally, as shown in Table 5. ${\widehat{\beta }}_2$ and ${\widehat{\beta }}_3$ did not fit well for the following estimators: MLE, LLE, LLTE, LTPE and LKLE 1.

6. Some Concluding Remarks

Kibria and Lukman (2020) developed the K-L estimator to circumvent the multicollinearity problem for the linear regression model. In this paper, we described the logistic Kibria-Lukman estimator (LKLE) to address the challenge of multicollinearity for the logistic regression model. We theoretically determined the superiority of LKLE over other existing estimators in terms of the MSE. The performance of the estimators was evaluated using the Monte Carlo simulation study. In the design of the experiment, factors such as the degree of correlation, the sample size and the number of explanatory variables were varied. The results showed that the performance of the estimators was highly dependent on these factors. Finally, to illustrate the efficiency of the proposed estimator, we applied a cancer dataset and observed that the results agreed with those of the simulation study to some extent. The findings of this study will be helpful for practitioners and applied researchers who use a logistic regression model with correlated explanatory variables.