Evaluating Estimator Performance Under Multicollinearity: A Trade-Off Between MSE and Accuracy in Logistic, Lasso, Elastic Net, and Ridge Regression with Varying Penalty Parameters

Abstract

Multicollinearity in logistic regression models can result in inflated variances and yield unreliable estimates of parameters. Ridge regression, a regularized estimation technique, is frequently employed to address this issue. This study conducts a comparative evaluation of the performance of 23 established ridge regression estimators alongside Logistic Regression, Elastic-Net, Lasso, and Generalized Ridge Regression (GRR), considering various levels of multicollinearity within the context of logistic regression settings. Simulated datasets with high correlations (0.80, 0.90, 0.95, and 0.99) and real-world data (municipal and cancer remission) were analyzed. Both results show that ridge estimators, such as $k_{\mathrm{A}\mathrm{L}1}$, $k_{\mathrm{A}\mathrm{L}2}, k_{\mathrm{K}\mathrm{L}1},$and $k_{\mathrm{K}\mathrm{L}2}$, exhibit strong performance in terms of Mean Squared Error (MSE) and accuracy, particularly in smaller samples, while GRR demonstrates superior performance in large samples. Real-world data further confirm that GRR achieves the lowest MSE in highly collinear municipal data, while ridge estimators and GRR help prevent overfitting in small-sample cancer remission data. The results underscore the efficacy of ridge estimators and GRR in handling multicollinearity, offering reliable alternatives to traditional regression techniques, especially for datasets with high correlations and varying sample sizes.

1. Introduction

Logistic regression stands as a key method in statistical modeling, particularly for binary classification, where the outcome variable is dichotomous. Its widespread use in various fields, including epidemiology, social sciences, and machine learning, is due to its ability to model the probability of an event occurring as a function of one or more predictor variables [1]. Unlike a linear model, which assumes a continuous outcome, logistic regression uses a logistic function to convert the linear combination of predictors into a probability, making it well-suited for categorical outcomes. This ability to estimate probabilities directly makes logistic regression invaluable for risk assessment, predictive modeling, and hypothesis testing in several applied contexts.

However, multicollinearity, characterized by a high correlation among predictor variables, adds complexity to the interpretation and reliability of logistic regression models [2]. Although multicollinearity does not introduce bias into the coefficient estimates in the same manner as in linear regression, it significantly inflates the standard errors of these coefficients [3]. Inflated standard errors lead to decreased t-statistics (or Wald statistics) associated with the coefficients, which reduces the likelihood of detecting statistically significant effects. This increases the risk of Type II errors, where genuine relationships between predictors and the outcome may be overlooked [4]. As a result, the practical implications of multicollinearity can lead to misleading conclusions, ultimately compromising the model’s usefulness for both predictive and explanatory purposes. Although the coefficients are not biased, their stability diminishes, resulting in substantial changes in the estimated coefficients when there are slight variations in the data. This instability makes it challenging to determine the true magnitude and direction of the predictors’ effects [5]. Multicollinearity also complicates the assessment of variable importance. The high correlation among predictors complicates the identification of their individual impacts on the outcome, risking misleading conclusions about their influence [6]. The interpretation of odds ratios, which are central to logistic regression, can also be problematic. With inflated standard errors, the confidence intervals around the odds ratios broaden, making it difficult to draw precise inferences about the magnitude of the effects. This complicates the practical application of the model findings [7].

To mitigate multicollinearity in logistic regression, researchers often use regularization techniques like Ridge Regression, Lasso, and Elastic Net [8]. These methods are intended to minimize the overlap between predictor variables, strengthen the consistency of coefficient estimates, and boost the overall reliability and clarity of the model.

Hoerl and Kennard (1970) introduced ridge regression to address multicollinearity in engineering data. Their work showed that a nonzero ridge parameter (k) can lower the MSE of ridge regression compared to the variance of the Ordinary Least Squares (OLS) estimator [9]. Since their pioneering contribution, extensive research has been conducted to refine and enhance the ridge regression methodologies. A multitude of scholars have proposed innovative estimators for the ridge parameter, making remarkable strides in the field. Their contributions not only enrich existing knowledge but also open new avenues for exploration and understanding. Notable contributions include those of McDonald & Schwing, 1973; Hoerl et al., 1975; McDonald & Galarneau, 1975; J. F. & P, 1976; Dempster et al., 1977; Gibbons, 1981; Schaeffer et al., 1984; Schaeffer, 1986; Walker & Birch, 1988; Kibria, 2003; Khalaf & Shukur, 2005 [10,11,12,13,14,15,16,17,18,19,20] and very recently Muniz & Kibria, 2009; Månsson et al., 2010; Kibria et al., 2012; Hefnawy & Farag, 2014; Aslam, 2014; A. V. Dorugade, 2014; Arashi & Valizadeh, 2015; Ayinde & Lukman, 2016; Lukman & Ayindez, 2017; Melkumova & Shatskikh, 2017; Lukman et al., 2018, 2019; Herawati et al., 2018, 2024; Yüzbaşı et al., 2020; Golam Kibria & Lukman, 2020; Kibria, 2023; Hoque & Kibria, 2023; Mermi et al., 2024; Nayem et al., 2024; Hoque & Kibria, 2024; Yasmin & Kibria, 2025, among others [20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40]. These contributions have played a critical role in the ongoing refinement of ridge regression techniques, ensuring their continued relevance in statistical modeling and analyses.

Despite extensive research on ridge regression, relatively little attention has been devoted to its application in logistic regression. Schaeffer et al. (1984) presented a logistic ridge regression (LRR) estimator, and later research investigated different approaches for estimating the ridge parameter (k) and evaluated their effectiveness using Monte Carlo simulations [16,17,23]. Recently, Mermi et al. (2024) carried out a thorough comparative evaluation of 366 ridge parameter estimators introduced over various periods, taking into account elements like the number of independent variables, sample size, correlation among predictors, and variance of errors [38]. In the present study, we incorporated 16 of the most effective estimators from Mermi et al. (2024). Furthermore, we utilized seven estimators that Kibria and Lukman (2020) and Kibria (2023) specifically suggested for non-normal population distributions, guaranteeing a thorough assessment of logistic ridge regression across different modeling scenarios [34,36].

In the presence of multicollinearity, selecting an optimal estimator requires a careful balance between minimizing the MSE and maximizing the classification accuracy. MSE minimization is crucial for obtaining precise parameter estimates and reducing prediction errors, particularly in regression contexts, where deviations from true values carry significant implications [5]. However, an exclusive focus on the MSE may fail to account for the importance of classification accuracy, especially in binary or categorical response settings, where correct classification is critical for decision-making [7]. The selection of an estimator must strategically weigh these competing metrics, emphasizing MSE reduction in scenarios where precise numerical predictions are paramount and prioritizing classification accuracy when robust categorical predictions are the primary objective. As evidenced by numerous simulation studies, the optimal estimator is often context-dependent, varying based on the specific application and the relative costs associated with prediction errors versus misclassifications. Unlike much of the existing literature that focuses predominantly on linear models or limits evaluation to MSE, this study emphasizes both MSE and classification accuracy to provide a more comprehensive assessment of the model performance in the presence of multicollinearity. While previous studies have explored various ridge estimators in linear regression contexts, there remains a significant gap in systematically evaluating their effectiveness within logistic regression frameworks, particularly under high multicollinearity and non-normal data conditions. By incorporating recently developed and robust ridge parameter estimators tailored for such scenarios, this study aims to address this gap and offer practical guidance for the selection of estimators in binary classification problems. Therefore, it is crucial to conduct a thorough assessment of both the MSE and classification accuracy to make a well-informed choice concerning estimator selection when multicollinearity is present.

This research intends to conduct an in-depth comparison of 23 ridge regression estimators alongside traditional LR, Lasso, EN, and GRR. This examination goes beyond the typical emphasis on MSE by also assessing classification accuracy in situations of multicollinearity. The objective of this study is to enhance the existing literature on ridge regression by systematically evaluating the performance of estimators under various levels of multicollinearity. Additionally, real-world datasets will be utilized to corroborate the findings obtained from simulations, thereby demonstrating the practical applicability of the methodologies employed.

2. Methodology

The OLS estimator of $\theta$ is the following:

$${\widehat{\theta }}_{OLS}={\left(X^{\prime }X\right)}^{-1}{X}^{\prime }Y$$

In regression analysis, let Y be the response, X be a matrix of predictors, and θ be a parameter vector. The ε denotes the random error. When the response variable is binary, standard OLS regression is no longer suitable, and logistic regression is employed instead. In this section, we present LRR, initially proposed by Schaefer et al. (1984), and explore its enhancements through more recent advancements [16]. The refinements aim to enhance the performance and stability of logistic ridge regression, making it a more robust alternative in cases of multicollinearity.

Let the $i$th observation of the response, $y_i$, follows a Bernoulli distribution, denoted as $y_i\sim \mathrm{B}\mathrm{e}\mathrm{r}\mathrm{n}\mathrm{o}\mathrm{u}\mathrm{l}\mathrm{l}\mathrm{i} \left({\pi }_i\right)$, where ${\pi }_i$ represents the probability of success associated with the $i$th observation. The parameter ${\pi }_i$ is modeled as a function of the predictors using the logit link, ensuring that predicted probabilities remain within the unit interval.

$${\pi }_i={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{x}}_i^{\prime }\theta \right)}{1+\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{x}}_i^{\prime }\theta \right)}}$$

where $x_i$ is the $n\times (m+1)$ data matrix with $m$ predictors, and $\theta$ is the $(m+1)\times 1$ coefficients vector. The approach of maximum likelihood for estimating $\theta$ involves maximizing the subsequent log-likelihood.

$$\mathcal{l}(\mathrm{X};\theta )={\mathrm{y}}^{\prime }\mathrm{l}\mathrm{o}\mathrm{g}\left(\pi \right)+(1-\mathrm{y}{)}^{\prime }\mathrm{l}\mathrm{o}\mathrm{g}(1-\pi )$$

This can be achieved by setting the first derivative of the expression equal to zero. As a result, the maximum likelihood estimates are found by solving the resulting equation:

$${\displaystyle \frac{\partial \mathcal{l}(\mathrm{X};\theta )}{\partial \theta }}={\mathrm{X}}^{\prime }(\mathrm{y}-\pi )=0$$

As this equation is nonlinear in nature, the Newton-Raphson method must be employed to find a solution. As a result, the solution to the previously mentioned equation can be obtained through the subsequent iterative weighted least squares method.

$${\widehat{\theta }}_{ML}={\left({\mathrm{X}}^{\prime }\mathrm{W}\mathrm{X}\right)}^{-1}\left({\mathrm{X}}^{\prime }\mathrm{W}\mathrm{Z}\right)$$

where $W=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left[\left({\widehat{\pi }}_i\left(1-{\widehat{\pi }}_i\right)\right)\right]$, and

$$Z_i=\mathrm{l}\mathrm{o}\mathrm{g}\left({\widehat{\pi }}_i\right)+{\displaystyle \frac{y_i-{\widehat{\pi }}_i}{{\widehat{\pi }}_i\left(1-{\widehat{\pi }}_i\right)}}$$

The asymptotic covariance matrix of the maximum likelihood estimator is the inverse of the matrix of second derivatives:

$$\mathrm{C}\mathrm{o}\mathrm{v}\left({\widehat{\theta }}_{ML}\right)=E\left(-{\displaystyle \frac{{\partial }^2\mathcal{l}(\mathrm{X};\theta )}{\partial \theta \partial {\theta }^{\prime }}}\right)={\left({\mathrm{X}}^{\prime }\mathrm{W}\mathrm{X}\right)}^{-1}$$

and the asymptotic MSE equals:

$$E\left({∥{\widehat{\theta }}_{ML}-\theta ∥}^2\right)=E{\left({\widehat{\theta }}_{ML}-\theta \right)}^{\prime }\left({\widehat{\theta }}_{ML}-\theta \right)=\mathrm{t}\mathrm{r}\left({\left({\mathrm{X}}^{\prime }\mathrm{W}\mathrm{X}\right)}^{-1}\right)=\sum _{j=1}^J{\displaystyle \frac{1}{{\lambda }_j}}$$

where ${\lambda }_j$ is the eigenvalue of $X^{\prime }WX$ matrix. A key drawback of the maximum likelihood estimate is that its asymptotic variance can inflate due to high correlation among independent variables, leading to small eigenvalues. To address this multicollinearity issue, Schaefer et al. (1984) proposed using the following LRR estimator [16]:

$${\widehat{\theta }}_{LRR}={\left({\mathrm{X}}^{\prime }\widehat{\mathrm{W}}\mathrm{X}+k\mathrm{I}\right)}^{-1}\left({\mathrm{X}}^{\prime }\widehat{\mathrm{W}}\mathrm{X}{\widehat{\theta }}_{ML}\right)=\mathrm{Z}{\widehat{\theta }}_{ML}$$

Numerous ridge estimators have been suggested in the literature for different kinds of models, as mentioned previously, to determine the ridge parameter k. For this study, we selected the 16 most effective estimators identified by Mermi et al. (2024) based on their performance in different statistical settings [38]. In addition, we incorporated seven estimators introduced by Kibria & Lukman (2020) and Kibria (2023) which are specifically designed for asymmetric data [34,36]. These 23 estimators were selected to ensure a comprehensive comparison that spanned both classical and modern approaches to ridge parameter selection. Including estimators suited for asymmetric data is particularly important because such data structures frequently arise in applied settings and can adversely affect the performance of conventional ridge estimators [9,19]. The comprehensive set of 23 ridge estimators utilized in this study, encompassing both simulation-based analyses and real-life applications, is systematically presented in

Table 1. Ridge penalties compared in this study.

Ridge Penalties	Reference
$k_{AS} :\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{1}{q_i}}\right)$	[41]
$k_{A1} :\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{s^2}{{{\widehat{\theta }}_i}^2}}\right)$	[42]
$k_D :{\displaystyle \frac{{\widehat{\sigma }}^2}{\mathrm{m}\mathrm{i}\mathrm{n} (\frac{{\widehat{\sigma }}^2}{{{\widehat{\alpha }}_i}^2}+\frac{1}{{\lambda }_i})}}$	[25]
$k_{LA1}:\sqrt{p{\sum }_{i=1}^p\left({\displaystyle \frac{\mathrm{m}\mathrm{a}\mathrm{x}\left({\lambda }_i\right){\widehat{\sigma }}^2}{(n-p){\widehat{\sigma }}^2+\mathrm{m}\mathrm{a}\mathrm{x}\left({\lambda }_i\right){\widehat{a}}_i^2}}\right)}$	[27]
$k_{K1}:\mathrm{m}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{n}\left({\displaystyle \frac{{\widehat{\sigma }}^2}{{\widehat{\theta }}_i^2}}+{\displaystyle \frac{1}{{\lambda }_i}}\right)$	[36]
$k_{K2}:\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{{\widehat{\sigma }}^2}{{\widehat{\theta }}_i^2}}+{\displaystyle \frac{1}{{\lambda }_i}}\right)$	[36]
$k_{K3}:\mathrm{m}\mathrm{i}\mathrm{n}\left({\displaystyle \frac{{\widehat{\sigma }}^2}{{\widehat{\theta }}_i^2}}+{\displaystyle \frac{1}{{\lambda }_i}}\right)$	[36]
$k_{K4}:\mathrm{G}\mathrm{e}\mathrm{o}\mathrm{m}\mathrm{e}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{mean}\left({\displaystyle \frac{{\widehat{\sigma }}^2}{{\widehat{\theta }}_i^2}}+{\displaystyle \frac{1}{{\lambda }_i}}\right)$	[36]
$k_{K5}:\mathrm{H}\mathrm{a}\mathrm{r}\mathrm{m}\mathrm{o}\mathrm{n}\mathrm{i}\mathrm{c}\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}\left({\displaystyle \frac{{\widehat{\sigma }}^2}{{\widehat{\theta }}_i^2}}+{\displaystyle \frac{1}{{\lambda }_i}}\right)$	[36]
$k_{LA2}:\mathrm{m}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{n}\left({\displaystyle \frac{\mathrm{m}\mathrm{a}\mathrm{x} \left({\lambda }_i\right){\widehat{\sigma }}^2}{(n-p){\widehat{\sigma }}^2+{\lambda }_i{\widehat{\alpha }}_i^2}}\right)$	[27]
$k_{L1}:\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{\mathrm{m}\mathrm{a}\mathrm{x} \left({\lambda }_i\right){\widehat{\sigma }}^2}{(n-p){\widehat{\sigma }}^2+\mathrm{m}\mathrm{a}\mathrm{x} \left({\lambda }_i\right){\widehat{a}}_i^2}}\right)$	[32]
$k_{LA3}:\sqrt{p{\displaystyle \sum _{i=1}^p}\left({\displaystyle \frac{{\lambda }_i{\widehat{\sigma }}^2}{(n-p){\widehat{\sigma }}^2+{\lambda }_i{\widehat{a}}_i^2}}\right)}$	[27]
$k_{A2}:{\displaystyle \frac{1}{p}}{\displaystyle \sum _{i=1}^p}{\displaystyle \frac{\mathrm{m}\mathrm{a}\mathrm{x} \left({\lambda }_i\right){\widehat{\sigma }}^2}{(n-p){\widehat{\sigma }}^2+{\lambda }_i{\widehat{\alpha }}_i^2}}$	[42]
$k_{M1}:\mathrm{m}\mathrm{a}\mathrm{x}\left({\displaystyle \frac{1}{\frac{{\lambda }_i{\widehat{\sigma }}^2}{(n-p){\widehat{\sigma }}^2+{\lambda }_i{\widehat{a}}_i^2}}}\right)$	[43]
$k_{M2}:{\left({\displaystyle \prod _{i=1}^p}{\displaystyle \frac{\mathrm{m}\mathrm{a}\mathrm{x} \left({\lambda }_i\right){\widehat{\sigma }}^2}{(n-p){\widehat{\sigma }}^2+{\lambda }_i{\widehat{\alpha }}_i^2}}\right)}^{{\displaystyle \frac{1}{p}}}$	[43]
$k_{L2}:\mathrm{m}\mathrm{a}\mathrm{x}\left(1/\sqrt{{\displaystyle \frac{{\lambda }_i{\widehat{\sigma }}^2}{(n-p){\widehat{\sigma }}^2+\left[\mathrm{m}\mathrm{a}\mathrm{x} \left({\lambda }_i\right)\right]{\widehat{\alpha }}_i^2}}}\right)$	[32]
$k_{L3}:\mathrm{m}\mathrm{a}\mathrm{x}\left(1/{\displaystyle \frac{{\lambda }_i{\widehat{\sigma }}^2}{(n-p){\widehat{\sigma }}^2+{\lambda }_i\mathrm{m}\mathrm{a}\mathrm{x} \left({\widehat{\alpha }}_i^2\right)}}\right)$	[32]
$k_{K6}:{\prod }_{i=0}^p{\left({\displaystyle \frac{1}{q_i}}\right)}^{{\displaystyle \frac{1}{p}}}$	[21]
$k_{L4}:\mathrm{m}\mathrm{a}\mathrm{x}\left(1/{\displaystyle \frac{{\lambda }_i{\widehat{\sigma }}^2}{(n-p){\widehat{\sigma }}^2+{\lambda }_i{\left(\mathrm{m}\mathrm{a}\mathrm{x} \left({\widehat{\alpha }}_i\right)\right)}^2}}\right)$	[32]
$k_{AL1}:{\displaystyle \frac{1}{\sqrt{\frac{2{\widehat{\sigma }}^2}{\mathrm{m}\mathrm{a}\mathrm{x} \left({\lambda }_i\right) \mathrm{m}\mathrm{a}\mathrm{x}\left({\widehat{a}}_i^2\right)}}}}$	[28]
$k_{AL2}:{\displaystyle \frac{1}{\sqrt{\frac{2p{\widehat{\sigma }}^2}{\mathrm{m}\mathrm{a}\mathrm{x} \left({\lambda }_i\right){\sum }_{i=1}^p{\widehat{a}}_i^2}}}}$	[28]
$k_{KL1}:\widehat{\sigma }{p}^{\left(1+{\displaystyle \frac{p}{n}}\right)}$	[34]
$k_{KL2}:\widehat{\sigma }\times \mathrm{m}\mathrm{a}\mathrm{x}\left(p^{\left(1+{\displaystyle \frac{p}{n}}\right)},p^{\left(1+{\displaystyle \frac{1}{p}}\right)}\right)$	[34]

.

2.1. Generalized Ridge Regression (GRR)

In 2017, Yang and Emura presented a GRR estimator that uses non-uniform shrinkage instead of the conventional uniform shrinkage method, replacing the identity matrix with a diagonal matrix$\widehat{N}\left(\Delta \right)$ [44].

For a binary response variable $y_i\in \left\{\mathrm{0,1}\right\}$, the GRR estimator can be adapted by incorporating it into the penalized logistic regression framework. The parameter estimates ${\widehat{\theta }}_{GRR}$ are derived by minimizing the penalized negative log-likelihood, as follows:

$${\widehat{\theta }}_{GRR}=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\theta }{\mathrm{m}\mathrm{i}\mathrm{n}}\left\{-\sum _{i=1}^n\left[y_i\mathrm{l}\mathrm{o}\mathrm{g}\left({\pi }_i\right)+\left(1-y_i\right)\mathrm{l}\mathrm{o}\mathrm{g}\left(1-{\pi }_i\right)\right]+\lambda \cdot {\theta }^{\mathrm{\top }}\widehat{N}(\Delta )\theta \right\}$$

where ${\pi }_i={\displaystyle \frac{1}{1+\mathrm{e}\mathrm{x}\mathrm{p}\left(-x_i^{\top }\theta \right)}}$ is the predicted probability from logistic regression, $\lambda >0$ is the shrinkage parameter, $\Delta \ge 0$ is the threshold parameter, and $\widehat{N}(\Delta )=\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\widehat{n}}_1(\Delta ),\dots ,{\widehat{n}}_p(\Delta )\right)$ is a diagonal matrix encoding non-uniform penalties for each coefficient [45].

$${\widehat{n}}_i(\Delta )=\left\{\begin{array}{ll}{\displaystyle \frac{1}{2}},& \mathrm{if} z_i\ge \Delta \\ 1,& \mathrm{if} z_i<\Delta \end{array}\right.$$

where $z_i={\displaystyle \frac{{\widehat{\theta }}_i^0}{\mathrm{S}\mathrm{D}\left({\widehat{\theta }}^0\right)}}$ is a standardized initial estimate of ${\theta }_i$, $\mathrm{S}\mathrm{D}\left({\widehat{\theta }}^0\right)={\left\{{\displaystyle \frac{1}{p-1}}\sum _{i=1}^p{\left({\widehat{\theta }}_i^0-{\displaystyle \frac{1}{p}}\sum _{j=1}^p{\widehat{\theta }}_j^0\right)}^2\right\}}^{1/2}$, and ${\widehat{\theta }}_i^0={\displaystyle \frac{x_i^{\mathrm{\top }}y}{x_i{x}_i}}$ serves as an initial estimator based on a simple componentwise pseudo-regression.

This formulation allows for adaptive shrinkage, where coefficients associated with stronger initial signals (larger $z_i$) are penalized less, while weaker signals are shrunk more aggressively, thus promoting both interpretability and generalization in high-dimensional binary classification problems.

2.2. Least Absolute Shrinkage and Selection Operator (Lasso)

Lasso (Tibshirani, 1996) is an effective regression method that conducts variable selection and regularization at the same time, enhancing both the precision and clarity of statistical models [46]. Unlike ridge regression, which uses the squared $l_2$ norm, Lasso employs the $l_1$norm to reduce the total of squared differences while adhering to a limitation on the overall absolute size of the coefficients. This constraint promotes sparsity in the model, effectively mitigating multicollinearity and identifying a subset of truly relevant predictors [46,47].

$${\widehat{\theta }}_{\mathrm{lasso} }=\mathrm{a}\mathrm{r}\mathrm{g} \underset{\theta }{\mathrm{m}\mathrm{i}\mathrm{n}}\left\{\sum _{i=1}^n\left[-{\mathcal{l}}_i\left(\theta \right)\right]+\lambda \sum _{j=1}^p\left|{\theta }_j\right|\right\}$$

where $\mathcal{l}\left(\beta \right)$ is the log-likelihood of the binary logistic model, $\lambda \ge 0$ is the tuning (regularization) parameter, $\theta$ is the coefficient vector, $\sum _{j=1}^p\left|{\beta }_j\right|$ is the $l_1$ norm to shrink coefficients toward zero, n is the sample size, and p is the number of predictors.

2.3. Elastic Net (EN)

The EN, which was introduced by Zou and Hastie in 2005, extends Lasso by combining penalties from both the $l_1$ norm (Lasso) and $l_2$ norm (ridge), addressing limitations in variable selection [48]. This dual regularization reduces coefficients like ridge regression while enforcing sparsity by setting some to zero, as in the case of Lasso. The $l_1$ norm ensures model sparsity, while the $l_2$ norm alleviates restrictions on the number of selected predictors, enabling more flexibility [49]. EN’s ability to select more predictors than Lasso makes it a powerful tool for variable identification, as demonstrated in spectral data analysis [50]. For a binary outcome variable, the EN reduces the penalized negative log-likelihood of the logistic regression model to calculate the best-parameter vector.

Let:

$${\pi }_i={\displaystyle \frac{1}{1+\mathrm{e}\mathrm{x}\mathrm{p}\left(-x_i^{\mathrm{\top }}\theta \right)}}$$

Then the EN estimator becomes:

$${\widehat{\theta }}_{EN}=\mathrm{a}\mathrm{r}\mathrm{g} \underset{\theta }{\mathrm{m}\mathrm{i}\mathrm{n}}\left\{-\sum _{i=1}^n\left[y_i\mathrm{l}\mathrm{o}\mathrm{g}\left({\pi }_i\right)+\left(1-y_i\right)\mathrm{l}\mathrm{o}\mathrm{g}\left(1-{\pi }_i\right)\right]+\lambda \left[(1-\alpha )\sum _{j=1}^p{\theta }_j^2+\alpha \sum _{j=1}^p\left|{\theta }_j\right|\right]\right\}$$

where ${\pi }_i$ is the predicted probability that $Y_i=1$, $\theta$ is the coefficient vector, $\lambda$ is the regularization strength, and $\alpha$ is the weight between the $l_1$ norm and $l_2$ norm ($0=$ Ridge, $1=$ Lasso$)$.

In this study for GRR, Lasso, and EN, the optimum $\lambda$, tuning parameter was obtained using 5-fold cross-validation.

2.4. Simulation Study

The objective of this research is to evaluate the effectiveness of the leading ridge estimators against LR, Lasso, EN, and GRR, focusing on minimizing MSE and enhancing accuracy. Due to the infeasibility of a theoretical comparison, a Monte Carlo simulation study has been performed utilizing the R programming language [51]. The method used for generating data for the models adheres to a recognized procedure [19].

$$X_{ij}={\left(1-{\rho }^2\right)}^{1/2}{d}_{ij}+\rho d_{i(p+1)},i=\mathrm{1,2},\dots ,n; j=\mathrm{1,2},\dots ,P$$

where $\rho$ indicates the correlation between two predictors, $d_{ij}$ denotes the independent pseudo-random variable, and P indicates the number of independent variables. Moreover, the response, $Y$ was obtained through the Bernoulli $\left({\pi }_i\right)$ distribution, where:

$${\pi }_i={\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{x}}_i^{\prime }\theta \right)}{1+\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathrm{x}}_i^{\prime }\theta \right)}}$$

In the modeling process, we considered three distinct values for the number of independent variables ($P=3,5$, and 10), the correlation between the predictors ($\rho =0.80,0.90,0.95$, and 0.99), and three sample sizes ($n=100,200$, and 300). The procedure for generating data for the predictors was carried out according to the established values of P, $\rho ,$ and n. In order to confirm the reliability of the findings, the experiment was conducted 5000 times.

Following the theoretical foundation established by Newhouse and Oman (1971), which posits that the MSE is minimized when the coefficient vector$\theta$ aligns with the normalized eigenvector corresponding to the largest eigenvalue of the $X^{\prime }X$ matrix, the simulation study computes MSE by comparing estimated coefficient vectors, $\theta$, derived from various models, to the true$\theta$ [52]. This true $\theta$ is determined for each simulation by extracting and normalizing the eigenvector associated with the largest eigenvalue of $X^{\prime }X$, ensuring ${\theta }^{\prime }\theta =1$, thereby providing a theoretically grounded benchmark for evaluating estimator performance through the calculation of the average squared difference between $\widehat{\theta }$ and $\theta$. The average estimated MSE was computed using the formula applied to all simulations [39].

$$MSE\left({\widehat{\theta }}_j^{*}\right)={\displaystyle \frac{1}{N}}{\sum }_{j=1}^P{\left({\widehat{\theta }}_j-\theta \right)}^{\prime }\left({\widehat{\theta }}_j-\theta \right)$$

where ${\widehat{\theta }}_j^{*}$ is the estimators, $\theta$ is the parameter, and N = 5000.

In the evaluation of the classification performance, accuracy was used as a key metric to quantify the proportion of correctly predicted binary outcomes. For each simulation, we applied a 70% training and $30\mathrm{\%}$ testing split on the simulated data. Models were trained on the $70\mathrm{\%}$ training subset, and predictions were generated for the held-out $30\mathrm{\%}$ test set. The predicted values, which were initially probabilities derived from the logistic regression link function, were converted into binary classifications using a threshold of 0.5. Specifically, if the predicted probability exceeded 0.5, the observation was classified as 1; otherwise, it was classified as 0.

The models’ overall accuracy was calculated using a confusion matrix based on the total number of correctly classified observations [53].

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

where TP—true positive, TN—negative, FP—false positive, and FN—false negative, accuracy serves as the comprehensive metric for the model’s ability to make correct predictions across the entire dataset. For each simulation, accuracy values were calculated, and the average accuracy was then computed across all simulations, considering different combinations of parameters (n, P, and $\rho$) and models.

3. Results & Discussion

In this part, we carried out an extensive simulation study to assess the effectiveness of ridge estimators relative to conventional logistic regression, elastic net, Lasso, and generalized ridge regression. The results, summarized in

Table 2. Estimated MSE values and Accuracy for correlation 0.80.

Model	MSE									Accuracy
	P = 3			P = 5			P = 10			P = 3			P = 5			P = 10
	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300
${\mathit{k}}_{\mathit{A}\mathit{S}}$	0.099	0.099	0.100	0.096	0.098	0.099	0.091	0.095	0.097	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{A}1}$	0.100	0.100	0.100	0.099	0.099	0.100	0.097	0.098	0.099	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{D}}$	0.297	0.223	0.192	0.247	0.154	0.111	0.602	0.187	0.120	0.730	0.732	0.733	0.757	0.763	0.766	0.787	0.799	0.808
${\mathit{k}}_{\mathit{L}\mathit{A}1}$	0.100	0.111	0.119	0.051	0.049	0.043	0.067	0.061	0.057	0.735	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{K}1}$	0.092	0.092	0.092	0.085	0.087	0.086	0.086	0.089	0.089	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{K}2}$	0.088	0.089	0.089	0.070	0.075	0.076	0.066	0.072	0.075	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{K}3}$	0.096	0.095	0.095	0.097	0.097	0.096	0.099	0.099	0.099	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{K}4}$	0.093	0.093	0.092	0.089	0.090	0.089	0.090	0.092	0.093	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{K}5}$	0.091	0.091	0.091	0.083	0.085	0.085	0.082	0.086	0.087	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{L}\mathit{A}2}$	0.358	0.341	0.328	0.145	0.143	0.125	0.056	0.063	0.061	0.730	0.731	0.732	0.762	0.764	0.766	0.809	0.809	0.813
${\mathit{k}}_{\mathit{L}1}$	0.358	0.342	0.328	0.145	0.143	0.125	0.056	0.063	0.061	0.730	0.731	0.732	0.762	0.764	0.766	0.809	0.809	0.813
${\mathit{k}}_{\mathit{L}\mathit{A}3}$	0.117	0.132	0.141	0.044	0.046	0.041	0.049	0.043	0.040	0.735	0.734	0.734	0.769	0.769	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{A}2}$	0.358	0.342	0.328	0.146	0.143	0.125	0.056	0.063	0.061	0.730	0.731	0.732	0.762	0.764	0.766	0.809	0.809	0.813
${\mathit{k}}_{\mathit{M}1}$	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{M}2}$	0.358	0.342	0.328	0.146	0.143	0.125	0.056	0.063	0.061	0.730	0.731	0.732	0.762	0.764	0.766	0.809	0.809	0.813
${\mathit{k}}_{\mathit{L}2}$	0.097	0.098	0.098	0.094	0.096	0.096	0.093	0.095	0.096	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{L}3}$	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{K}6}$	0.099	0.099	0.100	0.096	0.098	0.099	0.090	0.095	0.097	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{L}4}$	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{A}\mathit{L}1}$	0.086	0.086	0.087	0.056	0.055	0.051	0.062	0.057	0.054	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{A}\mathit{L}2}$	0.089	0.090	0.090	0.049	0.049	0.045	0.050	0.046	0.044	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{K}\mathit{L}1}$	0.090	0.090	0.090	0.085	0.086	0.085	0.089	0.089	0.089	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
${\mathit{k}}_{\mathit{K}\mathit{L}2}$	0.088	0.088	0.088	0.082	0.082	0.081	0.089	0.088	0.088	0.736	0.734	0.734	0.770	0.770	0.770	0.815	0.816	0.818
LR	0.593	0.417	0.369	0.509	0.237	0.171	0.431	0.260	0.151	0.721	0.727	0.729	0.746	0.758	0.762	0.773	0.794	0.804
Lasso	0.383	0.344	0.328	0.210	0.163	0.135	0.161	0.103	0.084	0.726	0.729	0.731	0.756	0.761	0.764	0.797	0.803	0.809
EN	0.332	0.305	0.297	0.164	0.133	0.112	0.125	0.078	0.065	0.728	0.730	0.732	0.760	0.763	0.765	0.803	0.806	0.811
GRR	0.280	0.268	0.264	0.114	0.100	0.087	0.064	0.047	0.041	0.732	0.732	0.733	0.764	0.766	0.768	0.809	0.812	0.815

,

Table 3. Estimated MSE values and Accuracy for correlation 0.90.

Model	MSE									Accuracy
	P = 3			P = 5			P = 10			P = 3			P = 5			P = 10
	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300
${\mathit{k}}_{\mathit{A}\mathit{S}}$	0.098	0.099	0.099	0.095	0.097	0.098	0.088	0.093	0.096	0.746	0.746	0.746	0.782	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{A}1}$	0.099	0.100	0.100	0.098	0.099	0.099	0.096	0.098	0.098	0.746	0.746	0.746	0.782	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{D}}$	0.357	0.266	0.236	0.342	0.207	0.159	1.000	0.318	0.205	0.742	0.744	0.745	0.773	0.776	0.779	0.806	0.817	0.822
${\mathit{k}}_{\mathit{L}\mathit{A}1}$	0.097	0.103	0.119	0.053	0.048	0.047	0.068	0.062	0.059	0.746	0.746	0.746	0.781	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{K}1}$	0.093	0.093	0.093	0.088	0.089	0.089	0.087	0.090	0.090	0.746	0.746	0.746	0.782	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{K}2}$	0.088	0.087	0.088	0.072	0.075	0.077	0.068	0.073	0.076	0.746	0.746	0.746	0.782	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{K}3}$	0.097	0.096	0.096	0.097	0.097	0.097	0.098	0.099	0.099	0.746	0.746	0.746	0.782	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{K}4}$	0.094	0.093	0.093	0.091	0.091	0.091	0.091	0.093	0.094	0.746	0.746	0.746	0.782	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{K}5}$	0.091	0.090	0.091	0.084	0.086	0.087	0.084	0.087	0.089	0.746	0.746	0.746	0.782	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{L}\mathit{A}2}$	0.350	0.343	0.347	0.134	0.143	0.145	0.044	0.053	0.061	0.742	0.743	0.744	0.778	0.778	0.780	0.827	0.828	0.829
${\mathit{k}}_{\mathit{L}1}$	0.350	0.343	0.347	0.134	0.143	0.145	0.044	0.053	0.061	0.742	0.743	0.744	0.779	0.778	0.780	0.827	0.828	0.829
${\mathit{k}}_{\mathit{L}\mathit{A}3}$	0.116	0.125	0.145	0.046	0.044	0.046	0.049	0.043	0.042	0.746	0.746	0.746	0.781	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{A}2}$	0.351	0.344	0.347	0.135	0.143	0.145	0.044	0.054	0.061	0.742	0.743	0.744	0.779	0.778	0.780	0.828	0.828	0.829
${\mathit{k}}_{\mathit{M}1}$	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.746	0.746	0.746	0.782	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{M}2}$	0.351	0.344	0.347	0.135	0.143	0.145	0.044	0.054	0.061	0.742	0.743	0.744	0.779	0.778	0.780	0.828	0.828	0.829
${\mathit{k}}_{\mathit{L}2}$	0.097	0.098	0.098	0.095	0.096	0.097	0.094	0.096	0.096	0.746	0.746	0.746	0.782	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{L}3}$	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.746	0.746	0.746	0.782	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{K}6}$	0.098	0.099	0.099	0.094	0.097	0.098	0.087	0.093	0.095	0.746	0.746	0.746	0.782	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{L}4}$	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.746	0.746	0.746	0.782	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{A}\mathit{L}1}$	0.084	0.081	0.087	0.059	0.055	0.055	0.065	0.060	0.058	0.746	0.746	0.746	0.781	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{A}\mathit{L}2}$	0.086	0.084	0.090	0.052	0.049	0.049	0.053	0.049	0.048	0.746	0.746	0.746	0.781	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{K}\mathit{L}1}$	0.089	0.088	0.090	0.084	0.084	0.084	0.088	0.088	0.089	0.746	0.746	0.746	0.782	0.782	0.782	0.830	0.831	0.832
${\mathit{k}}_{\mathit{K}\mathit{L}2}$	0.087	0.085	0.087	0.081	0.080	0.080	0.088	0.087	0.087	0.746	0.746	0.746	0.782	0.782	0.782	0.830	0.831	0.832
LR	0.786	0.490	0.424	0.871	0.362	0.262	2.987	0.519	0.294	0.731	0.738	0.741	0.759	0.770	0.775	0.787	0.811	0.818
Lasso	0.461	0.385	0.368	0.302	0.216	0.189	0.277	0.153	0.129	0.738	0.741	0.742	0.773	0.775	0.778	0.816	0.822	0.825
EN	0.376	0.326	0.319	0.222	0.160	0.144	0.205	0.103	0.090	0.741	0.743	0.743	0.776	0.777	0.779	0.822	0.825	0.827
GRR	0.289	0.263	0.268	0.129	0.104	0.099	0.069	0.052	0.048	0.743	0.744	0.745	0.779	0.780	0.781	0.826	0.828	0.830

,

Table 4. Estimated MSE values & Accuracy for correlation 0.95.

Model	MSE									Accuracy
	P = 3			P = 5			P = 10			P = 3			P = 5			P = 10
	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300
$k_{AS}$	0.098	0.099	0.099	0.094	0.097	0.098	0.087	0.092	0.095	0.752	0.749	0.751	0.787	0.787	0.789	0.837	0.837	0.837
$k_{A1}$	0.099	0.100	0.100	0.098	0.099	0.099	0.095	0.097	0.098	0.752	0.749	0.751	0.787	0.787	0.789	0.837	0.837	0.837
$k_D$	0.445	0.342	0.279	0.469	0.274	0.212	2.304	0.526	0.338	0.750	0.747	0.750	0.780	0.784	0.785	0.818	0.825	0.829
$k_{LA1}$	0.100	0.114	0.115	0.053	0.047	0.047	0.068	0.062	0.059	0.753	0.749	0.751	0.787	0.787	0.788	0.837	0.837	0.837
$k_{K1}$	0.095	0.096	0.095	0.091	0.092	0.092	0.091	0.092	0.092	0.752	0.749	0.751	0.787	0.787	0.789	0.837	0.837	0.837
$k_{K2}$	0.090	0.090	0.088	0.072	0.076	0.077	0.070	0.076	0.078	0.752	0.749	0.751	0.787	0.787	0.789	0.837	0.837	0.837
$k_{K3}$	0.098	0.098	0.097	0.097	0.098	0.098	0.099	0.099	0.099	0.752	0.749	0.751	0.787	0.787	0.789	0.837	0.837	0.837
$k_{K4}$	0.095	0.095	0.094	0.092	0.093	0.093	0.093	0.094	0.095	0.752	0.749	0.751	0.787	0.787	0.789	0.837	0.837	0.837
$k_{K5}$	0.092	0.093	0.092	0.086	0.088	0.089	0.087	0.089	0.091	0.752	0.749	0.751	0.787	0.787	0.789	0.837	0.837	0.837
$k_{LA2}$	0.343	0.366	0.354	0.113	0.132	0.145	0.037	0.044	0.051	0.751	0.748	0.750	0.786	0.786	0.787	0.836	0.836	0.836
$k_{L1}$	0.344	0.366	0.354	0.113	0.132	0.145	0.037	0.044	0.051	0.751	0.748	0.750	0.786	0.786	0.787	0.836	0.836	0.836
$k_{LA3}$	0.121	0.139	0.141	0.045	0.042	0.046	0.049	0.043	0.041	0.753	0.749	0.751	0.787	0.787	0.788	0.837	0.837	0.837
$k_{A2}$	0.344	0.367	0.355	0.114	0.132	0.145	0.037	0.044	0.051	0.751	0.748	0.750	0.786	0.786	0.787	0.836	0.836	0.836
$k_{M1}$	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.752	0.749	0.751	0.787	0.787	0.789	0.837	0.837	0.837
$k_{M2}$	0.344	0.367	0.355	0.114	0.132	0.145	0.037	0.044	0.051	0.751	0.748	0.750	0.786	0.786	0.787	0.836	0.836	0.836
$k_{L2}$	0.098	0.099	0.099	0.096	0.097	0.098	0.096	0.097	0.097	0.752	0.749	0.751	0.787	0.787	0.789	0.837	0.837	0.837
$k_{L3}$	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.752	0.749	0.751	0.787	0.787	0.789	0.837	0.837	0.837
$k_{K6}$	0.098	0.099	0.099	0.094	0.097	0.098	0.086	0.092	0.095	0.752	0.749	0.751	0.787	0.787	0.789	0.837	0.837	0.837
$k_{L4}$	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.752	0.749	0.751	0.787	0.787	0.789	0.837	0.837	0.837
$k_{AL1}$	0.086	0.088	0.085	0.060	0.055	0.056	0.065	0.061	0.059	0.753	0.749	0.751	0.787	0.787	0.788	0.837	0.837	0.837
$k_{AL2}$	0.088	0.092	0.088	0.052	0.048	0.050	0.054	0.050	0.049	0.753	0.749	0.751	0.787	0.787	0.788	0.837	0.837	0.837
$k_{KL1}$	0.089	0.090	0.089	0.083	0.083	0.084	0.088	0.088	0.088	0.752	0.749	0.751	0.787	0.787	0.789	0.837	0.837	0.837
$k_{KL2}$	0.087	0.088	0.086	0.080	0.079	0.080	0.088	0.087	0.087	0.752	0.749	0.751	0.787	0.787	0.789	0.837	0.837	0.837
LR	1.204	0.670	0.524	7.344	0.626	0.420	6.595	1.045	0.572	0.737	0.742	0.747	0.766	0.776	0.781	0.794	0.816	0.823
Lasso	0.600	0.481	0.427	0.424	0.297	0.256	0.416	0.232	0.188	0.747	0.746	0.749	0.780	0.782	0.784	0.828	0.830	0.832
EN	0.463	0.384	0.348	0.296	0.199	0.177	0.281	0.142	0.116	0.749	0.747	0.750	0.783	0.785	0.786	0.831	0.833	0.834
GRR	0.302	0.282	0.262	0.124	0.101	0.098	0.065	0.050	0.045	0.751	0.749	0.751	0.786	0.786	0.788	0.835	0.836	0.836

and

Table 5. Estimated MSE values & Accuracy for correlation 0.99.

Model	MSE									Accuracy
	P = 3			P = 5			P = 10			P = 3			P = 5			P = 10
	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300	n = 100	n = 200	n = 300
$k_{AS}$	0.098	0.099	0.099	0.094	0.097	0.098	0.086	0.092	0.094	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{A1}$	0.099	0.100	0.100	0.098	0.099	0.099	0.095	0.097	0.098	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_D$	0.824	0.528	0.382	1.250	0.617	0.413	4.806	1.625	1.010	0.754	0.755	0.754	0.790	0.790	0.790	0.828	0.835	0.838
$k_{LA1}$	0.099	0.111	0.111	0.058	0.052	0.049	0.069	0.063	0.060	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{K1}$	0.099	0.099	0.099	0.098	0.098	0.098	0.097	0.097	0.097	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{K2}$	0.090	0.090	0.088	0.078	0.080	0.080	0.078	0.082	0.083	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{K3}$	0.099	0.099	0.099	0.099	0.099	0.099	0.099	0.099	0.099	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{K4}$	0.097	0.097	0.097	0.097	0.097	0.097	0.097	0.098	0.098	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{K5}$	0.093	0.094	0.093	0.091	0.092	0.093	0.093	0.095	0.095	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{LA2}$	0.293	0.325	0.328	0.097	0.110	0.118	0.030	0.032	0.035	0.756	0.755	0.754	0.795	0.792	0.791	0.842	0.843	0.843
$k_{L1}$	0.293	0.325	0.328	0.097	0.110	0.118	0.030	0.032	0.035	0.756	0.755	0.754	0.795	0.792	0.791	0.842	0.843	0.843
$k_{LA3}$	0.120	0.137	0.137	0.052	0.049	0.048	0.050	0.044	0.042	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{A2}$	0.294	0.326	0.328	0.097	0.110	0.118	0.030	0.032	0.035	0.756	0.755	0.754	0.795	0.792	0.791	0.842	0.843	0.843
$k_{M1}$	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{M2}$	0.294	0.326	0.328	0.097	0.110	0.118	0.030	0.032	0.035	0.756	0.755	0.754	0.795	0.792	0.791	0.842	0.843	0.843
$k_{L2}$	0.099	0.099	0.099	0.098	0.099	0.099	0.098	0.098	0.099	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{L3}$	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{K6}$	0.098	0.099	0.099	0.093	0.097	0.098	0.085	0.092	0.094	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{L4}$	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.100	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{AL1}$	0.085	0.087	0.083	0.062	0.058	0.056	0.063	0.059	0.057	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{AL2}$	0.088	0.090	0.086	0.056	0.053	0.051	0.052	0.049	0.048	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{KL1}$	0.089	0.089	0.088	0.084	0.084	0.084	0.088	0.088	0.088	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
$k_{KL2}$	0.087	0.087	0.085	0.081	0.080	0.080	0.088	0.087	0.086	0.756	0.755	0.754	0.795	0.792	0.792	0.842	0.843	0.843
LR	4.497	2.044	1.369	7.543	2.699	1.709	9.653	5.259	2.754	0.740	0.748	0.749	0.774	0.780	0.783	0.796	0.822	0.829
Lasso	1.356	0.901	0.729	1.217	0.744	0.586	1.711	0.580	0.452	0.753	0.753	0.752	0.791	0.789	0.789	0.836	0.840	0.840
EN	1.051	0.671	0.522	0.918	0.477	0.359	1.024	0.332	0.252	0.753	0.754	0.753	0.792	0.790	0.790	0.839	0.842	0.842
GRR	0.281	0.265	0.244	0.114	0.097	0.089	0.045	0.037	0.034	0.756	0.755	0.754	0.795	0.792	0.791	0.842	0.843	0.843

, present the MSE and classification accuracy of the estimators under different conditions. Specifically, this study examines the impact of correlation coefficients (0.80, 0.90, 0.95, and 0.99). The findings, systematically detailed in these tables, illustrate the effects of different correlation structures, the number of independent variables, and sample sizes on the estimator performance.

Table 2 compares the MSE and Accuracy across various regression models under moderate multicollinearity (correlation = 0.80) for different predictor counts (P = 3, 5, 10) and sample sizes (n = 100, 200, 300). For P = 3, $k_{KL2}$consistently achieves the lowest MSE (0.088), demonstrating its effectiveness in low-dimensional settings. As P increases to 5, $k_{KL2}$ remains optimal, with an MSE of 0.082. However, for P = 10, GRR performs best, with an MSE of 0.064 at n = 300, highlighting its capacity for high-dimensional data. LR exhibits the highest MSE, indicating limitations in multicollinear environments. EN and Lasso show competitive performance, especially with larger P and n, benefiting from regularization.

In terms of accuracy, most models show stable performance, with slight improvements as sample size increases. For P = 3, ridge estimators like $k_{KL2}$, $k_{K5}$, and $k_{K1}$ maintain an accuracy of 0.736 at n = 100, rising to 0.738 at n = 300. With P = 5, models such as GRR, EN, and Lasso achieve accuracies of 0.764 to 0.766 at n = 300, demonstrating effectiveness in moderate complexity. For P = 10, GRR leads with an accuracy of 0.812 at n = 300, showing its robustness in high-dimensional spaces.

Table 3 shows that under high multicollinearity (correlation = 0.90), MSE values increase, indicating greater prediction difficulty. Focusing on the non-penalized methods first, we see that LR exhibits the highest MSE, particularly as $P$ increases, indicating its vulnerability to overfitting with more predictors. GRR substantially reduces MSE compared to LR, highlighting the benefits of $l_2$ regularization. As expected, increasing sample size ($n$) generally decreases MSE for all methods, as more data provides a better estimate of the underlying relationships. For the ridge estimators, we observe generally low and stable MSE across different $P$ and $n$, suggesting that these methods are less sensitive to the increase in predictor dimension in this context. However, some estimators like $k_D,{ k}_{LA2},k_{L1},{ k}_{A2}$, and $k_{M2}$ show relatively higher MSE compared to others, indicating potential challenges in capturing the underlying data structure with specific kernels. Notably, $k_{AL1},{ k}_{AL2},k_{KL1}$, and $k_{KL2}$ achieve some of the lowest MSE values, suggesting good performance in this scenario.

Regarding accuracy, the table displays the proportion of correctly classified instances for each estimator under the same conditions. Like the MSE trends, the accuracy tends to improve with increasing sample size ($n$) for most methods, reflecting the benefit of more data. LR, which suffered from a high MSE, also exhibits the lowest accuracy, reinforcing the detrimental effect of overfitting. In contrast, the penalized methods, Lasso and EN, demonstrate higher accuracy, with GRR achieving the highest accuracy among the estimators. The ridge estimators, except for $k_D,{ k}_{LA2},{ k}_{L1},k_{A2}$, and $k_{M2}$, achieve high and relatively consistent accuracy across different $P$ and $n$, often surpassing the traditional methods. This suggests that the ridge estimators are effective in capturing the non-linear relationships in the data, leading to better classification performance. The methods $k_{AL1},k_{AL2},{ k}_{KL1}$, and $k_{KL2}$, which had low MSE, also maintain high accuracy, further indicating their effectiveness. The methods $k_D$, $k_{LA2},{ k}_{L1},{ k}_{A2}$, and $k_{M2}$ that showed relatively higher MSE also show lower accuracy compared to other ridge estimators, suggesting a link between MSE and classification performance.

Table 4 presents the MSE for various estimators across different parameter settings (P = 3, 5, 10) and sample sizes (n = 100, 200, 300) given a correlation of 0.95. Firstly, within the ridge estimators, we observe a general trend of decreasing MSE as the sample size (n) increases, as expected due to the larger amount of information available for estimation. Notably, the estimators $k_{AL1}$,${ k}_{AL2}, k_{KL1},$and $k_{KL2}$consistently demonstrate lower MSE values, particularly as P increases, suggesting superior performance in capturing the underlying relationship under higher dimensionality. In contrast, the $K_{D }$estimator exhibits significantly higher MSE, indicating poor performance and potential instability. Among the traditional methods, GRR consistently achieves the lowest MSE across all settings, highlighting its effectiveness in handling high correlations and varying dimensions. EN and Lasso also demonstrate reasonable performance, with the MSE decreasing as the sample size increases, albeit not as effectively as the GRR. LR shows the highest MSE, especially with larger P values, indicating its sensitivity to high correlation and dimensionality.

The accuracy metric presented alongside the MSE provides insights into the classification performance of the estimators. Similar to the MSE trends, we observe that the accuracy generally improves with increasing sample size across all estimators. However, the ridge estimators show remarkably consistent and high accuracy, often reaching 83.7% for P = 10 and n = 300. This suggests that these estimators are robust and effective in capturing underlying patterns, even with increased dimensionality. GRR, EN, and Lasso also exhibit good accuracy, with GRR showing slightly better performance, especially at higher P values. LR, while showing improvement with larger sample sizes, lags the other methods in terms of accuracy, consistent with its higher MSE.

Table 5 reveals the impact of extremely high multicollinearity (correlation of 0.99) on the MSE. The near-perfect correlation increases the MSE values across all models, highlighting the difficulty of accurate prediction under extreme dependencies. The ridge estimators, particularly $k_{AL1}$,${ k}_{AL2}, k_{KL1},$and $k_{KL2}$ consistently exhibit the lowest MSE across all parameter settings (P = 3, 5, 10) and sample sizes. Importantly, their MSE values decrease as the sample size increases, demonstrating the desired consistency and effectiveness in handling the strong dependencies. In contrast, the $k_D$ estimator shows significantly higher MSE, indicating poor performance and a lack of robustness to high correlation. Among the classical methods, GRR stands out with the lowest MSE and exhibits a clear trend of decreasing MSE with increasing sample size, suggesting a superior ability to mitigate the impact of high correlation. EN and Lasso also show reasonable performance, with MSE decreasing as the sample size increases, although not as effectively as GRR. LR exhibits the highest MSE, especially with larger P values, and shows less consistent improvement with increasing sample size, indicating its struggle to handle the strong linear dependency of the data.

From Table 5, we observe that the accuracy generally improves with increasing sample size across all estimators. The ridge estimators, specifically $k_{AL1}$,${ k}_{AL2}, k_{KL1},$and $k_{KL2}$, demonstrate remarkably consistent and high accuracy, often reaching 84.3% for P = 10 and n = 300. This suggests that these estimators are robust and effective in capturing the underlying patterns, even with extreme correlations. GRR also exhibits good accuracy, showing slightly better performance, especially at higher P values. EN and Lasso show comparable accuracy to GRR, while LR lags the other methods in terms of accuracy, consistent with its higher MSE.

4. Applications

We conducted an analysis of two real-life datasets to effectively illustrate the findings derived from the simulation presented in this section.

4.1. Municipal Data

In this part, the models are applied to the municipal dataset from Statistics Sweden [54]. A binary logistic regression model is employed, with the dependent variable indicating whether the net population expands (coded as 1) or contracts (coded as 0). The model aims to explain the response variable using the following predictors: X1 (population), X2 (number of unemployed individuals), X3 (number of newly constructed buildings), and X4 (number of bankrupt firms).

The full dataset consists of 271 entries, which correspond to the municipalities in Sweden. The correlation plot illustrated in Figure 1 indicates that all correlation values are above 0.90, with some approaching 0.99. Furthermore, the condition number is 38.33, suggesting a notable issue with multicollinearity within the data [55].

Table 6. Comparison of MSE and Accuracy for municipal data.

Estimator	MSE	Accuracy
$k_{AS}$	0.243	0.671
$k_{A1}$	0.243	0.671
$k_D$	0.560	0.622
$k_{LA1}$	0.167	0.671
$k_{K1}$	0.250	0.671
$k_{K2}$	0.247	0.671
$k_{K3}$	0.250	0.671
$k_{K4}$	0.249	0.671
$k_{K5}$	0.249	0.671
$k_{LA2}$	0.160	0.659
$k_{L1}$	0.160	0.659
$k_{LA3}$	0.151	0.671
$k_{A2}$	0.160	0.659
$k_{M1}$	0.250	0.671
$k_{M2}$	0.160	0.659
$k_{L2}$	0.250	0.671
$k_{L3}$	0.250	0.671
$k_{K6}$	0.243	0.671
$k_{L4}$	0.250	0.671
$k_{AL1}$	0.199	0.671
$k_{AL2}$	0.179	0.671
$k_{KL1}$	0.217	0.671
$k_{KL2}$	0.209	0.671
LR	0.711	0.622
Lasso	0.187	0.646
EN	0.188	0.646
GRR	0.147	0.671

presents the performance of various statistical estimators, primarily focusing on shrinkage methods, evaluated based on MSE and Accuracy. Lower MSE values indicate a closer approximation of the estimated values to the true values, suggesting higher precision and a reduced prediction error. Conversely, higher accuracy scores reflect a greater proportion of correct predictions, implying a better classification performance. Examining the estimators, we observe that GRR exhibits the lowest MSE (0.1470747), indicating superior precision among the tested models, which aligns with the simulation results since this is a large sample. This suggests that the GRR, by introducing a form of regularization that shrinks coefficients towards zero, effectively mitigates overfitting and improves the model’s predictive accuracy. In contrast, LR, without any shrinkage, shows the highest MSE (0.7111922), highlighting the potential for substantial prediction errors in unregularized models, especially in datasets with multicollinearity or high dimensionality. The Lasso and EN estimators, which also incorporate regularization, fall within a moderate range of MSE values, demonstrating their ability to balance bias and variance, although not as effectively as GRR for this data (Figure 2).

In terms of accuracy, most of the estimators, including GRR and the various ridge estimators, such as $k_{AS}$, $k_{A1}$, and $k_D$ achieve an accuracy of approximately 67.1%, suggesting a consistent ability to correctly classify a significant portion of the observations. This uniform accuracy across several shrinkage estimators implies that, while the MSE varies significantly, the overall classification performance remains relatively stable. However, the LR estimator exhibits a lower accuracy of 62.2%, aligning with its higher MSE and indicating a less reliable classification capability. The Lasso and EN estimators, like their MSE results, show moderate accuracy levels (64.63%), further demonstrating their balanced performance. The consistency of high accuracy among the ridge estimators, despite variations in their specific shrinkage parameters, suggests that these methods are robust in classification tasks, potentially due to their specific adaptations to the dataset’s characteristics. Overall, the findings suggest that shrinkage methods, particularly GRR, are effective in minimizing prediction errors and maintaining high classification accuracy for large data, highlighting the importance of regularization in statistical modeling.

4.2. Cancer Remission Data

In contrast to the previous analysis, which examined a larger dataset of approximately 271 samples, we now focus on a smaller dataset to further evaluate the performance of the estimators. Specifically, we analyze the Cancer Remission dataset [56,57], which consists of 27 observations. This dataset provides a valuable opportunity to assess the performance of estimators in a setting with a limited sample size, offering insights into their robustness and effectiveness in small-sample scenarios. The response variable is binary, indicating whether a patient achieves complete cancer remission (Y = 1) or not (Y = 0). The dataset includes observations from 27 patients, of whom nine experienced complete remission. The explanatory variables in the dataset are standardized such that $X^{\prime }X$ corresponds to a correlation matrix.

The condition number computed as $k=\sqrt{{\displaystyle \frac{{\lambda }_{\mathrm{m}\mathrm{a}\mathrm{x}}}{{\lambda }_{\mathrm{m}\mathrm{i}\mathrm{n}}}}}$ $=201.33$. This high condition number and Figure 3 strongly suggest the presence of severe multicollinearity, which can adversely impact the stability and reliability of parameter estimates in regression models [55].

Table 7. Comparison of MSE and Accuracy for cancer remission data.

Estimator	MSE	Accuracy
$k_{AS}$	0.129	0.667
$k_{A1}$	0.123	0.667
$k_D$	68.356	1.000
$k_{LA1}$	0.434	0.667
$k_{K1}$	0.367	0.667
$k_{K2}$	2.041	1.000
$k_{K3}$	0.139	0.667
$k_{K4}$	0.156	0.667
$k_{K5}$	0.626	0.741
$k_{LA2}$	3.234	1.000
$k_{L1}$	3.138	1.000
$k_{LA3}$	0.761	0.778
$k_{A2}$	3.668	1.000
$k_{M1}$	0.127	0.667
$k_{M2}$	4.478	1.000
$k_{L2}$	0.114	0.667
$k_{L3}$	0.127	0.667
$k_{K6}$	0.244	0.667
$k_{L4}$	0.127	0.667
$k_{AL1}$	0.148	0.667
$k_{AL2}$	0.177	0.667
$k_{KL1}$	0.225	0.667
$k_{KL2}$	0.225	0.667
LR	376.414	1.000
Lasso	33.097	1.000
EN	19.657	1.000
GRR	0.717	0.778

presents the MSE and accuracy for various models, including ridge regression with different shrinkage estimators, LR, Lasso, EN, and GRR. Several models, particularly those with high MSE values like LR, Lasso, EN, and some ridge estimators, achieve perfect accuracy (100%). However, with a sample size of only 27, this perfect accuracy strongly suggests overfitting. Overfitting occurs when a model conforms too closely to the training data, picking up noise instead of true patterns, which results in weak performance on new, unseen data. The models with lower MSE values, such as $k_{L2}$, $k_{L3}$, and $k_{AS}$, while not achieving perfect accuracy, might offer a more robust representation of the underlying relationships in the data, as they are less prone to overfitting.

5. Concluding Remarks

This study aims to evaluate the effectiveness of ridge regression, GRR, Lasso, and Elastic Net within the context of a logistic regression model by balancing the MSE with prediction accuracy. Given that a theoretical assessment of the estimators cannot be performed, a comprehensive simulation study has been carried out to assess their performance across various parametric conditions. The simulation studies, focusing on varying levels of correlation (0.80, 0.90, 0.95, and 0.99), consistently demonstrate the superior performance of ridge estimators with varying penalties, particularly $k_{AL1}$, ${\mathrm{k}}_{\mathrm{A}\mathrm{L}2}, k_{KL1},$and $k_{KL2}$, for small samples and GRR for large samples in terms of both MSE and accuracy. These methods exhibit desirable characteristics, such as lower MSE, indicating a better fit and consistent reduction of MSE with increasing sample size, highlighting robustness. Furthermore, they achieve high accuracy, suggesting reliable prediction capabilities even under severe multicollinearity. The simulation findings are corroborated by the real-world applications involving municipal and cancer remission data. Specifically, in the municipal data, characterized by high correlations among predictors, the GRR achieves the lowest MSE, aligning with the simulation results and demonstrating its effectiveness in mitigating overfitting and improving prediction accuracy in large multicollinear datasets. In the cancer remission data, a small-sample scenario with high multicollinearity, the models with lower MSE, including several ridge estimators and GRR, exhibit more realistic accuracy, suggesting robustness and reduced overfitting compared to models with inflated accuracy and high MSE, which are indicative of overfitting.

The alignment of the simulation findings with actual applications highlights the significance of using suitable statistical techniques, especially when dealing with multicollinearity and different sample sizes. The ridge estimators and GRR emerge as robust and reliable choices, demonstrating their ability to balance bias and variance, minimize prediction errors, and maintain high classification accuracy. These findings have significant implications for practical applications, highlighting the potential of these methods to enhance predictive modeling in diverse fields where multicollinearity and limited sample sizes are common challenges.

While ridge estimators and GRR show strong predictive performance, it is important to note the limitations of logistic regression, particularly the assumption of linearity in the logit and the interpretability challenges posed by coefficient shrinkage, which can bias odds ratio estimates [58]. Future research could explore advanced regularization techniques such as Smoothly Clipped Absolute Deviation (SCAD), which was proposed by Fan and Li (2001) [59], and adaptive Lasso, which offers variable selection capabilities and improved theoretical properties. Bayesian ridge regression also provides a flexible framework for incorporating prior information and quantifying uncertainty. Additionally, machine learning models, such as random forests and gradient boosting, can effectively handle multicollinearity and capture complex nonlinear relationships, offering valuable alternatives for high-dimensional and correlated data [48].