Shrinkage Approaches for Ridge-Type Estimators Under Multicollinearity

Abstract

Multicollinearity is a common issue in regression analyses that occurs when some predictor variables are highly correlated, leading to unstable least squares estimates of model parameters. Various estimation strategies have been proposed to address this problem. In this study, we enhanced a ridge-type estimator by incorporating pretest and shrinkage techniques. We conducted an analytical comparison to evaluate the performance of the proposed estimators in terms of their bias, quadratic risk, and numerical performance using both simulated and real data. Additionally, we assessed several penalization methods and three machine learning algorithms to facilitate a comprehensive comparison. Our results demonstrate that the proposed estimators outperformed the standard ridge-type estimator with respect to the mean squared error of the simulated data and the mean squared prediction error of two real data applications.

1. Introduction

A regression analysis provides answers to queries concerning the dependence of a variable, known as the response, on one or a set of independent variables, known as predictors. These include problems like predicting the response value for a given collection of predictors or identifying the most significant group of predictors with a plausible impact on the response variable. A commonly used method to estimate the functional relationship between the response variable and the set of predictors is the ordinary least squares (OLS) method. This method provides the best linear unbiased estimator (BLUE) of the regression vector of coefficients based on the Gauss–Markov theorem. This result holds, assuming that the columns of the design matrix, which are represented by the independent variables, are not correlated. However, the OLS estimator becomes less efficient if a strong or near-to-strong linear relationship exists among the columns of the design matrix, known as multicollinearity.

The primary objectives of this research were threefold. The initial goal of this study was to improve the ridge estimator (RE) of the regression vector, proposed by Kibria and Lukman [1], by integrating pretest and shrinkage approaches in the presence of multicollinearity among the regressor variables. This enhancement primarily focuses on situations where particular regression coefficients are considered insignificant. The second purpose was to examine the analytical findings on the bias and quadratic risks of the estimators for the newly proposed set. The third step involved evaluating the performance of the estimators using a numerical simulation, specifically by measuring the mean squared error. Additionally, the estimators were assessed using real-world data to determine the prediction error.

The contributions of this research are as follows: (i) the formulation of novel ridge-type pretest and shrinkage estimators in the context of multicollinearity, (ii) the analytical derivation of their bias and quadratic risks, and (iii) extensive Monte Carlo simulations to evaluate their efficacy against established estimators, in parallel with an application to a real-data example in conjunction with penalization and machine learning techniques to evaluate the predictive accuracy.

The subsequent sections of this paper are structured in accordance with our objectives. Section 2 contains a discussion of relevant work. In Section 3, we provide the regression model and the KL estimator. The strategy improvements for the RE estimator are outlined in Section 4. In Section 5, we study some analytical properties of the array of estimators—specifically, the bias and quadratic risks. Some penalizing techniques are presented in Section 6, and we present three machine learning algorithms in Section 7. In Section 8, we conduct a comparison of the array of estimators using both Monte Carlo simulations and two real-data examples. Section 9 provides concluding remarks, and a Supplementary Material is appended at the conclusion of the manuscript, which includes the proofs of the analytical findings.

2. Related Work

The multicollinearity issue lessens the accuracy of the OLS-estimated coefficients; the coefficient estimates can also exhibit significant fluctuations depending on the inclusion of different independent variables in the model, in addition to a high degree of sensitivity to even minor changes in the regression model. Several estimation methods have been developed to improve the OLS estimator when multicollinearity exists. For instance, Hoerl and Kennard [2] proposed the ridge regression estimator and analytically and numerically demonstrated the superiority of this new estimator over the OLS estimator. Later, Liu [3] introduced a novel biased estimator and analytically and numerically verified its superiority over the OLS estimator, known as the Liu-type estimator. Recently, Kibria and Lukman [1] developed a new ridge-type estimator for the regression parameter vector. They showed that the new estimator performed better than the ridge and Liu estimators in terms of the MSE criterion.

The OLS technique uses the sample information obtained from the data set to draw inference about the unknown population regression vector of parameters, which refers to the classical inference method. However, in the Bayesian context, the sample information is combined with non-sample information (NSI) to make an inference about the regression parameters. Such NSI may not be available at all times. However, model selection and building procedures, such as the Akaike information criterion (AIC), Bayesian information criterion (BIC), penalization methods, and machine learning algorithms, can still be utilized to yield NSI. Bancroft [4] was one of the first to try estimating regression coefficients by merging NSI with sample information and produced what is known as the pretest estimator. The pretest estimator relies on evaluating the statistical significance of certain regression coefficients. After the determination is made, the pretest selects either the estimate of the full model or the estimator of the revised model, known as the sub-model, which has fewer coefficients. The pretest algorithm chooses either the full or sub-model estimators using binary weights. A modified version of the pretest estimator, known as the shrinkage estimator, was formulated by Stein [5]. This estimator utilizes smooth weights to merge the estimations from both the overall and sub-model estimators. The regression coefficients are modified to converge towards a desired value that is influenced by the NSI. Nevertheless, the enhanced shrinkage estimator sometimes encounters an excessive reduction. Following that, a more improved version of this estimator was proposed by Stein [6] to efficiently address the issue of excessive shrinkage, known as the positive shrinkage estimator.

Many studies have been drawn to the idea of using shrinkage and pretest estimation approaches. For instance, Al-Momani et al. [7] proposed a novel approach utilizing pretest and shrinkage approaches to accurately estimate the regression coefficient vector of the marginal model for multinomial responses. Al-Momani [8] presented Liu-type pretest, shrinkage, and positive shrinkage estimators for the conditional autoregressive regression model’s large-scale effect parameter vector, and demonstrated that these estimators are more efficient than Liu-type estimators. By employing the concept of shrinkage, Arashi et al. [9] proposed an enhanced version of the Liu-type estimator. The proposed method’s superiority was demonstrated using analytical and numerical data. Subsequently, Arashi et al. [10] proposed a rank-based Liu regression estimator that improves the robustness against non-normal errors and outliers while addressing multicollinearity issues in multiple linear regression. Yüzbaşı et al. [11] proposed the use of pretest and shrinkage ridge estimation techniques for the linear regression model. They demonstrated the advantages of employing the suggested estimators alongside specific penalty estimators. Yüzbaşı et al. [12] introduced pretest and shrinkage approaches that use generalized ridge regression estimation to address issues related to multicollinearity and high-dimensional situations. Al-Momani et al. [13] introduced effective shrinkage and penalty estimators for regression coefficients in spatial error models. This showcases the efficiency improvements using asymptotic and numerical studies. In a recent study, Al-Momani and Arashi [14] showed that the ridge-type pretest and shrinkage estimators outperformed the maximum likelihood estimator in the presence of multicollinearity in a spatial error regression model. To obtain further information on shrinkage estimators, please consult with Saleh et al. [15], Saleh et al. [16], and Nkurunziza et al. [17], among other relevant sources.

3. Statistical Model and New Ridge Estimation

To explain the problem, let us consider the linear regression model given below:

$$\begin{array}{ccc}\hfill \mathbf{Y}& =& \mathbf{X}\mathit{\theta }+\mathit{\upsilon },\hfill \end{array}$$

(1)

where $\mathbf{Y}$ is an $(n\times 1)$ vector of random responses; $\mathbf{X}={\left({\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_p\right)}^T$ is an $(n\times p)$ full-rank design matrix with ${\mathbf{X}}_j={(X_{j1},X_{j2},\dots ,X_{jp})}^T$; $\mathit{\theta }=({\theta }_1,{\theta }_2,\dots ,{\theta }_p)$ is a vector of unknown, but fixed, regression parameters; and $\mathit{\upsilon }={({\upsilon }_1,{\upsilon }_2,\dots ,{\upsilon }_n)}^T$ is an $(n\times 1)$ random error vector such that the expected value and the variance of ${\upsilon }_i$ are, respectively, $E\left({\upsilon }_i\right)=0$ and $Var\left({\upsilon }_i\right)={\tau }^2>0$. The ordinary least squares (OLS) estimator of $\mathit{\theta }$ is given by

$$\begin{array}{ccc}\hfill \widehat{\mathit{\theta }}& =& {\mathbf{G}}^{-1}{\mathbf{X}}^T\mathbf{Y},\hfill \end{array}$$

where $\mathbf{G}={\mathbf{X}}^T\mathbf{X}$. Based on the Gauss–Markov theorem, $\widehat{\mathit{\theta }}$ is the best linear unbiased estimator (BLUE) of $\mathit{\theta }$, which is the case when the columns of the design matrix $\mathbf{X}$ are not correlated, with $Var\left(\widehat{\mathit{\theta }}\right)={\tau }^2{\mathbf{G}}^{-1}$. However, $\widehat{\mathit{\theta }}$ becomes less efficient if a strong or near-to-strong linear relationship exists among the columns of $\mathbf{X}$, which is known as multicollinearity. The ridge-type estimator proposed by [1], which is an estimator of the regression coefficient $\mathit{\theta }$, is denoted by ${\widehat{\mathit{\theta }}}^{RE}$ and given by

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^{RE}& =& {(\mathbf{G}+k{\mathbf{I}}_p)}^{-1}(\mathbf{G}-k{\mathbf{I}}_p)\widehat{\mathit{\theta }},\hfill \end{array}$$

(2)

where ${\mathbf{I}}_p$ is a $(p\times p)$ identity matrix and $k\ge 0$ is known as a biasing parameter, to be estimated using the data.

The incorporation of NSI in a model often involves the inclusion of a hypothesized restriction on the model parameters, resulting in the emergence of potential sub-models. Bayesian statistical approaches have been developed in response to the need to incorporate NSI into models fitted to objective sample data. This allows for the consideration of the uncertainty introduced by both sources of information. If the NSI asserts that some of the regression coefficients are irrelevant, then it is possible to integrate this information into the estimation process via testing a linear hypothesis with the following form:

$$\begin{array}{ccc}\hfill H_0:\mathbf{R}\mathit{\theta }& =& {\mathbf{0}}_q,\hfill \end{array}$$

(3)

where $\mathbf{R}$ is a $(q\times p)$ known matrix of rank $(q\le p)$, and  ${\mathbf{0}}_q$ is a $(q\times 1)$ vector of zeros.

It is important to emphasize that the NSI represented by the linear hypothesis in Equation (3) is not obtained from the same data; instead, it represents external or prior knowledge about the original model. In several practical fields, such knowledge may be obtained from previous empirical studies, theoretical models, or, as in some cases, a researcher with sufficient expertise about the important covariates in the model. For instance, in medical frameworks, previous studies may indicate that specific regression coefficients are insignificant, comparable, or functionally related to the response variable. In such cases, the pretest and shrinkage estimation procedures will provide a methodological basis for incorporating NSI into the estimation process.

Traditional variable selection methods (such as forward, backward, or best subset) are usually employed just to illustrate the execution of a possible sub-model used for comparison purposes, they cannot be interpreted as the theoretical foundation of NSI, but rather as a practical demonstration of the fact that a researcher can formulate a sub-model in the absence of robust prior information. Therefore, the pretest and shrinkage estimators aim to combine the NSI with the sample information to obtain better estimators with respect to a specific metric.

The new sub-model often incorporates a reduced number of regression variables, facilitating interpretation and mitigating the complexity associated with a large number of irrelevant variables. A candidate sub-model may be obtained by using some known variable selection methods, such as the AIC or BIC, or by penalization techniques such as ridge estimation; the least absolute shrinkage and selection operator (LASSO); an elastic net which is a regularized method that combines the penalties of both the LASSO and ridge regression; smoothly clipped absolute deviation (SCAD); or an adaptive LASSO, among others. Under the restriction given by the hypothesis in Equation (3), and by using Lagrange multipliers, the sub-model estimator of $\mathit{\theta }$, denoted by ${\widehat{\mathit{\theta }}}^R$, is given by the following:

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^R& =& \widehat{\mathit{\theta }}-{\mathbf{G}}^{-1}{\mathbf{R}}^T{\left(\mathbf{R}{\mathbf{G}}^{-1}{\mathbf{R}}^T\right)}^{-1}\mathbf{R}\widehat{\mathit{\theta }}\hfill \end{array}$$

(4)

Obviously, if the restriction in Equation (3) is correct, then ${\widehat{\mathit{\theta }}}^R=\widehat{\mathit{\theta }}$, which is an unbiased estimator for $\mathit{\theta }$. On the other hand, when the restriction is not true, ${\widehat{\mathit{\theta }}}^R$ becomes less efficient than $\widehat{\mathit{\theta }}$, particularly in cases where there is ongoing multicollinearity among the $\mathbf{X}$ columns. This can be diminished by employing the Kibria and Lukman estimation method to calculate the coefficients of sub-model regression, which are denoted by ${\widehat{\mathit{\theta }}}^{RRE}$ and given as follows:

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^{RRE}& =& {(\mathbf{G}+k{\mathbf{I}}_p)}^{-1}(\mathbf{G}-k{\mathbf{I}}_p){\widehat{\mathit{\theta }}}^R\hfill \end{array}$$

(5)

4. Strategy Improvements for the RE Estimator

Given that a sub-model was acquired via the use of penalization methods or machine learning algorithms, we explored various shrinkage and pretest procedures to combine the estimations from both the full and sub-models. First, we established the definition of a test statistic, designated as ${\varphi }_n$, which was used to test the hypothesis stated in Equation (3) as follows:

$$\begin{array}{ccc}\hfill {\varphi }_n& =& {\displaystyle \frac{{\left(\mathbf{R}\widehat{\mathit{\theta }}\right)}^T{\left(\mathbf{R}{\mathbf{G}}^{-1}{\mathbf{R}}^T\right)}^{-1}\left(\mathbf{R}\widehat{\mathit{\theta }}\right)}{q{s}^2}},\hfill \end{array}$$

where $s^2={\displaystyle \frac{{(\mathbf{Y}-\mathbf{X}\widehat{\mathit{\theta }})}^T(\mathbf{Y}-\mathbf{X}\widehat{\mathit{\theta }})}{n-p}}$ is an estimator of $Var\left({\nu }_i\right)={\tau }^2$. Assuming the null hypothesis in Equation (3), the test statistic ${\varphi }_n$ is asymptotically distributed according to an F-distribution with $(q,r),r=n-p$ degrees of freedom. However, if the alternative hypothesis $\mathbf{R}\mathit{\theta }\ne {\mathbf{0}}_q$ holds, the test statistic follows a non-central $F$-distribution with a non-centrality parameter ${\Delta }^2={\displaystyle \frac{{\left(\mathbf{R}\mathit{\theta }\right)}^T{\left(\mathbf{R}{\mathbf{G}}^{-1}{\mathbf{R}}^T\right)}^{-1}\left(\mathbf{R}\mathit{\theta }\right)}{{\tau }^2}}$. An improved set of shrinkage estimators using the technique in [1] can be formulated as follows:

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^{RE-Shrinkage}& =& {\widehat{\mathit{\theta }}}^{RE}-\left({\widehat{\mathit{\theta }}}^{RE}-{\widehat{\mathit{\theta }}}^{RRE}\right)f\left({\varphi }_n\right),\hfill \end{array}$$

(6)

where $f\left({\varphi }_n\right)$ is a Borel measurable function of ${\varphi }_n$ and certain selections of $f\left({\varphi }_n\right)$ may provide shrinkage estimators that are both plausible and helpful.

4.1. Preliminary and Shrinkage Estimators

The preliminary test estimator can be obtained when the function $f\left({\varphi }_n\right)=I\left({\varphi }_n\le F_{q,r}\left(\alpha \right)\right)$; it is denoted by ${\widehat{\mathit{\theta }}}^{REPT}$ and given by the following:

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^{REPT}& =& {\widehat{\mathit{\theta }}}^{RE}-\left({\widehat{\mathit{\theta }}}^{RE}-{\widehat{\mathit{\theta }}}^{RRE}\right)I\left({\varphi }_n\le F_{q,r}\left(\alpha \right)\right),\hfill \end{array}$$

(7)

where $I(.)$ is the indicator function, $\alpha$ is the level of significance, and  $F_{q,r}\left(\alpha \right)$ represents the upper critical $\alpha$-value of the F-distribution with $(q,r)$ degrees of freedom. Clearly, if the indicator function is zero, then ${\widehat{\mathit{\theta }}}^{REPT}={\widehat{\mathit{\theta }}}^{RE}$; otherwise, ${\widehat{\mathit{\theta }}}^{REPT}={\widehat{\mathit{\theta }}}^{RER}$. One limitation of this estimator is its reliance on a binary choice of the two estimators, which is influenced by the level of significance $\alpha$. A more refined method of assigning weights may be attained by setting $f\left({\varphi }_n\right)=\pi {\varphi }_n^{-1},\pi ={\displaystyle \frac{(q-2)(n-p)}{q(n-p+2)}}$, the James–Stein shrinkage estimator, which can be formally stated as follows:

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^{RES}& =& {\widehat{\mathit{\theta }}}^{RE}-\left({\widehat{\mathit{\theta }}}^{RE}-{\widehat{\mathit{\theta }}}^{RRE}\right)\pi {\varphi }_n^{-1}.\hfill \end{array}$$

(8)

The shrinkage estimator ${\widehat{\mathit{\theta }}}^{RES}$ experiences over-shrinkage, resulting in negative $\mathit{\theta }$ coordinates when $({\varphi }_n<\pi )$. The positive James–Stein estimator solves this issue. It is denoted by ${\widehat{\mathit{\theta }}}^{REPS}$, and can by obtained by setting $f\left({\varphi }_n\right)=\pi {\varphi }_n^{-1}+(1-\pi {\varphi }_n^{-1})I({\varphi }_n\le \pi )$

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^{REPS}& =& {\widehat{\mathit{\theta }}}^{RE}\hfill \\ \hfill & -& \left({\widehat{\mathit{\theta }}}^{RE}-{\widehat{\mathit{\theta }}}^{RRE}\right)\left[\pi {\varphi }_n^{-1}+(1-\pi {\varphi }_n^{-1})I({\varphi }_n\le \pi )\right],\hfill \end{array}$$

(9)

which can be simplifies as follows:

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^{REPS}& =& {\widehat{\mathit{\theta }}}^{RES}-\left({\widehat{\mathit{\theta }}}^{RE}-{\widehat{\mathit{\theta }}}^{RRE}\right)\hfill \\ \hfill & & \left(1-\pi {\varphi }_n^{-1}\left)I\right({\varphi }_n\le \pi \right).\hfill \end{array}$$

(10)

4.2. Modified Preliminary and Shrinkage Estimators

Another way to enhance the regression estimate when the null hypothesis in Equation (3) is true, which could utilize any type of NSI, is to consider the linear shrinkage estimator of the sub- and full models, the RE estimator. Ahmed S. E. [18] provided evidence that the estimator is competitive when the NSI is correct. The estimator is denoted by ${\widehat{\mathit{\theta }}}^{LSRE}$ and given by the following:

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^{LSRE}& =& d{\widehat{\mathit{\theta }}}^{RRE}+(1-d){\widehat{\mathit{\theta }}}^{RE}\hfill \\ & =& {\widehat{\mathit{\theta }}}^{RE}-d\left({\widehat{\mathit{\theta }}}^{RE}-{\widehat{\mathit{\theta }}}^{RRE}\right),\hfill \end{array}$$

(11)

where $0\le d\le 1$ serves as a tuning parameter, which is selected to minimize the estimator’s mean squared error. Obviously, when $d=0$, ${\widehat{\mathit{\theta }}}^{LSRE}={\widehat{\mathit{\theta }}}^{RE}$, and when $d=1$, ${\widehat{\mathit{\theta }}}^{LSRE}={\widehat{\mathit{\theta }}}^{RRE}$. Additionally, we can employ the idea proposed by Ahmed S. E. [19] to produce a new RE estimator that is often referred to as the shrinkage pretest estimator. It is denoted by ${\widehat{\mathit{\theta }}}^{SPTRE}$ and given as follows:

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^{SPTRE}& =& {\widehat{\mathit{\theta }}}^{RE}-d\left({\widehat{\mathit{\theta }}}^{RE}-{\widehat{\mathit{\theta }}}^{RRE}\right)\hfill \\ & & I\left({\varphi }_n\le F_{q,r}\left(\alpha \right)\right)\hfill \end{array}$$

(12)

5. Analytical Properties

This section presents the bias and quadratic risk functions of the proposed RE estimators. For this purpose, assume ${\widehat{\mathit{\theta }}}^{\ast RE}$ is any of these RE-type estimators; consequently, the bias of ${\widehat{\mathit{\theta }}}^{\ast }$ is $B\left({\widehat{\mathit{\theta }}}^{\ast }\right)=E\left({\widehat{\mathit{\theta }}}^{\ast }-\mathit{\theta }\right)$. The bias expressions are given in the following Theorem 1.

Theorem 1.

Let ${\mathbf{A}}_k={(\mathbf{G}+k{\mathbf{I}}_p)}^{-1}(\mathbf{G}-k{\mathbf{I}}_p)$ and $\mathbf{B}={\mathbf{G}}^{-1}{\mathbf{R}}^T{\left(\mathbf{R}{\mathbf{G}}^{-1}{\mathbf{R}}^T\right)}^{-1}{\mathbf{R}}^T$; then,

1. 

$B\left({\widehat{\mathit{\theta }}}^{RE}\right)=[{\mathbf{A}}_k-{\mathbf{I}}_p]\mathit{\theta }$

2. 

$B\left({\widehat{\mathit{\theta }}}^{RRE}\right)=B\left({\widehat{\mathit{\theta }}}^{RE}\right)-{\mathbf{A}}_k\mathbf{B}\mathit{\theta }$

3. 

$B\left({\widehat{\mathit{\theta }}}^{REPT}\right)=B\left({\widehat{\mathit{\theta }}}^{RE}\right)-{\mathbf{A}}_k\mathbf{B}\mathit{\theta }{H}_{q+2,r}(c_{\alpha },{\Delta }^2)$

4. 

$B\left({\widehat{\mathit{\theta }}}^{RES}\right)=B\left({\widehat{\mathit{\theta }}}^{RE}\right)-\pi {\mathbf{A}}_k\mathbf{B}\mathit{\theta }E\left(F_{q+2,r}^{-1}\left({\Delta }^2\right)\right)$

5. 

$$\begin{gathered} B\left({\widehat{\mathit{\theta }}}^{REPS}\right)=B\left({\widehat{\mathit{\theta }}}^{RES}\right)-{\mathbf{A}}_k\mathbf{B}\mathit{\theta }\left\{\pi E\left(F_{q+r,2}^{-1}\left({\Delta }^2\right)\right) \\ +E\left((1-\pi F_{q+2,r}^{-1}(c_{\alpha };{\Delta }^2))I(F_{q+2,r}^{-1}\left({\Delta }^2\right)\le \pi )\right)\right\} \end{gathered}$$

6. 

$B\left({\widehat{\mathit{\theta }}}^{LSRE}\right)=B\left({\widehat{\mathit{\theta }}}^{RE}\right)-d{\mathbf{A}}_k\mathbf{B}\mathit{\theta }$

7. 

$B\left({\widehat{\mathit{\theta }}}^{SPTRE}\right)=B\left({\widehat{\mathit{\theta }}}^{RE}\right)-d{\mathbf{A}}_k\mathbf{B}\mathit{\theta }{H}_{q+2,r}(c_{\alpha },{\Delta }^2)$

where $H_{m,r}(.)$ is the non-central cumulative distribution function of F random variable with $(m,r)$ degrees of freedom, $c_{\alpha }$ is the $\alpha$-critical value for the F-distribution, and $={\displaystyle \frac{{\Delta }^2}{2}}$ is a non-centrality parameter. For the proof of the above theorem, see Appendix A.

In order to obtain the quadratic risk expressions, we used the quadratic loss function of any RE-type estimator ${\widehat{\mathit{\theta }}}^{\ast }$, which is defined for any positive definite $p\times p$ matrix $\mathbf{M}$ as follows:

$$\begin{array}{ccc}\hfill L({\widehat{\mathit{\theta }}}^{\ast },\mathit{\theta })& =& {({\widehat{\mathit{\theta }}}^{\ast }-\mathit{\theta })}^T\mathbf{M}({\widehat{\mathit{\theta }}}^{\ast }-\mathit{\theta })\hfill \end{array}$$

The quadratic risk function of any estimator is denoted by $\Gamma \left({\widehat{\mathit{\theta }}}^{\ast }\right)$ and  defined as $\Gamma \left({\widehat{\mathit{\theta }}}^{\ast }\right)=E\left(L({\widehat{\mathit{\theta }}}^{\ast },\mathit{\theta })\right)$. The quadratic risk expressions are given in the following Theorem 2.

Theorem 2.

Let $\mathbf{M}$ be a $(p\times p)$ positive definite matrix; then,

$$\begin{array}{c}\hfill \Gamma \left({\widehat{\mathit{\theta }}}^{RE}\right)={\tau }^{2}tr\left({\mathbf{A}}_k{\mathbf{G}}^{-1}{\mathbf{A}}_k^T\mathbf{M}\right)+{\mathit{\theta }}^T({\mathbf{A}}_k-{\mathbf{I}}_p)\mathbf{M}({\mathbf{A}}_k^T-{\mathbf{I}}_p)\mathit{\theta }\end{array}$$

$$\begin{array}{ccc}\hfill \Gamma \left({\widehat{\mathit{\theta }}}^{RRE}\right)& =& \Gamma \left({\widehat{\mathit{\theta }}}^{RE}\right)\hfill \\ & -& 2{\tau }^{2}tr\left({\mathbf{A}}_k{\mathbf{G}}^{-1}{\mathbf{B}}^T{\mathbf{A}}_k^T\mathbf{M}\right)\hfill \\ & -& 2{\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k({\mathbf{A}}_k-{\mathbf{I}}_p)\mathbf{M}({\mathbf{A}}_k^T-{\mathbf{I}}_p){\mathbf{B}}^T{\mathbf{A}}_k^T\right)\mathit{\theta }\hfill \\ & +& 2{\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k\mathbf{B}\mathbf{M}\right)\mathit{\theta }\hfill \\ & +& {\tau }^{2}tr\left({\mathbf{A}}_k\mathbf{B}{\mathbf{G}}^{-1}\mathbf{M}{\mathbf{B}}^T{\mathbf{A}}_k^T\right)\hfill \\ & +& {\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k\mathbf{B}({\mathbf{A}}_k-{\mathbf{I}}_p)\mathbf{M}({\mathbf{A}}_k^T-{\mathbf{I}}_p){\mathbf{B}}^T{\mathbf{A}}_k^T\right)\mathit{\theta }\hfill \end{array}$$

$$\begin{array}{ccc}\hfill \Gamma \left({\widehat{\mathit{\theta }}}^{REPT}\right)& =& \Gamma \left({\widehat{\mathit{\theta }}}^{RE}\right)\hfill \\ & -& 2tr\left({\mathbf{A}}_k\mathbf{B}{\mathbf{A}}_k^T\mathbf{M}\right)H_{q+2,r}(c_{\alpha },{\Delta }^2)\hfill \\ & -& 2\mathit{\theta }tr\left({\mathbf{A}}_k\mathbf{B}{\mathbf{A}}_k^T\mathbf{M}\right)\mathit{\theta }{H}_{q+4,r}(c_{\alpha },{\Delta }^2)\hfill \\ & -& 2\mathit{\theta }tr\left({\mathbf{A}}_k\mathbf{B}\mathbf{M}\right)\mathit{\theta }{H}_{q+2,r}(c_{\alpha },{\Delta }^2)\hfill \\ & +& tr\left({\mathbf{A}}_k\mathbf{B}\mathbf{M}{\mathbf{B}}^T{\mathbf{A}}_k^T\right)H_{q+2,r}(c_{\alpha },{\Delta }^2)\hfill \\ & +& {\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k\mathbf{B}\mathbf{M}{\mathbf{B}}^T{\mathbf{A}}_k^T\right)\mathit{\theta }{H}_{q+4,r}(c_{\alpha },{\Delta }^2)\hfill \end{array}$$

$$\begin{array}{ccc}\hfill \Gamma \left({\widehat{\mathit{\theta }}}^{RES}\right)& =& \Gamma \left({\widehat{\mathit{\theta }}}^{RE}\right)\hfill \\ & -& 2\pi tr\left({\mathbf{A}}_k\mathbf{B}{\mathbf{A}}_k^T\mathbf{M}\right)E\left(F_{q+2,r}^{-1}\left({\Delta }^2\right)\right)\hfill \\ & -& 2\pi {\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k\mathbf{B}{\mathbf{A}}_k^T\mathbf{M}\right)\mathit{\theta }E\left(F_{q+4,r}^{-1}\left({\Delta }^2\right)\right)\hfill \\ & -& 2\pi {\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k\mathbf{B}\mathbf{M}\right)\mathit{\theta }E\left(F_{q+2,r}^{-1}\left({\Delta }^2\right)\right)\hfill \\ & +& {\pi }^{2}tr\left({\mathbf{A}}_k\mathbf{B}\mathbf{M}{\mathbf{B}}^T{\mathbf{A}}_k\right)E\left(F_{q+2,r}^{-2}\left({\Delta }^2\right)\right)\hfill \\ & +& {\pi }^2{\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k\mathbf{B}\mathbf{M}{\mathbf{B}}^T{\mathbf{A}}_k\right)\mathit{\theta }E\left(F_{q+4,r}^{-2}\left({\Delta }^2\right)\right)\hfill \end{array}$$

$$\begin{array}{ccc}\hfill \Gamma \left({\widehat{\mathit{\theta }}}^{REPS}\right)& =& \Gamma \left({\widehat{\mathit{\theta }}}^{RE}\right)\hfill \\ & -& 2tr\left({\mathbf{A}}_k\mathbf{B}{\mathbf{A}}_k^T\right)\hfill \\ & & E[\pi F_{q+2,r}^{-1}\left({\Delta }^2\right)+(1-\pi F_{q+2,r}^{-1}\left({\Delta }^2\right))\hfill \\ & & I\left(F_{q+2,r}^{-1}\left({\Delta }^2\right)\le \pi \right)]\hfill \\ & -& 2{\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k\mathbf{B}{\mathbf{A}}_k^T\mathbf{M}\right)\mathit{\theta }\hfill \\ & & E[\pi F_{q+4,r}^{-1}\left({\Delta }^2\right)+(1-\pi F_{q+4,r}^{-1}\left({\Delta }^2\right))\hfill \\ & & I\left(F_{q+4,r}^{-1}\left({\Delta }^2\right)\le \pi \right)]\hfill \\ & -& 2{\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k\mathbf{B}{\mathbf{A}}_k^T\mathbf{M}\right)\mathit{\theta }\hfill \\ & & E[\pi F_{q+2,r}^{-1}\left({\Delta }^2\right)+(1-\pi F_{q+2,r}^{-1}\left({\Delta }^2\right))\hfill \\ & & I\left(F_{q+2,r}^{-1}\left({\Delta }^2\right)\le \pi \right)]\hfill \\ & +& tr\left({\mathbf{A}}_k\mathbf{B}\mathbf{M}{\mathbf{B}}^T{\mathbf{A}}_k^T\right)\hfill \\ & & E[\pi F_{q+2}^{-1}\left({\Delta }^2\right)+(1-\pi F_{q+2,r}^{-1}\left({\Delta }^2\right))\hfill \\ & & I\left(F_{q+2,r}^{-1}\left({\Delta }^2\right)\le \pi \right)]\hfill \\ & +& {\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k\mathbf{B}\mathbf{M}{\mathbf{B}}^T{\mathbf{A}}_k^T\right)\mathit{\theta }\hfill \\ & & E[\pi F_{q+4,r}^{-1}\left({\Delta }^2\right)+(1-\pi F_{q+4,r}^{-1}\left({\Delta }^2\right))\hfill \\ & & I\left(F_{q+4,r}^{-1}\left({\Delta }^2\right)\le \pi \right)]\hfill \end{array}$$

$$\begin{array}{ccc}\hfill \Gamma \left({\widehat{\mathit{\theta }}}^{LSRE}\right)& =& \Gamma \left({\widehat{\mathit{\theta }}}^{RE}\right)\hfill \\ & -& 2d{\tau }^{2}tr\left({\mathbf{A}}_k\mathbf{B}\mathbf{M}{\mathbf{G}}^{-1}{\mathbf{A}}^T\right)\hfill \\ & -& 2d{\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k({\mathbf{A}}_k-{\mathbf{I}}_p)\mathbf{M}({\mathbf{A}}_k^T-{\mathbf{I}}_p){\mathbf{B}}^T{\mathbf{A}}_k^T\right)\mathit{\theta }\hfill \\ & +& d^2{\tau }^{2}tr\left({\mathbf{A}}_k\mathbf{B}\mathbf{M}{\mathbf{G}}^{-1}{\mathbf{B}}^T{\mathbf{A}}_k^T\right)+2d{\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k\mathbf{B}\mathbf{M}\right)\mathit{\theta }\hfill \\ & +& d^2{\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k\mathbf{B}({\mathbf{A}}_k-{\mathbf{I}}_p)\mathbf{M}({\mathbf{A}}_k^T-{\mathbf{I}}_p){\mathbf{B}}^T{\mathbf{A}}_K^T\right)\mathit{\theta }\hfill \end{array}$$

$$\begin{array}{ccc}\hfill \Gamma \left({\widehat{\mathit{\theta }}}^{SPTRE}\right)& =& \Gamma \left({\widehat{\mathit{\theta }}}^{RE}\right)\hfill \\ & -& 2dtr\left({\mathbf{A}}_k{\mathbf{B}}^T{\mathbf{A}}_k^T\mathbf{M}\right)H_{q+2,r}(c_{\alpha },{\Delta }^2)\hfill \\ & -& 2d{\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k{\mathbf{B}}^T{\mathbf{A}}_k^T\mathbf{M}\right)\mathit{\theta }{H}_{q+4,r}(c_{\alpha },{\Delta }^2)\hfill \\ & -& 2d{\mathit{\theta }}^{T}tr\left(\mathbf{B}{\mathbf{A}}_k^T\mathbf{M}\right)\mathit{\theta }{H}_{q+2,r}(c_{\alpha },{\Delta }^2)\hfill \\ & +& d^{2}tr\left({\mathbf{A}}_k\mathbf{B}\mathbf{M}{\mathbf{B}}^T{\mathbf{A}}_k^T\right)H_{q+2,r}(c_{\alpha },{\Delta }^2)\hfill \\ & +& d^2{\mathit{\theta }}^{T}tr\left({\mathbf{A}}_k\mathbf{B}\mathbf{M}{\mathbf{B}}^T{\mathbf{A}}_k^T\right)\mathit{\theta }{H}_{q+4,r}(c_{\alpha },{\Delta }^2),\hfill \end{array}$$

For the proof of the above theorem, see Appendix A. In the following section, we shall enumerate some extant penalization methods to produce a new sub-model from the literature.

6. Commonly Used Penalizing Techniques

This section provides a brief overview of some common penalization techniques—ridge, LASSO, elastic net, SCAD, and adaptive LASSO—that perform a concurrent parameter estimation and variable selection. These techniques are incorporated for comparative analyses and to develop sub-models to support the proposed ridge-type pretest and shrinkage estimators. Utilizing these common techniques underscores the practical significance and relative benefits of the techniques presented in this study. Penalty estimators are produced as a consequence of simultaneous model selection and parameter estimation processes by applying a penalty to the least squares equation. As a result, model selection and estimating processes are included in penalty techniques. Some of the penalty techniques are provided below.

6.1. Ridge Estimator

The ridge estimator proposed by Hoerl and Kennard [2] efficiently addresses the issue of multicollinearity by using a technique known as coefficient shrinkage, which reduces the magnitudes of the coefficients associated with strongly correlated variables. This approach aids in achieving model stability and mitigating the influence of multicollinearity on the estimation of coefficients. It is denoted by ${\widehat{\mathit{\theta }}}^{Ridge}$, and can obtained by minimizing the penalized residual sum of squares. It is given by the following:

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^{Ridge}& =& \underset{\mathit{\theta }}{argmin}\left[{(\mathbf{Y}-\mathbf{X}\mathit{\theta })}^T(\mathbf{Y}-\mathbf{X}\mathit{\theta })+{\lambda }_R({\mathit{\theta }}^T\mathit{\theta }-c)\right],\hfill \end{array}$$

(13)

where ${\lambda }_R\ge 0$ is a constant known as a biasing parameter.

6.2. LASSO Estimator

The LASSO, or the least absolute selection and shrinkage operator, was proposed by Tibshirani [20]. It is a method for selecting variables and estimating parameters in linear models. In the usual least squares estimation of regression coefficients, the LASSO algorithm employs the $L1$ norm of the vector to define a penalty term. The LASSO estimator is denoted by ${\widehat{\mathit{\theta }}}^{lasso}$ and given by the following:

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^{lasso}& =& \underset{\mathit{\theta }}{argmin}\left[{(\mathbf{Y}-\mathbf{X}\mathit{\theta })}^T(\mathbf{Y}-\mathbf{X}\mathit{\theta })+{\lambda }_L\sum _{i=1}^p\left|{\theta }_i\right|\right],\hfill \end{array}$$

(14)

where ${\lambda }_L\ge 0$ is the tuning parameter to be estimated from the data. However, it is known that the LASSO technique may not be the most ideal approach when dealing with a set of columns in a design matrix that exhibit significant levels of correlation. As a solution, Zou et al. [21] introduced the elastic net (ELNT) method, which combines $L1$ and $L2$ penalty terms in a linear manner.

6.3. Elastic Net Estimator

The elastic net estimator is denoted by ${\widehat{\mathit{\theta }}}^{ELNT}$ and can be obtained as follows:

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^{ELNT}& =& \underset{\mathit{\theta }}{argmin}\left[{(\mathbf{Y}-\mathbf{X}\mathit{\theta })}^T(\mathbf{Y}-\mathbf{X}\mathit{\theta })+{\lambda }_L\sum _{i=1}^p\left|{\theta }_i\right|+{\lambda }_R\sum _{i=1}^p{\theta }_i^2\right],\hfill \end{array}$$

(15)

where ${\lambda }_L$ and ${\lambda }_R$ are, respectively, the LASSO and the ridge-tuning parameters. A variable selection process is said to possess the oracle property when it successfully identifies the correct subset of zero coefficients inside the regression model being examined. Additionally, the estimators of the remaining non-zero coefficients demonstrate consistency and asymptotic normality. The LASSO estimator does not enjoy this property. However, Fan and Li [22] and Zou [23] developed two approaches that possess the oracle property. In the following two subsections, we establish the definitions for these methods.

6.4. SCAD Estimator

The SCAD, or smoothly clipped absolute deviation estimator, is denoted by ${\widehat{\mathit{\theta }}}^{SCAD}$ and obtained as follows:

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^{SCAD}& =& {\displaystyle \underset{\mathit{\theta }}{argmin}\left\{{(\mathbf{Y}-\mathbf{X}\mathit{\theta })}^T(\mathbf{Y}-\mathbf{X}\mathit{\theta })+{\lambda }_S\sum _{i=1}^p{P}_{a,{\lambda }_S}\left({\theta }_i\right)\right\},}\hfill \end{array}$$

(16)

where $P_{a,{\lambda }_S}\left(t\right)$ is a continuous function of t. Its derivative is given by the following:

$$\begin{array}{ccc}\hfill P_{a,{\lambda }_S}^{\prime }\left(t\right)& =& {\lambda }_S\left[I\left(\right|t|\le {\lambda }_S)+{\displaystyle \frac{(a{\lambda }_S-{\left|t\right|)}_{+}}{(a-1){\lambda }_S}}I\left(\right|t|>{\lambda }_S)\right],\hfill \\ & & t\ge 0,\hfill \end{array}$$

where ${\lambda }_S>0$ and $a>2$ are known as tuning parameters. Note that, when $a=\infty$, the function $P_{a,{\lambda }_S}\left(t\right)$ is equivalent to the $L1$ penalty.

6.5. Adaptive LASSO Estimator

Using adaptive weights on the $L1$ penalties of regression coefficients, the adaptive LASSO modifies the LASSO penalty. Theoretically, it has been demonstrated that the adaptive LASSO estimator enjoys the oracle property. It is denoted by ${\widehat{\mathit{\theta }}}^{Alasso}$ and obtained as follows:

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\theta }}}^{Alasso}& =& \underset{\mathit{\theta }}{argmin}\left[{(\mathbf{Y}-\mathbf{X}\mathit{\theta })}^T(\mathbf{Y}-\mathbf{X}\mathit{\theta })+{\lambda }_A\sum _{i=1}^p{\widehat{h}}_i\left|{\theta }_i\right|\right],\hfill \end{array}$$

(17)

where ${\widehat{h}}_i$ is a weight function defined as ${\widehat{h}}_i={\displaystyle \frac{1}{|{\theta }_i^{★}{|}^{\gamma }}}\gamma >0$. The estimator $\widehat{{\theta }_i^{★}}$ is considered to be root-n consistent for ${\theta }_i$. The minimization process for the adaptive LASSO solution does not provide any computing challenges and can be easily solved. One possible choice for $\widehat{{\theta }_i^{★}}$ is the OLS estimate of ${\theta }_i$. Next, we will describe some machine learning algorithms that are used in this manuscript.

7. Machine Learning Algorithms

This section simply presents three machine learning algorithms—random forest, K-nearest neighbors, and neural networks—employed to enhance the penalization strategies and to further evaluate the predictive effectiveness of the suggested estimators. Integrating these algorithms allows for a more comprehensive comparison between traditional statistical estimators and contemporary data-driven techniques.

The area of regression analysis has been significantly transformed by the advent of machine learning, providing robust methodologies and strategies to extract meaningful insights and achieve precise predictions from intricate data sets. Yet, the emergence of machine learning has expanded the scope of regression analyses, allowing us to use the predictive capacities of algorithms in order to unveil latent patterns and make informed judgments based on data. The three machine learning algorithms are briefly described in the following subsections.

7.1. Random Forest

Ho [24] suggested a technique for creating tree-based classifiers that may be expanded as needed to improve the accuracy on both training and hidden data, which is known as random forest. Random forest is one of the most effective and flexible algorithms that can be used for both classification and regression. It is a kind of ensemble learning that takes the results of many different decision trees and uses them together to come up with more reliable results. It builds several decision trees on distinct subsets of the training data and allows them to generate their own predictions. Randomization is introduced via bootstrapping using random data points and training set replacements and selecting a random subset of features for each tree. Later on, Ho [25] discussed the problem of overfitting and obtaining the most accurate results by building a decision-tree-based predictor that stays as accurate as possible on the training data and becomes more accurate as it becomes more complicated. In a random forest, the final prediction is derived by aggregating the predictions of all the individual trees naturally, through regression averaging. This ensemble method reduces overfitting and increases the algorithm’s precision and stability. More details about the extensions and developments of the algorithm can be found in the studies by Breiman [26] and Liaw et al. [27], among others.

7.2. K-Nearest Neighbors

K-nearest neighbors (KNN) is an effective machine learning algorithm used for classification and regression, as with the Random Forest algorithm. It was initially proposed by Fix and Hodges [28], and later on, Cover and Hart [29] made modifications to it. The KNN method employs supervised learning in a non-parametric fashion. In the context of economic forecasting, Imandoust and Bolandraftar [30] examined the application of the KNN method and  showed that it exhibits a greater efficacy compared to alternative methodologies. Lubis et al. [31] examined the implementation of the Euclidean distance formula in KNN in comparison to the normalized Euclidean distance, Manhattan, and normalized Manhattan. Cunningham et al. [32] provided a comprehensive examination of several methodologies employed in nearest neighbor classification.

KNN uses instance-based, non-parametric learning. The method does not rely heavily on assumptions about the underlying data distribution. However, it produces predictions via evaluating similarities between the data points. In a regression context, the method computes the average or weighted average of the response value of the K-nearest neighbors to predict the response value for the new data point. The KNN algorithm is often used for initial machine learning applications due to its simplicity in comprehension and implementation. Yet, the performance of the system may be influenced by the selection of K and the specific distance measure used. Moreover, the computational cost associated with this approach is high, since it necessitates the calculation of distances for every data point in the data set during the prediction stage.

7.3. Neural Network

A neural network is a computer model that draws inspiration from the anatomical and functional characteristics of the human brain. One of the firsts attempts of this method was proposed by McCulloch and Pitts [33], who established the foundation for comprehending the ability of basic computational units, resembling neurons, to execute intricate logical operations. Subsequently, other improvements and enhancements have been incorporated into this approach. Masri et al. [34] introduced a detection approach based on a neural network method for identifying changes in the properties of systems with unknown structures. Angelini et al. [35] presented a case study showcasing the effective operation of neural networks in the evaluation of credit risk. Du [36] provided a thorough examination of clustering neural network approaches based on competitive learning. Adeli [37] provided an overview of academic articles on neural networks that have been published in archival research journals from 1989 to 2000.

The purpose of a neural network is to effectively process information and acquire knowledge via the analysis of data. Neural networks are composed of linked nodes, which are arranged in layers. These networks have an exceptional proficiency in addressing intricate issues, notably in domains like pattern recognition, picture and audio recognition, natural language processing, and other related fields.

After obtaining a sub-model estimator in addition to the full-model estimator, which contains all the available variables, it becomes possible to assess the accuracy of the sub-model using different criteria. The objective of this study was to identify estimators that are functions of both ${\widehat{\mathit{\theta }}}^{RE}$ and ${\widehat{\mathit{\theta }}}^{RER}$, with the purpose of mitigating the potential risks associated with either of these estimators over a wide range of parameter values. This objective can be accomplished by developing the pretest and shrinkage estimators in the subsequent section.

8. Numerical Illustrations

In this section, we investigate the finite-sample properties of the proposed estimators using Monte Carlo simulation experiments and two real-data examples.

8.1. Simulation Experiments

The properties of the estimator under investigation and the standards being applied to evaluate the findings will determine how the simulation experiment should be prepared. Following [1], the following equation was used to produce the regressor variables:

$$\begin{array}{ccc}\hfill {\mathbf{X}}_{ij}& =& \sqrt{1-{\rho }^2}{\mathbf{T}}_{ij}+\rho {\mathbf{T}}_{i,p+1},\hfill \\ & & i=1,2,\dots ,n,j=1,2,\dots ,p,\hfill \end{array}$$

(18)

where $T_{ij}$ is an independent and identically standard normal random variable and $\rho$ is the correlation between ${\mathbf{X}}_{ij}$ and ${\mathbf{X}}_{i^{\prime }j}$ for $i\ne i^{\prime }$. The response variable ${\mathbf{Y}}_i$ was then obtained via the following equation:

$$\begin{array}{ccc}\hfill {\mathbf{Y}}_i& =& {\theta }_0+{\theta }_1{\mathbf{X}}_{i1}+{\theta }_2{\mathbf{X}}_{i2}+\cdots ++{\theta }_p{\mathbf{X}}_{ip}+{ϵ}_i,\hfill \\ & & i=1,2,\dots ,n,\hfill \end{array}$$

(19)

where ${ϵ}_i$ is $i.i.d$ $N(0,{\tau }^2)$, the regression vector is partitioned as $\mathit{\theta }=({\mathbf{1}}_{\mathbf{p}-\mathbf{q}}^{\mathbf{T}},{\mathbf{0}}_{\mathbf{q}-\mathbf{1}}^{\mathbf{T}},{\Delta }^{\ast })$, and ${\Delta }^{\ast }$ varies over the set $\{0,0.1,0.2,\dots ,2\}$, which represents the degree of deviation from the null hypothesis in Equation (3). In this simulation, we chose the value of k as follows: Following [1,38], we rewrote the model in Equation (1) in its canonical form as $\mathbf{Y}=\mathbf{Z}\mathit{\beta }+\mathit{ϵ}$, where $\mathbf{Z}=\mathbf{X}\mathbf{T}$, $\mathit{\beta }={\mathbf{T}}^T\mathit{\theta }$, and ${\mathbf{Z}}^T\mathbf{Z}=\mathbf{D}=diag({\lambda }_1,{\lambda }_2,\dots ,{\lambda }_p)$ are the eigenvalues of $\mathbf{G}$ and $\mathbf{T}$ is an orthogonal $p\times p$ matrix whose columns are the corresponding eigenvectors. Thus, we have $\widehat{\mathit{\beta }}={\left({\mathbf{Z}}^T\mathbf{Z}\right)}^{-1}{\mathbf{Z}}^T\mathbf{Y}={\mathbf{D}}^{-1}{\mathbf{Z}}^T\mathbf{Y}$; then, we estimated the value k as ${\widehat{k}}_{min}={min}_i\left\{{\displaystyle \frac{s^2}{2{\widehat{\beta }}_i^2+(s^2/{\lambda }_i)}}\right\}$, and $s^2$ is the estimated value of ${\tau }^2$ obtained by fitting the regression model to the generated data. The biasing parameter was determined by utilizing the KL heuristic outlined in Appendix A.2—Algorithm A1. We set $d=0.10$, as suggested by [18], and ${\tau }^2=1$. The correlation coefficient $\rho$ was chosen to vary over the set $\{0.5,0.75,0.95\}$, $p\in \{10,15,20,25,30\}$, and $q\in \{5,10,15,20,25\}$. We set $\alpha =0.05$ for testing the hypothesis in Equation (3). It was seen that the performance of all the estimators had a similar pattern when the values of $\rho ,p$, and q were varied. To save space, we chose $\rho \in \{0.75,0.95\}$, $p\in \{10,15\}$, and $q\in \{5,10\}$; then, we ran the simulation over $B=1000$ iterations for $n\in \{50,100\}$. For each estimator, we computed the mean squared error (MSE) as follows:

$$\begin{array}{ccc}\hfill MS{E}_j\left({\widehat{\mathit{\theta }}}^{\ast }\right)& =& {\displaystyle \sum _{i=1}^p{({\theta }_i^{\ast }-{\theta }_i)}^2,j=1,2,\dots ,B,}\hfill \end{array}$$

(20)

where ${\widehat{\mathit{\theta }}}^{\ast }$ is any of the proposed estimators in this study.

For the purpose of comparison, we used the relative efficiency of the mean squared error (RE) with respect to ${\widehat{\mathit{\theta }}}^{RE}$, which is defined as follows:

$$\begin{array}{ccc}\hfill RE\left({\widehat{\mathit{\theta }}}^{\ast },{\widehat{\mathit{\theta }}}^{RE}\right)& =& {\displaystyle \frac{MSE\left({\widehat{\mathit{\theta }}}^{RE}\right)}{MSE\left({\widehat{\mathit{\theta }}}^{\ast }\right)}},\hfill \end{array}$$

(21)

The complete simulation framework used to calculate the MSEs and relative efficiencies of all estimators is provided in Appendix A.2—Algorithm A2. A number greater than one of the REs indicated the superiority of ${\widehat{\mathit{\theta }}}^{\ast }$ over ${\widehat{\mathit{\theta }}}^{RE}$, and vice versa. Figure 1 shows the graphs for the cases we considered. The following conclusion can be obtained:

• The sub-model estimate ${\widehat{\mathit{\theta }}}^{RRE}$ consistently beat all other estimators when the null hypothesis in Equation (3) was true or approximately true. However, its relative efficiency decreased and eventually approached zero as $\Delta$ increased. Moreover, all the estimators outperformed the regular ${\widehat{\mathit{\theta }}}^{RE}$ estimator in terms of the mean squared error across all values of $\Delta$.
• For all values of $\Delta$, the RE positive shrinkage estimator dominated over all other estimators except when the sub-model was true, in which case the RE sub-model and the pretest estimators outperformed it.
• The relative efficiencies exhibited a consistent pattern when the values of $p,q$, and $\rho$ were held constant for both sample sizes used in this simulation.

8.2. Example 1: Biomass Production in the Cape Fear Estuary

In this section, we consider a case study discussed by Rawlings et. al [39] in two different chapters of their book, also available for free within the "VisCollin"-R-package by Friendly [40]. The data were originally studied by Rick [41]. His goal was to detect the key soil factors that affect the aerial biomass in the marsh grass Spartina alterniflora in the Cape Fear Estuary of North Carolina at three different locations. At each location, three types of Spartina vegetation areas were sampled, namely devegetated "dead" areas, "short" Spartina areas, and "tall" Spartina areas. Five samples of the soil substrate from different sites were collected within each location and vegetation type. These samples were then evaluated for 14 different physico-chemical parameters of the soil monthly for several months, resulting in a total of 45 samples. Thus, there were 45 observations in this data set covering the following 17 variables: the location loc; area type type; hydrogen sulfide $H_{2}S$; salinity in percentage SAL; ester hydrolase Eh7; soil acidity in water pH; buffer acidity at pH 6.6 BUF; concentration of the following elements: phosphorus P, potassium K, calcium Ca, magnesium Mg, sodium Na, manganese Mn, zinc  Zn, copper Cu, and ammonium NH4; and aerial biomass BIO as the response variable.

One main objective was to employ the set of variables that can accurately predict the response variable. As a first investigation of multicollinearity, we constructed a correlation plot among these variables, which is given below in Figure 2.

The plot shows that there are many significant relationships between these variables, and some of these relations are very strong and highly significant with $p-value<0.001$. This indicates that multicollinearity exists among these variables, which can be easily detected by the variance inflation factor plot given below in Figure 3.

The VIF plot shows a serious multicollinearity problem, as some of the value are more than 5 in many cases. Hence, the RE estimator will reduce this problem and produce a better estimate of the parameters. Moreover, the proposed estimators will be an additional improvement to estimate and predict the target response variable BIO.

In the absence of prior knowledge, the limitation on the parameters is established either through the judgment of an expert or by utilizing existing methodologies for variable selection, such as the Akaike information criterion (AIC), forward selection (FW), backward elimination (BE), best subset selection (BS), or the Bayes information criterion (BIC), or by using some penalization algorithms, such as the LASSO, adaptive LASSO, and others, to produce a sub-model. In this example, we first employed the forward, backward, and best subset selection methods to produce a sub-model, and then obtained the RE, pretest, and shrinkage estimators. Secondly, we applied random forest, K-nearest neighbors, and a neural network as machine learning algorithms to compare the prediction error with the seven proposed estimators. The sub-models selected by the forward, backward, and best subset selection methods are summarized in

Table 1. Full and sub-models for Biomass data.

Full Model	FW	BW/BS	LASSO	ALASSO	SCAD	ELNT
SAL		✓				✓
pH	✓		✓	✓	✓	✓
K		✓				✓
Na
Zn		✓		✓
H2S			✓			✓
Eh7		✓			✓	✓
BUF				✓		✓
P					✓	✓
Ca	✓					✓
Mg	✓	✓	✓	✓	✓	✓
Mn						✓
Cu	✓	✓	✓	✓	✓	✓
NH4		✓	✓		✓	✓

below.

In our analysis, we examined two sub-models: the forward sub-model, which included the variables pH, Ca, Mg, and Cu along with the intercept, and the second one, which was the backward/best subset that included the variables SAL, K, Zn, Eh7, Mg, Cu, and NH4 along with the intercept. The two sub-models are designated as Sub.1 and Sub.2, respectively.

We fit a full model with all the available variables and the selected sub-model. The whole model yielded an estimated value of ${\widehat{\tau }}^2=s^2=333.9673$. Kibria and Lukman [1] presented several methods for estimating the biasing parameter $\left(k\right)$, but we chose the one with the lowest MSE, which was also provided by [38]. The estimated value of $\left(k\right)$ was determined by writing the model in Equation (1) in its canonical form as follows:

$$\begin{array}{ccc}\hfill \mathbf{Y}& =& \mathbf{Z}\mathit{\beta }+\mathit{ϵ},\hfill \end{array}$$

where $\mathbf{Z}=\mathbf{X}\mathbf{T}$, $\mathit{\beta }={\mathbf{T}}^T\mathit{\theta }$, and ${\mathbf{Z}}^T\mathbf{Z}={\mathbf{T}}^T{\mathbf{X}}^T\mathbf{X}\mathbf{T}=\mathbf{D}=diag\left({\lambda }_1,{\lambda }_2,\dots ,{\lambda }_p\right)$ are the eigenvalues of $\left({\mathbf{X}}^T\mathbf{X}\right)$, and $\mathbf{T}$ is an orthogonal $(p\times p)$ matrix whose columns are the eigenvectors corresponding to the eigenvalues in $\mathbf{D}$. In this case, $\widehat{\mathit{\beta }}={\left({\mathbf{Z}}^T\mathbf{Z}\right)}^{-1}{\mathbf{Z}}^T\mathbf{Y}={\mathbf{D}}^{-1}{\mathbf{Z}}^T\mathbf{Y}$, and the estimated value of k is given by the following:

$$\begin{array}{ccc}\hfill {\widehat{k}}_{min.}& =& \underset{i}{min}\left\{{\displaystyle \frac{s^2}{2{\widehat{\beta }}_i^2+(s^2/{\lambda }_i)}}\right\}\hfill \end{array}$$

Using the previous approach, we found ${\widehat{k}}_{min}=0.0000102$. Then, the RE-type estimators were calculated. To ensure full reproducibility and methodological transparency, the hyperparameter tuning and training settings for all machine learning models, including the neural network (NN), random forest (RF), and K-nearest neighbors (KNN), were trained and tuned using the train function provided by the caret package in R.

For the NN model, the nnet implementation was used. A grid search was applied over two main hyperparameters, the number of hidden neurons (size) and the weight decay parameter (decay). The grid systematically explored hidden-layer sizes ranging from 1 to 10 and decay values between 0.1 and 0.5 in increments of 0.1. This procedure aimed to balance the model complexity and generalization ability, while the weight-decay term controlled overfitting through regularization. The RF model was trained using the random forest method. The number of predictors randomly selected at each node split mtry was optimized through internal cross-validation, and the total number of trees (ntree) was fixed at 500 to maintain model stability and ensure comparability across repeated runs. For the KNN regression model, a grid search was conducted over the number of neighbors (k), covering odd integer values from 1 to 19 to prevent ties in distance-based voting. Prior to model fitting, all the predictor variables were standardized by centering around the zero mean and scaling to the unit variance to ensure fair distance computation. All the models were trained under identical resampling conditions using 5-fold CV, as specified through the trainControl function in caret. The model performance was assessed via the root mean square error, which served as the common evaluation metric across all algorithms. This consistent and fully specified setup guaranteed both the reproducibility and fair comparability of the RF, KNN, and NN models in the presented study.

In order to evaluate the estimators’ performance, we implemented a bootstrap method—see Ahmed et al. [42]—and calculated the mean squared prediction error in the subsequent manner:

• Select, with replacement, a sample of size $n=45$ from the data set K times, say $K=1000$.
• Partition each sample from (1) into separate training and testing sets. Divide the training ($n_{train}=31$) and testing ($n_{test}=14$) sets at a ratio of 70% and 30%, respectively. Then, fit the full and sub-models using the training data set, and obtain the values of all the RE-type estimators.
• Evaluate the predicted response values using each estimator based on the testing data set as follows:

  $$\begin{array}{ccc}\hfill {\widehat{{\mathbf{Y}}_i}}_{,test}& =& {\mathbf{X}}_{i_{,test}}{\widehat{{\mathit{\theta }}_i}}^{RE\ast },\hfill \end{array}$$

  where $i=1,2,\dots ,K$; ${\widehat{{\mathbf{Y}}_i}}_{,test}={\widehat{lps{a}_i}}_{,test}$; ${\mathbf{X}}_{i_{,test}}$ is the matrix of the other variables in the model; and ${\widehat{{\mathit{\theta }}_i}}^{RE\ast }$ is any of the proposed RE estimators.
• Find the prediction error of each estimator for each sample as follows:

  $$\begin{array}{ccc}\hfill P{E}_i\left({\widehat{{\mathit{\theta }}_i}}^{RE\ast }\right)& =& {\displaystyle \frac{1}{n_{i_{test}}}}{\mathbf{e}}_i^T{\mathbf{e}}_i,\hfill \end{array}$$

  where $\mathbf{e}={\widehat{{\mathbf{Y}}_i}}_{,test}-{\overline{{\mathbf{Y}}_i}}_{,train}-\left({\mathbf{X}}_{i_{test}}-{\overline{\mathbf{X}}}_{i,train}\right){\widehat{{\mathit{\theta }}_i}}^{RE\ast }$.
• Calculate the average prediction error of all the estimators as follows:

  $$\begin{array}{ccc}\hfill AVPE\left({\widehat{\mathit{\theta }}}^{RE\ast }\right)& =& {\displaystyle \frac{1}{K}\sum _{i=1}^{K}P{E}_i\left({\widehat{\mathit{\theta }}}^{RE\ast }\right)}\hfill \end{array}$$
• Finally, calculate the relative efficiency of the prediction error with respect to ${\widehat{\mathit{\theta }}}^{RE}$ as follows:

  $$\begin{array}{ccc}\hfill REPE\left({\widehat{\mathit{\theta }}}^{RE\ast },{\widehat{\mathit{\theta }}}^{RE}\right)& =& {\displaystyle \frac{AVPE\left({\widehat{\mathit{\theta }}}^{RE}\right)}{AVPE\left({\widehat{\mathit{\theta }}}^{R{E}^{\ast }}\right)}}\hfill \end{array}$$

The findings shown in

Table 2. The relative efficiency of the prediction error of the RE shrinkage estimators for the biomass data.

Estimator	Sub.1	Sub.2
$RE$	1.000	1.000
$RRE$	1.482	1.473
$REPT$	1.392	1.451
$RES$	1.263	1.129
$REPS$	1.355	1.404
$LSRE$	1.517	1.453
$SPTRE$	1.422	1.431

align with the outcomes of the simulations discussed in the preceding subsection.

Next, we employed various penalizations and three machine learning algorithms to analyze prostate cancer data. Our objective was to determine the prediction error. We first scaled all variables, including the responses (BIO), and then applied the penalizations or machine learning algorithms to avoid any differences in the variable units. The results of our investigation are summarized in the following table.

As shown in

Table 3. Relative efficiency of the prediction error for penalized and machine learning algorithms (MLMs) for biomass data.

Penalized Method	REPE	MLM	REPE
Ridge	0.891	RF	1.039
LASSO	0.983	KNN	1.053
ELNT	1.170	NN	1.138
SCAD	0.889
ALASSO	1.148

, every machine learning algorithm outperformed the traditional RE estimator. However, the performance of the ridge, LASSO, and SCAD penalization methods was inferior to that of the RE estimator. This discrepancy may be attributed, in part, to the presence of multicollinearity among the predictor variables.

Overall, the RE-type shrinkage estimators exhibited the highest relative efficiency, outperforming both the penalized and ML models; the penalized estimators achieved moderate gains under correlated predictors, while the ML algorithms performed well in capturing nonlinear interactions, but remained less efficient in structured linear settings. The superior performance of RE-type shrinkage estimators can be attributed to their ability to incorporate valid linear restrictions and simultaneously apply shrinkage, which reduces estimator variance under multicollinearity. When the underlying data structure is approximately linear, these estimators achieve a favorable bias–variance trade-off, whereas the penalized and ML models, despite their flexibility, may lose efficiency due to over-regularization or overfitting in such structured settings.

Upon the careful examination of the numerical data, it became apparent that the prediction error’s relative efficiencies differed from one method to another. To gain a deeper comprehension of the range of values and potential outliers in our prediction, we examined the associated prediction error using the following box plots, and based on 1000 replications.

The box plot in Figure 4 clearly illustrates the distribution of our data set, and with further analysis, it became apparent that there are clues of possible outliers. The extended whiskers and isolated data points beyond the usual range indicate variability and occurrences that diverge from the general pattern. Furthermore, applying a shrinkage technique to the (RE) estimator revealed a significant effect on the suppression of outliers. Applying shrinkage not only improves the estimate process, but also helps us identify and reduce the impact of outliers, resulting in a more flexible and dependable representation of the underlying data structure.

8.3. Example 2: Air Pollution Data Set

The air pollution data initially utilized by McDonald and Schwing [43] were subsequently employed by [12,44] to demonstrate universal ridge shrinkage methods. The data can be accessed freely at Carnegie Mellon University’s StatLib (https://lib.stat.cmu.edu/datasets/, accessed on 17 November 2025), and it has 15 covariates pertaining to air pollution and socio-economic and meteorological observations, with the mortality rate as the dependent variable for 60 US cities in 1960. These variables are the average annual precipitation in inches Precip, annual average percentage of relative humidity at 1:00 pm Humidly, average January temperature in degrees F JanTemp, average July temperature in degrees F JulyTemp, percentage of people aged 65 or older Over65, population per household House, median number of school years completed by persons over 22 years Educ, percentage of housing units that are sound and with all facilities Sound, population density per squared mile in urbanized areas Density, percentage of non-white population NonWhite, percentage of people employed in white-collar occupations WhiteCol, percentage of families with an income less than USD 3000 Poor, relative hydrocarbon pollution potential HC, relative nitric oxide pollution potential NOX, relative sulphur dioxide pollution potential SO2, and total age-adjusted mortality rate per 100,000 MORT.

As we did in the first example, to examine multicollinearity among the explanatory variables, we created the correlation matrix shown in Figure 5 below, which shows that a significant relationship exists among some of the variables. This result is also supported by the VIF plot given in Figure 6.

Using the ols_step_best_subset function from the olsrr package applied to the pollution data set, the best subset selection procedure identified the variables Precip, JanTemp, JulyTemp, Educ, NonWhite, and SO2 based on the Cp criterion, whereas the variables Precip, JanTemp, JulyTemp, House, Educ, NonWhite, and SO2 were selected according to the AIC.

Following the methods employed in the first example and detailed in Section 8.2, the relative efficiencies of the prediction error for the proposed estimators about ${\widehat{\mathit{\theta }}}^{RE}$ are presented below in

Table 4. The relative efficiency of the prediction error for RE shrinkage estimators for the air pollution data.

Estimator	Sub.1	Sub.2
$RE$	1.000	1.000
$RRE$	1.424	1.303
$REPT$	1.406	1.296
$RES$	1.290	1.180
$REPS$	1.318	1.242
$LSRE$	1.400	1.401
$SPTRE$	1.383	1.392

.

Table 4 reiterates the analogous conclusions derived in Example 1 regarding the preferences of the proposed estimators. Secondly, we utilized the identical penalized approaches and machine learning algorithms applied in Example 1, and a summary of our results is displayed in the subsequent table (see

Table 5. Relative efficiency of the prediction error for penalized and machine learning algorithms (MLMs) for the air pollution data.

Penalized Method	REPE	MLM	REPE
Ridge	0.930	RF	0.916
LASSO	0.936	KNN	0.697
ELNT	1.087	NN	1.076
SCAD	0.812
ALASSO	0.947

).

The results of the penalization methods and machine learning algorithms matched with those obtained in Example 1. Furthermore, we created box plots for the variables in the second data set in Figure 7, which again depicted the distribution of this data set. The figure indicates the presence of probable outliers in the data set; hence, utilizing shrinkage estimation methods will mitigate the influence of these outliers, enhance the estimating process, and yield a more adaptable and reliable representation of the underlying data structure.

9. Conclusions

This article introduces the pretest and shrinkage of the new ridge estimators when there is multicollinearity among predictor variables in a multiple linear regression model. Since the pretest and shrinkage technique relies on prior information, we developed a linear hypothesis to assess the significance of certain regressor variables in the regression model. We then combined the result in our estimation process. Subsequently, we benefited from Kibria Lukman’s idea by implementing the novel estimating approach to mitigate the extent of multicollinearity.

In order to compare, we implemented several penalization approaches along with three machine learning algorithms. We assessed the comparative efficacy of each method’s prediction error in relation to the usual RE estimator.

Our findings demonstrated that the utilization of the shrinkage estimation technique resulted in a significant enhancement in the mean squared error when applied to simulated data under various configurations of the correlation coefficient among the predictor variables. Furthermore, these findings were supported by two real data illustrations in which we employed the relative efficiency of the prediction errors as a benchmark for comparison.

The suggested estimation techniques can be expanded to encompass various different types of regression models, such as Poisson, logistic, and beta regression models, among others. In addition, the analysis can consider the case of high-dimensional data, evaluating the effectiveness of the suggested estimators in comparison to penalization methods and machine learning algorithms.