Performance Evaluation of the Robust Stein Estimator in the Presence of Multicollinearity and Outliers

Abstract

Multicollinearity and outliers are common challenges in multiple linear regression, often adversely affecting the properties of least squares estimators. To address these issues, several robust estimators have been developed to handle multicollinearity and outliers individually or simultaneously. More recently, the robust Stein estimator (RSE) was introduced, which integrates shrinkage and robustness to effectively mitigate the impact of both multicollinearity and outliers. Despite its theoretical advantages, the finite-sample performance of this approach under multicollinearity and outliers remains underexplored. First, outliers in the y direction have been the main focus of earlier research on the RSE, not considering that leverage points could substantially impact regression results. Second, this study addresses the gap by considering outliers in the y direction and leverage points, providing a more thorough assessment of the RSE robustness. Finally, to extend the limited existing benchmark, we compare and evaluate the RSE performance with a wide range of robust and classical estimators. This extends existing benchmarking, which is limited in the current literature. Several Monte Carlo (MC) simulations were conducted, considering both normal and heavy-tailed error distributions, with sample sizes, multicollinearity levels, and outlier proportions varied. Performance was evaluated using bootstrap estimates of root mean squared error (RMSE) and bias. The MC simulation results indicated that the RSE outperformed other estimators under several scenarios where both multicollinearity and outliers are present. Finally, real data studies confirm the MC simulation results.

1. Introduction

Linear regression analysis is an important statistical tool that helps model a relationship between the response variable ($\mathit{y}$) and the predictors ($\mathit{X}$), and it is often applied in all fields of study. A linear model has the following form:

$$\mathit{y}=\mathbf{X}\mathit{\beta }+\mathit{\epsilon },$$

(1)

where $\mathit{y}$ is an $n\times 1$ vector of observed values of the dependent variable, $\mathit{X}={[x_1,x_2,\dots ,x_n]}^{\prime }$ is an $n\times p$ matrix with a row vector $x_i={[x_{i1},x_{i2},\dots ,x_{ip}]}^{\prime }$ of components $x_{ij}$ of the regressors, $i=1,2,3,\dots ,p$, $\mathit{\beta }$ is a vector $p\times 1$ of unknown parameters and $\epsilon$ is an $n\times 1$ vector of errors. Under classical assumptions that the errors have zero mean, constant variance and are uncorrelated, then the ordinary least-squares (OLS) estimate $\widehat{\beta }$ is the best linear unbiased estimator (BLUE) of $\beta$ according to the Gauss-Markov theorem. The OLS estimate is BLUE even without assuming normally distributed residuals and minimizes the sum of squared residuals. If an error term ($\epsilon$) follows a normal distribution $\epsilon \sim \mathcal{N}(0,{\sigma }^{2}I)$, this assumption of normality is essential to perform hypothesis tests and construct confidence intervals. If there is more than one predictor variable, then the analysis is referred to as multiple linear regression analysis [1]. A multiple regression model could be used to describe this relationship:

$$y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_p{x}_p+\epsilon .$$

(2)

Applying the ordinary least squares estimator (OLSE) in simple or multiple linear regression always requires some assumptions, that is, normality of the error terms, equal variance of the error terms, absence of outliers, leverage points, and multicollinearity. Therefore, the OLS estimate is the best method in regression analysis if the assumptions are met. However, if these assumptions are not satisfied, the results can easily be affected [2].

Various problems can occur in the estimation of the model parameters, one of which is if the assumption of normality is not fulfilled, or due to outliers in the data [3]. According to [4,5], the normality of error distributions is based on the central limit theorem, which is a limit theorem based on approximations. Hampel and Huber [4] critically challenged the prevailing assumption that the OLSE remains approximately optimal under conditions of approximate normality. This is because it has been shown that typical error distributions of real-world datasets usually deviate slightly from normality and, in most cases, are tailed. Additionally, outliers in the dependent variable lead to large residual values, which further result in the failure of the normality assumption of the error terms. According to [6], the data that come from the regression line is an outlier. Outliers are data that have different characteristics from other non-outlier data. There exist several statistical methods used to detect outliers, such as the Cook distance index plot and the potential residual plot (e.g., [7]). The existence of outliers can affect the data analysis process, leading to incorrect decision making. However, the existence of outliers can be overcome by several methods, including robust regression [8]. Several robust regression estimators have been provided to address the problem of outliers in different regression models; these include the Huber maximum estimator (HME), the modified maximum likelihood estimator (MME), the least trimmed squares estimator (LTSE), the least median of squares estimator (LMSE), and the S estimator (SE) [9].

In addition, multicollinearity is another pressing challenge in linear regression models that occurs when predictor variables have a high degree of correlation. This problem causes maximum likelihood estimates to be unstable and inefficient. Therefore, many biased estimators have been provided to address this problem in different regression models [10,11]. However, both multicollinearity and outliers can exist simultaneously in a model. To address both issues, various estimators have been combined for robust estimation in order to handle multicollinearity and outliers [12,13,14]. Despite existing robust estimators such as HME, MME, SE, LTSE, and LMSE handling outliers effectively, they suffer a significant efficiency loss under multicollinearity due to increased variance of coefficient estimations. In contrast, shrinkage estimators, such as the James-Stein estimator (JSE), handle multicollinearity but entirely fail in the presence of outliers, since they rely on OLS-based estimation, which is extremely susceptible to contamination. Existing estimators struggle to maintain high efficiency under multicollinearity while maintaining resilience to outliers [9].

To address this gap, Lukman et al. [14] recently introduced the robust Stein estimator (RSE) by combining shrinkage to reduce multicollinearity with M-estimation to prevent outlier effect. Despite its promising theoretical foundation, the RSE has not received much attention in the literature and its practical performance has not been extensively examined under various contamination settings. There are particularly limited studies that investigate how well it holds up in the presence of leverage points, heavy-tailed error distributions, and varied degrees of multicollinearity. In this study, we examine a robust version of the Stein estimator, which is the combination of the HME and the JSE, which was originally proposed by [14] and can handle both multicollinearity and outliers simultaneously. The performance of this estimator is assessed and compared with that of other robust regression methods utilizing different simulation scenarios and real data applications.

The rest of the paper is organized as follows. In Section 2, we begin with the materials and methods. Section 3 presents the results of Monte Carlo (MC) simulation study and the empirical application results that shows how RSE performs on real-world data. The study concludes with a discussion and some conclusions that summarize the main findings and identify potential areas for further research in Section 4 and Section 5.

2. Materials and Methods

In this section, attention will be paid to the robust methods considered in this study and their properties. The selected estimators were chosen for their theoretical significance and their easy implementation and computational viability due to their availability in R.

2.1. Review on Robust Estimators

When the data are contaminated by one or a few outliers, the issue of detecting such observations becomes challenging. However, in general, data sets often contain multiple outliers or clusters of influential observations, and they can also exhibit multicollinearity among their variables. These two factors provide additional insight into the co-occurrence of both outliers and multicollinearity, which complicates regression analysis, with biased and unstable parameter estimates as a common result. Alma [15] points out that robust estimation is an important method for analyzing data contaminated by outliers. It is an approach to regression analysis that aims to overcome some of the limitations of classical parametric and nonparametric approaches. Well-performing estimates are robust estimates designed to become less sensitive to outliers and multicollinearity. Here, we present a review of existing robust estimators used in linear regression.

2.1.1. The Huber Maximum Likelihood Estimator (HME)

The most common general robust method is M-estimates, introduced by Huber et al. [5], which minimizes the objective function. In this study, we use a turning constant $k=1.345$, which offers around 95% efficiency while protecting against outliers [16]. The rlm() function in the MASS R statistical package was used to implement the HME. For y-direction outliers, this estimator serves as a baseline without considering multicollinearity into account:

$$\sum _{i=1}^n\rho \left(r_i\right)=\sum _{i=1}^n\rho \left({\displaystyle \frac{e_i\left(\beta \right)}{\sigma }}\right),$$

(3)

where $r_i={\displaystyle \frac{e_i\left(\beta \right)}{\sigma }}$ are called scaled residuals. Differentiating with respect to $\beta$, assuming $\sigma$ is fixed, and setting the partial derivatives to zero, results in the following normal equations:

$$\sum _{i=1}^n\psi \left({\displaystyle \frac{e_i\left(\beta \right)}{\sigma }}\right)x_i=0.$$

(4)

Let $X^{*}=\left[1_{n}X\right]$ denote the design matrix $n\times (p+1)$, where ${\mathbf{1}}_n$ is the intercept column, and $\dot{X}$ contains the predictors p. The ith observation can be written as ${\mathbf{x}}_i^{\prime }=(1,x_{i1},x_{i2},\dots ,x_{ip})$, where $i=1,2,\dots ,n$ (observations) and $j=1,2,\dots ,p$ (predictors). We consider spectral decomposition $T^{\prime }\left(X^{\prime }X\right)T=\Lambda =diag\left({\lambda }_j\right),j=1,2,\dots ,p^{*},(p^{*}=p+1)$. Following [14], the model in Equation (1) is transformed as follows:

$$y_i=\sum _{j=1}^{p^{*}}{\alpha }_j{h}_{ij}+{\epsilon }_i,i=1,2,\dots ,n$$

(5)

where ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_{p^{*}}$, and T is an orthogonal matrix $(p^{*}\times p^{*})$ whose columns are eigenvectors corresponding to these eigenvalues. The transformation is defined as $\mathbf{H}=\dot{X}T$, $\mathit{\alpha }=T^{\prime }\mathit{\beta }$, and $T\left(X^{\prime }X\right)T^{\prime }={\mathbf{H}}^{\prime }\mathbf{H}=\mathrm{\Lambda }.$ Therefore, the OLS of $\mathit{\alpha }$ is written as follows:

$${\widehat{\alpha }}_{LS}={\mathrm{\Lambda }}^{-1}{H}^{\prime }y.$$

(6)

The M-estimator of $\alpha$ is

$${\widehat{\alpha }}_M=min\sum _{i=1}^n\rho \left({\displaystyle \frac{{\epsilon }_i}{\sigma }}\right)=min\sum _{i=1}^n\rho \left({\displaystyle \frac{y_i-{\sum }_{j=1}^{p^{*}}{\alpha }_j{h}_{ij}}{\sigma }}\right),$$

(7)

where $\rho$ is a robust function and $\sigma$ is a scale parameter. The estimator ${\widehat{\alpha }}_M$ is obtained as the solution to the M-estimating Equations (3) and (4).

2.1.2. James Stein Estimator (JSE)

As a remedy to the problem of multicollinearity, the James-Stein estimator (JSE), originally proposed by [17,18], applies shrinkage to the ordinary least squares estimator to improve estimation efficiency under correlated regressors. In the standard regression context, the JSE is given by the following:

$${\widehat{\mathit{\beta }}}_{\mathrm{JSE}}=\left(1-{\displaystyle \frac{(p-2){\widehat{\sigma }}^2}{{\widehat{\mathit{\beta }}}_{\mathrm{OLS}}^{\prime }{\mathbf{X}}^{\prime }\mathbf{X}{\widehat{\mathit{\beta }}}_{\mathrm{OLS}}}}\right){\widehat{\mathit{\beta }}}_{\mathrm{OLS}},\mathrm{for}p>2,$$

(8)

where ${\widehat{\mathit{\beta }}}_{\mathrm{OLS}}$ is the ordinary least squares estimator and ${\widehat{\sigma }}^2$ is the estimated error variance. In the transformed $\alpha$-space defined in Equation (5), following [14], the JSE can be expressed as follows:

$${\widehat{\mathit{\alpha }}}_{\mathrm{JSE}}=c{\widehat{\mathit{\alpha }}}_{\mathrm{LS}},$$

(9)

where $0<c<1$ and ${\widehat{\alpha }}_{\mathrm{LS}}$ is an unbiased estimate of $\alpha$. If $c=1$, then ${\widehat{\alpha }}_{\mathrm{JSE}}={\widehat{\alpha }}_{\mathrm{LS}}$. The shrinkage factor c is given by the following:

$$c={\displaystyle \frac{{\widehat{\mathit{\alpha }}}_{\mathrm{LS}}^{\prime }{\widehat{\mathit{\alpha }}}_{\mathrm{LS}}}{{\widehat{\mathit{\alpha }}}_{\mathrm{LS}}^{\prime }{\widehat{\mathit{\alpha }}}_{\mathrm{LS}}+{\sigma }^2\mathrm{tr}\left({\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}\right)}}=\sum _{j=1}^{p^{*}}{\displaystyle \frac{{\lambda }_j{\widehat{\alpha }}_j^2}{{\sigma }^2+{\lambda }_j{\widehat{\alpha }}_j^2}}.$$

(10)

The equivalence between the two forms in Equation (10) follows from the spectral decomposition, noting that $\mathrm{tr}\left({\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}\right)={\sum }_{j=1}^{p^{*}}{\lambda }_j^{-1}$ and the canonical transformation diagonalize the covariance structure into independent components along the eigenvector directions.

2.1.3. The Robust Stein Estimator (RSE)

The robust Stein estimator was first introduced by [14] as an alternative method to the Stein and ridge estimators to improve the accuracy of the estimation. However, both methods are non-robust to outliers. In the study, it was indicated that the Stein estimator is sensitive to outliers in the y-direction. They proposed the RSE [14], which is defined as:

$${\widehat{\alpha }}_{M\text{-}JSE}=c^{*}{\widehat{\alpha }}_M,$$

(11)

where ${\widehat{\alpha }}_M$ is the M-estimate of $\alpha$, the shrinkage factor $c^{*}$ in the RSE (M–Stein) is often defined using the following components:

$$c^{*}=\sum _{j=1}^{p^{*}}\left({\displaystyle \frac{{\lambda }_j{\alpha }_M^2}{{\mathrm{\Psi }}_{jj}+{\lambda }_j{\alpha }_M^2}}\right),$$

(12)

where

• ${\lambda }_j$ are the ordered eigenvalues of the design information matrix (e.g., X′X in linear models)
• ${\alpha }_{M,j}$ are the components of the robust estimator ${\widehat{\alpha }}_M$,
• ${\mathrm{\Psi }}_{jj}$ are the diagonal entries of the variance-covariance matrix $Var\left({\widehat{\alpha }}_M\right)$.

The robust covariance matrix $\mathbf{\Psi }$ is estimated using the sandwich estimator [5]:

$$\widehat{\mathbf{\Psi }}={\widehat{A}}^{-2}\widehat{B},\mathrm{where}{\widehat{A}}^2={\displaystyle \frac{1}{n-p^{*}}}\sum _{i=1}^n{\left[{\psi }^{\prime }\left({\displaystyle \frac{e_i}{\widehat{\sigma }}}\right)\right]}^2,\widehat{B}={\displaystyle \frac{{\widehat{\sigma }}^2}{{(n-p^{*})}^2}}\sum _{i=1}^n{\left[\psi \left({\displaystyle \frac{e_i}{\widehat{\sigma }}}\right)\right]}^2,$$

(13)

and $\widehat{\sigma }$ is a robust scale estimate. The approximate covariance of ${\widehat{\mathit{\alpha }}}_{\mathrm{M}\text{-}\mathrm{JSE}}$ is the following:

$$Cov\left({\widehat{\mathit{\alpha }}}_{\mathrm{M}\text{-}\mathrm{JSE}}\right)\approx c^{*}Cov\left({\widehat{\mathit{\alpha }}}_M\right){\left(c^{*}\right)}^{\prime }=c^{*}\mathbf{\Psi }{\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}{\left(c^{*}\right)}^{\prime }.$$

(14)

The approximate bias is the following:

$$Bias\left({\widehat{\mathit{\alpha }}}_{\mathrm{M}\text{-}\mathrm{JSE}}\right)\approx \mathbb{E}\left(c^{*}{\widehat{\mathit{\alpha }}}_M\right)-\mathit{\alpha }=(c^{*}-1)\mathit{\alpha }.$$

(15)

The matrix mean squared error (MMSE) is approximately the following:

$$\mathrm{MMSE}\left({\widehat{\mathit{\alpha }}}_{M\text{-}JSE}\right)\approx c^{*}\mathbf{\Psi }{\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}{\left(c^{*}\right)}^{\prime }+Bias\left({\widehat{\mathit{\alpha }}}_{M\text{-}JSE}\right)Bias{\left({\widehat{\mathit{\alpha }}}_{M\text{-}JSE}\right)}^{\prime }.$$

(16)

The scalar mean squared error (SMSE) is approximately given by:

$$\mathrm{SMSE}\left({\widehat{\mathit{\alpha }}}_{M\text{-}JSE}\right)\approx {\left(c^{*}\right)}^2\sum _{j=1}^{p^{*}}{\displaystyle \frac{{\mathrm{\Psi }}_{jj}}{{\lambda }_j}}+{(c^{*}-1)}^2\sum _{j=1}^{p^{*}}{\alpha }_j^2,$$

(17)

which can also be expressed as:

$$\mathrm{SMSE}\left({\widehat{\mathit{\alpha }}}_{M\text{-}JSE}\right)\approx \sum _{j=1}^{p^{*}}{\displaystyle \frac{{\mathrm{\Psi }}_{jj}{\lambda }_j{\alpha }_j^4}{{({\mathrm{\Psi }}_{jj}+{\lambda }_j{\alpha }_j^2)}^2}}+\sum _{j=1}^{p^{*}}{\displaystyle \frac{{\mathrm{\Psi }}_{jj}^2{\alpha }_j^2}{{({\mathrm{\Psi }}_{jj}+{\lambda }_j{\alpha }_j^2)}^2}}.$$

(18)

2.1.4. The Least Median of Squares Estimator (LMSE)

In order to reach the highest possible 50% breakdown point (BP), the LMSE minimizes the median of squared residuals rather than their sum [19]. Using the robustbase R statistics package, the LMSE was implemented using the lmsreg() function. Siegel [19] defined the least median of squares (LMS) estimator as:

$${\widehat{\beta }}_{LMS}=\underset{\beta }{argmin}med\left\{e_i^2\right\},$$

(19)

which replaces the sum by the median in the Least Squares Estimator (LSE). The procedure is essentially based on the idea presented by [20]. The advantage of the LMSE is that it is very robust to outliers in both the y direction and the leverage points, and it has been shown that LMSE has the highest possible BP of 0.5. Unfortunately, it has a very low efficiency and can be unstable. Moreover, due to its slow convergence rate, the LMSE does not have a well-defined influence function. Due to these properties, the LMSE is usually used as the initial estimate of residuals for other more efficient methods such as MM-estimators [20].

2.1.5. The Least Trimmed Squares Estimator (LTSE)

Another regression estimator that has a BP of nearly 50% is the least trimmed square (LTS) estimator proposed by [20]. Traditional OLS methods are highly sensitive to outliers, meaning that a few extreme data points can dramatically affect the estimated model. LTSE is designed to be more robust to these outliers by focusing on a subset of the data with the smallest residuals. The estimator chooses the regression coefficients $\beta$ to minimize the sum of the smallest h of the squared residuals and is defined as follows:

$${\widehat{\beta }}_{LTS}=\underset{\beta }{argmin}\sum _{i=1}^h{e}_{\left(i\right)}^2\left(\beta \right),$$

(20)

where $e_{\left(i\right)}^2\left(\beta \right)$ represents the i-th ordered squared residuals $e_{\left(1\right)}^2\le \cdots \le e_{\left(n\right)}^2$ and h is called the trimming constant, which must satisfy $n/2<h\le n$. The constant h determines the BP of the LTSE. Typically, $h=\lfloor n/2\rfloor +\lfloor (p+1)/2\rfloor$ can reach the maximum, where $\lfloor \rfloor$ denotes the floor function (rounded to the next smallest integer). When $h=n$, LTSE is exactly equivalent to the LSE whose BP is zero. Although its convergence rate [21] makes it asymptotically normal, which is better than LMSE, it still suffers a very low efficiency of only 7%. In this study, we used the function ltsReg(), which is available in the robustbase R statistical package, and set $h=50\%BP$.

2.1.6. The S-Estimator (SE)

To find a simple high-breakdown regression estimator that shares the flexibility and nice asymptotic properties of the M-estimator, Rousseeuw et al. [21] introduced the S-estimate. The SE was developed as a robust alternative to traditional estimators like OLSE, particularly when the data contain outliers or high-leverage points. They called it the S-estimate because it is derived from the M-scale estimate equation:

$${\displaystyle \frac{1}{n}}\sum _{i=1}^n\rho \left({\displaystyle \frac{e_i\left(\mathit{\beta }\right)}{\widehat{\sigma }}}\right)=\delta .$$

(21)

In M-estimates, when $\sigma$ is unknown, we use Equation (21) to obtain the scale parameter and regression estimates simultaneously. Let $\delta ={\mathbb{E}}_{\varphi }\left[\rho \left(x\right)\right]$, where $\varphi$ represents the standard normal density, and let

$$d\left(\mathbf{e}\right)={\displaystyle \frac{\#\{i:1\le i\le n,e_i=0\}}{n}}.$$

When $d\left(\mathbf{e}\right)<1-\delta /a$ (where a is the upper bound of $\rho$ and $a\in (0,\infty )$), the scale has a unique positive solution. If $d\left(\mathbf{e}\right)=1-\delta /a$, it may have infinite solutions, including $\sigma =0$, and if $d\left(\mathbf{e}\right)>1-\delta /a$, then there is no solution. To avoid such indeterminacies, we define that whenever $d\left(\mathbf{e}\right)\ge 1-\delta /a$, we set $\sigma \left(\mathbf{e}\right)=0$. To define the S-estimator, let $\rho$ satisfy the following condition:

(A1)

$\rho$ is symmetric, continuously differentiable, and $\rho \left(0\right)=0$. There exists $c>0$ such that $\rho$ is strictly increasing in $[0,c]$ and constant in $[c,\infty ]$.

For each vector $\mathit{\beta }$, using Equation (21), we can calculate the dispersion of residuals $\widehat{\sigma }(e_1\left(\mathit{\beta }\right),\dots ,e_n\left(\mathit{\beta }\right))$, where $\rho$ satisfies (A1). Then, the S-estimator $\widehat{\mathit{\beta }}$ is defined by:

$$\widehat{\mathit{\beta }}=arg\underset{\mathit{\beta }}{min}\widehat{\sigma }(e_1\left(\mathit{\beta }\right),\dots ,e_n\left(\mathit{\beta }\right)).$$

(22)

The tuning constant k is set to $1.547$ to guarantee a $50\%$ breakdown point. We used the function lmrob.S(), which is available in the robustbase R statistical package.

2.1.7. The Modified Maximum Likelihood Estimator (MME)

The MM estimation is a special type of M-estimation developed by [20], and is particularly useful when using non-normal data or outliers, and combines a high breakdown point (50%) with high efficiency through the [20] three-stage procedure. The function lmrob(), which is part of the robustbase R statistical package, was used in this study, with a constant k set to $4.685$ to guarantee a $95\%$ efficiency. Similar to HME, MME does not directly address multicollinearity, although it does improve upon HME’s breakdown point while retaining efficiency.

Stage 1

Compute an initial consistent robust estimate ${\widehat{\mathit{\beta }}}_0$ with a high breakdown point (BP), possibly 50%, but not necessarily efficient.

Stage 2

Compute the M-scale $\widehat{\sigma }$ of the residuals $e_i\left({\widehat{\mathit{\beta }}}_0\right)$ using Equation (21), with a function ${\rho }_0$ satisfying (A1) and choosing a constant $\delta$ such that $\delta /a=0.5$, where $a=sup{\rho }_0\left(e\right)$. Thus, the asymptotic BP of $\widehat{\sigma }$ is 0.5 [16].

Stage 3

Let ${\rho }_1$ be another $\rho$-function satisfying (A1) such that

$$sup{\rho }_1\left(e\right)=sup{\rho }_0\left(e\right)=a,$$

(23)

and

$${\rho }_1\left(e\right)\le {\rho }_0\left(e\right).$$

(24)

Let ${\psi }_1={\rho }_1^{\prime }$ and define the objective function as follows:

$$L\left(\mathit{\beta }\right)=\sum _{i=1}^n{\rho }_1\left({\displaystyle \frac{e_i\left(\mathit{\beta }\right)}{\widehat{\sigma }}}\right),\mathrm{with}{\rho }_1(0/0)=0.$$

(25)

Then, the MM-estimate ${\widehat{\mathit{\beta }}}_1$ is defined as any solution to the following equation:

$$\sum _{i=1}^n{\psi }_1\left({\displaystyle \frac{e_i\left(\mathit{\beta }\right)}{\widehat{\sigma }}}\right)x_i=0,$$

(26)

that also satisfies

$$L\left({\widehat{\mathit{\beta }}}_1\right)\le L\left({\widehat{\mathit{\beta }}}_0\right).$$

(27)

Yohai [20] showed that any value of $\mathit{\beta }$ that satisfies Equations (26) and (27), for example, a local minimum, will have the same efficiency as the global minimum, and its BP is not less than that of ${\widehat{\mathit{\beta }}}_0$. Thus, although the absolute minimum of $L\left(\mathit{\beta }\right)$ exists, it is not necessary to find it. In the first stage, the robust initial estimate ${\widehat{\mathit{\beta }}}_0$ should satisfy regression, scale, and affine equivariance and have a high BP. LMS, LTS, and S-estimates are possible candidates. For Stage 2, one way to choose ${\rho }_0$ and ${\rho }_1$ is as follows:

$${\rho }_0\left(e\right)=\rho \left({\displaystyle \frac{e}{k_0}}\right),{\rho }_1\left(e\right)=\rho \left({\displaystyle \frac{e}{k_1}}\right).$$

where $\rho$ is a function that satisfies (A1). In order to satisfy Equation (23), we must have $0<k_0<k_1$. The value of $k_0$ should be chosen so that $\delta /a=0.5$ is maintained. k is set to $4.685$ to guaranty the efficiency of $95\%$. In this study, the function lmrob() was used, which is part of the robustbase R statistical package.

2.2. Criteria for Assessing Estimator Performance

Following a study by Fitrianto et al. [22], two typical precision measures for the estimator $\widehat{\theta }$ are RMSE and Bias. These measures can be estimated using bootstrap resampling. Specifically, given B = 5000 paired bootstrap samples. The bootstrap estimate of Bias and RMSE is as follows:

2.2.1. Bias Under the Bootstrap

$$\widehat{\mathrm{Bias}}={\displaystyle \frac{1}{B}}\sum _{b=1}^B{\widehat{\theta }}_b^{*}-\widehat{\theta }={\overline{\widehat{\theta }}}^{*}-\widehat{\theta },$$

(28)

where ${\widehat{\theta }}_b^{*}$ are bootstrap copies of $\widehat{\theta }$.

2.2.2. RMSE Under the Bootstrap

$$\widehat{\mathrm{RMSE}}=\sqrt{{\displaystyle \frac{1}{B}}\sum _{b=1}^B{\left({\widehat{\theta }}_b^{*}-\widehat{\theta }\right)}^2},$$

(29)

where ${\widehat{\theta }}_b^{*}$ are bootstrap copies of $\widehat{\theta }$ and B represents the number of bootstrap replications.

3. Results

3.1. Monte Carlo Simulation Study

The data generation process that follows was adapted from [23,24]. We generate n samples from the model using R studio:

$$y_i={\beta }_0+{\beta }_1{X}_{i1}+{\beta }_2{X}_{i2}+{\beta }_3{X}_{i3}+{\epsilon }_i,i=1,2,\dots ,n$$

(30)

where the error terms are generated as ${\epsilon }_i\sim \mathcal{N}(0,1)$, and the explanatory variables are generated using the following equation:

$$X_{ij}=Z_{ij}\sqrt{1-{\rho }^2}+\rho Z_{ij},i=1,2,\dots ,n,j=1,2,3,...,p$$

(31)

where $Z_{ij}$ are independent standard normal random numbers that are held fixed for a given sample of size n, and $\rho$ is the degree of multicollinearity between predictors. In this study, we consider three values of $\rho$ = 0.10, 0.50, and 0.98. The study by [14] mainly examined the RSE under high correlation; we extended the analysis to low and moderate levels to better understand its performance in varying degrees of multicollinearity.

The true values of the regression parameters are chosen as ${\beta }_0={\beta }_1=\cdots ={\beta }_{p-1}=1$, which is a common restriction in simulation studies (e.g., [12,14,25]). Additionally, the performance of the RSE is also evaluated for heavy-tailed error distributions, such as t-distributions and the Cauchy distribution.

Five different sample sizes are considered $n\in \{25,50,100,150,200\}$, with the number of predictors fixed at $p=3$, and different proportions of outliers are evaluated $\pi =0.00\%,10\%,25\%,50\%$. In each simulation setting, we perform $N=5000$ MC replications, which is chosen as a compromise between achieving a low MC error and keeping the computation time reasonable.

To introduce outliers in the data, we consider two types, that is, y direction outliers and leverage points. For outliers in the y direction, we first generate clean data according to Equations (30) and (31), then randomly select $\lfloor n\times \pi \rfloor$ observations and replace their response values by adding a large constant, that is, $y_i^{\mathrm{outlier}}=y_i+10\times {\sigma }_{\epsilon }$, where ${\sigma }_{\epsilon }$ is the standard deviation of the error term. For leverage points, we randomly select $\lfloor n\times \pi \rfloor$ observations and replace their predictor values with $X_{ij}^{\mathrm{outlier}}=X_{ij}+10\times \mathrm{SD}\left(X_j\right)$, where $\mathrm{SD}\left(X_j\right)$ is the standard deviation of the j-th predictor.

A summary of the simulation scenarios is provided in Table 1. In addition, we evaluate the performance of the proposed estimators in the following ten cases:

• Case I: ${\epsilon }_i\sim \mathcal{N}(0,1)$—Standard normal distribution with 0.10, 0.50, and 0.98 multicollinearity and 0.00% outliers.
• Case II: ${\epsilon }_i\sim t_7$—Student’s t-distribution with seven degrees of freedom with 0.10, 0.50, and 0.98 multicollinearity and 0.00% outliers.
• Case III: ${\epsilon }_i\sim t_2$—Student’s t-distribution with two degrees of freedom, 0.10, 0.50, and 0.98 multicollinearity and 0.00% outliers.
• Case IV: ${\epsilon }_i\sim \mathcal{C}(0,1)$—Cauchy distribution errors with 0.10, 0.50, and 0.98 multicollinearity and 0.00% outliers.
• Case V: ${\epsilon }_i\sim \mathcal{N}(0,1)$—Standard normal distribution with 10%, 25%, and 50% outliers with no multicollinearity.
• Case VI: ${\epsilon }_i\sim t_2$—Student’s t-distribution with two degrees of freedom, 10%, 25%, and 50% outliers with no multicollinearity.
• Case VII: ${\epsilon }_i\sim \mathcal{N}(0,1)$—10%, 25%, and 50% outliers in the y-direction with 0.10, 0.50, and 0.98 multicollinearity.
• Case VIII: ${\epsilon }_i\sim \mathcal{N}(0,1)$—10%, 25%, and 50% leverage points with 0.10, 0.50, and 0.98 multicollinearity.
• Case IX: ${\epsilon }_i\sim t_2$ (Student’s t-distribution with two degrees of freedom), with multicollinearity levels of 0.10, 0.50, and 0.98, and outliers in the y-direction at 10%, 25%, and 50%.
• Case X: ${\epsilon }_i\sim t_2$ (Student’s t-distribution with two degrees of freedom), with multicollinearity levels of 0.10, 0.50, and 0.98, and leverage points at 10%, 25%, and 50%.

Table 1. Summary of simulation parameters considered.

Sample Size (n)	Multicollinearity ($\mathit{\rho }$)	Outlier ($\mathit{\pi }$)
25	0.10	0.00
50	0.50	0.10
100	0.98	0.25
150		0.50
200

Table 1. Summary of simulation parameters considered.

Sample Size (n)	Multicollinearity ($\mathit{\rho }$)	Outlier ($\mathit{\pi }$)
25	0.10	0.00
50	0.50	0.10
100	0.98	0.25
150		0.50
200

3.2. Simulation Study Findings

Case I (Table S1 and Figure 1): JSE and RSE produce the smallest RMSE values in all sample sizes, consistent with their theoretical optimality properties. MME, HME, OLSE, and LTSE also achieve RMSE values comparable to those of JSE and RSE only when $\rho$ = 0.50 and 0.98. In contrast, LMSE and SE display relatively higher RMSEs, reflecting their lower efficiency in the presence of multicollinearity. JSE and RSE have achieved exactly zero bias for low multicollinearity $\rho$ = 0.10 and 0.50 in sample sizes at $n\ge 100$, and LMSE shows the highest bias in nearly all scenarios. All estimators exhibit monotonic behavior, with RMSE values consistently decreasing as the sample size increases.

Case II (Table S2 and Figure 2): The RSE consistently achieves the lowest RMSE under all the conditions in this case. The JSE consistently achieves low bias across different sample sizes and multicollinearity levels. The LMSE has the overall worst performance in all simulation scenarios.

Case III (Table S3 and Figure 3): In this case, the performance of the OLSE, JSE, and LMSE is substantially worse than that of any other robust estimator. The RSE, MME, and HME exhibit RMSE values that are almost similar, but the RSE outperforms all estimators in this case, as shown in Figure 3. HME and MME consistently achieve low bias in almost all sample sizes and multicollinearity levels, and JSE has the highest bias at $\rho$ = 0.50 and 0.98.

Case IV (Table S4 and Figure 4): Under Cauchy-distributed errors, OLSE and JSE demonstrate erratic RMSE behavior, with values increasing due to the heavy-tailed nature of the distribution. In contrast, robust estimators maintain lower RMSEs and a monotonic decrease with increasing sample size, highlighting their stability. Furthermore, as the correlation coefficient ($\rho$) increases, the performance of OLSE and JSE worsens further. The RSE is the best estimator under the Cauchy distribution, as shown by Figure 4. HME and MME continue to maintain a consistently lowest bias in all scenarios at $\rho$ = 0.98 and n = 200. OLSE and JSE have the highest bias with values exceeding one.

Case V (Tables S5 and S6, and Figure 5 and Figure 6): The MME performs the best under the y direction, and the LTSE performs the best under the x direction. The RSE yields a higher RMSE because it lacks robustness in both y direction outliers and leverage points. LTSE and SE maintain a small bias and low RMSE in the presence of leverage points, whereas RSE, JSE, and OLSE break down due to high sensitivity, as shown in Figure 5 and Figure 6.

Case VI (Tables S7 and S8 and Figure 7 and Figure 8): Table S7 and Figure 7 show that MME, SE, HME and LTSE consistently achieve the lowest RMSE and almost zero bias, making them the best performers, while RSE, OLSE and JSE perform the worst due to high RMSE values. Table S8 and Figure 8 indicate that LTSE, MME, and SE are the most robust and accurate, while RSE and JSE again show the poorest performance.

Case VII (Tables S9–S11, and Figure 9, Figure 10 and Figure 11): When the data contain multicollinearity and outliers in the response direction, the performance of OLSE, JSE, and LMSE worsens significantly, particularly as the percentage of outliers and the degree of multicollinearity increase, as shown by Figure 9, Figure 10 and Figure 11. The following estimators, LMSE, OLSE and JSE in Figure 11, completely fail when 50% of the observations are contaminated. In contrast, the RSE, SE, HME, and MME demonstrate superior performance relative to OLSE, JSE, and LMSE under the same conditions. In addition, LTSE and SE exhibit even better robustness, maintaining substantially lower RMSE values compared to OLSE, JSE and LMSE, particularly in the presence of low contamination, which is 10% in Figure 9. When there are 25% of outliers in Figure 10, we observe patterns similar to Figure 9. The RSE performs very well in cases of outliers and multicollinearity, followed by MME, HME, and LTSE.

Case VIII (Tables S12–S14, and Figure 12, Figure 13 and Figure 14): RSE outperforms all estimators in all scenarios when the data contain multicollinearity and leverage points. The performance of LMSE, SE, LTSE, and OLSE worsens significantly as the percentage of outliers and multicollinearity increases, as shown by Figure 12, Figure 13 and Figure 14. The bias is the smallest for SE and MME when $\pi =25\%$ and 50%.

Cases IX (Tables S15–S17 and Figure 15, Figure 16 and Figure 17): The RSE consistently performs better than average in all simulation scenarios. When 10% outliers and multicollinearity are present, HME, MME, LTSE, and SE work with similar results as shown in Figure 15. In contrast, JSE and OLSE always have the poorest performance in all of these scenarios. LMSE also shows less than ideal performance accuracy, specifically below 10% contamination, over which it generates the highest bias values. Overall, the HME and MME are the most accurate estimators with a consistently lower bias compared to other methods.

Cases X (Tables S18–S20, and Figure 18, Figure 19 and Figure 20): RMSE values are shown in Figure 18, Figure 19 and Figure 20 under different multicollinearity levels and the leverage points. The RMSE increases with both higher multicollinearity and larger proportions of leverage points. RSE consistently provides the lowest RMSE, reflecting good stability, while OLSE and JSE perform the worst. In terms of bias, HME and MME show the lowest bias in all scenarios, and JSE shows the highest bias. These findings also emphasize the better performance of RSE and therefore its robustness and precision and show the relative bias performance of the other estimators under challenging data conditions.

In summary, JSE and OLSE only work well when there are no outliers since they are very sensitive to outliers. The estimator’s bias and RMSE values increased in the case of degree of multicollinearity ($\rho$) and the outlier percentage ($\pi$), and decreased in the case of increasing sample size (n) when other factors were fixed. When only the multicollinearity problem existed in the model (as in Figure 1 and Figure 2, or when there were no outliers, i.e., $\pi =0.00\%$), OLSE, JSE, HME, MME and RSE were better than LMSE, LTSE and SE. But in Figure 4, OLSE and JSE performed the worst, even though they had no outliers. When both problems existed (as in Figure 9, Figure 10, Figure 11, Figure 12, Figure 13, Figure 14, Figure 15, Figure 16, Figure 17, Figure 18, Figure 19 and Figure 20 or when there were outliers, i.e., $\pi >0.00\%$), RSE, HME, and MME were better than LMSE, LTSE, SE, JSE and OLSE, respectively, for all values $\pi$, $\rho$, and n. Finally, the RSE achieved the best performance among all given estimators when there are outliers and multicollinearity.

3.3. Empirical Applications

In the previous section, we conducted an MC simulation study to compare the performance of the estimators. However, simulations are usually performed under some ideal conditions. In contrast to the MC simulation, this section considers three datasets as an illustrative example for handling outliers and multicollinearity in linear regression. The datasets are the milk dataset, the real estate valuation dataset and the Hawkins–Bradu–Kass data set. We adopted a systematic trimming procedure according to [26] to modify the outlier contamination levels in each data set because real datasets frequently contain variable and uncontrollable levels of outliers. This method allows us to direct comparison of theoretical, simulated and real-data results, which enables us to produce multiple versions of each data set with particular outlier percentages that match our simulation study design. A summary of the datasets considered for this study is provided in

Table A1. Summary of the datasets.

Study No.	Dataset	n	p	No. of Outliers	Outliers (%)
	Milk dataset (original)	86	8	20	20%
1	Milk trimmed dataset	62	8	0	0%
	Milk trimmed dataset	81	8	8	10%
	Real Estate Valuation (original)	414	7	88	21.25%
2	Real Estate Valuation trimmed dataset	212	7	0	0%
	Real Estate Valuation trimmed dataset	268	7	41	15.3%
3	Hawkins–Bradu–Kass dataset (original)	75	4	14	18.67%
	Hawkins–Bradu–Kass trimmed dataset	61	4	0	0%

.

Study 1: Milk Dataset

The milk dataset provided by [27] is a composition of milk with eight variables. The eight variables are density, fat content, protein content, casein content, cheese dry substance measured in the factory, cheese dry substance measured in the laboratory, milk dry substance, and cheese produced. According to [28], there are 17 outliers in this dataset, which makes the percentage of outliers 20%. Observations (1st–3rd, 12th, 13th–17th, 27th, 41st, 44th, 47th, 70th, 74th, 75th, and 77th) are outliers. The milk dataset has severe multicollinearity (CN = 164.0314) and 20% y-direction outliers, which makes it similarly equivalent to simulation Case VII ($\rho =0.98$, $\pi =0.10$ to $0.25$).

3.4. Analysis for the Milk Dataset

3.4.1. Multicollinearity Detection for Milk Dataset

The degree of multicollinearity in regression analysis is determined by the variance inflation factor (VIF). If $\mathrm{VIF}\ge 10$, it is generally considered that multicollinearity is high [29]. Another metric to identify multicollinearity in regression models is the condition number (CN). A CN between 30 and 100 indicates extremely high linear multicollinearity (see [30]).

Table A2. Variance inflation factor (VIF) of the milk, the real estate, and Hawkins Bradu Kass datasets.

Dataset	Variable	VIF	Variable	VIF
Milk dataset	$X_1$	2.2007	$X_5$	24.7561
	$X_2$	8.2865	$X_6$	3.2919
	$X_3$	7.2187	$X_8$	2.1834
	$X_4$	24.2253
Real estate valuation dataset	$X_1$	1.0147	$X_5$	1.6023
	$X_2$	1.0143	$X_6$	2.9263
	$X_3$	4.3230
	$X_4$	1.6170
Hawkins Bradu Kass datase	$X_1$	13.4320
	$X_2$	23.8535
	$X_3$	33.4325

presents the VIF for the milk data set, which is a crucial diagnostic tool to detect multicollinearity in regression analysis. It is noted that in the milk data set, variables $X_4$ and $X_5$ exhibit high VIF. In Figure 21, it is observed that strong positive correlations are present between the variables $X_2$, $X_3$, $X_4$ and $X_5$ ($\rho >0.60$), with $X_4$ and $X_5$ indicating severe ($\rho =0.98$).

3.4.2. Outlier Detection Using Cook’s Distance for Milk Dataset

The Cook’s distance plot for the milk data set, with a threshold value of 0.047, is shown in Figure 22. It was discovered that two observations (70 and 74) greatly exceed this threshold, with observation 70 achieving a value above 2.0, and these data points are considered to be very significant and deserve more investigation. Their existence indicates the possibility of anomalies or leverage points that might influence the predictions of the model.

3.4.3. Testing for Normality for Milk Dataset

For this study, both theoretical and graphical techniques will be used to assess the normality of the OLS residuals. Theoretically, we used the Shapiro–Wilk test to evaluate consistency to the normality assumption. Extensive empirical evidence demonstrates that the Shapiro–Wilk test is among the most powerful goodness-of-fit procedures for detecting departures from normality, including in the context of regression residuals [31,32,33,34,35]. Figure 23 shows a diagnostic plot to assess residual normality in milk data. Visual inspection of these plots reveals that the milk data display approximately normal residuals. The results of the theoretical test yielded a p-value of 0.1964 ($p>0.05$), indicating that the milk data follows a normal distribution.

3.4.4. Model Fit and Evaluation for Milk Dataset

To compare the performance of the methods, the regression model was fitted using the milk data set for each method, considering the sparsity of the models. Finally, bias and RMSE were used to evaluate how well the methods performed. The results presented in

Table 2. Estimated bias and RMSE of the estimators under multicollinearity and varying outlier levels for the milk dataset.

Estimator	0% Outlier		10% Outlier		20% Outlier
	Bias	RMSE	Bias	RMSE	Bias	RMSE
OLSE	0.0002	0.0247	0.0007	0.0295	0.0028	0.0270
HME	0.0010	0.0280	0.0032	0.0282	0.0035	0.0250
MME	0.0026	0.0472	0.0059	0.0126	0.0058	0.0340
LMSE	0.0044	0.0890	0.0056	0.0667	0.0064	0.0606
LTSE	0.0041	0.0767	0.0006	0.0531	0.0065	0.0489
SE	0.0043	0.0753	0.0054	0.0235	0.0064	0.0214
JSE	0.0006	0.0228	0.0014	0.0126	0.0029	0.0111
RSE	0.0013	0.0261	0.0030	0.0114	0.0037	0.0108

Note: Bold values indicate the lowest Bias or RMSE within each outlier category.

cover the bias and the RMSE of each estimation method for the milk dataset. The results show that the RSE has outperformed all estimates when there are both outliers and multicollinearity. When there is multicollinearity and 0% outliers, the JSE, OLSE, and other robust estimates have almost the same RMSE values, the same thing that was happening in our simulation study. The next estimators with similar performance when multicollinearity and outliers exist are JSE, MME, and HME. Figure 24 was used to assess bias and RMSE. The OLSE bias (red line) was observed to increase with increasing outlier percentage. The RSE RMSE values (black line) decrease with increasing outlier percentage.

In

Table 3. Coefficient estimates with 95% confidence intervals for the milk dataset.

Method		${\mathit{X}}_1$	${\mathit{X}}_2$	${\mathit{X}}_3$	${\mathit{X}}_4$	${\mathit{X}}_5$	${\mathit{X}}_6$	${\mathit{X}}_7$
OLSE	LCL	$-0.0102$	$-0.0147$	$-0.0235$	$-0.0176$	$-0.0357$	$0.0051$	$-0.0013$
	Estimate	$-0.0032$	$0.0016$	$0.0044$	$0.0066$	$-0.0172$	$0.0079$ *	$0.0187$
	UCL	$0.0030$	$0.0257$	$0.0169$	$0.0286$	$0.0022$	$0.0110$	$0.0409$
HME	LCL	$-0.0093$	$-0.0152$	$-0.0187$	$-0.0142$	$-0.0313$	$0.0051$	$-0.0042$
	Estimate	$-0.0023$	$-0.0006$	$0.0086$	$0.0060$	$-0.0145$	$0.0077$ *	$0.0123$
	UCL	$0.0031$	$0.0210$	$0.0193$	$0.0254$	$0.0029$	$0.0107$	$0.0320$
MME	LCL	$-0.0105$	$-0.0155$	$-0.0300$	$-0.0196$	$-0.0333$	$0.0044$	$-0.0235$
	Estimate	$-0.0019$	$0.0031$	$0.0098$	$0.0029$	$-0.0149$	$0.0078$ *	$0.0053$
	UCL	$0.0038$	$0.0260$	$0.0237$	$0.0233$	$0.0102$	$0.0112$	$0.0343$
LMSE	LCL	$-0.0184$	$-0.0373$	$-0.0589$	$-0.0456$	$-0.0603$	$0.0015$	$-0.0481$
	Estimate	$-0.0014$	$0.0061$	$0.0008$	$0.0046$	$-0.0122$	$0.0078$ *	$0.0056$
	UCL	$0.0101$	$0.0459$	$0.0571$	$0.0482$	$0.0487$	$0.0136$	$0.0577$
LTSE	LCL	$-0.0150$	$-0.0249$	$-0.0472$	$-0.0326$	$-0.0467$	$0.0037$	$-0.0319$
	Estimate	$-0.0015$	$0.0063$	$0.0013$	$0.0041$	$-0.0121$	$0.0079$ *	$0.0052$
	UCL	$0.0059$	$0.0381$	$0.0449$	$0.0322$	$0.0297$	$0.0121$	$0.0419$
SE	LCL	$-0.0178$	$-0.0178$	$-0.0464$	$-0.0366$	$-0.0430$	$0.0036$	$-0.0319$
	Estimate	$-0.0008$	$0.0059$	$0.0068$	$0.0030$	$-0.0157$	$0.0078$ *	$0.0037$
	UCL	$0.0054$	$0.0349$	$0.0474$	$0.0281$	$0.0298$	$0.0125$	$0.0352$
JSE	LCL	$-0.0101$	$-0.0134$	$-0.0201$	$-0.0140$	$-0.0310$	$0.0046$	$-0.0009$
	Estimate	$-0.0029$	$0.0012$	$0.0047$	$0.0055$	$-0.0153$	$0.0072$ *	$0.0165$
	UCL	$0.0026$	$0.0225$	$0.0175$	$0.0266$	$0.0023$	$0.0103$	$0.0341$
RSE	LCL	$-0.0094$	$-0.0139$	$-0.0140$	$-0.0124$	$-0.0300$	$0.0048$	$-0.0032$
	Estimate	$-0.0022$	$-0.0011$	$0.0089$	$0.0055$	$-0.0137$	$0.0072$ *	$0.0116$
	UCL	$0.0028$	$0.0161$	$0.0187$	$0.0244$	$0.0032$	$0.0098$	$0.0291$

Note: LCL (Lower Class Limit), UCL (Upper Class Limit), (*) indicates significance at $\alpha =0.05$ level.

we fit the models to the milk dataset. The estimates of the coefficients show that only $X_6$ is significant across all methods.

Study 2: Real Estate Valuation Dataset

Real estate valuation data was collected in 2018 from Sindian District, New Taipei City, Taiwan, and recently published by [36]. The response variable in this study is the house prices per unit area, measured in 10,000 New Taiwan Dollars per Ping, where one Ping corresponds to $3.3{\mathrm{m}}^2$. Explanatory variables include transaction date ($X_1$), house age in years ($X_2$), distance to the nearest metro station in meters ($X_3$), number of convenience stores within walking distance ($X_4$), geographic coordinate latitude in degrees ($X_5$) and geographic coordinate longitude in degrees ($X_6$). The real estate valuation dataset has moderate multicollinearity (CN = 14.7723) and 21.5% outliers, positioning it among simulation Cases I and VII with moderate correlation ($\rho =0.50$) and $\pi =0.25$. Furthermore, the non-normal error distribution ($p<0.0001$) confirms strong estimator performance under conditions comparable to simulation Cases III and VI (heavy-tailed errors with multicollinearity).

3.5. Analysis for the Real Estate Valuation Dataset

3.5.1. Multicollinearity Detection for Real Estate Valuation Dataset

The real estate valuation dataset exhibits low VIF in all predictors; therefore, the CN of 14.7723 indicates that moderate multicollinearity exists among predictors, as shown in Table A2. Figure 25 shows that there are strong negative correlations between variables $X_3$, $X_4$, $X_5$ and $X_6$, particularly between $X_3$ and $X_6$ ($\rho =-0.81$).

3.5.2. Outlier Detection Using Cook’s Distance for the Real Estate Valuation Dataset

Figure 26 displays the Cook’s distance plot with a 0.01 threshold for the real estate valuation dataset. In comparison to the other points, observation 271 shows a significantly higher Cook’s D value, indicating a significant impact on the regression results. Data points 149, 221, and 313 show additional moderate influences.

3.5.3. Testing for Normality for Real Estate Valuation Dataset

Figure 27 shows clear deviations from normality due to skewed residual distributions. Therefore, the theoretical test confirms that the data are not normally distributed as the p-value is less than 0.05 ($p<0.0001$).

3.5.4. Model Fit and Evaluation for Real Estate Valuation Dataset

Table 4. Estimated bias and RMSE of the estimators under multicollinearity and varying outlier levels for the real estate valuation dataset.

Estimator	0% Outlier		15.3% Outlier		21.5% Outlier
	Bias	RMSE	Bias	RMSE	Bias	RMSE
OLSE	168.2679	54,504.9900	2732.8280	66,897.1350	7291.0330	59,865.6200
HME	7683.2360	67,391.2100	1937.2760	62,329.2885	2379.8690	47,601.0400
MME	12,588.7800	67,391.2100	2322.3150	70,523.8053	720.0933	50,175.4500
LMSE	18,806.2100	167,922.3000	9802.2880	188,380.7012	8417.1460	137,477.5000
LTSE	17,386.5200	116,365.6000	5397.6670	90,165.8498	6875.5230	71,319.8900
SE	21,667.5800	89,613.2900	12,931.5900	28,573.5115	4669.2530	20,823.2300
JSE	1446.8980	29,700.3800	3393.4700	13,195.9917	8122.0880	13,441.6400
RSE	9355.8480	40,048.4200	3861.2500	13,122.0683	3046.3670	10,501.8300

Note: Bold values indicate the lowest Bias or RMSE within each outlier category.

presents the estimated bias and RMSE of various estimators with increasing levels of outlier contamination and multicollinearity. The RSE demonstrates the most consistent performance in terms of RMSE across all contamination levels, particularly excelling at higher outlier percentages. Although JSE shows the lowest RMSE under clean data, its performance deteriorates significantly with more outliers. HME and OLSE maintain relatively low bias at all levels, but often at the cost of higher RMSE. Figure 28 was used to assess bias and RMSE. It was observed that HME, OLSE, and RSE maintain a consistently low bias across all outlier levels. The RSE and JSE are the clear top performers with low RMSE values. Despite the fact that RSE outperforms all other estimators in terms of RMSE, it exhibits a higher bias compared to some of them. This indicates that while RSE achieves the best overall predictive accuracy by effectively balancing bias and variance, it does not perform best in terms of bias alone. Therefore, as the percentage of outliers increases, the RSE consistently outperforms the other estimators, demonstrating strong robustness to both multicollinearity and outliers.

Table 5. Coefficient estimates with 95% confidence intervals for real estate valuation dataset.

Method		${\mathit{X}}_1$	${\mathit{X}}_2$	${\mathit{X}}_3$	${\mathit{X}}_4$	${\mathit{X}}_5$	${\mathit{X}}_6$
OLSE	LCL	874.29	−352.62	−6.72	678.32	110,419.70	−153,479.40
	Estimate	3754.65 *	−269.16 *	−5.52 *	1136.75 *	201,935.60 *	−103,319.30 *
	UCL	6728.76	−184.54	−4.43	1521.31	285,090.50	−59,497.40
HME	LCL	93.95	−367.64	−5.89	895.64	131,361.90	−117,826.80
	Estimate	2357.41 *	−298.01 *	−5.01 *	1198.02 *	208,792.30 *	−81,590.80 *
	UCL	4548.51	−230.50	−4.07	1532.01	293,689.80	−44,261.00
MME	LCL	−2705.04	−432.29	−10.94	363.69	115,550.90	−122,244.10
	Estimate	1203.37	−302.09 *	−5.22 *	1182.22 *	208,400.50 *	−62,398.90
	UCL	4668.72	−151.12	−2.80	2003.35	316,911.50	9656.81
LMSE	LCL	−5535.30	−538.38	−13.25	−44.80	−34,083.48	−199,199.20
	Estimate	486.03	−267.82 *	−5.83 *	1173.77	199,015.50	−48,595.50
	UCL	8262.26	−48.12	−1.61	2589.28	442,348.00	53,781.28
LTSE	LCL	−1966.84	−428.95	−11.33	545.01	76,641.88	−124,414.00
	Estimate	686.87	−297.89 *	−5.61 *	1194.42 *	198,832.30 *	−51,877.60 *
	UCL	4318.37	−182.37	−3.19	1891.99	330,466.30	−8131.57
SE	LCL	−3289.21	−467.78	−11.85	269.45	83,853.34	−146,443.70
	Estimate	445.64	−275.69 *	−5.86 *	1173.76 *	197,567.90 *	−47,627.10
	UCL	5866.78	−115.15	−2.58	2161.03	360,706.00	20,103.39
JSE	LCL	1046.77	−345.96	−6.69	655.91	113,213.80	−152,953.80
	Estimate	3700.06 *	−265.66 *	−5.46 *	1120.25 *	200,256.10 *	−102,075.30 *
	UCL	6705.15	−177.59	−4.40	1496.77	278,830.30	−60,578.70
RSE	LCL	308.02	−360.72	−5.94	886.87	121,104.70	−116,835.90
	Estimate	2396.59 *	−296.85 *	−5.01 *	1179.11 *	207,385.00 *	−81,953.30 *
	UCL	4438.46	−228.47	−4.15	1492.47	290,164.10	−44,765.30

Note: LCL (Lower Class Limit), UCL (Upper Class Limit), (*) indicates significance at $\alpha =0.05$ level.

shows the coefficient estimates with 95% confidence intervals, which provide formal statistical validation beyond RMSE comparisons. The study found that all predictor variables are statistically significant for RSE, JSE, HME and OLSE.

Study 3: Hawkins Bradu Kass Dataset

We utilized an artificial Hawkins Bradu Kass dataset constructed by [37] consisting of one dependent and three independent variables with 75 observations, of which 14 are outliers. The Hawkins–Bradu–Kass dataset has high degrees of multicollinearity (CN = 102.5294, $\rho >0.95$ among all predictors) and 18.67% outliers, which makes it a suitable empirical validation for simulation Case VII at extreme multicollinearity levels ($\rho =0.98$, $\pi =0.10$ to $0.25$). The non-normal residual distributions correspond to those used in simulation Cases III and IX.

3.6. Analysis for the Hawkins Bradu Kass Dataset

3.6.1. Multicollinearity Detection for Hawkins Bradu Kass Dataset

It should be noted that all predictors exhibit exceptionally high VIF values in the Hawkins Bradu Kass dataset, as shown in Table A2, and the observed CN of 102.5294 indicates that there is strong multicollinearity among the regressors. Figure 29 shows that all regressors are highly correlated ($\rho >0.95$).

3.6.2. Outlier Detection Using Cook’s Distance for Hawkins Bradu Kass Dataset

The Cook’s distance plot for the Hawkins Bradu Kass dataset, with a threshold value of 0.053, is shown in Figure 30. Cook’s D values in observations 12 and 14 significantly surpass the threshold, highlighting the fact that they are highly significant outliers. If not adequately addressed, these points might bias the parameter estimates and the overall performance of the model.

3.6.3. Testing for Normality for Hawkins Bradu Kass Dataset

Figure 31 shows clear deviations from normality due to skewed residual distributions. Therefore, the theoretical test confirms that the data are not normally distributed as the p-value is less than 0.05 ($p<0.0001$).

3.6.4. Model Fit and Evaluation for Hawkins Bradu Kass Dataset

As shown in

Table 6. Estimated bias and RMSE of the estimators under multicollinearity and varying outlier levels for the Hawkins–Bradu–Kass dataset.

Estimator	0% Outlier		18.67% Outlier
	Bias	RMSE	Bias	RMSE
OLSE	0.0004	0.0633	0.0672	0.3983
HME	0.0002	0.0683	0.0671	0.3721
MME	0.0005	0.0721	0.0147	0.1127
LMSE	0.0058	0.2818	0.0122	0.2698
LTSE	0.0031	0.1542	0.0238	0.2135
SE	0.0092	0.1704	0.0078	0.1494
JSE	0.0012	0.0593	0.0588	0.3669
RSE	0.0025	0.0620	0.0152	0.0847

Note: Bold values indicate the lowest Bias or RMSE within each outlier category.

and Figure 32, in the absence of outliers, the JSE achieved the lowest RMSE, while the HME had the lowest bias. Under 18.67% outlier contamination with multicollinearity, SE produced the lowest bias, and RSE achieved the lowest RMSE. MME also performed well under contamination, demonstrating both low bias and RMSE. In contrast, OLSE experienced a significant increase in both bias and RMSE when exposed to outliers. In general, robust estimators such as RSE and MME display greater stability and resilience under data contamination. In

Table 7. Coefficient estimates with 95% confidence intervals for the Hawkins–Bradu–Kass dataset.

Method		${\mathit{X}}_1$	${\mathit{X}}_2$	${\mathit{X}}_3$
OLSE	LCL	$-0.2438$	$-0.7004$	$0.0006$
	Estimate	$0.0406$	$-0.2679$	$0.3962$ *
	UCL	$0.3007$	$0.3679$	$0.7260$
HME	LCL	$-0.2977$	$-0.7001$	$0.0064$
	Estimate	$-0.0751$	$-0.1623$	$0.4021$ *
	UCL	$0.2196$	$0.1594$	$0.7244$
MME	LCL	$-0.2534$	$-0.0893$	$-0.2494$
	Estimate	$0.0183$	$0.0209$	$-0.0329$
	UCL	$0.2108$	$0.1308$	$0.3916$
LMSE	LCL	$-0.4282$	$-0.3817$	$-0.4724$
	Estimate	$0.0901$	$-0.0093$	$-0.0764$
	UCL	$0.5907$	$0.6711$	$0.5469$
LTSE	LCL	$-0.3576$	$-0.2258$	$-0.2670$
	Estimate	$0.0100$	$0.0141$	$0.0077$
	UCL	$0.2831$	$0.5694$	$0.5269$
SE	LCL	$-0.1541$	$-0.2276$	$-0.3194$
	Estimate	$0.0712$	$0.0005$	$-0.0802$
	UCL	$0.3936$	$0.1350$	$0.1223$
JSE	LCL	$-0.2190$	$-0.6939$	$-0.0057$
	Estimate	$0.0224$	$-0.1955$	$0.3012$
	UCL	$0.2675$	$0.2750$	$0.6923$
RSE	LCL	$-0.2530$	$-0.6822$	$0.0000$
	Estimate	$-0.0646$	$-0.1111$	$0.3135$ *
	UCL	$0.1614$	$0.1412$	$0.7138$

Note: LCL (Lower Class Limit), UCL (Upper Class Limit), (*) indicates significance at $\alpha =0.05$ level.

we fit the models to the Hawkins–Bradu–Kass dataset. The estimates of the coefficients show that only $X_3$ is significant under OLSE, HME, and RSE.

4. Discussion

The findings are consistent with the current literature on robust regression estimators. The superior performance of RSE’s in the presence of simultaneous multicollinearity and outlier contamination is attributed to its dual-component structure, that is, the HME component reduces the influence of outliers, while the JSE component maintains estimates through shrinkage, thus alleviating variance inflation [14]. The MME performed better when outliers only appeared in the y-direction and there was no multicollinearity. This pattern is consistent with the findings of [38,39], who found that MME was the most effective estimator for pure vertical contamination. The failure of OLSE and JSE in the presence of Cauchy-distributed errors corroborates the conclusions of [24,40], which indicate that heavy-tailed distributions lead to extreme observations dominating the sum of squared residuals. A basic trade-off in robust estimation is demonstrated by the failure of high-breakdown estimators (LMSE, LTSE, SE) under multicollinearity without outliers [21]. The general findings of this study support the idea that no single estimator is always the best. MME maintains advantages in environments dominated by y-direction outliers without multicollinearity [38,39,41], while RSE performs best when both multicollinearity and outliers are present [14]. These results highlight how crucial diagnostic analysis is before deciding on an estimation technique.

5. Conclusions

In this study, multicollinearity and outliers from both the leverage points and the y direction are evaluated in relation to the performance of RSE, which combines the shrinkage factors of ME and JSE. We evaluate the efficiency of RSE in an extensive MC simulation study with bias and RMSE criteria. Currently, the simulation study is conducted under a number of distributional circumstances (normally, t-distributions and Cauchy distributed errors), sample sizes, multicollinearity levels and outliers from leverage points and y direction independently.

When there are multicollinearity and outliers from leverage points and the y direction, the RSE outperforms classical OLSE, JSE, HME, MME, LMSE, LTSE, and SE. Additionally, when comparing the RSE to other existing robust estimators, we see that in the majority of the simulation scenarios evaluated, the RSE performs better. Based on the simulation study and its application to real datasets, when there is multicollinearity and outliers from the leverage points and the y direction, we discover that the RSE is the optimal estimator. Future research may extend this work by comparing the RSE through simulation studies using various distributions, such as the Weibull, Gamma, and Poisson distributions, across different sample sizes, levels of outlier contamination, and degrees of multicollinearity.