Adaptive Penalized Regression for High-Efficiency Estimation in Correlated Predictor Settings: A Data-Driven Shrinkage Approach

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Major Revision Report
Title: Adaptive Penalized Regression for High-Efficiency Estimation in Correlated Predictor Settings: A Data-Driven Shrinkage Approach
General Comment:
The proposed methodology is interesting and addresses an important challenge in penalized regression models, especially under high collinearity. The paper is well-structured and generally well-written. However, several critical aspects require further clarification, elaboration, and validation before the manuscript can be considered for publication. Therefore, I recommend major revision. My detailed comments are as follows:
Major Comments
Template Compliance: The manuscript must be prepared using the official template of the target journal. Please ensure that all formatting (title, abstract, headings, references, figures, tables, etc.) adheres to the journal’s guidelines.
Verification of Mathematical Formulas: All mathematical expressions, especially those defining the proposed estimator and associated penalty terms, should be carefully checked for accuracy and clarity. Typographical or derivation-related errors must be avoided, as they may compromise reproducibility.
Asymptotic Properties of the Estimator: The manuscript does not discuss the asymptotic distribution or normality of the proposed adaptive estimator. While theoretical derivation may be complex, at the very least, a numerical investigation of the estimator’s asymptotic behavior should be included. This would enhance the understanding of its statistical properties and inferential utility.
Bias of the Estimator: In the simulation study, bias metrics (in addition to MSE) should be reported in both tabular and graphical formats. Presenting bias alongside MSE will provide a more complete picture of the estimator’s performance, especially in assessing the trade-off between variance and bias introduced by penalization.
Reproducibility and Code Availability: The authors must provide all computational codes used in simulations, real data analysis, and graphical outputs. This is crucial for ensuring transparency and allowing independent verification of results.
Standard Errors in Real Data Application: In the application section, the paper currently reports point estimates only. Please include standard errors (or confidence intervals) of the estimated coefficients to allow assessment of the estimator’s precision and inferential robustness.
Conclusion
The manuscript introduces a potentially valuable contribution to the penalized regression literature. However, the concerns listed above are substantial and must be thoroughly addressed. Once these revisions are incorporated, the paper will be in a stronger position for consideration.

Comments on the Quality of English Language

Major Revision Report
Title: Adaptive Penalized Regression for High-Efficiency Estimation in Correlated Predictor Settings: A Data-Driven Shrinkage Approach
General Comment:
The proposed methodology is interesting and addresses an important challenge in penalized regression models, especially under high collinearity. The paper is well-structured and generally well-written. However, several critical aspects require further clarification, elaboration, and validation before the manuscript can be considered for publication. Therefore, I recommend major revision. My detailed comments are as follows:
Major Comments
Template Compliance: The manuscript must be prepared using the official template of the target journal. Please ensure that all formatting (title, abstract, headings, references, figures, tables, etc.) adheres to the journal’s guidelines.
Verification of Mathematical Formulas: All mathematical expressions, especially those defining the proposed estimator and associated penalty terms, should be carefully checked for accuracy and clarity. Typographical or derivation-related errors must be avoided, as they may compromise reproducibility.
Asymptotic Properties of the Estimator: The manuscript does not discuss the asymptotic distribution or normality of the proposed adaptive estimator. While theoretical derivation may be complex, at the very least, a numerical investigation of the estimator’s asymptotic behavior should be included. This would enhance the understanding of its statistical properties and inferential utility.
Bias of the Estimator: In the simulation study, bias metrics (in addition to MSE) should be reported in both tabular and graphical formats. Presenting bias alongside MSE will provide a more complete picture of the estimator’s performance, especially in assessing the trade-off between variance and bias introduced by penalization.
Reproducibility and Code Availability: The authors must provide all computational codes used in simulations, real data analysis, and graphical outputs. This is crucial for ensuring transparency and allowing independent verification of results.
Standard Errors in Real Data Application: In the application section, the paper currently reports point estimates only. Please include standard errors (or confidence intervals) of the estimated coefficients to allow assessment of the estimator’s precision and inferential robustness.
Conclusion
The manuscript introduces a potentially valuable contribution to the penalized regression literature. However, the concerns listed above are substantial and must be thoroughly addressed. Once these revisions are incorporated, the paper will be in a stronger position for consideration.

Author Response

Comment to Authors

Template Compliance: The manuscript must be prepared using the official template of the target journal. Please ensure that all formatting (title, abstract, headings, references, figures, tables, etc.) adheres to the journal’s guidelines

Answer to Reviewer-I

We sincerely thank the reviewer for highlighting this point. The manuscript has now been thoroughly revised to comply with the official template of the Mathematics journal. All sections—including the title page, abstract, keywords, headings, references, figures, and tables have been reformatted in accordance with the journal’s submission guidelines. We have carefully cross-checked each component to ensure complete adherence 

Comment to Authors

Verification of Mathematical Formulas: All mathematical expressions, especially those defining the proposed estimator and associated penalty terms, should be carefully checked for accuracy and clarity. Typographical or derivation-related errors must be avoided, as they may compromise reproducibility.

Answer to Reviewer-I

All the mathematical expressions including those defining new terms were revalidated.

Comment to Authors

Asymptotic Properties of the Estimator: The manuscript does not discuss the asymptotic distribution or normality of the proposed adaptive estimator. While theoretical derivation may be complex, at the very least, a numerical investigation of the estimator’s asymptotic behavior should be included. This would enhance the understanding of its statistical properties and inferential utility.

Answer to Reviewer-I

Needful done, discussion provided on page 8.

Comment to Authors

Bias of the Estimator: In the simulation study, bias metrics (in addition to MSE) should be reported in both tabular and graphical formats. Presenting bias alongside MSE will provide a more complete picture of the estimator’s performance, especially in assessing the trade-off between variance and bias introduced by penalization.

Answer to Reviewer-I

The bias of the existing and proposed estimators for real-life datasets is presented in Tables 4 and 5, respectively. Furthermore, future studies will incorporate this bias into simulation analysis as well.

Comment to Authors

Reproducibility and Code Availability: The authors must provide all computational codes used in simulations, real data analysis, and graphical outputs. This is crucial for ensuring transparency and allowing independent verification of results.

Answer to Reviewer-I

All the codes are provided as Appendix-I

Reviewer 2 Report

Comments and Suggestions for Authors

Please see the attached file for my comments.

Comments for author File: Comments.pdf

Author Response

Comment to Authors

The authors did a quite extensive review on the existing estimators in terms of the regularization parameters in ridge regression, but they only use a short paragraph describing the proposed estimators. I am not convinced on an intuitive level why the proposed estimator will be beneficial. Although the authors demonstrated that the estimator is superior using simulation studies, the authors should mention at least in an intuitive level, why the proposed estimator is better compared to the existing ones. Also, the novelty of the proposed estimator should also be clearly mentioned

Answer to Reviewer-2

The logic behind proposed estimators is explained on Page 7 & 8 in the manuscript. 

Comment to Authors

When reading section 2.2, I am confused on which estimator is the proposed AATPR estimator, the one in (20) or the one in (21)? The demonstration should definitely be improved in this section

Answer to Reviewer-2

Needful done and clarification provided on Page 8 in the manuscript. 

Comment to Authors

Even though the authors mentioned the model dimensionality, throughout the simulation, only low-dimensional settings were considered. Since ridge regression is popular in the high-dimensional setting, more simulation studies need to be conducted.

Answer to Reviewer-2

The instructions are noted for future studies. 

Comment to Authors

Many of the notations in the paper are neither defined nor consistent. Below are some examples I detected.

1. What is ˆα_max in equation (11)? –
2. What is cond(X) in equation (18)? Is it the condition number? In addition, what is X?
3. What are λ_max and λ_min in equation (20)?
4. The subindices in the notation m_ji are not consistent in equation (23) and (24).

Answer to Reviewer-2

1. Needful done on page 5.
2. Needful done on page 7.
3. Needful done on page 8.
4. Needful done on page 9.

Comment to Authors

Some minor issues: –

1. In the definition of Υ on page 2, the dimension should be p + 1 instead of p.
2. Please specify whether λ1, λ2, . . . , λp are arranged in a decreasing order or an increasing order.
3. On page 5, the authors referenced “Eq. (2-4)”, which does not exist in the paper.

Answer to Reviewer-2

1. Since we have assumed the intercept term as zero, hence the dimension is p.
2. Needful done on page 5.
3. Corrected. 

Reviewer 3 Report

Comments and Suggestions for Authors

This paper presents a novel Auto-Adjusted Two-Parameter Ridge (AATPR) estimator, which dynamically adjusts its penalty structure based on key data features, including predictor collinearity, error variance, and model dimensionality. The main goal was to address the absence of universal robustness in existing ridge estimators, a crucial issue since multicollinearity is still a common challenge in applied regression modelling. The study's key strength is its thorough empirical evaluation. The extensive Monte Carlo simulation setup, covering various levels of multicollinearity (ρ), error variance (σ²), sample size (n), and number of predictors (p), offers a rigorous testing environment. The inclusion of two real-world datasets exhibiting severe multicollinearity further boosts its practical relevance. Results consistently show AATPR's superiority in reducing MSE, especially under high collinearity, marking a significant advancement.

However, several areas require clarification, expansion, or correction to enhance the paper's scientific rigour, clarity, and impact:

1. Insufficient Justification for Proposed Estimator Form: The derivation and intuitive justification for the specific functional form of the proposed AATPR estimator (Eqs. 20 & 21: k_t(k,q) = p * (λ_t^p * |α_t| / (1 + λ_Max/λ_min))^p and its median version) are lacking. While adopting the penalty selection framework from [26] is mentioned, the rationale behind this specific structure (e.g., why exponent p, why the ratio λ_Max/λ_min in the denominator, why the median) needs a stronger theoretical or heuristic motivation beyond the simulation results.
2. Lack of Theoretical Comparison Context: While stating that theoretical comparison is "intractable" (Page 8) justifies the simulation approach, it would be valuable to briefly reference any known theoretical results or bounds related to the performance of ridge-type estimators under the varying conditions studied, even if a direct analytical comparison isn't feasible. This situates the empirical work within the broader theoretical landscape.
3. Inadequate Justification for Simulation Parameter Choices: The rationale behind selecting the specific levels for the varying factors (ρ = 0.90, 0.95, 0.99, 0.999; σ² = 0.5, 1, 5, 10; sample sizes implied as n=20,50,100; p=4,10) is not provided. While these ranges cover common scenarios, explaining why these particular values were chosen (e.g., representing mild, moderate, high, extreme collinearity) would strengthen the design. It was stated that "three levels of sample size and two levels of number predictors" but all p and n combinations were not specified until the tables.
4. Undefined Terms in Competitor Estimators: The definitions of the CARE estimators (Eqs. 17-19) include the term cond(X) without definition. While readers familiar with multicollinearity diagnostics will recognize this as the condition number (likely λ_Max / λ_min), it should be explicitly defined for clarity within the paper. Similarly, the notation ∂² in the Toker & Kaciranlar equations (Eqs. 15 & 16) is undefined (presumably σ̂²).
5. Limited Discussion of Real Data Limitations: The Pakistan GDP Growth dataset exhibits extreme multicollinearity (VIFs > 100,000, condition number > 5 million). While this tests the method under duress, it also raises questions about the practical utility and interpretability of any regression model built on such data. A brief discussion acknowledging this limitation and the inherent challenges of interpretation when predictors are so highly confounded would be prudent. The authors should verify that the dependent variable is not a direct cumulant of the predictors, which might have caused the high VIFs. I will advise to drop this data if this doubt cannot be clarified.
6. Over-Reliance on MSE: While MSE is a valid and common criterion, focusing exclusively on it provides a limited view of estimator performance. Supplementing MSE with metrics like Mean Absolute Error (MAE) or predictive performance measures (e.g., out-of-sample R²) on the real data would offer a more comprehensive assessment, particularly regarding predictive robustness and potential sensitivity to outliers.

Author Response

Comment to Authors

The derivation and intuitive justification for the specific functional form of the proposed AATPR estimator (Eqs. 20 & 21: k_t(k,q) = p * (λ_t^p * |α_t| / (1 + λ_Max/λ_min))^p and its median version) are lacking. While adopting the penalty selection framework from [26] is mentioned, the rationale behind this specific structure (e.g., why exponent p, why the ratio λ_Max/λ_min in the denominator, why the median) needs a stronger theoretical or heuristic motivation beyond the simulation results.

Answer to Reviewer-3

The intuitive justification for proposed estimators are explained on Page 7 & 8 in the manuscript. 

Comment to Authors

Lack of Theoretical Comparison Context: While stating that theoretical comparison is "intractable" (Page 8) justifies the simulation approach, it would be valuable to briefly reference any known theoretical results or bounds related to the performance of ridge-type estimators under the varying conditions studied, even if a direct analytical comparison isn't feasible. This situates the empirical work within the broader theoretical landscape

Answer to Reviewer-3

It is correct that we have emphasized regarding theoretical comparison of ridge type estimators is intractable due to the complexity of deriving the exact distribution of such type of estimators, it is important to situate our work within the broader theoretical framework. In fact, there do exist some known results and bounds related to ridge-type estimators. For instance, it is well established that ridge regression reduces variance at the expense of introducing bias, which often results in lower mean squared error (MSE) compared to the ordinary least squares estimator, particularly in the presence of multicollinearity (see Hoerl & Kennard, 1970; Hoerl, Kennard & Baldwin, 1975). Moreover, several asymptotic properties of the ridge estimator have been studied: under mild regularity conditions, ridge estimators are consistent and asymptotically normal (Khalaf & Shukur, 2005; Saleh & Kibria, 1993).

 

Comment to Authors

Undefined Terms in Competitor Estimators: The definitions of the CARE estimators (Eqs. 17-19) include the term cond(X) without definition. While readers familiar with multicollinearity diagnostics will recognize this as the condition number (likely λ_Max / λ_min), it should be explicitly defined for clarity within the paper. Similarly, the notation ∂² in the Toker & Kaciranlar equations (Eqs. 15 & 16) is undefined (presumably σ̂²).u

Answer to Reviewer-3

Needful done on page 7 & 8.

Comment to Authors

Over-Reliance on MSE: While MSE is a valid and common criterion, focusing exclusively on it provides a limited view of estimator performance. Supplementing MSE with metrics like Mean Absolute Error (MAE) or predictive performance measures (e.g., out-of-sample R²) on the real data would offer a more comprehensive assessment, particularly regarding predictive robustness and potential sensitivity to outliers.

Answer to Reviewer-3

We thank the reviewer for this valuable suggestion. While MSE has been our primary evaluation criterion due to its wide usage in the existing literature, we agree that incorporating additional measures such as Mean Absolute Error (MAE), Ridge Variance Inflation Factor (RVIF), and predictive performance indicators would provide a more comprehensive assessment of the estimators. We have duly noted these recommendations and will incorporate them in our future work.

Reviewer 4 Report

Comments and Suggestions for Authors

The structure of the paper is reasonable and the topics discussed have some significance, but there are still several issues that need to be modified before acceptance.

(1)Suggest the authors to add a comparison with methods other than ridge regression and its improved version in the paper, which can further enhance the advantages of the proposed method.

(2)There are still many minor issues in the paper that need to be thoroughly read by the authors before making proper revisions. For example, the subscript in equation (9) is missing ')'; Equation (24) is missing a comma; The σ in the figures 1-2 should be σ²; The title of the  figure should be below the figure.

Author Response

Comment to Authors

Suggest the authors to add a comparison with methods other than ridge regression and its improved version in the paper, which can further enhance the advantages of the proposed method.

Answer to Reviewer-4

We thank the reviewer for this valuable suggestion. We agree that including comparisons with alternative shrinkage or biased estimation methods beyond ridge regression and its variants; such as the Liu estimator, James–Stein type estimators, Lasso, or Elastic Net, would further contextualize the advantages of the proposed method. Due to space and scope constraints, we have focused on ridge-type estimators in this study; however, we acknowledge this as an important direction for future work. We will highlight this point in the revised manuscript, noting that extending the comparison framework to a wider class of estimators can provide a more comprehensive assessment of the proposed method’s performance.

 

 

Comment to Authors

There are still many minor issues in the paper that need to be thoroughly read by the authors before making proper revisions. For example, the subscript in equation (9) is missing ')'; Equation (24) is missing a comma; The σ in the figures 1-2 should be σ²; The title of the figure should be below the figure

Answer to Reviewer-4

Needful done. In Figures 1-3, it is σ, if we are going to change it to σ² then its value will also be changed.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The revised manuscript is satisfactory. I think it is auitable for publication in the current form.

Author Response

We extend our heartfelt gratitude for your meticulous review and valuable feedback. Your comments and expert recommendations were instrumental in improving our manuscript and enhancing its clarity.

Reviewer 2 Report

Comments and Suggestions for Authors

The authors have addressed part of my previous comments, but not all of them. In particular, the authors added some description on pages 7 and 8. However, I am still not convinced on an intuitive level why the proposed estimators work. Why does the U-shaped relationship between MSE and k lead to the form of the estimator in (7) and (8)?

Additionally, some of the notations are still not specified in the revised manuscript.

1. What is cond(M'M) in equation (18)?
2. Please specify whether λ1, λ2, . . . , λp are arranged in a decreasing order or an increasing order.

Author Response

We extend our heartfelt gratitude for your meticulous review and valuable feedback. Your comments and expert recommendations were instrumental in improving our manuscript and enhancing its clarity.

Comment to Authors

The authors have addressed part of my previous comments, but not all of them. In particular, the authors added some description on pages 7 and 8. However, I am still not convinced on an intuitive level why the proposed estimators work. Why does the U-shaped relationship between MSE and k lead to the form of the estimator in (7) and (8)?

Answer to Reviewer-3

Thank you for this insightful comment and for engaging deeply with our work. Following is our explanation: It is well-established that the MSE of a ridge estimator is strongly influenced by multicollinearity and  model dimensionality. The degree of multicollinearity is precisely diagnosed using eigenvalues. Metrics like the Condition Number   and Condition Index  offer a robust framework for its detection. In the absence of multicollinearity, eigenvalues are balanced and moderate. However, under multicollinearity, this balance is disrupted, inflating the largest eigenvalue while others shrink toward zero. Moreover, multicollinearity may adversely impact the regression coefficients ( ) by significantly inflating their values with incorrect signs. Our proposed estimator synthesizes these insights, by formulating the ridge penalty k as a function of model dimensionality, regression coefficients and the condition indices of the data to achieve a unique balance that mitigates both overfitting and underfitting, ensuring robust performance. The numerator addresses overfitting through a function of the eigenvalues, regression coefficients, and data dimensionality. Simultaneously, the denominator safeguards against underfitting induced by an overly aggressive ridge penalty parameter.

The above explanation is incorporated in the manuscript on page 8.

Comment to Authors

What is cond(M'M) in equation (18)?

Answer to Reviewer-3

  is the index number of   matrix, the same is incorporated in the manuscript on page 7.

Comment to Authors

Please specify whether λ1, λ2, . . . , λp are arranged in a decreasing order or an increasing order.

Answer to Reviewer-3

The order of λ1, λ2, . . . , λp is decreasing. Th same has been incorporated in manuscript on page 5.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

All my comments have been sufficiently addressed. I recommend accepting the paper in its current form.

Author Response

We extend our heartfelt gratitude for your meticulous review and valuable feedback. Your comments and expert recommendations were instrumental in improving our manuscript and enhancing its clarity.

Round 3

Reviewer 2 Report

Comments and Suggestions for Authors

The authors have addressed my previous comments and I do not have further comments.

Author Response

We would like to once again express our sincere gratitude for your insightful and constructive comments.