Bayesian Model Averaging and Regularized Regression as Methods for Data-Driven Model Exploration, with Practical Considerations

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The work presents an empiric comparison of three methods in regression modeling – Bayesian model averaging (BMA), LASSO regularization, and stepwise regression. The work considers on several examples their quality of fit by cross-validation, and compares the models by the root mean squared error (RMSE) – see Figures 1-4. The work concludes that in general the LASSO is better than the others, and BMA outperforms the stepwise regression. These results could be helpful in practical applications of regression modeling.  

However, the focus of the paper is not well defined. The work claims that it mainly serves to the needs in adequate models for psychological and educational researchers, which is stated from the first phrase in the abstract. Then the words “psychological” and “educational” are used two dozen times in the text. It is specially indicated that the work is oriented to the “psychological and educational researchers interested in an explanation model rather than prediction” (lines 145-6), and that it “will offer practical insights about utilizing the tools for psychological and educational researchers who want to perform model exploration instead of a priori hypothesis testing” (lines 193-4).

Actually, the main interest in using regression modeling in the psychological and educational field consists in evaluation of the individual variables’ contribution to explanation of variability of the outcome variable, that is related to the multicollinearity and other specific issues of multiple regression modeling in interpretation and meaningful explanation. These problems are well described, for instance, here are several sources:

Budescu D.V. and Azen R. (2004) Beyond global measures of relative importance: some insights from dominance analysis, Organizational Research Methods, 7, 341–350.

Johnson J.W. (2000) A heuristic method for estimating the relative weight of predictor variables in multiple regression. Multivariate Behavioral Research, 35, 1-19.

Johnson J.W. and Lebreton J. M. (2004), History and use of relative importance indices in organizational research, Organizational Research Methods, 7, 238–257.

Kruskal W.H. and Majors R. (1989) Concepts of relative importance in recent scientific literature, The American Statistician, 43, 2–6.

Lipovetsky S. and Conklin M. (2015) Predictor relative importance and matching regression parameters, Journal of Applied Statistics, 42, 5, 1017-1031.

 

Resuming, the paper should be elaborated more in the following aspects:

1.      To reduce too lengthy description of not very much exiting experiments on different datasets;

2.      To add consideration of problems of variables importance;

3.      To give the main formulae of the methods applied;

4.      The given Figures 1-4 can be substituted by a table of all these results;

5.      To write concisely and precisely on the main results.

Then, subject to the major revision detailed above, the paper can be reconsidered for publication.

Comments on the Quality of English Language

it can be improved.

Author Response

Comment 1. The work presents an empiric comparison of three methods in regression modeling – Bayesian model averaging (BMA), LASSO regularization, and stepwise regression. The work considers on several examples their quality of fit by cross-validation, and compares the models by the root mean squared error (RMSE) – see Figures 1-4. The work concludes that in general the LASSO is better than the others, and BMA outperforms the stepwise regression. These results could be helpful in practical applications of regression modeling. 

 

However, the focus of the paper is not well defined. The work claims that it mainly serves to the needs in adequate models for psychological and educational researchers, which is stated from the first phrase in the abstract. Then the words “psychological” and “educational” are used two dozen times in the text. It is specially indicated that the work is oriented to the “psychological and educational researchers interested in an explanation model rather than prediction” (lines 145-6), and that it “will offer practical insights about utilizing the tools for psychological and educational researchers who want to perform model exploration instead of a priori hypothesis testing” (lines 193-4).

 

Actually, the main interest in using regression modeling in the psychological and educational field consists in evaluation of the individual variables’ contribution to explanation of variability of the outcome variable, that is related to the multicollinearity and other specific issues of multiple regression modeling in interpretation and meaningful explanation. These problems are well described, for instance, here are several sources:

 

Budescu D.V. and Azen R. (2004) Beyond global measures of relative importance: some insights from dominance analysis, Organizational Research Methods, 7, 341–350.

 

Johnson J.W. (2000) A heuristic method for estimating the relative weight of predictor variables in multiple regression. Multivariate Behavioral Research, 35, 1-19.

 

Johnson J.W. and Lebreton J. M. (2004), History and use of relative importance indices in organizational research, Organizational Research Methods, 7, 238–257.

 

Kruskal W.H. and Majors R. (1989) Concepts of relative importance in recent scientific literature, The American Statistician, 43, 2–6.

 

Lipovetsky S. and Conklin M. (2015) Predictor relative importance and matching regression parameters, Journal of Applied Statistics, 42, 5, 1017-1031.

 

Response 1. Thank you very much for your suggestion regarding the consideration for relative importance analysis. I agree with you that psychological and educational researchers may also be interested in the extent to which each predictor contributes to predicting the outcome variable of interest, so discussions on relative important should be informative in this paper. In the introduction section, I briefly overviewed research on relative importance and how it can be connected to my work on data-driven model exploration:

 

(in lines 62-89)

Furthermore, we should also consider the examination of the relative importance of predictors as a purpose for regression-based analysis in psychological and educational studies [69]. Researchers in psychology and education are frequently interested in the extent to which each predictor contributes to predicting the outcome variable of interest [69]. Although conventional indicators, such as p-values and R2, have been employed to examine the relative importance of variables, they do not demonstrate the importance accurately [70]. I already explained previously the reason why p-values cannot be used for variable selection or evaluation [10, 11]. In the case of R2, due to the intercorrelation between predictors and multicollinearity, it is practically difficult to calculate the pure contribution of each predictor for evaluating its relative importance [69, 71, 72]. Hence, methods providing accurate information about the true relative importance of each variable after considering the intercorrelation and multicollinearity, such as dominance analysis and relative weight calculation [71, 73], might be beneficial for quantitative studies in psychology and education [69]. Although their primary goal is not exploring best candidate models and predictors unlike data-driven exploration methods, their outcomes, relative importance, can offer some insights into which predictors should be seriously considered in explaining the association between predictors and the outcome variable in regression models of interest.

Hence, it would be informative to briefly overview and discuss methods to examine relative importance within the context of the current paper focusing on data-driven model exploration. Although data-driven model exploration methods pursue a different goal, identifying the best model and predictors, compared with conventional hypothesis testing methods, particularly testing p-values, the data-driven methods can provide better information regarding the relative importance of variables [69, 70]. For instance, it would be possible to determine that predictors survived data-driven exploration may be more important in predicting an outcome variable than others [74]. I will discuss implications of data-driven methods within the context of examining relative importance in the discussion section after overviewing exploration methods and testing results.

 

Then, I added some discussions on this point in the discussion section while reviewing the findings from the current study. These are about how data-driven analysis can contribute to examining relative importance plus some limitations and future directions for improvement.

 

(lines 415-450)

I will also consider how data-driven methods are related to efforts to examine the relative importance of candidate predictors. As I mentioned in the introduction, data-driven analysis and analysis of relative importance (e.g., dominance analysis and relative weight analysis) pursue different goals [74]. However, I assumed that results from data-driven analysis may inform psychological and educational researchers who are interested in investigating the extent to which predictors of interest contribute to predicting the outcome variable of interest. At the least, the outcomes of data-driven exploration, the suggested best model and candidate predictors, can indirectly inform such researchers. We can expect that variables survived the exploration process are deemed to be more important than those did not.

There are some caveats regarding the potential usefulness of data-driven methods in relative importance examination. Obviously, stepwise regression is likely to be least informative. It only suggests one candidate model and is susceptible to model uncertainty [9, 23]. Also, it does not provide any additional information other than the suggested model. That said, researchers cannot get more direct information about the relative importance of each survived predictor. The same issue regarding the lack of direct importance indicators can be applied to regularized regression [75]. BMA might be the best method among three candidates for this purpose. In fact, Shou and Smithson [74] suggested that BMA can provide useful information about relative importance when it was compared with dominance analysis. Perhaps the posterior probabilities of candidate predictors, which are about the extent to which the predictors are supposed to be included in prediction models, can be used as more direct indicators for the relative importance of the predictors.

However, there are several issues that we need to consider while utilizing BMA for a method for relative importance analysis. Although BMA and dominance analysis behave similarly in most cases, as mentioned previously, they pursue two different goals at the first place, model averaging (or exploration) and testing relative importance [74], at the beginning. Thus, BMA’s relative superiority in examining relative importance should be considered a collateral benefit. Also, the consistency between BMA and dominance analysis becomes lower as multicollinearity increases [74]. If multicollinearity becomes a serious concern, then researchers may need to test the multicollinearity issue before conducting BMA when relative importance is also of their interest. Despite the issue when multicollinearity is severe, I would like to note that one positive aspect of BMA related to multicollinearity is that it can potentially (and perhaps partially) provide one way to address the issue [76]. Future studies should be done to investigate the benefits of BMA in relative importance analysis more accurately.

 

Comment 2. Resuming, the paper should be elaborated more in the following aspects:

 

1. To reduce too lengthy description of not very much exiting experiments on different datasets;

 

Response 2. I appreciate your suggestion about improving the descriptions for datasets. For brevity, I moved some further details about the datasets to the supplementary materials so that interested readers can refer to them while making the methods section in the main text shorter.

 

Comment 3. 2.      To add consideration of problems of variables importance;

 

Response 3. Thanks for your comment about variable importance. I addressed this issue by adding suggested citations, brief theoretical overview and discussion to the main text. Please refer to my response to your Comment 1 for further details.

 

Comment 4. 3.      To give the main formulae of the methods applied;

 

Response 4. Thank you for your suggestion to add main formulae. When I submitted my paper to Stats, the editor suggested to make it a case report dealing with applications rather than theoretical further details. So, I assumed that the potential readers will be more interested in methodological applications rather than theoretical foundations. However, I agree with you that some readers may want to learn more about such foundations. To be able to accommodate both readers interested in applications and theoretical details, I added brief descriptions for formulae and theories related to Bayesian Model Averaging and regularized regression in the supplementary materials.

 

Comment 5. 4.      The given Figures 1-4 can be substituted by a table of all these results;

 

Response 5. Thanks for your suggestion to improve the visibility of the findings. I moved the figures to the supplementary materials for brevity.

 

Comment 6. 5.      To write concisely and precisely on the main results.

 

Then, subject to the major revision detailed above, the paper can be reconsidered for publication.

 

Response 6. I appreciate your suggestion to improve the language in the results section. I edited the results section briefly to improve its readability by removing redundant expressions and correcting some errors.

Reviewer 2 Report

Comments and Suggestions for Authors

In the opinion of reviewer, this manuscript obeys the quality expected of journal STATs. After the proposed suggestions/corrections be implemented, the reviewer considers the paper ready for publication.

Comments for author File: Comments.pdf

Comments on the Quality of English Language

-

Author Response

Comment 1. General Comments:

The manuscript is generally well written but along the text, instead refer I did

something, shall refer the author did something. Never refer the first person in

singular. Please, correct this issue along the text, including abstract.

For example, in abstract, where is “I intend to discuss practical considerations

regarding data-driven methods for end-user researchers without sufficient

expertise in quantitative methods. I tested three data-driven methods, i.e.,

Bayesian Model Averaging, LASSO as a form of regularized regression, and

stepwise regression, with datasets in psychology and education.” can be “the

author intends to discuss practical considerations regarding data-driven

methods for end-user researchers without sufficient expertise in quantitative

methods. Three data-driven methods, i.e., Bayesian Model Averaging, LASSO

as a form of regularized regression, and stepwise regression were tested, with

datasets in psychology and education.”

This manuscript is very interesting and a good contribution. The authors

performs an evaluation of data-driven methods, discussing practical

considerations regarding data-driven methods for end-user researchers .

This paper is well organized. The experimental part is detailed and justified.

Three datasets collected by studies in psychology were acquired from open

data repositories and used to examine the performance of the three data-driven

model exploration methods, i.e., stepwise regression, BMA, and LASSO.

The results are explained and displayed. The discussion is extense, very

interesting, with illustrative comparing the obtained results with other studies.

Depending on the purpose of the researcher and the criteria to obey, each one

of the model-driven can be chosen.

No claims about the contents of almost all sections of the work, but the

authors shall take attention to:

1- Introduction section is too long. Please, split it in two sections: the

general state of art until line 60, ending with a paragraph with the structure

of the manuscript. The remaining part corresponds to a Preliminaries (or

Background) Section.

 

Response 1. I sincerely appreciate your positive evaluation on my paper. Also, thank you for your suggestion to re-structuralize the introduction section. Following your suggestion, I created multiple subsections under the intro: 1.1. Background and 1.2. Current Study

 

Comment 2. 2- Still in the Introduction section, in lines 54, 55 it is written “Thus, it

cannot accurately predict the dependent variable outside the used dataset”.

In this point shall be clarified that a regression model is valid (statistically)

in the interval defined by the values that regressors dataset defines.

Outside these intervals, it is perform an extrapolation without statistically

significance guarantee of the obtained estimates.

 

Response 2. Thanks for your suggestion to further clarify the point about cross-validity. I added the additional description as per your suggestion:

 

(in lines 55-58)

More specifically, in such a case, the estimated regression model might be statistically valid in the interval defined by the values and data in the employed dataset. However, outside the intervals, it may perform an extrapolation without statistically significant guarantee of the estimates.

 

Comment 3. 3-A conclusion section shall be included. The author shall finish this section

with a paragraph about future work . Maybe the lines 439-453 fill

this gap.

 

Response 3. I appreciate your suggestion to add the conclusion section. At the end of the main text, I created the new section as per your suggestion:

 

(in lines 467-480)

5. Concluding Remarks

In this paper, I tried to suggest several practical guidelines about how to employ data-driven model exploration methods by examining widely available methods, stepwise regression, BMA, and LASSO. Researchers in psychology and education may refer to the suggestions to conduct more accurate data-driven analysis depending on the nature of their data to be examined. For additional information for practice, I suggested that future works should be done in the field. Perhaps, as I mentioned while discussing limitations, data collected from different groups may need to be further tested with data-driven methods to inform cross-cultural researchers. Furthermore, researchers may consider adjusting options for the R functionalities, and test and compare outcomes to examine how to maximize the performance of each method. With the findings from the proposed future studies, practical researchers will obtain more useful insights into how to utilize the data-driven model exploration methods in an optimal way based on their research purposes and situational factors.

 

Comment 4. 4-A detail in line 405, where is “abovementioned” shall be “above mentioned”.

In the opinion of reviewer, this manuscript obeys the quality expected of journal

STATs. After the proposed suggestions/corrections be implemented, the

reviewer accept the paper

 

Response 4. Thanks for your suggestion for the language improvement. I modified the word in the revised manuscript as per your suggestion.