Predictors of Early College Success in the U.S.: An Initial Examination of Test-Optional Policies

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

See reviewer comments uploaded in PDF file

Comments for author File: Comments.pdf

Author Response

see attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This study examines whether the test-optional policy predicts early college success, particularly in terms of first-year GPA and retention. I have enjoyed the study and believe that the present paper can be improved both theoretically and empirically by incorporating literature that better supports the authors' findings, conducting supplemental robustness checks, and clearly outlining the limitations of the analysis.  

Here are my questions and comments:

1. I was surprised to see the correlation among the independent variables (Table 3) because I expected to find stronger correlations. I recommend that the authors calculate VIFs to provide supporting evidence and address concerns about multicollinearity in the regression analysis.
2. I recommend avoiding the term 'five-factor model'. Readers may be confused about whether authors employed factor analyses, such as EFA or CFA, since this terminology is more commonly used in these types of analyses.
3. I noticed the word "impact" appears multiple times throughout the manuscript. I also recommend avoiding this term, as the methods in this study do not convincingly establish causality. Instead, consider using terms like "associations" or "correlations".  
4. The authors found no statistically significant relationship between the Test-Optional variable and early college success variables after accounting for other covariates. I wonder whether the authors did not find any statistical correlation for Test-Optional in the simple regression model. In other words, did authors still find no correlation when the Test-Optional variable was solely included in the model? If this variable is a significant factor predicting the outcome variable but loses statistical significance as other covariates are included in the model, it suggests that the effect of Test-Optional on the outcome is being attributed to other covariates. In other words, the Test-Optional variable's influence is captured by other factors. (The authors indeed discussed potential correlations among the independent variables in the discussion section.) This raises a fundamental question about which variable is more important in this research!  
5. Additionally, the test-optional variable has a selection issue. Students voluntarily opted into the test-optional policy. Various factors may influence the decision regarding the policy's use, including peer effects, school counselor engagement, and parental involvement. If these unobservable factors are confounded with the test-optional variable, it could lead to attenuation issues, resulting in a lack of statistical significance. It would be great if the authors could address this selection problem; however, if the authors cannot resolve this issue, it should be thoroughly discussed in the limitations section, informing readers of this limitation when interpreting the findings. Depending on the severity of the selection problem, the correlation for the test-optional variable might also be statistically significant.  
6. If the authors wish to tackle these selection issues, authors might consider using propensity score or instrumental variable approaches.  
7. There is a misinterpretation in line 282 (under the Discussion section). The sentence starting with "students who did submit a test score for admissions had a lower high school GPA" should read "higher".  
8. Finally, I would like to see some interaction effects between the test-optional variable and other covariates, such as HS GPA and student of color. The authors may identify particular patterns when adding interaction terms or quadratic forms of variables into the model.  
9. I strongly suggest that the authors rewrite the discussion section based on my comments above. 

Comments on the Quality of English Language

The manuscript is well written. 

Author Response

see attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

I have no additional comments.