Traits versus Grades—The Incremental Predictive Power of Positive Psychological Factors over Pre-Enrollment Achievement Measures on Academic Performance

Round 1

Reviewer 1 Report

The aim of this paper was to estimate a wide range of psychological factors and their predictive power on university performance over pre-enrollment achievement. To fulfill this goal authors tested 137 university students using a wide range of instruments and used machine learning methods to estimate the predictive power of psychological factors. The main conclusion of the manuscript is that psychological factors and prior academic achievement have better predictive power for university achievement comparing to academic achievement only.

Although this study has some interesting results, there exist several limitations that restricted my ability to recommend this paper to be published. Detailed comments are listed below:

• Introduction and research aims. The main concern is that this study has explorative aims and does not have a clear focus and hypothesis. It looks like the authors have collected plenty of data on a sample of students and have decided to analyze it to identify the significant predictors. If you have many variables, it is not surprising that some variables may have a significant effect. This strategy looks like as data snooping or data fishing. It would be better to focus on one-two clear hypotheses and test them more deeply.
• It is unclear the novelty of the obtained findings. The authors demonstrated that psychological factors have additive predictive power. Whether was it unknown before this study? What is the additive value of this research? The authors should identify study novelty and strengthens.
• The author considered too many variables and it is unclear criteria for their selection. These are psychological factors, but this is a too wide a framework. Due to this, it is hard to follow the authors’ logic. The introduction presented a short description of different psychological constructs. What does integrated these factors except they are psychological and associated with achievement? Plenty of other psychological factors associated with academic performance, e.g. self-efficacy or self-concept. It is impossible to take into account all factors. What are the criteria for factors selection or rejection?
• 1. Sample size. I think that the sample size is too small for this number of variables.
  2. Psychometric properties of instruments. The description of the instruments is too short. There is no description of psychometric properties of instruments (e.g. reliability, measurement error, etc.)
  3. There is no detailed description of the statistical approach. The authors mentioned that they used some algorithms of machine learning. Why these algorithms were chosen? What are the advantages of machine learning in comparison of other methods? Does the sample size is enough for machine learning? I think it is worth describing the main steps if this algorithm in order to readers can understand the main idea and procedure. Besides, it is unclear why the authors decided to convert the outcome variable to a binary variable and what advantage of this approach.
• Descriptive statistics would be helpful to get an idea regarding used variables. I did not understand the aim of identifying pairs, triplets, and quadruplets of factors. What are the advantages of this step?
• Weaknesses of discussion occur from the introduction and the lack of more focused research questions and hypotheses. It is unclear the theoretical value of the obtained results. The practical implications might be also discussed in more detail.

Author Response

Dear Reviewer,

We really appreciate your time and your work in reading our manuscript and thank you for your really helpful comments. We feel that all the issues that you raised are legitimate and we worked a lot to handle all your concerns. We believe that we managed to improve the paper substantially according to your very helpful comments.

First of all, we made the research aims clearer and reorganized the introduction. Hopefully, now we have a more focused aim, and this will solve many of your comments and questions. The exact aim of our research is to find what factors (and what groups of factors) have the strongest incremental predictive validity over pre-enrollment achievement measures. By identifying the most important factors, we can make recommendations for higher education institutions what psychological factors should be measured at the time of enrollment to build a model with strong predictive power. We used logistic regression with 10-fold cross-validation to obtain reliable results.

We also try hard to highlight what the novelty of the paper is. It was known before that psychological factors have some incremental predictive power, on the other, in this paper we identified what factors should be really measured if a higher education institute aims to build a more effective predictive model to identify at-risk students. We also put under the microscope positive psychological factors, that are rarely investigated in the literature.

About variable selection: We added a new paragraph detailing what selection criteria we used to select the additional psychological variables. Namely, it describes emotion-related competencies in a positive way, it can be easily developed even in a university context and it has already research evidence that the phenomenon is linked with academic success (see rows 80-83).

We increased the sample size substantially by considering not only the student cohort enrolled in 2018 but also the class that enrolled in 2019.

About psychometric properties: Thank you for pointing it out, we integrated all the important psychometric attributes of the used measurement tools and also detailed the psychometric properties and the connected descriptive statistics in Table 1 (see row 221 for the table with our descriptives and rows 170-204 for the available psychometric attributes of the used test materials).

We changed from the used machine learning algorithm to a more traditional approach: 10-fold-cross-validated logistic regression. We believe that our results are more reliable using a logistic regression model. We converted the outcome variable to binary since we aim for a predictive model that can identify at-risk students. A student with a first-year GPA below the threshold (3.0) that we used can be considered at-risk since students lose their state-funded scholarship below this threshold.

We studied the predictive power of pairs, triplets, quadruplets, etc. because that way we can see which and how many factors higher education institutes should measure to increase the predictive power of a risk detection model. If one aims to build a model with 2/3/4... factors what factors should be measures.

Both in the introduction and in the discussion, we put more emphasis on the practical implications of the research project and the paper, namely our research reveals what psychological factors should be measured to build a better model to identify at-risk students.

Best regards,

The Authors

Reviewer 2 Report

I congratulate you on your research.

Indeed, the findings cannot be generalized to other contexts, as such research is limited to a specific sample of students.

As a consequence of the findings obtained and as you say you need to continue researching to explore the relationship between soft psychological skills and academic performance.

Author Response

Dear Reviewer,

We really appreciate your time and your work in reading our manuscript and thank you for the kind words and support. We are aware of the fact that we have a homogenous sample. We totally agree with your suggestion and at the moment, we are working on collecting more data to explore the relationship between psychological skills and academic performance.

Best regards!

The Authors

Reviewer 3 Report

Good Paper

Author Response

Dear Reviewer,

We really appreciate your time and your work in reading our manuscript.  We also thank you for the positive feedback.

Best regards,

The Authors

Reviewer 4 Report

I appreciate the opportunity to review this manuscript.

Congratulation to the authors for the nice work conducted. I really enjoy reading this document. The document is well structured and consistent with the object of research.

Introduction: the author/s provides data on the importance of expansion in this field of research.

Methodology: this section explains how the study was carried out and details the research design and measures used.

Results: this section explains the results obtained in an orderly and concise form, being easy to understand and consistent with what was stated in the theoretical framework.

Discussion: An analysis of the results is made and it is related to other studies so that the importance of the data obtained can be seen.

The author/s have made a good job, although I contribute here some suggestions for the improvement of the quality of the document:

• In the document: grammar check by a native speaker.
• In the Discussion part: add more previous studies to confirm or not the results obtained and explain in more detail the conclusions reached.
• Explain more the possible practical applications of the study carried out.

Author Response

Dear Reviewer,

we really appreciate your time and your effort in reading our manuscript. First of all, we want to say thank you for your kind words and positive feedback.

We improved our paper based on your suggestions. based on your suggestions.

• Point 1: We did a grammar check. We hope that the paper now is in a good shape from a grammar point of view.
• Point 2: In the Discussion section we cited more studies and compared our findings with the literature. We also explained our findings in more detail.
• Point 3: Thank you for pointing out to emphasize the possible practical application in more detail. We explained in detail the practical applications of the study at the beginning and in the last section of the Discussion section.

Best regards,

The Authors

Round 2

Reviewer 1 Report

My previous recommendation to this paper was to reject. However, the authors did a job addressing previous comments, and the revised manuscript was improved. However, some previous problems is still unresolved and some additional improves are needed.  Below are my comments:

Major concerns: Although authors tried to make their introduction more focused and presented the conception of “positive psychological factors”, I still have some concerns regarding to including the large number of different variables into models as a variables of interest (not as just a covariates). Particularly, authors noted that value of their study is that they aim to select the important factors and that this “short list” save time: “These factors are measured using lengthy questionnaires that makes it cumbersome to collect all these measures from all the incoming students. On the other hand,  by identifying a small set of psychological factors that have significant incremental predictive power, we suggest a more feasible strategy with direct applications: measuring only these important factors at the time of enrollment and using the psychological data together with the pre-enrollment achievement measures to identify students at risk of academic failure more efficiently”.

However, it is not clear what instruments were included into this “short list” and what the length of all instruments within this list. If authors aim to focus on practical value of their research, they should pay more attention to practical recommendation. Particularly, how much time is needed to fulfill all questionnaires from “short list”? Is it possible to fulfill these questionnaires for applicants? What system of support for students from “group of risk” can be suggested?

Authors wrote: “measuring only one or two psychological factors has also significant incremental predictive power”. However, it is not clear if the increasing AUC was statistically significant and what does it mean from practical view? Maybe it would be helpful to estimate predicted probability to drop out for student with different value of “main” psychological factors.

Finally, authors did not convince me, that this research strategy (to include the long list of variables and to select the important variables) have theoretical value.  I thing that authors should pay more attention to theoretical interpretations of obtained results and their value from theoretical perspectives.

Minor concerns:

• Author used some abbreviations without transcription (e.g. PERMA, AUC).
• Presenting the grit scale, author used lowercase letters, but further in the text they wrote GRIT (capital letters). Does it mean any sense?
• Table 1. The name of column “Alpha value” should be replaced by Cronbach’s alpha.
• Table 1. The Cronbach’s α for EPI was negative. Is it mistake? Although theoretically Cronbach’s α can have negative value, it indicated the existence of some problems with scale, for example, problems with coding of negative and positive items. Why there is no Cronbach’s α for Grit scale?
• Authors noted in their answers that they replaced machine learning algorithm to a more traditional approach: 10-fold-cross-validated logistic regression. First, in previous review I did not criticized machine learning algorithms or their using, but I suggested to describe statistical approach in more details. Second, in current version of manuscript authors used 10-fold-cross-validated logistic regression. This algorithm is one of the approach in machine learning models. Why authors wrote that they changed statistical approach? Although current approach was described clear enough, I recommend to drop this part: “We assume that the reader is familiar with the basic concepts of statistical learning, otherwise, we recommend [63]” (lines 237-238). Instead, it is possible to save reference on detailed description of approach after first sentence in method section: e.g. “We use 10-fold cross-validated logistic regression models to measure the predictive and incremental predictive validity of the factors [63]”.
• Table 2. In the title of Table 2 authors wrote: “The AUC of the UES alone is 0.737, the other AUC values should be interpreted in comparison with this value”. I think, that this sentence should be replaced into table’s notes or in description of results.
• Table 1 and 2 should be in accordance. Particularly, it is unclear, why in the Table 1 Cronbach’s α is presented for whole scale, e.g. PERMA, but in In Table 2 AUC for different subscales are presented. I assume, that in Table 1 it should be presented descriptive statistics and Cronbach’s α for those variables, which included into models. For example, if authors test models with whole PI self-control (or coping) scale, the descriptive statistics and Cronbach’s α should be presented for this scale. Vice versa, if authors include different subscales of PERMA, Cronbach’s α should be presented for these subscales separately.
• Line 269: The coping factor has the most significant predictive power that increased the AUC from 0.737 to 0.778, i.e. by more than 0.04”. This change is significant? (This comment can be extended on other parts where it was reported about increasing).
• Table 4. Title:The number of groups (pairs, triplets, quadruplets, quintuplets, sextuplets) and the ratio of all groups with additional incremental predictive power where the given factor contributes to. What does the “ratio of all groups” mean? It is worth to describe it before or after table.

Recommendation: Major revision

Author Response

Dear Reviewer,

thank you very much for highlighting that the revised manuscript was improved. We also believe that in the present round, we also managed to improve the manuscript based on your important suggestions.

In the followings, we comment on the issues, questions that you raised in your report

1) You mentioned your concerns about the inclusion of a large number of different variables as variables of interest (not as just covariates). We aimed to identify what psychological factors are worth measuring the most at the time of enrollment to build a model with the high predictive power of identifying at-risk students. We had the opportunity to measure a high number of psychological factors, we aimed to select those that have strong incremental predictive. This research design requires to use of these variables as variables of interest.

2) You asked how much time is needed to fulfill all questionnaires from the “shortlist”. That is indeed a very good question. To be more reader-friendly, we add a new table that shows the completion time of the surveys. (See please Table 6, row 470.)

3) You asked whether it is possible for applicants to fulfill the questionnaires. We suggest using these surveys at the time of enrollment for identifying students at risk of academic failure in order to provide them targeted help. We definitely do not suggest using psychological tests as selection criteria. From our point of view, this would lead to an undesirable admission procedure.

4) You asked what system of support for students from a “group of risk” can be suggested. This is indeed a very interesting question. In this article we focused on the identification of at-risk students, however, we mentioned some possible ways in the Discussion section and cited some relevant articles We feel that a more detailed description of the support system is out of the scope of this article but is a natural next step for us.

5) You raised a very important question about the significance of the improvement in AUC scores. We added a new sophisticated statistical test the null hypothesis that the two AUC scores are equal. We also commented on the statistical significance of the results in the paper. (See Table 5 and rows 251-264.)

6) You missed the theoretical interpretations of obtained results and their value from theoretical perspectives. We completely agree with your remark. On the other hand, we note that we had a different approach in this paper. Theoretically, the importance of our contribution finding is that we identified the psychological factors with the strongest incremental predictive power over pre-enrollment achievement measures.

7) We described the abbreviations that you requested.

8) We modified the captions as you requested, we also unified the inconsistent writing of the factors.

9) We analyze the Cronbach alpha values again and now we have the right numbers from which we can see that each scale or subscale has acceptable psychometric properties. The negative value was a mistake. Thank you for pointing this out!

10) We removed the sentence about machine learning as you suggested. We changed the more advanced machine learning model to a simpler one because logistic regression is more widespread and easier to interpret.

11)  We clarified the meaning of the “Ratio of all groups” column below Table 4.

Best Regards,

The Authors