Unlocking the Potential of Competency Exam Data with Machine Learning: Improving Higher Education Evaluation

Round 1

Reviewer 1 Report

Dear authors, 

Congratulations on your study! Please see some comments I have made below. Besides, my main comment would be that your paper is not connected to the topic of sustainability. It belongs to an educational journal, or you could make a clear connection to Sustainable Education Goal 4 (SDG 4).

Lines 57-60: Review this paragraph, doesn't make sense: "Taking into consideration the standard of the graduates produced by Jordan's higher education institutions, as well as including information that is helpful in the process of formulating rules governing higher education, Find out how far the university has come and what improvements have been made."

Lines 67-69: I think you wanted to connect the two sentences by a comma: "In order to provide an accurate prediction model for student performance, the data from the competency exams will be merged with ML models. After classifying every individual subset of data according to its features."

Line 74: In spite of the fact that the competency exam in Jordan (replace with "Although")

Line 123: In Section 2, the related works is presented. (are)

Line 460: The experiments is based on a CE- dataset (are)

Page 7: try splitting the text in paragraphs, it would read easier.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

In your introduction, you make a claim about all of the factors that affect student performances on CBEs, [lines 48-55.] but what seems to be missing here is evidence to support this claim. How do you know all of these things affect student’ performances on these exams? What have others found that support your claims? This section is currently under-supported by evidence.

Line 57-60 sentence structure needs work.

Line 83-85 not a full sentence

What are “academic characteristics” being operationalized as in this paper? An explanation of this will be helpful to readers. (After having read further, this is maybe what’s described in lines 200-203?)

In your purpose statement you call out looking at demographic and academic elements for how they may affect student performance on CE. Why those things? My first note called out your claims about elements of students' internal beliefs and/or behaviors – but what might be more appropriate is to instead use this space to describe how academic elements and demographic may be impactful to student performance to help set up your purpose. So far, you don’t appear to be at all investigating student motivation, self-regulation, goal setting, or self-efficacy, so I am confused as to why you tell the reader they are important to understand if your research never seeks to understand them.

In the first paragraph of related works, you list what ML can successfully predict, and then you make the claim that ML can be used to support student success and improve decision making. I think readers are missing a “how” connection. I would recommend more explicitly connecting these two things. How can ML’s prediction of X, Y, Z lead to better decisions making and student success?

Sentence in lines 196-199 needs citations.

Your introductory sentence into the paragraph starting at line 200 leads me to believe I’m about to read a paragraph on the factors that affect student performance, but instead this is a paragraph that describes categorical data in ML? I would recommend changing the first sentence of that paragraph – maybe to a clearer explanation of the information the reader is about to get? I’m not sure of what these categories are or how they are useful to my understanding of this work as a reader. Also, are there citations to support the creation of these categories? Is this something your research team did? If so, how? Is this standard for the field / ML in student success prediction? If so, please cite.

It is unclear to me in sections 2 and 3 which sources you are describing to detail additional work in the field, and which sources have directly influenced the research you conducted and are reporting on in this paper – I would recommend a review of grammar to ensure consistency in how other’s work is described.

Any citations for data segmentation? For readers who are inspired to use it in their own work after reading about it in yours?

Figure 1 is distorted/stretched, and the text is too small to read.

In section 5 the authors start using the word “you” to describe methods (second person). 1) the use of second person is inconsistent with the style in the rest of the paper so far, I would recommend a read-through for consistency one way. 2) the explanation in which these appear I think belong in the methods – possibly a sub-section that describes the methods used to compare results and assess accuracy?

Your discussion section seems to be a summarization of the results, but I find it lacks discussion of the implications of the results. Your purpose statement for the paper states that this work was provide guidance to decision makers in this space, however the discussion section lacked connections between your results and relevant literature that could inform decisions in academic programs using ML. This comes out in the conclusion, that the results can be used to improve student support and learning, but there is never a connection as to how – how do the insights your ML model found inform decisions?

The last few sentences of your conclusion focus on “improvements to the paper” with “additional time” – I would recommend not framing these last pieces in this way – it opens the opportunity for reviewers and readers to ask the question why you didn’t delay publication to make these improvements? Instead, I would recommend talking about limitations of the research and the implications of those limitations. You could have similar points, but I would recommend framing it as “limitations of the research design” rather than “ways we could have made this better with more time”. That shows readers you considered and acknowledge possible shortcomings of your work, but they were all intentional research decisions that were made by you as professionals, and not shortcomings because you ran out of time. The points you brought up could be framed as “extended / future research opportunities.”

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

Dear Authors,

I have the following suggestions for you:

1- The introduction part was written without citing relevant sources which is quite alarming. I suggest authors to link the ideas in the introduction part to the corresponding references.

2-All three research questions proposed between lines 109-113 should be answered in the results section. 

3- Please make sure the contributions mentioned between lines 116-120   are warranted with the findigns of the study.

4- Related work seems allright with minor additions. Pls go over table 1 and add ML models used in the summarized articles in related work. 

5- Please add summary of literature review and the main gaps in the literature which your study contributes.

6- Please give detailed description of the dataset used in this study. It is not clear.

7- figure one should be combined and should be depicted as only 1 figure.

8- Research methodology should be reorganized. Pls separate the phases into subtitles and group the information underneath each phase.

9- The machine learning techniques used in this study should be explicitly mentioned in the methodology section.  The ratio of the dataset allocated for testing and training should be mentioned.

10- For relevant ML algorithms used, neural network model with number of input /output land hidden layers should be specified.

11- Please give detailed description of how Decision Trees (DT), Support Vector Machines 446 (SVM), Multi-layer Perceptron (MLP), K-Nearest Neighbors (KNN), and Logistic Regression (LR)  were applied.

12- Authors should give the rationale for performing 4 different experiments  in the results section.

13- Please give the reason why earlier experiment results yielded poor accuracy. 

14- Pls explain Table 8 . It says experiment 6-14 were performed which did not inside the text.

15 Discussion did not involve the discussion of the findings with related studies. What are the pros and cons of this study findings as compared to others.

15-Please provide practical implications and future recommendations.

16- Is  the dataset that was used in this study accessible to other researchers?

 17- References should be improved. Pls add up-to-date sources.

Good luck!

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Dear Authors,

I have read your response and the edited manuscript. I still think your work is valuable, but you should mention SDG 4 to establish a clear and undisputable link with sustainability in education.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

Line 57-60 sentence structure needs work – This did not appear to be addressed / edited in the revision.

Line 83-85 not a full sentence – This did not appear to be addressed / edited in the revision.

What are “academic characteristics” being operationalized as in this paper? An explanation of this will be helpful to readers when the content is first introduced in the introduction. Your cover letter indicated that you’d make edits to the introduction section to explain this terminology, but I don’t see any edits in the revision?

In your purpose statement you call out looking at demographic and academic elements for how they may affect student performance on CE. Why those things? I think it would be valuable to briefly cover in the background evidence of ways in which demographic and academic elements have been impactful to competency so support your further investigation into them. You had this content in your response to me (“we included demographic and academic elements as factors that may affect student performance on competency-based exams because previous research has shown that they can have an impact. For example, studies have found that students from different demographic backgrounds may have different levels of access to resources and support systems that can affect their performance. Additionally, academic characteristics such as prior knowledge and study habits can also influence how well students perform on exams.”), but I think it also belongs in the paper with appropriate citations to convince readers of its relevance, not just me.

Sentence in lines 196-199 needs citations. This did not appear to be addressed / edited in the revision.

You stated in your cover letter “Regarding the categories in machine learning, they were created by our research team based on domain knowledge and experience in working with student performance data. We did not come across any standard categorization in the literature for this specific purpose. However, we will provide more information and clarification on how these categories were created and their relevance to our study. We will also include appropriate citations to support our claims.” – I greatly appreciated the clarification (I assume these edits happened in lines 201-214), but this section is still missing citations as to what literature supported your creation of these categories.

Thanks for including citations about data segmentation in your cover letter response to me! I would recommend adding them to the paper where you see most appropriate to provide these resources to your readers as well.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

Dear Authors, 

I'm satisfied with the revised version of the manuscript. However,

I have few additional suggestions. 

Discussion requires thorough compare and contrast of your findings with relevant sources. 

Please separate conclusion section and add practical implications and future recommendations are  independent titles.

I appreciate if authors could add more relevant references to their introduction, related work and discussion sections.

Also I suggest authors to go over English proof reading of the manuscript.

Apart form those issues, I believe that the study has potential and the topic is interesting. 

Best of luck!

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 3

Reviewer 2 Report

Thank you for addressing the concerns / suggested revisions. There exist a few small typos that a careful editorial review will hopefully address.

Author Response

Dear reviewer,

Thank you for your feedback and for taking the time to review our work. We appreciate your comments and have carefully reviewed the manuscript to address the concerns and suggested revisions. We apologize for any  typographical errors that may have been overlooked during the editing process. We have conducted a thorough review to eliminate the remaining typos (please see the revised version).

Thank you again for your valuable feedback.

Author Response File: Author Response.pdf