Development and Evaluation of Adaptive Learning Support System Based on Ontology of Multiple Programming Languages

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Pleas se th attached file.

Comments for author File: Comments.pdf

Comments on the Quality of English Language

The overall quality of the English is good. The language is clear. There are small things so it would benefit from a proof reading in the end. In several places the paragraphs start with a sentence without a verb. Please consider developing the sentences but preserve the short and concise formulation which gives nice introductions to the paragraphs.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

This study aims to address the challenges that beginners often encounter in programming education and proposes an adaptive learning support system that combines programming ontology, Elo scoring mechanism, and dynamic practice recommendation. The study used the CONTINUOUS programming concept ontology to organize cross-language programming knowledge and designed a visual, prompt-based, and ability-adaptive practice platform. Finally, an empirical comparison was conducted using a total of 1,186 code submissions from three groups of students. The authors used random recommendation as a control group and analyzed the effectiveness of the system from multiple learning performance indicators. The research design was systematic, and the results also showed that the adaptive recommendation group had significant advantages in learning efficiency and concept mastery.

This study has a considerable level of technical integration and empirical value. The following are the areas where improvements can be made:

1. The paragraph introducing ADVENTURE in Figure 1 needs to be clearly described in detail, including which operation steps are for learners with programming experience and which operation steps are for beginners without programming foundation. The current description combined with Figure 1 does not clearly show how learners use the system.
2. Figure 2 shows the programming concept map. How is the order of learning each concept determined? For example, how do you decide that you need to learn list before learning list_method?
3. It is recommended that the content of Figure 2 can show that some concepts are completed (green represents Complete) and some concepts are in progress (In progress). This will help to better understand what a learner will see.
4. How do you define the difficulty of programming exercises? Are there any standards to refer to?
5. The author mentioned that "the system adjusts the current level according to the changes in learners' skills." What do you mean by "learners' skills"? Programming skills? Or programming concepts?
6. The author mentioned that "When learners reach the required score, the system will advance them to the next level. Conversely, if the learner's performance at the current level drops to zero, the system will reallocate them to the previous level." How is the required score defined? How do I determine when I have reached the required score to proceed to the next level? Secondly, when learners enter a new level, does the score have an initial value, or does it start from zero? If starting from scratch, if the learner accidentally answers a question at this level incorrectly at the beginning, will he or she be reallocated to the previous level? Please describe this part clearly.
7. If learners have difficulty solving a problem, they can click the "Try another exercise" button to receive other appropriate problems. "How will the learner's answer to this question be evaluated? Will it be considered a failure? Will there be a penalty? Or will there be no impact? Is there a limit on the number of times a learner can send a code to execute? Under what circumstances will the learner be considered to have a penalty for this question? Please explain clearly.
8. How are the concepts related to "conditionals" in Figure 10 selected? That is how we came up with the list of questions related to "conditionals" in Figure 10.
9. Since learners can be promoted to the next level when their skills reach 0.85, 0.85 is an important threshold value. Although it is not the value proposed by this study, it is recommended to describe the basis of 0.85.
10. How is the learner’s initial level calculated? For example, if a learner's skill level starts at 0.36, how is 0.36 derived?
11. How to calculate the Learning Rate (𝐾)? Is it enough to meet the minimum number of correct answers required for the Learning Rate to advance to the next level? Will this lead students to use an exhaustive approach, that is, to keep trying, even if they make mistakes, as long as they can find a way to accumulate the correct number.
12. The authors state that the learning assessments are based on supervision by experienced teachers. However, there is no mention of whether the test has been validated to ensure that it correctly measures programming learning. The following indicators should be included: reliability of the instrument (e.g., Cronbach’s alpha) and content validity (expert review). Therefore the perception questionnaire has not been psychometrically validated. Second, the lack of reliability and validity testing limits the interpretation of the data.
13. If the MANOVA and ANOVA results show no significant differences between groups, it is not recommended to over-interpret them.
14. For quantitative results, please report effect sizes (e.g., eta-squared, Cohen's d) alongside p-values. Effect sizes provide critical information about the practical significance of the findings.
15. The presentation of qualitative data requires more rigor. Simply presenting selected quotes is insufficient. Present the findings in a more organized manner (e.g., using themes supported by illustrative quotes). Explicitly analyze and interpret the qualitative data rather than just presenting it. Consulting recent Q1 publications in educational technology or related fields for examples of rigorous qualitative data presentation is recommended.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

Thank you for a thorough revision of the paper and you have addressed and replied to the comments and questions.

The structure and argumentation throughout the paper is improved by a more clearly stated aim, additional research question, and by structuring the paper according to these. As this is also a developmental work, you highlight your solutions and contributions in the field.

Both the introduction and discussion are strengthened by more references.

Section 3 is easier to follow by frequent references to the figures and detailed explanation of the workflow.

Section 4 is clearer thanks to the additional explanations of the adaptation of the ERS in your system.

Section 5 (Experimental methods) There is now a clearer distinction between methods and results. You provide also valuable information about the data collection. The section is also more complete with the addition of discussion of limitations.

Section 6 (Results) – suggestion for revision

NB! Line 652:  You start by: “To address the second research question …” it is now the THIRD.

The additional details and explanations provide a clearer picture.

The addition of qualitative results provides an interesting and deeper understanding. The presentation is well structured, and you use the qualitative results to support the quantitative results. However, in the section on “technology acceptance” the use of qualitative results is less connected to the research question and quantitative results. In the beginning you state that the quantitative results show no significant differences. That is a valid result and as you write the system is “beneficial for all participants”. However, when you focus on each system’s features (line 697-763) it is not stated from which group the statements are, they become more of a general evaluation, and it is difficult to see how they are related to the research question. Also, the quotes represent a substantial part of the results section. Could some of them be summarized? My suggestion is not to remove the qualitative results but please consider adding some explanation and arguments how they contribute to answering the research question (focusing “learning performance, including learning achievement and perception”). For example, these features of the system are similar (same) for all groups (experimental, control, preliminary) and therefore how can the similar quantitative results be interpreted. When you quote positive/negative feedback, is it relevant to mention which group it comes from, like you do when explaining the “satisfaction” (Line 764-777)?

Line 795-796 and 822-826: Please explain what is meant by “group assignment” how does it affect the performance you describe in these two places?  

 

Section 7 (Discussion and conclusion)

The section is greatly improved by the addition of discussion of your results in light of earlier studies. This also highlights your contribution to the field.

Section 7.3 (Limitations) – suggestion for revision

The additional explanation related to the data collection is also mentioned in the experimental methods (Section 5). Please add a short reflection on the methodological limitations (for example generalisability).

In summary, the article is significantly revised and improved. I have only minor suggestions for revision in Section 4 and 7.3.

 

 

Comments on the Quality of English Language

I think the overall quality of English is fine. However, the text will benefit from a thorough reading before final submission. I found a few things and there may be more.  

Line 652 – THIRD research question

Line 694 – There is no negation between the two last sentences in the paragraph. Please consider replacing “nevertheless” with “and” or similar.

Line 815-815 – sentence starting “Based on …” something is missing

Line 974 – try -> tried

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

We thank the authors for their comprehensive response to the reviewers' comments and their careful revisions. Based on the responses and the revised manuscript submitted in this review round, we acknowledge the following improvements and offer a few additional suggestions for consideration:

1. Enhancement of System Architecture and Workflow Description: In response to previous concerns about the lack of clarity in Figures 1 and 2 of the ADVENTURE system, the authors have added five operational steps and annotated usage scenarios for both beginners and experienced users. The inclusion of the newly added prompt message interface (Figure 2) has significantly improved the readability and interpretability of the manuscript.

2. Clarification of Ontology and Concept Hierarchy: In response to prior feedback regarding the conceptual ordering of "list" and "list_method," the authors have clarified in Figure 3 and Section 3.1 that the structure is based on the CONTINUOUS ontology. It is now also specified that students can freely select concepts, and the system provides adaptive support based on difficulty. This clarification offers a sound logical foundation and completes the structural explanation.

3. Explanation of Difficulty and Skill Assessment Mechanisms: The authors have now clearly defined the difficulty levels using Effenberger et al. (2019), described the skill calculation method, and cited the promotion threshold of 0.85 from Zheng et al. (2022). Additionally, the rules for skill upgrading and downgrading are detailed, enhancing transparency in the adaptive mechanism's logic.

4. Clarification of Interactive Behavior Evaluation and Penalty Logic: Regarding whether the "Try another exercise" function is equivalent to an error and whether there is a usage limit, the authors have clarified that this behavior is penalized similarly to an error and is recorded as an incomplete attempt. This design choice is reasonable and well explained.

5. Correction of Statistical Presentation and Avoidance of Overinterpretation: In response to concerns about potential overinterpretation of non-significant MANOVA and ANOVA results, the authors have revised the corresponding text in Sections 6.1 and 6.2, providing effect size (η²) values and appropriately tempering their interpretations.

6. Improvement of Qualitative Analysis Presentation: Previously, the qualitative analysis was somewhat disjointed, listing only quotations without sufficient context. In the revised manuscript, the authors have reorganized learner feedback around five key system functions, supplemented by interpretation and illustrative quotations. This aligns more closely with best practices for qualitative reporting in contemporary educational technology journals.

What could be further improved is the presentation of the reliability and validity indicators for the instruments used. While it is noted that the test was developed by teachers with over ten years of experience and the questionnaire was based on Wang et al. (2014, 2020), we recommend including explicit psychometric indicators such as Cronbach’s α and details about the validity verification process to strengthen the credibility of the measurement instruments.

Overall, the authors have responded thoroughly to all review comments, and the revisions have significantly improved the clarity, methodological rigor, and presentation quality of the manuscript. If no further major concerns are raised, we recommend this manuscript be accepted for publication with minor editorial edits.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf