Educational Design Principles of Using AI Chatbot That Supports Self-Regulated Learning in Education: Goal Setting, Feedback, and Personalization

Round 1

Reviewer 1 Report

This article presents an important topic on ChatGPT and generative AI technologies as concerns arise regarding students potentially exploiting these tools unethically, misrepresenting their work, or gaining academic merits without active participation in the learning process. They also present details on Zimmerman's self-regulated learning (SRL) framework and Judgment of Learning (JOL).

Some comments

Please add a Reference to Figure 2. An example of reverse prompting from a Chatbot.

Add a Table on the advantages and challenges of using ChatGPT and generative AI technologies.

It is better to add Topic Recommendations on using ChatGPT and generative AI technologies.

Figure 5 should be before the concluding remarks.

Milor checks.

Author Response

Dear Reviewer 1 – We would like to thank you for your attention-to-detail feedback! We have addressed your suggestions, please see below, or refer to our attachment.

Here’s our revision:

Reviewer 2 Report

Dear authors,

For me it was very interesting to read this article as it delivers topics that have the high potential and might be important for the current needs. The authors try to identify several instructional dimensions that should be helpful for the design of educational chatbots to facilitate effective learning for students or at least to supplement classroom instructions. They managed to show the role of chatbots as the facilitators of the teaching process. However, the articles content especially in terms of research results presentation demands very significant improvements. It’s rather a draft of the article (theoretical framework) and it should be adjusted to fulfil the minimum standards for scientific research article. Firstly, the main issues to be solved: - not sufficient description of methodological background and lack of some inevitable statistical analysis, - under-analysed and overgeneralized interpretation of data in the results part of the article, - missing explanitary part in the article. Secondly, please describe the experimental part of your research to confirm the stated hyposis relevant to your own research ( otherwise your research looks like incorporation of Zimmerman's SRL theoretical framework and JOL into the existing capacity of AI in Education). The authors do not compare their own research results with other authors findings in missing discussion section. There is not a lot of use done from the introduction part in discussion (topics for consideration – it could be a bit more “soaked” with it). As a result authors repeat some very general statements, but they do not justify it well in other authors concepts/research. Thirdly, it’s necessary to mention in conclusions also: further research plan, limitations section. Finally, the logic chain to support this research problem may be not clear. What problems the educational community is facing today? Why chatbots necessary to resolve these problems? How educational chatbots help teachers and students and what is the latest technology used in practice? For example, the Itroduction of this article has mentioned the marked demand of new technology, but the necessity of using these particular principles in education is indistinct. The arrangement of empirical study in this article is not interpreted clearly. I hope that these recommendations will be helpful for further development of the article.

The article delivers topics that have the high potential and might be important for the current needs. The authors try to identify several instructional dimensions that should be helpful for the design of educational chatbots to facilitate effective learning for students or at least to supplement classroom instructions. They managed to show the role of chatbots as the facilitators of the teaching process. However, the articles content especially in terms of research results presentation demands very significant improvements. The current version of the article should be rejected from publication, as it’s rather a draft of the article (theoretical framework) and it should be adjusted to fulfil the minimum standards for scientific research article. Firstly, the main issues to be solved: - not sufficient description of methodological background and lack of some inevitable statistical analysis, - under-analysed and overgeneralized interpretation of data in the results part of the article, - missing explanitary part in the article. Secondly, please describe the experimental part of your research to confirm the stated hyposis relevant to your own research ( otherwise your research looks like incorporation of Zimmerman's SRL theoretical framework and JOL into the existing capacity of AI in Education). The authors do not compare their own research results with other authors findings in missing discussion section. There is not a lot of use done from the introduction part in discussion (topics for consideration – it could be a bit more “soaked” with it). As a result authors repeat some very general statements, but they do not justify it well in other authors concepts/research. Thirdly, it’s necessary to mention in conclusions also: further research plan, limitations section. Finally, the logic chain to support this research problem may be not clear. What problems the educational community is facing today? Why chatbots necessary to resolve these problems? How educational chatbots help teachers and students and what is the latest technology used in practice? For example, the Itroduction of this article has mentioned the marked demand of new technology, but the necessity of using these particular principles in education is indistinct. The arrangement of empirical study in this article is not interpreted clearly. I hope that the recommendations will be helpful for further development of the article.

Proof reading is required. F.ex,

line 238

Prøitz [51] mentioned: “the two terms [learning outcomes and learning objectives] are often inter-twined and interconnected in the literature makes it difficult to distinguish between 238 them” [p. 122].

line 242

Therefore, These orientations

Author Response

Reviewer 2

Thank you for your comment and wonderful suggestions! We want to acknowledge that you indeed have provided lots of useful suggestions for empirical research work, and we will keep these in mind while proceeding to our future research route. Here, we would like to clarify that this article is not based on any experimental or correlational studies and/or the data gathered in controlled settings. Instead, our article addresses the existing instructional issues regarding the use of AI chatbots, in particular, ChatGPT, in teaching and learning. Based on Zimmerman’s SRL and several existing pedagogical strategies, our article proposes that teachers and instructional designers can formalize ways to resolve the existing challenges encountered by teachers and students when they use generative AI tools. Despite the abundance of approaches suggested by other researchers, we agree that when students use AI tools, they are demonstrating self-regulation, in which the process can involve several cognitive and metacognitive components that may provide educators with tools such as improved prompts and tasks to optimize the use of AI in learning. We acknowledge that future empirical studies are required to shed light on the efficacy of the suggested model. We have incorporated your suggestions and this requirement in our concluding remarks section so that the researchers who are interested in continuing this line of work can have a solid foundation that motivates their work. 

Below is our addition of the limitation section. We also add future direction for empirical research in our conclusion. 

-----------

4. Limitations

Lo's [78]comprehensive rapid review indicates three primary limitations inherent in generative AI tools: 1. biased information, 2. Constrained access to current knowledge, and 3. Propensity for disseminating false information [78]. Baidoo-Anu and Ansa [79] underscore that the efficacy of generative AI tools is intricately linked to the training data that were fed into the tool, wherein the composition of training data can inadvertently contain biases that subsequently manifest in the AI-generated content, potentially compromising the neutrality, objectivity, and reliability of information imparted to student users. Also, the precision and accuracy of the information generated by generative AI tools further emerge as a key concern. Scholarly investigations have discovered several instances where content produced by ChatGPT has demonstrated inaccuracy and spuriousness, particularly when tasked with generating citations for academic papers [80],[79].

Amidst these acknowledged limitations, our position leans toward an emphasis on students' educational use of these tools, transcending the preoccupation with the tools’ inherent characteristics of bias, inaccuracy, or falsity. Based on our proposal, we want to develop students' capacity for self-regulation and discernment when evaluating received information. Furthermore, educators bear an important role in guiding students on harnessing the potential of generative AI tools to enhance the learning process, instead of the generative AI tools can provide information akin to a textbook. This justifies the reason why we integrate Zimmerman's SRL model, illustrating how the judicious incorporation of generative AI tools can foster students' self-regulation, synergizing with the guidance of educators and the efficacy of instructional technology design.

[Here we have Figure 5]

5. Concluding remarks

This paper explores how educational chatbots, or so-called conversational agents, can support student self-regulatory processes and self-evaluation in the learning process. As shown in Figure 5 below, drawing on Zimmerman’s SRL framework, we postulate that chatbot designers should consider pedagogical principles, such as goal setting and planning, self-assessment, and personalization, to ensure that the chatbot effectively supports student learning and improves academic performance. We suggest that such a chatbot could provide personalized feedback to students on their understanding of course material and promote self-assessment by prompting them to reflect on their learning process.  We also emphasize the importance of establishing the pedagogical functions of chatbots to fit the actual purposes of education and supplement teacher instruction. The paper provides examples of successful implementations of educational chatbots that can inform SRL process as well as self-assessment and reflection based on JOL principles. Overall, this paper highlights the potential benefits of educational chatbots for personalized and interactive learning experiences while emphasizing the importance of considering pedagogical principles in their design. Educational chatbots may supplement classroom instruction by providing personalized feedback and prompting reflection on student learning progress. However, chatbot designers must carefully consider how these tools fit into existing pedagogical practices to ensure their effectiveness in supporting student learning.

Through the application of our framework, future researchers are encouraged to delve into three important topics of inquiry that can empirically validate our conceptual model. The first dimension entails scrutiny of educational principles. For instance, How can AI chatbots be designed to support learners in setting and pursuing personalized learning goals, fostering a sense of ownership over the learning process? Addressing this questions involves exploring how learners can form a sense of ownership over their interactions with the AI chatbots, while working towards the learning objectives.

The second dimension involves a closer examination of the actual self-regulated learning (SRL) process. This necessitates an empirical exploration of the ways how AI chatbots can effectively facilitate learners' self-regulated reflections and the honing of self-regulation skills. For example, how effective is AI’s feedback to a student’s essay and how do students develop subsequent SRL strategies to address the AI’s feedback and evaluation? Additionally, inquiries might also revolve around educators' instructional methods in leveraging AI chatbots to not only nurture learners' skills in interacting with the technology but also foster their self-regulatory processes. Investigating the extent to which AI chatbots can provide learning analytics as feedback that harmonizes with individual learners' self-regulation strategies is also of significance. Moreover, ethical considerations must be taken into account when integrating AI chatbots into educational settings, ensuring the preservation of learners' autonomy and self-regulation.

The third dimension is related to user interface research. A research endeavor could revolve around identifying which conversational interface proves the most intuitive for learners as they engage with an AI chatbot. Additionally, an inquiry might probe the extent to which the AI chatbot should engage in dialogue within educational contexts. Furthermore, delineating the circumstances under which AI chatbots should abstain from delivering outcome-based outputs to learners constitutes a worthwhile avenue of investigation. Numerous additional inquiries can be derived from our conceptual model, yet the central message that we want to deliver remains clear: our objective is to engage educators, instructional designers, and students in the learning process while navigating in this AI world. It is important to educate students on the potential of AI chatbots to enhance their self-regulation skills while also emphasizing the importance of avoiding actions that contravene the principles of academic integrity.

Reviewer 3 Report

The investigation into the potential implications of generative AI technologies on education is undoubtedly a topic of immense interest and relevance to the scholarly community. The paper cogently advocates for the seamless integration of pedagogical principles into chatbot design, accentuating the considerable benefits of educational chatbots in fostering personalized and interactive learning experiences. However, some concerns and recommendations for improvement have been identified.

1. The abstract should be more informative, providing details about the author's pedagogical principles and guidelines for implementing AI in teaching and learning contexts. Additionally, the rationale for the study could be more concise.

2. The theoretical framework section would benefit from slight reorganization. Starting each subsection with an introduction that provides context and rationale for the topic will strengthen the author's arguments and support for including each framework. For example, in Section 2.1, the authors begin with an informative statement about Zimmerman's multi-level SRL framework, but it's essential to explain why this framework is relevant for establishing guidelines for AI in teaching and learning contexts.

3. The limitations of generative AI chatbots, such as limited context understanding, domain knowledge, and adaptability, should be discussed to manage readers' expectations. Moreover, the paper's major limitation of not providing sufficient real-world applications to demonstrate the proposed concepts should be explicitly addressed.

4. The authors should clarify if this concept is original to their work. If there are any references available for this concept, they should be provided.

5. Concepts like "learning analytics" and "data-driven insights and algorithms" require more explanation, references, and examples to help readers fully understand their significance and implications.

6. The conclusion section could be improved in terms of organization. It would be beneficial to provide Figure 6 in a separate section with more detailed explanations. Additionally, the conclusion should address the study's limitations and offer recommendations for future research issues, as this is a conceptual paper proposing ideas with many unanswered questions.

The manuscript displays grammatical accuracy, but for enhanced scholarly quality, authors should revise sentence structures, improve clarity, and conciseness. Incorporating additional explanations or examples will aid readers' comprehension. Addressing these aspects elevates the manuscript's scholarly excellence.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 4 Report

Dear authors:

- This paper proposes three key pedagogical principles for integrating AI chatbots into classrooms, informed by Zimmerman's Self-Regulated Learning (SRL) and Judgmental Learning (JOL) framework.

The goal is to provide educators with guidelines for implementing AI in teaching and learning contexts, with a focus on promoting student self-regulation through AI-assisted pedagogy and instructional design.

Additionally, these guidelines are helpful for chatbot designers to consider how these tools fit into existing pedagogical practices to ensure their effectiveness in supporting student learning.

- Undoubtedly, the topic is extremely important and topical and addresses an aspect such as the incorporation of intelligent technologies, such as chatbots, in learning processes that has not been studied in depth.

- The dimensions analyzed are relevant and are central to the integration of these learning technologies. Additionally, the document provides examples of successful implementations of educational chatbots.

- The references used are current and pertinent.

Congratulations

Author Response

Dear Reviewer 4, 

We want to say thank you for your positive review of our manuscript. Your encouragement and support give value to our manuscript and our research team. Your words really mean a lot to us. Thank you so much.

Best Regards, 

Manuscript Team

Round 2

Reviewer 2 Report

I’m grafeful to the authors of this article for their fruitful work and well-grounded comments, especially for the concluding remarks done in a logical way. The authors presented enough information for me to think that they upgraded the methodological background of the article and added some inevitable proofs addressing the existing instructional issues regarding the use of AI chatbots, in particular, ChatGPT, in teaching and learning based on Zimmerman’s SRL and several existing pedagogical strategies. To my mind it contributed to the upgrading of the missing explanitary part in the article and confirmed the stated hyposis relevant to their own research. Inspite of the lack of experimental studies the authors modernised the discussion section showing the abundunce of approaches suggested by other researchers comparing their own research results with other authors findings. In conclusions the authors presented further research plan and limitations section. I think that with the help of the done improvments the logic chain of the research problem is clear and the necessity of using the suggested educational principles is distinct. The research comprises further perspectives and the results fitting the scope of the journal are significant for the practical purpuses. I would recommend future empirical studies for the authors to have solid foundations that make their research results more applicable.