Enhancing Online Learning Through Multi-Agent Debates for CS University Students

Abstract

As recent advancements in large language models enhance reasoning across various domains, educators are increasingly exploring their use in conversation-based tutoring systems. However, since LLMs are black-box models to users and lack human-like problem-solving strategies, users are hardly convinced by the answers provided by LLMs. This lack of trust can potentially undermine the effectiveness of learning in educational scenarios. To address these issues, we introduce a novel approach that integrates multi-agent debates into a lecture video Q&A system, aiming to assist computer science (CS) university students in self-learning by using LLMs to simulate debates between affirmative and negative debaters and a judge to reach a final answer and presenting the entire process to users for review. This approach is expected to lead to better learning outcomes and the improvement of students’ critical thinking. To validate the effectiveness of this approach, we carried out a user study through a prototype system and conducted preliminary experiments based on video lecture learning involving 90 CS students from three universities. The study compared different conditions and demonstrated that students who had access to a combination of video-based Q&A and multi-agent debates performed significantly better on quizzes compared to those who only had access to the video or video-based Q&A. These findings indicate that integrating multi-agent debates with lecture videos can substantially enhance the learning experience, which is also beneficial for the development of students’ high-order thinking abilities in the future.

1. Introduction

With the increasing availability of online courses, a growing number of undergraduate and graduate students are turning to these resources for learning. Millions now access free educational videos on platforms like Khan Academy, Coursera, edX, Udacity, MIT OpenCourseWare, YouTube, and Chinese MOOC platforms. This trend underscores how Internet technology is enhancing educational access and broadening the range of available courses, particularly serving as a valuable alternative for students unable to attend in-person classes during the COVID-19 pandemic [1]. However, most current online courses still rely primarily on one-way lectures from an instructor, supplemented by a few post-lecture quizzes or self-regulated learning prompts [2,3]. There is a lack of necessary interaction and discussion, making them not significantly different from traditional, teacher-centered instruction, except that they are delivered through the Internet. Previous research on human–computer interaction has focused on enhancing educational videos through various techniques, such as interactive transcripts, word clouds, keyword search, and storyboard highlights [4]. Other studies have segmented videos to display key moments alongside snapshots and transcripts [5]. Additionally, some researchers have developed video digests structured in a textbook-like format, organized into chapters, titles, and sections with accompanying text summaries [6]. Moreover, Shimada et al. [7] found that students who used summarized slide previews achieved higher pre-quiz scores and required less time compared to those who previewed the original learning materials. Therefore, using large language models (LLMs) to summarize video content and subsequently perform retrieval-augmented generation (RAG) may also provide an effective human–computer interaction (HCI) approach to enhance students’ learning efficiency.

The advent of LLMs, such as Generative Pre-trained Transformer 4 (GPT-4) [8], has provided significant technological advancements in various educational activities, including engineering education [9], language teaching and learning [10], and personalized learning [11]. Despite their potential, LLMs still face challenges [12,13] such as hallucination, reliance on static training data, lack of real-time knowledge, limited problem-solving capabilities, memory continuity issues, ethical and safety concerns, and insufficient personalization. Careful use of LLMs poses challenges for both teachers and learners [14]. However, these limitations can potentially be mitigated by advanced prompt engineering techniques, such as prompt engineering [15], chain of thought (CoT) [16], self-consistency (SC) [17], and retrieval-augmented generation (RAG) [18]. Ongoing advancements in generative AI, including RAG, are progressively enhancing the factual reliability and responsiveness of large language models (LLMs). Autonomous agents have long been a central research area in both academia and industry. Large language models have been applied as autonomous agents in many fields, especially in educational assistance [19,20]. This advancement suggests a promising pathway for implementing multi-agent systems based on large language models, enabling novel online teaching and learning experiences.

Accordingly, we present an online learning-assisted scheme that offers an intuitive and user-friendly solution. This scheme supports both question-and-answer sessions and topic discussions through video-based multi-agent debates. The purpose of this paper is to explore the role of open-source LLM-based agents in facilitating self-learning among undergraduate students, particularly in the field of computer science.

The research question for this study is the following:

• RQ. How can video-based multi-agent debates enhance online learning outcomes for CS students?

Regarding the above-mentioned research question, we further elaborate and propose the following hypotheses to explore the effects of multi-agents on students.

• H1. Video-based multi-agent debates are more effective than video-only and video-based Q&A approaches for online learning among CS students in terms of objective questions.
• H2. Video-based multi-agent debates are more effective than video-only and video-based Q&A approaches for online learning among CS students in terms of enriching the answer dimensions of open-ended questions.

2. Literature Review

In this section, we conducted a review of the relevant literature. Prior to conducting this review, we first carried out a literature survey on Clarivate Analytics’ Web of Science using the keywords “online learning and video”, resulting in 3057 documents published since 2012. Keyword analysis reveals that it is relatively rare to use multi-agent debates to enhance online learning in a video-based environment, as shown in

Table 1. The top ten most relevant keywords were extracted from 3057 documents retrieved from Clarivate Analytics’ Web of Science using the keywords “online learning and video”, constrained to the educational research field.

	Author Keywords	Articles	Keywords Plus	Articles
1	ONLINE LEARNING	270	EDUCATION	264
2	LEARNING	227	STUDENTS	231
3	E-LEARNING	189	ONLINE	186
4	EDUCATION	172	PERFORMANCE	145
5	FLIPPED CLASSROOM	137	IMPACT	141
6	HIGHER EDUCATION	135	TECHNOLOGY	131
7	BLENDED LEARNING	121	VIDEO	114
8	ONLINE	121	ENGAGEMENT	113
9	VIDEO	112	MOTIVATION	95
10	TECHNOLOGY	103	PERCEPTIONS	95

. We performed a topic analysis on the abstracts of relevant literature, as illustrated in Figure 1. The findings reveal that cutting-edge technologies such as “virtual reality” and “artificial intelligence” have garnered significant attention in recent years, while research on online courses and online learning environments has consistently remained a central focus in the field. The contribution of this paper lies in presenting a novel approach to online learning. It leverages LLMs as multiple debaters to generate answers and, through this process, aims to inspire students’ high-order thinking abilities. The aforementioned literature highlights the application and transformative impact of “online learning and video” in education. Therefore, we summarize the prior literature about online learning from three perspectives: massive open online courses (MOOCs), online learning environments, and LLM-based multi-agents.

2.1. MOOCs

MOOCs represent a widely studied innovation in distance education, first introduced in 2008 [21]. They gained prominence as a popular learning format in 2012, a year often referred to as the “Year of the MOOC” [22]. MOOCs are distinguished by their openness, scalability, and ability to cater to diverse audiences. Openness facilitates free enrollment and unrestricted access to course materials, thereby advancing the democratization of education [23]. Scalability allows instructors to interact with thousands of learners simultaneously, surpassing the constraints of traditional classroom settings [24]. The global pandemic of COVID-19 underscored the role of MOOCs in delivering educational and learning opportunities to a wide audience [25,26,27]. A particularly appealing feature of MOOCs for adult learners is the high degree of control and autonomy they offer over the learning process [28,29]. However, despite their significant potential, MOOCs continue to face challenges related to high dropout rates and low completion rates, which hinder their broader adoption [30].

Despite offering convenient online learning, MOOCs face notable limitations. Completion rates typically range between $5\%$ and $15\%$, with longer course durations correlating with lower completion rates [31]. Margaryan et al. [32] highlighted shortcomings in traditional MOOCs’ learning outcomes, attributing them to a lack of personalized support, limited feedback mechanisms, and insufficient opportunities for deep learning. The one-size-fits-all approach of MOOCs often fails to address the diverse learning styles and needs of participants [33]. Additionally, the absence of personalized feedback and direct interaction with instructors remains a critical challenge [34].

To address the aforementioned issues, educators and technologists have explored various interactive teaching strategies. The integration of interactive elements into MOOCs has been shown to significantly enhance learning outcomes. Interactive approaches in MOOCs include discussion forums, gamification, and interactive videos. MOOC discussion forums serve as a vital channel for communication between instructors and learners, with their effectiveness shaped by various factors. Wei et al. [35] demonstrated that small group discussions, particularly those with clear objectives and assessment criteria, enhance learner engagement. Similarly, Neha et al. [36] found that structured topics encourage participation, while immediate feedback helps sustain engagement. Galikyan et al. [37] highlighted the dual role of forums in facilitating knowledge exchange and fostering social connections, contributing to deeper learning when instructors actively guide discussions and incorporate peer review. Gamification in MOOCs has demonstrated significant positive impacts. Freitas et al [38] found that gamification enhances motivation and engagement, with achievement badges serving as powerful incentives. Similarly, Meng et al. [39] showed that a well-designed scoring system promotes skill development and multidimensional engagement but cautioned against the risk of goal misalignment if gamification is overused. Maintaining clear rules and mechanisms is essential to ensure that gamification remains aligned with educational objectives. MOOCs incorporate interactive videos with various elements to enrich the learning experience. Embedded quizzes, implemented by platforms such as Coursera and Udacity [40], serve as checkpoints for self-assessment of knowledge. Furthermore, Chen et al. [41] introduced time-anchored discussions, enabling learners to engage in conversations tied to specific video timestamps, thereby fostering interaction and enhancing content understanding. Some platforms have further innovated by combining linear discussions with live comments, providing more flexible and dynamic interaction opportunities.

Integrating large language models (LLMs) into MOOCs represents a transformative development. These models enable personalized learning experiences, as exemplified by Khan Academy’s Khanmigo, which supports students through adaptive, progressive questioning. Similarly, the Massive AI-Empowered Course (MAIC) [42] establishes an intelligent classroom environment, leveraging AI-driven roles to create personalized learning paths that were previously unattainable in traditional MOOCs.

2.2. Online Learning Environment

The online learning environment is a diverse and digital educational ecosystem that provides learners with flexible educational platforms that transcend geographical, temporal, and cultural boundaries through the Internet, cloud computing, and advanced digital interaction technologies [43]. This complex learning ecosystem is not merely a technological medium but an innovative paradigm that integrates education, cognitive science, and information technology, with the core value of offering highly personalized, adaptive, and inclusive learning experiences [44]. Educational theories provide the essential theoretical foundation and normative guidance for the cognitive framework and practical paradigms of online learning. Self-Determination Theory (SDT) extensively analyzes the inherent mechanisms of learners’ basic psychological needs—autonomy, competence, and relatedness—regarding learning motivation and performance, offering significant theoretical insights into learner agency in online learning contexts [45]. The Cognitive–Affective Immersive Learning (CAMIL) model deconstructs the immersive experience of learners in digital learning environments in various dimensions, elucidating the complex dialectical relationships between cognitive engagement, emotional involvement, and interaction experiences throughout the learning process [46]. The Inquiry-Based Learning (IBL) framework provides a structured methodology for online learning, emphasizing the cultivation of learners’ autonomy in exploration, critical thinking, and problem-solving skills within the knowledge construction process [47]. Self-Regulated Learning (SRL) strategies reveal the complex pathways through which learners regulate their behaviors and achieve goals during autonomous learning [48,49]. Deeper investigations indicate that learners’ emotional trajectories and levels of interaction critically influence learning outcomes, not only impacting cognitive knowledge acquisition but also reflecting the development of learners’ metacognitive skills and learning adaptability [50]. This study demonstrates that cognitive presence, social presence, and teaching presence are significantly correlated with autonomy, competence, relevance, and perceived learning in the context of online educational settings in China [51].

The COVID-19 pandemic, in particular, has accelerated the application of digital technologies in education, leading to a growing number of technological innovations in online learning platforms. For example, virtual reality (VR) technology provides a revolutionary learning paradigm for disciplines that require high levels of practicality and interactivity by creating highly immersive, interactive learning environments [46,52]. At the same time, artificial intelligence (AI) optimizes learning processes and dynamically adapts learning paths through intelligent algorithms tailored to the individual needs of learners [53].

By breaking the structural limitations of traditional education, online learning environments achieve global sharing of educational resources, promote learner autonomy, and implement the concept of lifelong learning, thereby advancing educational equity and knowledge democratization in a globalized context. Despite advancements in technology, online education environments still face significant structural challenges. First, traditional online courses primarily rely on one-way lecturing models, which closely mirror conventional classrooms by simply transferring content to the Internet, thus lacking substantial pedagogical innovation. As a result, students remain in a highly passive state during the learning process, hindering their ability to think independently and explore knowledge actively. Second, the lack of interactivity leads to substantial declines in student satisfaction and learning outcomes due to insufficient opportunities for immediate questioning and deep communication [54]. Existing online learning platforms often fail to adapt precisely to learners’ cognitive characteristics and learning paces, resulting in a general lack of personalized learning paths and, consequently, low efficiency and engagement and poor learning outcomes [55]. More critically, these platforms tend to overlook the multidimensional nature of learning, failing to adequately stimulate learners’ cognitive autonomy and emotional engagement, which causes online education to become a mechanical and soulless tool for knowledge transmission. These inherent limitations severely restrict the developmental potential of online education and hinder the genuine transformation of the digital learning ecosystem.

2.3. LLM-Based Learning Systems

LLMs have recently demonstrated remarkable potential, achieving reasoning and planning capabilities comparable to those of humans. This aligns precisely with human expectations for autonomous agents capable of perceiving their surroundings, making decisions, and taking appropriate actions in response [56,57]. Building on the impressive capabilities of single LLM-based agents, LLM-based multi-agent systems have been proposed to harness collective intelligence and specialized skills across multiple agents. Compared to systems employing a single LLM-powered agent, multi-agent systems offer enhanced functionalities by (1) specializing LLMs into distinct agents, each with unique capabilities, and (2) facilitating interactions among these agents to effectively simulate complex real-world environments. Recent studies [58] have shown promising results in leveraging LLM-based multi-agents for diverse tasks, such as software development, multi-robot systems, societal simulations, policy modeling, and game simulations, with particular success in simulating human behavior. Notable examples include Generative Agents [59], Ghost in the Minecraft [60], and GPT-Bargaining [61].

As LLMs continue to advance, intense discussions have arisen regarding methodologies in this field [62,63]. A key focus lies in fully harnessing the capabilities of large models to simulate real classrooms with multiple agents for automated teaching. LLMs have already been applied to various intelligent educational tasks to support teaching. Ref. [64] introduced a multi-agent classroom simulation framework involving user participation, demonstrating that LLMs can effectively replicate traditional classroom interaction patterns while enhancing the user experience. This work pioneered the use of LLM-powered multi-agent systems in virtual classroom teaching.

In summary, how to effectively utilize AI technology to advance education has long been a key research focus, with scholars exploring this topic from various perspectives [9]. For example, Huber et al. [65] explored the use of LLMs through playful and game-based learning methods, while [66,67] implemented LLM prompt-based personalized tutoring. This study approaches the topic from the perspective of video-assisted online learning. AI may negatively impact independent thinking by fostering student dependency, leading to a lack of initiative in problem-solving. This can hinder deeper memory retention of knowledge and the development of critical thinking skills. Mohamed et al. [68] examined the benefits and drawbacks of integrating AI tools into language instruction, while [69] explored various issues caused by ChatGPT (gpt-3) in a broader range of educational scenarios. Zhang et al. [70] discussed students’ dependency on AI and identified the top five negative effects of AI technology: increased laziness, the spread of misinformation, decreased creativity, and reduced critical and independent thinking. Efficient, reliable, and valid assessment for online learning is also a topic worthy of in-depth investigation [71]. Further detailed investigation and assessment are needed in the area of online learning. Finally—and most importantly—the use of artificial intelligence tools like ChatGPT in education raises ethical dilemmas and challenges, including concerns about data privacy, over-reliance on technology, intrinsic biases, and the integrity of student outputs [72]. Achieving a rational and ethically balanced integration of LLMs and conversable agents in educational settings remains a significant challenge for researchers leveraging AI to enhance education. Although there have been quite a few attempts to integrate LLMs into learning systems, the design of such models for personalized learning still has room for further enhancement. Future research could explore the use of LLMs to achieve personalized learning through a conversation-based tutoring system enhanced with more interactive elements such as quizzes or debates that inspire higher-order thinking.

3. Materials and Methods

3.1. Design of Evaluation System

After defining the research questions and hypotheses, the study was conducted in four steps, as outlined in Figure 2. Step 1 focused on obtaining initial scores through a pre-video test, where participants answered questions via an online quiz. Step 2 involved having participants watch a lecture video and interact with a software interface. This step incorporated a learn-as-you-go approach, supported by multi-agent debates to facilitate understanding of key concepts. In Step 3, a post-video test was conducted, which also included answering an online quiz. Step 4 analyzed learning outcomes and subjective evaluations by comparing pre-test and post-test scores, along with responses from a follow-up questionnaire. The results of the questionnaire were analyzed using SPSS software (29.0.1.0(171)).

3.1.1. LLM RAG-Based Video Understanding Function

Videos inherently exhibit multimodal properties, encompassing both visual and audio features. The visual features can be categorized into image captioning, object detection, and optical character recognition (OCR) tasks. We extracted keyframes from the video using CLIP features and used the BLIP model to generate image captions, while employing GRiT and RAM++ for object detection. Additionally, we utilized Whisper to transcribe the audio into text. These multimodal features provide the foundational data for subsequent QA tasks, as shown in Figure 3.

The method proposed in this paper is compared with the current state-of-the-art approaches, including VideoChat2 (https://github.com/OpenGVLab/Ask-Anything, accessed on 5 September 2024) and the open-source VLog project. Unlike VideoChat2, which primarily utilizes visual features, our approach incorporates additional modalities, such as audio and OCR features. As a result, our method demonstrates a superior ability to capture finer details in video content. Figure 4a shows that current state-of-the-art methods for video understanding, such as VideoChat2 [73], can only answer questions related to the visual content of the video but struggle with finer details. In contrast, the multi-modal video analysis method based on the VLog project (https://github.com/showlab/VLog, accessed on 27 August 2024) can answer detailed questions about the video, though not always with full accuracy. The results presented in Figure 4b build on the VLog project, where ChatGPT is replaced with the free online “WizardLM-2” LLM API by Together.ai (https://api.together.ai/, accessed on 27 August 2024), along with English-to-Chinese translation using the pre-trained “Helsinki-NLP/opus-mt-en-zh” model. WizardLM-2’s capabilities are very close to those of cutting-edge proprietary models such as GPT-4-1106-preview and significantly ahead of all other open-source models. However, as shown in Figure 4c, our system, enhanced with OCR recognition, significantly improves accuracy in capturing detailed information within the video content. Compared to existing systems, our system accurately extracts video information, then provides it to the RAG module for answer generation.

3.1.2. Multi-Agent Debate-Assisted Learning Function

Using large language models as agents to assist users in completing various tasks is increasingly becoming a trend. In particular, multi-agent collaboration can address the limitations of a single model, resulting in improved accuracy and robustness.

The study titled Multi-Agent Debate (MAD) [74], demonstrates that reflection-based methods are prone to the Degeneration-of-Thought (DoT) problem. Once a large language model becomes confident in its initial solution, it struggles to generate new ideas through further reflection, even if the initial stance is incorrect. Multi-Agent Debate, inspired by another fundamental strategy of human-like problem solving, employs two agents to express their own arguments in a “tit-for-tat” state. A judge then monitors and manages the debate process to generate a final answer. The core idea of MAD is outlined as follows: (1) Through debate, the distorted thinking of one agent can be corrected by the others. (2) Through debate, the resistance of one party can be counterbalanced by the support of other members participating in the debate. (3) The answers of each agent can serve as context for mutual feedback. Therefore, MAD is less vulnerable to the occurrence of the DoT problem and can explore divergent CoTsto reduce hallucinations. As shown in

Table 2. The MAD process for an example in Andrew Ng’s lecture video on regularization (original text is in Chinese).

〈 debate topic 〉: Is regularization always necessary? Affirmative Argument: As a debater for the affirmative side, I believe that in many cases, regularization is essential. Here are several points to support this argument: 1. Preventing Overfitting: The primary goal of regularization is to prevent the model from overfitting the training data. Overfitting occurs when a model becomes too complex and overly learns the details and noise of the training data, leading to poor performance on unseen data. By introducing a regularization term, we can constrain the model parameters, thus enhancing its generalization ability. 2. Resource Optimization: Regularization helps strike a balance between model complexity and performance. By controlling the complexity of the model, we can avoid building overly complex models, which may consume excessive computational resources and time. 3. Knowledge Transfer: In transfer learning and multi-task learning, regularization promotes knowledge sharing and transfer between tasks. It prevents each task from learning independently, thus improving overall performance. 4. Algorithm Stability: Regularization can improve the stability of the algorithm, making the model less sensitive to small changes in the data and increasing its robustness. 5. Parameter Constraints: In fields such as medicine or finance, it is sometimes important for model parameters to have physical or economic significance. Regularization can help constrain these parameters within reasonable bounds. However, this does not imply that regularization is necessary in all cases. In situations where the dataset is very large, the relationship between features and the target is simple, or where there is no strict need to limit model complexity, it may be feasible to forgo regularization. Overall, considering the risk of overfitting and resource efficiency, regularization is a valuable tool in most machine learning problems and should be regarded as an important step. Negative Argument: As the opposing debater, I understand the affirmative side’s points, but I believe that regularization is not always necessary in every situation. Here are my arguments: 1. Data Quality: If the training data is large enough and evenly distributed, without noise or outliers, the model may generalize well without regularization. In such cases, the model can naturally learn the main patterns in the data without overfitting. 2. Simple Problems: For simple linear problems or problems with a clear structure, such as linear regression, the model might already have good generalization ability and may not need additional regularization to prevent overfitting. 3. Preprocessing and Feature Selection: Effective data preprocessing and feature selection can reduce noise and irrelevant features, thereby lowering the risk of overfitting, making regularization less necessary. 4. Non-Parametric Methods: Certain non-parametric methods, such as decision trees and random forests, have inherent regularization properties (e.g., limiting tree depth or using bagging) and may not require additional regularization. 5. Hyperparameter Tuning: By carefully adjusting hyperparameters like model complexity and learning rate, it is possible to find an appropriate model without the need for regularization. This is particularly feasible for small or medium-sized datasets. 6. Real-Time and Online Learning: In environments where data streams are continuously updated, regularization may not be required since the model constantly adapts to new data and doesn’t overfit early data points. In conclusion, while regularization is an effective tool for improving generalization in many cases, it is not always necessary. Whether regularization is needed depends on the nature of the data, the complexity of the problem, and the available computational resources. In some specific situations, satisfactory results can be achieved without regularization. After three rounds of debates… Arbitration opinion: Regularization is an important tool in machine learning, but its necessity depends on the specific application scenario. In many cases, regularization can improve the generalization ability and stability of the model. However, in other cases, it may be necessary to combine with other methods to achieve the best effect. Arbitration reason: In the third round of debates, both debaters further deepened their understanding of the role of regularization in machine learning, emphasizing its importance and complementarity with other technologies. Both sides agreed that regularization is a valuable tool, but not the only solution in all cases. They jointly pointed out that the decision on whether and how to use regularization should be made according to specific problems, data, and model selection.

, for questions that require a more comprehensive understanding through multiple perspectives and dialectical discussion, the debate function is clearly more suitable than the chat function. Multi-agent debate can be applied not only to STEM (Science, Technology, Engineering, and Mathematics) education but also to humanities and history studies.

We followed the idea from the work on divergent thinking based on MAD, employing three LLM-based agents to conduct multi-round debates. We utilized three roles, which are elaborated as follows: moderator, affirmative debater, and negative debater (

Table 3. Design of Multi-Agent Debate. We assigned different roles to the agents with the aim of creating an atmosphere of debate. The prompts used in our system are all in Chinese.

Roles	Prompt (English Translation)
Moderator	You are the moderator. Two debaters will participate in the debate. They will present their answers and discuss their respective viewpoints on the following topic: 〈debate topic〉. At the end of each round, you will evaluate the answers and determine which one is correct.
Affirmative Debater	You are affirmative side. Please express your viewpoints.
Negative Debater	You are negative side. You disagree with the affirmative side’s points. Provide your reasons and answer.

). Based on the given topics, both sides present their arguments and provide supporting reasons. After each round, the moderator evaluates the responses and determines the winning side. After three rounds of debate, the moderator provides a final summary evaluation, along with the corresponding rationale. The specific details of the debate result in Figure 5 can be found in Table 2, which presents an example of a specific debating process. Through the debate, the reasons for the arguments can be presented, which can inspire students’ dialectical thinking. The answers to the QA rely on the LLM and RAG technologies and are provided based on the speech information extracted from the video, the OCR information, and the object detection results of the objects in the images. Text generation in the debate is completely accomplished based on the knowledge inherent in the pre-trained large language model we use.

Regarding the implementation of multi-agents, we referred to the open-source project available at https://github.com/Skytliang/Multi-Agents-Debate, accessed on 6 August 2024. Specifically, we used the free online “Qwen2.5-72B-Instruct-Turbo” LLM API provided by Together.ai. Each agent is an instance of a large language model, and different responsibilities are assigned through different role prompts. Each agent has its own memory. The output is stored in the memory as the content of historical conversations, and its own memory is referred to when generating new responses. The moderator controls the advancement of the rounds, and its output is broadcast to everyone, indirectly affecting the next-round input of other agents. We preset the meta prompt (role setting) used by each role, the dynamic prompt template required for the moderator’s judging, and the alternative judgment prompt in case of an inability to make a ruling. We also provide the detailed step-by-step procedure of MAD in Algorithm 1.

Algorithm 1 Multi-Agent Debate Process

Input: The debate topic provided by the user and the number of rounds (n). Output: The debate topic and a complete debate process text.  Step 1: Initialize Agents  Create three agents: Affirmative, Negative, and Moderator  Inject meta-prompts for each agent role  Embed the debate topic into the preset prompt templates  Step 2: Commence the Debate  Affirmative agent presents the initial argument  Negative agent reads the affirmative argument and generates a refutation  Moderator evaluates both arguments and broadcasts a neutral judgment  Step 3: Make the Judging   while debate less than n rounds do      if Moderator identifies a clear preferred side then           Terminate the debate, summarize reasoning and final decision and go to Step 4      else           Proceed to the next round: Affirmative and Negative agent update their memory          and perform turn-based rebuttals      end if  end while  Invoke the alternative judgment prompt for Moderator:        a. Extract all answer candidates        b. Summarize reasoning and produce final decision  Step 4: Finish the Debate  Return the debate topic, arguments from both sides, full speech log, final decision, and reasoning

3.2. Participants and Design

In this section, we present a controlled study conducted to evaluate the potential benefits of providing students with video lectures, supplemented by video chat and LLM agents’ debates. Below, we outline the study’s participants, the testing conditions, and the results.

Participants. We recruited 197 undergraduate and postgraduate students to participate in the study. These students, specializing in computer science, were from Hangzhou Normal University, Nanjing University of Information Science and Technology, and Hebei University of Technology. Apart from the pre-test and post-test quiz, we also provided a follow-up questionnaire. Due to the time-consuming nature of the questionnaire, a total of 90 valid responses were ultimately returned. Undergraduate students accounted for 44.4%, and postgraduate students accounted for 55.6%.

Study design. We evaluated student performance on materials related to two machine learning topics. These topics were covered in two short online lecture videos by Andrew Ng: one on regularization (approximately 9 min) and the other on numerical approximation of gradients (approximately 6 min). For each topic, we assessed the following three learning conditions.

Testing conditions. (1) Watching only the lecture video; (2) watching the lecture video with the ability to ask questions via video-based Q&A; (3) watching the lecture video with the ability to ask questions via video-based Q&A and starting a multi-agent debate on specific topics.

All students were randomly assigned to a different learning condition for each topic. Prior to watching the lecture videos, students completed pre-test quizzes consisting of five single-choice questions and one open-ended question per topic to assess their initial understanding of the concepts. Once students had watched the video and indicated that they were ready, we administered the post-test quiz. The post-test quiz comprised the six questions from the pre-test, along with four additional, previously unseen single-choice questions. Each correct answer earned 1 point, while incorrect answers received 0 points. All the test questions were designed by teachers who teach relevant courses. Moreover, the difficulty level was determined through tests carried out by an undergraduate and a graduate student who are beginners in this area. Most students completed the test through WeChat groups, while a smaller number participated in face-to-face testing in the laboratory. To mitigate the biases introduced by heterogeneous assessment conditions, all participants, whether recruited via WeChat groups or undergoing face-to-face testing in the laboratory, were required to complete the quiz on our runnable system (https://huggingface.co/spaces/dj86/VLog4CustomLLMsPlusDebate, accessed on 1 September 2024). Our runnable system can be considered a prototype for LLM-based online video learning. It is equipped with features such as question generation based on retrieval-augmented generation and multi-agent debate functionality.

Measures. Due to differences in students’ prior knowledge, we assessed the effectiveness of the learning methods by measuring the relative improvement in test scores before and after the learning videos. We calculated the relative percentage-point increase as follows:

$$\Delta ={\displaystyle \frac{\mathrm{Post\; score}(\%)-\mathrm{Pre\; score}(\%)}{\mathrm{Pre\; score}(\%)}}$$

(1)

4. Results

4.1. Data Analysis

According to the pre-test and post-test results, as shown in

Table 4. Student performance was evaluated through pre-tests and post-tests after watching the lecture videos under each of the three learning conditions: video only, video-based Q&A, and video-based Q&A plus multi-agent debate.

Condition	N	Pre-Test ($\mathit{\mu },\mathit{\sigma }$)	Post-Test ($\mathit{\mu },\mathit{\sigma }$)	$\Delta$
V	30	$64.00\%$, 27.50	$78.97\%$, 18.83	$23.39\%$
V+Q&A	31	$63.87\%$, 22.76	$73.26\%$, 22.00	$14.70\%$
V+Q&A+MAD	29	$64.14\%$, 25.29	$83.24\%$, 20.88	$29.78\%$

, the mean score on the pre-test questions was below $70\%$. We limited our analysis to the single and multiple-choice questions, as grading the open-ended questions proved more challenging. Instead, we employed a word cloud analysis to examine the open-ended questions. After reviewing the lecture videos, students under all conditions demonstrated learning effectiveness. The learning method using MAD achieves the greatest improvement in outcomes. We employed a paired t-test and Cohen’s d [75] to analyze test correctness scores before and after “video-only” [$t\left(29\right)=-3.946$, $p<0.001$, 95% CI [−22.724, −7.210], Cohen’s $d=-0.720$], “video-based Q&A” [$t\left(30\right)=-2.765$, $p=0.005$, 95% CI [−16.320, −2.454], Cohen’s $d=-0.497$], and “video-based Q&A + MAD” [$t\left(28\right)=-3.260$, $p=0.001$, 95% CI [−31.108, −7.099], Cohen’s $d=-0.605$] interventions. These findings indicate that the “video-based Q&A + MAD” approach is more effective than “video-only” and “video-based Q&A” methods for online learning. These results support H1. Question samples and SPSS 29.0.1.0 results can be found at https://osf.io/4vt9h/, accessed on 26 November 2024.

Those in condition 2 exhibited the smallest improvement, whereas those in condition 3 showed the greatest improvement. It is generally believed that the use of large language models demonstrates an advantage over traditional video-watching methods. Table 4 indicates that the post-test performance of students in condition 3 shows a relative improvement of $29.78\%$, compared to $14.70\%$ for students in condition 2, and $23.39\%$ for students in condition 1. The quiz questions consisted of a mix of true/false questions and single and multiple-choice questions. A possible explanation is that the video-based Q&A approach might encourage students to take shortcuts, as they may not watch the video attentively and, instead, rely on asking questions directly to learn. This behavior could result in overlooking some video content, thereby leading to lower scores. In contrast, the “video-based Q&A + MAD” approach likely enhances students’ interest in learning through debates. The presentation of affirmative viewpoints may prompt students to question specific knowledge points, extending their study time and fostering a more comprehensive understanding of the video content. The pre-test and post-test questions are presented in Appendix A. As shown in Figure 6, we used violin plots to present the specific distributions of the scores of the three groups of samples in pre-test and post-test. It can be observed that the score distributions of the video-based Q&A + MAD method before and after the pre-test and post-test show significant differences, and these differences are much more pronounced compared to those of the other two groups.

Therefore, we conducted a further analysis from the perspective of students’ video-viewing duration. Specifically, we examined the variations in the duration of students’ engagement with online lecture videos under different scenarios, with and without the assistance of large language models. By comparing these durations in Figure 7, we can discern students’ interest in utilizing large language models and determine whether the use of such models can enhance students’ interest in learning videos. As shown in Figure 7, there is significant variation in the time students spent watching the video in the “traditional learning” condition. Some students spent only a few minutes, while others spent over an hour. One possible factor is that the learning material is in English, which can be challenging for Chinese students. As a result, some students may quickly skim through the video, while others might repeatedly listen or pause to focus on the video visuals. In contrast, the other two learning conditions, with the intervention of large language models, were more effective in engaging students’ interest. However, it is inevitable that some students, seeking convenience, may skip watching the video and, instead, rely solely on asking questions to learn from it. The “video-based Q&A + MAD” approach encouraged students to ask questions and initiate debates, helping them better understand the video content. These findings support Hypothesis (H2), which suggests that using video-based multi-agent debates helps students stay more focused on learning through videos.

The word frequency analysis presented in Figure 8 and Figure 9 indicates notable differences in responses to open-ended questions before and after watching the video. Post-test responses demonstrate a noticeably richer set of keywords. Additionally, inter-group comparisons reveal that under the third testing condition (“video-based Q&A + MAD”), the keywords are significantly more diverse. These findings suggest that MAD has a positive impact on online learning.

We present representative quotes from student responses to open-ended question for each group during the post-test for the topic of “Numerical Approximation of Gradients”:

Response of using the “video only” (English translation): One-sided difference refers to performing difference operations in one direction, typically used to eliminate errors in that direction. Two-sided difference involves difference operations in both directions, usually employed for further error elimination.

Response using the “video-based Q&A” (English translation): One-sided difference approximates derivatives based on changes in the function between x and $x+h$, while two-sided difference uses changes between x and $x-h$ for a more precise approximation with smaller errors.

Response of using the “video-based Q&A + MAD” (English translation): One-sided difference focuses on the deviation of a single data point from the reference point, often used for preliminary data processing. Two-sided difference, by comparing two related data points, effectively reduces noise, highlights features, and optimizes model performance.

Similarly, representative quotes from student responses to open-ended questions for each group during the post-test on the topic of “regularization” are presented as follows:

Response using the “video only” (English translation): Although L2 regularization is more commonly used in deep learning, its widespread application does not render L1 regularization obsolete. L1 regularization offers unique advantages in feature selection and generating sparse models, making it highly effective in specific scenarios. Therefore, depending on the requirements of a particular task, both L1 and L2 regularization have their irreplaceable roles.

Response using the “video-based Q&A” (English translation): In machine learning and deep learning, L1 and L2 regularization are two commonly used techniques to prevent model overfitting. L1 regularization penalizes large weights by adding the absolute values of the weights to the loss function, while L2 regularization achieves this by adding the squared values of the weights.

Response using the “video-based Q&A + MAD” (English translation): Preventing overfitting is one of the primary objectives of regularization. While both L1 and L2 regularization can mitigate overfitting, they achieve this through different mechanisms. L1 regularization reduces the number of non-zero weights, whereas L2 regularization minimizes the magnitude of the weights. This diversity allows for the selection of the most suitable regularization method based on the specific problem and dataset. L1 regularization offers an alternative strategy for combating overfitting, complementing L2 regularization rather than opposing it.

These responses illustrate that the “video-only” approach enables students to acquire basic and accurate knowledge, the “video-based Q&A” approach allows for more precise answers, and the “video-based Q&A + MAD” approach further enhances the breadth of student responses. In other words, there is an increasing depth and breadth of understanding as the learning condition becomes more interactive. These findings are also consistent with the research results of previous papers. Through AI-assisted learning technologies, high-quality personalized learning can be achieved, improving students’ abstract thinking and critical thinking [42].

4.2. Student Feedback Analysis

To further analyze the role of “video-based Q&A + MAD” in students’ learning, we administered a follow-up questionnaire. The questionnaire included eight questions, five of which were rating-based using a five-point Likert scale, while the remaining three questions solicited users’ suggestions for improvement.

Table 5. Ratings in the follow-up questionnaire.

General Satisfaction	Easy to Use	System Response Speed	Eval of Q&A	Eval of MAD
4.31	4.43	3.96	4.13	4.34

presents the detailed ratings from the follow-up questionnaire. Each item, representing an evaluation question in the follow-up questionnaire, was rated on a scale from 1 (lowest) to 5 (highest). Users who experienced the MAD functionality generally gave high scores, indicating a positive learning experience and supporting Hypothesis H2. We conducted a word cloud analysis of user improvement suggestions. The questionnaire focused on the “video-based Q&A” function, the MAD function, and the additional features that users most desired to include in future updates. According to the word cloud analysis in Figure 10, students evaluated the “video-based Q&A” function as accurate, fast, and effective in addressing the content of the videos. The multi-agent debate feature was rated as helpful, offering different viewpoints for understanding the learning material. The most requested additional features were video voice translation (as many Chinese students prefer not to listen to lengthy English lectures) and summaries of key points. These suggestions reflect users’ subjective experiences and align closely with the responses to the earlier open-ended questions.

5. Discussion

Interaction between learners and instructors, as well as among peers, plays a crucial role in helping learners construct meaning from educational materials by articulating their ideas, engaging with diverse perspectives, and refining their understanding through feedback. However, facilitating such interaction in MOOCs presents a considerable challenge. The use of LLM-driven agents to support both teaching and learning in online educational environments is expected to become a significant trend, as highlighted in various studies [42,63,64]. Experiments demonstrate that introducing multi-agent systems into online learning environments can inspire students to think critically and help deepen their understanding of course content. Although this research focuses on lecture videos in computer science, the methodology is applicable to other engineering courses.

This study does not focus on the impact of video lecture summaries on learning, as existing research [76] has already concluded that incorporating automatically generated summaries with videos can enhance students’ overall learning experience. Chen et al. [77] also emphasized that designing and implementing an interactive video lesson alone is insufficient. They highlighted that incorporating summarizing strategies and prompts can enhance video-based learning by improving learning outcomes, reducing cognitive load, increasing intrinsic motivation, fostering metacognition, and further engaging learners.

Large language models have already started to drive the transformation of the teaching model from the “teacher–student” binary structure to the “teacher–machine–student” ternary structure [78]. This structural change propels teaching from a “teaching-centered” approach to a “learning-centered” one. The educational paradigm will shift from supply-dominated to demand-dominated, reshaping and updating the role functions and competency requirements of teachers. Compared with the traditional binary structure, the technology-empowered “teacher–machine–student” ternary structure liberates teachers from inefficient, repetitive, and onerous knowledge imparting, allowing them to return to the core business of educating. Teachers can then focus on the more creative and affectionate educational values, emphasizing the cultivation of students’ autonomous learning abilities and creativity. Singapore is the first country globally to publicly support the use of ChatGPT in the education system. Its education authorities will guide teachers to effectively use intelligent tools like ChatGPT to enhance learning [79].

Our user study evaluates students using two short, pre-recorded, and carefully edited lecture videos. Future research should explore the effectiveness of multi-modal LLMs under broader conditions, including prerecorded versus live lectures, varying lengths, and diverse topics. While our experiments compared three conditions, additional conditions could provide further insights into the value of multi-agent chat and debate. For instance, limiting the time students have to answer questions, tracking the frequency of their use of agents, and surveying their willingness to use the tool before attempting the questions could yield valuable findings. It is worthwhile to explore the possibility of customizing the behavior of multi-agent systems according to students’ knowledge levels or their behaviors when watching video materials. Designing a closed-loop system to generate dynamic feedback is of great significance. Multi-agents integrated into formal educational settings, such as online lectures, should be able to support learners for long-term use. The design principles of digital tutors and the technologies of AI-augmented classrooms shed light on our future research [42,80]. Additionally, exposing students to a different condition, analyzing their changes by comparing their results and feedback, and collecting students’ questions to analyze the patterns of their inquiries will all be highly meaningful future work. Moreover, this study focuses on students’ immediate performance after learning. However, it would be more valuable to evaluate long-term memory retention, changes in learning habits, or the impact on metacognition. Therefore, we plan to incorporate the evaluation of long-term memory retention, changes in learning habits, and the influence on metacognition in our future research.

Additionally, due to GPU hardware limitations, the sample size of participants was relatively small, and most assessments were conducted as independent online tests. In future studies, we will focus on improving the experimental design and selecting model parameters that match the GPU computing power. On one hand, we can integrate regular assignments in computer teaching to expand the sample size of the questionnaire. On the other hand, in the future, we can rely on the university’s servers to deploy open-source models with relatively low hardware consumption, such as the recently popular DeepSeek [81], to break through the computing-power limitations of GPUs. To date many large language models have introduced small models of 7B and 14B, as well as distilled models. When combined with RAG technology, it is believed that these models will be sufficient to handle answer generation for knowledge-based questions in most scenarios in the future.

6. Conclusions

This study focuses on video-based online learning, comparing the traditional “video-only” learning approach with methods that incorporate large language models, specifically “video-based Q&A” and “video-based multi-agent debate” approaches. These methods represent some of the latest trends in LLM applications, offering a more interactive learning experience compared to the traditional unidirectional MOOC format. Additionally, generative responses based on retrieval-augmented generation technology provide students with more comprehensive knowledge summaries. In summary, the method proposed in this paper utilizes multi-agent debates to provide students with reference answers from different perspectives, aiming to deepen students’ understanding of knowledge and cultivate their critical thinking. However, compared with in-person teaching by teachers, it lacks the cultivation of other high-order thinking abilities, such as deep thinking and abstract thinking. Additionally, it fails to offer life thinking, life enlightenment, or certain enlightenment and thinking beyond the realm of artificial intelligence. This is also one of the common drawbacks of using AI tools for learning.