Evaluating the Use of the SCAI Chatbot to Support Student Engagement and Academic Preparation in Secondary-School Physics

Abstract

Despite the widespread adoption of educational technologies, a critical gap remains in Generative Artificial Intelligence (GenAI) tools that are both pedagogically grounded and responsive to the practical needs of teachers and students. This study examines the use of the Scaffolding Cognitive Artificial Intelligence (SCAI) chatbot to support students’ pre-class preparation and classroom engagement in secondary-school physics. The SCAI chatbot was designed by applying scaffolding theory and aligning chatbot interactions with the educational curriculum, with the aim of providing adaptive explanations, formative questioning, and teacher-facing preparation analytics. A Design Science Research (DSR) approach with a mixed-methods design was employed. Quantitative data were collected through pre-class preparation tests, while system-generated log data captured student interaction patterns. Qualitative data were obtained through semi-structured interviews with both teachers and students. The findings suggest that SCAI-supported preparation was associated with higher student preparation and more active engagement when compared with textbook-based preparation. Interview and log data further indicated that the chatbot helped simplify difficult concepts, support students’ confidence, and provide teachers with useful information about students’ progress before classroom instruction. However, because the study was conducted in a limited educational context and the intervention combined AI-supported interaction, scaffolding, flipped preparation, and teacher analytics, the findings should be interpreted as exploratory evidence of a meaningful association rather than definitive proof of causation. Overall, the study contributes to understanding how a scaffolding-based GenAI chatbot can support pre-class preparation and teacher-informed instructional planning.

1. Introduction

Artificial intelligence (AI) is transforming education by enabling personalized learning, adaptive support, and intelligent feedback that are difficult to achieve in traditional classroom settings (Bressane et al., 2024). Recent studies suggest that AI-based adaptive learning methods, personalized feedback, and interactive tools can support student engagement, motivation, and achievement in different learning environments (Yaseen et al., 2025). In addition, academic performance may improve when pedagogical principles are integrated into the design of educational tools (Mirdad et al., 2024).

Conversational agents and GenAI also show considerable promise in education. They can support creativity, computational thinking, and knowledge building through adaptive conversation and dynamic task feedback (Jiang et al., 2024). These developments highlight the value of combining AI with educational scaffolding, as such integration can provide customized and responsive support tailored to the learner’s needs (Ward et al., 2024). Research on flipped classroom models further supports this direction by showing that pre-class preparation activities can improve both student engagement and conceptual understanding (Yao et al., 2025). When combined with AI-supported interaction, flipped learning may therefore become a practical approach to improving student readiness before classroom instruction begins. Recent research does not position AI as a replacement for teachers, but rather as a tool that can support teaching and learning (Chen et al., 2024). In this context, teachers remain central to the educational process because they interpret learning data and adapt instruction to students’ different needs (Liao et al., 2024). Human-AI collaboration is therefore essential for creating responsive learning environments that support both student learning and teacher decision-making.

At the same time, the educational use of GenAI requires careful attention to potential risks. These include inaccurate or unsupported responses, student overreliance on automated explanations, unequal access caused by differences in digital literacy or internet availability, and the ethical use of student interaction data. For this reason, GenAI tools should be designed as supportive educational systems that remain aligned with curriculum objectives, teacher judgment, and responsible data practices.

In this study, AI is used as the broad term for technologies that provide automated, adaptive, or intelligent educational support. GenAI refers more specifically to AI systems that can generate explanations, questions, feedback, and learning support in response to learner input. The term “chatbot” refers to the conversational interface through which students interact with the system. SCAI refers to the specific curriculum-aligned GenAI chatbot developed and evaluated in this study. Therefore, the findings are interpreted in relation to the SCAI-supported preparation model rather than AI tools in general.

Despite these developments, limited empirical attention has been given to curriculum-aligned GenAI chatbots that apply scaffolding theory to support pre-class preparation and teacher-informed planning in secondary-school physics classrooms. Many studies focus on student performance and participation, but pay less attention to how teachers can use early learning data to understand student preparation before class. This gap is especially important in subjects where students may hesitate to participate and may begin lessons with different levels of conceptual understanding. To address this issue, the present study designs, implements, and evaluates the SCAI chatbot, an AI-based learning tool that supports pre-class preparation through a scaffolding strategy. The system is designed to provide adaptive lesson explanations, question generation, and teacher-facing analytics that help identify student readiness before classroom instruction.

This study makes four main contributions. First, it operationalizes scaffolding theory within a curriculum-aligned GenAI chatbot designed to support students’ pre-class preparation in secondary-school physics. Second, it extends chatbot design beyond student interaction by integrating a teacher analytics dashboard that provides early visibility into student preparation, strengths, weaknesses, and response patterns before classroom instruction. Third, it contributes empirical evidence from secondary-school physics in Saudi Arabia. Fourth, it compares SCAI-supported preparation with traditional textbook-based preparation. Together, these contributions position SCAI as a pedagogically grounded GenAI artifact and a practical tool for teacher-informed planning in a flipped classroom.

Consequently, this study addresses the following research question:

• Q1: How does a scaffolding-based SCAI chatbot support students’ pre-class preparation in a flipped classroom setting?

Based on this research question, the study examines the following hypotheses.

H1. 

The use of the SCAI chatbot will be associated with higher levels of pre-class preparation and classroom participation compared to traditional textbook-based preparation.

H2. 

The use of the SCAI chatbot will be associated with reduced participation barriers, such as hesitation and lack of confidence, and with more active student engagement.

H3. 

The data and analytics generated by the SCAI chatbot will be perceived by teachers as useful for identifying students’ preparation levels and planning appropriate instructional activities.

Section 2 presents the theoretical foundation. Section 3 presents the related work. Section 4 is devoted to the materials and methods applied. Section 5 presents the data analysis, while Section 6 presents the results. Section 7 discusses the research results. The conclusion and future work are presented in Section 8.

2. Theoretical Foundation

This section presents the theoretical foundation of the study and explains how it informs the design of the SCAI chatbot. The study is grounded in the flipped classroom as the instructional model and scaffolding as the teaching strategy. Together, these perspectives explain how the SCAI chatbot supports students before class, improves preparation, and provides teachers with early information about student learning needs.

2.1. Teaching Strategy and the Flipped Classroom

Teaching strategies are a fundamental component of effective educational practice because they shape how learners engage with content, build understanding, and develop higher-order thinking skills. Among these strategies, educational scaffolding provides structured and gradual support that helps students move from assisted learning toward greater independence, while collaborative learning, inquiry-based learning, and formative feedback further promote active participation and deeper understanding. In the flipped classroom, part of the learning process takes place before the lesson, often through videos, readings, or digital tools, while classroom time is used for deeper explanation, discussion, and active participation (Yao et al., 2025). The SCAI chatbot supports this model by shifting part of the learning process to the pre-class stage. Students use the chatbot to review lesson content, receive explanations, answer adaptive questions, and build background knowledge before entering the classroom. This pre-class interaction is important because flipped learning depends on students’ preparation and their active responsibility for initial learning. By giving students structured support before the lesson, SCAI may reduce the difficulty of entering classroom discussion without prior understanding. At the same time, the SCAI chatbot provides teachers with early information about student progress and learning gaps, which supports more informed instructional planning. In this sense, the flipped classroom is not only the setting for the study but also a key reason why the chatbot is educationally useful.

2.2. Scaffolding Theory

This study is based on scaffolding theory, which is derived from Vygotsky’s concept of the zone of proximal development (ZPD) (Vygotsky, 1978). Scaffolding refers to the process of providing temporary and structured support as students develop new knowledge and skills. In traditional classroom settings, this support often begins with the introduction of a new concept, followed by a series of questions that move from simple to more complex levels. This gradual progression helps teachers assess students’ understanding and adjust the depth of explanation according to their needs. In the present study, scaffolding is applied through the design of the SCAI chatbot. The system begins with the lesson topic, checks the student’s prior knowledge, provides simple explanations, and then offers questions and feedback based on the student’s level of understanding. If the student needs more help, the chatbot provides further explanation before moving to the next step. If the student demonstrates understanding, the system moves forward to the next question or learning goal. This sequence reflects contingent scaffolding because the amount and type of support change according to the student’s response. It also reflects a limited form of fading, as the support is reduced when the student is able to continue and increased when additional explanation is needed. In this way, SCAI is designed to support student agency by requiring students to select learning goals, respond to questions, request support, and actively progress through the lesson. Figure 1 shows how scaffolding theory is operationalized in the SCAI chatbot system through prior-knowledge elicitation, adaptive support, preparation analytics, and teacher planning.

Figure 1. Conceptual model showing how SCAI applies scaffolding through prior-knowledge checking, adaptive support, preparation analytics, and teacher-informed planning.

However, AI-based scaffolding remains limited because the system may not fully capture students’ emotional states, classroom context, or deeper misconceptions in the same way that the teacher can. Therefore, SCAI is positioned as a supportive tool that complements teacher judgment rather than replaces it.

3. Related Work

Recent research has shown that artificial intelligence (AI) can play an important role in improving student engagement, motivation, and learning outcomes. Adaptive learning techniques and intelligent feedback systems have been found to support personalized learning environments that encourage active participation and sustained motivation (Yaseen et al., 2025). Other studies have shown that AI-supported methods can help students develop study strategies that are more responsive to individual learning differences, which may contribute to improved academic performance (Bressane et al., 2024). In addition, AI-based e-learning systems have been associated with greater participation and higher levels of student satisfaction when they are effectively integrated into an educational setting (Eltahir & Babiker, 2024). Together, these studies suggest that AI-driven tools can support performance and classroom engagement (Mirdad et al., 2024).

Chatbot-based learning environments represent one practical application of AI in education, particularly when conversational interfaces are used to provide explanations, feedback, recommendations, or learning support. For example, Tayseer, an AI-powered Arabic chatbot framework, was reported to support student interaction and provide responsive academic assistance, which may contribute to motivation and comprehension (Xu et al., 2024). Similarly, AI-supported learning systems have been reported to support students’ determination and growth mindset (Kim & Su, 2024). Studies have also shown that integrating natural language processing into AI-based chatbots can improve accessibility and learning efficiency by providing academic guidance and support to the learner (Fryer et al., 2017). In addition, chatbots that provide educational recommendations may personalize learning pathways and offer students access to learning content beyond classroom time (Xu et al., 2024). These findings indicate that conversational AI can support self-regulated learning and pre-class preparation when it is aligned with clear educational goals.

Previous work has shown that scaffold-based support strategies can improve learner confidence and learning outcomes (Alghamdy, 2024). Other studies have explored how ChatGPT 3.5-based support can foster computational thinking and how domain-based support can enhance creativity and skill development (Su et al., 2024). More recent research has combined scaffolding with generative AI systems and found that adaptive scaffolding can improve conceptual understanding by aligning support with students’ current level of understanding of students (Yan et al., 2025). These findings suggest that AI-supported learning tools may be more educationally useful when they are combined with structured pedagogical strategies rather than used as general conversational systems. AI tools also offer value in supporting educational decision-making through learning analytics. Previous studies have shown that AI-based analytics can help identify conceptual gaps, visualize student performance, and provide teachers with useful information before instruction begins (Ng et al., 2024; Xu et al., 2024). In flipped classroom settings, AI-supported pre-class materials have been associated with higher levels of student engagement and more efficient classroom learning (Yao et al., 2025).

Taken together, previous studies show three main directions in AI-supported education: conversational support for learners, scaffolding-based assistance, and learning analytics for instructional decision-making. However, these directions are often examined separately. Some studies emphasize chatbot interaction and academic support, while others focus on adaptive scaffolding or teacher-facing analytics. In contrast, SCAI integrates these elements within one pedagogical model by combining curriculum-aligned chatbot interaction, scaffolding-based pre-class preparation, adaptive questioning, and teacher-facing analytics. This integration is important because it links student preparation before class with teacher planning before instruction, rather than treating chatbot interaction as an isolated learning activity.

However, these benefits depend on careful educational design, because chatbot-based learning may also create risks related to response accuracy, learner dependence, unequal access, cognitive overload, and the interpretation of learning analytics. Despite the contributions of previous studies, several gaps remain in the literature. First, much of existing research focuses on AI-supported students during or after instruction, while less attention has been given to structured pre-class preparation. Second, limited attention has been paid to how AI-generated analytics can support teacher planning before lessons begin. Third, relatively few studies have directly compared AI-supported preparation with traditional textbook-based preparation. Finally, only a small number of studies have examined whether AI tools can reduce hesitation, anxiety, and other barriers to equitable classroom participation. These gaps provide the foundation for the present study, which evaluates SCAI as a curriculum-aligned, scaffolding-based GenAI chatbot designed to support pre-class preparation while also providing teachers with insight into student preparation before classroom instruction.

4. Materials and Methods

This study adopted Design Science Research (DSR) with a mixed-methods design to support the systematic development, implementation, and evaluation of the SCAI chatbot for secondary-school physics education. DSR was selected because it supports both the development of practical educational artifacts and the evaluation of their usefulness in a real learning context. The mixed-methods design allowed the study to combine quantitative evidence of students’ pre-class preparation with qualitative insights from teachers and students regarding their experience with the system. The quantitative component examined students’ preparation scores and behavioral log data, while the qualitative component explored participants’ perceptions of the system and its role in supporting learning and teaching.

4.1. Research Design

The DSR methodology provides an appropriate approach to designing and implementing SCAI as a GenAI chatbot that applies educational scaffolding to support pre-class preparation (Ojeniyi et al., 2019). The study followed the main stages of DSR: awareness of the problem, suggestion, development, and evaluation, as shown in Figure 2. In the awareness stage, a literature review and semi-structured interviews with three experienced female physics teachers were conducted to identify barriers to student participation and to define the educational needs that the system should address. The participating teachers had a mean teaching experience of 12.33 years (SD = 3.4), and each interview lasted an average of 26 min (SD = 5.77). These interviews helped identify key issues such as low participation, hesitation, difficulties in understanding abstract concepts, and the need for early insight into student preparation. In the suggestion stage, the findings of the literature review and teacher interviews were translated into design requirements that emphasize usability, adaptability, immediate feedback, Arabic-language support, teacher reporting, and scalability. In the development stage, these requirements were converted into system features and functions, followed by further refinements to improve the quality and relevance of the system’s responses. In the evaluation stage, the system was first assessed by pilot usability testing and then evaluated in classroom practice.

Figure 2. Design Science Research (DSR) process used in the development and evaluation of SCAI.

4.2. Study Setting and Participants

The study was conducted during the first semester of the 2025–2026 academic year in a girls’ secondary school in Jeddah, Saudi Arabia. The initially enrolled cohort comprised 74 third-year secondary-school students studying physics. For the intervention, 30 students actively participated in the study. Of these, 15 students were assigned to the experimental group, which used the SCAI chatbot for pre-class preparation, and 15 students were assigned to the control group, which used traditional textbook-based preparation.

Both groups were drawn from the same grade level and school context, studied the same physics units, completed the same pre-class readiness assessments, and were supervised by the same physics teacher in order to reduce variation in lesson delivery. The intervention lasted four weeks and was implemented in two physics units. However, because the students were not randomly assigned and no separate baseline test of prior physics achievement, motivation, or digital literacy was administered, group comparability should be interpreted in terms of shared instructional context rather than full baseline equivalence.

During the study period, a total of 278 response-level observations were obtained from 11 formative assessments, including 133 responses from the experimental group and 145 responses from the control group. These observations represent repeated preparation responses collected from the same active participants across multiple assessments. Therefore, the 278 observations should not be interpreted as 278 independent student cases. The study should be understood as a small-scale classroom implementation rather than a large-scale evaluation.

4.3. SCAI System Architecture and Technical Design

The SCAI system was implemented as a web-based educational application designed to integrate AI-supported scaffolding, student interaction, teacher analytics, authentication, and secure data management. The system followed a modular architecture to support usability, scalability, maintainability, and adaptability in an educational context.

The system consisted of six main components: the AI scaffold chatbot, the student interface, the teacher dashboard, the data management layer, the authentication service, and the evaluation module. The AI scaffold chatbot generated adaptive explanations, hints, examples, and formative questions based on the selected lesson objective and the learner’s response. The student interface allowed learners to log in, select the relevant physics content, choose the lesson objective, and interact with the chatbot. The teacher dashboard presented student progress indicators, generated questions, correct answers, and areas of strength and weakness to support pre-class instructional planning.

The frontend was implemented using React 18.3.1, Vite 6.0.11, Tailwind 4.0.0, and Lucide 0.474.0 React to support responsive interaction and a clear user interface. The backend was implemented using FastAPI, AI integration services, SMTP, and token authentication to manage chatbot logic, data routing, authentication, and communication between system components. The analytics module used Recharts and teacher-reporting interfaces to generate real-time visual summaries of student performance, while the database module used PostgreSQL/SQLite to store interaction logs, performance data, and teacher feedback securely.

During runtime, the student interface, the AI component, the teacher dashboard, and the database communicated through RESTful APIs and secure data communication protocols. This structure allowed student responses to be processed by the AI component, stored in the database, analyzed by the evaluation module, and displayed through the teacher dashboard.

The generative AI component of SCAI was implemented using the OpenAI API. The system did not retrain or fine-tune the OpenAI model using a separate dataset. Instead, SCAI guided the generated explanations, questions, feedback, and response analysis through curriculum-aligned prompts linked to the selected physics lesson objectives. This design helped to keep the chatbot interaction aligned with the physics curriculum and the instructional goals of the study.

4.4. Intervention Procedure

The SCAI chatbot was designed to support pre-class preparation through adaptive interaction. Students accessed the system through a web interface by logging in and selecting the relevant chapter, lesson, and learning goal. Each interaction began with a welcome message and the lesson topic, as shown in Figure 3. The SCAI chatbot then asked students about their prior knowledge and allowed them to choose a level of interaction, as shown in Figure 4. A simple explanation was provided first, followed by a question to assess understanding. If the student answered correctly, the system moved to the next question or learning goal, representing a reduction in support when the learner demonstrated readiness to continue. If the student answered incorrectly or requested further support, the SCAI chatbot provided additional explanation and questioning aligned with the student’s current level of understanding. At the end of the lesson, the SCAI chatbot generated a real-time dashboard showing student progress, the number of generated questions, the number of correct answers, and reports highlighting strengths and weaknesses, as shown in Figure 5. These indicators were intended to help the teacher identify students’ preparation levels and areas of difficulty before classroom instruction. This allowed the teacher to direct class time in a way that suits the students’ needs and provided the necessary support for explanation. At the beginning of each lesson, both groups completed a short readiness quiz before classroom instruction. The quiz questions were worth one mark each, except for the third question, which was worth 1.5 marks because it was designed to assess higher-order understanding and required more complex reasoning than the other quiz questions, which focused on basic and intermediate comprehension. These tests were used to assess students’ preparation prior to the lesson.

Figure 3. Students interface of SCAI, through which learners access the system, select physics content, and begin pre-class preparation.

Figure 4. Interaction-Level Selection screen showing how students choose the level of support before receiving scaffolded explanations and questions.

Figure 5. Teacher dashboard showing student progress, generated questions, correct answers, and areas of strength and weakness for instructional planning.

4.5. Data Collection

Three main sources of data were used. First, quantitative preparation data were collected through a short pre-class test administered to both groups at the start of each lesson. These assessments were used to examine students’ preparation before classroom instruction. Second, system-generated behavioral log data was collected from the SCAI chatbot. These logs included the number of questions generated and the number of correct answers. The data were exported from the system in CSV format for analysis.

Third, qualitative data were collected through semi-structured interviews conducted with the physics teacher and all participating students from both groups after the intervention. The student interview sample therefore included all 30 active participants: 15 students from the SCAI group and 15 students from the textbook-preparation group. Because all participating students were interviewed, no separate student-selection procedure was applied. The interviews lasted between 20 and 30 min and were conducted in Arabic, which was the participants’ native language and the language of classroom instruction.

The interview responses were transcribed for analysis. The transcripts were analyzed in Arabic, and the selected quotations used in the manuscript were translated into English for reporting. The interviews explored students’ preparation, confidence, content clarity, motivation, learning experience, and views on the usefulness of the SCAI chatbot. No independent second coder, external checking procedure, or formal audit process was used. Therefore, qualitative findings are interpreted cautiously as exploratory evidence of the perceptions and experiences of the participants.

4.6. Pilot Evaluation

Before the main classroom implementation, a pilot usability assessment was conducted to evaluate the clarity and usability of the SCAI chatbot. A five-point Likert-scale questionnaire was administered to 16 students, with response options ranging from 1 (strongly disagree) to 5 (strongly agree) (Likert, 1932). The questionnaire assessed the usability of the interface, the clarity of the content, the adaptability of the system, and general satisfaction. Because the pilot evaluation was conducted with a small sample and item-level reliability analysis was not available, the pilot findings are interpreted cautiously as initial usability evidence. The pilot results showed a high initial acceptability, with an average satisfaction score of 4.80 (SD = 0.14). The mean scores included interface usability (5.00), content clarity (4.75), response accuracy (4.56), and system adaptability (4.81). These results suggest that the students found the system easy to use, clear, and generally helpful before the main implementation. However, the high rating may also have been influenced by novelty effects, positive response tendencies, or the short duration of initial exposure to the chatbot.

4.7. Measures

In this study, student participation was treated as a multidimensional construct rather than a single outcome. It was operationalized through three forms of evidence: pre-class preparation, measured through short preparation tests administered before each lesson; behavioral engagement, represented by SCAI log indicators such as the number of generated questions and correct answers; and affective and participatory engagement, explored through interviews focusing on confidence, hesitation, motivation, and classroom participation. Therefore, quantitative scores are reported as preparation-related measures, while a broader engagement is interpreted through the combined evidence of the log and interview.

4.8. Ethical Considerations and Data Protection

Ethical considerations were addressed because the study involved secondary-school students and collected preparation scores, chatbot interaction logs, dashboard-related learning analytics, and interview data. The study was approved by the Ministry of Education in the Kingdom of Saudi Arabia under approval number 4601000876. Informed consent was obtained before participation, and all student data were handled confidentially. The dataset used for analysis was anonymized so that students were not identified by name in the reported results. Access to the data was limited to the research team, and the findings were reported in aggregate form to protect the privacy of the participants. Chatbot interaction logs and teacher-dashboard data were used only for research and instructional analysis related to the study objectives. No individual student was identified in the published findings.

5. Data Analysis

Quantitative data were analyzed using descriptive statistics to calculate average preparation scores. Because the pre-class preparation data included repeated responses from the same students in multiple formative assessments, student-level average preparation scores were first calculated for each participant. The group comparisons were then conducted using these student-level averages instead of treating all response-level observations as independent cases.

The main inferential analysis was conducted as a single group comparison using student-level average preparation scores. Therefore, no multiple comparison correction was applied to the primary analysis. Comparisons at the lesson level or the assessment level were treated as descriptive or exploratory and were not used as the main evidence for the conclusion of the study.

The Kolmogorov-Smirnov test was used to examine whether the score distributions were normal (Kolmogorov, 1933). Given the small sample size and the exploratory nature of the study, the Mann-Whitney U test was used to compare the experimental and control groups (Mann & Whitney, 1947). The effect size and 95% confidence intervals were reported to support the interpretation of practical significance, rather than relying only on statistical significance.

Behavioral log data from the SCAI chatbot were analyzed to examine patterns during chatbot-supported preparation. These logs included the number of generated questions and the number of correct answers. For each student, an accuracy rate was calculated by dividing correct answers by generated questions and multiplying the result by 100. The students were then classified into three proficiency levels, where scores below 40% indicated substantial difficulty answering the chatbot questions correctly, scores between 40% and 70% indicated developing accuracy, and scores above 70% indicated relatively consistent accuracy. These thresholds were used as practical interpretive categories to summarize the accuracy of chatbot interactions in the context of the study.

Descriptive statistics and correlation analyzes using Spearman and Pearson tests were conducted to examine performance patterns and the relationship between the number of generated questions and the correct answers, as it is appropriate to assess monotonic relationships when data are not normally distributed (Schober et al., 2018). This analysis was interpreted as evidence of association only and was not used to make causal claims about the effect of question generation on performance.

Qualitative interview data were analyzed inductively using the thematic analysis approach outlined by Braun and Clarke (2006). The analysis began with repeated reading of the interview responses to become familiar with the data. The initial codes were then generated from meaning segments related to preparation, confidence, participation, understanding, motivation, technical challenges, and teacher planning. Similar codes were grouped into candidate themes that were reviewed and refined to ensure that they accurately reflected the responses of the participants.

The final themes were organized separately for the teacher, the experimental group, and the control group to compare the preparation supported by SCAI and the preparation based on textbooks. Semi-structured interviews were conducted with the physics teacher and the selected students after the intervention to explore their experience with preparation, confidence, participation, content clarity, and perceived usefulness of the SCAI chatbot. The interview responses were reviewed and organized by participant group, including the teacher, the SCAI group, and the textbook-preparation group. Themes were developed through repeated reading, initial coding, grouping of related codes, and refinement of candidate themes. Because the coding was conducted by the researcher, the qualitative findings are interpreted with caution, and the absence of independent coding is acknowledged as a limitation.

6. Results

The study began with an enrolled cohort of 74 students of the same grade level. Of these, 30 students actively participated in the intervention, with 15 students in the SCAI group and 15 students in the textbook-preparation group. Both groups were taught under the same instructional conditions, completed the same test questions, studied the same physics units, and were supervised by the same teacher. These shared conditions helped reduce instructional variation between groups, although they do not fully establish a baseline equivalence.

6.1. Pre-Class Test Scores

A total of 278 valid observation levels of responses were obtained in 11 formative assessments, including 133 responses from the SCAI group and 145 responses from the textbook-preparation group. These values are reported as descriptive statistics at the response level because they represent repeated responses from the same participating students. For the main inferential analysis, each student’s average preparation score across the completed formative assessments was calculated, and the two groups were compared using these student-level averages. Because the primary analysis used each student’s average preparation score across the completed formative assessments, the reported group comparison represents one main inferential test rather than separate tests for each of the 11 assessments.

The descriptive statistics indicated that the SCAI group achieved a higher average preparation score (M = 2.14, SD = 0.28) than the textbook-preparation group (M = 1.89, SD = 0.29), as shown in Table 1. However, these values should be interpreted descriptively because they are based on observations at the level of repeated responses rather than on independent cases at the level of students. Although the median was also higher in the SCAI group, the relatively small difference between the two medians indicates that the practical difference between the groups was modest.

Table 1. Descriptive Statistics of Student Preparation Scores.

Group	N	Mean	SD	Median	Q1	Q3	Min–Max
AI group	15	2.14	0.28	2.18	1.99	2.35	1.67–2.50
Traditional group	15	1.89	0.29	1.95	1.62	2.14	1.4–2.30
Total	30	—	—	—	—	—	—

Note: N refers to students, not response-level observations. Descriptive statistics were calculated using each student’s average preparation score across Units 3 and 4. The effect size is reported as the rank-biserial correlation.

The Kolmogorov–Smirnov test did not indicate a significant departure from normality for the average preparation scores at the student level in either group. The test was evaluated using the D statistic, which indicates the distance between the observed and normal distributions, together with the corresponding p-value, where p > 0.05 indicates no significant deviation from normality. The results were as follows: SCAI group D(15) = 0.174, p = 0.695; textbook-preparation group, D(15) = 0.172, p = 0.704. However, because each group included only 15 students, the Mann–Whitney U test was used as a cautious non-parametric comparison between the two groups. The results showed a statistically significant difference in the preparation scores between the SCAI and textbook-preparation groups (U = 167.00, p = 0.025). The effect size was 0.484, with a mean difference of 0.245 points and a confidence interval of 95% of [0.048, 0.439].

Although the SCAI group showed higher average preparation scores than the textbook-preparation group, the practical magnitude of the difference should be interpreted cautiously. Therefore, the quantitative result should be interpreted together with the effect size, confidence interval, behavioral log data, and interview findings rather than as evidence of a large standalone performance gain.

6.2. Behavioral Log Results

Behavioral log data from the SCAI chatbot showed a variation in the frequency of question generation and the number of correct answers among students in the SCAI group. Scatter plot analysis showed a positive relationship between the number of questions generated and the number of students who achieved accuracy levels between 60% and 80%, while a smaller number of students had accuracy levels below 40%, as shown in Figure 6. The Spearman correlation analysis also showed a strong positive relationship between the number of generated questions and the number of correct answers (Spearman’s rs = 0.78, p < 0.01), suggesting that a more active interaction with the chatbot was associated with better performance. However, this association should not be interpreted as evidence that generating more questions caused stronger performance, because motivation, study habits, or digital confidence may also have influenced students’ interaction patterns. Therefore, behavioral log findings are interpreted as supportive evidence of active engagement with SCAI rather than as independent proof that question generation alone improved student performance.

Figure 6. Generated Questions and Student Accuracy Rates.

6.3. Interview Results

To complement the quantitative findings, semi-structured interviews with the teacher and students were conducted to explore their experiences with the SCAI chatbot and their views on preparation, participation, and classroom learning. Qualitative data were analyzed inductively to identify recurring themes grounded in the responses of the participants. The interview findings are presented in three groups: the teacher, the SCAI group, and the textbook-preparation group.

For the teacher interview, six main themes were identified: positive and enjoyable experience, growth in students’ motivation and confidence, improved understanding and academic performance, AI as a source of updated information beyond the teacher’s own material, technical and environmental challenges, and recommendations for further improvement. The teacher described the experiment as enjoyable and reported that the students became more eager to participate over time. According to the teacher, shy students gained confidence and students understood lessons more easily, the teacher stated. “It was a beautiful and enjoyable experience. At first, we had difficulty convincing the students to participate, but they wanted to participate themselves later. The teacher also noted that the SCAI chatbot sometimes provided additional updated information beyond the teacher’s original material. The teacher stated, “AI provided students with new and updated information, sometimes newer than the teacher’s own materials.” At the same time, internet problems and students’ competing commitments during the exam period limited participation in some cases. The teacher suggested that future implementation could be improved by using the application at school rather than only at home and by adding a teacher-side chat feature to support lesson preparation.

For the experimental group, six themes also emerged: enjoyable and positive learning experience (n = 5), simplification and clarification of complex information (n = 6), increased confidence and preparation (n = 5), technical and time-related challenges (n = 3), suggestions for improvement, and preference for digital learning over traditional textbooks (n = 6). The student commonly described the experience as “fun,” and “we really enjoyed using it.” Many reported that the SCAI chatbot made lessons easier to understand by providing examples that clarify difficult concepts. Some students stated, “The book is complex, but the app simplified the information and gave us examples.” Students also stated that “We had background on the lesson, so we felt more confident answering in class.” However, some students reported that “sometimes the internet was weak” or “we felt it was extra work when we had many tasks.” Their suggestions for improvement included “adding simulation tests so we can practice for exams” and “including more videos and visuals to explain the lessons.” A notable theme was, as one student said, “We liked the app more than the book.”

For the control group, five themes were identified: recognition of the value of preparation (n = 4), commitment and performance through traditional study (n = 3), different levels of motivation and participation, preference for collaborative learning (n = 5), and openness to technology integration (n = 4). The student in this group recognized that “preparation is important; it helps me understand what the teacher is explaining.” Some students reported that regular textbook preparation supported strong academic performance, and in some cases their results were comparable to those of students in the AI-assisted group. At the same time, some students described traditional preparation as “boring” or “repetitive,” while others reported inconsistent study habits. Several students expressed a preference for collaborative learning with friends, noting that “It is more fun when we solve questions with our friends; discussion helps us understand better.” In addition, some students showed interest in trying the SCAI chatbot and believed that technology could make learning more interactive and easier to follow.

In general, the interview findings suggest that SCAI influences engagement not only by providing additional content but also by changing the way students approached preparation before class. The clarity of the SCAI group’s comments, confidence, and enjoyment indicates that the chatbot may have reduced some barriers to participation by making physics content more accessible before classroom discussion. The teacher-facing dashboard represents an additional component of the SCAI-supported preparation model. The dashboard summarized student progress, generated questions, correct answers, and areas of strength and weakness before classroom instruction. Based on the teacher interview, this feature was perceived as useful for identifying students’ readiness levels and possible learning gaps before the lesson began. In this way, SCAI may help connect students’ pre-class preparation with teacher-informed planning. However, because the dashboard evidence was based primarily on qualitative perception data, these findings should be interpreted as exploratory evidence of perceived instructional usefulness rather than direct evidence of measurable changes in teacher practice.

7. Discussion

This study examined whether the SCAI-supported preparation model was associated with students’ pre-class preparation in a flipped secondary-school physics classroom. Using student-level averages across two units, the SCAI group showed higher average preparation scores than the textbook-preparation group. The Mann-Whitney U test indicated a statistically significant difference between the two groups, with a moderate rank-biserial effect size. However, the mean difference was modest. Therefore, the result should be interpreted as evidence of a meaningful but cautious association between SCAI-supported preparation and student readiness-related outcomes, rather than as definitive proof that SCAI alone caused the improvement.

The findings can also be understood through the theoretical logic of scaffolding and flipped learning. SCAI was designed to support students before class by helping them select lesson goals, activate prior knowledge, receive explanations, answer questions, and obtain additional support when needed. This sequence reflects contingent scaffolding because the level of support changes according to the learner’s response. It also reflects a limited form of fading, as students move forward when they demonstrate understanding and receive further explanation when they need additional support. In the context of the flipped classroom, this structured pre-class interaction may help students enter the lesson with better preparation and greater confidence.

At the same time, the findings of the textbook-preparation group show that traditional preparation can also support learning when students are motivated and consistent. Some students in the control group showed that they recognized the value of preparing before class and reported that textbook-based preparation helped them follow the lesson. This means that the study does not position SCAI as a replacement for traditional learning. Rather, SCAI is better understood as a complementary preparation model that can provide additional interactive support through scaffolded explanations, immediate feedback, and adaptive questioning.

The teacher-facing dashboard is another important element of the SCAI-supported preparation model. The dashboard summarized student progress, generated questions, correct answers, and areas of strength and weakness before classroom instruction. This feature may help teachers identify students’ readiness levels and possible learning gaps before the lesson begins. In this way, SCAI connects students’ pre-class preparation with teacher-informed planning. However, the dashboard findings should be interpreted as evidence of perceived instructional usefulness rather than direct evidence of measurable changes in teacher practice.

Several limitations should also be acknowledged. First, the study was conducted with a small active sample of 30 students, with 15 students in each group. This limits statistical power and reduces the extent to which findings can be generalized to other schools, grade levels, or subject areas. Second, the study used a quasi-experimental, non-randomized comparative design. Although both groups shared the same grade level, physics content, teacher supervision, and assessment conditions, the absence of random assignment means that selection bias cannot be ruled out. Third, no separate baseline measure of prior physics motivations, digital literacy, or study habits was administered. Therefore, group comparability should be interpreted in terms of a shared instructional context rather than a complete baseline equivalence. Fourth, the intervention lasted four weeks and covered two physics units, which limits conclusions about the long-term effects of the SCAI-supported preparation. Fifth, the intervention combined several components, including AI-supported interaction, scaffolding theory, flipped pre-class preparation, adaptive questioning, and teacher-facing analytics. Therefore, the study cannot isolate the independent effect of the AI component alone. Sixth, technical and contextual constraints, such as poor internet access, time pressure, and students’ workload, may have influenced participation. Finally, qualitative findings are based on a limited number of interviews and should be interpreted as explanatory evidence of participants’ experiences rather than as generalizable evidence.

Additional limitations related to AI should also be considered. Although SCAI was aligned with the objectives of the physics lesson, chatbot responses may still require teacher monitoring to ensure accuracy and to reduce the possibility of inaccurate or incomplete explanations. Students may also become dependent on immediate chatbot support if the tool is not balanced with independent thinking, peer discussion, and teacher-guided learning. In addition, the use of interaction logs and teacher-facing analytics requires careful attention to privacy, transparency, and responsible interpretation, especially because the participants were secondary-school students. These issues reinforce the importance of positing SCAI as a teacher-supported educational tool rather than as an autonomous replacement for instruction.

In general, the convergence of preparation scores, behavioral log data, and interview findings suggests that the SCAI-supported preparation model was positively associated with student preparation and learning experience under the condition of this small-scale classroom implementation. Its value appears to lie in the combination of curriculum-aligned GenAI interaction, scaffolding-based preparation, adaptive questioning, and teacher interface analytics.

8. Conclusions

This study provides exploratory evidence that students who used the SCAI chatbot showed higher pre-class preparation scores than students who used textbook-based preparation under the conditions of this study. This difference was supported not only by the statistical results but also by the platform log data and the interview findings. The evidence showed that the SCAI chatbot promoted active engagement, helped simplify difficult content, increased students’ confidence, and supported preparation for classroom participation.

In general, the findings suggest that the SCAI-supported preparation model may serve as a supportive educational approach when used alongside structured teaching and effective study practices. The contribution of this study lies not simply in building and testing a chatbot but in showing how a scaffolding-based GenAI artifact can support pre-class preparation in a flipped classroom. The study also includes a teacher-facing analytics component that may support instructional monitoring, although its usefulness should be interpreted as exploratory and perception-based. The study extends evidence from secondary-school physics in Saudi Arabia and provides a direct comparison with textbook-based preparation.

However, because the study was non-randomized and conducted over a relatively limited period, the findings should be interpreted as evidence of a meaningful association rather than definitive proof of causation. Because the intervention combined chatbot interaction, scaffolding, flipped classroom preparation, and teacher analytics, future studies should examine the independent contribution of each component under stronger research conditions. Random assignment should be used where possible to provide stronger evidence of causation. Additional research should also examine how technical conditions, especially internet quality and study workload, influence students’ engagement with the tool. In practical terms, future development of the application could include simulated tests, more explanatory visual materials, and a teacher-facing SCAI chatbot interface to support teacher development.