Promoting Reflection Skills of Pre-Service Teachers—The Power of AI-Generated Feedback

Abstract

Reflection skills are a key but challenging element in teacher training. Feedback on reflective writing assignments can improve reflection skills, but it is affected by challenges (high variability in judgments and time investment). AI-generated feedback offers many options. Therefore, the aim of this study was to examine the potential of AI-generated feedback compared to that provided by lecturers for developing reflective skills. A total of 93 randomly selected pre-service teachers (70% female) in a course at a German university wrote two reflections and received feedback from either lecturers or ChatGPT 4.0 based on the same prompts. Pre-service teachers’ written reflections were assessed, and an online questionnaire based on standard instruments was applied. Control variables included metacognitive learning strategies and reflection-related dispositions. Based on a linear mixed model, the main effects on reflective skills were identified for time ($\widehat{\mathsf{\beta }}$ = 0.41, p = 0.003) and feedback condition ($\widehat{\mathsf{\beta }}$ = −0.42, p = 0.032). Both forms of feedback similarly fostered reflective skills over time, with academic self-efficacy emerging as a pertinent disposition ($\widehat{\mathsf{\beta }}$ = 0.25, p = 0.014). The limitations of this study and implications for teacher training are discussed.

1. Introduction

Teachers’ professional development is becoming increasingly important. Teachers need not only to ensure that students’ academic achievement is improving but also to develop learning environments in schools based on 21st-century skills (Benade, 2017; Chan & Lee, 2021; Idel et al., 2020). The ability to (self-)reflect is seen as a core concept for almost all professional trainings (Chan & Lee, 2021; Ryan, 2013). For example, reflection processes have been shown to positively influence decision-making and problem-solving strategies (Sudirman et al., 2024). In particular, in professional teacher training, the ability to reflect is a core skill (Borko et al., 1997; Albert, 2016). Reflection serves as an important bridge between theoretical knowledge and professional action (Korthagen, 2018). Teachers must take an active role in their ongoing professional development process and be able to analyze and reflect on their skills (Albert, 2016). The ability to reflect is also a core component of self-directed (Zimmerman, 2000) and lifelong learning (Abdalina et al., 2022). (Self-)reflection processes are especially relevant for developing metacognitive skills (Desautel, 2009). However, research has shown that reflection skills are often poorly developed among (pre-service) teachers (Zhang et al., 2024). Therefore, teacher training programs need approaches that systematically support pre-service teachers’ reflection skills (Artmann & Herzmann, 2016). Feedback is particularly vital for learning as well as for professional development in general (Darling-Hammond & Bransford, 2007; Korthagen & Kessels, 1999). Implementing feedback in teacher training is associated with two major challenges for lecturers: first, they often perceive the evaluation of pre-service teachers’ reflections as extremely complex (Ryan, 2013), and second, providing individual feedback requires considerable resources (Fleckenstein et al., 2023; Ullmann, 2015). As a result, approaches that incorporate computer-assisted and automated feedback into teaching and learning processes are increasingly being developed and tested (Chong et al., 2020; Fleckenstein et al., 2023; Gibson et al., 2016; Wongvorachan et al., 2022; Wulff et al., 2023). Artificial intelligence (AI) is creating new opportunities, particularly in this context (Zhang et al., 2024), but it also introduces new challenges and problem areas, for example, regarding privacy and data protection (Sebastian, 2023) or AI ethics (Huang, 2023). Given the increasing importance of AI-supported feedback concepts in teaching and learning, this study compared AI-supported feedback with feedback provided by lecturers and examined their effectiveness in promoting pre-service teachers’ reflection skills.

2. Theoretical Background

2.1. Reflection and Reflective Skills in Teacher Education

The ability to reflect has been deemed essential for tackling the growing challenges in schools (Chan & Lee, 2021; Ryan, 2013). In Germany, the KMK Standards for Teacher Education (KMK—Ständige Konferenz der Kultusminister der Bundesrepublik Deutschland, 2022) outline quality assurance guidelines that specifically promote the cumulative development of experience and skills through theory-based reflection. Reflection processes are primarily intended to support the crucial link between theory and practice (Korthagen, 2018) and to constructively integrate the various professional domains of teaching (Albert, 2016). Above all, reflection should contribute to the primary goal of initiating change and innovation: fostering a reorientation towards improved and individualized learning through new insights (Saric & Steh, 2017). Reflection, particularly in the context of teacher training, can be theoretically situated in various ways (Korthagen, 1999). Many academic discussions build upon the ideas of Dewey (1910/2022), which are meaningfully supplemented by the “reflective practitioner” approaches of Argyris and Schön (2024). The definition by Lenske and Lohse-Bossenz (2023) also draws on these principles, defining professional reflection in pedagogical contexts as “an occasion-related mental process that, with self-reference, aims at an expanded understanding of pedagogical practice” (p. 118). Given the many different classifications and forms of implementing reflection (Aeppli & Lötscher, 2016), teacher training generally focuses on analyzing situations or events (Korthagen, 1999) in order to generate options for future action (Schön, 1987). Importantly, it addresses not only cognitive reflection processes and options for action but also motivational and volitional aspects (Leonhard & Rihm, 2011). Almost all models depicting reflection processes build on these fundamental assumptions and differentiate the complex process into distinct phases and sub-competencies (e.g., Aeppli & Lötscher, 2016; Von Aufschnaiter et al., 2019; Blömeke et al., 2015). Given the heterogeneous nature of the teaching profession (e.g., different school types, subjects, grade levels, degrees, etc.), the model by Von Aufschnaiter et al. (2019) provides a particularly suitable framework because it is designed to be domain-independent and dynamically accommodates varying emphases and focuses depending on the context. This model divides the reflection process into (1) “reflection-related dispositions” (fundamental characteristics underlying situation-related thought processes); (2) “reflection-related thought processes” (a situation that serves as a trigger for reflection, ideally within a prepared learning environment); and (3) “reflection-related performance” (the quality or output of the reflection process) (Von Aufschnaiter et al., 2019, p. 152). According to this model, in addition to a specific and well-explained impulse for reflection, a successful reflection process primarily depends on reflection-related dispositions. These include, for example, prior experience with reflection, attitudes and beliefs about its relevance, and the individual’s self-efficacy, i.e., the confidence to carry out the reflective process independently (Von Aufschnaiter et al., 2019). Empirical results show that previous experience with reflection is related to reflection skills (Fraij & Kirschner, 2017), and that self-efficacy expectations influence reflective capability (Stender et al., 2021). The influence of attitudes on reflection and the necessity of developing reflection skills (Göbel & Neuber, 2020; Stender et al., 2021) is also well supported and aligns with findings from portfolio research and the importance of attitudes of pre-service teachers towards portfolio work (Feder et al., 2021).

Pre-service teachers often struggle with reflection and tend to achieve only low levels of reflective depth (Zhang et al., 2024). Consequently, several studies have investigated the conditions for successfully implementing reflection in teacher training (e.g., Hansen & Rachbauer, 2018; Hatton & Smith, 1995; Jaeger, 2013; Jones & Jones, 2013; Saric & Steh, 2017). It seems essential to systematically foster the development of reflective skills from the beginning of teacher training through targeted approaches (Artmann & Herzmann, 2016). Written engagement with the subject of reflection has proven to be an effective strategy for developing reflective skills (Häcker, 2019; Sudirman et al., 2024). Moreover, students require various forms of support to develop this complex skill; most frequently mentioned are detailed explanations, examples, feedback, and specific workshops (Ross et al., 2024). These measures are necessary to positively influence enthusiasm and attitudes toward reflective writing and to help students overcome difficulties and obstacles (Sudirman et al., 2024).

Different models can be used to assess written reflections (Boyd & Fales, 1983; Gibbs, 1988; Gore & Zeichner, 1991; Hatton & Smith, 1995). To determine levels of reflection, one usually attempts to reduce the complexity of reflections by employing depth or breadth models, or to classify the reflective thoughts according to specific criteria (Ullmann, 2019). Research on teacher education often draws on the depth model of Hatton and Smith (1995), which has been adapted multiple times (e.g., Riel, 2022; Zhang et al., 2023, 2024). In university-based teacher training, contextually tailored criteria grids specifying quality characteristics of successful reflections are commonly used for evaluation (Von Aufschnaiter et al., 2019; Landesinstitut für Lehrerbildung und Schulentwicklung Hamburg, LI Hamburg, 2020). These criteria grids serve as the basis for assessment, feedback, or (self-)reflection questionnaires (e.g., Berger, 2022).

2.2. Feedback and AI

Numerous meta-analyses have shown positive effects of feedback on promoting learning processes and performance (e.g., Hattie & Timperley, 2007; Kluger & DeNisi, 1996). Feedback is also highly relevant for reflective writing (Darling-Hammond & Bransford, 2007; Korthagen & Kessels, 1999), and its positive effects have already been demonstrated (Jahncke et al., 2018; Schellenbach-Zell et al., 2023). However, university lecturers consider evaluating written reflections extremely complex and time-consuming (Ryan, 2013). Furthermore, due to limited resources, it is often difficult to provide timely, detailed, and didactically prepared feedback on reflections to a larger number of students (Deeva et al., 2021; Fleckenstein et al., 2023; Ullmann, 2015).

Rapid technological advances, especially in artificial intelligence (AI), are fundamentally transforming the educational system (Zhai et al., 2021). Various computer-assisted and automated methods for feedback in learning and instruction have been developed (Chong et al., 2020; Fleckenstein et al., 2023; Gibson et al., 2016; Suvendu & Deb, 2024; Wongvorachan et al., 2022; Wulff et al., 2023). AI approaches based on (open) large language models (LLMs), such as ChatGPT, are becoming increasingly popular (Chang et al., 2023). These models can be integrated into learning and instruction, for example, by providing feedback to students in instructional settings where feedback was previously not feasible due to the lack of this technological advancement (Deeva et al., 2021; Ullmann, 2019). The use of LLMs enables timely, personalized, and frequent feedback (Russell & Korthagen, 2013); however, lecturers need to know which systems can be applied in which contexts, and they must possess the skills to use them accordingly (Deeva et al., 2021). Initial trials mostly employ relatively simple task formats to test machine-generated feedback. Such automatic feedback systems have already proven effective in these areas, as (moderate) effects on the respective competence development have been observed (Fleckenstein et al., 2023).

However, reflections pose an enormous challenge for analysis and evaluation due to their complexity and multi-perspective nature (Körkkö et al., 2016; Poldner et al., 2014; Ullmann, 2019). The enormous substantial innovations in LLM training over the past two years have now enabled meaningful reflection assessment (Nehyba & Štefánik, 2023; Wulff et al., 2023). These advances allow the use of AI feedback on reflections in learning and instruction (Barthakur et al., 2022; Jung et al., 2022; Zhang et al., 2024), making it possible for a large number of students to receive feedback on their reflections promptly, or even during the work phase (Kovanović et al., 2018; Ullmann, 2019).

While there is great potential associated with the development of artificial intelligence (Trust et al., 2023; Winkel, 2025; Wongvorachan et al., 2022), including in the field of teacher training (Van den Berg & Du Plessis, 2023), it also brings new obstacles and challenges (Chang et al., 2023; Eden et al., 2024; Suvendu & Deb, 2024). Open AI products such as ChatGPT in particular often face the issue that data protection aspects are not yet fully addressed and privacy is insufficiently safeguarded (Huang, 2023; Sebastian, 2023). Thus, innovations and improvements in the context of AI ethics are urgently needed (Sebastian, 2023; Winkel, 2025). This also implies that teachers, learners, and the entire educational landscape must develop competencies in these areas if technological advances are to be used effectively without compromising privacy (Buckingham Shum et al., 2023; Sperling et al., 2024; Watini et al., 2024; Wiese et al., 2025). Furthermore, new, technology-supported concepts also require new implementation strategies (Eden et al., 2024; Suvendu & Deb, 2024) as well as new pedagogical and didactic approaches to designing learning environments (Bearman & Ajjawi, 2023; Chang et al., 2023; Deeva et al., 2021; Eden et al., 2024; Li & Kim, 2024; Van den Berg & Du Plessis, 2023; Wiese et al., 2025). In addition to the lecturers, students also need to adapt to make meaningful use of AI feedback: when well-designed learning environments are in place, the challenge lies less in the ability to use automatic feedback systems and more in the ability to integrate them meaningfully into (self-regulated) learning processes and to critically evaluate the results (Bearman & Ajjawi, 2023; Li & Kim, 2024). Furthermore, skills such as creativity, teamwork, and problem-solving, which cannot (currently) be performed by AI systems, must be increasingly emphasized and developed (George, 2023).

This study focuses on the significant possibilities offered by AI tools in teaching and learning processes (Eden et al., 2024; Trust et al., 2023; Van den Berg & Du Plessis, 2023). In particular, the option of generating automatic feedback (Shaik et al., 2022; Wongvorachan et al., 2022) promises considerable potential, especially in connection with the complex skill of reflection (e.g., Chang et al., 2023).

Nevertheless, the effects of AI-based feedback compared to instructor feedback on the development of learners’ reflection skills have not yet been investigated.

3. Research Questions

Considering the current state of research outlined above, it can be considered certain that pre-service teachers should possess or develop reflective competence. However, it remains unclear how this competence can be most effectively imparted. A promising approach is feedback, as the fundamental effects of feedback on learners’ competency development are well-established. Positive effects of feedback have also been repeatedly demonstrated in the specific area of reflective ability. The question of whether AI-generated feedback similarly enhances the quality of pre-service teachers’ written reflections remains to be clarified, particularly in comparison to feedback provided by lecturers. Furthermore, we aim to investigate whether reflection-related dispositions influence the use of (AI-supported) feedback. The current study therefore addresses the following research questions:

RQ 1:

What is the effect of AI-supported feedback compared to instructor feedback on the reflection level in pre-service teachers’ written reflections?

RQ 2:

How do reflection-related dispositions influence the reflection level in pre-service teachers’ written reflections?

4. Method

4.1. Participants

A total of 93 pre-service teachers from four different teacher training programs at a German university participated in this study. The instructional concept, developed to improve reflection skills, was integrated into a basic module in the field of educational science. In this module, participants encountered, for the first time, a learning concept involving interaction with AI. Data were collected at the beginning (t1) and midpoint (t2) of the summer semester of 2024. A total of 93 pre-service teachers participated in the initial survey at t1, distributed across teacher training programs as follows: primary school (44.1%), lower secondary school (7.5%), intermediate secondary school (23.7%), and academic secondary school (24.7%). Eight participants discontinued the course, resulting in a final sample of the study with eighty-five pre-service teachers at t2 (see

Table 1. Sociodemographic sample characteristics.

Characteristic	FeedbackInstructor		FeedbackAI		Sample
	M (SD)/%	n	M (SD)/%	n	M (SD)/%	n
Age	21.82 (3.24)	45	21.00 (2.21)	48	21.40 (2.27)	93
Gender
Female	66.7%	30	72.9%	35	69.9%	65
Male	33.3%	15	25.0%	12	29.0%	27
Other	-	0	2.1%	1	1.1%	1
Teacher Training Program
Primary	46.7%	21	41.7%	20	44.1%	41
Lower Secondary	6.7%	3	8.3%	4	7.5%	7
Intermediate Secondary	17.8%	8	29.2%	14	23.7%	22
Academic Secondary	28.8%	13	20.8%	10	24.7%	23

).

4.2. Design

4.2.1. Procedure

The current study is part of the research project PetraKIP (“Personal Transparent AI-Based Portfolio for Teacher Training”) funded by the German Federal Ministry of Education and Research.

To promote the pre-service teachers’ reflection skills, an instructional concept was developed as part of a basic teacher training module in the domain of school education. The concept comprised a lecture, a seminar, and two individual learning phases covering two topics (1. Diagnostics and 2. Classroom Management). The individual learning phase entailed the subsequent processing of a case study, each presenting a problem situation from the school context. After the first survey (t1), the participants submitted a written reflection on each case study over the semester, for which they received feedback.

Following the first written reflection, 48 participants were randomly assigned to receive AI-generated feedback, while 45 participants received lecturer-provided feedback. The pre-service teachers could then use this feedback when preparing the second written reflection. After submitting the second reflection (t2), all participants received AI-generated feedback on their work. By comparing the two feedback groups from t1 to t2, it was possible to evaluate the effects of AI-supported feedback versus lecturer feedback (RQ 1), while also considering the influence of reflection-related dispositions (RQ 2). Table 1 presents sociodemographic information for both feedback groups as assessed at t1.

4.2.2. Instruments

Feedback Grid and Reflection Scores

To assess the quality of pre-service-teachers’ written reflections, a grid with quality criteria was used (e.g., Von Aufschnaiter et al., 2019; Philipp, 2023). This grid served as the basis both for an evaluation form for instructors and as a prompt for ChatGPT 4.0 (Berger, 2022) (see Appendix A for the full prompt). The starting point for the evaluation criteria was the compilation of quality characteristics according to Meyer-Siever and Levin (2018), which identify four major categories: (1) analysis based on theoretical models and empirical findings, (2) consideration of different perspectives, (3) developing alternative courses of action or considerations for future action, and (4) relating to one’s professionalization. An existing instrument built on this foundation and already successfully used is the form developed by the State Institute for Teacher Training and School Development Hamburg (Landesinstitut für Lehrerbildung und Schulentwicklung Hamburg, LI Hamburg, 2020). Their framework originally included the following ten criteria: (1) Linguistic clarity, (2) Structure, (3) Description of the experience, (4) Theoretical relevance, (5) Professional accuracy, (6) Richness of perspectives, (7) Own viewpoints (and their justification), (8) Self-critical perspective, (9) Constructive solutions and conclusions, and (10) Systemic perspective (linking of the individual aspects and perspectives). This framework was slightly modified because some criteria, particularly those relevant for practical use, were not considered or required more specific explanation or supplementation. A comparison with other quality criteria lists (e.g., Von Aufschnaiter et al., 2019; Hansen & Rachbauer, 2018; Leonhard & Rihm, 2011; Landesinstitut für Lehrerbildung und Schulentwicklung Hamburg, LI Hamburg, 2020; Philipp, 2023) informed these adjustments. For example, the highly detailed compilation by Von Aufschnaiter et al. (2019) provided the criterion “scope”, while “structure of reflection” was differentiated further by including explicitly stated reflection goals (including their justification). Furthermore, some criteria were reformulated (e.g., “describing the experience” became “object of reflection”) or combined due to a lack of clarity in practice (e.g., “own perspectives” and “self-critical perspectives”).

Thus, for this study, the following criteria were included in the grid: (1) Linguistic design, (2) Structure, (3) Scope, (4) Formulation of goals, (5) Justification of these goals, (6) Reference to the object of reflection, (7) Use of professional knowledge, (8) Own perspectives, (9) Different temporal perspectives, (10) Richness of perspectives, (11) Linking of the individual aspects and perspectives, and (12) Further thoughts and conclusions. According to Leonhard and Rihm (2011), a suitable instrument should reflect both the structure of the construct of reflective ability and the corresponding levels of expression. Each listed criterion was assessed on a scale of 0–3 (Philipp, 2023), with a maximum score of 36. Detailed coding guidelines for instructors were developed and iteratively refined to ensure uniform and consistent scoring.

Most coders had prior experience in grading and providing feedback on reflective writing from previous semesters of the project. To further ensure consistency, all coders participated in a three-hour re(training) session each semester, which was led by the main author and focused on the twelve criteria and distinction of quality levels (0–3). After training, three coders independently rated a subset of reflection texts, but they were encouraged to consult with one another via chat exchange in cases of uncertainty. A second team of three coders conducted an independent second round of coding with similar opportunities for collegial exchange.

Interrater reliability was then assessed based on 540 coded segments (12 criteria across 45 pre-service teachers’ written reflections in the instructor feedback group at t1), each rated by 2 of k = 6 university instructors. Following Shrout and Fleiss (1979), intraclass correlation coefficients were calculated in R using a one-way random effects model for single measures (psych: ICC, Revelle, 2025). The resulting interrater reliability was excellent (ICC (1,1) = 0.93, 95% CI [0.93, 0.94]) according to benchmarks by Koo and Li (2016), supporting the use of reflection scores in subsequent analyses.

Online Questionnaire

All additional data were collected as part of an online survey at the beginning of the semester (t1) and after completing the first reflective writing assignment (t2). Apart from socio-demographic data, the following reflection-related dispositions were collected:

(1)

Metacognitive Learning Strategies (t1)

Metacognitive Learning Strategies (MCLS) encompass the dimensions of planning, monitoring, and regulation in the context of learning activities. The scale was adapted from the German LIST inventory (Learning Strategies in Higher Education, Wild & Schiefele, 1994). Responses were recorded on a 5-point Likert scale ranging from 1 (not at all true) to 5 (completely true). The scale comprises 18 items and demonstrated good internal consistency, with Cronbach’s α = 0.85, 95% confidence interval (CI) [0.81, 0.90], n = 93. A sample item is: “If I do not understand a text upon first reading, I go through it again step by step.”

(2)

Academic Self-Efficacy (t1)

Academic self-efficacy (ASE) was measured using a six-item Likert scale based on the work of Schiefele et al. (2002). Responses were rated on a 5-point scale ranging from 1 (not at all true) to 5 (completely true), in response to items such as: “Even if an exam is very difficult, I know what I have to do to pass it.” The scale indicated good reliability, with Cronbach’s α = 0.89, 95% CI [0.86, 0.92], n = 93.

(3)

Attitude towards reflective writing tasks (t1)

Positive attitude towards reflective writing tasks (ARW) was assessed using a self-developed 5-point Likert scale, with response options ranging from 1 (not at all true) to 5 (completely true). The brief scale consisted of three items (e.g., “I believe that engaging in written reflections during teacher training enhances reflection on personal development.”) and yielded good reliability estimates (Cronbach’s α = 0.79, 95% CI [0.72, 0.87], n = 93).

(4)

Previous experience with reflective writing tasks (t1)

Pre-service teachers’ prior experience with reflective writing tasks (ERW) was measured via a self-developed single item: “I have already engaged in written reflection tasks prior to attending the Module 1 school pedagogy course.” Responses could be given on a scale from 1 (not at all true) to 5 (completely true).

(5)

Feedback Engagement (t2)

To assess pre-service teachers’ engagement with the feedback provided on their first reflective writing assignment (FE, treatment check), we employed a self-developed three-item Likert scale (e.g., “I carefully considered the feedback on my first reflection before composing the second one.”). Response options ranged from 1 (not at all true) to 5 (completely true). The scale demonstrated acceptable reliability (Cronbach’s α = 0.60, 95% CI [0.46, 0.74], n = 84).

4.3. Data Analysis

In the first step, eight instructors assessed written reflections from both feedback groups (AI vs. instructor) and both measurement points (t1 and t2), following the coding guideline. These reflection scores (RSs) served as the basis for further analysis.

In the second step, a linear mixed effects model (LMM) with random intercepts was applied to account for individual variability and the dependency of repeated measures in the RS. LMMs allow for more precise estimation of fixed and random effects and offer increased statistical power, even with small sample sizes (Brauer & Curtin, 2018).

The dataset was transformed into long format (two rows per participant) to model differences between measurement occasions, with RS collapsed into a single variable and measurement time point added as a factor (Time; 0 = t1, 1 = t2). The RS variable was z-standardized and grand-mean centered for better comparability and interpretation of results. The final model included RS as an outcome predicted by time, feedback condition (including their interaction), and reflection-related dispositions (metacognitive learning strategies, academic self-efficacy, attitude towards reflective writing tasks, previous experience with reflective writing tasks, and engagement with provided feedback). Analyses were conducted in R (R Core Team, 2025) using the lme4 package with restricted maximum likelihood estimation (Bates et al., 2015).

5. Results

5.1. Descriptive Statistics

Table 2. Means and standard deviations for all variables across both measurement points.

Variable	FeedbackInstructor		FeedbackAI		Sample
	M (SD)	n	M (SD)	n	M (SD)	n
(t1)
Academic Self-Efficacy (ASE)	4.11 (0.88)	45	4.12 (0.92)	48	4.12 (0.90)	93
Metacognitive LS (MCLS)	3.59 (0.53)	45	3.79 (0.51)	48	3.70 (0.53)	93
Experience with RW (ERW)	2.29 (1.39)	45	2.06 (1.28)	48	2.17 (1.33)	93
Attitude towards RW (ARW)	3.92 (0.66)	45	3.94 (0.74)	48	3.93 (0.70)	93
Reflection Score (RS1)	23.38 (4.69)	45	19.31 (7.64)	48	21.28 (6.67)	93
(t2)
Feedback Engagement (FE)	3.94 (0.56)	43	3.91 (0.55)	41	3.92 (0.55)	84
Reflection Score (RS2)	26.28 (6.59)	43	21.55 (7.85)	42	23.94 (7.58)	85

Note. LS = learning strategies, RW = reflective writing.

presents the means (M) and standard deviations (SD) for the first (t1) and second (t2) measurement points across both feedback groups. For the first measurement point, the table lists descriptive values for reflection scores (RS1), metacognitive learning strategies (MCLS), and reflection-related disposition variables (academic self-efficacy, ASE; experience with reflective writing, ERW; attitude towards reflective writing, ARW). For the second measurement point, feedback engagement (FE) and reflection scores (RS2) are reported. Both groups scored very high on academic self-efficacy (ASE) and moderately high on metacognitive learnings strategies (MCLS) and (positive) attitudes towards reflective writing (ARW). Most students had little prior experience with reflective writing (ERW), although the standard deviation indicates considerably variability within the sample. Descriptive statistics further indicate that metacognitive learning strategies (MCLS) and reflection-related disposition variables (ASE, ERW, ARW) are similar across the two groups, with the main differences observed in reflection scores (RS1 and RS2).

Reflection scores were moderately correlated between measurement points, r = 0.56, p < 0.001. Among the predictor variables considered for inclusion in the LMM, the most significant intercorrelation coefficients fell into the small effect range (0.44 > r > 23, p < 0.05), except for metacognitive learning strategies (MCLS; see also

Table 3. Correlation coefficients for reflection scores and reflection-related dispositions.

	ASE (t1)	MCLS (t1)	ERW (t1)	ARW (t1)	FE (t2)	RS (t1)
MCLS (t1)	*** 0.64
ERW (t1)	0.03	0.03
ARW (t1)	* 0.25	** 0.27	0.15
FE (t2)	*** 0.38	*** 0.43	0.07	* 0.24
RS (t1)	* 0.24	0.20	0.09	0.06	* 0.26
RS (t2)	** 0.28	0.21	* 0.23	* 0.25	** 0.29	*** 0.56

Note. *** p < 0.001, ** p < 0.01, * p < 0.05.

). MCLS showed a strong correlation with academic self-efficacy (ASE) at t1, r = 0.64, p < 0.001; a moderate correlation with feedback engagement (FE) at t2, r = 0.43, p < 0.001; and a weak correlation with attitude towards reflective writing (ARW) at t1, r = 0.27, p = 0.009. Correlations between MCLS and reflection scores at both measurement points were non-significant. Due to the risk of multicollinearity, MCLS was excluded as a predictor in the LMM.

5.2. Linear Mixed Effects Model

A Linear Mixed Effects Model was used to address both research questions. The full model (including fixed and random effects) explained 60.6% of the total variance in reflection scores (Conditional R²).

Regarding the fixed effects, time emerged as a significant predictor, with $\widehat{\mathsf{\beta }}$ = 0.41, 95% CI [0.14, 0.68], t (81.90) = 3.02, p = 0.003, indicating a general increase in reflection scores from t1 to t2 by nearly half a standard deviation. Feedback condition also reached significance in the model, $\widehat{\mathsf{\beta }}$ = −0.42, 95% CI [−0.80, −0.04], t(125.82) = −2.17, p = 0.032, with participants in the AI feedback group scoring, on average, nearly half a standard deviation lower than those in the instructor feedback group. However, the interaction term (RQ 1) was non-significant, suggesting that both feedback groups showed comparable gains in reflection scores from pre- to post-measurement.

Concerning reflection-related dispositions (RQ 2), only academic self-efficacy emerged as a significant positive predictor of reflection scores, $\widehat{\mathsf{\beta }}$ = 0.25, 95% CI [0.05, 0.45], t(81.10) = 2.52, p = 0.014. The fixed effects accounted for 23.7% of the variance in reflection scores (Marginal R²).

The intraclass correlation coefficient (ICC) was 0.48, indicating that a substantial proportion of variance in reflection scores was attributable to differences between participants (random effects).

6. Discussion

6.1. Summary

In accordance with the first research question (RQ 1), the main objective of this study was to analyze the effects of AI feedback on the reflection level of pre-service teachers, specifically compared to instructor feedback. The results indicate that pre-service teachers in both groups slightly increased their reflection scores from the first to the second measurement point. This is consistent with previous findings and indicates that, on the one hand, reflection skills can be improved (e.g., Von Aufschnaiter et al., 2019; Häcker, 2019) and, on the other hand, the provision of feedback may have a positive impact on this learning process (Darling-Hammond & Bransford, 2007; Jahncke et al., 2018; Korthagen & Kessels, 1999; Schellenbach-Zell et al., 2023). Nonetheless, pre-service teachers’ reflections generally remained on a lower level, consistent with previous findings (e.g., Zhang et al., 2023, 2024), highlighting that the ability to reflect (in writing) is a complex skill (e.g., Aeppli & Lötscher, 2016) that requires time and multiple interventions to develop (e.g., Häcker, 2019). Furthermore, other factors appear to determine the development of reflection skills (e.g., Von Aufschnaiter et al., 2019), the handling of feedback (e.g., Deeva et al., 2021), and the use of AI tools, as well as the acceptance of the output results (Zhang et al., 2023). Pre-service teachers who received AI feedback increased their reflection skills similarly to the lecturer feedback group. Previous research has shown that modern technologies such as AI and machine learning can be used to assess the quality of reflective writing (Barthakur et al., 2022; Jung et al., 2022; Zhang et al., 2024). Our findings add novel evidence that AI-based feedback, when guided by didactically informed prompts, can be at least as effective as feedback from experienced instructors. This suggests that, with future developments in this field (e.g., Chong et al., 2020; Fleckenstein et al., 2023; Suvendu & Deb, 2024; Wongvorachan et al., 2022; Wulff et al., 2023), the use of AI can positively impact learning processes, and its effectiveness may further increase. However, we also found that the group receiving feedback from instructors performed slightly better than the group receiving feedback from AI—even if this difference was not significant in our model. Therefore, it is certainly worth investigating whether, despite similar effects, a closer examination could reveal differences between the two forms of feedback. For example, despite uniform criteria and formal alignment, minor (qualitative) differences could exist, which may ultimately lead to minor differences in their impact on the development of (written) reflection skills. This could further indicate that the use of AI has limitations and, at least currently, does not (yet) achieve better results.

Regarding the second research question (RQ 2), we found evidence of a significant relationship between academic self-concept and the development of reflection skills, which aligns with existing research (Stender et al., 2021). However, other factors in the model did not prove to be significant predictors of the development of reflection skills. This is only partially consistent with existing findings: several empirical studies have reported relationships between prior experience with (written) reflections (e.g., Fraij & Kirschner, 2017) and attitude towards reflections (e.g., Göbel & Neuber, 2020; Stender et al., 2021) and the development of reflective skills. Instead, our results suggest that individual differences between pre-service teachers exert a stronger influence on reflection levels, which is again in line with prior research (e.g., Aeppli & Lötscher, 2016; Lambrecht & Bosse, 2020; Landesinstitut für Lehrerbildung und Schulentwicklung Hamburg, LI Hamburg, 2020) and indicates that individual learning prerequisites and learning arrangements play a significant role, especially in the development of reflective skills (e.g., Saric & Steh, 2017). Furthermore, the various theoretical models for mapping reflection processes (e.g., Aeppli & Lötscher, 2016; Von Aufschnaiter et al., 2019; Blömeke et al., 2015) already indicate that a multitude of (additional) factors influence the development of reflective skills (e.g., Sudirman et al., 2024).

6.2. Limitations

This study faces several limitations that warrant acknowledgment and consideration. Due to the complexity of the intervention, only a relatively small sample of fully evaluated cases was available, and only a few covariates could be used for the analysis, with corresponding implications for the power and generalizability of the findings. Consequently, several known direct or indirect factors influencing the development of reflective skills (e.g., Aeppli & Lötscher, 2016) could not be fully considered in the analyses, such as subject-specific competencies (Henderson, 1992) and motivational or emotional learning prerequisites (Leonhard & Rihm, 2011). Furthermore, other constructs beyond metacognitive learning strategies have not been included (Möllers, 2014; Pintrich & Garcia, 1993). Regarding the opportunities and limitations of feedback concepts based on machine learning or AI, it should be noted that the aforementioned personal characteristics of learners also play a role in this context: factors such as AI acceptance or attitudes towards automated learning processes could influence the effectiveness of the feedback provided (e.g., Zhang et al., 2024). Additionally, no specific pedagogical concepts or implementation strategies targeting AI use were applied (e.g., Eden et al., 2024; Suvendu & Deb, 2024). Such concepts could potentially mitigate negative attitudes towards the use of AI in learning processes. In this context, it should also be noted that this study was conducted at a German university, and the results cannot be easily generalized to other countries with different teacher training structures. This limitation becomes even more pronounced if one were to extrapolate the results to other fields. In our design, the intervention only provided for a one-time provision of feedback. Feedback research has shown that multiple iterations are often necessary to realize the full potential of feedback (Hattie & Timperley, 2007; Kluger & DeNisi, 1996). Furthermore, to keep feedback conditions constant, specific attributes of the two feedback forms were not integrated into the intervention: for example, the near-immediate availability of AI feedback to learners (e.g., Deeva et al., 2021; Fleckenstein et al., 2023; Ullmann, 2015) or the ability of instructor feedback to consider previously unspecified criteria and respond more empathetically (e.g., O’Donovan et al., 2019; Ossenberg et al., 2019) was deliberately not utilized. Studies using LLMs such as ChatGPT often face the problem of non-replicable results: despite using the same prompt at different times, different outputs may occur (e.g., Cao et al., 2025; Mondal et al., 2024). This ensures, on the one hand, that different results could potentially be obtained at a different study point in time; on the other hand, the study cannot be easily repeated or expanded. Another limitation concerns the assessment of reflections: reflective ability is often evaluated using stage models (e.g., Hatton & Smith, 1995). The small number of stages and the major changes often required to reach a higher level can obscure small developments (Zhang et al., 2023). For this reason, among others, the present study used a multi-layered criteria grid as the basis for assessment and feedback (e.g., Ullmann, 2019). However, although we applied an assessment method based on a theoretical and multi-layered criteria grid (e.g., Von Aufschnaiter et al., 2019), allowing for detailed ratings of different aspects of reflective ability, the criteria used may still limit the thoroughness of our evaluation. Therefore, the criteria grid and associated coding guidelines require further testing. Overall, it remains unclear whether AI and instructor feedback can be expected to have different effects on the various aspects of reflection skills, as only the overall score was included in the tested model. Additionally, little attention was paid to the form of feedback given to students, which research indicates can influence its effects (e.g., Hattie & Timperley, 2007). For more significant improvement in reflection skills (Aeppli & Lötscher, 2016; Argyris & Schön, 2024; Leonhard & Rihm, 2011), multiple measures addressing different aspects of reflection would be necessary (Artmann & Herzmann, 2016; Borko et al., 1997).

6.3. Implications and Future Directions

The study results show that pre-service teachers can develop their written reflection skills both through instructor feedback and, similarly, when receiving AI-generated feedback. This important finding may contribute to the further development of instructional quality in teacher education, offering pre-service teachers the opportunity to receive anonymous, criteria-based, and frequent feedback almost immediately, at nearly any time of day. Since reflections not only support learning processes in university courses in teacher training but also play a significant role in practical phases, their potential range of applications could be significantly expanded (e.g., Häcker, 2019). However, our results indicate that feedback alone is not sufficient to significantly improve the reflection skills of pre-service teachers. To achieve real improvement, several concepts must be used in combination: in addition to feedback, detailed explanations, prompts, examples, and the consideration of individual learning requirements and motivation are necessary (Artmann & Herzmann, 2016; Borko et al., 1997). Developing corresponding, more complex concepts requires further empirical insights. In this context, it is particularly important to examine the quality of feedback, both in general and in comparison, between the two actors. Specifically, qualitative analyses of the feedback provided could generate valuable insights in this area. Closely linked to these considerations is the idea of generating higher-quality feedback by varying the prompt or by improving prompt engineering and/or using powerful LLMs, thereby potentially enhancing the development of reflective skills. Furthermore, it should be noted that the integration of AI in learning environments offers research potential beyond technical developments; the ethical and moral implications of AI use must also increasingly be addressed in future research.