Human and AI Judgment in EFL Speaking Assessment: Psychometric Dependability, Feedback Quantity and Coverage, and Exploratory Teacher Perspectives

Abstract

This study examined human-AI judgment in EFL speaking assessment as a measurement problem relevant to research on individual differences: how consistently do expert raters and repeated AI sessions preserve between-person differences in a complex performance? Thirty oral-performance samples from first-year Chinese university English majors were evaluated by four experienced EFL teachers and by ChatGPT-4o across four repeated sessions using the same holistic and accuracy-fluency-complexity framework. Score distributions were summarized descriptively; human-rater intraclass correlations and generalizability-theory G- and D-studies evaluated dependability. Feedback quantity and thematic coverage were summarized descriptively, and post-results interviews with the same four teacher-raters were analyzed as exploratory participant reflections. Teachers produced more dependable single ratings. Holistic phi coefficients were 0.54 for one ChatGPT-4o session and 0.65 for one teacher, increasing to 0.83 and 0.88 with four assessments. Analytic single-assessment coefficients ranged from 0.54 to 0.56 for ChatGPT-4o and 0.61 to 0.65 for teachers. Across holistic and analytic scoring, a phi coefficient of at least 0.80 was reached with four repeated ChatGPT-4o sessions but three independent teachers. ChatGPT-4o’s analytic dimensions showed limited differentiation from its holistic scoring: accuracy matched the holistic score in all 120 AI ratings, while fluency and complexity differed in only three and four ratings, respectively. ChatGPT-4o also produced more rubric-relevant comments descriptively, while the four teacher-raters emphasized human oversight for contextual and interpersonal interpretation. The study contributes to intelligence research at the level of psychometric measurement and human-AI judgment: assessor systems can differentially preserve observed individual differences in complex performance. EFL speaking was not treated as an intelligence test, and unmeasured cognitive mechanisms were not inferred from the score patterns.

1. Introduction

Research on intelligence depends on valid measurement of individual differences. Latent abilities are not identical to any single observable task, and scores from complex performance tasks are meaningful only to the extent that the measurement system preserves systematic between-person differences rather than assessor-related error (Deary et al., 2010; Kovacs & Conway, 2016). This problem is especially important when responses require judgment. Recent Journal of Intelligence research has shown that rater effects can alter scores on divergent-thinking tasks and that large language models (LLMs) are increasingly being used as evaluators of open-ended responses (Pellegrino et al., 2025; Saretzki et al., 2025; op ‘t Hof et al., 2025). These developments make the assessor—human or AI—part of the measurement problem rather than a neutral conduit from performance to score.

EFL speaking offers an applied setting in which to examine that problem. Oral performance draws on linguistic knowledge together with rapid lexical retrieval, formulation, monitoring, and attentional control, and complexity, accuracy, and fluency (CAF) are observable outcomes of interacting knowledge, processing resources, task demands, and performance conditions (de Jong, 2023; Linck et al., 2014; Skehan, 2009). An EFL speaking score is not a measure of general intelligence. The intelligence-relevant question addressed here is narrower and psychometric: whether different assessor systems preserve between-person performance differences with comparable dependability.

Generative AI creates a new assessor system for this purpose. LLMs can classify performance features, generate scores, and produce feedback at a scale that is difficult for individual teachers. Language-assessment research has documented rapid and extensive AI feedback (Barrot, 2023; Guo & Wang, 2024; Lu et al., 2024; Song & Song, 2023; Su et al., 2023), while recent speaking studies have directly examined LLM scoring and human-AI agreement (Aryadoust et al., 2026; Choi, 2026; Voskoboinik et al., 2025). In parallel, intelligence research has compared human and LLM evaluators in creativity assessment and examined rater effects in open-ended cognitive-performance tasks (Pellegrino et al., 2025; Saretzki et al., 2025; op ‘t Hof et al., 2025). Together, these strands motivate a design that separates stable person differences from variability attributable to independent human raters or repeated applications of one AI system.

The general objective of the present study was to evaluate human-AI complementarity in EFL speaking assessment by integrating psychometric evidence, feedback analysis, and teacher reflections. Two specific objectives followed: (1) to compare how dependably repeated ChatGPT-4o sessions and experienced teachers differentiate learners’ holistic and analytic speaking performances; and (2) to characterize differences in feedback quantity and thematic coverage and explore how the participating teacher-raters viewed human oversight. The empirical contribution to intelligence research is therefore measurement-focused: the study quantifies how much observed score variation is attributable to persons versus assessor/occasion facets and how aggregation changes dependability when judgments come from independent experts versus repeated applications of the same AI system.

2. Literature Review

2.1. Human Intelligence, Language, and EFL Speaking Performance

Psychometric theories distinguish latent cognitive abilities from the tasks through which they are observed. General intelligence is inferred from covariance across multiple cognitive measures rather than from a single performance (Deary et al., 2010). Process Overlap Theory likewise emphasizes that complex tasks recruit overlapping domain-general and domain-specific processes (Kovacs & Conway, 2016). EFL speaking is therefore treated here as a complex performance domain, not as a proxy for IQ or g.

Second-language research provides a process-oriented basis for this framing. Working memory is associated with L2 comprehension and production (Linck et al., 2014), and CAF dimensions capture different observable aspects of real-time language control: accuracy reflects form control and monitoring, fluency reflects continuity and efficiency of production, and complexity reflects the range and elaboration of linguistic resources deployed (de Jong, 2023; Skehan, 2009). The present analysis concerns the dependability with which these performance differences are measured, rather than the direct measurement of any cognitive ability.

2.2. Psychometric Dependability, Human Judgment, and Generalizability Theory

Reliable measurement is essential whenever scores are used to characterize individual differences. The Standards for Educational and Psychological Testing emphasize measurement precision, relevant sources of error, fairness, and the intended universe of generalization (American Educational Research Association et al., 2014). In open-ended intelligence-related tasks, assessor effects are not merely nuisance details: recent divergent-thinking research has modeled differences in rater severity and discrimination because these differences can change the scores attributed to individuals (Pellegrino et al., 2025). Speaking assessment raises the same general psychometric issue because human judges may differ in severity and in how they apply performance criteria (Huang et al., 2021).

Generalizability (G-) theory decomposes multiple sources of score variation and estimates the dependability of decisions made from performance measures (Brennan, 2001; Cronbach et al., 1972; Shavelson & Webb, 1991). In a person-by-rater design, person variance represents systematic between-person differences, whereas assessor and person-by-assessor components represent sources that can weaken dependability. In the present study, the human facet is teacher-rater identity and the AI facet is repeated session/occasion within the same model. Their common 30 × 4 structure permits parallel dependability analyses, but the two facets are not substantively equivalent.

2.3. Human and Artificial Judgment in Language Assessment

Generative AI can perform several assessment functions, including applying explicit criteria, categorizing performance features, assigning scores, and generating explanations. Recent studies reveal a heterogeneous rather than uniformly human-like pattern in AI evaluation. In L2 speaking, Voskoboinik et al. (2025) showed that LLM scoring improved when human-rated examples across proficiency levels were supplied, underscoring dependence on prompting and reference examples; Choi (2026) found useful reliability in pronunciation scoring but also limited discriminative power, problems in handling missing data, reduced sensitivity to L1 interference, and weak pragmatic or contextual interpretation; and Aryadoust et al. (2026) showed that a customized GPT-based conversational system can support EAP speaking assessment when measurement design, evaluation logic, and learner experience are jointly considered. Parallel intelligence research reaches a similar conclusion from open-ended creativity tasks: Saretzki et al. (2025) reported substantial correspondence between LLM and human ratings but model- and condition-dependent performance, op ‘t Hof et al. (2025) showed that assessor identity (human or LLM) can systematically affect creativity judgments, and Pellegrino et al. (2025) demonstrated that human rater severity and discrimination can alter person scores. Taken together, recent evidence suggests that human-AI evaluation should be compared along multiple dimensions run-to-run reproducibility, agreement with human raters, construct coverage, score compression or bias, rater/occasion effects, and sensitivity to context--rather than reduced to a single agreement coefficient. These findings support studying AI as an assessor system without assuming interchangeability with expert humans.

Accordingly, the present comparison focuses on measurement behavior rather than on attributing human psychological capacities to the model. Agreement and disagreement are treated as properties of two assessor systems applied to the same performance samples; possible mechanisms behind those patterns are considered only as hypotheses for future research.

2.4. Feedback Quantity, Coverage, Usefulness, and Human-AI Complementarity

Feedback is not merely an output of scoring; it is information intended to support learners’ interpretation, self-monitoring, and subsequent action. Effective feedback should identify relevant performance features, make the basis of judgment understandable, and offer usable directions for improvement (Carless & Boud, 2018; Carless et al., 2011; Hattie & Timperley, 2007). In speaking tasks, comments on pronunciation, lexical choice, grammatical control, pausing, repair, and linguistic complexity can direct attention toward components of performance. Generative AI may increase the amount and breadth of such information (Lu et al., 2024; Song & Song, 2023; Su et al., 2023), but quantity, thematic coverage, usefulness, and educational quality are not equivalent constructs. The present study directly measures only the number of comments meeting an operational usefulness criterion and the themes those comments cover; it does not test whether recipients learn more from AI feedback or whether AI feedback is superior in pedagogical quality.

2.5. Teacher Acceptance, Trust, and Human Oversight

The educational value of AI assessment also depends on trust, interpretability, and compatibility with professional judgment. EFL educators and learners generally report perceived usefulness alongside concerns about transparency, reliability, access, and appropriate reliance (Dehghani & Mashhadi, 2024; Liu & Ma, 2024). Human-AI interaction research similarly emphasizes critical evaluation rather than uncritical acceptance of model output (Li et al., 2025). Speaking assessment is particularly useful for exploring these issues because communicative meaning may depend on pragmatics, interpersonal appropriateness, learner history, and cultural context. In this study, such issues are examined only through exploratory reflections from the four participating teacher-raters.

2.6. Research Gap, Objectives, Hypotheses, and Research Questions

Recent research has begun to evaluate LLMs in speaking assessment (Aryadoust et al., 2026; Choi, 2026; Voskoboinik et al., 2025), but three gaps remain. First, little evidence separates run-to-run variability within one AI system from variability among independent expert raters using parallel repeated-rating designs. Second, AI feedback studies often blur amount or category coverage with educational quality, although these are distinct outcomes. Third, psychometric comparisons are rarely integrated with post-assessment reflections from the same expert raters about trust, context, and appropriate human oversight. The present mixed-methods design addresses these gaps with G-theory, a descriptive feedback-counting framework, and exploratory teacher interviews.

For the quantitative components, two theory-informed, analysis-guiding expectations were made explicit. H1: A single independent teacher rating would be more dependable than a single ChatGPT-4o session, while aggregation would improve dependability for both assessor systems. H2: ChatGPT-4o sessions would generate more rubric-relevant comments and cover more CAF categories than individual teachers. H2 concerns observed quantity and coverage, not pedagogical quality.

The following research questions were examined: RQ1. How dependable are ChatGPT-4o and teacher holistic and analytic scores of the 30 EFL speech samples, and how do person, teacher-rater/AI-session, and interaction components differ? RQ2. How do ChatGPT-4o and teacher feedback differ descriptively in the quantity of comments meeting the operational useful-comment criterion and in thematic coverage across accuracy, fluency, and complexity? RQ3. How do the four participating teacher-raters reflect on the appropriate division of labor between AI assessment and human judgment, particularly for context-sensitive aspects of speaking performance?

3. Materials and Methods

3.1. Research Ethics Approval

The study followed the ethical principles of the American Educational Research Association (2011). Ethical approval was obtained from the Research Ethical Review Board of the corresponding authors’ institution. Participants were informed of the purpose and procedures of the study, the voluntary nature of participation, and the confidentiality of their responses. Informed consent was obtained before data collection.

3.2. EFL Speech Samples

The source population comprised first-year English-major undergraduates at the participating Chinese university who completed the common final English-speaking examination. The examination required an approximately one-minute oral presentation on ‘How to maintain a good relationship with your friends or classmates?’ Speech recordings from this common examination formed the source pool for the assessor-comparison study. The study did not create separate treatment and control groups of students; instead, the same performance sample was evaluated under both assessor systems.

Inclusion required first-year English-major status, completion of the target final-examination task, availability of a complete and assessable speech recording, and informed consent for research use. A recording not meeting these requirements was not eligible. From the source pool, 30 complete eligible recordings constituted the analytic sample and represented a range of observed EFL speaking performance. No student-level intervention or random assignment was involved. The comparison was at the assessment-system level: each of the same 30 recordings was rated by four independent teachers and on four repeated ChatGPT-4o occasions. This common-sample design permits exploratory estimation of person, assessor/occasion, and interaction variance; larger and more diverse samples are required for stronger population generalization.

3.3. Assessors

Four EFL speaking teachers from the same Chinese university were selected through convenience sampling as human assessors. The group included three women and one man, each with more than 10 years of experience teaching and assessing university EFL speaking. Their experience supports their use as an experienced-rater comparison panel, but the four-person, single-institution panel was neither demographically balanced nor population-representative; gender was not a sampling or analytic factor. Accordingly, the teacher ratings are treated as an experienced human benchmark for this study, not as an unbiased gold standard. ChatGPT-4o served as the AI assessor. The human condition therefore involved four independent human raters, whereas the AI condition involved four repeated applications of the same model. An AI session is a measurement occasion, not an independent artificial rater.

3.4. Rating Procedures and Reproducibility

The same CAF domain framework was applied in the human and AI conditions. Each speech received four standalone ratings: one holistic score and one overall score each for accuracy, fluency, and complexity. Accuracy was judged with reference to pronunciation, vocabulary choice/precision, and grammatical form; fluency with reference to rate/pacing, pauses or hesitations, and repair behaviors such as self-correction and reformulation; and complexity with reference to lexical range/sophistication and grammatical elaboration. These sub-features served as criteria informing one overall score for each analytic domain; they were not separately scored and then summed or averaged. The holistic score was likewise a standalone judgment and was not computed from the three analytic scores. Higher stored values indicate stronger performance. The same domain definitions were used across assessor systems. The original score sheets include the integer scores assigned by teachers and ChatGPT-4o. Therefore, we report the observed score ranges in Table 1.

Table 1. Descriptive statistics for rating scores by ChatGPT-4o session and teacher.

Assessor/Session	Holistic	Accuracy	Fluency	Complexity
ChatGPT-4o session 1	6.77 (0.68), 5–8	6.77 (0.68), 5–8	6.77 (0.68), 5–8	6.77 (0.68), 5–8
ChatGPT-4o session 2	6.97 (0.67), 5–8	6.97 (0.67), 5–8	6.97 (0.67), 5–8	6.90 (0.66), 5–8
ChatGPT-4o session 3	7.07 (0.78), 5–8	7.07 (0.78), 5–8	7.03 (0.76), 5–8	7.07 (0.78), 5–8
ChatGPT-4o session 4	6.90 (0.71), 6–8	6.90 (0.71), 6–8	6.83 (0.70), 6–8	6.90 (0.71), 6–8
Teacher 1	7.53 (0.94), 6–9	7.93 (0.91), 6–9	7.27 (1.28), 5–9	7.47 (0.97), 5–9
Teacher 2	6.90 (1.16), 5–9	7.47 (1.04), 5–9	6.80 (1.13), 5–9	6.83 (1.15), 5–9
Teacher 3	6.77 (1.04), 5–9	7.20 (1.10), 5–9	6.40 (1.22), 4–9	6.53 (1.20), 4–9
Teacher 4	7.00 (1.05), 5–9	7.23 (1.04), 5–9	6.80 (1.06), 5–9	7.03 (1.00), 5–9

Note. Cells report M (SD), observed range; n = 30 speech performances per assessor/session. Ranges are observed in the original score files.

Data collection procedures

Human-rater calibration and data collection

Before formal rating, the four teachers participated in online calibration workshops. The protocol documents three calibration stages: (a) joint review of the holistic and accuracy-fluency-complexity criteria and domain definitions; (b) independent practice scoring of three speech samples using the same framework intended for formal rating; and (c) comparison and discussion of rating decisions, especially disagreements in how accuracy, fluency, complexity, and overall performance were interpreted. The purpose was to establish a shared understanding of the rubric, not to force identical scores or remove genuine rater variability. After the discussion, formal ratings were completed independently so that inter-rater and person-by-rater variability remained observable.

ChatGPT-4o was accessed through the ChatGPT interface; the study record identifies 19 August 2024 as the access date. Fixed task instructions were used across sessions. The model was instructed to evaluate the same speech samples, first holistically and then analytically for accuracy, fluency, and complexity, and to provide written feedback after analytic scoring. No fine-tuning was performed, and human assessors did not interact with the model during scoring. The ChatGPT interface did not expose a backend snapshot identifier or researcher-controlled decoding settings (e.g., temperature, top_p, frequency or presence penalties, and random seed), so these cannot be retrospectively recovered. The verbatim historical prompt logs and the full original scale-anchor document were not retained. Appendix A therefore reports all recoverable settings and provides task instructions for replication.

Human ratings were collected in two phases. Each teacher first independently assigned holistic scores to all 30 speech samples. After a one-week interval, the teachers completed analytic ratings of accuracy, fluency, and complexity and provided written feedback. Separating the holistic and analytic phases reduced immediate carryover between scoring formats.

ChatGPT-4o ratings were collected in four separate sessions spaced three days apart. Within each session, holistic scoring preceded the analytic phase, and a one-week interval was maintained between holistic and analytic scoring. The same 30 speech recordings and retained task specification were used across all four AI sessions. For G-theory, the teacher and AI data were analyzed in separate person × facet studies with the same 30 × 4 data structure: person × teacher-rater for the human condition and person × AI-session for the model condition. This structural parallelism permits comparison of variance decompositions and D-study curves while preserving the substantive distinction between differences among four people and run-to-run variation within one AI system.

The protocol indicates direct use of the speech recordings with ChatGPT-4o. No separate researcher-generated transcript or acoustic-feature extraction procedure is documented in the study materials. File format, sampling rate, and platform-side audio preprocessing details were not preserved. Because the platform’s internal audio-processing pipeline was not exposed to the researchers, the manuscript does not infer whether the system relied on native audio features, an internal transcription stage, or a combination of processes (Appendix A).

After scoring was completed, the same four teachers who had served as raters participated in interviews about ChatGPT-4o as a classroom assessment aid. They were shown the major quantitative findings before the interviews and asked to reflect on observed similarities, differences, and appropriate uses of AI. This component was designed as an exploratory, post-results follow-up rather than as an independent qualitative sample. Exposure to the quantitative results may have shaped the interview responses, so the themes are interpreted as participant reflections that help contextualize the rating study, not as population-level evidence about teacher attitudes.

3.5. Data Analysis Methods

3.5.1. Data Analysis for the First Research Question

To address RQ1, human-rater intraclass correlation coefficients (ICCs) and G-theory analyses were used. Human inter-rater reliability was specified as a two-way random-effects, absolute-agreement, single-measure ICC [ICC(2,1), equivalent to ICC(A,1)] because the four teachers were treated as a sample from the broader population of qualified EFL assessors and absolute score agreement was relevant (Koo & Li, 2016; McGraw & Wong, 1996). Univariate G-theory was used for holistic scores and multivariate G-theory for the three analytic domains (Crick & Brennan, 1983). The two G-studies were fitted separately: the human facet represented teacher-rater identity, whereas the AI facet represented repeated session/occasion within the same ChatGPT-4o system. Person, facet, and person-by-facet components were estimated within each system. D-studies then projected criterion-referenced phi coefficients for aggregating one through ten independent teacher ratings or one through ten repeated AI assessments. Because the facets differ substantively, facet-level variance is interpreted within system rather than as if teacher-rater variance and AI-session variance represented the same psychological source.

3.5.2. Data Analysis for the Second Research Question

To address RQ2, feedback from ChatGPT-4o and the teachers was analyzed quantitatively and qualitatively. The unit for quantitative counting was a distinct feedback statement assigned to one of the three performance domains. A statement met the study’s operational “useful-comment” criterion if it satisfied at least one of two conditions: (a) it identified a specific, rubric-relevant performance feature, or (b) it provided an understandable and supportive direction for improvement. The two conditions were inclusive rather than cumulative; a statement meeting both was counted once. Generic praise or evaluation that neither identified a rubric-relevant feature nor provided an improvement direction was not counted. This operational label does not establish pedagogical effectiveness or comparative feedback quality.

For feedback quantity, the data include one domain-specific total for each of the four ChatGPT-4o sessions and four teachers, with every total aggregating comments across the same 30 speech performances. The underlying speech-by-assessor/session count matrix is not available. Because a valid inferential comparison would need to model repeated observations of the same speeches (for example, with a crossed mixed-effects Poisson or negative-binomial count model), Welch tests and standardized effect sizes are not reported. We therefore summarize total counts, assessor/session means, and standard deviations descriptively. These results support claims about observed feedback quantity and coverage only.

For qualitative analysis, the first two researchers independently double-coded feedback from two speech samples, compared decisions, resolved discrepancies by discussion, and refined the coding rules before the remaining material was analyzed. Codes were organized into themes within accuracy, fluency, and complexity. Formal intercoder reliability (e.g., Cohen’s kappa or Krippendorff’s alpha) was not calculated because the double-coding phase served as calibration and consensus rather than an independent reliability study; this is explicitly a limitation.

3.5.3. Data Analysis for the Final Research Question

The four teacher interviews were analyzed thematically as an exploratory post-results component. The first two researchers independently coded initial material, compared categories, resolved disagreements through discussion, and consolidated themes (Creswell & Creswell, 2023). No formal intercoder reliability coefficient was calculated. Because the interviewees were the same four teacher-raters and had already seen the principal quantitative findings, the analysis is used to document their reflections on the rating comparison; it is not treated as an independent triangulation sample, evidence of thematic saturation, or a basis for generalizing teacher perceptions beyond these participants.

4. Results

4.1. Findings for the First Research Question

4.1.1. Descriptive Statistics for Rating Scores

Before the reliability analyses, the raw rating files were summarized by assessor/session. Table 1 reports the mean, standard deviation, and observed range for holistic, accuracy, fluency, and complexity scores for each of the four ChatGPT-4o sessions and each of the four teachers. These summaries permit direct inspection of scale use and dispersion.

The four ChatGPT-4o sessions used a relatively narrow portion of the observed score distribution: holistic means ranged from 6.77 to 7.07 and holistic SDs from 0.67 to 0.78. Teacher means and dispersions varied more across raters; for example, holistic means ranged from 6.77 to 7.53 and SDs from 0.94 to 1.16. Analytic scores showed the same general contrast, including observed teacher scores as low as 4 in fluency and complexity. These descriptive patterns are not tests of assessor severity; they provide distributional context for the ICC and G-theory analyses that follow.

4.1.2. Reliability of Holistic Scoring

Univariate G-theory was used to evaluate holistic ratings across the 30 speech samples. The conventional human-rater ICC benchmark was 0.65, 95% CI [0.46, 0.80], for holistic scoring. Analytic ICCs were 0.65 for accuracy, 95% CI [0.45, 0.80], 0.61 for fluency, 95% CI [0.42, 0.76], and 0.65 for complexity, 95% CI [0.44, 0.80]. These ICCs describe agreement among the four human raters; the repeated AI sessions are evaluated below as an occasion facet rather than as four independent raters.

Table 2 decomposes holistic-score variance separately for the two assessor systems. For ChatGPT-4o, person variance accounted for 54.24% of total variance, AI-session variance for 1.59%, and person-by-session residual variance for 44.17%. The small session component indicates little shift in average scoring level across repeated model runs; it should not be interpreted as agreement among independent AI raters.

Table 2. Univariate G-study variance components for holistic scoring.

Rater type	Sources of Variability	DF	Variance	%
ChatGPT-4o	Person	29	0.2799	54.24
	AI session	3	0.0082	1.59
	Person × AI session	87	0.2279	44.17
	Total	119	0.5160	100
Teachers	Person	29	0.7852	65.33
	Teacher rater	3	0.1025	8.53
	Person × teacher rater	87	0.3142	26.14
	Total	119	1.2019	100

Note: The human facet is teacher-rater identity; the AI facet is repeated ChatGPT-4o session/occasion. These facet components have different substantive meanings and are interpreted within assessor system.

For teachers, person variance accounted for 65.33% of holistic-score variance, teacher-rater variance for 8.53%, and person-by-teacher residual variance for 26.14% (Table 2). Teacher-rater variance represents differences among four people in average severity or scale use; AI-session variance represents run-to-run occasion effects within one model. The matched 30 × 4 data structures make dependability comparisons useful, but the facet variances are not substantively interchangeable.

The D-studies translated the variance components into criterion-referenced phi coefficients for aggregating one through ten assessments (Table 3; Figure 1). For teachers, k denotes the number of independent teacher ratings; for ChatGPT-4o, k denotes the number of repeated model sessions. These projections are planning illustrations for this design rather than universal reliability thresholds.

Table 3. D-study phi coefficients for holistic ratings.

Assessments Aggregated (k)	Repeated ChatGPT-4o Sessions, phi	Independent Teachers, phi
1	0.54	0.65
2	0.70	0.79
3	0.78	0.85
4	0.83	0.88
5	0.86	0.90
6	0.88	0.92
7	0.89	0.93
8	0.90	0.94
9	0.91	0.94
10	0.92	0.95

Figure 1. Comparison of holistic scoring reliability by ChatGPT-4o and human teachers.

As shown in Table 3 and Figure 1, phi was 0.54 for one ChatGPT-4o session and 0.65 for one independent teacher, increasing to 0.83 and 0.88, respectively, at k = 4. Using phi >= 0.80 as an illustrative planning target, the holistic design required four repeated AI sessions but three independent teachers; phi >= 0.90 required eight AI sessions but five teachers.

Overall, both assessor systems differentiated among the observed speaking performances, but teacher ratings were more dependable per assessment. Repeated AI assessment narrowed the dependability gap, while requiring more occasions to reach the same illustrative planning targets.

4.1.3. Reliability of Analytic Scoring

Multivariate G-theory was used to evaluate analytic scores for accuracy, fluency, and complexity (Brennan, 2001). Table 4 reports the variance components for each domain under the separate person × teacher-rater and person × AI-session designs.

Table 4. Multivariate G-study variance components for analytic ratings.

Assessor System	Source	Accuracy		Fluency		Complexity
		Variance	%	Variance	%	Variance	%
ChatGPT-4o	Person	0.2799	54.24	0.2751	54.70	0.2883	56.15
	AI session	0.0082	1.59	0.0075	1.49	0.0079	1.54
	Person × AI session	0.2279	44.17	0.2203	43.81	0.2172	42.31
	Total	0.5160	100	0.5029	100	0.5134	100
Teachers	Person	0.7490	65.03	0.9058	60.60	0.8546	64.95
	Teacher rater	0.1044	9.06	0.1096	7.33	0.1427	10.85
	Person × teacher rater	0.2984	25.91	0.4793	32.07	0.3184	24.20
	Total	1.1518	100	1.4947	100	1.3157	100

Note: AI-session components represent repeated occasions within ChatGPT-4o; teacher-rater components represent differences among independent human raters.

Across all three analytic domains, person variance was the largest component in both systems. ChatGPT-4o person variance ranged from 54.24% to 56.15%, AI-session variance remained below 2%, and person-by-session residual variance ranged from 42.31% to 44.17%. The low occasion component therefore coexisted with substantial person-specific run-to-run variation.

Teacher person variance ranged from 60.60% to 65.03%, teacher-rater variance from 7.33% to 10.85%, and person-by-teacher residual variance from 24.20% to 32.07% (Table 4). The teacher facet again showed more between-rater severity variation but less person-specific residual variation than the repeated AI-session design.

These component patterns should be read according to facet type: differences among teachers concern independent human judges, whereas AI-session differences concern repeated occasions of one model. Accordingly, no psychological equivalence between the two facet components is assumed.

The analytic D-studies projected phi coefficients for one through ten aggregated assessments (Table 5; Figure 2), again distinguishing independent teacher ratings from repeated ChatGPT-4o sessions.

Table 5. D-study phi coefficients for analytic ratings.

k	AI Accuracy	Teacher Accuracy	AI Fluency	Teacher Fluency	AI Complexity	Teacher Complexity
1	0.54	0.65	0.55	0.61	0.56	0.65
2	0.70	0.79	0.71	0.75	0.72	0.79
3	0.78	0.85	0.78	0.82	0.79	0.85
4	0.83	0.88	0.83	0.86	0.84	0.88
5	0.86	0.90	0.86	0.88	0.86	0.90
6	0.88	0.92	0.88	0.90	0.88	0.92
7	0.89	0.93	0.89	0.92	0.90	0.93
8	0.90	0.94	0.91	0.92	0.91	0.94
9	0.91	0.94	0.92	0.93	0.92	0.94
10	0.92	0.95	0.92	0.94	0.93	0.95

Figure 2. Comparison of analytic scoring reliability by ChatGPT-4o and human teachers.

At k = 1, phi coefficients were 0.54, 0.55, and 0.56 for ChatGPT-4o accuracy, fluency, and complexity, compared with 0.65, 0.61, and 0.65 for teachers. At k = 4, the corresponding coefficients were 0.83, 0.83, and 0.84 for repeated AI sessions and 0.88, 0.86, and 0.88 for independent teachers.

Using phi >= 0.80 as an illustrative target, all three analytic domains required four repeated AI sessions but three independent teachers. For phi >= 0.90, accuracy required eight AI sessions versus five teachers, fluency eight AI sessions versus six teachers, and complexity seven AI sessions versus five teachers.

Thus, aggregation improved dependability in both systems, but fewer independent teacher ratings than repeated AI sessions were needed to reach the same illustrative thresholds. This is a statement about measurement dependability in the present design, not about the intelligence or general competence of either assessor system.

Taken together, the analytic results show that ChatGPT-4o captured systematic between-person performance differences and benefited substantially from repeated assessment, while independent teacher ratings had a per-assessment dependability advantage across all three domains.

4.2. Findings for the Second Research Question

Descriptive Feedback Quantity and Thematic Coverage

Feedback on the 30 speech samples was summarized to compare the number and thematic coverage of comments meeting the operational useful-comment coding rule. Because only assessor/session-level aggregate counts are available, this component is descriptive rather than inferential.

Table 6 shows a pronounced descriptive difference in comment volume. Across the four ChatGPT-4o sessions, total counts were 351 for accuracy, 331 for fluency, and 325 for complexity, corresponding to assessor/session means of 87.75, 82.75, and 81.25. Teacher totals were 173, 145, and 85, with means of 43.25, 36.25, and 21.25. These aggregates all derive from the same 30 performances; no p values or standardized effect sizes are reported.

Table 6. Descriptive feedback quantity under the useful-comment coding rule.

Domain	Assessor/Session	Assessors/ Sessions (n)	Total	Mean	SD
Accuracy	ChatGPT-4o sessions	4	351	87.75	0.87
	Teachers	4	173	43.25	3.12
Fluency	ChatGPT-4o sessions	4	331	82.75	0.93
	Teachers	4	145	36.25	4.18
Complexity	ChatGPT-4o sessions	4	325	81.25	1.02
	Teachers	4	85	21.25	4.75

The larger ChatGPT-4o totals are therefore interpreted only as an observed difference in feedback quantity under the coding rule. A speech-level count matrix would be required for an inferential mixed-effects comparison that accounts for repeated assessment of the same performances.

The thematic analysis examined what the comments addressed. As summarized in Table 7, feedback clustered into pronunciation, vocabulary, and grammatical accuracy; speed, breakdown, and repair fluency; and lexical and grammatical complexity. These categories describe feedback content and coverage, not comparative pedagogical quality.

Table 7. Themes and codes used to characterize feedback coverage.

Domain	Themes	Codes
Accuracy	Accuracy in pronunciation	correct articulation of sounds; stress; intonation; rhythm and a natural flow
	Accuracy in vocabulary choices	word choice appropriateness; correct use of collocations; precision and specificity
	Accuracy in grammar structures	sentence structure; verb tenses and forms; subject-verb agreement; correct use of articles, prepositions, and pronouns; correct word order
Fluency	Speed fluency	speech rate; pacing; natural rhythm
	Breakdown fluency	filled pauses; unfilled pauses; hesitations and interruptions
	Repair fluency	self-corrections; reformulations; false starts; repetitions
Complexity	Lexical complexity	lexical range (variety of words); lexical sophistication (advanced vocabulary); lexical density (content words); lexical diversity (variety within specific categories)
	Grammar complexity	sentence length; sentence structure (variety of sentences); subordination (dependent clauses); embedding (nested clauses); use of passive voice

Eight major themes were identified. Accuracy feedback addressed pronunciation, vocabulary choice, and grammatical structure. Fluency feedback addressed speech rate and pacing, pauses and interruptions, and self-correction or reformulation. Complexity feedback addressed lexical range and sophistication and the structural elaboration of grammar. Both assessor systems produced comments that could be mapped onto this common CAF framework.

The two systems differed descriptively in breadth and presentation. ChatGPT-4o more often addressed multiple rubric categories within a performance, whereas teacher feedback was typically shorter and more selective. Illustratively, one AI response separately addressed pronunciation/stress, lexical precision, grammar, pacing, pauses/repairs, lexical range, and grammatical elaboration, while one teacher response combined pronunciation, vocabulary misuse, grammar, coherence, sentence simplicity, and fluency in a shorter holistic paragraph. The examples illustrate coverage and format only; they do not establish which feedback is more educationally effective.

In summary, the descriptive pattern was consistent with H2: the four ChatGPT-4o sessions produced more comments meeting the coding criterion and broader CAF-category coverage than the individual teachers. Because speech-level count data and learning-outcome evidence are unavailable, this finding should not be converted into an inferential claim or a claim of superior feedback quality.

4.3. Findings for the Third Research Question

Exploratory Teacher-Rater Reflections on Human-AI Complementarity

Interview analysis identified four themes. Because the interviewees were the same four teacher-raters and had reviewed the major quantitative findings beforehand, the themes are reported as exploratory post-results reflections rather than as an independent sample of teacher perceptions.

Theme 1: AI as a supplement rather than a substitute for expert judgment

Teachers’ positions ranged from cautious experimentation to strong support, but the responses converged on a supplementary role. Teacher A suggested that the tool “might be tried on a small scale, but not implemented widely in teaching.” Teacher B was “strongly in favor of applying AI tools to classroom assessment” if technical barriers were resolved, and Teacher C stated “I tend to use” the tool. Teacher D most explicitly articulated the human-AI boundary, arguing that ChatGPT-4o could be an auxiliary tool but “cannot fully replace the professional judgment of human teachers,” especially when nuanced interpretation is required.

Theme 2: Dependability is necessary but not sufficient for defensible assessment

Teachers generally acknowledged that ChatGPT-4o could apply explicit criteria consistently, particularly for accuracy and fluency, but they did not treat score reliability as the sole criterion for good assessment. Teacher B described the model as “a very powerful tool,” and Teacher C noted that large models “are performing well.” Teacher A nevertheless emphasized that “human grading is crucial” for oral assignments requiring deeper interpretation. Teacher D similarly argued that creative expression and humor may be difficult for AI to evaluate. These comments show how the participating teachers themselves distinguished psychometric consistency from broader interpretive competence.

Theme 3: Broad AI feedback versus contextualized human feedback

Teachers valued the speed and comprehensiveness of ChatGPT-4o feedback but questioned whether breadth translated into depth or personalization. Teacher A considered feedback on accuracy and fluency useful but said complexity feedback “needs human verification to ensure accuracy.” Teacher C emphasized efficiency, while Teacher D argued that AI feedback “may lack depth and personalization” because it is less sensitive to emotion and cultural context. These are perceptions of relative strengths and limitations rather than independent demonstrations of feedback quality.

Theme 4: Trust, transparency, and socially situated human judgment

Trust was closely tied to transparency and contextual fit. Teacher B argued that understanding how AI generates scores would make it “easier for teachers to trust and adopt its assessments.” Teacher D warned that over-reliance might “overlook individual differences among students and emotional factors in the learning process.” Teacher C questioned whether the model’s training data adequately represented Chinese university students, and Teacher A raised concerns about strategic misuse. These accounts identify context, learner knowledge, culture, and consequences as areas the teachers believed require human oversight.

Overall, the four participating teacher-raters valued AI speed, coverage, and structured criteria while preferring human oversight for emotion, humor, culture, learner history, and pedagogical purpose. These reflections help explain how the participants interpreted the psychometric findings but do not establish population-level teacher views or independently measured cognitive mechanisms.

5. Discussion

5.1. Discussion of Findings

This study compared independent expert teacher ratings with repeated ChatGPT-4o sessions as two assessor systems applied to the same 30 speaking performances. Its contribution to intelligence research is psychometric: observed individual differences on complex performance tasks depend not only on the persons being assessed but also on the system that converts performance into scores. This measurement issue parallels recent Journal of Intelligence work on rater effects in divergent-thinking assessment and on LLM scoring of open-ended cognitive-performance responses (Pellegrino et al., 2025; Saretzki et al., 2025; op ‘t Hof et al., 2025).

First, teachers produced more dependable measurements per assessment. A single teacher reached phi = 0.65 holistically versus 0.54 for one ChatGPT-4o session, and the same advantage appeared across accuracy, fluency, and complexity. Repetition substantially improved AI dependability, but the D-study translates the difference into a practical design implication: phi >= 0.80 required four repeated AI sessions but three independent teachers across all four scoring outcomes. For more stringent phi >= 0.90 targets, the AI design generally required seven or eight sessions compared with five or six teachers.

Second, the G-study clarifies why the same 30 × 4 structure should not be mistaken for equivalent facets. Teacher-rater variance reflects differences among people in average severity or scale use, whereas AI-session variance reflects occasion-to-occasion shifts within one model. ChatGPT-4o showed very little session-level mean shift but comparatively large person-by-session residual variation, indicating that a model can be stable in overall severity while still varying in how particular performances are treated across runs. Teacher ratings showed more between-rater severity variation but less person-specific residual variation. Explanations involving expert cue integration, prompt sensitivity, stochastic model behavior, or audio processing remain hypotheses because those mechanisms were not directly measured.

Third, ChatGPT-4o generated substantially more comments meeting the operational coding rule and covered more CAF categories in the retained aggregates. These differences are presented descriptively rather than through Welch tests or standardized effect sizes because all assessor/session totals derive from the same 30 performances and the speech-level count matrix is unavailable. More feedback is not necessarily better feedback: learner uptake, revision quality, retention, motivation, and later speaking performance were not measured.

Fourth, the interview component adds exploratory participant interpretation rather than independent confirmation. The same four teacher-raters, after seeing the major quantitative findings, described AI as efficient and broad but emphasized human oversight for context-sensitive and interpersonal judgments. These reflections are useful for generating hypotheses about professional judgment, trust, and human-AI division of labor, but the design does not support generalization to teachers as a population.

5.2. Implications for Intelligence Research and Pedagogy

For intelligence and assessment research, the results emphasize a broader methodological point: when AI is used to score open-ended or complex performance tasks, the assessor system itself should be modeled as a potential source of measurement error. Human-AI agreement alone is insufficient; researchers should estimate occasion or rater effects, person-by-assessor interactions, and the gain in dependability from aggregation. Recent intelligence research on LLM scoring and rater effects reaches a similar conclusion that scoring method can influence the measurement of individual differences (op ‘t Hof et al., 2025; Pellegrino et al., 2025; Saretzki et al., 2025). Reproducible AI assessment also requires archiving model snapshots, prompts, scoring instructions, and technical input-processing details.

For pedagogy, the present results support a complementary workflow. Repeated AI assessments can add rubric-relevant observations and reduce run-specific instability through aggregation, whereas teachers provide an independent source of judgment and retain responsibility for consequential interpretation. The study does not identify a universally optimal human-AI combination; rather, it shows how G-theory can make the reliability consequences of different combinations explicit.

5.3. Limitations and Future Directions

Several limitations constrain interpretation. The speaking task was not designed as an intelligence test, and no independent measures of working memory, attention, reasoning, or other cognitive processes were administered. The intelligence contribution is therefore limited to psychometric measurement of individual differences and assessor effects. The original rating files preserve the assigned numerical scores, but the original documentation of the full nominal score range and point-by-point verbal scale anchors was not retained; exact equivalence of the historical numerical anchors across assessor systems cannot therefore be independently verified.

Additional limitations include the 30 speech performances, four teacher-raters, one institution and cultural context, one speaking task, and one AI model. The teacher panel was a small convenience sample from one institution (three women and one man), so the study cannot establish that the human ratings were unbiased or generalizable to the wider population of expert EFL assessors; gender and other rater characteristics were not modeled. The feedback archive retains only assessor/session-level domain totals, so a speech-by-assessor mixed-effects count analysis cannot be reconstructed and feedback quantity is reported descriptively. The interviews involved the same four raters, were conducted after they had seen the quantitative findings, and used consensus coding without a formal intercoder-reliability coefficient; they should therefore be treated as exploratory participant reflections rather than independent triangulation. Model provenance is partly incomplete: the study record identifies 19 August 2024 as the ChatGPT-4o access date, but the exact backend snapshot, verbatim historical prompts, researcher-controlled decoding settings, the full historical scale-anchor document, and platform-side audio-processing details were not retained or exposed. Only ChatGPT-4o was studied. A post hoc test with a competing model was not added because a model run conducted in 2026 would use noncontemporaneous model versions and settings and would introduce a new analysis rather than a controlled comparison with the August 2024 ChatGPT-4o condition. Future work should prospectively compare multiple models using the same archived audio, contemporaneous model versions, identical prompts and documented settings, and should recruit larger, gender-balanced, multi-institutional expert panels; it should also model speech-level feedback counts, use independent teacher samples and blinded interviews, and directly measure cognitive constructs when making cognitive-process claims.

Overall, the evidence is consistent with a potentially complementary use of human and AI assessment in this setting while specifying the measurement trade-off more precisely: independent teachers yielded higher dependability per rating, repeated ChatGPT-4o sessions improved dependability through aggregation, and AI produced more rubric-relevant feedback comments descriptively. The most direct contribution is not a claim about which system is more intelligent, but evidence that human and AI assessor systems introduce different patterns of measurement variability when individual differences in complex performance are scored.

6. Conclusions

The study addressed its general objective by integrating psychometric, descriptive feedback, and exploratory interview evidence on human-AI assessment. Specific Objective 1 was met: teachers showed higher single-assessment dependability, while repeated ChatGPT-4o sessions progressively narrowed the gap; H1 was consistent with the observed D-study pattern. However, ChatGPT-4o;s analytic dimensions showed limited differentiation from its holistic scoring: accuracy matched the holistic score in all 120 AI ratings, while fluency and complexity differed in only three and four ratings, respectively; the AI analytic-domain findings should therefore be interpreted cautiously. Specific Objective 2 was also addressed: the aggregate feedback records showed more rubric-relevant comments and broader CAF-category coverage from ChatGPT-4o, a descriptive pattern consistent with H2, while the four teacher-raters’ post-results reflections favored human oversight for context-sensitive interpretation. For intelligence research, the main implication is methodological: assessor-system variance and person-by-assessor interaction should be modeled whenever human or AI judgment is used to measure individual differences in complex performance. For language assessment, the findings suggest that carefully designed human-AI complementarity may be useful in comparable settings, but the present single-site study does not establish its superiority over other assessment arrangements.