Engagement Profiles in AI-Assisted ESAP: Associations with AI Use Indicators and Speaking Performance

Abstract

Generative AI (GenAI) offers opportunities for repeated practice and feedback in English language learning. However, how students participate in these learning activities requires further investigation. This study examined learning engagement profiles and their associations with GenAI use and speaking performance among 500 STEM master’s students in an AI-assisted ESAP speaking course at a Chinese university. Latent profile analysis used self-reported behavioral, emotional, and cognitive engagement scores collected in Week 15. Separate R3STEP models examined associations with self-reported weekly GenAI use frequency and average weekly online duration. BCH analyses compared posttest speaking performance across profiles while adjusting for pretest scores. Four profiles emerged: Low Engagement Profile (P1, 22.4%), High Engagement Profile (P2, 38.2%), Low Emotional Engagement Profile (P3, 24.0%), and Low Cognitive Engagement Profile (P4, 15.4%). Proportions were based on most likely assignments. After Holm correction, longer online duration was associated with higher odds of membership in P2, P3, and P4 relative to P1. More frequent GenAI use was associated with membership in P2 and P4 relative to P3. P2 had significantly higher baseline-adjusted overall speaking scores than all other profiles. These findings provide person-centered evidence of learner heterogeneity in AI-assisted ESAP. They highlight the value of considering multidimensional engagement configurations alongside GenAI use indicators when interpreting differences in speaking performance.

1. Introduction

1.1. Research Background

Academic English proficiency is important for STEM postgraduate students’ participation in scholarly communication. Their academic activities often involve presenting research findings, explaining technical processes, interpreting data, and discussing disciplinary issues. These tasks require students to integrate language skills with disciplinary knowledge and respond to questions in real time. Recent research has identified spoken communication as a challenge for international STEM postgraduates (Gleason & Leandro, 2023; Tan & Botha, 2026; Willoughby et al., 2025). In the English for Specific Academic Purposes (ESAP) curriculum, speaking courses address these needs by developing oral expression, content organization, and interactive responsiveness. Speaking instruction therefore helps students communicate disciplinary knowledge clearly and coherently while adapting their explanations to the audience.

Generative artificial intelligence (GenAI) offers new opportunities for speaking practice. Conversational systems can support contextualized interaction, language revision, repeated practice, and personalized feedback (W. Y. W. Lai & Lee, 2024; Lee et al., 2026). Research on AI-assisted speaking has reported improvements in oral performance, greater willingness to communicate, and more positive learning experiences (Ding & Muhyiddin B Yusof, 2025; Wu & Li, 2025). Beyond performance outcomes, related studies have examined affective engagement and self-regulated learning in AI-supported language education (J. Chen & Alibakhshi, 2026; Ekizer, 2025; Ma et al., 2025; Pan et al., 2025). However, these benefits depend on interaction arrangements, task support, and instructional design (Sok & Shin, 2025; Tai & Chen, 2024; Yang & Li, 2024). A recent systematic review indicates growing attention to learning processes and performance in this field (B. Li et al., 2025).

Learning engagement provides a multidimensional perspective on students’ participation in language learning (Hiver et al., 2020). Behavioral engagement reflects students’ effort and persistence; emotional engagement involves interest, value, and enjoyment, while cognitive engagement involves strategy use and cognitive processing (Fredricks et al., 2004). Recent studies have examined engagement across AI-supported educational settings (Jallali et al., 2026; Lo et al., 2024; L. Wang & Wang, 2025; Zhou & Liu, 2026). Intervention studies and meta-analyses have also examined engagement outcomes (Heung & Chiu, 2025; Y. Wang & Xue, 2024; Xia et al., 2025). However, improvements at the group level do not establish that all dimensions develop together within individual students. Understanding AI-assisted ESAP learning requires attention to both technology use and students’ specific engagement.

1.2. Person-Centered Research and Research Gaps

Existing research has examined AI-assisted language learning through instructional effectiveness, learning experiences, and technology usage (Law, 2024). Quasi-experimental studies compare language performance across instructional conditions (Fathi et al., 2024). Correlational analyses and structural equation modeling investigate relationships among learner perceptions, learning emotions, and engagement (Wu & Li, 2025; Zhang et al., 2024). These methods provide evidence about average effects and relationships among these variables. However, behavioral, emotional, and cognitive engagement may combine differently across learners. Sample averages may conceal these differences. Latent profile analysis (LPA) provides a person-centered approach to examine this heterogeneity. It identifies latent subgroups based on joint response patterns across continuous indicators. It captures both the overall levels and internal configurations across dimensions. Person-centered research has identified distinct configurations of motivation and engagement in foreign language learning, English-medium instruction, and online learning. These configurations have been linked to learning environments and outcomes (Guo et al., 2026; Hoi & Hang, 2022; Lei & Ye, 2025; F. Wang et al., 2025). Related studies have examined course experiences, teacher support, and heterogeneity at both student and institutional levels (L. Wang & Wang, 2025; Xing & Teng, 2026). This work extends profiling beyond class identification to examining associated learning conditions. In AI-supported language learning, profiles reflect both overall level differences and the relative strength of behavioral, emotional, and cognitive engagement (Xing & Teng, 2026; Xu & Li, 2024). These findings support understanding learner heterogeneity through multidimensional configurations. However, studies differ in their psychological indicators and course contexts. The meaning and applicability of identified profiles therefore need to be interpreted within specific learning settings. Research on AI use has further broadened the basis for classification. Learners have been grouped according to self-reported reliance on ChatGPT across tasks or observed speaking interaction behaviors. These groupings represent reported reliance and observable participation (Y. Chen et al., 2026a; Stojanov et al., 2024). Speaking research has also combined engagement questionnaires, dialogue records, and interviews to deepen understanding of learning processes (Min et al., 2026). These developments suggest that technology use indicators and psychological engagement configurations offer complementary perspectives. Use frequency and online duration describe the extent of use but do not, by themselves, capture emotional experiences or cognitive processing. The outcome measures also differ. The motivation–engagement studies discussed above mainly linked profiles to academic efficacy, satisfaction, or perceived skills. Chen compared speaking gains across behavioral clusters and found differences in fluency and coherence (Y. Chen et al., 2026a). These studies connect classification with educational outcomes, but their classifications and outcome measures differ. Engagement profiles therefore need to be examined in relation to specific usage characteristics and observed learning changes within a clearly defined course context.

Building on these developments, three gaps remain. First, engagement profiles have been studied across different learning contexts. However, evidence from AI-assisted ESAP speaking courses for STEM master’s students remains limited. These courses require both the communication of disciplinary content and real-time interaction. How behavioral, emotional, and cognitive engagement are configured among students in this setting therefore requires direct examination. Second, previous studies have investigated technology perceptions, use types, or specific interaction behaviors. However, the respective associations of AI use frequency and online duration with engagement profile membership remain insufficiently understood. Third, further evidence is needed on whether students in different profiles show different outcomes in speaking performance and its component dimensions in ESAP courses. Overall scores may obscure differences in professional content, organization and coherence, linguistic accuracy and range, fluency and intelligibility, and interactive responsiveness. Examining these dimensions would provide a more detailed understanding of speaking development.

1.3. Theoretical Framework

This study uses Kahu and Nelson’s student engagement framework in higher education (Kahu & Nelson, 2018). The framework situates engagement within the “educational interface,” where student and institutional factors interact to shape individual learning experiences. Engagement is understood as a dynamic psychosocial state, with reciprocal links to its associated conditions and learning outcomes. Guided by this perspective, the study focuses on behavioral, emotional, and cognitive engagement (Fredricks et al., 2004). These dimensions encompass task participation and persistence, emotional experiences during learning, and the mental effort involved in understanding content and applying strategies.

The framework provides a conceptual basis for the study’s three analytical components. First, its emphasis on individual learning experiences supports examining engagement heterogeneity within a shared course context. LPA is used to explore configurations of engagement among STEM master’s students in an AI-assisted ESAP speaking course. The analysis identifies profiles that differ in overall engagement levels or patterns across dimensions. Second, the distinction between engagement and its associated conditions helps clarify the relationship between technology use and psychological engagement. The study treats self-reported GenAI use frequency and average weekly online duration as external measures of technology use. Their associations with engagement profile membership are examined separately. Finally, the framework connects engagement with learning outcomes, providing a basis for comparing speaking development across profiles. The study examines overall speaking performance and its five dimensions to assess the pattern of profile differences across components.

The framework guides variable organization and the interpretation of statistical associations; its full psychological mechanisms and mediation pathways are not tested. Figure 1 summarizes the conceptual framework and its application to the study’s three analytical components.

Figure 1. Theoretical Framework of This Study.

1.4. The Present Study

This study examines STEM master’s students enrolled in an AI-assisted English for Specific Academic Purposes (ESAP) speaking course. First, latent profile analysis (LPA) identifies distinct learning engagement profiles. Second, the study investigates the relationships between GenAI use indicators and profile membership. Third, the study summarizes pretest scores, posttest scores, and speaking gains descriptively. It then compares posttest speaking performance across profiles after adjustment for the corresponding pretest scores.

Given the observational design, the findings are interpreted in terms of statistical associations and profile differences rather than causal effects. The study addresses the following research questions: RQ1: What latent learning engagement profiles can be identified among STEM master’s students in an AI-assisted ESAP speaking course? RQ2: How are self-reported GenAI use frequency and system-recorded weekly online duration each associated with engagement profile membership? RQ3: How does speaking performance differ across engagement profiles?

The theoretical framework views engagement as a process linking educational experiences with learning outcomes. Different engagement configurations may therefore correspond to different performance outcomes. Based on this framework and previous research, two hypotheses are proposed for the profile comparisons in RQ3: H1: Overall posttest speaking performance will differ across engagement profiles after adjustment for the overall pretest score. H2: Posttest performance in the five speaking dimensions will differ across engagement profiles after adjustment for the corresponding pretest scores.

2. Materials and Methods

2.1. Research Design

2.1.1. Participants

The participants were full-time, first-year master’s students enrolled in an English for Specific Academic Purposes (ESAP) speaking course at a science and engineering university in Northeast China. At the beginning of the course, 536 students from 15 classes were invited through convenience sampling, with approximately 36 students in each class. They represented science, technology, engineering, and mathematics (STEM) disciplines. Eligibility for the analytic sample required formal course registration and completion of the main learning activities. During screening, 36 cases were excluded for attendance below 80% (n = 12), missing pretest or posttest speaking scores (n = 5), missing responses to key questionnaire items (n = 8), low-quality responding (n = 5), or unmatched questionnaire and assessment records (n = 6). Each case was counted only under the first applicable exclusion criterion. Finally, 500 complete and valid cases were retained with an effective retention rate of 93.3%. The demographic characteristics of the analytic sample are summarized in Table 1. Most students had no full-time work experience before the course, whereas most had previously used GenAI for learning.

Table 1. Demographic and Learning Background Characteristics of the Participants.

Variable	Category	n	%
Gender	Male	337	67.4
	Female	163	32.6
Age	22 years or younger	181	36.2
	23–25 years	269	53.8
	26–28 years	43	8.6
	29 years or older	7	1.4
Academic discipline	Science (S)	133	26.6
	Technology (T)	123	24.6
	Engineering (E)	163	32.6
	Mathematics (M)	81	16.2
Work experience	No	462	92.4
	Less than 1 year	9	1.8
	1 to less than 3 years	25	5.0
	3 years or more	4	0.8
GenAI experience	No prior use	30	6.0
	Less than 6 months	279	55.8
	6 to less than 12 months	122	24.4
	12 months or longer	69	13.8

Note. Work experience refers to the cumulative duration of full-time employment before the course began. Students entering directly from undergraduate study generally have no prior full-time work experience. Internships, part-time employment, and periods of unemployment are excluded.

2.1.2. English for Specific Academic Purposes Speaking Course

This study was conducted in an ESAP speaking course for graduate students in STEM disciplines. The course carried three credits and included 48 contact hours across 16 weeks. The course aimed to develop academic speaking skills relevant to students’ disciplines. Activities progressed from basic expression to integrated academic communication. Its main content included explaining professional concepts, describing technical processes, delivering academic presentations, participating in discussions, and responding to questions. Specifically, the course was organized into three stages. The early stage focused on ESAP speaking requirements, professional terminology, and concept explanation. The middle stage covered technical processes, research methods, research findings, and academic literature. The final stage emphasized presentations, argumentation, spontaneous Q&A activities, and integrated professional tasks.

The course used Chaoxing and the FiF Oral Training System developed by iFLYTEK. Chaoxing was used to distribute course materials, assign tasks, and collect student work. FiF supported speaking practice, interaction, and automated feedback. Its AI companion module provided generative dialogue for simulated professional exchanges and academic Q&A practice. Teachers designed tasks involving concept explanation, technical process descriptions, and academic presentations. They provided scenarios, practice objectives, and initial prompts. Subsequent exchanges did not follow fixed scripts. Students could formulate their own responses, ask follow-up questions, request clarification, or rephrase their answers. The system provided automated scores and written comments. Feedback mainly addressed pronunciation accuracy, fluency, vocabulary, and grammar. Depending on the task, it also addressed the relevance of students’ responses. Students could review the feedback and practice again. They could also select relevant topics for supplementary practice beyond the required tasks. The platform’s automated scores and comments were used for practice feedback. They were not used as the speaking outcome scores in this study. The pretest and posttest were scored independently by human raters, as described in Section 2.2.3.

All classes followed the same core materials, task objectives, GenAI usage protocols, and assessment criteria to maintain instructional consistency. The 15 classes were taught by eight instructors. Some instructors taught more than one class. The average teaching assignment was approximately two classes per instructor. At the beginning of the course, teachers introduced the system’s basic functions and explained task requirements, automated feedback, and academic integrity guidelines. Students remained responsible for their final submissions and presentations. Teachers assessed students’ final performance and checked the accuracy of disciplinary content when necessary.

2.1.3. Data Collection

Data were collected in three stages. The first stage took place in Week 1. Before the course, the participants completed the baseline ESAP speaking evaluation (Pretest). The second stage occurred in Week 15. After the main course activities, they completed the learning engagement questionnaire. They also reported their average weekly GenAI use frequency outside the required course tasks. In the same week, each class teacher manually checked the online duration records in the AI system. Each teacher then exported these records for analysis. In Week 16, the participants independently finished the end-of-course assessment (Posttest). AI and other external aids were not permitted during either speaking test. After data collection, all data were matched using participants’ anonymous identification codes. These included pretest scores, questionnaire responses, system-recorded online duration data, and posttest scores. The matched data were then screened for completeness, response quality, and variable ranges according to the predefined criteria.

2.1.4. Ethical Considerations

The study was approved by the Ethics Review Committee of the College of Foreign Languages at Jilin Agricultural University (Approval No. 20260226S1). Participants were informed of the study purpose, procedures, data use, and voluntary nature of participation. They had the right to refuse or withdraw from the study at any time. Their decisions did not affect their course participation, teacher evaluations, or course grades.

Research data were coded and matched using unique identifiers, and no direct personal identifiers were retained in the analytical files. Because the study used internal evaluation data from a course-based teaching reform, the raw data are not publicly available. Researchers may request access for academic purposes by contacting the first author after approval by the ethics review committee.

2.2. Measures and Measurement

2.2.1. Learning Engagement Scale

Learning engagement was measured using an adapted School Engagement Measure (Fredricks et al., 2004, 2005). This framework distinguishes behavioral, emotional, and cognitive engagement. Behavioral engagement concerns attention, effort, and persistence. Emotional engagement reflects affective experiences such as interest and enjoyment. Cognitive engagement involves the use of cognitive and metacognitive strategies. All behavioral and emotional items were retained, while four cognitive items were omitted. The omitted items concern studying at home without an upcoming test, watching course-related television programs, reading extra books, and discussing classroom learning with people outside school. These items emphasize extracurricular activities or opportunities that are less directly aligned with the task-related strategies targeted in this study. The original English items were translated into Chinese and contextualized for the GenAI-assisted ESAP speaking course. For example, “I pay attention in class” was adapted to “I pay attention during ESAP speaking activities.” Relevant experts reviewed the adapted items for semantic equivalence, clarity, contextual relevance, and consistency with their intended dimensions.

Before the main survey, the questionnaire was piloted with 30 students to assess item comprehension, response options, and completion time. Response distributions and preliminary internal consistency were also examined to guide revisions. Pilot participants were excluded from the main analytic sample. The final questionnaire comprises 15 items. Responses used a five-point agreement scale adapted from the original frequency format, ranging from 1 (strongly disagree) to 5 (strongly agree). Higher mean scores indicate greater engagement in the corresponding dimension.

2.2.2. GenAI Use Measures

Self-reported GenAI use frequency was measured using a single self-report item administered in Week 15. The item focused on voluntary use beyond required course tasks. Participants answered the following question: “Apart from the required AI-assisted learning tasks in class, how many times per week did you use GenAI for ESAP learning during this semester?” For this item, one use was defined as a student-initiated GenAI session devoted to a relatively independent ESAP learning activity. Examples included practicing oral dialogues, simulating academic Q&A, refining spoken expressions, obtaining language feedback, and preparing professional presentations. Multiple interactions within the same continuous session were counted as one use. Participants reported their average weekly use as an integer from 0 to 10. Values from 0 to 9 represented the corresponding number of uses per week. A value of 10 indicated 10 or more uses per week.

System-recorded average weekly online practice time supplemented self-reported use frequency. Data came from a GenAI speaking platform used for practice outside class. Records covered required tasks and additional use within its speaking practice module. The system automatically removed suspected idle or anomalous periods according to its timing rules. Some idle or non-practice time may have remained undetected. The measure was therefore treated as an estimate of platform practice time. Records were exported after Week 15. They were matched to questionnaire and speaking assessment data using anonymous identifiers. Each student’s total time after screening was divided by 15 to obtain average weekly online practice time in minutes per week. System-recorded time reflected platform practice duration, while self-reported frequency reflected GenAI use beyond required course tasks. These complementary indicators were analyzed separately as continuous variables.

2.2.3. Speaking Performance

Students’ ESAP speaking performance was assessed at the beginning and end of the semester. Performance was scored across five dimensions: professional content, organization and coherence, linguistic accuracy and range, fluency and intelligibility, and interactive responsiveness. Each dimension was scored out of 20, giving a total score out of 100. They completed both tests independently, without AI tools, dictionaries, or other assistance. Each assessment contained two parts: a professional academic presentation and a real-time Q&A session. Students were given 7 min to prepare based on provided materials. They then delivered a 5-min academic oral presentation, followed by a 3-min real-time Q&A session. The pretest and posttest used different tasks within the same presentation and Q&A format. Both assessments used identical preparation and response times, instructions, and a common five-dimensional scoring rubric. These shared features supported comparability in task structure and administration. Experts also reviewed the two tasks for comparability in content requirements, topic familiarity, and cognitive demands.

All speaking performances were recorded and presented in a randomized order. Two primary raters and one external adjudicating rater participated in the scoring process. Both primary raters had experience in academic English teaching or speaking assessment and scored independently. Recordings were assigned across classes to reduce potential bias associated with raters’ recognition of students. For recordings that did not require adjudication, the final score was the mean of the two primary raters’ scores. An external rater independently reassessed a recording when the two initial total scores differed by more than 3 points or their scores on any dimension differed by more than 2 points. The external rater was not involved in teaching the course and had experience in English speaking assessment. For adjudicated recordings, the median of the three scores was used. All final scores were retained to one decimal place. Speaking gain was calculated by subtracting the pretest score from the posttest score. A positive value indicated improvement, a value of zero indicated no change, and a negative value indicated a lower posttest score. This measure represented change from the beginning to the end of the course. However, because no non-AI control group was established, this change should not be directly considered as a net effect of AI-assisted instruction.

2.3. Data Analysis

IBM SPSS Statistics 27.0 was used for data screening, descriptive statistics, and internal consistency analysis. CFA, LPA, R3STEP, and manual BCH models were estimated in Mplus 8.3. Python 3.11 was used for ICC calculations, design effects, and Holm adjustment. Holm correction was implemented using custom Python code. BCH effect sizes and their confidence intervals were obtained from the fitted models and checked in Python.

First, preliminary analyses were conducted. The raw data were screened for missing values, potential outliers, and invalid response patterns. Pretest overall speaking scores were summarized by STEM discipline. Welch’s one-way ANOVA was used to compare scores across the four discipline groups. Demographic and background characteristics were summarized descriptively and were not included as additional covariates in the R3STEP or BCH models. The internal consistency of the learning engagement scale was assessed using Cronbach’s α (Hayes & Coutts, 2020). Confirmatory factor analysis (CFA) was used to examine the three-factor structure. Standardized factor loadings, composite reliability (CR), and average variance extracted (AVE) were also reported to evaluate measurement quality. Because the students were nested within classes, ICC(1) values and their 95% confidence intervals were estimated using one-way random effects models to assess class-level clustering in learning engagement, weekly GenAI use frequency, average weekly online duration, and speaking outcomes. Design effects were subsequently calculated using the average class size and the corresponding ICC estimates. The ICC estimates, confidence intervals, and design effects were considered jointly when determining whether the subsequent analyses could be conducted at the individual level (Campbell et al., 2012; M. H. C. Lai & Kwok, 2015).

Second, latent profile analysis (LPA) was used to identify heterogeneous profiles of learning engagement (Masyn, 2013). The three engagement mean scores were treated as continuous observed indicators. Models with one to six classes were estimated in Mplus 8.3 using robust maximum likelihood estimation (MLR). Indicator means were freely estimated across classes. Each indicator’s variance was constrained to be equal across classes, and within-class covariances were fixed at zero. Variances were not constrained to be equal across different indicators. Each model used 1000 initial random starts and 250 final-stage optimizations, with a maximum of 20 initial-stage iterations. For the final candidate model, these settings were increased to 2000 initial random starts, 500 final-stage optimizations, and 50 initial-stage iterations to assess estimation stability. Two alternative specifications were also examined: class-specific variances with zero within-class covariances, and a full variance–covariance matrix constrained to be equal across classes. Under each specification, models with one to six profiles were estimated and compared using the BIC. Model selection was based on the Akaike information criterion (AIC), Bayesian information criterion (BIC), sample-size-adjusted Bayesian information criterion (aBIC), entropy, Lo–Mendell–Rubin adjusted likelihood ratio test (LMR), and bootstrap likelihood ratio test (BLRT). The final profile solution was determined by class sizes, profile distinctiveness, model parsimony, and theoretical interpretability (Nylund-Gibson & Choi, 2018; Spurk et al., 2020).

Third, R3STEP analyses were conducted after the final profile solution was selected. Two separate multinomial logistic regression models were estimated to answer RQ2. Each model included one predictor: average weekly online duration or weekly GenAI use frequency. R3STEP first estimates the LPA without predictors and derives classification-error probabilities from posterior probabilities. It then fits multinomial logistic regressions with these error probabilities fixed, preserving the original profile definitions (Vermunt, 2010). Both indicators were entered as continuous predictors. Odds ratios (ORs) corresponded to a one-minute increase in average weekly online duration or a one-unit increase in the 0–10 frequency code. The reference profile was changed in turn to obtain all unique pairwise comparisons among the retained profiles. ORs and their 95% confidence intervals were reported. Two-sided Wald p values were obtained for the regression coefficients. Holm correction was applied separately across all pairwise profile comparisons for each use indicator. Statistical significance was determined using the adjusted p values (α = 0.05). A sensitivity analysis excluded participants who selected the top-coded frequency category of 10 or more uses per week. The frequency model was re-estimated using a manual three-step approach. The original four-profile solution and classification-error probabilities were retained. Holm correction was applied across the six pairwise comparisons.

Fourth, overall speaking gain was calculated as the posttest score minus the pretest score. Pretest scores, posttest scores, and gain distributions were summarized descriptively. The Bolck–Croon–Hagenaars (BCH) method used weights from the LPA to account for classification error. These weights were used in MLR regressions of posttest outcomes on corresponding pretest scores. Pretest scores were centered at their full-sample means, and residual variances were freely estimated across profiles (Asparouhov & Muthén, 2014). The primary models constrained pretest regression slopes to be equal across profiles. Wald chi-square tests assessed profile differences in the overall speaking score and five dimensions. Holm correction was applied across the six omnibus tests. Pairwise comparisons of the overall score were adjusted as one set. All pairwise comparisons across the five dimensions were adjusted jointly as a separate set. Residual-standardized effect sizes (d) and 95% confidence intervals were reported for overall score comparisons. Equality of pretest slopes across profiles was also tested, with Holm correction across the six outcomes. Sensitivity models allowed these slopes to vary across profiles. In these models, profile comparisons were evaluated at the full sample mean of each corresponding pretest score, using the same Holm correction procedures. Holm correction was used to control the family-wise error rate without requiring independent tests (Holm, 1979). Separate families addressed each use indicator and distinguished overall from dimension-specific speaking performance.

3. Results

3.1. Preliminary Analyses

3.1.1. Measurement Quality Assessment

Learning engagement was assessed using a 15-item questionnaire measuring behavioral engagement (5 items), emotional engagement (6 items), and cognitive engagement (4 items). Confirmatory factor analysis supported the fit of the three-factor model: χ²(87) = 119.477, p = 0.012, CFI = 0.993, TLI = 0.992, RMSEA = 0.027, 90% CI [0.013, 0.039], and SRMR = 0.032. For behavioral engagement, standardized factor loadings ranged from 0.606 to 0.756, with Cronbach’s α = 0.809, composite reliability (CR) = 0.833, and average variance extracted (AVE) = 0.501. For emotional engagement, factor loadings ranged from 0.681 to 0.775, with α = 0.844, CR = 0.869, and AVE = 0.525. For cognitive engagement, factor loadings ranged from 0.762 to 0.796, with α = 0.832, CR = 0.859, and AVE = 0.603. These results supported the internal consistency and convergent validity of the three engagement dimensions.

3.1.2. Speaking Assessment Reliability

Inter-rater agreement was assessed separately for the pretest and posttest using two-way mixed-effects, absolute-agreement intraclass correlation coefficients (ICCs). The analyses included both examiners’ initial total scores for all students, including cases referred for adjudication. Adjudicated scores were not substituted for the initial ratings. The single-measure ICC, ICC(A,1), was 0.824 (95% CI [0.791, 0.853]) at pretest and 0.864 (95% CI [0.839, 0.885]) at posttest. For the mean of the two examiners’ initial ratings, ICC(A,2) was 0.904 (95% CI [0.883, 0.921]) at pretest. The corresponding posttest value was 0.927 (95% CI [0.912, 0.939]). Confidence intervals were obtained using 4000 bootstrap resamples at the student level. Each resample preserved the paired ratings from the two examiners. At both the pretest and posttest, the two primary raters’ initial scores differed by no more than 2 points on any dimension. A third examiner adjudicated recordings when the two initial total scores differed by more than 3 points. This procedure was triggered for 52 pretest recordings (10.4%) and 61 posttest recordings (12.2%). These estimates support the inter-rater reliability of the initial ratings and the mean of the two examiners’ initial scores.

3.1.3. Classroom Clustering

Because the 500 students were nested within 15 classes, ICCs were estimated using one-way random-effects models to assess class-level clustering. The ICC was −0.009 for behavioral engagement, 95% CI [−0.019, 0.022]; 0.001 for emotional engagement, 95% CI [−0.014, 0.045]; and 0.012 for cognitive engagement, 95% CI [−0.008, 0.071]. The ICC was 0.015 for self-reported GenAI use frequency, 95% CI [−0.006, 0.077], and 0.014 for online practice time, 95% CI [−0.007, 0.075]. The ICC was −0.001 for baseline speaking scores, 95% CI [−0.015, 0.040]; 0.003 for end-of-course speaking scores, 95% CI [−0.013, 0.051]; and −0.004 for speaking gains, 95% CI [−0.016, 0.035]. Negative ICC estimates were treated as zero when calculating design effects, which ranged from 1.00 to 1.49. The ICC point estimates and design effects suggested limited class-level clustering in the main study variables. The main analyses were conducted at the individual level. However, some 95% confidence intervals still allowed for non-negligible class-level clustering. The findings should therefore be interpreted with caution. Mean pretest speaking scores across the four STEM disciplines ranged from 82.142 to 82.650. Welch’s one-way ANOVA showed no statistically significant differences across disciplines, F(3, 249.613) = 0.753, p = 0.522. These comparisons concern only baseline ESAP speaking performance. General English proficiency and prior English reading experience were not assessed.

3.2. Latent Profiles of Learning Engagement

Latent profile models with one to six classes were estimated using behavioral, emotional, and cognitive engagement as indicators. In the primary specification, indicator variances were constrained to equality across profiles, and within-profile covariances were fixed at zero. As the number of profiles increased, the AIC and aBIC generally decreased. The BIC reached its lowest value for the four-profile model (BIC = 3423.219) and increased to 3432.382 and 3441.895 for the five- and six-profile models. The adjusted LMR test favored the four-profile model over the three-profile model (p < 0.001). The corresponding tests for five and six profiles were nonsignificant. Although the BLRT remained significant for the five- and six-profile models, the five-profile model contained a small class representing 4.0% of the sample, and the six-profile model exhibited two small classes representing only 2.4% and 2.0% of the sample. Consequently, the four-profile solution was retained. The model fit results are presented in Table 2.

Table 2. Model Fit Indices for Latent Profile Analysis.

Model	AIC	BIC	aBIC	Entropy	LMR	BLRT	Most Likely Class (%)
1	3741.833	3767.120	3748.076	—	—	—	100
2	3468.583	3510.729	3478.989	0.879	<0.001	<0.001	26.60/73.40
3	3414.970	3473.975	3429.538	0.768	<0.001	<0.001	23.60/24.80/51.60
4	3347.356	3423.219	3366.086	0.782	<0.001	<0.001	22.40/38.20/24.00/15.40
5	3339.660	3432.382	3362.552	0.788	0.529	0.013	23.20/16.20/4.00/20.20/36.40
6	3332.316	3441.895	3359.370	0.809	0.128	<0.001	23.20/20.60/15.00/36.80/2.40/2.00

Note. AIC = Akaike information criterion; BIC = Bayesian information criterion; aBIC = sample-size-adjusted Bayesian information criterion; BLRT = bootstrapped likelihood ratio test; LMR = Lo–Mendell–Rubin adjusted likelihood ratio test. Lower AIC, BIC, and aBIC values indicate better model fit, whereas higher entropy values indicate greater certainty in profile assignment. Class proportions represent the percentages of participants assigned to each profile based on their most likely class membership.

The model-estimated proportions of the four profiles were 22.12%, 37.55%, 23.67%, and 16.65%. Classification assigned 112 students (22.4%), 191 students (38.2%), 120 students (24.0%), and 77 students (15.4%). Entropy was 0.782, and the average posterior probabilities for profiles 1 to 4 were 0.932, 0.877, 0.845, and 0.883. These indices indicated acceptable classification quality. The best log-likelihood of the four-profile model was −1655.678 and was replicated across multiple random starts. Increasing the initial starts to 2000 and final-stage optimizations to 500 reproduced the same value, supporting the numerical stability of the solution. Alternative variance specifications were also examined. Under both the class-specific diagonal-variance specification and the common full variance-covariance specification, the BIC continued to favor four profiles. The four-profile class-specific variance model had a BIC of 3439.767, and the four-profile model with a common full variance-covariance matrix had a BIC of 3436.745. Both values exceeded the BIC of 3423.219 for the primary four-profile model.

In Figure 2, the four profiles differed across the three engagement dimensions. The profile labels describe relative engagement configurations within the present sample and should not be interpreted as learner types or externally validated cutoffs.

Figure 2. Estimated Mean Profiles of Learning Engagement.

Profile 1 included 112 students and represented 22.4% of the sample. This profile showed low behavioral engagement (M = 2.652) and cognitive engagement (M = 2.535). Emotional engagement was close to the midpoint of the scale (M = 3.152). Because the overall level of engagement was relatively low, this profile was labeled Low Engagement. Profile 2 was the largest group and represented 38.2% of the sample. Students reported high mean scores across behavioral (M = 4.285), emotional (M = 4.003), and cognitive engagement (M = 4.238). This profile was named High Engagement. Profile 3 contained 120 students (24.0% of the sample). Students in this profile showed high behavioral engagement (M = 3.829) and cognitive engagement (M = 3.766), but lower emotional engagement (M = 2.772). Therefore, Profile 3 was labeled Low Emotional Engagement. The smallest group was the fourth profile, accounting for 15.4% of the sample. Students in this profile reported high behavioral engagement (M = 4.168) and emotional engagement (M = 4.446). However, their cognitive engagement was relatively low (M = 2.694). This profile was named Low Cognitive Engagement.

3.3. Weekly GenAI Use and Engagement Profile Membership

R3STEP analyses were used to examine the associations between GenAI use indicators and engagement profile membership. Average weekly online duration and weekly GenAI use frequency were entered into separate single-predictor models. Each model provided 6 unique pairwise comparisons among the four profiles. Holm correction was applied separately to the six comparisons for each indicator. Descriptive means were based on most likely profile assignments. Average weekly online duration was relatively high in P2 and P4 (M = 25.185 and 25.127 min/week). It was lower in P1 (M = 19.781 min/week). The mean frequency code was highest in P4 (M = 4.844) and lowest in P3 (M = 3.767). These means summarize the two use indicators within each profile. R3STEP results with P1 as the reference profile are presented in Table 3.

Table 3. R3STEP Comparisons of GenAI Use Indicators with Engagement Profile Membership.

Use Indicator	P2 vs. P1 OR (95% CI)	Ad. p	P3 vs. P1 OR (95% CI)	Ad. p	P4 vs. P1 OR (95% CI)	Ad. p
Weekly online duration	1.479 (1.259–1.737)	<0.001	1.122 (1.028–1.224)	0.029	1.432 (1.199–1.709)	<0.001
Weekly GenAI use frequency	1.141 (1.003–1.298)	0.178	0.883 (0.758–1.029)	0.223	1.188 (1.004–1.406)	0.178

Note. Each row represents a separate single-predictor R3STEP model. P1 = Low Engagement Profile; P2 = High Engagement Profile; P3 = Low Emotional Engagement Profile; P4 = Low Cognitive Engagement Profile. ORs correspond to a one-minute increase in average weekly online duration or a one-unit increase in the frequency code. Holm correction was applied separately across the six pairwise comparisons for each indicator. For ease of interpretation, a five-minute increase in average weekly online duration corresponded to ORs of approximately 7.077 (95% CI [3.163, 15.812]) for P2, 1.778 (95% CI [1.148, 2.747]) for P3, and 6.022 (95% CI [2.478, 14.578]) for P4, all relative to P1. The approximate five-minute ORs and confidence limits were calculated by raising the reported per-minute values to the fifth power.

In the online duration model, the ORs per additional minute/week were 1.479 for P2, 1.122 for P3, and 1.432 for P4, all relative to P1. All three comparisons remained significant after Holm correction. Online duration was negatively associated with membership in P3 relative to P2 (OR = 0.759, 95% CI [0.634, 0.908], adjusted p = 0.010). It was positively associated with membership in P4 relative to P3 (OR = 1.276, 95% CI [1.059, 1.538], adjusted p = 0.029). The comparison between P2 and P4 was nonsignificant (adjusted p = 0.649).

In the use frequency model, two of the six comparisons remained significant after Holm correction. These were the comparisons of P3 with P2 and P4 with P3. Weekly use frequency was negatively associated with membership in P3 relative to P2 (OR = 0.774, 95% CI [0.673, 0.890], adjusted p = 0.002). The comparison of P4 with P3 showed the opposite direction, with an OR of 1.346 (95% CI [1.138, 1.591], adjusted p = 0.003). These ORs corresponded to a one-unit increase in the frequency code. The other four comparisons were nonsignificant. These included all three comparisons with P1 and the comparison between P2 and P4 (adjusted p = 0.625).

Ten participants (2.0%) selected the top-coded category of 10 or more uses per week. After excluding these participants, the sensitivity analysis (n = 490) retained the same pattern of significant comparisons. The OR was 0.774 for P3 relative to P2 (95% CI [0.669, 0.895], adjusted p = 0.003) and 1.329 for P4 relative to P3 (95% CI [1.113, 1.586], adjusted p = 0.008). The other four comparisons remained nonsignificant.

3.4. Speaking Performance Across Engagement Profiles

Descriptive statistics showed that the four profiles had similar speaking scores at the beginning of the course. The mean pretest scores were 82.413 for the Low Engagement Profile, 82.562 for the High Engagement Profile, 82.487 for the Low Emotional Engagement Profile, and 82.486 for the Low Cognitive Engagement Profile. Their corresponding posttest means were 85.085, 86.636, 85.222, and 85.147. All four profiles therefore showed positive mean gains over the course. The High Engagement Profile showed the largest descriptive gain, whereas the other three profiles showed similar average gains. Individual gains nevertheless varied within each profile and included negative values. Figure 3 shows unadjusted speaking gains (posttest minus pretest total scores), grouped by most likely profile assignment. Violin widths indicate density, boxes span the 25th–75th percentiles, and horizontal lines mark medians. Small dots represent individual gains; larger circles and M labels indicate profile means. These distributions are descriptive, are not BCH estimates, and do not account for classification uncertainty.

Figure 3. Distributions of individual speaking gain scores across the four engagement profiles.

The BCH method was used to compare posttest performance across engagement profiles. The analyses controlled for the corresponding pretest scores and accounted for classification uncertainty. All six omnibus tests were significant after Holm correction (Table 4). For overall speaking performance, the omnibus difference was significant, Wald χ²(3) = 49.523, adjusted p < 0.001. The High Engagement Profile had the highest adjusted posttest mean (M = 86.795, SE = 0.171). After Holm correction, this profile scored significantly higher than the other three profiles (all adjusted p < 0.001). Differences among the remaining three profiles were nonsignificant. Relative to the Low Engagement, Low Emotional Engagement, and Low Cognitive Engagement Profiles, the residual-standardized effect sizes were d = 0.785 (95% CI [0.489, 1.081]), d = 0.849 (95% CI [0.509, 1.189]), and d = 0.826 (95% CI [0.452, 1.200]).

Table 4. Comparisons of Baseline-Adjusted Speaking Performance Across the Engagement Profiles.

Variable	P1 (n = 112)	P2 (n = 191)	P3 (n = 120)	P4 (n = 77)	Wald χ2(3)	Ad. p	Comparisons
OSP	85.165 (0.236) a	86.795 (0.171) b	85.052 (0.253) a	84.981 (0.329) a	49.523	<0.001	P2 > P1, P3, P4
PC	17.854 (0.027) a	17.910 (0.021) a	17.809 (0.026) a	17.801 (0.031) a	10.378	0.016	None
OC	17.054 (0.022) a	17.177 (0.016) b	17.056 (0.021) a	17.039 (0.028) a	32.113	<0.001	P2 > P1, P3, P4
LA	17.555 (0.051) a	18.048 (0.041) b	17.671 (0.059) a	17.411 (0.074) a	81.387	<0.001	P2 > P1, P3, P4
FI	16.959 (0.072) a	17.375 (0.056) b	16.951 (0.081) a	17.117 (0.095) ab	27.521	<0.001	P2 > P1, P3
IR	15.741 (0.101) a	16.274 (0.070) b	15.564 (0.103) a	15.640 (0.147) a	37.957	<0.001	P2 > P1, P3, P4

Note. Values are BCH-estimated posttest means adjusted for the corresponding pretest scores, with standard errors in parentheses. OSP = overall speaking performance; PC = professional content; OC = organization and coherence; LA = linguistic accuracy and range; FI = fluency and intelligibility; IR = interactive responsiveness. P1 = Low Engagement Profile; P2 = High Engagement Profile; P3 = Low Emotional Engagement Profile; P4 = Low Cognitive Engagement Profile. Sample sizes reflect most likely class assignments. Omnibus p values were Holm-adjusted across the six outcomes. Pairwise comparisons were Holm-adjusted separately across the six overall-score comparisons and the 30 dimension-specific comparisons. Within each row, means sharing a superscript letter did not differ significantly after correction.

The High Engagement Profile scored significantly higher than the other three profiles in organization and coherence, linguistic accuracy and range, and interactive responsiveness. For fluency and intelligibility, its adjusted score was significantly higher than those of the Low Engagement and Low Emotional Engagement Profiles. However, its difference from the Low Cognitive Engagement Profile was nonsignificant. The omnibus test for professional content was significant (adjusted p = 0.016), but no pairwise comparisons remained significant after correction. Differences among the remaining three profiles were nonsignificant across all five dimensions. All six tests of slope equality were nonsignificant after Holm correction. However, the unadjusted test for organization and coherence suggested possible slope differences (Wald χ²(3) = 10.418, unadjusted p = 0.015, adjusted p = 0.092). A sensitivity analysis therefore allowed pretest slopes to vary across profiles. Profile comparisons were evaluated at the full-sample mean of each corresponding pretest score. With the same Holm corrections, the pattern of significant pairwise differences remained unchanged for all six outcomes. Thus, at these pretest reference values, the main profile differences did not depend on the common-slope constraint.

4. Discussion

4.1. Learning Engagement Profiles in AI-Assisted ESAP Speaking

The findings for RQ1 identified four distinct engagement profiles: Low Engagement, High Engagement, Low Emotional Engagement, and Low Cognitive Engagement. These profiles reveal different engagement configurations among STEM master’s students within a shared course context. The High and Low Engagement Profiles mainly reflected differences in overall levels. The other two profiles showed uneven engagement across dimensions. Based on most likely class assignments, these two profiles accounted for 39.4% of the sample. These findings suggest that a total engagement score may obscure differences in how the three dimensions combine within this sample. The identified configurations are consistent with earlier profile research in language learning (Hoi & Hang, 2022; J. Li et al., 2022). Xu and Li identified low, moderate, and high engagement profiles, showing variation in overall engagement (Xu & Li, 2024). The present study also found that stronger behavioral engagement could coexist with different emotional and cognitive engagement levels. Similarly, Xing and Teng emphasized emotional–behavioral or cognitive engagement (Xing & Teng, 2026). However, the present Low Emotional Engagement Profile combined relatively strong behavioral and cognitive engagement. It should not be equated with their cognitively oriented group. These studies support engagement heterogeneity, but do not establish identical learner categories across populations and courses.

Students in the High Engagement Profile reported relatively high behavioral, emotional, and cognitive engagement. This configuration may reflect a good fit between their learning needs, course tasks, and available support. In contrast, the Low Engagement Profile showed low behavioral and cognitive engagement. Emotional engagement in this profile was close to the scale midpoint. From the educational interface perspective, this configuration may reflect a weaker alignment between some students’ learning needs and course tasks or support (Kahu & Nelson, 2018).

Students in the Low Emotional Engagement Profile reported lower levels of emotional engagement with relatively higher behavioral and cognitive engagement. They might value disciplinary communication or course goals and continue to participate despite experiencing less interest or enjoyment. The academic and professional purposes of ESAP speaking instruction provide a contextual basis for this interpretation (Gleason & Leandro, 2023). However, engagement scores alone cannot establish whether professional needs or course requirements motivated these students. The Low Cognitive Engagement Profile was characterized by relatively higher behavioral and emotional engagement, and its emotional engagement mean was the highest across the four profiles. However, these students reported comparatively lower cognitive engagement. Related EFL research has found uneven changes in self-regulated learning processes under AI support. Task strategies and time management improved, whereas changes in self-evaluation were more limited (Gu & Liu, 2025). A systematic review also found that educational chatbots provided uneven support across self-regulated learning processes (Guan et al., 2025).

From Kahu and Nelson’s framework, a shared course offers similar learning opportunities, but students may respond differently to its tasks and resources (Kahu & Nelson, 2018). The present study applies this perspective to an AI-assisted ESAP speaking course for STEM master’s students. It provides context-specific evidence on how participation, emotional experiences, and cognitive effort combine within individual learners.

4.2. Weekly AI Use in AI-Assisted ESAP Speaking

A quasi-experimental study using Duolingo reported improvements in engagement, enjoyment, and motivation among Chinese EFL learners (Yuan & Liu, 2025). Xu and Li also found that the relationship between AI affordances and engagement varied across engagement groups (Xu & Li, 2024). In the present study, online duration and use frequency showed different patterns of association with profile membership. Both indicators distinguished P3 from P2 and P4. However, only online duration showed significant associations in all three comparisons with P1 as the reference profile. The indicators also differed in task coverage. Online duration included required and supplementary platform practice, whereas reported frequency concerned voluntary GenAI use beyond required tasks.

The duration findings are consistent with P1’s relatively low behavioral engagement. More time on the platform may provide additional opportunities to practice speaking, encounter model responses, and receive feedback. These opportunities may help explain the contrast between P1 and the other profiles. However, P4 also showed relatively long online duration alongside comparatively low cognitive engagement. Greater platform use may therefore coexist with uneven engagement across dimensions. AI-assisted speaking activities can involve formulating responses, listening to AI-generated examples, reading feedback, and using generated content. These activities all take time online but place different cognitive demands on learners.

The significant frequency comparisons involved P3, which showed lower emotional engagement than P2 and P4. Higher use frequency also corresponded to higher odds of membership in P2 or P4 relative to P3. The same significant comparisons remained after the top-coded responses were excluded. This suggests that these observations were not solely responsible for the frequency findings. The analysis does not recover actual frequencies within the 10-or-more category or remove recall error. Use frequency and engagement were reported in the same Week 15 questionnaire. Shared response tendencies may therefore contribute to the observed association. GenAI’s immediate responses make practice readily available. However, this accessibility does not ensure that students continue to find practice interesting or valuable. Kahu and Nelson’s educational interface framework emphasizes the role of psychological experiences in connecting learning resources and student engagement (Kahu & Nelson, 2018). From this perspective, P3’s use pattern may reflect an emotional aspect of voluntary participation. Task participation and cognitive effort may coexist with less positive practice experiences.

The contrast between P2 and P4 raises a question about how students and GenAI share cognitive activities. Neither online duration nor use frequency showed a significant association with membership in P2 relative to P4. However, P4 showed relatively low cognitive engagement. These nonsignificant results do not establish that P2 and P4 used AI to the same extent or in the same way. GenAI can provide practice materials and feedback. It can also help organize ideas, generate language, and revise responses. Students may use AI suggestions to compare expressions, check disciplinary content, and revise their own answers. They may also draw more heavily on ready-made content to complete tasks (Darvishi et al., 2024; Kalimira & Jazayeri, 2026). These possible uses could involve different levels of independent judgment and reflection. Godwin-Jones’s discussion of shared human–AI agency provides a theoretical basis for this interpretation (Godwin-Jones, 2024). Differences in this collaboration may help explain why high behavioral and emotional engagement can coexist with relatively low cognitive engagement. These accounts remain exploratory theoretical interpretations that require future empirical testing. The specific AI-use behaviors described above were not directly measured. Cognitive offloading and self-regulation were not assessed as separate constructs. The present data therefore cannot establish whether these processes explain the profile differences.

Related speaking research has also documented links between engagement configurations and interaction processes. Chen found that AI interaction sequences differed across engagement profiles in repeated attempts and behavioral continuity (Y. Chen et al., 2026b). This finding is consistent with the interpretation offered here. Engagement differences may emerge in how interactions unfold, how students respond to feedback, and how they develop their spoken responses. The frequency and duration measures used here describe the amount of use but do not reveal these interaction processes.

4.3. Speaking Performance in AI-Assisted ESAP Speaking

All four profiles showed positive mean changes in speaking scores during the course. Previous studies have also reported speaking improvements in AI-supported learning (Fathi et al., 2024; Ma et al., 2025). The present findings add a profile perspective by examining performance differences within a shared course. After baseline adjustment, the High Engagement Profile scored significantly higher than the other three profiles. Thus, students with relatively high behavioral, emotional, and cognitive engagement also showed stronger assessed speaking performance. The course provided opportunities for speaking practice, interaction, and feedback. However, students may have responded differently to these opportunities. Higher engagement across all three dimensions may help learners remain involved in practice while considering feedback and revising their expression. This is particularly relevant to ESAP speaking, which requires students to organize disciplinary ideas, use language accurately, and respond to questions. The uneven profiles further clarify the findings. Both the Low Emotional Engagement and Low Cognitive Engagement Profiles had lower adjusted overall scores than the High Engagement Profile. Notably, the Low Cognitive Engagement Profile reported the highest emotional engagement among the four profiles. Its lower overall speaking score suggests that positive emotional experiences alone provide an incomplete picture of assessed performance. The results therefore support considering the full engagement configuration.

The detected differences also varied across speaking components. The High Engagement Profile scored higher than all three other profiles in organization and coherence, linguistic accuracy and range, and interactive responsiveness. For fluency and intelligibility, however, its difference from the Low Cognitive Engagement Profile was nonsignificant. Chen similarly found differences in fluency and coherence gains across learner groups during GenAI-supported speaking practice (Y. Chen et al., 2026a). Their groups were based on observable interaction behaviors. The present study provides complementary evidence by relating psychological engagement configurations to assessed ESAP performance after baseline adjustment. Professional content requires a more cautious interpretation. Its significant omnibus test indicated overall differences, but the corrected pairwise comparisons did not identify which profiles differed. The High Engagement Profile therefore did not show a statistically established advantage in every speaking component. Nonsignificant comparisons also do not demonstrate equivalent performance. These findings highlight the importance of examining component scores alongside the overall score when interpreting the educational significance of engagement profiles.

The BCH sensitivity analyses allowed pretest slopes to differ across profiles. At the sample-mean pretest values, they retained the same pattern of significant comparisons. Nevertheless, engagement was measured late in the course, so its temporal relationship with performance changes remains unclear. Learning progress may also have influenced students’ reported engagement. The findings therefore indicate associations between engagement configurations and baseline-adjusted performance. They do not establish that engagement or AI-assisted instruction caused the observed improvements.

4.4. Contributions and Implications

This study connects multidimensional engagement configurations, GenAI use, and speaking performance. It provides person-centered evidence of learner differences among STEM master’s students in a shared AI-assisted ESAP course. The four profiles reflect differences in overall engagement. They also show that high behavioral participation can coexist with relatively low emotional or cognitive engagement. These configurations add detail to descriptions based on overall engagement scores. Online duration and use frequency showed different associations with profile membership. The two indicators provided different information about students’ engagement configurations. The study also examined baseline-adjusted speaking performance across profiles, both overall and in specific skill dimensions. P2 scored higher overall and in organization and coherence, linguistic accuracy and range, and interactive responsiveness. These advantages clarify the educational relevance of engagement differences. Together, these findings offer a more detailed account of learner heterogeneity in AI-assisted academic speaking.

In practice, support priorities can be informed by students’ reported difficulties, GenAI use indicators, and classroom performance. Students showing P1’s low behavioral and cognitive engagement may benefit from guidance on planning practice and periodic feedback. Such support could help clarify task requirements and goals for improvement. P3 showed relatively strong behavioral and cognitive engagement, so support could focus more on practice experiences. Options include topic choices related to students’ research interests or academic communication needs. Brief discussions could also help teachers understand students’ interest in tasks and the difficulties they encounter. For students showing P4’s characteristics, feedback could address how they understand AI suggestions, judge their suitability, and explain revisions to their own responses. High online activity should not be a reason to reduce support. P2’s performance in organization and coherence, linguistic accuracy and range, and interactive responsiveness also suggests specific priorities for speaking feedback. Teachers could identify problems in the connections between ideas, the accuracy of key expressions, and responses to follow-up questions. This would clarify the focus of subsequent practice. Support should remain responsive to students’ actual needs, with profile membership serving as one source of information. These suggestions have not been experimentally evaluated. Their effectiveness in supporting students with different engagement profiles requires further investigation.

4.5. Limitations and Future Directions

This study has several limitations. First, learning engagement was measured only at the end of the course. The study could not capture changes in engagement profiles over time or establish causal relationships among engagement, AI use, and speaking performance. Moreover, no non-AI control group was included. Speaking gains therefore cannot be interpreted as the net effect of AI-assisted instruction. Future studies could use longitudinal measurements and controlled designs to examine these relationships. Second, this study used convenience sampling. Participants were STEM master’s students in China, which limits the generalizability of the findings. Course requirements and assessment criteria were standardized, but differences between classes and instructors may still have influenced the results. Although the ICC point estimates and design effects were small, some confidence intervals still allowed for non-negligible class-level clustering. The single-level analyses did not explicitly account for this potential dependence among students within the 15 classes. Demographic and background characteristics were described but were not included as additional covariates. Their potential contribution to the observed associations and profile differences cannot be ruled out. Current employment status, workplace AI use, and English use at work were not assessed. These unmeasured factors may also have influenced the findings. Future research across institutions and instructional settings should assess these background factors prospectively and examine the associations after appropriate adjustment. Third, learning engagement and GenAI use frequency were measured through self-report questionnaires. These measures may be affected by recall bias and common method bias. The top-coded frequency response also limited differentiation among students reporting 10 or more uses per week. System-recorded usage duration provided additional information. However, it primarily reflected the amount of use and could not fully capture usage purposes, feedback processing, or interaction quality. Future studies could combine dialogue logs and interviews to examine AI use processes more comprehensively. Fourth, the pretest and posttest tasks shared the same structure and scoring criteria. However, formal evidence of task equivalence was not provided. No delayed assessment was conducted to examine the retention and transfer of learning. The teaching recommendations for different profiles were also not tested through interventions. They should therefore be viewed as directions for practice that require further empirical validation.

5. Conclusions

This study used LPA to examine self-reported engagement among 500 STEM master’s students in an AI-assisted ESAP speaking course. Engagement was measured in Week 15. Four profiles were identified: Low Engagement Profile, High Engagement Profile, Low Emotional Engagement Profile, and Low Cognitive Engagement Profile. In separate models, self-reported GenAI use frequency and system-recorded online duration each distinguished some profiles after Holm correction. Longer platform use was associated with higher odds of membership in the other three profiles relative to the Low Engagement Profile. More frequent supplementary GenAI use was associated with the High Engagement and Low Cognitive Engagement Profiles relative to the Low Emotional Engagement Profile. After baseline adjustment, the High Engagement Profile had higher overall posttest speaking scores than the other profiles. Its advantages were most evident in organization and coherence, linguistic accuracy and range, and interactive responsiveness. These findings suggest that AI-assisted ESAP instruction could provide targeted support informed by students’ technology use and their behavioral, emotional, and cognitive engagement. Further research is needed to determine whether such support improves engagement and speaking performance.