A Systematic Approach to Evaluate the Use of Chatbots in Educational Contexts: Learning Gains, Engagements and Perceptions

Qiu, Wei; Su, Chit Lin; Jamil, Nurabidah Binti; Thway, Maung; Ng, Samuel Soo Hwee; Zhang, Lei; Lim, Fun Siong; Lai, Joel Weijia

doi:10.3390/computers14070270

Open AccessEditor’s ChoiceArticle

A Systematic Approach to Evaluate the Use of Chatbots in Educational Contexts: Learning Gains, Engagements and Perceptions

by

Wei Qiu

,

Chit Lin Su

,

Nurabidah Binti Jamil

,

Maung Thway

,

Samuel Soo Hwee Ng

,

Lei Zhang

,

Fun Siong Lim

and

Joel Weijia Lai

^*

Applications of Teaching and Learning Analytics for Students, Institute for Pedagogical Innovation, Research and Excellence, Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798, Singapore

^*

Author to whom correspondence should be addressed.

Computers 2025, 14(7), 270; https://doi.org/10.3390/computers14070270

Submission received: 2 June 2025 / Revised: 24 June 2025 / Accepted: 4 July 2025 / Published: 9 July 2025

(This article belongs to the Special Issue Smart Learning Environments)

Download

Browse Figures

Versions Notes

Abstract

As generative artificial intelligence (GenAI) chatbots gain traction in educational settings, a growing number of studies explore their potential for personalized, scalable learning. However, methodological fragmentation has limited the comparability and generalizability of findings across the field. This study proposes a unified, learning analytics–driven framework for evaluating the impact of GenAI chatbots on student learning. Grounded in the collection, analysis, and interpretation of diverse learner data, the framework integrates assessment outcomes, conversational interactions, engagement metrics, and student feedback. We demonstrate its application through a multi-week, quasi-experimental study using a Socratic-style chatbot designed with pedagogical intent. Using clustering techniques and statistical analysis, we identified patterns in student–chatbot interaction and linked them to changes in learning outcomes. This framework provides researchers and educators with a replicable structure for evaluating GenAI interventions and advancing coherence in learning analytics–based educational research.

Keywords: generative artificial intelligence; application of retrieval-augmented generation; student–chatbot interaction; learning analytics; artificial intelligence in education

Share and Cite

MDPI and ACS Style

Qiu, W.; Su, C.L.; Jamil, N.B.; Thway, M.; Ng, S.S.H.; Zhang, L.; Lim, F.S.; Lai, J.W. A Systematic Approach to Evaluate the Use of Chatbots in Educational Contexts: Learning Gains, Engagements and Perceptions. Computers 2025, 14, 270. https://doi.org/10.3390/computers14070270

AMA Style

Qiu W, Su CL, Jamil NB, Thway M, Ng SSH, Zhang L, Lim FS, Lai JW. A Systematic Approach to Evaluate the Use of Chatbots in Educational Contexts: Learning Gains, Engagements and Perceptions. Computers. 2025; 14(7):270. https://doi.org/10.3390/computers14070270

Chicago/Turabian Style

Qiu, Wei, Chit Lin Su, Nurabidah Binti Jamil, Maung Thway, Samuel Soo Hwee Ng, Lei Zhang, Fun Siong Lim, and Joel Weijia Lai. 2025. "A Systematic Approach to Evaluate the Use of Chatbots in Educational Contexts: Learning Gains, Engagements and Perceptions" Computers 14, no. 7: 270. https://doi.org/10.3390/computers14070270

APA Style

Qiu, W., Su, C. L., Jamil, N. B., Thway, M., Ng, S. S. H., Zhang, L., Lim, F. S., & Lai, J. W. (2025). A Systematic Approach to Evaluate the Use of Chatbots in Educational Contexts: Learning Gains, Engagements and Perceptions. Computers, 14(7), 270. https://doi.org/10.3390/computers14070270

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Menu

A Systematic Approach to Evaluate the Use of Chatbots in Educational Contexts: Learning Gains, Engagements and Perceptions

Abstract

Share and Cite

Article Metrics

Article Access Statistics

Further Information

Guidelines

MDPI Initiatives

Follow MDPI