Evaluating the Effectiveness of AI Chatbots in University Admissions: Exploring Student Assistance and Satisfaction ^†

Abstract

Universities increasingly rely on digital self-service channels to manage high volumes of time-sensitive admissions enquiries. AI enabled chatbots represent a prominent solution because they can provide round-the-clock responses, standardize guidance, and potentially reduce uncertainty for applicants. Yet evidence on whether such chatbots meaningfully assist students and improve their satisfaction with admissions support remains limited in many developing higher education contexts. This quantitative study evaluates the perceived effectiveness of AI chatbots used for university admissions in Pakistan, with a focus on student assistance and satisfaction as key outcomes. Using a cross-sectional survey design, data were collected from students who had recently engaged with university admissions information services (e.g., website chat widgets, messaging-based virtual assistants, and admissions enquiry portals) across private universities in Pakistan. Admissions chatbot effectiveness was measured through established information systems and service quality constructs system quality (ease of use, responsiveness, accessibility), information quality (accuracy, clarity, completeness), and service quality and trust cues (assurance, privacy confidence, and appropriateness of conversational support). Student assistance captured the extent to which chatbot interactions helped participants complete admissions related tasks and navigate application procedures. Student satisfaction reflected overall evaluation of the admissions support experience. The results indicate a positive association between perceived chatbot quality and perceived student assistance, and a further positive association between student assistance and student satisfaction with admissions support. The overall pattern suggests that student assistance functions as a key mechanism through which chatbot effectiveness translates into satisfaction. At the same time, respondents highlighted limitations in resolving complex or exception based queries, emphasizing the importance of transparent escalation to human admissions staff. The study contributes context specific evidence from Pakistan and offers an empirically grounded framework that university administrators can use to evaluate and improve admissions chatbots. Practical implications emphasize maintaining accurate knowledge bases, designing clear handoff pathways, and implementing governance practices that strengthen students’ confidence in information reliability and data privacy.

1. Introduction

Objectives: The study pursues four objectives:

(i)

To assess students’ perceptions of AI chatbot effectiveness in universities in Pakistan.

(ii)

To quantify the relationship between chatbot quality dimensions and perceived student assistance.

(iii)

To examine the relationship between AI chatbot assistance and satisfaction with admissions support.

(iv)

To test whether students are satisfied with the empathetic response of AI-based chatbots.

Theoretical background: We focus on the Information Systems Success Model, which positions system quality, information quality, and service quality as antecedents of use and user satisfaction, ultimately yielding net benefits [1,2]. This lens is complemented by the Technology Acceptance Model’s (TAM) emphasis on perceived usefulness and ease of use as mechanisms linking technology characteristics to user attitudes and outcomes [3,4]. Consistent with meta-review evidence that highlights variation in how information success dimensions are operationalized across contexts [5], the present study tailors these constructs to admissions interactions where the primary value proposition is task guidance and reduced uncertainty.

Literature review: Prior studies in higher education show generally positive student perceptions of chatbots as convenient assistants, while also highlighting concerns about the accuracy and reliability of responses [6]. In learning-oriented contexts, students’ satisfaction can vary by educational level and usage patterns, suggesting that user characteristics may shape perceived value even when a chatbot is available [7]. Admissions and enrolment contexts, however, add distinct service quality expectations. Responses must be consistent with the policies, up-to-date, and easily auditable, and a clear escalation pathway is essential for exceptions or ambiguous cases. These considerations motivate the focus of the current study on assistance and satisfaction as measurable outcomes of chatbot effectiveness in Pakistan’s university admissions landscape. Parallel work on chatbot satisfaction highlights that relevance, completeness, and assurance are central drivers of satisfaction and usage intention, alongside privacy concerns [8].

2. Proposed Methodology

This study employed a quantitative, cross-sectional survey design. The setting comprised private universities in Pakistan that provide digital admissions enquiry support through web-based or messaging-based chatbot interfaces.

The target population included prospective and current students who had interacted with admissions related digital support channels during a recent admissions cycle. Convenient sampling, a non-probability sampling approach was used to recruit respondents across Pakistan. Participant demographics (e.g., age, gender, educational level, and experience with chatbots) were recorded to describe the sample and to support robustness checks.

The questionnaire used multi-item Likert scales to measure key constructs: (a) system quality (ease of navigation, response speed, availability), (b) information quality (accuracy, clarity, completeness), (c) service quality and trust cues (assurance, perceived appropriateness of responses, privacy confidence), (d) perceived student assistance (effectiveness in guiding admissions tasks and reducing uncertainty), and (e) student satisfaction with admissions support. Scale items were adapted from established information system success and technology acceptance literature and contextualized to tasks related to admissions.

Data will be screened for completeness and outliers. Internal consistency and construct reliability will be assessed (e.g., Cronbach’s alpha and/or composite reliability), and validity will be examined using standard psychometric criteria. Hypotheses will be tested using correlation and regression/structural modeling appropriate to the study’s design. These tests will be performed through Structural Equation Modeling (SEM) Smart PLS [9].

3. Expected Results

The analysis process is yet to be carried out, but data collection to date shows that respondents were mostly from three universities, which are NUCES-FAST, University of Central Punjab and the University of Sialkot. The respondents reported recent use of digital admissions enquiry channels. The final dataset will be finalized after full conclusion of the ongoing survey. The distribution of responses would be across undergraduate and postgraduate applicants/students.

Validity checks seem to support measurement adequacy based on the criteria applied in the study.

Inferential results seem to have a positive relationship between admissions chatbot quality dimensions and perceived student assistance. Student assistance apparently exhibits a positive association with student satisfaction with admissions support.

Overall, the findings indicate that admissions chatbots are valued primarily as task-oriented support systems; students report higher satisfaction when the chatbot provides accurate policy-aligned information, responds promptly, and communicates in a clear and reassuring manner. Students may report limitations related to complex cases when it comes to non-routinely asked questions.

The final accurate findings will be included once actual analysis is performed.

4. Conclusions (Implications, Originality Value, Contribution, etc.)

The universities in Pakistan are not much into using chatbots for query handling, which may hinder the footfall on campus sites during the admission process. The study provides empirical evidence that AI chatbots can enhance university admissions support in Pakistan when they deliver high-quality system performance and policy-consistent information in a reassuring service style. Student assistance emerged as a pivotal mechanism: applicants are more satisfied not simply because a chatbot exists, but because it reduces uncertainty and helps them complete admissions tasks. The services are available around the clock.

For practice, the results underscore the need to treat admissions chatbots as continuously maintained information systems rather than static “FAQ bots.” Universities should invest in (i) content governance and rapid policy updates (deadlines, merit criteria, fee revisions), (ii) design for clarity (step-by-step guidance, checklists, and links to authoritative pages), (iii) transparent escalation pathways to human staff for exception-based cases, and (iv) privacy-forward communication that explains what data are collected and how they are used.

By examining universities in Pakistan and centering student assistance and satisfaction as outcomes, the study extends the higher education chatbot literature to an underrepresented admissions service context and offers a practical evaluation framework grounded in information system success and technology acceptance perspectives. The framework can support benchmarking across institutions and guide evidence-based improvements to admissions service delivery.

The study remained limited to the private sector universities in Punjab and Islamabad. Further studies can be conducted in the universities of other provinces and the enhancements can be applied for further efficiency of the system.