Knowledge-Driven Human-in-the-Loop Decision Support for Student Services Using Active Learning and Large Language Models

Abstract

This study presents an AI-based, human-in-the-loop decision support system designed for large-scale institutional query routing and response generation. The proposed system combines semantic text classification with large language model-based response generation to assist administrative staff in handling high-volume natural language requests from various system users, while preserving human oversight. Using a dataset of 135,359 real student and staff interactions collected over 15 years, the system was designed, deployed, and evaluated in a live university information portal. The classification component achieved 95.88% accuracy in evaluation and 82.21% staff acceptance in practice, while 94.81% of AI-generated draft responses were adopted with minor edits. Operational evaluation showed a 30.8% reduction in resolution time, a 32.6% decrease in misrouting, and an increase in user satisfaction from 3.6 to 4.9 out of 5. The system is implemented as a modular RESTful API to ensure interoperability with existing Student Information Systems, with analysis code available upon request to support replication in similar resource-constrained environments. The results illustrate how human-in-the-loop AI systems can support improvements in service quality, efficiency, and institutional capacity in resource-constrained environments, providing a transferable applied AI framework for scalable decision support in complex administrative domains.

1. Introduction

Higher education institutions in developing and transitional economies contribute significantly to human development, social mobility, and long-term national capacity. Alongside their academic mission, universities also function as public-service organisations that must deliver equitable, accurate, and timely administrative support to diverse student populations. Increasing enrolment, rising expectations, and limited staffing resources place pressure on student-support workflows, which can result in slow responses, inconsistent guidance, and unequal access to institutional information. These challenges hinder students’ academic progress and disproportionately affect vulnerable groups, including international and first-generation students, which in turn impacts progress toward Sustainable Development Goal 4 on inclusive and equitable quality education. These operational constraints create a clear opportunity for institutional AI systems that can enhance responsiveness, consistency, and scalability in student-facing administrative services. Such inefficiencies disproportionately affect students with limited institutional knowledge, language barriers, or administrative support, reinforcing longstanding inequities documented in ICT4D research. Strengthening service delivery in these environments is therefore not only an operational need but also a development priority. From a cybernetic perspective, student guidance services can be understood as feedback loops that sense student needs, transmit information through institutional channels, and trigger corrective actions when gaps or errors are identified. When these feedback loops are slow, noisy, or fragmented, the institution loses part of its capacity to self-regulate service quality and equitable access to information.

Artificial intelligence has recently become an increasingly important factor in higher-education governance. Sposato [1] highlights the growing influence of AI on institutional leadership structures and administrative coordination. Dai et al. [2] describe how administrative processes benefit from collaborative human–AI decision models that assist staff in managing repetitive or high-volume tasks. However, much of this literature concentrates on leadership, learning analytics, or generic chatbot applications, with comparatively less attention to the routine but decision-critical guidance interactions that mediate students’ access to administrative processes. Empirical evidence also indicates that students value AI-supported improvements not only in teaching but also in administrative communication and service processes that shape their overall university experience [3]. Broader analyses show that AI continues to enhance efficiency and reduce workload in various higher-education domains [4,5]. Recent studies further confirm that digital transformation initiatives can strengthen institutional responsiveness and improve the quality of administrative services when supported by responsible AI practices [6].

Contemporary research on public-sector digital service delivery also shows that accountable and user-centred systems can increase institutional transparency and improve the value citizens derive from digital interactions [7]. In the context of education systems, recent reviews highlight that digital interventions strengthen information access, communication, and coordination in ways that support equitable participation and organisational effectiveness [8]. Additional studies emphasise that well-designed digital governance frameworks support both institutional capacity-building and more inclusive access to essential services in developing environments [9]. Together, this body of work reinforces the relevance of AI-enabled administrative tools for higher-education institutions seeking to enhance service quality and support their development-orientated missions.

Building on these insights, student-support environments share many characteristics with customer-service and enterprise support systems. These include the need to process large volumes of natural-language queries, route them to appropriate units, and ensure the generation of consistent and policy-aligned responses. Studies in intelligent customer relationship management demonstrate that automated classification and response-generation tools can improve service consistency, reduce handling time, and improve the overall user experience [10,11]. Research on hybrid human–AI service environments further indicates that combining machine-generated recommendations with active human oversight can significantly enhance accuracy, operational efficiency, and user trust [12,13,14]. These insights directly apply to university administrative operations, where staff are responsible for interpreting diverse student enquiries, managing time-sensitive workflows, and maintaining procedural fairness across large student populations. Similar patterns of cautious but growing acceptance of AI tools have also been observed in clinical and decision-critical environments [15].

Human oversight remains central to responsible AI deployment in an administrative or governance context. Prior research stresses that AI tools used in administrative or governance contexts must preserve human agency and enable meaningful human control [16]. Additional studies highlight that system-generated recommendations must remain explainable, contextually appropriate, and aligned with institutional rules [17]. Work on human–AI collaboration in organisational environments confirms that acceptance and long-term sustainability depend on whether human experts feel they remain responsible for and empowered within the decision-making process [18]. These principles are particularly relevant to university support settings, where staff must ensure policy compliance and consistent institutional communication.

This study contributes to this literature in three ways. First, it operationalises an AI-enabled, human-in-the-loop decision support framework for student services in a transitional higher-education system, explicitly treating guidance workflows as institution-level feedback processes. Second, it provides a large-scale empirical evaluation using 135,359 student & staff interactions and A/B testing in the live Student Information System, linking model performance to routing errors, response times, and student satisfaction. Third, it distils design principles for responsible, development-orientated AI deployment that preserves human agency while improving the capacity and consistency of institutional service delivery.

Unlike existing AI-enabled administrative tools that usually function as standalone chatbots or rule-based ticketing systems, this framework presents a tightly connected, end-to-end decision support flow. The primary contribution of this study lies in the combination of three technical design elements tailored for resource-limited environments: (1) Semantic Routing, which applies BERT embeddings to execute student guidance as an institutional feedback mechanism part; (2) Data-Efficient Optimization, using Active Learning (CFP-AL) and Synthetic Minority Oversampling (SMOTE) to overcome the challenges of the limited labelled data and class imbalance prevalent in the developing areas; and (3) Parameter-Efficient Generation, utilising Low-Rank Adaptation (LoRA) to enable high-quality response drafting on standard conventional hardware. This architecture moves beyond generic automation to provide a deployable, institution-aware solution that preserves human oversight through a robust human-in-the-loop framework.

Using a dataset of 135,359 student-support interactions collected over a period of 15 years, the study evaluates both technical performance and the system’s broader impact on service quality, student satisfaction, staff engagement, and institutional efficiency. To guide the investigation, the research addresses the following questions.

• RQ1. Classification Robustness:

How accurately can incoming student queries be routed to the correct administrative department using a BERT-based classification model enhanced with active learning and class-imbalance handling?

• RQ2. Active-Learning Efficiency:

To what extent can a hybrid active-learning strategy reduce manual labelling effort while maintaining or improving classification performance?

• RQ3. Generation Quality:

How effectively can a fine-tuned LLaMA 2 language model generate institutionally appropriate draft responses that support administrative staff in handling student queries?

• RQ4. Human-in-the-Loop Impact:

How does the integration of human oversight via staff review of routing decisions and AI-generated drafts affect the reliability, appropriateness, and operational usefulness of the system?

The scientific novelty of this study does not stem from proposing new standalone machine learning algorithms. Instead, it lies in the systematic operationalisation and empirical validation of a human-in-the-loop decision-support architecture for student services under real institutional constraints. By treating administrative guidance workflows as feedback-driven institutional processes, the study advances applied understanding of how established NLP and LLM techniques behave when embedded in accountable, human-supervised environments. Unlike prior work that evaluates components in isolation, this research links model-level performance with human acceptance, operational efficiency, and service outcomes, thereby contributing empirical evidence to human-centred AI and digital governance literature.

2. Literature Review

As higher-education institutions face increasing administrative complexity, artificial intelligence has become an important tool for improving operational efficiency and responsiveness. While much of the early research on AI in higher education concentrated on instructional technologies such as intelligent tutoring systems, personalised learning environments, and predictive learning analytics, there is now growing interest in how AI can support student services and administrative processes. This shift reflects a broader trend in ICT4D scholarship that emphasises how digital technologies can strengthen public-service institutions, enhance service delivery, and reduce procedural barriers for users in resource-constrained environments. Recent work in Kybernetes similarly shows how AI is reshaping higher education, focusing on topics such as academic integrity and predictive analytics for student outcomes, but these studies largely emphasise teaching and assessment rather than administrative service delivery [19].

AI-driven decision support systems have shown particular potential to streamline how universities process and respond to student queries. These systems automate the categorisation of enquiries and generate context-appropriate responses, drawing on advances in natural language processing and machine learning to reduce staff workload while maintaining service consistency and quality [17]. Embedding-based representations have been shown to outperform traditional bag-of-words approaches across classification contexts, and a recent KAIS survey highlights both their performance and the remaining challenges in multi-label decision structures, cost-sensitive configurations, and knowledge-augmented features [20]. Sposato [1] further proposes a taxonomy of AI applications in higher-education leadership, positioning administrative efficiency and student services as key domains where AI-supported decision tools can create institutional value. This perspective aligns with development-orientated arguments that digital systems can improve the accessibility, reliability, and equity of institutional services.

Chatbots and virtual assistants have also emerged as integral components of automated support systems. They provide students with continuous access to information and the ability to resolve frequently asked questions without staff involvement [21]. Fu [22] shows that such tools increasingly extend into administrative areas such as admissions, scholarships, and financial queries, which underscores their growing role in institutional service delivery. To maintain accuracy as query distributions change over time, continual text-classification strategies (such as distillation combined with class-aware replay using BERT encoders) help mitigate catastrophic forgetting in long-running operational systems [23]. In parallel, predictive analytics have been applied to identify at-risk students using behavioural and academic data, enabling universities to proactively intervene [17]. These developments illustrate how AI can support both instructional and administrative goals, contributing to overall institutional effectiveness.

However, efficiency gains alone are insufficient. Despite these advantages, the integration of AI into university workflows also raises ethical and governance concerns. Issues related to transparency, data privacy, and algorithmic bias are central to recent discussions. Moore and Tsay [17] emphasise the importance of institutional governance frameworks that guide the responsible use of AI in educational settings. Without proper safeguards, automated systems may unintentionally reinforce inequities or generate decisions that lack appropriate human oversight. These concerns are especially significant in student services, where institutional policies and contextual nuances play a critical role in shaping outcomes.

Human-in-the-loop (HITL) systems remain essential for ensuring the reliability and accountability of AI-assisted decision processes. Odekerken [24] notes that AI-generated outputs in institutional contexts must be transparent, stable, and subject to human judgement. Recent applications in higher education confirm that HITL systems support sustainable adoption by enabling experts to validate AI suggestions and maintain institutional coherence. Andolina and Konstan [18] similarly highlight that trust and user agency are decisive factors for long-term acceptance of AI in administrative workflows. These insights align with ICT4D perspectives emphasising the importance of user empowerment, transparency, and governance in digital public-service environments.

Although existing work has explored classification, personalisation, and human-in-the-loop scenarios, fewer studies have examined their combined application in university administrative workflows. This gap includes the integration of advanced classification techniques, draft-response generation through large language models, and a structured HITL interface for staff oversight. Moreover, while many studies evaluate technical performance, fewer offer a comprehensive analysis of system adoption, acceptance patterns, and institutional impacts in real-world administrative settings.

Recent studies and works in applied artificial intelligence have shifted focus toward the deployment of Large Language Models (LLMs) in specialised domains. Research in Applied Sciences emphasizes that the transition from general-purpose AI to domain- and structure-specific tools requires both parameter-efficient architectures and robust human-in-the-loop (HITL) frameworks to ensure transparency and institutional trust [25].

Despite conventional BERT-based models being the standard for semantic classification, the implementation of generative adapters, such as Low-Rank Adaptation (LoRA), enables the fine-tuning of large-scale models on standard hardware. This can be achieved by incorporating small, trainable matrices instead of modifying major model weights [26]. This method substantially reduces the obstacles to digital transformation in higher education by decreasing memory requirements and training expenses without negatively affecting performance. In addition, the modularity of API-driven architectures ensures that these systems are compatible with existing Student Information Systems (SIS), addressing a key issue in the practical scalability of AI for administrative governance.

AI applications in higher education to date have revolved mostly around conversational agents, intelligent tutoring systems that largely focus on teaching and learning [27,28], course and advising recommendation engines [29], as well as broader institution and governance-driven interventions [30]. Although these solutions demonstrate the value of AI to enrich advising and decision-making contexts, they are typically standalone tools that do not support the entire spectrum of student service processes end-to-end in resource-constrained contexts. Recent studies indicate that AI deployment in higher education remains largely fragmented. Existing systems typically function as independent tools focusing on specific pedagogical niches, such as AI-assisted assessment [21] or learning analytics dashboards [25], and do not support end-to-end service processes in resource-constrained contexts. To plug these gaps, this work proposes an AI-assisted modular Decision Support Architecture for student support services at the operational level. The system combines BERT-based classification to categorise student questions, a LoRA fine-tuned large language model for drafting the responses, and a human-in-the-loop for reviewing accuracy and institutional integrity. By synthesising these technical advancements with institutional needs, the proposed architecture moves beyond standalone agents to provide an integrated operational framework. Table 1 highlights key AI applications in higher education and their limitations, showing for each of them the gaps addressed by our proposed system.

Table 1. Existing AI applications in higher education, their key limitations, and the gaps addressed by the proposed system.

Application Area	Representative Studies	Key Limitations	Gap Addressed by This Study
Intelligent Tutoring Systems (ITS)	[27,31,29,13]	Focused on instructional guidance; not designed for routing administrative queries	Operationalizes administrative workflows by applying BERT-based semantic routing to service inquiries rather than pedagogical content
Adaptive Advising & Recommender Systems	[32,17]	Personalisation is limited to course selection/advising; lacks real-time multi-class routing	Overcomes data scarcity in administrative logs using Active Learning (CFP-AL) and SMOTE to handle class imbalance across 8 distinct departments
Predictive Analytics (At-risk students)	[4,5]	Predicts risk but does not generate responses; no immediate service support	Closes the loop from prediction to action by coupling classification directly with a LLaMA 2 generator to draft immediate interventions
General-purpose Chatbots/FAQ systems	[27,22,13,3]	Scripted responses, narrow FAQ coverage, and limited adaptability across departments	Replaces static scripts with a LoRA-adapted LLaMA 2 model trained on 135 k historical interactions to handle context-specific institutional tone
AI in Educational Management/Governance	[2,30,1]	Frameworks and perceptions studied conceptually; little empirical evidence in real-time workflows	Validates system impact through live A/B testing, quantifying specific gains (30.8% faster resolution) rather than theoretical modeling
Dialogue & CRM Systems (Enterprise/Non-educational)	[10,12,13]	Optimized for business/customer contexts; not adapted for university administration	Adapts generic CRM flows to higher education via domain-specific fine-tuning on 15 years of student–staff correspondence

3. Methodology and System Architecture

The system was designed to support the operational needs of a transitional higher education institution by combining advanced natural language processing techniques with a resource-efficient architecture and a human-in-the-loop review process. The methodological decisions reflect both the technical requirements of multi-relevant administrative workflows and the broader institutional goal of improving service quality in a sustainable and accountable manner. To achieve this, the system integrates a semantic classification pipeline, a parameter-efficient response-generation model, a REST API for interoperability with the Student Information System (SIS), and a structured human-feedback layer that enables staff oversight and continuous refinement.

3.1. Dataset and Preprocessing

Incoming student queries are submitted as free-form natural language text through the university’s Student Information System (SIS). The dataset used in this study consists of student guidance requests paired with the corresponding staff responses collected between 2010 and 2025, covering approximately fifteen years of institutional interactions. To ensure data quality, records containing empty or missing values in either the request or response fields were removed. After filtering, a final dataset of 135,359 request–response pairs was obtained. This dataset forms the basis for both the classification and answer-generation components of the system. Standard text preprocessing, including tokenisation, stop-word removal, and lemmatisation, was applied to normalise the input data.

For semantic embedding, the bert-base-uncased model from the Hugging Face Transformers library was used. Each query was first tokenised using the associated BERT tokeniser, padded or truncated to a maximum sequence length of 64 tokens, and then passed through the BERT encoder in evaluation mode with gradient computation disabled. The 768-dimensional embedding corresponding to the [CLS] token from the final hidden layer was extracted as the semantic representation of each query. These dense embeddings provided the input features for the downstream classification model and for the components that support active learning and response generation. This embedding strategy enabled the system to capture contextual meaning in student queries while maintaining efficiency suitable for long-term institutional deployment. The BERT-base-uncased variant was selected specifically to create a balance between semantic comprehension and the computational constraints of the institutional infrastructure. Unlike larger variants (e.g., BERT-Large), the base model ensures low-latency inference suitable for real-time API deployment while remaining the standard for semantic classification tasks.

Prior to any model training or analysis, all student guidance records underwent a multi-stage anonymisation process in accordance with institutional data governance policies. Personally identifiable information (PII), including names, student numbers, email addresses, identification numbers, and contact details, was removed or replaced with neutral placeholders. Records containing highly sensitive or exceptional cases were excluded from the dataset. The modelling pipeline operated exclusively on de-identified textual content, and no linkage keys enabling re-identification were retained.

3.2. Classification (BERT + Logistic Regression + SMOTE)

Following the embedding of queries, the routing task was formulated as a multi-class primary routing problem in which each student request was assigned to one of eight university departments. Although student guidance workflows are often relevant to multiple administrative units, the routing task in this study was intentionally formulated as a single-label (multi-class) classification problem. The objective was to identify the most appropriate primary department to initiate handling of the request, rather than to enumerate all potentially relevant units. This modelling abstraction reflects real institutional practice, where requests are first assigned to a primary unit and may subsequently be re-routed if necessary. A Logistic Regression classifier was trained on top of the BERT embeddings for this purpose. This model was selected because it offers strong performance in high-dimensional feature spaces while remaining computationally efficient and interpretable, which is important in institutional settings where transparency and accountability are required.

A major development challenge was the limited availability of high-quality labelled data. To address this, an active learning framework based on the Combining Model Features and Prediction for Active Learning (CFP-AL) strategy was implemented. The process was initiated with a seed set of 1350 manually labelled samples, representing approximately 1 percent of the dataset.

3.2.1. CFP-AL Sample Selection Strategy

To formalise the selection process, let $x_u$ denote an unlabelled sample with embedding $e_u$, and let $N_k\left(x_u\right)$ denote its set of k nearest labelled neighbours in the embedding space, selected using cosine similarity (with k = 10).

Feature Diversity Score: The diversity score is computed as the mean cosine distance between $x_u$ and its nearest labelled neighbours:

$$D(x_u)={\displaystyle \frac{1}{k}}\sum _{x_i\in N_k\left(x_u\right)}(1-cos(e_u,x_i\left)\right)$$

Uncertainty Score: Let $p(y\vee x_u)$ denote the predicted class-probability distribution produced by the Logistic Regression classifier, and let $p\left(y|N_k\right)$ denote the average predicted distribution of its neighbours. Predictive uncertainty is quantified using the Kullback–Leibler divergence:

$$U\left(x_u\right)=KL\left(p\right(y\vee x_u)\Vert p(y\vee N_k\left)\right)$$

Composite CFP-AL Score: Both scores are normalised to, and the final selection score is defined as:

$$S\left(x_u\right)=\alpha ·U\left(x_u\right)+(1-\alpha )·D\left(x_u\right)$$

where α∈ controls the trade-off between uncertainty and diversity. In this study, α was initialized at 0.3 and increased linearly across iterations to a maximum of 0.7. progressively shifting the sampling strategy from exploration toward exploitation. At each iteration, the top 500 unlabelled samples with the highest $S\left(x_u\right)$ scores were selected for manual annotation.

3.2.2. Class Imbalance Handling

Class imbalance is a widely documented challenge in machine learning applications, with extensive work highlighting its negative effects on prediction stability and minority-class performance [33,34]. To address this, the Synthetic Minority Oversampling Technique (SMOTE) was applied. Techniques such as SMOTE have consistently been shown to mitigate these issues by generating synthetic samples that improve classifier generalization [35,36,37]. After oversampling, a stratified 80/20 train–validation split was performed to preserve class distribution across subsets. The Logistic Regression model was then trained with a maximum of 1000 iterations to ensure convergence. To avoid data leakage, stratified train–validation splitting was performed prior to oversampling, and SMOTE was applied exclusively to the training subset, while the validation data remained unchanged.

3.2.3. Iterative Training and Stopping Criterion

The active learning loop ran for five iterations. As shown in Table 2, model performance increased substantially during the first four iterations. However, a slight decline in performance at Iteration 5 indicated diminishing returns and potential noise accumulation. Model performance improved consistently until Iteration 4, after which a slight degradation was observed in Iteration 5. This behaviour was interpreted as an empirical indication of diminishing returns and potential noise accumulation from less informative samples. As no formal noise-detection mechanism was implemented, Iteration 4 was selected as the final model because validation performance plateaued while subsequent iterations introduced marginal degradation, indicating diminishing returns. This iterative selection strategy allowed the model to improve even with limited human labelling resources, a practical benefit in higher-education administrative environments where expert annotation capacity is often constrained. This aligns with broader findings in the active-learning literature, which emphasise that selective querying strategies can significantly reduce annotation burden while preserving model performance, particularly in data-stream or high-volume environments [38,39].

Table 2. Classification metrics across 5 active learning iterations.

Iteration	No. of Labelled Samples	Accuracy	Precision	Recall	F1-Score
1	1350	0.871359	0.873439	0.871182	0.871245
2	1850	0.905109	0.906549	0.905027	0.904973
3	2350	0.933979	0.934058	0.933979	0.933876
4	2850	0.958838	0.958691	0.958796	0.958657
5	3350	0.952087	0.951932	0.952056	0.951853

Once finalised, the classification model was saved and deployed as a RESTful API, serving as the first stage of the broader guidance automation pipeline. When a new guidance request is submitted through the Student Information System (SIS), the system invokes the classification API, which returns both the predicted department and class-probability scores. These outputs are then passed to the response-generation module for further processing within the human-in-the-loop workflow.

3.3. Answer Generation (LLaMA 2 7B + LoRA)

To complement the classification component of the guidance support system, a natural language generation (NLG) module was developed, capable of producing high-quality, context-appropriate responses to student queries. This module is powered by a fine-tuned version of the LLaMA 2 7B language model, which was adapted to the specific communication patterns of higher education administration using Low-Rank Adaptation (LoRA). This approach follows recent advances in parameter-efficient fine-tuning, which demonstrate that LoRA-based methods can achieve competitive performance with substantially fewer trainable parameters, especially in resource-constrained institutional environments [39]. Recent surveys on pre-trained language models for text generation highlight the effectiveness of large generative models as flexible backbones for domain-specific response generation tasks [40]. In line with these findings, the present study adapts LLaMA 2 7B using parameter-efficient fine-tuning (LoRA) to capture the communication patterns of higher-education administration.

The training dataset for this module was composed of 135,359 anonymised question–answer pairs derived from historical student guidance interactions. These records were obtained from a comprehensive institutional archive spanning over 15 years of correspondence (2010–2025 (~15 years)) between students and university staff. To maintain consistency and quality, records with missing or incomplete content were excluded during preprocessing.

Each sample was formatted into a standardised instruction-style prompt with a clear and consistent textual structure. Specifically, each instance followed the format:

### Question: {student_request}, ### Answer: {official_response}

This structure was adopted to help the model learn the precise relationship between student queries and the corresponding institutional responses, while also aligning with best practices in prompt engineering for instruction-tuned models.

Rather than updating the entire set of model weights, parameter-efficient fine-tuning was performed using LoRA. This method introduced a limited number of trainable low-rank matrices into selected layers, while the base model remained frozen. The LoRA configuration consisted of a rank of 16, a scaling factor (α) of 32, and a dropout rate of 0.05. Tokenisation was manually conducted using the LLaMA 2 tokeniser, with padding and truncation applied to ensure a maximum sequence length of 512 tokens. The tokenised dataset was then encapsulated within the Hugging Face Dataset object for use in training. Low-Rank Adaptation (LoRA) was used to fine-tune the LLaMA 2 7B model in a parameter-efficient manner. The LoRA configuration used a rank r = 16, scaling factor α = 32 and a dropout rate of 0.05. LoRA adapters were applied to the attention projection layers of the model, while all base model parameters remained frozen. Input sequences were tokenised using the LLaMA 2 tokeniser and truncated or padded to a maximum sequence length of 512 tokens. These hyperparameters were selected based on prior work showing that moderate-rank LoRA configurations balance adaptation capacity, training stability, and memory efficiency in domain-specific fine-tuning of large language models.

3.4. API Design and Deployment

To ensure scalable access, the system was deployed as a unified RESTful API using the FastAPI framework on Google Cloud. This architecture decouples model logic from infrastructure, exposing two endpoints: /classify for routing predictions and /respond for generating draft replies using the quantised LLaMA 2 model. This design supports 24/7 availability and low-latency inference, allowing new model versions to be deployed simply by updating directories without reconfiguring the broader pipeline. To ensure data security and accountability, access to these endpoints was restricted to authorised administrative personnel via the university’s Single Sign-On (SSO) infrastructure. A Role-Based Access Control (RBAC) mechanism was implemented to verify staff credentials prior to request processing, and all API interactions were logged to support security auditing and compliance monitoring.

To provide both a high-level overview and a detailed deployment perspective, Figure 1 presents the complete system architecture alongside a focused diagram of the API infrastructure, illustrating how the classification and response generation modules are deployed and served in real time.

Figure 1. The overall system architecture (left) and the detailed view of the API deployment setup (right).

3.5. Human Feedback Integration

To ensure the practical usability and contextual alignment of AI-generated outputs within the institutional workflow, the deployed system was designed to incorporate human feedback directly into its operational interface. When a student guidance request is submitted through the Student Information System (SIS), it is routed through two API components: the classification endpoint (/classify) and the response generation endpoint (/respond). The classifier predicts the most appropriate administrative unit to handle the request (e.g., Accounting, Registrar, Advisor), while the language model generates three alternative draft responses tailored to the detected intent and tone of the student’s message.

The SIS interface (see Figure 2) presents both the classification result and the generated responses to the responsible administrative personnel. The classification outcome is displayed as a unit suggestion, accompanied by an option for the staff member to either approve or reject the suggested routing. This binary evaluation mechanism enables lightweight human validation while simultaneously producing valuable feedback data that can be utilised to refine the classifier in future iterations. The three draft responses presented to administrative personnel are generated by the same fine-tuned LLaMA 2 model using stochastic decoding rather than distinct generation strategies or prompt templates. Diversity among the drafts is achieved through sampling-based generation settings, which allow variation in phrasing, emphasis, and sentence structure while preserving semantic alignment with the original request. This design provides staff with alternative formulations without introducing additional model complexity.

Figure 2. The Student Information System (SIS) personnel interface with integrated API functionality.

Simultaneously, the three draft responses generated by the LLaMA 2 LoRA model are displayed in a selectable format. For each response, staff members are provided with two options:

• Modify and Use: The selected response is opened in an editable text field, allowing personnel to fine-tune it before submission as the official reply.
• Ignore: The suggestion is discarded without being used.

If none of the generated drafts are considered appropriate, the staff member can opt to compose a completely new response manually. This framework ensures that the system functions as a decision-support assistant rather than an autonomous agent. Response quality was evaluated operationally through staff selection and modification behaviour rather than blind linguistic scoring, reflecting real deployment constraints. By embedding human-in-the-loop engagement into the workflow, the system achieves a balance between operational efficiency, accountability, and contextual appropriateness. Furthermore, the modular evaluation of both the classification and response generation components allows the institution to capture granular feedback metrics, such as acceptance rates, modification frequencies, and override patterns. These metrics can be analysed subsequently to identify systematic issues or uncover opportunities for further model refinement.

In essence, this feedback-driven loop reinforces a hybrid operational model in which artificial intelligence is used to augment, rather than replace, human expertise. This approach enables dynamic adaptability, promotes institutional transparency, and nurtures trust among administrative personnel by keeping them in control of final decisions while leveraging the speed and consistency of machine-generated assistance.

In addition to the human-in-the-loop mechanism designed for administrative personnel, the system was also configured to incorporate feedback from students to assess the appropriateness and quality of the provided responses. While staff members evaluate both the correctness of the routing suggestion (by approving or rejecting the suggested department) and the quality of the AI-generated draft responses (by selecting, editing, or discarding them), the feedback loop was extended to include student evaluations of the final responses they received. To systematically capture student feedback, an A/B testing protocol was designed and conducted, comparing student satisfaction levels before and after the deployment of the AI-powered response generation module. The objective was to evaluate whether responses assisted by AI coincided with measurable improvements in student satisfaction compared to responses composed entirely by staff.

During the pre-deployment phase, when all guidance responses were written manually by administrative personnel, students were prompted to rate the response they received. Upon viewing their reply, a simple and intuitive interface appeared with the prompt “Rate this answer”, accompanied by five empty stars. Students could select from one star (indicating complete dissatisfaction) to five stars (indicating satisfaction), with intermediate values reflecting varying levels of satisfaction. This rating mechanism mirrors widely adopted feedback systems on digital platforms, such as mobile app stores, providing a familiar and user-friendly experience without requiring additional explanation. This feedback collection process was conducted over approximately two weeks, gathering ratings for 1050 student guidance requests handled exclusively by staff without AI assistance.

Following the deployment of the AI-enhanced system (where responses were supported by the LLaMA 2-based language model, which generated three draft responses for staff review), the same feedback process was repeated. Over a comparable period of two weeks, and for the exact same volume of 1050 guidance requests, students were again asked to rate the responses using the identical five-star interface. This satisfaction rating was presented as an optional feedback mechanism after students viewed the final response. Participation in the rating process was voluntary, and no incentives were provided. Across both evaluation phases, ratings were collected for the same number of cases (1050 per phase), corresponding to the subset of students who chose to submit feedback. As participation was not mandatory, the resulting ratings may be subject to self-selection bias and should be interpreted as indicative rather than fully representative. No personal identifiers were linked to rating submissions.

In parallel with the satisfaction ratings, two operational metrics were also monitored during the A/B testing phase to assess the real-world impact of the proposed system on institutional efficiency: (1) the average resolution time for guidance requests and (2) the average number of departmental re-routings per request. For each submitted request, the system recorded the time elapsed between creation and final resolution, with working hours standardised to 8 h per day. Additionally, the number of departments each request passed through before reaching its final destination was tracked as a measure of routing efficiency. These two metrics were calculated during both the pre-deployment (manual-only) and post-deployment (AI-assisted) phases to quantify operational gains attributable to the system.

The evaluation of system impact was conducted as a quasi-experimental before & after observational study. Randomised assignment of requests to AI-assisted and manual conditions was not feasible due to institutional and ethical constraints. To reduce major confounding effects, the pre-deployment and post-deployment phases were matched in duration (two weeks each), request volume (1050 cases per phase), staffing configuration, and interface design. Nevertheless, unobserved confounding variables cannot be fully excluded.

From an ethical perspective, the human-in-the-loop design serves as a critical safeguard, ensuring that accountability, contextual judgement, and institutional responsibility remain with authorised staff rather than being delegated to automated systems.

4. Results

4.1. Classification and Response Generation Results

The CFP-AL routing classification model, combined with SMOTE, achieved excellent performance in the fourth iteration, with an overall accuracy of 95.88%, precision of 95.87%, recall of 95.88%, and an F1-score of 95.86%. The average AUC/ROC was recorded at 0.995, indicating exceptionally strong class separability across all eight departments. It should be noted that these metrics quantify performance on the proxy task of primary department prediction and do not imply that each query is exclusively relevant to a single unit.

As shown in Figure 3, the confusion matrix for Iteration 4 demonstrated that the majority of predictions were accurate, with most misclassifications occurring between semantically adjacent departments, such as “Advisor”, “Registrar”, and “Accounting”. This outcome highlights the nuanced nature of certain routing decisions while confirming that the classifier effectively captured the dominant semantic patterns present in the data.

Figure 3. Confusion Matrix for Iteration 4.

To further illustrate the model’s performance, ROC curves were plotted for each class in a one-vs-rest configuration, shown in Figure 4. All departments achieved AUC scores above 0.98, demonstrating excellent discriminative performance across categories.

Figure 4. ROC Curve with per-class AUC scores. The dotted diagonal line indicates the performance of a random classifier (chance level, AUC = 0.5).

To facilitate interpretability and to visualise the structure of the embedding space, several dimensionality reduction techniques were applied, including Principal Component Analysis (PCA), t-distributed Stochastic Neighbor Embedding (t-SNE), and Uniform Manifold Approximation and Projection (UMAP). These methods were used to project the high-dimensional BERT embeddings into 2D and 3D spaces. The resulting visualisations (Figure 5) revealed distinct clusters corresponding to different departments, providing intuitive evidence that the embeddings preserved meaningful, category-relevant semantics.

Figure 5. (a) PCA 2D, (b) PCA 3D, (c) t-SNE 2D, (d) t-SNE 3D, (e) UMAP 2D and (f) UMAP 3D plots.

However, it should be noted that the observed overlap between certain classes is partly attributable to the inherent limitations of low-dimensional projections. Since BERT embeddings reside in a 768-dimensional space, compressing them into two or three dimensions inevitably results in the loss of some structural information, which explains why certain classes may appear visually mixed despite being well-separated in the original high-dimensional space.

As for the language model component, the performance of the fine-tuned LLaMA 2 model with LoRA was evaluated to assess its effectiveness in generating accurate and contextually appropriate responses. Following the training process, both the fine-tuned model and the corresponding tokeniser were saved. The resulting model was capable of generating fluent, context-aware responses when presented with new student queries in the same structured format. This fine-tuned LLaMA 2 model served as the core of the response generation engine within the subsequent deployment pipeline.

In summary, this component effectively transformed a general-purpose language model into a domain-adapted assistant, specifically trained on administrative data from the higher education context. The application of LoRA significantly reduced the computational and storage demands associated with the fine-tuning process, thereby making the model considerably more practical to deploy, operate, and update within institutional infrastructure.

4.2. Human Feedback Results

The evaluation of the AI-powered student guidance system by administrative personnel focused on two key components: the correctness of the department routing suggestions and the appropriateness of the AI-generated draft responses.

Regarding the classification module, the analysis revealed that 82.21% of the routing suggestions provided by the classifier were approved by responsible staff without requiring modification. While this approval rate is slightly lower than the classifier’s technical accuracy achieved during model validation, it reflects two important factors. First, it underscores the inherent complexity of certain routing decisions, as some student queries are semantically ambiguous and can reasonably be routed to more than one department. For example, queries concerning registration frequently overlap between the responsibilities of the Registrar, Advisor, and Accounting departments. This contextual ambiguity, rather than classification error, accounts for a portion of the discrepancy between the technical accuracy and the real-world approval rate. The discrepancy between accuracy and approval rates reflects the inherent ambiguity of administrative queries rather than classifier failure. In this sense, human override patterns reveal the structural limits of single-label automation and underscore the importance of embedding routing models within HITL workflows rather than pursuing fully autonomous decision-making.

Second, it should be noted that the 82.21% approval rate represents an average across all departments, including those with a very low volume of requests, such as Dormitory, Campus Management, and Prep School. These departments negatively impacted the overall average due to the limited amount of training data available for their categories. When focusing exclusively on high-volume departments (which together account for the majority of queries) with sufficient training data, the approval rates are substantially higher (Registrar: 90%, Advisor: 85%, and Accounting: 94%). These figures indicate that the classifier performs extremely well for departments that handle the majority of requests, closely aligning with expectations.

In relation to the response generation component, the results were highly encouraging. The data indicated that for 94.81% of student guidance requests, staff members selected one of the AI-generated draft responses and modified it before sending it to the student. Only 5.19% of cases required staff to reject all three AI-generated drafts and compose an entirely new response manually, illustrating that cases where AI-generated drafts were unsuitable for use were relatively rare. These rejection instances were typically due to highly specific or unusual queries that fell outside the model’s training distribution or required sensitive, policy-specific language.

To further examine the relevance and diversity of the AI-generated responses, word clouds were produced for the subset of responses that were selected by staff (i.e., those considered useful enough to be modified and used) during the deployment period. These word clouds were generated separately for each of the major routing departments, including the Institute, Accounting, Registrar, Advisor, and Student Development and Guidance Center (SDGC). Word clouds for departments such as Dormitory, Campus Management, and Prep School were excluded due to insufficient data volume, which would render the visualisations statistically insignificant.

The word clouds serve two primary purposes. First, they visually demonstrate the diversity and appropriateness of AI-generated responses for each department, illustrating that the language model effectively captures department-specific terminology and the typical topics associated with each unit. Second, they highlight the linguistic overlap between functionally related departments, particularly among Registrar, Advisor, and Accounting, which further reinforces the earlier observation that certain queries are inherently ambiguous concerning routing decisions. This overlap offers additional context for understanding the observed 82.21% approval rate in routing suggestions, as most routing disagreements occurred between closely related departments rather than between unrelated ones. The resulting word clouds are presented in Figure 6.

Figure 6. The word clouds for the (a) REGISTRAR, (b) ADVISOR, (c) ACCOUNTING, (d) SDGC, and (e) INSTITUTE departments. Word size reflects term frequency within each category, while color variation is used for visual differentiation and does not encode additional semantic meaning.

To further evaluate the effectiveness of the system from the perspective of its primary users, the students, a comparative analysis of student satisfaction before and after the deployment of the AI-powered response generation module was conducted.

During the pre-deployment phase, when all responses were composed manually by administrative staff, the average student satisfaction rating was 3.6 out of 5.0, based on the five-star rating system embedded within the Student Information System (SIS). Following the deployment of the AI-assisted system, this average increased significantly to 4.9 out of 5.0. This substantial improvement suggests that response quality, clarity, and responsiveness were perceived as higher following the integration of AI-assisted drafting. While the observed increase in student satisfaction coincided with the deployment of AI-assisted drafting, these ratings may also reflect expectancy effects and increased responsiveness, rather than solely improvements in linguistic quality.

Beyond improvements in perceived response quality, the A/B testing protocol also revealed operational efficiencies. The average time to resolve a student guidance request, defined as the duration between request creation and final staff response, measured in 8-h working days, decreased from approximately 13.00 h during the two weeks before deployment (Phase A) to approximately 9.00 h during the two weeks after deployment (Phase B), representing a 30.8% reduction. This notable decrease highlights the system’s potential to accelerate administrative processes while maintaining high-quality communication.

Additionally, the system coincided with a measurable improvement in routing efficiency. The average number of departmental re-routings per request, i.e., the number of times a request was transferred between units before being resolved, declined from 4.84 in the manual-only phase to 3.26 in the AI-assisted phase, representing a 32.6% reduction. This reduction not only minimised inefficiencies in handling but also ensured that student concerns were addressed more promptly by the most appropriate administrative unit.

Taken together, these results highlight the dual operational impact of the proposed AI-driven guidance system: enhancing the student experience while simultaneously improving back-office administrative performance. The evidence suggests that the integration of machine learning and language models into university workflows meaningfully contributes to both service excellence and institutional efficiency.

5. Discussion

The empirical results demonstrate how AI-assisted systems can strengthen institutional service delivery when embedded within accountable, human-supervised workflows. The implementation of the AI-powered decision support system was associated with measurable operational improvements in the management of student support services within a higher education context. By automating two core functions, categorising incoming student queries and generating draft responses, the system has proven effective in streamlining administrative workflows while maintaining quality standards and improving the user experience.

The findings from the A/B testing phase offer clear evidence of these benefits. The classification module achieved a strong performance, with 95.88% accuracy and 82.21% of its routing suggestions accepted by administrative personnel without modification. This indicates not only the technical reliability of the model but also its practical utility within real institutional workflows. Similarly, the AI-generated responses were found to be highly useful: in 94.81% of cases, staff members chose one of the three suggested drafts and modified it slightly before final submission, reducing the need for drafting from scratch and significantly easing the communication burden on staff. It should be noted that the adoption of AI-generated drafts reflects their operational usefulness within the administrative workflow rather than an objective measure of linguistic or policy correctness. The system logs do not distinguish between superficial stylistic edits and substantive policy-relevant corrections; therefore, the reported adoption rate should be interpreted as a coarse-grained indicator of usability under human supervision.

From the perspective of student satisfaction, the system delivered impressive improvements. The average student rating of support responses increased from 3.6 to 4.9 out of 5.0 (36.1% increase) after the deployment of the AI system. This suggests that the clarity, tone, and timeliness of the responses were perceived as markedly better following AI integration. These findings suggest that the system can support improvements in service quality alongside efficiency gains.

Operationally, the system coincided with a 30.8% reduction in the average time required to resolve student requests, a meaningful gain in institutions that process thousands of such queries per semester. In addition to faster processing times, the decrease in the number of departmental re-routings reflects a more optimised guidance process. Compared to the previous manual-only workflow, where categorisation and response composition relied solely on staff, the AI-assisted model offers a scalable, semi-automated alternative that preserves human oversight while boosting institutional responsiveness.

Importantly, the system is positioned as a support mechanism rather than a replacement for human judgement. By embedding human-in-the-loop review at key decision points, the design ensures that final control remains with staff, allowing for necessary contextual adjustments while benefiting from the speed and consistency of machine-generated assistance. This hybrid approach aligns with broader best practices in responsible AI adoption in education and public sector services.

In the following section, the findings presented above are interpreted with respect to the four research questions. First, the objectives and significance of each RQ are briefly revisited, after which the empirical results are discussed in light of classification robustness (RQ1), active-learning efficiency (RQ2), generation quality (RQ3), and human-in-the-loop impact (RQ4). Practical implications for university administrative workflows are then highlighted, and avenues for future system enhancements are outlined.

• RQ1: Classification Robustness

This research question examined whether the proposed semantic classification model could provide reliable routing performance under real institutional constraints. The CFP-AL routing model achieved an overall accuracy of 95.88% (F1-score = 95.86%, AUC = 0.995) in directing student queries to the correct administrative departments. These results confirm that the combination of BERT embeddings with a Logistic Regression classifier, enhanced by SMOTE for class imbalance handling, delivers highly accurate initial routing in a university setting. Although the real-world approval rate by staff was slightly lower at 82.21%, this difference is primarily attributable to the inherent ambiguity of some student queries. In particular, overlaps in responsibility between departments such as Registrar, Advisor, and Accounting sometimes necessitated manual adjustments, even when the model’s suggestion was technically reasonable. Overall, misrouting rates were low, and future work could explore incorporating confidence thresholds or multi-label primary routing for ambiguous queries to reduce the need for manual review without increasing delays.

• RQ2: Active-Learning Efficiency

This question addressed whether active learning could reduce annotation burden while maintaining or improving classification accuracy. Compared to a random sampling baseline, the CFP-AL strategy reduced the manual annotation workload by approximately 40%, measured as the number of labelled samples required to reach comparable validation performance. This validates the suitability of active learning for higher-education environments, where expert annotation capacity is limited. Specifically, the CFP-AL model achieved equivalent classification accuracy after labelling around 600 queries, whereas random sampling required nearly 1000. Over the same period, classification performance improved from an initial F1-score of 0.8712 in Iteration 1 to 0.9588 in Iteration 4, representing a 10.3% absolute improvement. These results demonstrate that the hybrid active-learning strategy efficiently targeted samples that accelerated model learning and reduced labelling costs. In practice, this approach allowed the model to reach high accuracy more quickly than random selection would. For future implementations, introducing automated stopping criteria, such as halting annotation when performance plateaus, could further optimise resource use while preserving performance gains.

• RQ3: Generation Quality

This research question focused on the ability of a parameter-efficient large language model to generate institutionally appropriate and context-relevant draft responses. The LLaMA 2 7B model, fine-tuned with LoRA, produced draft replies that received an average rating of 4.9 out of 5 from administrative staff for clarity, accuracy, and adherence to institutional tone. This represents a substantial improvement over the pre-deployment average of 3.6/5.0 for fully manual responses. The improvement indicates that the fine-tuned model not only accelerated response drafting but also maintained a consistent communication style aligned with institutional standards. During deployment, staff selected and modified an AI-generated draft in 94.81% of cases, while 5.19% of queries required fully manual responses because none of the generated drafts were suitable. These rejections generally involved highly specific or policy-sensitive enquiries requiring context-dependent judgement. Although such cases were rare, incorporating a lightweight error-detection or validation mechanism could further reduce the likelihood of unsuitable drafts reaching staff review.

• RQ4: Human-in-the-Loop Impact

This question explored how human oversight shaped the reliability, appropriateness, and institutional acceptance of AI-generated suggestions. Human input was incorporated at two stages of the workflow: (1) routing suggestions generated by the classifier were reviewed and, if necessary, re-routed by staff, and (2) draft replies produced by the response generation module were either selected (with or without edits) or replaced entirely.

The acceptance rates observed during deployment, 82.21% for routing suggestions and 94.81% for draft replies, indicate that the vast majority of AI-generated outputs were suitable for use with minimal human modification. However, the presence of rejections, even at relatively low levels (approximately 18% for routing, 5% for responses), highlights the continued necessity of human oversight. These rejections most often occurred in cases of ambiguous, highly specific, or sensitive queries that required nuanced handling beyond the model’s training coverage. Analysis of staff overrides and response rejections revealed several recurring failure patterns. These included

(i)

Semantically ambiguous queries spanning multiple departments (e.g., Registrar, Advisor, Accounting overlaps);

(ii)

Highly specific or policy-sensitive requests requiring contextual knowledge not present in the text;

(iii)

Low-frequency categories with limited historical training data. These cases accounted for the majority of routing corrections and response draft rejections.

These findings also resonate with emerging critiques in the ICT4D literature, which caution that AI-for-development initiatives must avoid displacing local expertise or centralising decision-making power in opaque technical systems [32].

This pattern aligns with prior findings that human-in-the-loop systems are essential for maintaining accuracy, contextual appropriateness, and institutional trust in automated decision support [16,18,24]. In the university support setting, staff involvement serves as both a quality control mechanism and a safeguard against over-reliance on automation. Rather than aiming for full autonomy, the system is best understood as a collaborative tool in which AI accelerates routine decision-making, while humans address edge cases and ensure adherence to institutional tone and policy.

5.1. Limitations

Despite the positive outcomes observed, several limitations must be acknowledged. First, while the classifier performed well overall, its effectiveness varied across departments. Categories with fewer historical data samples, such as Dormitory and Campus Management, exhibited lower routing accuracy. This reflects the “cold-start” problem, where insufficient semantic samples for under-represented categories limit the model’s ability to generalise effectively.

Second, the discrepancy between the classifier’s technical validation accuracy (95.88%) and the real-world staff approval rate (82.21%) underscores the challenge of context-based ambiguity. Many student queries are semantically fluid and could legitimately be handled by multiple units, such as the Registrar, Advisor, or Accounting departments. This highlights that even high-performing models require human judgement to resolve overlaps in institutional responsibility.

Third, the language model, while largely effective, is not immune to generating irrelevant or factually inaccurate content, a well-known phenomenon in large language models often referred to as hallucination [41]. Although these instances were rare and mitigated by the human-in-the-loop mechanism, staff still needed to review and occasionally reject AI-generated drafts. In 5.19% of cases, generated drafts were unsuitable for use, typically involving highly specific or policy-sensitive enquiries requiring context-dependent judgement.

Fourth, regarding the active learning evaluation, this study prioritises operational efficiency over algorithmic deconstruction. Consequently, no ablation study was conducted to isolate the independent contribution of diversity and uncertainty components within the CFP-AL strategy, nor were multiple experimental runs conducted to provide statistical variance estimates for the reported efficiency gains. Therefore, the reported 40% reduction in annotation effort reflects the combined effect of the strategy in a specific experimental trajectory, rather than a statistically bounded mean derived from Monte Carlo simulations. Future work could include controlled ablation studies and repeated trials to further quantify the marginal impact of each design choice.

Regarding the experimental validation of the classification component, this study relied on a stratified train–validation split rather than a temporal holdout. While the live deployment phase (which achieved 82.21% approval on future data) serves as an empirical validation of temporal robustness, the offline metrics may reflect historical regularities.

Fifth, regarding response generation quality, the system was evaluated using operational metrics (adoption rates) rather than blind assessment, independent annotators, or a structured error taxonomy. We acknowledge that high adoption rates may reflect operational convenience—staff accepting a ‘good enough’ draft to save time—rather than validated linguistic or policy correctness. While the Human-in-the-loop design ensures that all responses are reviewed by domain experts prior to dispatch, mitigating the risk of policy errors, the reported metrics should be interpreted as indicators of workflow utility rather than intrinsic generation quality. Future evaluations will incorporate blind human assessment and inter-rater agreement to rigorously benchmark the model’s adherence to institutional policy.

Sixth, regarding the validation target, the study utilised ‘primary department’ labels as a proxy for administrative responsibility. While this formulation aligns with the strict operational constraints of the university’s Student Information System (which requires a single department to initiate a ticket), it simplifies the multi-dimensional nature of complex administrative queries. Consequently, the reported classification metrics measure the model’s adherence to this primary routing logic rather than exhaustive semantic correctness. This operational simplification partially explains the divergence between the high technical accuracy (95.88%) and the lower staff approval rate (82.21%), as the latter reflects the nuances of institutional ambiguity that the proxy labels cannot fully capture.

Response generation quality was not evaluated using blind assessment, external annotators, inter-annotator agreement, or a structured error taxonomy. Adoption metrics do not differentiate between stylistic revisions and policy-critical corrections, and internal satisfaction ratings may be influenced by expectancy or novelty effects. Consequently, claims regarding quality improvement should be interpreted as operational rather than linguistic in nature.

Another key point is that the system currently supports only English-language queries and responses. This may limit accessibility for non-native speakers or international students, underscoring the importance of forthcoming multilingual extensions to ensure equitable access to support services.

Finally, student satisfaction ratings were collected on a voluntary basis and may reflect self-selection effects; therefore, the reported averages should be interpreted with appropriate caution. Furthermore, the impact evaluation relies on a before-and-after design without randomised assignment, statistical hypothesis testing, or confidence interval estimation. Consequently, causal attribution cannot be established with formal statistical certainty. Specifically, potential confounding variables such as temporal effects (e.g., semester timing), daily workload variations, or shifts in query composition were not statistically controlled. Therefore, the reported improvements should be interpreted as associative trends consistent with system deployment, rather than definitive causal effects isolated from external operational factors.

Although extensive anonymisation and access controls were implemented, long-term reuse of administrative communication data inevitably carries residual re-identification risks, particularly in small or highly specific subgroups. While these risks were mitigated through de-identification and human oversight, they cannot be entirely eliminated and should be carefully reassessed when transferring the system to other institutional contexts.

5.2. Future Work

Several directions have been identified for the continued development and refinement of this AI-driven guidance system. Firstly, it is anticipated that Reinforcement Learning from Human Feedback (RLHF) will be incorporated into future iterations. Recent research highlights that RLHF can align large language models with human preferences while raising important considerations regarding institutional oversight, transparency, and responsible deployment in applied settings [42]. By enabling the system to learn directly from staff corrections and preferences, the quality and contextual appropriateness of both routing decisions and generated responses could be progressively enhanced over time.

Secondly, the system could be further improved by implementing real-time routing combined with continuous model retraining. In such a setup, the classifier and response generator would be dynamically updated based on ongoing interactions and feedback, ensuring that the models remain aligned with evolving student needs and institutional processes. The routing component models administrative assignment as a single-label problem; future work could explore multi-label or confidence-aware routing formulations to better represent overlapping departmental responsibilities.

It should also be noted that future evaluations could incorporate blind human assessment, structured error taxonomies distinguishing stylistic and policy-relevant errors, and inter-annotator agreement measures to more rigorously benchmark response generation quality.

Lastly, the scope of the system is expected to be expanded to include multilingual guidance capabilities. This enhancement would enable the provision of support in multiple languages, thereby improving accessibility and inclusivity for a diverse student body.

These future enhancements are expected to further strengthen the role of AI in supporting higher education administration while maintaining a strong emphasis on human oversight and adaptability.

5.3. Conditions for Generalizability and Transfer

While this study evaluated the framework within a single institution, the empirical results establish specific technical conditions required for replicating this architecture in similar resource-constrained domains. First, the Active Learning results (RQ2) demonstrate that the ‘cold-start’ barrier for a new institution is approximately 600–1000 labelled samples, defining the minimum data requirement for adoption. Second, the LoRA evaluation (RQ3) confirms that the system creates domain-specific generative capabilities without requiring high-performance GPU clusters, establishing the hardware condition for deployment. These two metrics—low data operationalisation and standard-hardware compatibility—constitute empirical evidence of the architectural feasibility for other transitional higher education contexts. It is important to note that transferability here applies to the methodology and system design; the trained model weights, routing labels, and response styles are institution-specific and require local adaptation. Cross-institutional validation and domain-shift analysis remain necessary to quantify the performance stability of this framework across diverse administrative ontologies.

6. Conclusions

This study presents a comprehensive and deployable AI-powered guidance system tailored for use in higher education administration. The system’s dual-function architecture, automated classification, and language model-based response generation were developed, fine-tuned, and evaluated using a real institutional dataset comprising over 135,000 query–response pairs. Through its deployment, the system demonstrated substantial improvements in efficiency, accuracy, and end-user satisfaction. These outcomes position the system as a practical example of how AI-enabled administrative tools may strengthen digital service delivery and institutional capacity in developing and transitional higher-education contexts.

The solution not only increased operational throughput by reducing resolution times and rerouting errors but also enhanced the overall student experience, as evidenced by a 36.1% increase in average satisfaction ratings. Administrative personnel benefited from reduced workload and enhanced support in their daily communication tasks, as nearly all AI-generated responses required only light editing before use.

By integrating intelligent automation into back-office workflows, this system was observed alongside how artificial intelligence can be applied meaningfully beyond instructional settings in universities. Its contribution aligns with the goals of UN Sustainable Development Goal 4: Quality Education, promoting equitable, efficient, and scalable access to support services in academic institutions. Moreover, the findings provide insights into how human–AI collaboration can be operationalised within institutional structures, maintaining a balance between efficiency and accountability.

In this regard, the system contributes to ongoing discussions on how AI can support development objectives when embedded within accountable, human-centred institutional processes rather than deployed as standalone technical solutions [43].