A Generalizable Agentic AI Pipeline for Developing Chatbots Using Small Language Models: A Case Study on Thai Student Loan Fund Services

Abstract

The rising deployment of artificial intelligence in public services is constrained by computational costs and limited domain-specific data, particularly in multilingual contexts. This study proposes a generalizable Agentic AI pipeline for developing question–answer chatbot systems using small language models (SLMs), demonstrated through a case study on the Thai Student Loan Fund (TSLF). The pipeline integrates four stages: OCR-based document digitization using Typhoon2-3B, agentic question–answer dataset construction via a clean–check–plan–generate (CCPG) workflow, parameter-efficient fine-tuning with QLoRA on Typhoon2-1B and Typhoon2-3B models, and retrieval-augmented generation (RAG) for source-grounded responses. Evaluation using BERTScore and CondBERT confirmed high semantic consistency (F_BERT = 0.9807) and stylistic reliability (F_BERT = 0.9839) of the generated QA corpus. Fine-tuning improved the 1B model’s domain alignment (F_BERT: 0.8593 → 0.8641), while RAG integration further enhanced factual grounding (F_BERT = 0.8707) and citation transparency. Cross-validation with GPT-5 and Gemini 2.5 Pro demonstrated dataset transferability and reliability. The results establish that Agentic AI combined with SLMs offers a cost-effective, interpretable, and scalable framework for automating bilingual advisory services in resource-constrained government and educational institutions.

1. Introduction

Recent advances in agentic artificial intelligence (AI) and small language models (SLMs) have reshaped the landscape of intelligent system design, enabling modular, self-improving architectures that combine reasoning and action loops with efficient parameter tuning. Unlike monolithic large language models (LLMs) that require extensive computational resources, SLMs integrated within agentic frameworks can autonomously plan, verify, and adapt tasks—achieving competitive performance with minimal infrastructure. Such characteristics are crucial for real-world deployments in low-resource or public-sector environments where cost efficiency, interpretability, and operational transparency are essential [1].

The integration of AI technologies into public-sector workflows has already demonstrated significant potential to enhance accessibility and efficiency in citizen services. Yet, many governmental and educational organizations—especially in developing nations—still rely heavily on document-centric and labor-intensive processes. The Thai Student Loan Fund (TSLF) provides a compelling example: although it has enabled millions of low-income students to access higher education, its loan-origination and servicing workflows remain largely manual, resulting in slow processing times, redundant paperwork, and frequent clerical errors. These limitations reflect broader challenges in digitizing regulated services that must respect linguistic diversity, data privacy, and infrastructure constraints.

In response to these issues, recent progress in Optical Character Recognition (OCR) and Natural Language Processing (NLP) has provided partial yet valuable solutions. Traditional OCR systems such as Tesseract [2] and modern deep-learning approaches like CRAFT [3] and DocTr [4] now enable high-fidelity digitization of complex multilingual documents. In parallel, the transformer-based NLP architecture [5] has powered large pretrained models such as BERT [6], T5 [7], and GPT-3 [8], achieving state-of-the-art performance in question–answer and summarization tasks. Nevertheless, their massive computational demands and memory footprints hinder their practical deployment in resource-constrained or institutional environments.

Open-weight models such as LLaMA [9] and RoBERTa [10] have improved accessibility for downstream adaptation, while parameter-efficient fine-tuning (PEFT) methods like LoRA [11] and QLoRA [12] further reduce computational cost without sacrificing accuracy. In addition, domain-adaptive and task-adaptive pretraining (DAPT/TAPT) [13] can substantially enhance domain transfer, as exemplified by BioBERT’s biomedical gains [14]. To improve model alignment with human intent, reinforcement learning from human feedback (RLHF) [15] and self-instruction frameworks such as Self-Instruct [16] have become effective strategies for tuning generative behavior in instruction-following models. Complementary evaluation metrics like BERTScore [17] and text detoxification approaches [18] ensure quality and stylistic safety in generated outputs.

Despite these advancements, most existing chatbot systems continue to rely on large-scale LLMs or manually curated datasets, which limit scalability, reproducibility, and affordability. Moreover, multilingual adaptation—particularly for languages such as Thai—remains underexplored due to the lack of domain-specific corpora and standardized benchmarks. Consequently, public-service chatbots often struggle with contextual grounding, factual consistency, and domain regulation compliance [19]. Retrieval-Augmented Generation (RAG) [20,21] has emerged as a robust strategy to enhance factual reliability by combining dense retrieval [22], vector search [23], and sentence embeddings [24,25], offering verifiable and source-grounded conversational responses. However, integrating these modules into a coherent, autonomous workflow remains a practical challenge for domain-specific and low-resource scenarios.

Recent research indicates that smaller, modular models coupled with Agentic AI principles—where reasoning and acting are decomposed into iterative, verifiable loops—can provide a cost-effective alternative while maintaining interpretability and robustness [1,26,27]. This direction is particularly relevant for public-sector environments, where computational budgets, privacy requirements, and operational constraints limit the feasibility of deploying large-scale foundation models [28]. In multilingual and low-resource settings, additional challenges arise from limited annotated corpora and linguistic variability, underscoring the importance of structured workflows that can reliably bootstrap high-quality training data [29].

Within this broader landscape, the CCPG (Clean–Check–Plan–Generate) workflow operationalizes agentic design through explicit task decomposition across specialized sub-agents responsible for text normalization, quality validation, structured question planning, and evidence-grounded answer generation. Unlike generic Self-Instruct pipelines or ReAct- and tool-based agents that rely heavily on large models for reasoning [30], CCPG incorporates quality gating, provenance tracking, and modular control, enabling reproducible and auditable dataset construction that aligns with regulatory requirements in government workflows.

Moreover, CCPG is intentionally optimized for resource-constrained deployment, reducing computational load during fine-tuning, minimizing annotation cost, and ensuring compatibility with the modest hardware typically available in public institutions [31,32]. When paired with retrieval-augmented generation, this structured agentic framework equips SLMs to deliver verifiable, regulation-aligned answers with minimal infrastructure. As a result, the approach establishes a practical and extensible reference model for building source-grounded chatbots in multilingual, low-resource, and compliance-sensitive governmental settings.

To address these challenges, this study aims to develop and rigorously examine a generalizable Agentic AI pipeline tailored for building question–answer (QA) chatbot systems in low-resource public-service settings. The proposed pipeline introduces a structured clean–check–plan–generate (CCPG) agentic workflow for dataset construction, combined with (1) OCR-based document digitization; (2) role-specialized and quality-gated QA generation; (3) parameter-efficient fine-tuning of small language models; and (4) retrieval-augmented, source-grounded response generation. As a representative case study, we apply the pipeline to the Thai Student Loan Fund (TSLF) to evaluate its effectiveness in a real governmental context. The investigation focuses not only on system performance but also on methodological contributions, including the conceptual advances of the CCPG workflow, the ability of SLMs to operate under resource constraints, and the practical costs of deployment. By empirically demonstrating how compact models, when guided by a structured agentic control mechanism, can produce accurate, bilingual, and regulation-grounded responses, this work establishes a reproducible and extensible foundation for developing trustworthy, multilingual chatbot systems across diverse public-service domains.

2. Materials and Methods

This section describes the end-to-end methodology for developing an advisory chatbot designed for TSLF. The proposed pipeline integrates four main components—document digitization, agentic construction of a high-quality QA dataset, parameter-efficient fine-tuning of SLMs using QLoRA, and a RAG architecture for generating verifiable and source-grounded responses.

2.1. System Overview

This section outlines the end-to-end methodology for developing the advisory chatbot system for TSLF. The proposed pipeline consists of four main stages: (1) document digitization, (2) agentic construction and screening of QA datasets, (3) parameter-efficient fine-tuning using QLoRA, and (4) deployment through a RAG-based chatbot. The overall workflow is depicted in Figure 1, and the internal coordination of agentic tasks is illustrated in Figure 2.

Figure 1. End-to-end system overview.

Figure 2. Agentic AI controller with sub-agents for text cleaning, quality assessment, question planning, and QA generation.

2.2. Stage 1: Document Digitization (OCR)

In this stage, TSLF documents are ingested as scanned images and transcribed using the Typhoon2-3B OCR engine. The system preserves hierarchical metadata at the page, paragraph, and section levels, ensuring that each extracted text segment maintains its contextual structure. This metadata enables reliable provenance tracking and verification prior to downstream processing, thereby enhancing the trustworthiness of later QA generation and model fine-tuning.

2.3. Stage 2: Agentic Pipeline for QA Construction and Screening

To move beyond monolithic self-instruct methods or one-shot QA generation, we design an agentic clean–check–plan–generate (CCPG) controller with a well-structured workflow. Instead of letting a single LLM respond to a single prompt and directly generate question–answer pairs, the controller decomposes the process into four role-specialized agents and coordinates them through explicitly defined feedback loops. This design makes iterative quality refinement a first-class component of the pipeline, rather than relying on ad hoc manual filtering as is common in many existing “self-instruct” or “RAG + fine-tuning” approaches.

• Text_Cleaner: Responsible for normalizing and restructuring the extracted text by removing headers, footers, and boilerplate content that are irrelevant. It also standardizes numerical formats, dates, and Thai-language tokens to yield text segments that are consistent, ready for downstream processing, and compliant with the formatting conventions of TSLF documents.
• Text_Quality_Checker: Evaluates candidate text segments in terms of completeness, readability, level of formality, and suitability for QA generation using a five-point scale (1–5, where 1 = very poor and 5 = excellent). In this work, we set an acceptance threshold of 3 on this scale: segments with an overall quality score below 3 are automatically routed back to the Text_Cleaner for further normalization and restructuring. This creates a clean–check feedback loop in which the same passage can be refined multiple times until it meets the quality criteria, providing a systematic and automated quality-control mechanism instead of a single-shot preprocessing step. The threshold of 3 was chosen empirically based on manual inspection of sample segments, where texts with scores below 3 frequently contained truncated sentences, OCR artefacts, or inconsistent formality that made them unsuitable for reliable QA generation, whereas segments scoring 3 or higher were generally adequate for downstream CCPG-based QA construction.
• Question_Planner: Constructs an explicit QA “blueprint” by specifying the target number of items, question types (factoid, boolean, numeric, or multiple-choice), difficulty levels (easy/medium/hard), and topical coverage over the regulatory content. When the subsequent question generation is conditioned on this blueprint, the controller can enforce diversity and coverage at a global level, thereby reducing redundancy and topical drift that commonly arise in conventional self-instruct pipelines.
• QA_Generator: Generates question–answer pairs conditioned on the blueprint and the curated text segments, while annotating rich metadata such as source provenance (line range or section ID), answer type, and content-safety flags. These metadata are later used in both the PEFT fine-tuning stage and the RAG layer to support citation-style responses and deployment in risk-aware environments.

With this systematically coordinated agentic workflow, our system is not merely “an LLM that calls tools”; it implements a closed-loop, feedback-driven QA data pipeline that (a) enforces iterative quality control through the clean–check loop, (b) optimizes QA coverage via a dedicated planning agent, and (c) preserves fine-grained provenance for use in subsequent RAG and PEFT stages. The CCPG controller, therefore, forms the core novelty of the proposed generalizable agentic pipeline: the orchestration logic is domain-agnostic and reusable, while this study instantiates and evaluates it concretely on Thai Student Loan Fund (TSLF) documentation as a focused case study.

2.4. Stage 3: QLoRA Fine-Tuning of Small Language Models

To adapt the language models to the TSLF domain, Typhoon2-1B and Typhoon2-3B checkpoints are fine-tuned using QLoRA on moderate hardware resources. The base model weights are quantized into the NF4 (4-bit) format, while only the LoRA adapters inserted into key transformer layers are trainable.

The dataset used for training consists exclusively of QA pairs generated in Stage 2, divided approximately into 80% training, 10% validation, and 10% testing subsets. Fine-tuning employs causal language modeling (CLM) over prompt–answer pairs, with maximum sequence length = 2048, initial learning rate = 0.0001, and fixed random seed = 42 for reproducibility. Early stopping is applied based on validation loss to prevent overfitting under VRAM constraints. This approach ensures efficient domain adaptation while maintaining model compactness and stability.

2.5. Stage 4: RAG-Based Chatbot Deployment

In the final stage, a RAG architecture is deployed to provide verifiable and bilingual conversational responses. TSLF regulatory documents are segmented into chunks of approximately 1000–1500 tokens with 10–15% overlap. Each segment is cleaned and embedded using the BGE-M3 multilingual encoder. The resulting vector representations are indexed using FAISS for high-speed retrieval.

During inference, the system retrieves the top-k = 2 most relevant document chunks, constructs a contextual prompt window, and queries the QLoRA-tuned model to synthesize an answer in a formal and regulation-aligned tone. Each response includes explicit citations to the corresponding document spans to ensure transparency and traceability.

The serving stack consists of a Next.js + Tailwind + shadcn/ui frontend connected to a FastAPI backend that integrates FAISS indexing and the model engine (Ollama/vLLM) behind Nginx for scalability and production readiness. This architecture ensures low-latency inference, secure access, and reproducible deployment in institutional environments.

3. Results

This section presents both quantitative and qualitative evaluation results. The analysis proceeds from the quality assessment of the agentically generated QA dataset to the fine-tuning outcomes of the 1B and 3B SLMs, followed by an examination of the incremental impact of the RAG component on factual accuracy. Finally, overall system-level indicators—including latency and resource utilization—are reported to demonstrate the feasibility of the proposed pipeline in real-world public-service environments.

3.1. Quality of the Agentically Generated QA Dataset

The agentically constructed QA dataset was evaluated using BERTScore for semantic consistency and CondBERT for stylistic and safety compliance. Both metrics were computed in terms of precision (P), recall (R), and F1-score (F) across the aggregated corpus.

Table 1 summarizes the results, indicating that the generated QA pairs achieved near-perfect semantic alignment (F_BERT = 0.9807) and stylistic conformance (F_CondBERT = 0.9839). These high scores demonstrate the reliability of the proposed CCPG workflow in producing linguistically coherent and contextually faithful question–answer pairs suitable for downstream fine-tuning.

Table 1. QA dataset quality (semantic vs. style/safety).

Metric	PBERT	RBERT	FBERT	PCondBERT	RCondBERT	FCondBERT
Score	0.9866	0.9748	0.9807	0.9916	0.9763	0.9839

These results indicate high semantic consistency (F_BERT = 0.9807) together with strong stylistic and safety adherence (F_CondBERT = 0.9839), confirming the effectiveness of the CCPG loop embedded in the agentic workflow. The close alignment across both semantic and stylistic dimensions demonstrates that the proposed QA-generation process can produce data of sufficient quality for downstream model adaptation.

To further evaluate the reliability of the Agentic AI–generated dataset, we conducted a cross-model validation using two large foundation models, ChatGPT-5.1 and Gemini 2.5 Pro. In this setting, both models acted as independent QA baselines, rather than as “LLM judges” directly scoring our corpus. We first randomly sampled 20 QA pairs from the CCPG-generated corpus, stratified by question type and difficulty. For each sampled item, the question and its associated context passage were submitted to ChatGPT-5 and Gemini using a shared, standardized prompt template. The system message explicitly instructed the model to answer strictly based on the provided TSLF passage, to avoid using any prior knowledge or external resources.

Answers generated by both large models were then evaluated against expert-validated TSLF reference answers, rather than against the Agentic AI outputs themselves. The evaluation pipeline was implemented using Python-based software libraries, including TSLF and Agentic AI modules developed in-house, with semantic similarity assessed using BERTScore (implemente d via the transformers library, version ≥ 4.43.0) and stylistic and safety conformity evaluated using CondBERT (based on transformers ≥ 4.43.0 and torch ≥ 2.2.0). The experimental framework additionally relied on pandas (≥2.0), numpy (≥1.24), tqdm (≥4.66), faiss-cpu (≥1.8.0), sentence-transformers (≥3.0.0), langchain and langchain-community (≥0.2.12), and ollama (≥0.3.0). The quantitative outcomes of this cross-model baseline experiment are summarized in Table 2.

Table 2. Comparison of ChatGPT-5 and Gemini 2.5 Pro answers on 20 random QA pairs from Agentic AI.

Model	PBERT	RBERT	FBERT	PCondBERT	RCondBERT	FCondBERT
Chat GPT-5.1	0.9225	0.9409	0.9312	0.9989	0.9989	0.9989
Gemini 2.5 Pro	0.9219	0.9358	0.9284	0.9988	0.9988	0.9988

The evaluation reveals that both ChatGPT-5 and Gemini 2.5 Pro achieve consistently high stylistic conformity (F_CondBERT ≈ 0.99) and strong semantic agreement (F_BERT ≈ 0.93) with the expert-verified TSLF references. These findings indicate that the QA pairs generated by the Agentic AI CCPG pipeline are readily interpretable and answerable by external architectures without significant semantic drift, reinforcing both the reliability and cross-model transferability of the proposed automatic QA synthesis mechanism.

3.2. Model Performance After Fine-Tuning (1B vs. 3B)

Learning convergence for both the 1B and 3B small language models was analyzed during QLoRA fine-tuning. As illustrated in Figure 3, the training and validation losses exhibit a consistent downward trend across steps, reflecting effective optimization and stable convergence under constrained GPU memory conditions. On a single NVIDIA A100 GPU environment on Google Colab, the Typhoon2-3B model required 10:22:21 (hh:mm:ss) of wall-clock time to complete two training epochs, whereas the Typhoon2-1B model finished the same two-epoch schedule in approximately 3 h. This gap highlights the expected trade-off between model size and fine-tuning time under identical hardware and optimization settings.

Figure 3. Training and validation loss curves for the Typhoon2-based 3B and 1B models fine-tuned with QLoRA. The 3B model (left) corresponds to scb10x/llama3.2-typhoon2-t1-3b-research-preview, while the 1B model (right) corresponds to scb10x/llama3.2-typhoon2-1b-instruct.

After the initial warm-up phase, the validation loss plateaus smoothly, confirming that the QLoRA adapters achieved stable learning dynamics without overfitting. Importantly, performing only two epochs of fine-tuning is standard practice for large language models, as pretrained LLMs already possess strong foundational representations; thus, small numbers of epochs are typically sufficient for effective task adaptation while avoiding unnecessary computational cost or potential overfitting.

From Figure, the 3B model (left panel) converged faster and reached a slightly lower final validation loss compared with the 1B model (right panel), suggesting that moderate model scaling can enhance representation learning even under low-precision (NF4) quantization. Both models demonstrated steady learning behavior and reproducibility across runs with the same random seed, validating the robustness of the fine-tuning procedure within the proposed agentic pipeline.

To evaluate end-task QA performance, a domain-specific benchmark was constructed by posing representative questions to TSLF domain experts and collecting their validated responses. These expert-annotated answers served as reference ground truth for model evaluation. The outputs of both the base and fine-tuned versions of the Typhoon2 1B and 3B models were compared against this benchmark using BERTScore (semantic alignment) and CondBERT (stylistic and safety conformance). Each metric was reported in terms of precision (P), recall (R), and F1-score (F). The results are summarized in Table 3.

Table 3. QA quality before and after fine-tuning, benchmarked against expert-annotated answers.

Model	PBERT	RBERT	FBERT	PCondBERT	RCondBERT	FCondBERT
typhoon2-t1-3b	0.8460	0.8606	0.8531	0.8085	0.8384	0.8229
typhoon2-t1-3b Finetune	0.8419	0.8624	0.8518	0.8039	0.8399	0.8210
typhoon2-1b-	0.8819	0.8381	0.8593	0.8452	0.8223	0.8331
typhoon2-1b- Finetune	0.8823	0.8471	0.8641	0.8423	0.8278	0.8345

Overall, fine-tuning led to modest but consistent improvements in both semantic fidelity and stylistic consistency across model sizes. The 1B model achieved a slight gain in F_BERT (from 0.8593 to 0.8641) and maintained stylistic stability, while the 3B model exhibited stable recall with minimal degradation in precision—consistent with its larger capacity and quantized representation. These results suggest that even lightweight SLMs, when trained with high-quality agentic QA data, can attain strong domain adaptation and semantic robustness comparable to larger models.

Observations

1.

Performance Gains in the 1B Model:

The 1B model demonstrated measurable improvements after fine-tuning, achieving higher semantic fidelity (F_BERT: 0.8593 → 0.8641) and stronger style/safety conformance (F_CondBERT: 0.8331 → 0.8345) when evaluated against expert-validated answers. These gains confirm the effectiveness of domain adaptation through QLoRA, highlighting that lightweight small language models can substantially benefit from targeted fine-tuning using high-quality agentic QA data.

2.

Stable but Slight Regression in the 3B Model:

In contrast, the 3B model exhibited minor declines following fine-tuning (F_BERT: 0.8531 → 0.8518; F_CondBERT: 0.8229 → 0.8210). This small regression may be attributed to domain over-specialization or catastrophic forgetting, where the model adapts too narrowly to the stylistic and structural patterns of the training QA corpus. The phenomenon suggests that large models, when trained on limited-domain data, may require broader exposure to maintain general linguistic robustness.

3.

Key Takeaways:

Fine-tuning provides clear and consistent advantages for smaller models (1B) under compute-constrained environments, offering efficient and reliable domain adaptation. For larger models (3B), sustaining broader linguistic competence and stability may demand more diverse expert QA datasets, lower learning rates, or regularized training regimes such as dropout, weight decay, or mixed-domain fine-tuning that balances domain specificity with generalization.

3.3. Qualitative Evaluation

To assess qualitative performance in a practical context, we examined model responses to a regulatory question from the TSLF as follows:

Question. “ชั่วโมงจิตสาธารณะ กยศ. ต้องเก็บอย่างน้อยกี่ชั่วโมง” (How many hours of community service are required under the TSLF regulations?)

Expert Answer. ผู้กู้รายเก่าที่เลื่อนชั้นปี ต้องเก็บชั่วโมงจิตอาสาไม่น้อยกว่า **36 ชั่วโมงต่อปีการศึกษา** เพื่อขอกู้ในปีการศึกษาถัดไป ส่วนผู้กู้รายใหม่ หรือผู้กู้ย้ายสถานศึกษา ไม่มีกำหนดชั่วโมงตายตัว แต่ต้องเข้าร่วมกิจกรรมจิตอาสาที่เป็นประโยชน์ต่อสังคม (Continuing borrowers who advance to the next academic year are required to complete at least 36 h of community service per academic year in order to be eligible for funding in the following year. New borrowers or borrowers who transfer to a different institution are not subject to a fixed minimum number of hours but must participate in socially beneficial volunteer activities.)

The model outputs are summarized in Table 4.

Table 4. Comparison of model outputs and expert reference.

Model	Generated Answer
Typhoon2-1B	ต้องเก็บชั่วโมงจิตสาธารณะอย่างน้อย 120 ชั่วโมงต่อปี (At least 120 h of community service must be completed per year.)
Typhoon2-1B Finetune	ต้องไม่น้อยกว่า 36 ชั่วโมงต่อปีการศึกษา (Not fewer than 36 h per academic year.)
Typhoon2-3B	ต้องทำชั่วโมงจิตสาธารณะตามเงื่อนไขของกองทุน ซึ่งอาจแตกต่างกันไป (Community service hours must be completed in accordance with the Fund’s requirements, which may vary depending on the applicable conditions.)
Typhoon2-3B Finetune	[1] (source: ชั่วโมงจิตสาธารณะ กยศ.pdf|chunk:0) (TSLF Community Service Hours.pdf) ผู้กู้ กยศ. ที่เป็นนักศึกษารายเก่าที่เลื่อนชั้นปี ต้องเก็บชั่วโมงจิตอาสาอย่างน้อย 36 ชั่วโมงต่อปีการศึกษา เพื่อขอกู้ในปีการศึกษาถัดไป ส่วนผู้กู้รายใหม่ หรือผู้กู้ยืมรายเก่าที่ย้ายมาจากสถานศึกษาอื่น จะไม่มีการกำหนดจำนวนชั่วโมง แต่ให้ทำกิจกรรมจิตอาสาที่เป็นประโยชน์ต่อสังคม (TSLF borrowers who are continuing students and progressing to the next academic year are required to complete at least 36 h of community service per academic year in order to remain eligible for funding in the following year. New borrowers, as well as previous borrowers transferring from another institution, are not subject to a fixed minimum number of hours but must participate in volunteer activities that provide social benefit.) [2] (source: QA_คู่มือสำหรับสถานศึกษาในการดำเนินงาน2561.pdf|chunk:33) (QA_Manual for Educational Institutions on Operational Procedures (2018).pdf) สาธารณะที่น่าเชื่อถือ ตามจำนวนชั่วโมงที่กำหนดสำหรับผู้ขอกู้ยืมเงินแต่ละกลุ่มดังต่อไปนี้ (ก) กรณีเป็นผู้กู้ยืมเงินรายใหม่ หรือเป็นผู้กู้ยืมเงินรายเก่าเปลี่ยนระดับการศึกษาที่ศึกษาในระดับอนุปริญญาหรือปริญญาตรี ไม่มีการกำหนดจำนวนชั่วโมง (ข) กรณีเป็นผู้กู้ยืมเงินรายเก่าเลื่อนชั้นปีทุกระดับการศึกษาไม่น้อยกว่า 36 ชั่วโมง (Community service must be completed in accordance with verifiable public-benefit activities, with the required number of hours depending on the borrower category as follows: (a) For new borrowers or continuing borrowers who change their level of study and are enrolled in diploma or bachelor’s degree programs, no minimum number of hours is specified. (b) For continuing borrowers who progress to the next academic year at any level of study, a minimum of 36 h of community service is required.)

This case study highlights clear differences between base and fine-tuned models

(i)

1B base model produced a substantially incorrect requirement (120 h), reflecting a lack of domain alignment.

(ii)

1B fine-tuned model matched the expert reference (36 h), confirming the effectiveness of QLoRA fine-tuning for small models under constrained resources.

(iii)

3B base model provided only general guidance without specifying the required hours.

(iv)

3B fine-tuned model generated the precise requirement (36 h), showing improved domain grounding after adaptation.

This qualitative evaluation aligns with the quantitative findings (Table 4), demonstrating that fine-tuning yields tangible benefits for smaller models (1B), while also enhancing factual alignment for larger models (3B). The ability to replicate expert-validated answers underscores the practical utility of fine-tuned SLMs for TSLF advisory services.

3.4. Performance of RAG-Integrated Models

To further evaluate the effect of retrieval grounding, both the base and fine-tuned variants of the Typhoon2-1B model were integrated with an RAG module. The RAG mechanism enables the model to retrieve relevant document chunks prior to answer generation, thereby enhancing factual grounding and traceability. Quantitative results for the RAG-integrated models are summarized in Table 5.

Table 5. Quantitative results of RAG-integrated models.

Model	PBERT	RBERT	FBERT	PCondBERT	RCondBERT	FCondBERT
typhoon2-1b—RAG	0.8842	0.8501	0.8655	0.8470	0.8303	0.8384
typhoon2-1b—Finetune RAG	0.8869	0.8576	0.8707	0.8540	0.8351	0.8422

Integration of the RAG module yielded consistent improvements in semantic alignment and stylistic stability across both configurations. The fine-tuned RAG model achieved a higher F_BERT (0.8707 vs. 0.8655) and F_CondBERT (0.8422 vs. 0.8384) compared with the non-fine-tuned variant, confirming that retrieval grounding complements domain adaptation by reinforcing factual consistency.

The observed performance gain indicates that RAG effectively reduces hallucination and enhances factual recall, allowing the model to generate more accurate, source-grounded responses even under constrained model capacity. These results further demonstrate that the Agentic AI + SLMs + RAG (ASR) pipeline provides a practical and reliable framework for domain-specific chatbot systems where explainability and trustworthiness are essential.

To complement the quantitative findings presented earlier, qualitative inspection was conducted to assess how well the RAG-integrated models handle Thai-language regulatory queries in realistic advisory scenarios. This evaluation focused on the model’s ability to retrieve the correct context, maintain factual accuracy, and generate answers consistent with the official TSLF regulations.

The example below illustrates one of the test cases drawn from the expert-annotated benchmark. Both the question and the reference answer were provided in Thai to evaluate native-language comprehension and formal response generation by the model.

Question: “ชั่วโมงจิตสาธารณะ กยศ. ต้องเก็บอย่างน้อยกี่ชั่วโมง” (How many hours of community service are required under the TSLF regulations?)

Expert Answer: ผู้กู้รายเก่าที่เลื่อนชั้นปี ต้องเก็บชั่วโมงจิตอาสาไม่น้อยกว่า 36 ชั่วโมงต่อปีการศึกษา เพื่อขอกู้ในปีถัดไป ส่วนผู้กู้รายใหม่หรือย้ายสถานศึกษาไม่มีกำหนดชั่วโมงตายตัว (Continuing borrowers who progress to the next academic year are required to complete at least 36 h of community service per academic year in order to remain eligible for funding in the following year. New borrowers or those transferring from another institution are not subject to a fixed minimum number of community service hours).

Before illustrating the system outputs, Table 6 summarizes representative responses generated by the Typhoon2-1B RAG and Typhoon2-1B Finetune RAG models for the same question.

Table 6. Example responses of RAG-integrated models to TSLF regulatory question.

Model	Generated Answer
Typhoon2-1B RAG	ต้องเก็บชั่วโมงจิตสาธารณะอย่างน้อย 36 ชั่วโมงต่อปีการศึกษา สำหรับผู้กู้รายเก่าที่เลื่อนชั้นปี ส่วนผู้กู้รายใหม่หรือย้ายสถานศึกษา ไม่มีกำหนดชั่วโมงที่แน่นอน (At least 36 h of community service per academic year are required for continuing borrowers who advance to the next academic year. New borrowers or those transferring from another institution are not subject to a fixed minimum number of hours.)
Typhoon2-1B Finetune RAG	ต้องไม่น้อยกว่า 36 ชั่วโมง Reference: ชั่วโมงจิตสาธารณะ กยศ.pdf (Not less than 36 h. Reference: TSLF Community Service Hours.pdf)

The qualitative results confirm that both RAG-enabled models successfully retrieved the relevant regulatory content and generated contextually appropriate Thai-language answers. However, the fine-tuned RAG model demonstrated superior factual grounding by explicitly citing its document source (“ชั่วโมงจิตสาธารณะ_กยศ.pdf”) (TSLF Community Service Hours.pdf), reflecting stronger integration between retrieval and generation.

These findings indicate that the combination of fine-tuning and retrieval grounding substantially enhances factual reliability and interpretability in Thai regulatory contexts. Consequently, the proposed ASR framework demonstrates strong potential for deployment in TSLF advisory systems and other government-facing chatbot applications requiring bilingual accuracy, traceability, and compliance transparency.

3.5. Expert Human Evaluation

The results of a human evaluation conducted by domain experts from the Thai Student Loan Fund (TSLF) on 17 representative questions. For each question, answers from the Typhoon2-1B RAG model and the Typhoon2-1B Fine-tuned RAG model were rated along two dimensions—answer accuracy and practical usability—on a 10-point scale, see in Table 7.

Table 7. Domain-expert evaluation of answer accuracy and usability.

Model	Accuracy Score	Usability Score
Typhoon2-1B RAG	0.6765	0.6765
Typhoon2-1B Finetune RAG	0.7647	0.7647

The fine-tuned RAG model achieved higher scores on both dimensions, with average accuracy and usability scores increasing from 0.6765 to 0.7647. These improvements indicate that the proposed agentic AI pipeline, which automatically generates, cleans, and filters QA pairs, is able to produce training data of sufficiently high quality to yield responses that experts perceive as more factually reliable and more useful for real borrower interactions, thereby strengthening the chatbot’s suitability for deployment in a public-service context.

4. Discussion

This study highlights both the strengths and limitations of the developed Agentic AI-based advisory chatbot system for the Thai Student Loan Fund (TSLF). The discussion below integrates empirical findings with conceptual insights, emphasizing the system’s data quality, model behavior, retrieval effectiveness, and deployment implications.

A.

Quality of the QA Dataset

Evaluation using P_BERT (0.9807) and F_CondBERT (0.9839) confirms that the proposed agentic workflow—comprising the CCPG loop—can automatically construct QA pairs that are both semantically faithful and stylistically consistent with official TSLF documentation. The decomposition of data preparation into role-specialized agents (cleaner, checker, planner, and generator) effectively reduces noise, resolves redundancy, and improves contextual coherence across the dataset. This structured collaboration among agents demonstrates that task modularization in Agentic AI can yield dataset quality comparable to manually curated corpora while maintaining scalability for other domains.

Beyond the empirical observations described above, an important methodological contribution of this work lies in the design of the CCPG (clean–check–plan–generate) agentic workflow, which introduces structural capabilities not present in existing instruction-generation or tool-augmented LLM frameworks such as Self-Instruct, ReAct/tool-using agents, or RAG-classic pipelines. These conceptual advances help explain why the proposed pipeline is particularly effective for Thai regulatory content and small model adaptation. First, unlike Self-Instruct—which relies on a single-model iterative prompt–refinement cycle—the CCPG workflow employs explicit multi-agent role specialization, separating the data-generation process into four coordinated agents: cleaner, checker, planner, and generator. This modular decomposition prevents noise propagation, enforces linguistic consistency in Thai text, and systematically improves QA quality by routing low-quality segments back through the refinement loop. Second, while ReAct-style agents interleave reasoning with dynamic tool calls, they do not enforce predictable or quality-controlled process flows. In contrast, CCPG adopts a graph-based controller that governs deterministic transitions between stages (clean → check → plan → generate), ensuring reproducibility and reducing variance in dataset quality—an important consideration for SLMs, which are more sensitive to data noise than larger LLMs. Third, RAG-classic frameworks apply retrieval at inference time but do not constrain how training QA corpora are constructed. CCPG introduces a domain-governed planning mechanism that explicitly encodes regulatory categories, coverage requirements, question difficulty, and answer format constraints. This ensures that the resulting QA dataset reflects the actual informational structure, obligations, and exceptions found in SLF regulations, thereby reducing ambiguity in downstream fine-tuning.

Finally, CCPG is deliberately optimized for small-model adaptation under resource constraints. The workflow produces compact, high-quality training signals tailored to QLoRA fine-tuning, minimizing catastrophic forgetting and enhancing domain adherence. This design choice aligns with the practical realities of public-sector deployment, where access to large-scale models and compute resources is limited. Collectively, these conceptual advances distinguish CCPG from more generic instruction-generation or retrieval-augmented systems and provide a foundation for a generalizable, controllable, and reproducible agentic pipeline suitable for multilingual, document-intensive government workflows.

B.

Model Performance After Fine-Tuning

After fine-tuning with QLoRA, the Typhoon2-1B model achieved measurable gains across both semantic and stylistic metrics, underscoring the suitability of parameter-efficient training for small models under limited computational resources. These improvements illustrate that compact architectures can retain strong representational power when supported by high-quality, domain-adaptive QA data. Conversely, the Typhoon2-3B model exhibited marginal degradation in both BERTScore and CondBERT metrics after fine-tuning. This decline is likely due to over-specialization and partial catastrophic forgetting, arising from the narrow domain coverage of the training corpus. Future optimization for larger models should incorporate more diverse and heterogeneous datasets and apply regularization strategies—such as dropout, weight decay, or mixed-domain training—to preserve general linguistic competence while enhancing domain precision.

C.

Effectiveness of the RAG Architecture

The integration of fine-tuned models with RAG significantly enhanced factual reliability by grounding responses in verifiable document sources. This improvement, reflected in both semantic alignment (FBERT = 0.8707) and style consistency (FCondBERT = 0.8422), demonstrates the practical strength of retrieval grounding for regulatory and evidence-based applications. Importantly, RAG mitigates hallucination tendencies and ensures traceable, citation-driven responses, making it a robust mechanism for use in government or legal information systems where trust, auditability, and transparency are mission-critical.

D.

Practical Implications

The developed system demonstrates clear benefits for automating document-intensive workflows within the TSLF, reducing manual effort and improving the responsiveness of public information services. By integrating OCR, agentic data generation, QLoRA fine-tuning, and RAG retrieval into a unified pipeline, the framework supports bilingual Thai–English interaction and maintains alignment with official TSLF policy texts. For real-world deployment, however, continuous dataset maintenance and model retraining are essential to accommodate evolving regulations, linguistic variations, and user intents. Future implementations should also integrate user feedback loops and adaptive retraining schedules to sustain long-term system reliability and user satisfaction.

E.

Limitations and Future Directions

• Domain Narrowness: The current QA corpus is centered on TSLF rules and procedures, which may not adequately capture diverse, context-dependent borrower inquiries. Expanding the dataset to include real user interactions and frequently asked questions would enhance generalizability and conversational coverage.
• Model Forgetting: Larger models exhibited partial loss of general linguistic capabilities after fine-tuning, suggesting that catastrophic forgetting remains a challenge under domain-specific training. Future research should explore hybrid fine-tuning strategies that blend domain and general-purpose corpora, possibly using multi-objective optimization or continual learning frameworks to sustain balanced language understanding.

Overall, the findings highlight that Agentic AI combined with SLMs offers a cost-effective and interpretable pathway for building trustworthy, bilingual chatbot systems in public-sector settings. While further domain expansion and optimization are needed, the demonstrated ASR framework provides a replicable blueprint for intelligent automation in other government and educational service contexts.

5. Conclusions

This study presented an end-to-end advisory chatbot framework for the Thai Student Loan Fund (TSLF) that integrates four major components: OCR-based document digitization, Agentic AI–driven QA dataset generation, parameter-efficient fine-tuning using QLoRA, and retrieval-augmented generation (RAG) for verifiable responses.

Experimental evaluations demonstrated the effectiveness of the proposed approach, particularly in optimizing SLMs under constrained computational environments while maintaining semantic precision and factual reliability.

The key contributions of this study can be summarized as follows:

1.

Comprehensive automation pipeline.

A practical and reproducible blueprint that unifies OCR, Agentic AI, SLMs fine-tuning, and RAG retrieval for automating document-intensive workflows in public-sector contexts.

2.

High-quality QA dataset construction.

The CCPG agentic loop produced QA pairs with strong semantic alignment (FBERT = 0.9807) and stylistic reliability (FCondBERT = 0.9839), ensuring robust training data for domain adaptation.

3.

Model adaptation under resource constraints.

Fine-tuning via QLoRA significantly improved the performance of the 1B model, demonstrating that compact architectures can achieve domain-level accuracy while remaining computationally efficient.

4.

Trustworthy and interpretable chatbot responses.

The integration of RAG enabled the system to deliver source-grounded and transparent answers, an essential capability for government and educational advisory systems.

5.

Actionable insights for future research.

The study identified challenges such as domain narrowness and model forgetting in larger architectures, suggesting future directions involving continual learning, data diversification, and mixed-domain fine-tuning to enhance long-term adaptability.

In practice, the proposed system streamlines administrative workflows, reduces manual verification burdens, and improves accessibility to financial aid information for TSLF stakeholders. Beyond this case study, the ASR pipeline serves as a scalable, interpretable, and cost-effective framework for developing advisory chatbot systems across broader public-sector, educational, and multilingual service domains.