A Criteria-Governed Human-in-the-Loop Information System Architecture for Explainable AI-Assisted Learning and Preliminary Assessment

Abstract

This paper presents a criteria-governed, human-in-the-loop information system architecture integrating formative AI-assisted learning with preliminary assessment of spoken and written responses. It supports two modes: a retrieval-augmented assistant answering from course materials and an AI examiner scoring responses against a predefined rubric, criterion by criterion. The language model is confined to analysis: evidence extraction, aggregation, input-quality control, uncertainty estimation and escalation are performed by a controlled pipeline, and each result is stored as a structured decision object carrying criteria, evidence, provenance, uncertainty and teacher-verification status. Across 168 examination sessions and 1074 criterion-level decisions, every decision carried a verbatim evidence passage (100% coverage), the overall score matched the programmatic aggregation in all finalized sessions, 39.1% of responses were routed for secondary analysis and 5.4% of sessions were escalated. A survey of 52 students rated question clarity 4.81, criterion-level feedback 4.46, and allotted time 4.42, while concerns persisted about scoring reliability and recognition errors. In a preliminary blind calibration, the AI ranked 13 spoken answers like human raters (r = 0.75), but scored them 22 points lower. The contribution is the architecture and its feasibility in deployment; the study does not establish automated-scoring accuracy, and final decisions remain with the teacher.

1. Introduction

The use of artificial intelligence in higher education is increasingly moving beyond reference and consultation functions toward direct support for teaching and learning. Current applications include student recommendations, explanations of learning content, identification of learning difficulties, preliminary response analysis, preparation of assessment judgments, and feedback generation. Systematic reviews of AI in higher education typically group these applications under personalized learning, intelligent tutoring systems, prediction, student support, assessment, and feedback [1]. Educational RAG systems, which connect a language model to a controlled corpus of learning materials, represent a particularly fast-growing strand of this work. A recent review of educational RAG chatbots found that they are being used to support learning, provide access to domain knowledge, advise students, and assist instructors, although their evaluation procedures and human-oversight arrangements remain heterogeneous [2].

These applications nevertheless entail different levels of risk. In a learning-support setting, an AI error can be corrected through dialogue; in an assessment setting, the same error may affect a student’s academic trajectory [3]. Research on AI-based assessment therefore emphasizes not only potential gains in efficiency and scalability but also threats to validity, algorithmic bias, transparency, and fairness [4].

Assessment applications consequently require a more stringent control architecture incorporating human oversight, explainability, accountability, and pedagogical validity, because the assessment of learning outcomes is among the higher-risk uses of AI in education [5,6].

The central scientific and engineering challenge is thus not merely to use a language model to generate an answer or score, but to design a governable mechanism for producing and justifying decisions. Such a mechanism must establish traceable links among the input data, formalized criteria, computed features, supporting evidence, estimated uncertainty, and the teacher’s final decision.

This study proposes an intelligent educational system in which two modes operate within a common software and algorithmic pipeline. The AI assistant processes student queries and course materials through a RAG workflow, whereas the AI examiner analyzes student responses against an assessment rubric using criterion-based natural language processing (NLP). Both modes share mechanisms for formalizing criteria, retrieving supporting evidence, estimating output uncertainty, logging system activity, and enabling teacher verification.

Existing research tends to address educational RAG assistants, automated assessment, explainable AI, and human-control mechanisms as separate or only partly integrated areas [2,7,8]. Prior studies have proposed institutional frameworks for generative AI in assessment, local RAG assistants, and retrieval-augmented automated assessment systems. Far less attention, however, has been paid to software and algorithmic architectures in which formative learning support and preliminary assessment function as two modes of a single information system and share a common decision object, supporting evidence, explicit uncertainty estimates, audit logs, and recorded teacher verification. Integrating these components and formalizing the lifecycle of a preliminary decision constitute the research gap addressed in this study.

2. Literature Review and Problem Statement

2.1. Review Questions and Search Protocol

The positioning claims made in this paper require a fuller account of prior work than the previous version of this section provided. We therefore conducted a structured, protocol-driven review organised around four questions, each corresponding to one property that the proposed architecture asserts.

RQ1. How do retrieval-augmented systems in higher education ground their output in course materials, and how is that grounding measured?

RQ2. When language models score student responses against analytic rubrics, what computes the final score, and is criterion-level evidence a mandatory part of the output?

RQ3. How is uncertainty represented in language-model outputs, and under what conditions does a system abstain instead of deciding?

RQ4. How is human oversight of AI-assisted assessment formalised, and is teacher confirmation a recorded state in the lifecycle of a decision?

Searches were run against OpenAlex over the title and abstract fields for the window from 1 January 2023 to 1 October 2026. The search strings are reproduced verbatim below.

RQ1: (“retrieval-augmented generation” OR “retrieval augmented generation”) AND (education OR university OR “higher education”) AND (grounded OR citation OR evidence).

RQ2: (“large language model” OR “large language models” OR LLM) AND (rubric OR “analytic scoring” OR “criterion-level”) AND (grading OR scoring OR assessment).

RQ3: (“large language model” OR “large language models” OR LLM) AND (“uncertainty estimation” OR “selective prediction” OR abstention OR “confidence calibration”).

RQ4: (“human-in-the-loop” OR “human oversight”) AND (grading OR assessment OR scoring) AND (AI OR “language model” OR automated).

Selection proceeded in four steps, summarised in Figure 1, which follows the PRISMA 2020 flow-diagram template [9]. First, the four searches, run on 1 October 2026, identified 592, 2548, 1804 and 5798 records respectively, 10,742 in total; overlap between questions was not removed at this stage. Second, the records returned for each question were ranked by OpenAlex relevance, and the fifty highest-ranked records per question, 200 in total, were screened by title and abstract against the eligibility criteria stated below; the remaining 10,542 records were not screened. Third, because the architectural properties examined here are seldom named in titles or abstracts, the screened set was supplemented by targeted searching within each theme and by backward citation tracking from the records that passed screening. Reports from both routes were assessed in full text against the same criteria, which yielded 43 eligible studies. Fourth, a second author reviewed the list of eligible studies, and the five studies judged least relevant to the four review questions were removed. In total, 38 primary studies were retained: 5 had already been cited in the previous version of this manuscript and 33 are newly added [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42]. The number of records excluded at title-and-abstract screening and the number of included studies contributed by each route were not recorded separately, and Figure 1 therefore reports the two routes jointly from the full-text stage onwards. Because the OpenAlex index is updated continuously, re-running the same queries at a later date returns different counts. All but one of the retained studies carry a digital object identifier that we verified against the registering agency; the exception is a proceedings paper indexed without a DOI [42].

Figure 1. Flow of records through identification, screening, eligibility assessment and inclusion in the structured review (adapted from the PRISMA 2020 flow diagram).

Studies were included if they were published between 2023 and 2026; were peer-reviewed articles, conference papers or preprints with a verifiable identifier; addressed language-model-based assistance or assessment in an educational or evaluative setting, or addressed uncertainty estimation and abstention in language models as methods applicable to such settings; and reported sufficient architectural or methodological detail for the four dimensions above to be determined. Studies were excluded if they discussed generative AI in education only at the level of policy or opinion without a system or a method, if the object of assessment was not a student response, or if they duplicated an already retained record.

This is a structured review of a bounded corpus and not a registered systematic review. The protocol was not pre-registered, the full set of 10,742 identified records was not screened exhaustively, and selection within each theme continued until additional records ceased to introduce architectural arrangements not already represented. Screening was not duplicated by an independent second screener; the list of retained studies was reviewed by a second author. The review is therefore sufficient to position the contribution of this paper, but it should not be read as an exhaustive mapping of the field.

2.2. Grounding of Retrieval-Augmented Assistance (RQ1)

Retrieval-augmented generation is the standard means of binding an educational assistant to a defined corpus, and recent surveys classify such systems by audience, knowledge base and underlying model [12]. In deployed educational studies, however, grounding is evaluated after the fact rather than enforced. A pilot across four courses at two universities had experts code a random sample of conversations and found 1.5% of answers incorrect and 16.5% outside the supplied context [10]; a medical teaching assistant restricted to teacher-curated material reports usage patterns and student trust rather than the share of statements traceable to a source [11]; In each case the system answers even when support is weak, and citation is a design intention rather than a precondition of output.

The measurement of grounding is more developed outside education. Benchmarks require a model to cite retrieved passages and score citation precision and recall [13]; reference-free frameworks decompose an answer into claims and compute the proportion entailed by the retrieved context; trustworthiness has been attributed to the grounding of individual statements [14]; and counterfactual methods bound the risk that a retrieval-augmented answer is unsupported [16]. These methods establish that grounding can be quantified automatically, but they quantify it for an answer that has already been produced.

The distinction that matters for the present work is therefore not whether grounding is measured but when. In the architecture proposed here, a criterion-level judgment is not emitted unless a verbatim passage from the student’s own response is attached to it, so grounding is a condition of release rather than a property estimated afterwards on a sample.

2.3. Criterion-Level Scoring of Student Responses (RQ2)

Automated assessment has moved from a single holistic score toward rubric-aligned, trait-specific judgments, and large models are now reported as graders of student writing and of open-ended responses in university settings [43,44]. Recent systems decompose the rubric and score each criterion separately [18,19,21] and are evaluated explicitly for criterion-level adaptation [18]. A group of very recent systems anchors each rubric point in evidence drawn from the response [17,20,35], which is the closest published position to our own, and one of them audits whether the credit awarded actually responds to the rubric point it claims to address [35].

The decisive dimension, however, is what computes the final number. In most retained work the language model produces the score itself, whether directly or after an internal critique step [21,26], and the rubric shapes the prompt rather than the arithmetic. Reported agreement with human graders is then a property of a particular model and prompt, and it changes when either changes. In the architecture proposed here the model is confined to analysis, and the aggregate is computed by a deterministic weighted function outside it (Section 3.6): identical criterion scores always yield an identical total, and the total can be recomputed and audited without re-invoking the model. Section 3.11.5 sets out this comparison against direct model scoring and classical feature-based scoring in detail.

2.4. Representation of Uncertainty and Abstention (RQ3)

A substantial body of method work now addresses when a language model should decline to answer. Surveys map abstention across the model lifecycle [15] and review uncertainty quantification both for deep learning generally [45] and for large models specifically [27]; semantic entropy detects confabulation by measuring disagreement among sampled meanings [28]; conformal prediction has been extended to adaptive abstention policies [31]; and the relationship between model confidence and the confidence users attribute to the model has itself been measured [29].

Within grading, this line of work appears as deferral and escalation. Systems learn when to defer a short answer to a human [24], estimate confidence in automatic short-answer grading [23], combine self-consistency with selective prediction [32], calibrate a human-in-the-loop short-answer grader [22], and provide guarantees that an escalated judgment will agree with a human [30]; self-evaluation has also been used to improve selective generation [42].

In nearly all of this work, abstention is a decision rule applied to a model output. In the proposed architecture it is a state of the system: criterion-level confidence, input quality and evidentiary groundedness are combined and compared against thresholds (Section 3.8 and Section 3.9), and crossing a threshold does not merely suppress an answer but moves the decision into a recorded lifecycle state in which a teacher is the required next actor.

2.5. Human Oversight as a Formal Step (RQ4)

Regulatory analysis has made human oversight a named obligation rather than a design preference, and has examined what the requirement means in practice and who must discharge it [41]. Empirical teacher-facing work reports systems in which teachers review or adjust model output [33,34,36,39], including dual-process designs that divide the work between model and human explicitly [25], examines teacher agency when assessment is mediated by generative models [40], and documents how such arrangements can fail: grading assistants can be manipulated by content embedded in the submission itself [37]. Frameworks for fair and reliable educational assessment place a human in the loop as a structural element [38], building on established guidelines for human–AI interaction [46].

Across this literature, oversight is usually expressed as a recommendation, an interface affordance or a governance principle. What is rarely present is oversight as a recorded transition: a decision object that carries its own verification status, so that whether a human confirmed, revised or rejected a given preliminary result is part of the stored record and can be audited afterwards. That is the form of oversight the present architecture implements (Section 3.10).

2.6. Problem Statement

The reviewed corpus supplies each of the required components separately. Retrieval grounding is measurable but is measured after generation; rubric-level scoring is well developed but leaves the arithmetic to the model; abstention is formalised but as a rule over model outputs; and human oversight is mandated but seldom recorded as a state. What is missing is an arrangement in which these properties hold jointly and are bound to a single governed object.

The problem addressed in this paper follows from that observation. In an examination setting, three properties must hold at once: the score must not be generated by the language model itself; every criterion-level judgment must be traceable to evidence in the student’s own response; and the system must be able to withhold a decision and route it to a teacher when its own reliability conditions are not met. No retained study binds all three into one decision object with recorded state transitions, which is the gap this work addresses.

Accordingly, the contribution claimed here is the architecture and a demonstration of its feasibility in deployment. It is not a validation of scoring accuracy: establishing that the preliminary scores agree with expert judgment requires a teacher-annotated corpus of a size and reliability that the present deployment does not yet provide; a preliminary blind calibration (Section 4.6) indicates the direction of agreement but not its level, and the boundary is stated explicitly in the abstract, in Section Limitations and Planned Validation and in Section 6.

The aim of this study is to develop a criteria-governed architecture for an intelligent system that supports and explains decisions in university learning and assessment.

The study pursues the following objectives:

-

To design an overarching architecture that integrates a RAG assistant with an AI examiner;

-

To formalize decision criteria for the learning-support and assessment modes;

-

To propose a mathematical model that aggregates criterion-level scores into a preliminary decision;

-

To develop a mechanism for estimating uncertainty and escalating decisions to a teacher;

-

To define technical and user-centered indicators for pilot evaluation of system performance and user perceptions;

-

To demonstrate the applicability of the proposed architecture using pilot data from the AI assistant and AI examiner.

The principal contribution of this study is an intelligent information-system architecture in which two modes with different purposes and risk profiles—formative learning support and preliminary assessment—operate through a single criteria-governed pipeline. Unlike solutions based on unconstrained language-model output, the proposed architecture separates probabilistic intelligent processing from deterministic result construction, links every criterion-level inference to supporting evidence, represents uncertainty explicitly, and provides a controlled decision lifecycle extending from automated analysis to teacher confirmation, revision, or rejection.

More specifically, the study introduces a shared representation for AI-assistant and AI-examiner decisions; a formal mechanism for deterministic aggregation of criterion-level scores together with a reliability-aware extension; an escalation rule triggered by insufficient evidence or high uncertainty; an architecture for audit logging and teacher verification; and a reproducible software implementation evaluated using interaction logs and user-perception data.

3. Materials and Methods

3.1. General Architecture of the Intelligent System

The proposed system adopts a multilayer architecture comprising a user layer, an API layer, a data layer, an intelligent-processing layer, a criterion-analysis layer, a decision-explanation layer, and an expert-verification loop. This design extends the component logic of intelligent tutoring systems, which distinguish among a domain model, a learner model, a pedagogical model, and a user interface [47], and is consistent with contemporary human-centered AI architectures that combine intelligent processing, explainability, decision support, and user feedback [48].

The user layer comprises separate student and teacher interfaces. Through the student interface, users can submit questions, receive explanations, complete tasks, and provide spoken or written responses. The teacher interface supports the upload of course materials, rubric configuration, review of preliminary judgments, examination of disputed cases, and analysis of interaction logs.

The API layer handles request routing, session management, authorization, logging, language-model calls, and data exchange among subsystems.

The data layer comprises relational and vector stores, a course-material database, a verified-answer database, an assessment-rubric repository, model-call logs, and the outcomes of teacher verification.

The intelligent-processing layer incorporates the RAG module, automatic transcription, NLP analysis, feature extraction, the LLM Gateway, structured-output generation, and explanation generation.

The criterion layer maps input data to formalized criteria, computes criterion-level indicators, aggregates the decision, estimates uncertainty, and determines whether teacher review is required.

The architecture supports two operating modes (Figure 2): an AI assistant for formative learning support and an AI examiner for preliminary assessment. Both modes draw on shared components, including the knowledge base, vector store, LLM Gateway, criteria module, uncertainty module, audit log, and teacher-verification loop.

Figure 2. General architecture of the criteria-governed intelligent educational system.

3.2. Unified Representation of Intelligent-Processing Results

To align the two operating modes, we introduce a structured intelligent-decision object. Whether the system processes a learning query or a student’s response, the result is represented not as unconstrained language-model text but as a set of interconnected elements:

$$\mathcal{D}\left(x,m\right)=⟨m,C,S,E,U,Q,P,H,\sigma ⟩,$$

where x denotes the input data; $m$ denotes the system’s operating mode; C is the set of applied criteria; S contains criterion-level scores or indicators; E comprises supporting evidence; U contains uncertainty estimates; Q describes input quality and processing reliability; P records the provenance of the data, models, and materials used; H contains the outcome of teacher verification; $\sigma$ denotes the current decision state.

In AI-assistant mode, the criteria include relevance, groundedness, source availability, and the sufficiency of the retrieved context. In AI-examiner mode, the criteria are defined by the assessment rubric and linked to linguistic, semantic, and structural features of the response. The common representation standardizes logging, explanation, uncertainty control, and teacher review across both modes.

3.3. Technical Architecture of the AI-Assistant Mode

The AI-assistant mode is implemented as a controlled RAG pipeline that produces answers from the materials of a specific course. A user query q enters a preprocessing module for text normalization, language identification, key-term extraction, and construction of a retrieval representation. Before initiating a new retrieval-and-generation cycle, the system checks whether a suitable answer is already available in the repository of teacher-verified responses. This sequence prioritizes verified content and invokes the language model only when no appropriate verified answer exists.

3.3.1. Retrieval and Reuse of Verified Answers

To improve reproducibility and reduce the risk of repeatedly generating unreliable content, the AI-assistant architecture includes a repository of verified answers. It stores responses that a teacher has reviewed, confirmed, or revised. Unlike a conventional semantic cache, this repository contains only results with an explicit teacher-verification status.

Following query preprocessing, the system performs a two-stage search. The first stage checks for an exact match by computing a cryptographic identifier for the normalized query text:

$$h\left(q\right)=\mathrm{S}\mathrm{H}\mathrm{A}256\left(N\left(q\right)\right),$$

where q is the original user query, $N\left(q\right)$ is its normalized form, and $h\left(q\right)$ denotes the cryptographic identifier of the normalized query.

The resulting value is compared with the query identifiers stored in the verified-answer repository. An exact match eliminates repeated retrieval and generation calls and allows the system to return a previously approved answer immediately.

Because semantically equivalent queries may be phrased differently, the second stage performs semantic matching. A vector representation $e_q$ is generated for the normalized query and compared with the vector representations $e_{q_j}$ of queries associated with previously verified answers:

$$j^{*}=\begin{array}{c}\arg \underset{\mathit{j}\in \mathcal{V}}{\max }\mathrm{c}\mathrm{o}\mathrm{s}\left(e_q,e_{q_j}\right)\\ \end{array},$$

where $\mathcal{V}$ is the set of records in the verified-answer repository and $j^{*}$ is the index of the query that is closest in semantic space.

The maximum semantic similarity is defined as follows:

$$s_{\mathrm{v}\mathrm{e}\mathrm{r}}\left(q\right)=\cos \left(e_q,e_{q_{j^{*}}}\right).$$

If the cryptographic identifiers match exactly, or if the following condition is satisfied:

$$s_{\mathrm{v}\mathrm{e}\mathrm{r}}\left(q\right)\ge {\tau }_{\mathrm{v}\mathrm{e}\mathrm{r}},$$

where ${\tau }_{\mathrm{v}\mathrm{e}\mathrm{r}}$ is a configurable semantic-matching threshold, the system returns the verified answer $a_{j^{*}}^{\mathrm{v}\mathrm{e}\mathrm{r}}$ associated with the query. The interface must identify the answer as teacher-verified. A verified answer takes precedence over generating a new language-model output.

If no exact or sufficiently close semantic match is found among the verified answers, the query is passed to the relevant-context retrieval and new-answer generation pipeline.

3.3.2. Retrieving Relevant Context and Generating an Answer

When no verified answer is available, the query is sent to a hybrid retrieval module that combines full-text lexical ranking with vector-based semantic matching. The full-text component captures exact matches for technical terms and domain vocabulary, whereas the vector component detects semantic similarity between differently worded queries and passages [49,50]. Hybrid retrieval is performed over the materials of the relevant course and provides the evidentiary context for subsequent answer generation.

The relevance of document (d) to query (q) is defined as follows:

$$Rel(q,d)=\lambda \cdot \mathrm{c}\mathrm{o}\mathrm{s}\left(E_q,E_d\right)+(1-\lambda )\cdot BM25(q,d),$$

where $E_q$ and $E_d$ are the vector representations of the query and document, respectively; (BM25(q,d)) is the lexical relevance measure; and ($\lambda$) is the balancing coefficient between semantic and full-text retrieval.

The set of retrieved passages is defined as follows:

$$R\left(q\right)=T\mathit{opK}\left\{d_j:Rel\left(q,d_j\right)\ge {\tau }_R\right\}.$$

If the relevance of the retrieved context falls below a predefined threshold, the system must not produce a confident answer. Instead, it should request clarification or refer the query to a teacher.

Accordingly, the following groundedness measure is introduced to evaluate the factual support for an answer:

$$G(a,R)={\displaystyle \frac{\left|Claims\left(a\right)\cap Supported\left(R\right)\right|}{\left|Claims\left(a\right)\right|}},$$

where Claims(a) is the set of substantive claims in the answer and Supported(R) is the set of claims supported by the retrieved course-material passages.

The presence of a relevant passage in the context does not, by itself, guarantee that the generated answer uses it correctly. Evaluation of the RAG pipeline should therefore distinguish among retrieval quality, answer relevance, and the extent to which the answer is actually grounded in the retrieved documents. This distinction is consistent with the multicomponent logic of RAGAS and with research on groundedness in retrieval-augmented generation [51,52].

Figure 3 presents the technical workflow of the AI assistant. If no suitable verified answer is found, the query enters the hybrid retrieval module, after which relevant course-material passages are sent through the LLM Gateway to the language model. This sequence follows the basic retrieval-augmented generation architecture, in which a generative model formulates an answer using previously retrieved documents or passages from an external corpus [53]. The generated response is logged together with the context used, source references, relevance and groundedness indicators, and the operational parameters of the model call.

Figure 3. Technical workflow of the AI-assistant RAG pipeline.

A newly generated answer is initially marked as unverified. If the context is insufficiently relevant, the answer is weakly grounded, or other uncertainty signals are present, the system must not present the response as reliable and instead routes it to a teacher. The teacher may confirm the answer unchanged, revise it, or reject it. These review outcomes are used to manage the verified-answer repository, as described in the following subsection.

3.3.3. Teacher Verification and a Cumulative Reliability Model

After reviewing a newly generated answer, the teacher records one of three outcomes: confirmation without substantive changes, revision, or rejection. A confirmed answer can be added to the repository directly. A teacher-revised version is stored as a new verified record, whereas a rejected version is excluded from subsequent automated reuse. To incorporate the outcomes of teacher verification, the verified-answer repository may be extended with a mechanism that accumulates reliability estimates for individual query classes. A query class is a set of semantically similar requests for which the system uses the same verified answer or a common selection rule. Accumulating statistics by query class rather than by individual record enables the reliability history of an answer to develop as it is reused.

For a class of semantically similar queries j, let the unknown parameter θ_j denote the probability that the answer proposed by the system will be confirmed by a teacher without substantive revision. We assign it the following beta prior:

$${\theta }_j\sim \mathrm{Beta}\left({\alpha }_0,{\beta }_0\right)$$

where α₀ and β₀ are the prior parameters before any teacher-review outcomes have accumulated.

After $A_j$ confirmations and $R_j$ rejections have been observed for the relevant query class, the posterior distribution becomes

$${\theta }_j\mid A_j,R_j\sim \mathrm{Beta}\left({\alpha }_0+A_j,{\beta }_0+R_j\right).$$

The posterior mean is calculated as follows:

$${\widehat{p}}_j=\mathbb{E}\left[{\theta }_j\mid A_j,R_j\right]={\displaystyle \frac{{\alpha }_0+A_j}{{\alpha }_0+{\beta }_0+A_j+R_j}}.$$

Under the beta-binomial model used here, a confirmation is defined as a teacher accepting the answer without substantive changes. A revision or rejection of the original answer is treated as a failure to confirm it:

$$A_j=N_j^{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{f}\mathrm{i}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{d}},$$

$$R_j=N_j^{\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}}+N_j^{\mathrm{r}\mathrm{e}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}},$$

where $N_j^{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{f}\mathrm{i}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{d}}$ is the number of unchanged confirmations for the relevant answer class, $N_j^{\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}}$ is the number of substantive revisions, and $N_j^{\mathrm{r}\mathrm{e}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}}$ is the number of rejections. A teacher-revised version is stored as a new verified record and may subsequently be used to process semantically similar queries.

The cumulative estimate ${\widehat{p}}_j$ may serve as an additional operational indicator of verified-answer reliability. It can inform the ranking of semantically similar records, the decision to initiate renewed review, and the identification of answers that should be reconsidered after the course materials are updated.

In the implemented configuration, the verified-answer repository therefore reduces repeated calls to the language model and creates an expanding layer of expert-confirmed knowledge. The proposed Bayesian extension would additionally account for the history of confirmations, revisions, and rejections, rank query classes by operational reliability, and identify answers requiring renewed expert review.

3.4. Technical Architecture of the AI-Examiner Mode

The AI-examiner mode performs a preliminary, criterion-based analysis of a student’s response. The inputs are an examination task z, an assessment rubric Rb, and the student’s response x. If the response is spoken, automatic transcription is performed first:

$$x=ASR\left(audio\right).$$

Transcription quality is treated as a separate technical indicator because speech-recognition errors can distort all subsequent analyses. When a reference transcript is available, quality may be estimated using the word error rate (WER):

$$WER={\displaystyle \frac{S+D+I}{N}},$$

where S is the number of substitutions, D the number of deletions, I the number of insertions, and N the number of words in the reference transcript.

In automated spoken-response assessment, automatic speech recognition forms part of the measurement pipeline because transcription errors can affect the subsequent extraction of fluency, lexical-diversity, grammatical-accuracy, and content-coverage features. Zechner et al. implemented automated scoring of spontaneous non-native speech as a sequence of speech recognition, speech-feature computation, and score prediction, underscoring the need to account explicitly for intermediate transcription quality when producing assessment results [54].

The quality of the speech-input pipeline is defined as follows:

$$Q_{ASR}=1-WER.$$

If $Q_{ASR}<{\tau }_{ASR}$, the system must not produce a confident assessment judgment without manual review.

After transcription, the response is segmented into meaningful units:

$$x=x_1,x_2,\dots ,x_l.$$

For each segment, the system computes features relating to fluency, lexical diversity, grammatical accuracy, coherence, task coverage, and response structure.

Figure 4 shows the generalized architecture of the AI examiner. The examination task, rubric, and student response enter the system; its core performs criterion-based NLP analysis, extracts response features, calculates a preliminary score together with an uncertainty signal, and sends the result to the teacher for review.

Figure 4. Generalized architecture of the AI examiner.

3.5. Extracting Interpretable Response Features

Unlike holistic scoring, in which a model produces an overall score without disclosing its basis, the proposed system uses interpretable features associated with individual rubric criteria. This approach follows the logic of analytic rubrics and trait-specific automated scoring, in which a response is treated as a combination of structural, semantic, and criterion-relevant characteristics [55,56]. For IT-related responses, lexical features must also be interpreted in light of the domain-specific nature of professional discourse. Prior research [57] shows that the professional linguistic worldview of IT specialists draws on specialized terminology, English loanwords, professional jargon, transformed idioms, and other conventional linguistic units. The resulting design requirement is important: professional and borrowed terms must not automatically be classified as lexical errors or penalized. They should instead be evaluated against a domain lexicon, the task context, and the relevant rubric criteria.

The following groups of features may be computed for a spoken response:

$$F\left(x\right)=F_{fluency}\left(x\right)\cup F_{lex}\left(x\right)\cup F_{gram}\left(x\right)\cup F_{sem}\left(x\right)\cup F_{struct}\left(x\right).$$

Fluency features:

$$SpeechRate={\displaystyle \frac{N_{words}}{T_{total}}},ArticulationRate={\displaystyle \frac{N_{words}}{T_{speech}}},$$

$$PauseRatio={\displaystyle \frac{T_{pause}}{T_{total}}}.$$

Lexical diversity may be estimated using the type-token ratio:

$$TTR={\displaystyle \frac{N_{unique}}{N_{total}}},$$

A normalized measure may be used to reduce dependence on response length:

$$MATTR={\displaystyle \frac{1}{L-w+1}}{\displaystyle \sum _{j=1}^{L-w+1}}{\displaystyle \frac{N_{unique}\left(x_{j+w}\right)}{w}}.$$

The semantic relevance of a response to the task may be measured as the cosine similarity between their embeddings, a widely used approach to semantic textual similarity based on sentence embeddings [58]:

$$SemRel(x,z)=\mathrm{c}\mathrm{o}\mathrm{s}\left(E_x,E_z\right).$$

Response coherence may be estimated as the mean semantic similarity between adjacent units of meaning. This approach is consistent with automated text-coherence research, in which local coherence is modeled through the interaction and semantic similarity of neighboring sentences or text segments [59]:

$$Coh(x)={\displaystyle \frac{1}{l-1}}{\displaystyle \sum _{j=1}^{l-1}}\mathrm{c}\mathrm{o}\mathrm{s}\left(E_{x_j},E_{x_{j+1}}\right).$$

Figure 5 illustrates the pipeline for extracting interpretable features from a spoken response. The audio response is converted into a word-level timestamped transcript, after which the system performs segmentation and extracts features related to fluency, lexical diversity, syntactic complexity, semantic relevance, and coherence. The results are then passed to the criterion-based assessment module.

Figure 5. Transparent pipeline for extracting interpretable features from a spoken response.

3.6. Formalized Model of a Criterion-Based Decision

Let the set of criteria be $C=c_1,c_2,\dots ,c_m$. This representation follows the logic of analytic assessment rubrics, under which performance is evaluated not only holistically but also against separate criteria or dimensions of quality [55]. Each criterion is assigned a weight $W=w_1,w_2,\dots ,w_m, {\displaystyle {\sum }_{i=1}^m}{w}_i=1$, consistent with multi-criteria decision-making approaches in which the overall value is an aggregate function of criterion-level scores and their weights [60]. The notation used in Section 3.6, Section 3.7, Section 3.8 and Section 3.9 is summarized in Appendix A (Table A1).

For each criterion (c_i), the system produces a criterion-level score $s_i\left(x\right)={\varphi }_i\left(F_i\left(x\right),R_i,E_i\right)$. Here, R_i is the criterion’s textual description in the rubric, F_i(x) is the set of computed response features, and E_i comprises the evidence passages supporting the score. Contemporary trait-specific automated-scoring methods treat a response as a set of characteristics mapped to rubric criteria rather than solely as undifferentiated text. TRATES, for example, uses a rubric to generate and evaluate features specific to individual traits of a written response [56].

When a criterion is evaluated on an ordinal scale, its score may be calculated from a probability distribution over performance levels. This is consistent with the general logic of models for ordinal outcomes [61] and with converting a probabilistic forecast into a point estimate through a statistical functional of the distribution [62]:

$$s_i(x)={\displaystyle \sum _{k=0}^{K_i}}k\cdot P\left(y_i=k|x,c_i\right).$$

The implemented decision rule uses deterministic weighted aggregation of the criterion-level scores. This follows the additive logic of multi-criteria decision making, in which the preliminary overall score is calculated from individual criterion scores according to their predefined importance [63]:

$$S\left(x\right)={\displaystyle \sum _{i=1}^m}{w}_i{s}_i\left(x\right),{\displaystyle \sum _{i=1}^m}{w}_i=1$$

where s_i(x) is the score assigned to criterion i and w_i is its predefined weight. The aggregation is performed programmatically outside the language model, ensuring that the overall preliminary score cannot be generated or altered directly by the LLM.

As an architectural extension, the same decision model can incorporate criterion-specific reliability by reducing the contribution of criteria whose input quality or evidentiary basis is insufficient. A reliability-adjusted normalized form is therefore defined as

$$S_q\left(x\right)={\displaystyle \frac{{\displaystyle {\sum }_{i=1}^m}{w}_i{q}_i\left(x\right)s_i\left(x\right)}{{\displaystyle {\sum }_{i=1}^m}{w}_i{q}_i\left(x\right)}},$$

where q_i(x) ∈ [0,1] is a criterion-specific reliability coefficient reflecting the quality of the input data and the sufficiency of the supporting evidence for criterion i. A lower value of q_i(x) reduces the influence of the corresponding criterion without changing its rubric-defined weight w_i.

In the current implementation, the deterministic weighted form S(x) is used to calculate the preliminary score, whereas reliability indicators are handled separately through uncertainty control, confidence tiers, and escalation to teacher review. The reliability-adjusted form S_q(x) represents an architectural extension intended for subsequent empirical calibration on a teacher-annotated response corpus.

3.7. Decision-Explanation Model

The proposed educational AI system achieves explainability not by exposing the internal parameters of the language model, but by reconstructing the complete analytical path from the original response and transcript to the computed features, evidence passages, and criterion-level scores. In accordance with transparency requirements for AI in education [64], this approach allows the result to be presented in a form tailored to teachers’ needs, with explicit disclosure of the data, features, inferential basis, and model limitations involved.

For explainability, the system output is represented by the following structure:

$$J\left(x\right)={\left(c_i,s_i,e_i,r_i,u_i\right)}_{i=1}^m,$$

where c_i is a criterion, s_i is its score, e_i denotes the supporting evidence, r_i is the textual explanation, and u_i is the criterion-level uncertainty.

For the AI assistant, the explanation structure is defined as follows:

$$J_{assist}(q)={\left(a,d_j,Rel\left(q,d_j\right),G_j,status\right)}_{j=1}^k.$$

For the AI examiner, it is defined as follows:

$$J_{exam}\left(x\right)={\left(c_i,s_i,E_i\left(x\right),r_i,u_i,revie{w}_i\right)}_{i=1}^m.$$

This representation enables the teacher to inspect not only the overall result, but also the basis for it: which criteria were applied, which features were extracted, which parts of the response served as evidence, and where the system detected uncertainty.

The division of labor between the probabilistic and the deterministic parts of this construction requires explicit statement, because the guarantees claimed for the architecture rest on it. The language model performs one analytical step: for each rubric criterion it identifies the span of the student response on which its assessment is based. This step is probabilistic and is not presented here as otherwise.

Everything downstream of that step is governed by program rules that the model cannot alter, and these rules form the contract of the decision object. First, every criterion-level score must be accompanied by a verbatim span drawn from the response, which is stored with the score. Second, a criterion for which no such span is available is scored zero or marked as insufficiently substantiated, by program rule rather than by model judgment. Third, the overall preliminary score is computed as the programmatic weighted aggregation defined in Section 3.6, outside the model, so that the model cannot generate or alter it. Fourth, the complete decision object, including spans, criterion-level confidence and provenance, is persisted and remains auditable after the fact.

The architecture therefore does not make a probabilistic component deterministic; it constrains that component to an analytical role, requires it to expose the textual basis of each inference, and computes every consequential quantity outside it. Section 4.5 reports the empirical check of this contract on the deployment: across 1074 criterion-level decisions, all carried a non-empty verbatim evidence span, and in every finalized session the stored overall score reproduced the programmatic aggregation of the stored criterion scores.

Each construct of the formal model in Section 3.6, Section 3.7, Section 3.8 and Section 3.9 has a counterpart in the implementation (Appendix A, Table A1). The criterion-level score s_i(x), its band and the evidence E_i are fields of the per-criterion entry of the stored decision object, and the operational confidence signal r_i(x) is the confidence value recorded with that entry. The overall score is computed by the program from the stored criterion scores, the escalation predicate is implemented as the routing rule that sets the recorded session state, and the confidence tier is derived from the stored confidence values (Section 4.4). The schema of the decision object, the assessment prompt templates, the rubrics, the database schema and the architectural decision records are provided as Supplementary Materials (S1–S6).

3.8. Model of Uncertainty and Decision Escalation

Uncertainty estimation is used as an operational mechanism for controlling the reliability of a preliminary decision. For each criterion c_i, the system records an operational confidence signal r_i(x) ∈ [0,1] together with the criterion-level score and supporting evidence. This signal reflects the model’s self-reported confidence in the corresponding rubric-based inference and is not interpreted as a calibrated posterior probability.

Criterion-level operational uncertainty is defined as

u_i(x) = 1 − r_i(x)

The aggregate uncertainty of the preliminary decision is calculated as a weighted combination of the criterion-level uncertainties:

$$U(x)={\displaystyle \sum _{i=1}^m}{w}_i{u}_i(x).$$

where w_i is the weight of criterion i. Thus, higher values of U(x) indicate lower operational confidence in the preliminary decision. The measure is used for routing and escalation rather than as a calibrated estimate of the probability that the assessment is incorrect.

The rule for referring a decision to a teacher follows the principle of abstaining from automated inference whenever reliability is insufficient. If the system detects high uncertainty, poor input quality, inadequate evidence, or disagreement among criteria, the preliminary decision is not applied automatically. Instead, the student’s response is referred to a teacher for expert review [65]:

$$Esc\left(x\right)=1, if U\left(x\right)>{\tau }_U\vee Q\left(x\right)<{\tau }_Q\vee G\left(x\right)<{\tau }_G\vee D_{conflict}\left(x\right)>{\tau }_D.$$

Here, U(x) is the aggregate operational uncertainty defined above, Q(x) denotes input quality, G(x) denotes evidentiary groundedness, and C(x) represents a critical conflict among criteria or a substantial discrepancy from the reference decision. The thresholds used in this rule are configuration parameters of the operational system rather than statistically calibrated probabilities of error.

This rule establishes when escalation is required. To manage subsequent processing, however, the continuous and logical reliability indicators must be translated into a discrete operational status. We therefore introduce a three-tier confidence model that specifies how a preliminary result may be used and the extent to which teacher involvement is mandatory.

3.9. Confidence Tiers and the Boundaries of Decision Automation

A numerical uncertainty estimate does not, by itself, determine how a result may be used. The proposed architecture therefore introduces a three-tier confidence model that maps input quality, evidentiary support, uncertainty, and agreement among criteria to specific actions by the information system.

Here, confidence does not denote proven automated-scoring accuracy or a user’s subjective trust in AI. Rather, it is an operational status assigned to a preliminary result; this status determines how the result is processed, the priority of teacher review, and whether an assessment judgment may be generated.

Table 1 summarizes the conditions for assigning each confidence tier and the corresponding system actions.

Table 1. Confidence tiers for preliminary results and corresponding information-system actions.

Tier	Assignment Conditions	Result Status	Information-System Action
T1. Operational readiness	Input data are of sufficient quality; supporting evidence has been identified for the material criteria; uncertainty does not exceed the predefined threshold; and there are no critical conflicts among criterion-level scores.	The preliminary result has sufficient internal reliability for standard teacher review.	Store the result, criteria, evidence, and reliability indicators, and send the result to the teacher for confirmation.
T2. Mandatory expert review	Local uncertainty is detected for one or more criteria; the evidence is incomplete; the result lies in a borderline range; or a transcription error or discrepancy among individual indicators may be present.	The result may be used only as an analytical recommendation.	Highlight the affected criteria and passages, record the reason for escalation, and place the result in the priority queue for teacher review.
T3. Automated inference not viable	Input quality is critically low; supporting evidence is absent; the transcript does not allow the response to be reconstructed reliably; the criteria are materially inconsistent; or the rubric cannot be applied.	A substantively justified preliminary judgment cannot be produced.	Do not generate an assessment judgment; request a new response or clarification or refer the case entirely to the teacher.

A T1 status neither establishes scoring accuracy nor authorizes the system to make a final academic decision autonomously. It indicates only that the architecture’s internal reliability conditions have been met: adequate input quality, available evidence, no critical conflict among criteria, and acceptable uncertainty. The teacher retains final authority in every assessment scenario.

A T2 status indicates that the system has produced some useful analytical elements, but the identified limitations require closer scrutiny. At T3, the system abstains from generating an assessment conclusion because the input or evidence is insufficient to support a defensible preliminary decision.

The system’s functions are divided into three zones according to their permissible degree of automation. The automated-support zone covers material retrieval, transcription, segmentation, feature computation, and evidence extraction. The hybrid zone includes criterion-based analysis, calculation of a preliminary score, identification of strengths and weaknesses, and drafting of feedback. The human-decision-only zone comprises final academic grading, appeals, resolution of disputed cases, and any decision with a direct bearing on a student’s academic trajectory.

Accordingly, T1 results may be used in the automated and hybrid zones, subject to subsequent teacher confirmation. T2 results are permitted only in the hybrid zone and require priority expert review. At T3, the automated assessment judgment is withheld and the decision is referred entirely to the teacher.

3.10. Lifecycle of a Preliminary Decision and Teacher Verification

In the proposed architecture, human-in-the-loop operation is not treated as an external organizational constraint, but as a formal component in the lifecycle of the information object representing the system’s preliminary decision. Regardless of whether the AI assistant or AI examiner is used, each result passes through a fixed sequence of states: input acquisition, preprocessing, feature and evidence extraction, reliability checks, preliminary inference, possible escalation, and teacher verification.

When reliability is sufficient, the system generates a preliminary decision and submits it to the teacher for review. When reliability is inadequate because of weak evidence, poor input quality, or elevated uncertainty, the decision is not treated as ready for use and is escalated immediately. Following teacher review, the preliminary conclusion may be confirmed, revised, or rejected.

A decision therefore exists within the system not as a one-off model response, but as a governable object with recorded state transitions. This ensures traceability and accountability and allows the system to document how the outcome was reached and at what stage human judgment entered the process. Figure 6 shows the lifecycle of a preliminary decision and its verification by a teacher.

Figure 6. Lifecycle of a preliminary decision and teacher verification.

3.11. Implementation Features and Architectural Decisions

The operational implementation of the AI examiner separates probabilistic intelligent processing from deterministic result construction. The language model acts as a constrained analyzer of a response against a predefined rubric; it does not determine the final score. The software pipeline computes criterion-level and overall scores, verifies the presence of evidence, applies threshold rules, and routes the decision to a teacher. This design makes every preliminary conclusion reproducible, traceable, and auditable.

3.11.1. Criterion-Based Scoring and the Evidentiary Basis

The assessment module follows a two-pass, “critique-first” procedure. On the first pass, the model identifies missing or insufficiently developed elements of the response and then assigns scores for each rubric criterion. Every score must be accompanied by a verbatim passage from the response that supports the corresponding inference. If no supporting passage is available, the criterion receives a score of zero or is marked as insufficiently substantiated.

The overall preliminary score is computed programmatically using the deterministic weighted aggregation defined in Section 3.6. The language model produces criterion-level analyses and scores, but the aggregation itself is performed outside the model. Reliability and uncertainty signals do not directly modify the score in the current implementation; instead, they determine whether the result requires additional model review, receives a lower operational confidence tier, or must be escalated to the teacher. An optional second pass performs a self-check in which the system identifies unsupported scores and may revise them. Any revision in a single pass is capped at 25% of the maximum score for the relevant criterion.

Vector-based comparison of the response with the task’s reference answer and rubric-level exemplars provides an additional source of justification. The nearest level and the margin between the two closest exemplars are not used to assign a score directly; rather, they serve as independent diagnostic indicators available to the teacher.

In addition to scores, the system stores several types of interpretable indicators:

-

Vector similarity between the response and the exemplars;

-

Matches of key terms and entities;

-

n-gram measures of textual similarity;

-

Groundedness indicators; and

-

Discourse coherence in multistep responses.

Each indicator is linked to the relevant criterion, intermediate inference, and response passage. The result is therefore not a single numerical value, but a reconstructable body of evidence supporting the preliminary decision.

3.11.2. Uncertainty, Speech Recognition, and Teacher Review

For each criterion, the system records the operational confidence signal r_i(x) defined in Section 3.8. If this signal falls below the predefined threshold, the corresponding criterion is flagged for manual review. The criterion-level signals are also used to calculate the aggregate operational uncertainty U(x). If the overall assessment result falls within the predefined “gray zone”, the system initiates a single reassessment using a more capable model. This second analysis does not replace teacher verification; it provides an additional diagnostic signal for examining a borderline case.

The overall score is withheld from the student until the teacher has reviewed it. It is stored as a preliminary result that the teacher may confirm, revise, or reject. Human-in-the-loop operation is thus a mandatory stage in the lifecycle of an assessment decision, rather than an optional external procedure.

For spoken responses, the reliability of automatic speech recognition is evaluated separately from the substantive quality of the response. The transcript is stored with word-level timestamps, and low-reliability segments are explicitly flagged in the interface. A fallback route is used if the primary recognition service is unavailable.

Separate treatment of ASR quality is essential because recognition performance may vary across speech and social groups [66]. Although the relevant evidence largely concerns English-language speech, it supports a general architectural requirement: transcription errors must not automatically be interpreted as deficiencies in a student’s subject-matter competence. The recognition model used, processing route, timestamps, and transcription-confidence measure are therefore stored separately from the substantive response features.

3.11.3. Governability and Fault Tolerance

All language-model requests pass through a single router that supports both primary and fallback routes. Each call is logged with the model name, request time, parameters, and processing outcome. Identical-request coalescing and a semantic cache are used to limit redundant computation.

If external providers are unavailable, the system must not produce an unpredictable or silently incomplete result. Instead, it switches to a fallback model or an explicitly signaled degraded mode. This approach ensures controlled behavior during technical failures and allows the sequence of actions leading to a particular result to be reconstructed.

Table 2 summarizes the principal architectural mechanisms and their purposes.

Table 2. Key mechanisms for generating and controlling a preliminary assessment decision.

Architectural Mechanism	Implementation	Purpose
Constrained role of the language model	Rubric-based response analysis without determining the overall score	Reduce the result’s dependence on unconstrained LLM output
Deterministic aggregation	Programmatic weighted aggregation of criterion-level scores outside the LLM	Ensure reproducibility and prevent the LLM from directly determining the overall preliminary score
Mandatory evidentiary basis	Link each criterion to a passage from the response	Make the assessment inference auditable and explainable
Two-pass analysis	Initial critique followed by a constrained self-check	Identify unsupported or inconsistent scores
Comparison with exemplars	Vector similarity to the reference answer and rubric levels	Provide an independent diagnostic indicator
Uncertainty control	Operational criterion-level confidence, aggregate uncertainty, thresholds, and a “gray zone”	Identify cases requiring additional review
ASR quality control	Timestamps, a confidence measure, and a fallback route	Separate transcription errors from response quality
Human-in-the-loop	Teacher confirmation, revision, or rejection	Preserve human professional responsibility
Logging and failover	Unified router, call logs, and a degraded mode	Ensure accountability and predictable behavior during failures

As Table 2 illustrates, the distinctive feature of the implementation is not any single language model or retrieval algorithm, but the integration of multiple control mechanisms into one software pipeline. Probabilistic components perform recognition, retrieval, and semantic analysis; deterministic components compute the result, apply thresholds, maintain logs, and govern state transitions.

3.11.4. Implementation Parameters

The reference implementation, Caracal AI, is a self-hosted web application built with Next.js and PostgreSQL 18. Vector representations are produced by the gemini-embedding-2 model at an initial dimensionality of 3072, truncated to 768 dimensions using the Matryoshka method, and stored in pgvector with a hierarchical navigable small world (HNSW) index.

A structure-aware chunking algorithm divides course materials into passages with a target length of 500 characters and a maximum length of 800 characters. Retrieval returns the five nearest passages and uses a full-text fallback route when necessary.

Primary answer generation and criterion-based scoring use gemini-2.5-flash-lite. Results in the 40–70% range are reassessed once using gemini-2.5-flash or gemini-2.5-pro. The semantic cache is applied at a cosine-similarity threshold of 0.95, while teacher-verified answers take precedence over new generation at a threshold of 0.92. During assessment, the two nearest calibration exemplars are retrieved.

Spoken responses in Russian, Kazakh, and English are transcribed through a configurable speech-recognition route that includes Vosk, transcription by gemini-2.5-flash, and the Web Speech API as a fallback; the recognizers that produced the transcripts in the reported deployment are given in Section 4.5. The distinction between the pilot and integrated configurations is detailed in the following subsection.

These thresholds are configuration parameters of the pilot implementation and are not claimed to be empirically optimal or domain-independent. Their calibration should be performed on a teacher-annotated response corpus in subsequent validation studies.

3.11.5. Conceptual Comparison with Existing Approaches

A direct experimental comparison with commercial systems would require the examinations to be repeated on external platforms. We therefore compare architectural characteristics instead. The comparison reflects configurations commonly described in the literature and does not imply that every system in a given class has identical properties (Table 3).

Table 3. Conceptual comparison of typical automated-assessment approaches.

Dimension	Direct LLM-Based Scoring	Classical Feature-Based Automated Essay Scoring (AES)	Proposed Architecture
Basis of the decision	General model inference based on an instruction	Model trained on predefined features	Rubric-based analysis with deterministic aggregation
Criterion-level evidence	May be generated as a textual explanation	Partly represented by feature values	Mandatory response passages for every criterion
Explanation generation	Usually produced after the result	Based on feature contributions	Produced concurrently with criterion-level scores
Representation of uncertainty	Depends on the implementation	May be incorporated into the model	Criterion-level confidence, thresholds, and a “gray zone”
Teacher involvement	Determined by the use scenario	Typically used for labeling and calibration	Mandatory for the final decision
Reproducibility	Depends on the model and generation parameters	Usually high	Supported by audit logging and deterministic computation
Adaptation to another discipline	Revise the instruction and context	Often requires a new sample or model retraining	Replace the rubric, corpus, and domain-specific exemplars

The comparison suggests that the proposed architecture occupies a middle ground between the flexibility of generative scoring and the reproducibility of classical feature-based models. Its defining characteristic is the formal separation of probabilistic analysis from deterministic preliminary decision construction, while teacher oversight remains mandatory.

3.12. Software Versions and the Scope of the Pilot Evaluation

The system was developed iteratively. The technical indicators for the AI-assistant mode, derived from 347 processed queries, relate to a pilot configuration that combined local and external language-model routes through a unified gateway. This pilot configuration and its initial operational indicators were reported previously in a component-level study [67]. Here, the same data are used to characterize the original functional component and to trace the system’s evolution toward an integrated architecture; they are not presented as a new, independent experiment.

In the subsequent integrated configuration, the architecture was extended with an AI-examiner module, a structured result representation, criterion-based analysis, uncertainty-estimation mechanisms, and additional model-routing capabilities.

Table 4 summarizes the principal differences between the pilot AI-assistant configuration and the integrated system. This distinction is necessary for correct interpretation: the AI-assistant indicators describe the performance of that particular component, not the results of load or comparative testing of the complete integrated architecture.

Table 4. Evolution of the software architecture from the pilot AI assistant to the integrated system.

Component	Pilot AI-Assistant Configuration	Integrated System Configuration	Purpose of the Change
API layer and orchestration	FastAPI and LiteLLM Gateway	Next.js API layer, unified model-call router, and fallback routes	Standardize subsystem interaction and centralize model-call management
Language models	Llama 3.2 and external providers	Gemini 2.5 Flash Lite for primary processing; Gemini 2.5 Flash and Gemini 2.5 Pro for reassessing borderline cases	Separate primary processing from additional review of decisions in the “gray zone”
Embedding model	Embedding model used in the pilot configuration	Gemini Embedding with vector representations truncated to 768 dimensions	Standardize the representation of queries, course materials, reference answers, and rubric levels
Vector storage and retrieval	PostgreSQL and pgvector	PostgreSQL 18, pgvector, and an HNSW index	Improve semantic-retrieval performance and support exemplar records
Speech recognition	Not used	Configurable speech-recognition route: Vosk, gemini-2.5-flash transcription, and the Web Speech API as a fallback	Support spoken responses and record the quality of the speech-input pipeline
Output control	Database of teacher-verified answers	Formalized criteria, supporting evidence, deterministic aggregation, uncertainty estimation, and escalation rules	Move from answer generation to controlled construction and justification of a preliminary decision
Human-in-the-loop	Teacher review and confirmation of AI-assistant responses	Confirmation, revision, or rejection of the preliminary result; mandatory verification of assessment decisions	Formalize human oversight within the decision lifecycle
Fault tolerance	Switching between local and external models through a unified gateway	Fallback providers, a circuit breaker, semantic cache, and an explicitly signaled degraded mode	Ensure predictable system behavior when individual services are unavailable

As Table 4 shows, the transition to the integrated configuration involved more than replacing the language model. The central changes concerned the logic used to process and control results: the system gained a structured decision representation, criterion-based analysis, evidence passages, uncertainty estimation, escalation to teachers, and support for spoken responses. These additions transformed a specialized RAG assistant into a multifunctional information architecture that integrates formative learning support with preliminary assessment.

The empirical data must be interpreted within the scope of each system version. Response-time, query-success, source-reference, and technical-failure indicators were obtained for the pilot AI assistant. The student survey results concern the pilot deployment of the AI examiner. The data therefore support the functionality and user acceptability of individual subsystems, but do not characterize the performance of the complete integrated system when all modules operate concurrently.

3.13. Pilot Design and Survey-Data Processing

The pilot evaluation of the AI examiner included a post-use student survey administered after interaction with the system in an examination setting. The original dataset contained 58 records obtained using the main ten-item student questionnaire. Before statistical analysis, six test and administrative records were removed based on prespecified account and session identifiers; item responses were not used as exclusion criteria. The final analytical sample comprised 52 questionnaires. Additional records collected with a separate questionnaire containing a different set of items were not merged with the main sample and were not analyzed in this study. All exclusions were applied before descriptive statistics, confidence intervals, and composite indices were calculated.

Responses were recorded on a five-point Likert scale. Higher values on positively worded items indicated a more favorable perception of the procedure, whereas higher values on negatively worded items indicated a more pronounced difficulty or risk [68].

For the AI-assistant mode, the analysis used system interaction logs containing the query-processing status, response-generation time, presence of references to course materials, request-completion type, and recorded technical errors. A query was considered successful if it produced an answer without a technical error. The success rate was calculated as the number of successful queries divided by the total number logged. The mean response time, the median, and the 90th percentile of processing duration were also calculated.

For each questionnaire item, the mean and 95% confidence interval were calculated as follows:

$${\overline{x}}_j={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{x}_{ij},C{I}_{95}={\overline{x}}_j\pm t_{0.975,n-1}{\displaystyle \frac{s_j}{\sqrt{n}}},$$

where $x_{ij}$ is respondent i’s answer to item j, n is the number of questionnaires, s_j is the sample standard deviation for the item, and $t_{0.975,n-1}$ is the critical value of Student’s t distribution.

The distribution of responses was further examined by calculating the proportion in each scale category:

$$p_{jr}={\displaystyle \frac{n_{jr}}{n}}\cdot 100$$

where $p_{jr}$ is the proportion of responses in category r for item j, n_jr is the number of respondents who selected category r, and $r\in 1,2,3,4,5$.

Negatively worded items were reverse-coded before the composite indices were calculated:

$$x_{ij}^{*}=6-x_{ij}.$$

The questionnaire items were formally denoted by $S1$–$S10$: $S1$, question clarity; $S2$, adequacy of the allotted time; $S3$, alignment of the result with expectations; S4, usefulness of criterion-level feedback; S5, perceived fairness of the assessment; S6, uncertainty about the score; S7, discomfort with AI involvement; S8, concern about automatic speech-recognition errors; S9, technical failures; and S10, willingness to use the system.

For each respondent i, composite index k was calculated as the mean of its constituent items after aligning their direction:

$$I_{ik}={\displaystyle \frac{1}{m_k}}\sum _{j\in J_k}{x}_{ij}^{*},$$

where $J_k$ denotes the set of items included in composite index $k$, $m_k$ is the number of those items, and $x_{ij}^{*}$ is the original or reverse-coded value of item j for respondent i.

The mean composite-index value for the analytical sample was calculated as follows:

$${\bar{I}}_k={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{I}_{ik},$$

where n = 52 is the number of questionnaires included in the statistical analysis.

The procedural-transparency and acceptability index included items S1, S2, S4, and S10. The perceived-fairness and alignment-with-expectations index included items S3, S5, and reverse-coded item S6. The trust-in and acceptance-of-the-AI-examiner index included reverse-coded items S7, S8, and S9. The overall perception index was calculated from all ten items after reverse-coding the negatively worded items S6–S9.

This procedure aligned all items in the same interpretive direction, such that a higher composite value consistently represents a more favorable perception of the system.

3.14. Preliminary Calibration Against Human Raters

To obtain a first estimate of how the AI examiner’s criterion-level scores relate to human judgment, a blind calibration study was conducted on the deployment data. Of 215 AI-scored examination answers, 139 were eligible after excluding answers from the trial examination (50), answers outside the examination window (19), failed transcriptions (3), near-empty answers (2), incomplete sessions (1) and staff accounts (1). A stratified random sample of 35 answers (25%) was drawn once with a fixed seed, stratified by answer mode and rubric and, for spoken answers, by whether the original recording had been retained: 6 spoken answers with a recording, 7 spoken answers rated from the automatic transcript that the AI had graded, and 22 written answers, with at most one answer per student in each stratum. Raters scored every criterion on the same five-band descriptors that the AI examiner uses, without access to the AI scores, to one another’s ratings or to student identity, each in an individually randomized order and through a personal link that exposed no AI output. Four raters were invited; one of them was assigned only the spoken answers, and three of the four are co-authors of this study.

Inter-rater reliability was estimated with Krippendorff’s α using the ordinal metric [69]. Agreement between the AI band and the mean band of the raters was summarized with quadratic-weighted Cohen’s κ [70], Spearman’s ρ and the proportion of AI bands within one band of the rater mean. Criterion bands were converted to 0–100 totals with the examiner’s own band-to-points arithmetic, and the totals were compared by Pearson’s r with a Fisher z-transformed 95% confidence interval, the mean difference between AI and raters with a t-based 95% confidence interval, and Bland–Altman limits of agreement [71]. All available ratings were analyzed; Krippendorff’s α accommodates the missing ratings of a rater who did not complete the assignment.

4. Results

4.1. Technical Results for the AI-Assistant Mode

The AI-assistant mode was evaluated using interaction logs containing response times, query statuses, references to course materials, and technical errors. Table 5 presents the resulting indicators.

Table 5. Technical indicators for the AI-assistant mode.

Indicator	Value	Interpretation
Number of queries	347	Pilot-use volume
Successfully processed queries	336	Completed responses
Success rate	96.8%	Technical stability
Responses with source references	89.0%	Source-reference coverage and traceability to course materials
Mean response time	1247 ms	Interactivity
Median response time	980 ms	Typical interaction time
P90	2847 ms	Upper-range latency

Of the 347 logged queries, 11 did not produce a successful response: controlled refusals accounted for 2.0%, and technical errors for 1.2%. Response time reached 3200 ms on a cold start and fell to 980 ms for repeated queries. This final measure is not a direct indicator of educational effectiveness; it does, however, show that operational logs can help identify topics that require further explanation or revision of the learning materials.

4.2. Functional Implementation of the AI-Examiner Mode

The implemented AI-examiner prototype supports the principal stages of criterion-based analysis: receiving the examination task, rubric, and student response; transcribing a spoken response; segmentation; feature extraction; calculating a preliminary score; identifying strengths and weaknesses; and recording problematic passages and uncertainty signals.

The current pilot data should be interpreted as evidence of the technical pipeline’s feasibility and of user perceptions, not as conclusive proof of automated-scoring accuracy. Full validation will require a corpus of responses independently labeled by multiple experts, criterion-level comparisons between AI and teacher scores, inter-rater agreement estimates, and the proportion of cases in which a teacher changes the system’s preliminary judgment. Validation should address not only agreement between automated and expert scores, but also the defensibility of score interpretations, model generalizability, intended conditions of use, and the potential consequences of automated decisions [72].

4.3. Student Perceptions of the AI Examiner

Student perceptions of the AI-examiner mode were assessed using post-use questionnaires collected after the system had been piloted in an examination setting (Figure 7). After test and administrative records were excluded, the analytical sample comprised 52 responses. The questionnaire combined positively worded items addressing procedural clarity, the usefulness of criterion-level feedback, adequacy of the allotted time, perceived fairness, and willingness to use the system with negatively worded items addressing uncertainty about the score, concern about speech-recognition errors, discomfort with AI involvement, and technical failures. Values are reported as mean scores on a 1–5 Likert scale with 95% confidence intervals. Blue denotes positively worded items, whereas orange denotes negatively worded items representing potential risks or limitations.

Figure 7. Student perceptions of the AI examiner by questionnaire item (n = 52).

The highest ratings concerned question clarity (4.81), the usefulness of criterion-level feedback (4.46), and the adequacy of the allotted response time (4.42). Students thus found the interaction procedure clear and organizationally acceptable. Relatively high scores for perceived fairness (4.15) and alignment with expectations (4.10) further suggest that the system’s preliminary judgments were generally not perceived as arbitrary or opaque.

Trust in the AI examiner was nevertheless conditional. The comparatively high ratings for the negative items “uncertainty about the score” (3.79) and “concern about ASR errors” (3.31) reinforce the need for teacher review, particularly for spoken responses, where transcription quality may affect subsequent criterion-based analysis. Ratings for discomfort with AI involvement (2.54) and technical failures (2.06) were moderate, suggesting that the system is technically acceptable for pilot use as a preliminary-analysis tool.

The survey findings therefore accord with the central design logic of the proposed architecture: the AI examiner can structure responses, conduct criterion-level analysis, and prepare explainable feedback, but should not act as an autonomous source of final academic decisions. The best-supported use case is a human-in-the-loop model in which the system produces a preliminary judgment and the teacher retains the authority to review, revise, and decide.

Figure 8 shows the proportion of responses in each Likert category (1–5) for every item. For positively worded items, a predominance of categories 4 and 5 indicates favorable perceptions of the procedure. For negatively worded items, high values indicate a more pronounced risk or difficulty.

Figure 8. Distribution of responses to the student-questionnaire items (n = 52).

The response distributions corroborate the findings based on the means. For “question clarity,” high ratings account for 98.1% of responses: 82.7% of students selected category 5 and 15.4% selected category 4. This indicates a strong consensus that the procedure was clear. Categories 4 and 5 account for 90.4% of responses to “usefulness of criterion-level feedback,” supporting the pedagogical value of explainable feedback.

The distributions for “alignment with expectations” and “perceived fairness” are more reserved: appreciable proportions of mid-scale responses remain alongside the high ratings. Trust in the assessment inference therefore appears to be developing but is not unconditional. The most important limitations emerge in the negative items. A substantial proportion of students selected categories 4 and 5 for “uncertainty about the score,” indicating that teacher review remains necessary. The sizeable share of high ratings for “concern about ASR errors” likewise confirms the importance of monitoring transcription quality in spoken-examination settings.

By contrast, responses to “technical failures” are concentrated at the lower end of the scale; the predominance of categories 1 and 2 indicates that technical problems are not the leading source of negative perceptions. Further development should therefore focus less on basic operability than on confidence in the scoring logic, spoken-language recognition quality, and the transparency of teacher verification.

To summarize the structure of user perceptions, composite indices were calculated using the procedure described in Section 3.13. Table 6 presents the results.

Table 6. Composite indices of AI-examiner perceptions after reverse-coding negatively worded items.

Index	Included Items	Mean	Interpretation
Procedural transparency and acceptability	Clarity, time, usefulness of feedback, and willingness to use	4.39	The procedure is clear and generally accepted by students.
Perceived fairness and alignment with expectations	Alignment with expectations, fairness, and reverse-coded uncertainty	3.49	Confidence in the assessment inference is developing but remains limited.
Trust in and acceptance of the AI examiner	Reverse-coded discomfort, concern about ASR errors, and technical failures	3.37	A priority area for further development.
Overall perception index	All items, with negatively worded items reverse-coded	3.81	The system is viewed positively but still requires teacher oversight.

The composite indices show that procedural transparency and acceptability were the strongest outcomes of the pilot (4.39). Lower values for perceived fairness and alignment with expectations (3.49), and for trust in and acceptance of the AI examiner (3.37), indicate that the system may be used as a preliminary-analysis tool but should not be treated as an autonomous mechanism for final assessment. The overall perception index (3.81) reflects a generally positive student response, while confirming the continued need for teacher oversight.

4.4. Operational Behaviour of the Confidence Thresholds

The confidence tiers and routing thresholds introduced in Section 3.8 and Section 3.9 were specified as configuration parameters of the operational system. This subsection reports how those parameters behaved in deployment, so that their operating points can be inspected rather than assumed. The analysis covers 1074 criterion-level decisions recorded across 12 assessment criteria belonging to two rubric families: a five-criterion rubric for spoken responses and a five-criterion rubric for written answers in a software-engineering course.

Across these decisions the criterion-level operational confidence had a mean of 0.837 and ranged from 0.50 to 1.00. Table 7 reports the distribution of decisions across the three confidence tiers together with the mean criterion score observed in each tier.

Table 7. Distribution of criterion-level decisions across confidence tiers.

Confidence Tier	Criterion Decisions	Share of Decisions	Mean Criterion Score
Tier 1 (confidence ≥ 0.90)	480	44.7%	33.4%
Tier 2 (0.70 ≤ confidence < 0.90)	331	30.8%	40.2%
Tier 3 (confidence < 0.70)	263	24.5%	18.9%
Total	1074	100.0%	—

The lowest-confidence tier corresponds to the lowest mean criterion score. This is the behaviour the escalation rule presupposes: the system reports least confidence on responses that are weakest, shortest, or least intelligible, which are precisely the cases in which an unreviewed preliminary score would be least defensible. The pattern is consistent across both rubric families; the criterion with the highest mean confidence (0.914) is also the criterion with the lowest mean score (20.4%), which corresponds to responses in which a required element was confidently identified as absent.

The gray-zone threshold governs secondary analysis. Of 215 scored responses, 84 (39.1%) received a first-pass score within the 40–70% band and were reassessed once by a stronger model, so the threshold referred approximately two responses in five for additional processing; after reassessment, 55 final scores (25.6%) remained within the band. At the session level, 9 of 168 sessions (5.4%) were flagged for review by the input-quality and behaviour-signal conditions. The behaviour-signal subsystem recorded 303 integrity signals in 54 sessions, most of them losses of window focus (165) and hidden-tab events (51).

The 215 scored responses were processed along three routes of the model router: gemini-2.5-flash-lite scored 122 responses (mean score 23.0% of the maximum), gemini-2.5-flash scored 84 (49.2%), and deepseek-chat, the fallback provider, scored 9 (22.1%). Because the stronger model is invoked on gray-zone responses, these means describe different response populations and are not a comparison of model quality.

These figures describe the measured operating points of the thresholds; they are not a calibration. Establishing optimal threshold values would require ground-truth labels against which the trade-off between escalation rate and undetected error could be scored, and such labels were not available for the deployment reported here. The limitation and the planned calibration study are stated in Section 5.

4.5. End-to-End Traversal of the Decision Pipeline

Section 4.1 and Section 4.3 report the assistant and examiner subsystems separately. This subsection traces the complete decision pipeline specified in Section 3.10 on the operational data, so that the integrated architecture is evidenced as a whole rather than as two components presented together.

The traversal covers 168 examination sessions. For each stage of the specified sequence the system persists the artefact the architecture requires: retrieval operates over a course corpus of 1665 indexed passages; spoken responses are transcribed, yielding 113 transcripts, each with a recognizer confidence; response features and evidence passages are extracted; criterion-level analysis produces the 1074 scored decisions described in Section 4.4; criterion-level uncertainty is aggregated into the overall operational uncertainty; a confidence tier is assigned; the decision is routed for teacher verification; and the decision object reaches a recorded terminal state (Table 8).

Table 8. Terminal states of examination sessions in the deployment.

Recorded Session State	Sessions	Share
Preliminary assessment completed by the system	108	64.3%
Verified and completed by the teacher	23	13.7%
Created, not started by the candidate	20	11.9%
Flagged for review	9	5.4%
In progress at the time of extraction	6	3.6%
Abandoned	2	1.2%
Total	168	100.0%

Two architectural claims were verified directly against the stored decision objects rather than against the specification. First, every one of the 1074 criterion-level decisions carries a non-empty verbatim evidence passage drawn from the student’s own response, so the mandatory evidentiary basis described in Section 3.11.1 holds in practice at a coverage of 100%. Second, for every finalized session the stored overall preliminary score reproduces the programmatic weighted aggregation of the stored criterion-level scores, which confirms that the overall score is computed outside the language model as specified in Section 3.6.

For spoken responses the system recorded one recognizer confidence per transcript. Over the 113 transcripts this confidence had a mean of 0.770 (SD 0.282, range 0.10–1.00); 19 transcripts (16.8%) fell below 0.70, 16 of them failed transcription attempts recorded with the minimum value of 0.10. The transcripts were produced by gemini-2.5-flash (75, including the 16 failed attempts) and, on 12–13 May 2026, by the Vosk recognizer running in the browser (38); word-level timestamps were stored for 33 of them.

This traversal is a demonstration of traceability, not a validation of end-to-end accuracy. It establishes that each specified stage executes, persists its required artefact, and transitions the decision object to a recorded terminal state, and that the two structural guarantees above hold on real data. Whether the resulting preliminary scores agree with human judgment is a separate question; a preliminary blind calibration is reported in Section 4.6, and its limits are discussed in Section Limitations and Planned Validation.

4.6. Preliminary Agreement with Human Raters

At the time of writing, three raters had contributed 94 ratings: two completed their full assignment of 35 answers and one rated 24; the rater assigned only to spoken answers had not yet started. Every sampled answer received two or three ratings (median three).

For the 13 spoken answers, agreement among the raters was positive but modest: Krippendorff’s α ranged from 0.35 to 0.58 across the five criteria, below the value of 0.667 that Krippendorff regards as the lowest acceptable for tentative conclusions [69]. Against this reference, the AI examiner ordered the spoken answers consistently with the raters: on the 0–100 total, Pearson’s r was 0.75 (95% CI 0.34–0.92); at the criterion level, Spearman’s ρ ranged from 0.58 to 0.81, quadratic-weighted κ was 0.34, and 68% of AI bands lay within one band of the rater mean. The AI was, however, systematically more severe: its totals were on average 21.8 points lower than the rater mean (95% CI 14.3–29.3), with Bland–Altman limits of agreement from −46.2 to 2.7 points.

The 22 written answers cannot be interpreted. Inter-rater α was negative for every criterion (−0.37 to −0.34); each rater assigned an almost constant band across all written answers (within-rater standard deviation 0.0–0.3 bands; median rating time 8–9 s per answer); and two of the three raters are language teachers rather than teachers of the course subject. Without a reliable human reference, the near-zero correlation between AI and rater totals in this stratum (r = 0.05) carries no information about the AI examiner.

These results do not establish automated-scoring accuracy. They indicate that, for spoken answers, the AI’s ordering of responses is broadly consistent with human raters while its absolute scores are markedly stricter, which is consistent with the architectural decision to treat AI scores as preliminary and to require teacher confirmation, revision or rejection before any score becomes final.

5. Discussion

The results should be interpreted at two levels. At the architectural level, they demonstrate the feasibility of a unified criteria-governed pipeline integrating information retrieval, answer generation, student-response analysis, evidence construction, uncertainty estimation, and teacher verification. At the empirical level, they support the technical operability of the AI assistant and the acceptability of the AI-examiner interaction procedure. They do not, however, establish automated-scoring accuracy or justify autonomous use of the system for final grading.

From an information-systems perspective, the key result is the separation of probabilistic and deterministic pipelines. The probabilistic pipeline performs retrieval, speech recognition, semantic analysis, and the generation of structured recommendations. The deterministic pipeline applies predefined weights, verifies that required evidence is present, calculates the overall preliminary indicator, records threshold violations, and manages transitions between decision states. The language model consequently remains a replaceable service component and does not control the full lifecycle of an academic decision.

The central finding is that the AI assistant and AI examiner can be implemented as two modes of a single intelligent system for producing and explaining decisions. Technically, such a system requires more than a language model: it must also include formalized criteria, an evidentiary layer, an uncertainty layer, audit logging, and a human-in-the-loop verification pathway.

The proposed architecture reduces the risk of uncontrolled use of a generative model. The language model does not make the final decision; it performs bounded functions such as text interpretation, feature extraction, structured-explanation generation, and preparation of a preliminary judgment [73]. The teacher retains final authority in consequential cases. This restriction on AI autonomy is consistent with the risk-based approach to educational AI systems [6], under which systems used to assess learning outcomes are classified as high-risk and are therefore subject to requirements for human oversight, transparency, interpretability, record keeping, and a person’s ability to disregard or override the system output. This allocation of authority is consistent with the human-agency principle of the Safe AI in Education Manifesto, which places educational decisions under human control [74], and with the hybrid-intelligence view of human–AI collaboration, in which the system extends rather than replaces the teacher’s agency [75].

An earlier component-level pilot of the AI assistant found that teachers rated the factual and terminological accuracy of the responses most highly, whereas comprehensive treatment of complex topics remained more vulnerable [67]. In this study, those findings are not presented as independent, repeated evidence of quality. Rather, they provide an empirical rationale for including evidence-control, teacher-verification, and verified-answer accumulation modules in the overarching architecture.

The principal limitation at this stage is the absence of a student-response corpus independently annotated by multiple teacher-experts, which precludes calculation of automated-scoring accuracy metrics. The findings should therefore not be interpreted as evidence of pedagogical effectiveness, complete fairness, or actual system accuracy [72,76]. They support only technical feasibility, procedural clarity, favorable user perceptions, and the pre-validation status of the system; moreover, the survey data reflect a limited pilot. Rigorous future validation should include multi-expert annotation, analysis of inter-rater agreement and teacher revisions, criterion-level comparison of AI preliminary judgments with expert ratings, and a separate examination of how automatic speech-recognition errors affect the final decision [77,78].

The reliability of an educational AI system cannot be reduced to language-model confidence or retrieval quality. At the institutional level, student, teacher, and administrator roles must be separated; access to course materials and responses must be controlled; models and versions must be logged; the grounds for automated inferences must be recorded; personal data must be protected; and every preliminary decision must remain reversible by the responsible teacher. The proposed architecture addresses these requirements through authorization, separate user interfaces, storage of evidence passages, model-call auditing, and mandatory teacher verification of assessment decisions.

Limitations and Planned Validation

The scope of the evidence presented here is narrower than the scope of the architecture, and we state the boundary explicitly so that the contribution is not read more strongly than the data support.

First, and most importantly, this study does not establish automated-scoring accuracy. The teacher records accumulated during the deployment cannot serve as a reference because they were produced while verifying the grading interface rather than while assessing student work: of 23 finalized sessions, 21 carry the maximum score on every rubric criterion. The preliminary calibration in Section 4.6 is small: its interpretable part is limited to 13 spoken answers, inter-rater reliability did not reach conventional thresholds, two of the three contributing raters are co-authors of this study, and the written-answer ratings could not be interpreted. It indicates a direction, consistent ordering with marked severity on spoken answers, rather than a validated level of agreement.

Second, the threshold values reported in Section 4.4 describe measured operating points, not a calibration. Determining optimal thresholds requires ground-truth labels against which the trade-off between escalation rate and undetected error can be scored; such labels are not yet available at the required scale (Section 4.6). The reported tier distribution and gray-zone selection rate therefore characterize how the configured thresholds behaved, not whether their values are optimal.

Third, no horizontal comparison against external human-in-the-loop grading systems was performed. Two constraints are binding: the evaluation corpus consists of student examination responses collected under an institutional authorization that does not permit transferring them to third-party platforms, and the comparison metrics proposed in the literature are defined against human ground truth, which is available here only for the small preliminary sample of Section 4.6. The internal comparison across the three routed models reported in Section 4.4 is not a substitute, because the stronger model is invoked on gray-zone responses and therefore operates on a different response population.

Fourth, the propagation of transcription error into criterion-level scores was not quantified. The system records a recognizer confidence for every spoken response, and its distribution is reported in Section 4.5, but recognizer confidence is a proxy for transcription quality rather than a measurement of transcription error. A quantitative error-propagation analysis requires reference transcripts produced by human transcription of the same audio, which were not collected; the confirmatory study will therefore collect reference transcripts at the time of recording.

Fifth, the Bayesian reliability estimation for the verified-answer repository is implemented but not empirically validated. In the present deployment the repository governs too few records to support the controlled comparison that validation would require, and we therefore present this component as a specified architectural extension rather than as an evaluated mechanism.

Sixth, the empirical base is confined to one institution, two rubric families and a single academic period, and the student survey reflects 52 respondents. The findings characterize feasibility and procedural acceptability in that setting; they do not support generalization to other disciplines, languages or assessment cultures.

The instrumentation required to remove the first two limitations is in place. The calibration module draws a stratified sample with a fixed seed, collects blind criterion-level ratings through personal rater links, records rating time, and computes Krippendorff’s α, AI–rater κ, correlation with confidence intervals, systematic bias and Bland–Altman limits directly from the retained data. What remains is a larger rating round by independent teachers of the course subjects who are not co-authors, preceded by rater training on the band descriptors and accompanied by monitoring of rating time, so that inter-rater reliability reaches a level at which agreement, systematic bias, the teacher revision rate and a threshold sensitivity analysis can be interpreted. We regard this as the necessary next step before any claim about automated-scoring accuracy can responsibly be made.

6. Conclusions

This study developed and implemented a criteria-governed, human-in-the-loop information system architecture integrating formative AI-assisted learning with the preliminary assessment of student responses. Its distinguishing feature is that results are represented not as unconstrained language-model output, but as structured objects containing the criteria, criterion-level indicators, supporting evidence, input quality, uncertainty, result provenance, and teacher-verification status. The architecture separates probabilistic intelligent processing from deterministic preliminary-decision construction, supports audit logging and controlled escalation, and enables teachers to confirm, revise, or reject system outputs.

Pilot data supported the technical operability of the AI-assistant mode and the procedural acceptability of the AI-examiner mode. A preliminary blind calibration showed that, on spoken answers, the AI examiner orders responses consistently with human raters while scoring them markedly lower, which reinforces the need for teacher verification. Although the findings do not establish automated-scoring accuracy, they demonstrate the feasibility of a governable and auditable educational AI system in which the language model supports decision making without displacing the teacher’s professional responsibility.