From Black-Box Grading to Pedagogically Aligned AI Assessment: A Hybrid LLM–RAG Framework for Explainable and Scalable Automated Code Evaluation

Featured Application

The proposed hybrid LLM–RAG framework is deployed in EvaluaTeC, a real-world platform used to automatically grade and provide pedagogically aligned feedback on programming assignments, validated on 1287 student submissions across engineering courses at Universidad Politécnica de Madrid.

Abstract

Automated assessment of programming assignments remains a major challenge in higher education, particularly in large-scale courses where timely, consistent, and pedagogically meaningful feedback is difficult to provide, while Large Language Models (LLMs) have shown strong capabilities in code understanding and feedback generation, their use as standalone evaluators is fundamentally limited by inconsistency, lack of transparency, and weak alignment with instructional objectives. This paper argues that these limitations are not intrinsic to LLMs, but rather arise from their deployment as isolated components. In response, we propose a system-centric approach to AI-assisted assessment, introducing a hybrid framework that integrates LLMs within a structured, context-aware, and pedagogically aligned evaluation pipeline. The framework combines (i) explicit rubric-based decomposition of evaluation criteria, (ii) pedagogically guided prompting, and (iii) Retrieval-Augmented Generation (RAG) grounded in course-specific materials. Together, these components transform the evaluation process from a black-box prediction task into a traceable and reproducible decision process. The proposed approach is implemented in a real-world educational platform, EvaluaTeC, and evaluated on a dataset of 1287 programming submissions from 429 students. Experimental results show that the hybrid framework improves agreement with consolidated instructor reference grades ($r=0.9059$ vs. $0.7207$ baseline), reduces evaluation error (MAE = 0.5134), and exhibited lower output variability in the recorded aggregate statistics, while maintaining practical latency and cost. Beyond numerical improvements, the system approximates key statistical properties of human grading and generates structured, pedagogically aligned feedback. These findings demonstrate that reliable AI-assisted assessment emerges from the integration of LLMs within structured and context-aware systems, rather than from model capabilities alone. This work contributes a principled framework for explainable and scalable automated assessment, advancing the design of trustworthy AI systems in education. This shift reframes automated assessment as a systems problem rather than a purely model-centric task.

1. Introduction

The rapid advancement of Large Language Models (LLMs) has fundamentally reshaped the landscape of programming education, enabling automated code understanding, feedback generation, and interactive tutoring at an unprecedented scale [1,2,3]. These capabilities have sparked increasing interest in leveraging LLMs for automated assessment, particularly in large-scale courses where traditional grading approaches struggle to provide timely, consistent, and pedagogically meaningful feedback [4,5,6,7,8].

However, the direct application of LLMs to high-stakes educational assessment introduces critical challenges. When deployed as standalone evaluators, LLMs exhibit stochastic grading behavior, sensitivity to prompt formulation, lack of transparency, and susceptibility to hallucinations. These limitations undermine reliability and reproducibility, which are essential requirements in educational contexts. As a result, current LLM-based grading systems often behave as black-box evaluators, producing outputs that are difficult to interpret, justify, or align with instructional objectives [6,7,8,9,10,11].

This reveals a fundamental mismatch between the probabilistic nature of generative models and the normative requirements of educational assessment. While LLMs excel at semantic reasoning and natural language generation, they lack inherent mechanisms to enforce consistency, incorporate domain-specific context, or adhere to structured evaluation criteria. Consequently, similar student submissions may receive inconsistent evaluations, and generated feedback may be technically plausible yet pedagogically misaligned [12,13,14,15].

Despite recent progress, existing approaches suffer from three fundamental limitations. First, they are predominantly model -centric, treating evaluation as a prompt engineering problem rather than a system design challenge. Second, they lack pedagogical grounding, failing to explicitly incorporate structured evaluation criteria such as rubrics or learning outcomes. Third, they do not provide system-level integration, overlooking the need for architectures that ensure traceability, scalability, and robustness. As a result, current solutions remain insufficient for reliable deployment in real educational environments [15,16,17,18].

In this work, we argue that automated assessment with LLMs must be reframed as a system-level problem rather than a purely model-driven task. We argue that grading reliability depends not only on the underlying LLM, but also on the structured integration of contextual and pedagogical mechanisms within the evaluation system [18].

To operationalize this hypothesis, we propose a hybrid framework for AI-assisted programming assessment that integrates three key components: (i) rubric-based decomposition of evaluation criteria, which constrains the evaluation process and aligns it with instructional objectives; (ii) structured prompting with pedagogical guidance, which reduces ambiguity and improves consistency; and (iii) Retrieval-Augmented Generation (RAG), which grounds model outputs in course-specific materials, providing contextual accuracy and reducing hallucinations. These components are embedded within a modular system architecture designed to ensure scalability, traceability, and reproducibility.

The proposed framework is implemented in a real-world educational platform, EvaluaTeC (available at https://evaluatec.industriales.upm.es, accessed on 3 February 2026), and evaluated on a dataset of 1287 programming submissions from 429 students across multiple engineering disciplines. Experimental results demonstrate that the hybrid approach substantially improves agreement with consolidated instructor reference grades ($r=0.9059$, versus $0.7207$ for the baseline), reduces evaluation error (MAE = $0.51$), and exhibited lower output variability in the recorded aggregate statistics, compared to baseline LLM-based methods. Furthermore, the system reproduces key statistical properties of human grading and generates structured, pedagogically aligned feedback.

This paper makes three main contributions:

• Conceptual contribution: We introduce a system-centric perspective on automated assessment, reframing LLM-based grading as an architectural problem rather than a model optimization task.
• Methodological contribution: We propose a hybrid LLM–RAG framework that integrates structured rubrics, contextual grounding, and guided prompting to improve reliability, consistency, and pedagogical alignment.
• Empirical contribution: We provide a large-scale evaluation in real educational settings, demonstrating significant improvements in grading accuracy, consistency, and statistical alignment with human evaluators.

Unlike prior work, which primarily focuses on prompt optimization or model selection, this study emphasizes system design as the primary driver of reliable AI-assisted assessment. By shifting the focus from model capabilities to structured integration, this work contributes to the development of trustworthy, scalable, and pedagogically grounded AI systems for education.

2. Related Work

2.1. Automated Code Assessment

Automated code assessment has been extensively studied in computer science education, particularly in the context of large-scale courses where manual grading is infeasible [19,20]. Traditional approaches are primarily based on test-case execution frameworks, where student submissions are evaluated against predefined input–output specifications [21,22]. Widely used systems such as CodeRunner, AutoGrader, and competitive programming platforms provide scalable and objective evaluation of functional correctness [23,24,25]. However, these approaches are inherently limited to verifying observable program behavior and fail to capture higher-level aspects such as semantic correctness, algorithmic efficiency, code quality, and the reasoning process underlying a solution.

To address these limitations, prior work has explored static code analysis and rule-based feedback systems, which aim to provide richer insights into code structure, style, and potential errors, while these methods offer improvements in feedback quality, they require extensive manual rule definition and are difficult to generalize across diverse programming tasks and domains. Intelligent Tutoring Systems (ITSs) further extend this paradigm by incorporating domain knowledge and adaptive feedback strategies, but their scalability remains constrained due to the high cost of knowledge engineering [26,27].

More recently, LLMs have introduced a paradigm shift in automated code assessment. Unlike traditional approaches, LLMs enable semantic analysis of code and can generate natural language feedback that explains errors, suggests improvements, and supports conceptual understanding. This capability positions LLMs as promising candidates for addressing the long-standing limitations of automated grading systems [28,29].

However, existing LLM-based approaches remain fundamentally limited when applied to assessment. Empirical studies indicate that LLM outputs exhibit stochastic variability, prompt sensitivity, and lack of calibration, leading to inconsistent grading across semantically equivalent submissions. Moreover, these systems often operate without explicit evaluation criteria, resulting in feedback that is technically plausible but misaligned with instructional objectives. As a consequence, current LLM-based grading systems struggle to meet the reliability and accountability requirements of educational assessment [30,31,32].

Overall, prior approaches present a trade-off between scalability and pedagogical richness: traditional systems offer reliability but limited feedback, while LLM-based systems provide richer feedback but lack consistency and alignment as shown in

Table 1. Comparison of Automated Code Assessment Approaches.

Approach	Strengths	Limitations
Test-based systems	Scalable, objective	No semantic understanding
Static analysis	Code quality insights	Limited context awareness
ITS/rule-based	Personalized feedback	Hard to scale
LLM-based (baseline)	Semantic understanding, natural feedback	Inconsistent, unaligned, black-box

. Figure 1 illustrates the conceptual evolution of these code assessment methods. These limitations motivate the need for a new generation of automated assessment systems that integrate the semantic capabilities of LLMs with structured pedagogical criteria, contextual knowledge, and robust architectural design—an approach that we address in the following sections.

2.2. LLMs in Education

The rapid advancement of LLMs has significantly influenced the landscape of educational technologies, enabling new forms of interaction, content generation, and intelligent assistance. In recent years, LLMs have been increasingly integrated into educational contexts as virtual tutors, conversational agents, and feedback generators, supporting both students and instructors in a wide range of learning activities [33,34,35,36].

One of the most prominent applications of LLMs in education is their use as intelligent tutoring systems, where they provide real-time explanations, answer questions, and guide students through problem-solving processes [33]. Unlike traditional rule-based systems, LLMs can generate contextually rich and natural language responses, allowing for more flexible and adaptive interactions. This capability has been particularly valuable in programming education, where understanding conceptual errors and reasoning about code is essential for learning [37,38].

In addition to tutoring, LLMs have been employed to support automated feedback generation in educational settings [39]. By analyzing student submissions, these models can produce detailed comments that explain errors, suggest improvements, and highlight best practices. This represents a significant advancement over traditional automated systems, which typically provide limited or purely functional feedback. Empirical studies have shown that such AI-generated feedback can enhance student engagement and contribute to more effective learning processes, particularly when the feedback is timely and personalized [39,40].

Furthermore, LLMs enable the development of adaptive and scalable learning environments, where instructional content and assessment can be dynamically tailored to individual student needs. This aligns with broader trends in educational technology that emphasize personalization and continuous assessment. In programming courses, where large student cohorts are common, LLM-based systems offer the potential to deliver individualized feedback at scale, addressing one of the most significant challenges faced by educators.

Despite these promising capabilities, the integration of LLMs into education also presents several critical challenges. A primary concern is the reliability and consistency of model outputs, as LLMs may generate responses that are partially incorrect, incomplete, or misleading. This issue is particularly relevant in assessment contexts, where inaccuracies can directly affect student evaluation and learning outcomes. Additionally, LLM-generated feedback may lack alignment with specific course objectives or pedagogical frameworks, limiting its educational effectiveness [41,42].

Another important challenge relates to transparency and explainability [41]. LLMs operate as probabilistic models, and their decision-making processes are often opaque, making it difficult for both instructors and students to understand the rationale behind the generated feedback. This lack of interpretability raises concerns regarding fairness, accountability, and trust in AI-assisted educational systems [41,43,44].

Moreover, the use of LLMs in education introduces ethical and regulatory considerations, particularly in relation to data privacy, bias, and the potential over-reliance on automated systems [12,13]. Educational applications of AI are increasingly subject to regulatory frameworks that emphasize human oversight, transparency, and the protection of student data. As a result, LLM-based systems must be designed not only for performance but also for compliance with emerging standards of responsible AI [14,17,45].

In summary, while LLMs offer substantial potential to enhance teaching and learning through intelligent assistance, personalized feedback, and scalable solutions, their effective adoption requires addressing challenges related to reliability, pedagogical alignment, and system transparency. These limitations highlight the need for structured approaches that integrate LLM capabilities within well-defined educational frameworks.

Table 2. Applications of LLMs in Education.

Application	Benefits	Limitations
Virtual tutors	Personalized explanations	Possible inaccuracies
Automated feedback	Immediate, scalable	Lack of alignment with curriculum
Content generation	Flexible and adaptive	Quality variability
Assessment support	Semantic understanding	Lack of reliability and explainability

summarizes key applications of LLMs in education, while Figure 2 illustrates their role within the educational pipeline, showing the key roles LLMs play in transforming student input into comprehensive learning support.

These challenges underscore the necessity of moving beyond isolated LLM-based applications toward integrated architectural solutions that ensure reliability, pedagogical alignment, and scalability—an aspect that is addressed through emerging frameworks for LLM integration in software systems.

2.3. Architectural Approaches for LLM Integration

The increasing adoption of LLMs in real-world applications has prompted a conceptual shift from treating these models as standalone tools to integrating them as core components within complex software systems. This perspective is particularly relevant in educational contexts, where LLMs must operate reliably, interact with multiple subsystems, and support critical processes such as automated assessment and feedback generation. Consequently, recent research has focused on the development of architectural paradigms that enable the effective, scalable, and responsible integration of LLMs into such environments [4,46].

A prominent trend in this direction is the encapsulation of LLMs as autonomous or semi-autonomous services within modular architectures. Rather than embedding models directly into monolithic applications, contemporary system designs frequently adopt microservice-based approaches, where LLMs are deployed as independent services communicating through well-defined interfaces. This architectural choice enhances flexibility, scalability, and maintainability, as individual components can be updated, replaced, or scaled independently. Moreover, it facilitates the integration of multiple specialized models, enabling more sophisticated and distributed processing workflows [47,48,49].

Beyond microservice architectures, the emergence of hybrid cognitive systems represents another significant development. These systems combine LLMs with symbolic reasoning techniques, rule-based components, and external knowledge sources to address the limitations of purely generative approaches. By integrating complementary methodologies, hybrid systems introduce structure, validation, and contextual grounding into the decision-making process. For example, combining LLM-based semantic analysis with traditional code evaluation techniques—such as static analysis or test-case execution—can substantially improve the reliability, interpretability, and pedagogical relevance of automated assessment systems. This integration is particularly valuable in educational domains, where both correctness and explainability are essential [41,46,47,48,49].

In addition, architectural patterns such as RAG and workflow orchestration have gained increasing attention as mechanisms for enhancing LLM performance and robustness. RAG-based approaches enable systems to dynamically retrieve relevant information from external knowledge sources and incorporate it into model prompts, thereby reducing hallucinations and improving contextual accuracy. Similarly, orchestration frameworks allow LLMs to coordinate interactions with external tools and services, effectively acting as controllers that delegate subtasks to specialized components. These patterns contribute to the development of more context-aware and resilient systems capable of handling complex, multi-stage processes. Figure 3 illustrates a reference architecture that integrates these components within an educational setting [47].

Despite these advancements, several challenges remain in the integration of LLMs into software architectures. Scalability is a primary concern, as LLMs typically require substantial computational resources, particularly in real-time environments with large user populations. Efficient resource allocation, model optimization, and system design are therefore essential to ensure practical deployment in large-scale educational contexts.

Another critical challenge is ensuring traceability and accountability. Given the probabilistic nature of LLM outputs, it is necessary to maintain detailed records of inputs, outputs, and intermediate processing steps to support auditing, debugging, and evaluation. This requirement is especially important in educational applications, where assessment decisions must be transparent, reproducible, and justifiable. Accordingly, architectural designs must incorporate comprehensive logging, monitoring, and explanation mechanisms.

Finally, issues related to hallucinations and output reliability remain central concerns. Without appropriate safeguards, LLMs may generate inaccurate or misleading information, potentially compromising the integrity of the system. Addressing this limitation requires the integration of validation layers, contextual grounding mechanisms (e.g., RAG), and, where appropriate, human oversight to ensure that system outputs meet required standards of accuracy and trustworthiness.

In summary, contemporary architectural approaches emphasize the integration of LLMs within modular, hybrid, and context-aware systems that mitigate their inherent limitations while leveraging their strengths. However, despite the emergence of these design patterns, there remains a lack of comprehensive frameworks specifically tailored to educational assessment scenarios, where requirements for scalability, explainability, and pedagogical alignment intersect.

Table 3. Architectural Patterns for LLM Integration.

Pattern	Description	Benefits	Limitations
Microservices	LLM as independent service	Scalability, modularity	Integration complexity
Hybrid systems	LLM combined with symbolic methods	Reliability, explainability	Higher design complexity
RAG	Integration with external knowledge sources	Reduces hallucinations	Dependency on data quality
Orchestrated workflows	LLM coordinates multi-step tasks	Flexibility	Latency overhead

summarizes key architectural patterns for LLM integration.

These limitations highlight the need for structured frameworks that not only incorporate LLMs as modular components but also embed pedagogical alignment and contextual grounding mechanisms, particularly in the domain of automated educational assessment.

2.4. Limitations of Current Approaches

Despite substantial progress in automated code assessment and the increasing adoption of LLMs in educational contexts, existing approaches remain fundamentally limited when examined from both pedagogical and systems perspectives [46,47,48,49]. These limitations restrict their applicability in real-world educational environments, particularly in scenarios where reliability, alignment with learning objectives, and scalability are essential [13].

A primary limitation of current LLM-based approaches is the lack of contextual grounding. When deployed in isolation, LLMs operate without access to course-specific materials, instructional guidelines, or domain constraints, often resulting in inconsistent or contradictory evaluations. The absence of such contextual information increases the likelihood of hallucinations and reduces the coherence and relevance of the generated feedback. Consequently, similar student submissions may receive divergent evaluations due to minor variations in prompts or stochastic model behavior, thereby undermining the reliability of the assessment process.

Closely related to this issue is the lack of pedagogical alignment [50]. Most existing systems do not explicitly incorporate structured evaluation criteria derived from course objectives, such as grading rubrics or defined learning outcomes [51]. As a result, while LLMs may generate technically accurate feedback, such feedback is frequently misaligned with instructor expectations and intended pedagogical goals. This disconnect diminishes the educational value of automated assessment and limits instructor confidence in AI-assisted evaluation [51,52].

Another critical limitation lies in the absence of comprehensive system-level integration. Many current approaches treat LLMs as standalone components, focusing primarily on prompt engineering or model optimization without addressing broader architectural requirements [46,47]. This leads to fragmented solutions that lack essential properties such as scalability, robustness, and traceability. In contrast, real-world educational platforms require tightly integrated systems that combine multiple components, including code evaluation engines, data management layers, user interfaces, and monitoring infrastructures. The lack of such holistic designs significantly constrains the deployment of LLM-based assessment systems in authentic academic settings [51].

Furthermore, existing approaches often fail to ensure evaluation consistency and auditability. In educational contexts, assessment outcomes must be transparent, reproducible, and justifiable [12,13,51]. However, the probabilistic nature of LLM outputs, combined with insufficient logging and traceability mechanisms, makes it difficult to reproduce or explain evaluation decisions. This limitation raises important concerns regarding fairness, accountability, and compliance with emerging regulatory standards for AI in education [13].

In addition, the integration of hybrid evaluation strategies remains underexplored. Current systems rarely combine LLM-based reasoning with traditional assessment techniques, such as test-case execution or static code analysis. As suggested in prior research, hybrid approaches can enhance both reliability and interpretability by leveraging complementary strengths. Nevertheless, such integrations are often implemented in an ad hoc manner rather than as part of a coherent and principled architectural framework.

Taken together, these limitations reveal a fundamental gap between the capabilities of current AI-driven approaches and the requirements of robust, pedagogically meaningful assessment systems. Existing solutions tend to prioritize scalability at the expense of feedback quality, or alternatively, provide rich feedback without ensuring consistency, alignment, and system-level reliability.

These challenges motivate the development of a new generation of automated assessment systems that integrate LLMs within modular, context-aware, and pedagogically aligned architectures. In response, the following section introduces the proposed hybrid framework, which combines structured evaluation, Retrieval-Augmented Generation, and a scalable architectural design to enable reliable, explainable, and educationally effective code assessment in higher education.

Despite recent advances in LLM-based feedback generation and retrieval-augmented educational systems, existing approaches still address these dimensions in isolation. Most prior work focuses either on prompt engineering, pedagogical feedback, or contextual retrieval, but lacks integrated frameworks that jointly optimize: (i) pedagogical alignment through structured evaluation criteria, (ii) system-level traceability and explainability, and (iii) statistical reliability under real-world assessment conditions. Recent rubric- and feedback-oriented systems illustrate this fragmentation: CodEv [7] and BeGrading [9] target consistent and constructive feedback but do not incorporate course-specific retrieval, whereas Rubric Is All You Need [51] shows the value of question-specific rubrics without contextual grounding or a reproducibility analysis. Furthermore, only a limited number of studies validate their approaches using large-scale authentic educational datasets while simultaneously analyzing consistency and reproducibility of grading behavior.

Table 4. Comparison of recent LLM-based educational assessment approaches.

Work	Rubrics	RAG	Real Dataset	Consistency Analysis
Tseng et al. [7]	No	No	Yes	No
Emirtekin et al. [6]	No	No	Yes	No
Aditya et al. [51]	Yes	No	No	No
Chondamrongkul et al. [47]	No	Yes	No	No
Barredo et al. [41]	Partial	No	No	No
This work	Yes	Yes	Yes	Yes

positions our work against these approaches along the dimensions of rubric use, retrieval augmentation, use of a real dataset, and consistency analysis. We emphasize that this comparison is conceptual: because these systems were evaluated on different datasets, tasks, and (often proprietary or unavailable) implementations, a fully controlled head-to-head reproduction under a unified API, prompt budget, and hardware configuration was outside the scope of this study and is left for future work.

3. Proposed Framework

This section introduces the proposed hybrid framework, which constitutes the central contribution of this work. The framework is designed to transform LLM-based code assessment from a predominantly black-box process into a structured, explainable, and pedagogically aligned system. To achieve this, it integrates architectural design principles, contextual knowledge through RAG, and explicit evaluation criteria grounded in instructional practices.

3.1. Design Principles

The framework is guided by a set of design principles derived from both software engineering best practices and the specific requirements of educational assessment systems. These principles ensure that LLMs are integrated as reliable and accountable components within a broader system architecture.

Modularity. The system is structured as a collection of loosely coupled components, allowing independent development, maintenance, and substitution of modules such as the LLM engine, evaluation logic, and retrieval subsystem. This design enables extensibility and facilitates experimentation with alternative models and configurations.

Scalability. To accommodate large student cohorts typical of programming courses, the framework supports scalable deployment through distributed services and asynchronous processing. This ensures that high volumes of submissions can be handled efficiently without performance degradation.

Explainability. A central objective of the framework is to produce interpretable outputs. The system generates structured feedback explicitly linked to evaluation criteria, enabling both students and instructors to understand the rationale behind each assessment decision.

Traceability. All stages of the evaluation process are logged and structured to support auditing and reproducibility. Each assessment can be traced back to its inputs, retrieved contextual information, applied criteria, and generated outputs, addressing transparency requirements in educational environments.

Pedagogical alignment. The framework explicitly incorporates course objectives and instructional practices into the evaluation process. Structured rubrics and domain-specific knowledge derived from course materials guide the assessment, ensuring that feedback is consistent with instructor expectations.

Collectively, these principles extend existing architectural approaches for LLM integration by incorporating pedagogical constraints as first-class design elements, effectively bridging the gap between technical capabilities and educational requirements.

3.2. System Architecture

The proposed framework adopts a modular hybrid architecture that integrates multiple components to enable reliable and context-aware code assessment. Each component fulfills a specific role within the evaluation pipeline:

• LLM Engine: Responsible for the semantic analysis of student code and the generation of feedback. It operates using structured prompts augmented with contextual information and supports the use of multiple models for comparative evaluation.
• RAG Module: Retrieves relevant domain-specific knowledge from course materials previously provided by instructors and integrates it into the evaluation process to provide contextual grounding.
• Rubric Generator: Defines structured evaluation criteria aligned with learning outcomes. It decomposes the assessment into explicit dimensions, avoiding purely holistic judgments.
• Evaluation Engine: Orchestrates the overall assessment process by combining LLM outputs, rubric-based evaluations, and traditional analysis methods such as test execution.
• Data Layer: Stores student submissions, evaluation results, rubrics, logs, and retrieved contextual information, enabling traceability and system monitoring.
• User Interface (EvaluaTeC): Provides interaction mechanisms for students and instructors, including submission interfaces, feedback visualization, and configuration of evaluation parameters.

The interaction among these components forms a structured workflow in which each stage contributes to improving the reliability, consistency, and interpretability of the final evaluation.

Figure 4 illustrates the overall architecture of the proposed system.

3.3. Pedagogical Alignment Layer

A key innovation of the proposed framework is the introduction of a dedicated pedagogical alignment layer, which ensures that the evaluation process is explicitly grounded in instructional objectives and assessment criteria. The framework adopts a rubric-driven evaluation strategy in which submissions are assessed across explicit dimensions such as correctness, code quality, efficiency, and readability. Each rubric dimension was evaluated on a normalized 0–10 scale and included explicit descriptors to guide assessment consistency. The rubric used in the experiments decomposed the evaluation into four pedagogically motivated dimensions:

• Correctness: functional validity of the solution and compliance with exercise requirements.
• Code quality: structure, modularity, and adherence to programming best practices.
• Readability: clarity of variable naming, formatting, and code organization.
• Efficiency: appropriate algorithmic and computational resource usage.

Rather than requesting a single holistic score, each dimension is decomposed, for every exercise, into a set of discrete, checkable items (e.g., specific correctness conditions, style conventions, or efficiency requirements relevant to that exercise). For each item, the LLM returns a binary judgment (satisfied/not satisfied) together with a short justification; the model is not given the item weights, only the items to be checked. Weighting and aggregation are performed afterward, in the local backend: the proportion of satisfied items within each dimension yields a dimension score $s_d$ (rescaled to 0–10), and the final grade is computed as a weighted sum $\widehat{y}={\sum }_d{w}_d{s}_d$, where $w_d$ is the weight assigned to dimension d, with ${\sum }_d{w}_d=1$. Rather than being fixed globally, the weights $w_d$ are configured by the instructor on a per-exercise basis, so that the relative importance of correctness, code quality, readability, and efficiency can reflect the specific learning objectives of each task (for example, weighting correctness more heavily in an introductory exercise). In addition to the per-item judgments, the model output always includes two further qualitative fields: a justification for the set of responses, and a holistic comment providing improvement tips and an overall assessment of the submission; together with the per-item results, these form the basis of the feedback delivered to students. Instructor grades were normalized to the same 0–10 scale to ensure comparability across human and automated evaluations. No post hoc score calibration (e.g., isotonic regression or Platt scaling) was applied; the reported agreement therefore reflects the raw aggregated output of the system.

This rubric-driven approach enables the decomposition of complex evaluation tasks into smaller, interpretable components. Instead of producing a single aggregated judgment, the system generates partial evaluations for each criterion, which are subsequently combined into a final score. This not only improves consistency but also provides students with detailed, actionable feedback.

To further enhance consistency, the framework incorporates few-shot prompting techniques. Representative examples of evaluated submissions are included in the prompt, guiding the LLM toward a consistent interpretation of rubric criteria and reducing variability in outputs. The combination of explicit evaluation dimensions and guided examples significantly improves alignment with instructor expectations.

Overall, the pedagogical alignment layer transforms the evaluation process from an opaque, model-driven task into a structured, criteria-based assessment that closely mirrors the practices of human instructors.

3.4. Retrieval-Augmented Evaluation

To overcome the limitations of context-free LLM evaluation, the framework incorporates a RAG mechanism that integrates course-specific knowledge into the assessment process.

In this approach, instructors provide teaching materials, including lecture notes, slides, and other educational resources. These materials are processed and stored as a structured knowledge base reflecting the terminology, methodologies, and level of rigor expected in the course.

During evaluation, the RAG module retrieves the most relevant fragments of this knowledge base based on the student submission and the exercise description. The retrieved content is then incorporated into the LLM prompt, enabling the generation of feedback that is both technically accurate and pedagogically aligned with the course context.

The integration of RAG provides several key benefits:

• Improved coherence: Feedback is consistent with course content and instructional context.
• Increased precision: Domain-specific grounding reduces hallucinations and improves evaluation accuracy.
• Enhanced pedagogical alignment: Feedback reflects the language, examples, and conceptual emphasis used in teaching.

This approach enables a shift from generic evaluation toward context-aware assessment, where feedback is tailored to the specific educational environment. Figure 5 presents the retrieval-augmented evaluation pipeline.

The proposed framework unifies architectural design, pedagogical structure, and contextual knowledge into a coherent system for automated code assessment. By combining modular components, structured evaluation criteria, and retrieval-based contextualization, it addresses the key limitations of existing approaches and establishes a foundation for reliable, explainable, and scalable AI-assisted assessment systems.

4. System Implementation

This section describes the implementation of the proposed framework within a real-world educational environment. The objective is to operationalize the conceptual and architectural design into a functional system that demonstrates the feasibility of integrating LLMs, pedagogical alignment mechanisms, and contextual retrieval within a scalable platform for automated code assessment.

4.1. Platform Overview

The proposed framework has been implemented in EvaluaTeC, a web-based platform designed to support automated assessment and feedback generation in programming courses. The platform is publicly accessible at https://evaluatec.industriales.upm.es (accessed on 3 February 2026) and has been deployed in authentic academic settings to evaluate programming assignments and provide formative feedback to students.

A key feature of the platform is its support for controlled evaluation environments. During assessment sessions, the system enforces a full-screen mode to ensure academic integrity. If the student exits full-screen or the application loses focus, the submission is automatically finalized and delivered. This mechanism minimizes opportunities for external assistance and ensures that collected data accurately reflects student performance under controlled conditions.

EvaluaTeC follows a modular architecture in which each component is responsible for a specific stage of the evaluation pipeline. The system integrates multiple technologies to ensure scalability, flexibility, and interoperability:

• Backend framework: Laravel (PHP), responsible for user management, request handling, and orchestration of system components.
• Processing layer: Python-based services for code analysis, interaction with LLM APIs, and data processing.
• LLM integration: External LLM APIs, including OpenAI GPT-4 and Google Gemini 2.5, enabling semantic code analysis and feedback generation.
• Database layer: Relational database for storing users, submissions, evaluation results, and system logs.
• Execution environment: Secure sandbox for executing student code and validating functional correctness.

The evaluation process follows a structured pipeline: student submissions are processed by an evaluation engine that combines traditional techniques (e.g., test-case execution) with LLM-based analysis and contextual augmentation through RAG. The system produces both a quantitative score and detailed qualitative feedback. Figure 6 illustrates the overall architecture of the EvaluaTeC platform.

4.2. Prompt Design

The effectiveness of LLM-based evaluation is highly dependent on prompt design. In this work, prompts are systematically structured to ensure consistency, interpretability, and alignment with pedagogical objectives.

Each prompt is composed of three main components:

• Context definition: Description of the exercise, expected learning outcomes, and evaluation constraints.
• Student submission: Source code provided by the student, optionally including execution results or detected errors.
• Evaluation instructions: Explicit grading rubric, including criteria such as correctness, code quality, efficiency, and clarity, along with instructions for generating structured feedback.

A central element of the prompt design is the incorporation of few-shot learning. Representative examples of correct and incorrect solutions are included to guide the model toward consistent evaluation behavior and reduce output variability. Few-shot examples were manually selected by instructors to represent common correct solutions, partially correct implementations, and typical programming errors observed in prior course editions.

Furthermore, the evaluation process explicitly iterates over each rubric criterion, ensuring that every dimension is independently assessed. This structured decomposition improves both the interpretability and reproducibility of the evaluation outcomes.

Table 5. Prompt Structure for LLM-Based Evaluation.

Component	Description	Purpose
Context	Exercise description and objectives	Align evaluation with course
Submission	Student code	Input for analysis
Rubric	Structured evaluation criteria	Ensure consistency
Examples	Few-shot cases	Guide model behavior
Output format	Structured response	Improve interpretability

summarizes the structure of the prompts used in the system.

4.3. RAG Implementation

To overcome the limitations associated with context-free LLM evaluation, the proposed system integrates a RAG subsystem that incorporates domain-specific educational knowledge into the assessment workflow. By grounding the evaluation process in course materials, the system generates feedback that is pedagogically aligned, context-aware, and consistent with instructor expectations.

The RAG subsystem was implemented using educational resources extracted from the course textbook Fundamentals of Programming in C [53]. Following the methodology proposed in previous work [54], the documents were preprocessed and segmented into semantically coherent text chunks prior to embedding generation. This segmentation strategy preserves conceptual consistency while enabling fine-grained retrieval during inference.

The RAG pipeline consists of the following stages:

• Document processing: Course materials are collected and preprocessed. Textual content is extracted and divided into semantically meaningful chunks to facilitate efficient retrieval and contextual relevance. These materials reflect the terminology, conceptual scope, and pedagogical objectives defined by course instructors. Documents were segmented into chunks of approximately 300–500 tokens with overlapping windows to preserve semantic continuity.
• Embedding generation and indexing: Each text chunk is transformed into a dense vector representation using OpenAI’s text-embedding-3-small model. The resulting embeddings are indexed using a FAISS vector database with cosine similarity, enabling efficient semantic similarity search over the educational corpus.
• Context retrieval: During evaluation, the system performs semantic similarity retrieval using both the student submission and the exercise description as query inputs. The system selects the top-3 most semantically similar chunks ($k=3$) for prompt augmentation, a value chosen empirically to balance contextual coverage against prompt length.
• Prompt augmentation: The retrieved contextual fragments are incorporated into the LLM prompt as a dedicated context block, concatenated in decreasing order of similarity. No voting, summarization, or conflict-resolution mechanism is applied over the retrieved chunks; they are provided verbatim as grounding material, and the LLM is instructed to treat them as reference context rather than as authoritative solutions. These fragments typically include theoretical explanations, syntax examples, implementation guidelines, and canonical programming patterns relevant to the evaluated exercise.

The incorporation of retrieved educational context substantially improves the reliability and pedagogical consistency of the generated feedback. In particular, the grounding mechanism reduces hallucinations and aligns the evaluation with the instructional approach adopted in the course. This is especially important in educational environments where similar programming tasks may require different levels of abstraction, rigor, or implementation detail depending on the academic level and learning objectives (e.g., undergraduate versus graduate courses).

Furthermore, the use of RAG enables the assessment framework to adapt dynamically to course-specific content without requiring retraining of the underlying LLM. As a result, the system can maintain both technical accuracy and curricular alignment while supporting scalable and reusable automated assessment workflows.

A specific concern in retrieval-augmented assessment is the risk of solution leakage, i.e., the possibility that the retrieval step exposes reference solutions or answer keys to the model and thereby inflates agreement artificially. In our setting this risk is structurally limited: the knowledge base is built exclusively from the course textbook [53], which contains theoretical explanations, syntax descriptions, and generic programming patterns, but not the graded exercises, their expected outputs, or canonical solutions. Consequently, retrieval provides conceptual grounding rather than task-specific answers. Moreover, for every evaluation the identifiers of the retrieved chunks and their content are logged together with the prompt and the model output, so that the exact context supplied to the model can be reconstructed and audited by instructors, and, if required, disclosed to students.

Figure 7 illustrates the RAG-based evaluation pipeline.

4.4. Security and Academic Integrity

Given the sensitive nature of educational assessment, the system incorporates multiple mechanisms to ensure security, integrity, and controlled usage.

First, a secure sandbox environment is used to execute student code in isolation, preventing malicious behavior and ensuring system stability.

Second, the platform supports a controlled evaluation mode for supervised scenarios—such as examinations—in which:

• Access to external resources is restricted
• User actions are monitored and logged
• Evaluation conditions are standardized

Third, the system implements user authentication and role-based access control, ensuring that only authorized users can access specific functionalities. Instructors retain full control over evaluation parameters, including rubric design and feedback configuration.

Finally, all interactions with the LLM are logged, enabling full traceability and auditability of evaluation decisions. This is essential for ensuring transparency, fairness, and compliance with educational and regulatory requirements.

Table 6. Security and Integrity Mechanisms.

Mechanism	Description	Purpose
Sandbox execution	Isolated code execution	Prevent malicious behavior
Controlled mode	Restricted evaluation environment	Ensure assessment integrity
Access control	User roles and permissions	Secure system usage
Logging	Recording interactions and outputs	Enable traceability

summarizes the key security and integrity mechanisms implemented in the system.

4.5. Ethical Considerations and Data Protection

Because the system processes student-generated data and relies on third-party LLM APIs, data protection was treated as a first-class design requirement rather than an afterthought. The data handled by the platform fall into three categories: (i) student source-code submissions, (ii) evaluation results (scores, rubric breakdowns, and generated feedback), and (iii) interaction logs, including retrieved context and, in supervised mode, controlled-environment events. All of these are stored in the institutional database under the university’s data-protection framework, in line with the requirements of the EU General Data Protection Regulation (GDPR).

A key design decision concerns the separation between personal data and the data exposed to external services. Personally identifiable information (student name, institutional e-mail, and account details) is managed exclusively in the local database and is never transmitted to the LLM providers. The payload sent to the external APIs (OpenAI and Google) is restricted to the source code, the exercise statement, and the evaluation context (e.g., the rubric and retrieved course fragments); it contains no direct identifiers, and internal student references are pseudonymized. A fresh, isolated API session is created for each evaluation, so that no cross-student context is shared or accumulated on the provider side. Under this design, the external models cannot associate a submission with the identity of a specific student. The mapping between pseudonymized references and real identities is retained solely in the local database to support later review, auditing, and grade attribution.

Data minimization is thus enforced at the boundary with third parties, while full traceability is preserved internally: for each evaluation, the system stores the exact prompt, the retrieved context, and the model output, enabling instructors to audit or reconstruct any decision. Informed consent was obtained from all participating students, and only aggregated, de-identified data are reported in this article. We note that the study was conducted as part of the normal assessment activities of the courses involved. One limitation of the present work is the absence of formal audits of algorithmic fairness and demographic bias, which we identify as priorities for future work.

The implementation of the proposed framework in EvaluaTeC demonstrates the practical feasibility of integrating LLMs within a modular, context-aware, and pedagogically aligned system. By combining structured prompt design, retrieval-based contextualization, and robust architectural components, the platform provides a solid foundation for reliable and scalable AI-assisted assessment.

The following section presents the experimental design used to evaluate the effectiveness of the proposed approach in real educational settings.

5. Experimental Design

This section presents a rigorous empirical evaluation of the proposed framework in real educational settings. The objective is to assess whether the integration of structured rubrics, RAG, and modular system design improves the reliability, consistency, and pedagogical value of automated code assessment when compared to baseline LLM-based approaches.

5.1. Dataset and Experimental Setting

The evaluation is conducted on a dataset composed of real programming assignments collected from undergraduate- and graduate-level courses in computer science and engineering. The dataset includes a diverse set of exercises covering fundamental programming concepts, such as control structures, data manipulation, and algorithmic problem-solving. The evaluated assignments were implemented in ANSI C and executed within the EvaluaTeC platform under controlled examination conditions, which was selected due to its widespread use in introductory programming education, its strict syntax and low-level semantics, and the fact that it requires students to demonstrate a clear understanding of fundamental programming concepts without relying on high-level abstractions. The experimental dataset comprised 1287 submissions corresponding to 429 students, with each student completing three exercises. The use of a shared exercise set allowed direct comparison between human and automated grading configurations while minimizing task-dependent variability.

The dataset comprises the following elements:

• Student submissions: Source code solutions produced under realistic coursework and examination conditions.
• Exercise descriptions: Problem statements specifying requirements, constraints, and expected learning outcomes. These exercises covered progressively complex concepts, including conditional structures, loops, array manipulation, and function-based problem solving.
• Instructor evaluations (ground truth): Grades and qualitative feedback provided independently by at least two instructors. In cases of significant disagreement, a third instructor provides arbitration, ensuring high-quality and reliable reference labels.

The use of authentic student data ensures ecological validity, capturing the variability, ambiguity, and complexity inherent in real educational environments.

5.2. Compared Models and Configurations

To evaluate the contribution of each component of the proposed framework, three configurations are compared:

• Baseline LLM evaluation: The LLM is used without structural guidance or contextual augmentation. Evaluation is performed using a generic prompt describing the task, representing a typical “black-box” usage scenario.
• LLM + rubric: The evaluation process is guided by explicit grading criteria. Prompts include structured rubrics and few-shot examples, improving consistency and alignment with instructor expectations. This configuration is referred to as LLM + rubric throughout the manuscript.
• Hybrid LLM + RAG (proposed approach): The evaluation integrates structured rubrics with contextual retrieval from course materials. Prompts are augmented with domain-specific knowledge, representing the full implementation of the proposed framework.

The experiments were conducted using multiple commercial LLMs, including OpenAI GPT-4-based models and Google Gemini 2.5 Flash Lite configurations. Model selection was informed by previous studies evaluating the trade-off between evaluation quality, latency, and API cost in educational assessment scenarios [54,55].

To isolate the effect of the architectural components under evaluation, the three configurations compared in Section 6 (baseline, rubric-guided, and hybrid LLM + RAG) were all executed using the same underlying model, Gemini 2.5 Flash Lite, selected due to its lower latency and improved cost-efficiency when processing extended contextual prompts. GPT-4-based configurations were additionally evaluated in a separate robustness check, without contextual augmentation, to assess whether the observed trends generalize across model families.

This experimental setup enables a systematic analysis of the incremental impact of structured prompting and contextual grounding. Figure 8 illustrates the overall experimental workflow.

5.3. Evaluation Metrics

A comprehensive set of quantitative and qualitative metrics is employed to evaluate system performance. These metrics are designed to assess grading accuracy, consistency, reliability, and pedagogical relevance across configurations.

Table 7. Summary of Evaluation Metrics.

Category	Metric	Description
Accuracy	Correlation, MAE	Agreement with instructor evaluation
Consistency	Variance, repeatability	Stability of outputs
Pedagogical	Feedback quality	Educational usefulness
Explainability	Traceability	Transparency of evaluation
Robustness	Bias, prompt sensitivity	Reliability across conditions
Efficiency	Latency, cost	Practical deployment feasibility

summarizes the evaluation dimensions.

5.3.1. Agreement with Human Evaluation

Correlation with instructor grading: The alignment between automated scores ($\widehat{y}$) and human evaluations (y) is measured using the Pearson correlation coefficient:

$$r={\displaystyle \frac{{\sum }_{i=1}^n(y_i-\overline{y})({\widehat{y}}_i-\overline{\widehat{y}})}{\sqrt{{\sum }_{i=1}^n{(y_i-\overline{y})}^2}\sqrt{{\sum }_{i=1}^n{({\widehat{y}}_i-\overline{\widehat{y}})}^2}}}$$

(1)

where $y_i$ denotes the ground-truth score assigned by instructors for the i-th submission, ${\widehat{y}}_i$ represents the corresponding automated score, $\overline{y}$ and $\overline{\widehat{y}}$ are the mean values of human and automated scores, respectively, and n is the total number of evaluated samples.

Mean Absolute Error (MAE): The average deviation between automated and human scores is quantified as:

$$\mathrm{MAE}={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}|y_i-{\widehat{y}}_i|$$

(2)

where $y_i$ and ${\widehat{y}}_i$ are defined as above, and n is the total number of samples. This metric provides an interpretable measure of average grading error.

5.3.2. Consistency and Reliability

Intra-model consistency: To evaluate output stability, each experimental configuration was independently executed three times using identical prompts, evaluation settings, and submission subsets. Repeated executions were performed to quantify stochastic variability in model outputs and to assess evaluation reproducibility under fixed experimental conditions. The selection of the models was informed by prior studies, with the aim of achieving an optimal balance between cost-efficiency and output quality [54,55]. The stability of model outputs across repeated evaluations of the same submission is measured as:

$${\sigma }_{\mathrm{intra}}^2={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{\left({\widehat{y}}_i^{\left(1\right)}-{\widehat{y}}_i^{\left(2\right)}\right)}^2$$

(3)

where ${\widehat{y}}_i^{\left(1\right)}$ and ${\widehat{y}}_i^{\left(2\right)}$ are the scores assigned to the same submission in two independent runs, and n is the total number of submissions. Lower values indicate higher consistency.

Inter-sample stability: The variability of scores across semantically similar submissions is evaluated using variance:

$${\sigma }^2={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{({\widehat{y}}_i-\overline{\widehat{y}})}^2$$

(4)

where ${\widehat{y}}_i$ represents the automated score for the i-th submission, $\overline{\widehat{y}}$ is the mean score across the considered group of similar submissions, and n is the number of samples. This metric captures the robustness of the evaluation with respect to input variability.

Consistency score. To provide an interpretable normalized indicator of output stability across repeated evaluations, we introduce a consistency score derived from prediction variance:

$$C={\displaystyle \frac{1}{1+{\sigma }^2}}$$

(5)

where ${\sigma }^2$ denotes the variance of the predicted scores obtained from multiple evaluations of the same submission. The score is normalized in the range $[0,1]$, where higher values indicate greater consistency and stability of the model.

This metric is not intended as a standardized statistical measure, but as a descriptive normalization that facilitates comparison of stability across evaluation configurations.

5.3.3. Feedback Quality

Feedback quality is explored through qualitative analysis and indirect user interaction signals. Specifically, feedback is analyzed in terms of clarity, usefulness, pedagogical relevance, and alignment with predefined rubric criteria.

To complement this analysis, student interaction with the platform is used as an indirect validation signal. Students are allowed to either accept the feedback, request a reassessment, or provide no response. As reported in Section 6, 39.0% of students explicitly accepted the feedback, while only 1.4% requested revision. Given the academic context, the low rate of revision requests may be interpreted as an indirect indicator of student acceptance, although no formal user perception study was conducted.

Furthermore, alignment with rubric criteria is evaluated by verifying whether the generated feedback explicitly addresses each assessment dimension (e.g., correctness, code quality, efficiency, readability).

5.3.4. Explainability and Traceability

The explainability of the system is evaluated through two complementary metrics.

Explanation completeness. This metric measures the extent to which generated feedback covers all relevant evaluation dimensions. Completeness is assessed by checking whether feedback explicitly addresses key aspects such as correctness, style, efficiency, and clarity. The structured prompting strategy and rubric decomposition enable systematic coverage of these dimensions.

Traceability index. This metric evaluates whether feedback can be directly linked to specific rubric criteria and reasoning steps. Traceability is ensured by the modular evaluation process, in which each criterion is assessed independently and explicitly referenced in the generated feedback. Additionally, system-level logging enables full reconstruction of the evaluation process, including inputs, retrieved context, and model outputs.

5.3.5. Pedagogical Effectiveness

The pedagogical effectiveness of the framework is assessed using both direct and proxy measures.

Learning gain proxy. When multiple submissions from the same student are available, improvement is quantified as:

$$\Delta ={\widehat{y}}_{\mathrm{revised}}-{\widehat{y}}_{\mathrm{initial}}$$

(6)

where ${\widehat{y}}_{\mathrm{initial}}$ and ${\widehat{y}}_{\mathrm{revised}}$ denote the predicted scores before and after feedback, respectively. Positive values of $\Delta$ indicate that feedback contributes to performance improvement.

Error identification accuracy. This metric evaluates the ability of the system to correctly identify and explain student errors. It is assessed qualitatively by comparing generated feedback with instructor annotations and by analyzing whether key misconceptions are explicitly addressed. The hybrid configuration demonstrates improved diagnostic capability, particularly for conceptual errors.

5.3.6. Fairness and Robustness

The reliability of the framework is further evaluated in terms of fairness and robustness.

Bias considerations. The evaluation dataset includes students from multiple engineering programs and heterogeneous educational backgrounds, providing a degree of diversity in the assessed submissions. In addition, the use of multi-instructor ground truth (at least two independent evaluations per submission) helps reduce potential subjectivity in reference labels. However, the present study does not perform a formal fairness or demographic bias analysis, which remains an important direction for future work.

Robustness to prompt variation. Robustness is evaluated by analyzing the sensitivity of outputs to variations in prompt formulation. The lower variance values recorded in Section 6 suggest that structured prompting and contextual grounding reduce output variability, which would mitigate one of the key limitations of standalone LLM-based evaluation; as discussed in the threats to validity; however, this observation could not be re-verified with formal dispersion statistics.

5.3.7. Efficiency and Scalability

Latency: Average time per evaluation. Cost: API usage cost per evaluation.

The combination of these metrics enables a multi-dimensional evaluation of the proposed framework, capturing not only grading accuracy but also consistency, interpretability, fairness, and pedagogical impact. This comprehensive evaluation strategy is essential for assessing the suitability of AI-assisted systems in real educational environments, where reliability, transparency, and educational value are critical requirements.

To ensure experimental reproducibility, all students were evaluated using the same exercise statements, identical rubric structures, and equivalent prompt templates across configurations. The prompt templates used across the experimental configurations are provided in Appendix A. The only varying components between experiments were the inclusion of structured rubric guidance and the activation of the RAG-based contextual retrieval mechanism. This controlled setup enabled isolation of the contribution of each architectural component to grading reliability and pedagogical alignment.

6. Results

The quantitative evaluation compares the three configurations defined in Section 5: (i) baseline LLM, (ii) LLM with a structured rubric, and (iii) the proposed hybrid LLM + RAG framework.

The evaluation is conducted on a dataset comprising 429 students, each completing three independent programming exercises, resulting in a total of 1287 submissions. The students belong to different engineering programs (e.g., industrial, chemical, and management engineering), with heterogeneous educational backgrounds and levels of programming experience. Notably, the majority of participants were novice programmers, ensuring that the dataset reflects realistic learning conditions.

For each submission, four independent evaluations are obtained: three automated configurations and one human ground truth. The human grade is computed as the average of at least two independent instructors. In cases where the discrepancy exceeded two points (less than 3% of submissions), an additional evaluation was performed to ensure reliability.

6.1. Agreement, Consistency, and Stability

Table 8. Quantitative comparison of evaluation methods.

Model	r	MAE	$\mathit{\sigma }$	Consistency Score
Baseline LLM	0.7207	1.1614	0.81	0.71
LLM + rubric	0.7782	0.9211	0.52	0.82
LLM + RAG (proposed)	0.9059	0.5134	0.29	0.91

summarizes the quantitative performance of the three evaluation configurations in terms of agreement with instructor grading and output stability.

As shown in Table 8, the baseline LLM achieves only moderate agreement with instructor evaluations, highlighting the limitations of unstructured, context-free model usage. The relatively high MAE and variance further indicate that, despite its semantic capabilities, the model lacks reliability when used as a standalone evaluator.

The introduction of structured rubrics leads to a consistent improvement across all metrics. In particular, the increase in r and the reduction in MAE suggest that explicit evaluation criteria effectively constrain the model’s reasoning process, aligning it more closely with human grading behavior. Additionally, the decrease in variance indicates that structured prompting reduces output instability.

The proposed hybrid LLM + RAG approach achieves the highest performance across all dimensions among the evaluated configurations. The substantial increase in r (from 0.7207 to 0.9059) and the significant reduction in MAE indicate a strong alignment with the consolidated instructor reference grades. In addition, the recorded reduction in variance and the increase in the consistency score suggest that contextual grounding also improves output stability.

Two qualifications apply to these variability figures. First, the reported $\sigma$ and consistency scores are aggregate statistics recorded at experiment time; because the raw per-run outputs were not retained, they could not be re-verified with formal dispersion statistics (standard deviations or confidence intervals over repetitions), and the observed reduction in variability should therefore be interpreted as a recorded trend rather than a statistically verified effect. Second, all agreement metrics are computed against consolidated instructor reference grades rather than against independent human judgment (see Section 6.2 and the threats to validity).

These results suggest that improvements in accuracy and stability are closely related and jointly influenced by the incorporation of structure and contextual grounding, while rubrics primarily constrain the evaluation space, RAG provides domain-specific grounding, enabling the model to produce both consistent and contextually appropriate judgments.

Figure 9 further illustrates the improvement in correlation across configurations, showing a clear convergence of the hybrid approach toward human grading patterns.

Overall, the results reveal a systematic trend: each additional structural component contributes incrementally to both alignment and stability, with contextual grounding playing a dominant role in bringing the automated evaluation closer to the instructor grading used as reference. It should be emphasized that this reflects a closer approximation to instructor grading, rather than evidence of independently valid or “human-level” assessment, since instructor grades themselves constitute a reference standard subject to its own uncertainty. These results suggest that contextual grounding may contribute to reducing output variability while improving alignment with human evaluation.

6.2. Distributional Analysis of Grades

To further assess robustness, a statistical analysis of grade distributions was conducted. All configurations exhibit approximately normal distributions, consistent with expected grading behavior in educational settings.

However, a more detailed comparison reveals that the LLM + RAG configuration is the one most closely aligned with the human evaluation across most descriptive statistics, matching its median exactly and showing the smallest deviations in third quartile and standard deviation; the baseline configuration attains a marginally closer mean, but with a first quartile close to human as well and a substantially larger standard deviation, reflecting a less stable distribution overall. This suggests that the proposed framework approximates not only individual grading decisions but, to a reasonable extent, aspects of the overall statistical structure of human evaluation, although the comparison is based on descriptive statistics rather than a formal distributional equivalence test (e.g., Kolmogorov–Smirnov).

Table 9. Descriptive Statistics of Grade Distributions.

Model	Min	Q1	Median	Q3	Max	Mean	Std Dev
Human	1	3.2	4.0	6.2	10	5.0	2.45
Baseline LLM	0	3.3	4.6	7.1	10	5.1	2.91
LLM + rubric	0	2.8	3.9	6.0	10	4.5	2.79
LLM + RAG	0	3.1	4.0	6.1	10	4.8	2.6

summarizes the descriptive statistics, while Figure 10 presents the distributional comparison.

6.3. Impact of Retrieval-Augmented Generation

The integration of RAG produces improvements beyond numerical accuracy. Specifically, it reduces hallucinated or irrelevant feedback, enhances semantic consistency, and improves alignment across similar submissions.

Importantly, the hybrid approach approximates not only individual grading decisions but also, to a reasonable extent, the global statistical behavior of human evaluators, which strengthens its case as a formative, instructor-supervised assessment aid rather than establishing independent validity on its own.

6.4. Qualitative Analysis

In addition to quantitative performance, qualitative analysis provides insight into the pedagogical effectiveness of the system.

Student interaction data shows that among users who revisited the platform:

• A total of 39.0% explicitly accepted the feedback as correct and helpful,
• Only 1.4% requested a revision,
• The remaining students did not contest the evaluation.

Given the academic context, the low rate of revision requests may be interpreted as an indirect and limited indicator of student acceptance, although no formal user perception study was conducted.

Figure 11 illustrates the distribution of student responses.

A qualitative comparison of the generated feedback also reveals clear differences across configurations. The baseline LLM tends to produce generic and inconsistent feedback with limited pedagogical value. The LLM + rubric configuration yields structured feedback aligned with the evaluation criteria, but often lacking contextual depth. The LLM + RAG configuration produces context-aware, pedagogically aligned feedback, with explanations that more closely resemble instructor responses.

6.5. Cost and Performance Analysis

The proposed framework is also evaluated in terms of computational efficiency and deployment feasibility.

Table 10. Latency and cost comparison across evaluation configurations.

Configuration	Latency (s)	Cost (€)
Baseline LLM	$7.21\pm 1.12$	0.07
LLM + rubric	$7.53\pm 1.22$	0.15
LLM + RAG	$9.25\pm 0.83$	0.22

summarizes the average evaluation latency and cost per submission across the three configurations.

The results show that the baseline configuration achieves the lowest latency and cost, as expected due to its simpler prompt structure and lack of additional processing steps. Incorporating structured rubrics introduces a marginal increase in latency and a moderate increase in cost, primarily due to longer prompts and more complex evaluation instructions.

The proposed hybrid LLM + RAG approach incurs the highest latency and cost, driven by the additional retrieval and prompt augmentation steps. However, the observed overhead remains moderate and within acceptable bounds for near-real-time feedback scenarios.

Importantly, these increases are justified by the substantial improvements in grading accuracy, consistency, and pedagogical quality observed in previous sections, indicating a favorable trade-off between performance and evaluation quality.

We report efficiency in terms of end-to-end latency and monetary cost per submission, rather than in terms of parameter counts or floating-point operations (FLOPs). This choice is deliberate: the underlying evaluators (OpenAI GPT-4 and Google Gemini 2.5 Flash Lite) are accessed as proprietary, closed-weight APIs whose architectures, parameter counts, and per-query FLOPs are not publicly disclosed, so such figures cannot be reported reliably. For the components under our control, the additional overhead is modest and dominated by network round-trips to the LLM API rather than by local computation: the retrieval step performs an approximate nearest-neighbour search with FAISS over a single-course corpus, whose index fits comfortably in main memory and adds negligible latency compared with LLM inference. From a deployment standpoint, wall-clock latency and cost per submission are therefore the operationally meaningful efficiency metrics, and they are the ones a course could actually budget against. Regarding scalability, the system supports scalable deployment through the parallel processing of submissions, a distributed architecture, and the independent scaling of individual system components.

Taken together, the results point to three key findings: structured evaluation improves consistency but only partially enhances accuracy; contextual grounding through RAG is the main factor driving alignment with the instructor reference; and the hybrid approach closely approximates both individual grading decisions and, to a reasonable extent, the statistical behavior of human evaluators. These findings suggest that reliable automated assessment benefits from the integration of semantic reasoning, structured evaluation, and contextual knowledge, rather than relying on LLMs in isolation. We emphasize that this empirical evidence supports conclusions only for the configurations, dataset, and course setting evaluated in this study, and should not be read as a broader claim of superiority of the proposed framework over approaches or settings that were not tested.

7. Discussion

This section provides a critical interpretation of the experimental results and situates the proposed framework within the broader context of AI-assisted educational assessment. Beyond reporting performance improvements, the objective is to understand why these improvements occur, under which conditions they hold, and what implications they have for the design of reliable AI-based evaluation systems.

7.1. From Black-Box Grading to Structured Assessment

The results show that the main limitations of LLM-based grading are not caused by the models themselves, but by how they are deployed. In the baseline configuration, where the LLM acts as evaluator itself, grading is highly sensitive to prompt variations and produces moderate agreement with instructors, together with relatively high variance (Table 8).

Introducing a structured evaluation pipeline significantly improves this behavior. Explicit rubrics reduce ambiguity by constraining the model’s reasoning process, while RAG provides course-specific context that aligns the evaluation with instructor expectations. As a result, the system achieves both greater consistency and higher agreement with human grading.

These mechanisms play complementary roles: structure improves stability, and contextual grounding improves alignment. Their combination produces a more reliable and reproducible assessment process, suggesting that robustness in AI-assisted grading emerges primarily from system design.

7.2. Implications for AI-Assisted Educational Assessment

The findings of this study have broader implications for the design of AI-based assessment systems in education.

First, they challenge the prevailing paradigm of using LLMs as standalone tools. The results indicate that such approaches are inherently limited in terms of reliability and pedagogical alignment. Instead, effective deployment requires integrating LLMs within structured, context-aware architectures that explicitly encode evaluation criteria and domain knowledge.

Second, the proposed framework demonstrates that automated assessment systems can approximate not only individual grading decisions but also the statistical properties of human evaluation (Section 6.2). This suggests that AI systems can approximate both micro-level (individual scores) and macro-level (distributional behavior) aspects of human assessment, which is a relevant property for their adoption in real educational settings. We stress, however, that reproducing the grading patterns of instructors does not by itself establish independent validity: agreement is measured against instructor grades, which serve as a reference standard rather than an infallible ground truth. Consequently, the present evidence supports the use of the framework as a formative, instructor-supervised assistant, and does not on its own justify fully autonomous deployment in high-stakes summative assessment.

Third, the ability to generate structured, traceable, and context-aware feedback represents a significant step toward explainable AI in education. Unlike traditional black-box models, the proposed approach enables the reconstruction of evaluation decisions, supporting transparency, accountability, and trust.

7.3. Limitations and Failure Cases

Despite these advantages, the framework exhibits limitations that must be considered.

A primary challenge arises in open-ended or creative programming tasks, where multiple valid solutions exist and evaluation criteria are inherently less constrained. In such scenarios, even structured rubrics may fail to capture the full diversity of acceptable approaches, potentially leading to inconsistent evaluations.

The effectiveness of the RAG module is also dependent on the quality and coverage of the underlying knowledge base. Incomplete or poorly aligned instructional materials may reduce the coherence and pedagogical relevance of the generated feedback.

Furthermore, although the recorded results indicate reduced variability under the proposed approach, it is not eliminated entirely. Residual stochasticity remains inherent to LLM-based systems, particularly in borderline or ambiguous cases.

Finally, tasks requiring advanced reasoning, such as complex algorithm design or optimization, may exceed the capabilities of current LLM-based approaches and require integration with formal analysis or verification techniques.

7.4. Threats to Validity

Several threats to validity should be acknowledged.

From a construct validity perspective, while the selected metrics capture agreement, consistency, and feedback quality, they do not directly measure long-term learning outcomes or knowledge retention.

A further construct-validity consideration concerns the nature of the reference standard. All agreement metrics are computed against instructor grades, which are themselves subject to a degree of subjectivity and inconsistency. To mitigate this, each submission was graded by at least two instructors, with a third acting as arbiter when the discrepancy exceeded two points (fewer than 3% of submissions), which provides indirect evidence that the reference labels are reasonably consistent. However, only the consolidated (averaged) instructor grade was retained, and the individual grader scores were not stored. As a consequence, we could not compute formal inter-rater agreement statistics (e.g., ICC or Cohen’s $\kappa$), nor directly compare system–human agreement against human–human agreement. High agreement with the reference should therefore be interpreted as close approximation to instructor grading, not as proof of independent or “human-level” correctness. More broadly, the reliability of reference labels is a well-recognized challenge across machine learning, where a substantial body of work studies learning and evaluation under noisy labels [56]; although our framework does not train a model on these labels, the same concern motivates our reliance on multi-instructor arbitration and our emphasis on reporting inter-rater reliability and system–human versus human–human agreement in future work.

Regarding internal validity, system performance remains partially dependent on prompt design and model configuration. Although structured prompting reduces sensitivity, suboptimal prompt design may still affect results.

A limitation that deserves particular emphasis concerns reproducibility. Several experimental artifacts were not preserved alongside the reported results: the exact decoding parameters (temperature, top-p), random seeds, and the precise versions of the prompts used in each run, as well as the raw per-run outputs of the three repetitions executed per configuration. As a consequence, the quantitative tables report aggregate figures recorded at experiment time that cannot be independently re-verified, and mean ± standard-deviation summaries and confidence intervals across repetitions could not be reconstructed for this version. This materially limits the independent reproducibility of the reported numbers, and all conclusions should accordingly be restricted to the evaluated dataset and experimental setting. A fully pinned and versioned experimental release, reporting dispersion statistics over repeated runs, is planned for the extended study described in Section 8.

In terms of external validity, the evaluation is conducted on programming assignments within specific courses, while representative, the generalization of the framework to other domains (e.g., mathematics or natural sciences) requires further validation.

7.5. Implications for Educational Practice

Overall, the results support a central insight: LLMs are not reliable evaluators when used in isolation; however, when integrated within structured, context-aware systems, they can become effective and pedagogically meaningful assessment tools, particularly as instructor-supervised assistants rather than as fully autonomous replacements for human judgment. This highlights the need to shift from model-centric to system-centric approaches in the design of AI-assisted educational technologies. Beyond this research-level implication, the findings of this work have several practical implications for the integration of artificial intelligence in educational contexts, particularly in the design of innovative assessment practices.

First, the proposed framework supports a shift from traditional summative assessment toward continuous and formative evaluation. By enabling automated, timely, and structured feedback on programming assignments, the system promotes iterative learning processes in which students can refine their solutions based on actionable insights.

Second, the integration of LLMs with structured rubrics and contextual knowledge enables the delivery of personalized feedback at scale. Unlike conventional automated systems that produce generic responses, the proposed approach generates detailed, context-aware explanations aligned with course materials, thereby enhancing student understanding and engagement.

Third, the framework may help reduce instructor workload, particularly in large-scale courses. By automating repetitive aspects of evaluation and feedback generation, instructors could allocate more time to higher-value activities such as pedagogical design, mentoring, and individualized support. We note, however, that the present study does not directly quantify workload reduction: it does not measure how many automated evaluations would still require human verification, which cases would need review, or the net effort compared to conventional manual grading. A dedicated study measuring instructor time and validation effort is therefore left for future work, and the workload-related benefits should be interpreted as expected rather than empirically established.

Finally, the modular and scalable architecture is designed to support deployment in large educational environments and to maintain consistent performance across high volumes of submissions. This scalability is increasingly relevant for modern educational systems facing growing demand.

Overall, this work demonstrates that the integration of generative AI with pedagogical structure and robust system design can enable more effective, scalable, and learner-centered assessment practices, supporting ongoing innovation in education.

8. Limitations and Future Work

Despite the promising results, several limitations of the proposed framework must be acknowledged, which also define important directions for future research.

First, the generalizability of the approach across programming languages and domains remains limited. The present study was deliberately scoped to a single, well-characterized setting: introductory programming exercises written in ANSI C within an authentic, large-scale university course. This choice enables a controlled and ecologically valid evaluation on 1287 real submissions with multi-instructor reference grades, but it necessarily leaves cross-language and cross-domain generalization untested. We did not evaluate the framework on public code-evaluation benchmarks (e.g., competitive-programming datasets) or on other languages such as Python or Java, and we therefore refrain from claiming performance beyond the studied setting. Although the architecture is designed to be model- and language-agnostic, its effectiveness in other languages, course levels, and institutional contexts remains to be demonstrated empirically. Extending the evaluation is a natural next step: a subsequent course edition is already being prepared in which the platform will operate in open (non-exam) mode across additional programming languages, enabling cross-language and cross-benchmark validation as well as stratified analyses by problem type, difficulty, and code length.

Second, the performance of the RAG module is inherently dependent on the quality and availability of instructional materials. When course resources are incomplete, outdated, or misaligned with assessment tasks, the benefits of contextual grounding may be reduced. Further research is needed to improve knowledge base construction, including automated curation, quality assessment, and dynamic updating of educational content.

Third, although the framework improves consistency compared to baseline LLM approaches, it relies on a single-model evaluation paradigm, which may limit robustness in complex or ambiguous cases. A promising direction is the exploration of multi-agent architectures, where specialized models handle distinct aspects of evaluation (e.g., correctness, style, pedagogical feedback) under a coordinated framework. Such approaches could enhance reliability, reduce hallucinations, and improve interpretability.

Fourth, the experimental analysis follows an incremental design in which the rubric and retrieval components are added cumulatively (baseline, +rubric, +rubric+RAG). While this isolates the marginal contribution of each stage, it does not include the full set of reverse ablations (e.g., RAG without rubrics, or rubrics without few-shot guidance) that would independently quantify the effect of every component and their interactions. Likewise, and as noted in the threats to validity, the raw per-run outputs were not retained, preventing the reporting of mean ± standard-deviation and confidence intervals across repetitions, and the individual instructor scores were not stored, preventing a formal comparison between system–human and human–human agreement. A planned extended study will address these aspects jointly: a factorial ablation with repeated runs and reported dispersion, inter-rater reliability analysis, and evaluation across additional programming languages and stronger, more recent models, so that the contribution of each mechanism can be disentangled under a broader and statistically richer protocol.

A further direction concerns open-ended and creative programming tasks, for which multiple valid solutions exist and fixed rubrics are less effective; adaptive or dynamic rubrics, together with additional validation layers such as static analysis or formal verification, could extend the framework’s reliability to these more complex scenarios. The development of standardized benchmarks for AI-assisted assessment would likewise enable more rigorous, reproducible comparison across approaches, including the controlled re-implementation of competing systems discussed in Section 2.

Additional research opportunities include the incorporation of adaptive learning mechanisms, where feedback is tailored to individual student profiles and learning progress, as well as the integration of human-in-the-loop or reinforcement learning strategies to continuously refine evaluation quality. Moreover, longitudinal studies are needed to assess the long-term impact of the framework on learning outcomes beyond immediate feedback improvements.

Finally, practical deployment at scale requires further optimization of cost and performance. Techniques such as model compression, caching, and hybrid deployment strategies combining local and cloud-based models could improve efficiency while maintaining evaluation quality. Operational maturity will also require lifecycle mechanisms such as prompt and rubric versioning, model-version pinning, and monitoring of rubric-adherence drift across successive course editions.

In summary, while the proposed framework provides a robust foundation for reliable and pedagogically aligned AI-assisted assessment, addressing these limitations will be essential to ensure its scalability, generalizability, and long-term impact across diverse educational contexts.

9. Conclusions

This work presents a hybrid framework for AI-assisted code assessment that combines structured rubrics, RAG, and modular system design within a real educational platform. Within the evaluated configurations, dataset, and course setting, the results show that integrating contextual grounding and explicit evaluation criteria substantially improves agreement with the consolidated instructor reference grades, was associated with lower recorded output variability, and enables the generation of more coherent and pedagogically aligned feedback. These conclusions are grounded in, and limited to, the evaluated experimental setting, and are not intended as broader claims of superiority over untested approaches, languages, or educational contexts.

Beyond the quantitative improvements, the study highlights the importance of treating automated assessment as a systems problem rather than a purely model-centric task. The findings show that the reliability of LLM-based evaluation depends not only on the capabilities of the underlying models, but also on the architectural mechanisms that constrain, contextualize, and trace the evaluation process.

The implementation and large-scale evaluation carried out in authentic academic settings provide evidence that structured AI-assisted assessment can be deployed in practice while maintaining scalability, interpretability, and educational usefulness. In this sense, the proposed framework contributes to the development of more trustworthy and operationally viable AI systems for higher education.

Future research should focus on extending the framework to more open-ended programming tasks, improving robustness through multi-agent or hybrid verification strategies, and evaluating the long-term pedagogical impact on student learning outcomes. Additionally, broader validation across disciplines and educational contexts will be necessary to assess the generalizability of the proposed approach.

Ultimately, the results suggest that the most promising role of generative AI in education is not in the replacement of instructors, but the augmentation of educational processes through structured, transparent, and context-aware assessment systems.