Automatic Assessment Tools for Grading Coding Assignments: A Systematic Literature Review

Abstract

The rapid growth of programming education and online learning environments has increased the demand for scalable and reliable assessment methods. Although many automated grading approaches exist, the literature spans traditional test-based methods and recent solutions based on artificial intelligence, making it difficult to obtain a coherent overview of the field. This study conducts a systematic literature review on automated programming assessment, analysing grading approaches, the use of artificial intelligence techniques, educational contexts, and comparisons between automated and human grading. The review synthesises evidence from 39 primary studies to identify technological trends and research gaps. The results show a shift from execution-based grading toward deep learning and large language model approaches, with most systems applied in higher education and limited research on visual programming assessment. Automated grading is evolving into an intelligent educational support tool, but challenges remain with regard to reliability and pedagogical impact. Future work should prioritise empirical validation, hybrid grading models, and broader educational applications.

1. Introduction

In recent years, the rapid expansion of computer science education, online learning platforms, and large programming courses has created a growing demand for scalable assessment methods [1,2]. Traditional manual grading of programming assignments is time-consuming, resource-intensive, and often inconsistent between evaluators [3]. To address these challenges, automated assessment tools (AATs) have been widely adopted in programming education [4,5]. These tools, commonly referred to as autograders, automatically evaluate programming assignments and generate feedback for students without direct instructor intervention. Such systems are widely used in programming education to support large classes, online learning environments, and scalable assessment workflows [1,2].

Initially, most autograders relied on deterministic approaches, such as test-case execution or rule-based code analysis [4,6], evaluating programs by executing predefined test cases and comparing outputs against expected results. However, recent advances in artificial intelligence (AI) and large language models (LLM) appear to be transforming automated grading [7,8,9].

Over time, automated assessment methods have evolved beyond execution-based approaches. Machine learning (ML) methods have been introduced to analyse the structural, syntactic and semantic properties of source code, enabling predictive grading models and similarity-based assessment [10,11]. Deep learning (DL) approaches further extend this concept by learning code representations through neural architectures, allowing systems to capture higher-level semantic patterns and improve grading accuracy [12,13,14]. The most recent trend in automated assessment research involves the application of LLMs. These models leverage pretrained transformer architectures to interpret code, generate feedback, and simulate rubric-based evaluation processes [7,8,9]. Unlike earlier approaches, LLM-based systems can analyse program intent, style, and reasoning, allowing more human-like evaluation and richer formative feedback [15,16]. Recent studies also investigate the reliability and acceptance of such systems in educational settings [17,18].

The development of reliable AATs is crucial for several reasons. First, they enable scalable evaluation in large university courses, MOOCs, and online learning environments [1,19]. Second, they allow students to receive immediate feedback, which is known to improve learning outcomes and engagement [15,16]. Third, AI-driven grading tools may reduce instructor workload while maintaining evaluation consistency [17,18].

With the increasing integration of generative AI into education, understanding the capabilities and limitations of AATs has become essential for both educators and researchers. Evaluating how these systems are used, how reliable they are, and in which contexts they perform best is therefore an important research challenge. Moreover, assessing programming solutions in robotics-based tasks remains a challenge. Robotics activities involve both program logic and interactions with physical hardware, making manual evaluation time-consuming and potentially subjective. In addition, research has shown that students often develop diverse mental models of robots and programming concepts, which can further complicate the evaluation of their submissions [20].

Despite the growing number of studies on automated grading, the literature remains fragmented [3,21]. Several review and survey studies have explored automated assessment in programming education. These approaches typically rely on automated testing or static code analysis to determine the correctness of the program [22,23,24]. However, such methods are not always directly applicable in educational robotics environments, where program behavior depends on interactions with physical hardware and sensor input. Ref. Puskar and Sovic [25] proposes a method for the automatic evaluation of LEGO Mindstorms EV3 programming solutions. The approach converts block-based programs in visual programming (VP) into a machine-readable representation, allowing structural analysis and comparison with reference solutions. This allows for automated evaluation of the correctness of student programs while considering the sequence and logic of programming blocks. The method is intended to support scalable and objective assessment in robotics-based programming education. By reducing the need for manual grading, the approach can assist instructors in efficiently evaluating a large number of student submissions.

Earlier surveys primarily focused on traditional and ML-based grading techniques [26], while more recent reviews analyse the emerging role of LLMs in programming education and assessment workflows [3,21]. Studies highlight the educational potential of robotics-based learning activities, showing that they can foster problem-solving skills, collaboration, and computational thinking among students [27]. Recent research has also explored the use of robotic process automation to support educational processes [28]. However, existing literature often addresses grading methods, AI integration, educational contexts, and reliability issues separately. A comprehensive synthesis that jointly analyses these dimensions remains limited, motivating the need for a systematic review of the literature that provides a structured overview of automated grading research across methodological and technological perspectives. Many works focus on proposing new grading models without systematically analysing their educational applicability or comparative performance. In particular, several gaps can be identified:

• limited synthesis of how different grading approaches (test-based, ML, DL, LLM) compare [26],
• insufficient evaluation of AI-based graders against human assessment [8,29],
• limited research on automated grading in visual or block-based programming (BP) environments [30],
• lack of consolidated understanding of educational contexts in which autograders are applied [3].

These gaps indicate the need for a systematic overview that maps existing approaches and evaluates their role in modern programming education.

Previous literature reviews have primarily focused on specific aspects of automated assessment in programming education. For example, Paiva et al. [27] provided a broad overview of automated assessment systems, focusing on testing techniques, security measures, feedback generation, and assessment infrastructure. Keuning et al. [23] examined automated formative feedback in programming exercises, while Messer et al. [3] specifically analyzed the application of machine learning techniques in automated grading and feedback generation. In contrast, this study provides a unified and up-to-date overview of both traditional and AI-based automated assessment approaches, including dynamic analysis, static analysis, ML-, DL-, and LLM-based systems.

The purpose of this study is to provide a systematic overview of AATs for programming assignments, with a particular focus on the role of AI techniques. To achieve this, a systematic literature review (SLR) is conducted to analyse existing approaches to automated code assessment, examine how AI techniques are applied, and evaluate their reliability in educational contexts, particularly in comparison with human grading. By synthesizing and categorizing the existing research, this study aims to identify current research trends, highlight existing limitations, and identify directions for future research in intelligent automated assessment systems.

The rest of this paper is structured as follows. Section 2 presents the methodology used in the preparation of this research. Section 3 provides an overview of the included studies and a categorization of the scientific articles according to the research questions of this paper, obtained through the author’s own analysis during the SLR. Section 4 outlines the interpretation of the research results. Section 5 sets out the main findings, research gaps, and proposals for further research. Finally, Section 6 summarizes the conclusions.

2. Methodology

In this study, an SLR was conducted to identify, evaluate, and analyse research related to automated assessment tools in programming education. Following the methodology proposed by [31], the review was used to synthesise existing evidence and identify research gaps relevant to the studied topic. The study selection process was conducted by the first author and subsequently reviewed by both supervisors. Any uncertainties regarding study inclusion were resolved through discussion and consensus.

This section describes our protocol for the SLR, including the definition of research questions, inclusion/exclusion criteria, search, and filtering procedures.

2.1. Research Questions

To guide the SLR, research questions were defined to specify the objectives of the study and support the identification, selection, and analysis of relevant publications.

Initially, the research questions were designed to investigate the range of methodologies used in automated assessment tools. AI-based approaches were further examined due to their increasing role in educational assessment systems, while ML methods were considered foundational AI-driven techniques commonly applied in automated grading. Additionally, LLM-based approaches were analyzed separately because their human-like interaction and reasoning capabilities introduce distinct opportunities and challenges in programming assessment.

These questions structured the search and evaluation process, ensuring that the review remained focused on the research topic [31]. They were designed to address methodological approaches, AI-based assessment techniques, educational applications, the identified research gap in VP/BP assessment, and the reliability of LLM-based grading systems in comparison with human evaluators. In order to guide our review of AATs for program code, we used the following research questions:

• RQ1. (methods) What methods are used for the automatic assessment of programming code?
• RQ2. (AI and ML) How are AI and ML methods used in the automatic assessment of programming code, and what advantages do they provide compared to traditional approaches?
• RQ3. (VP and BP) To what extent do automated systems exist for assessing visual and block-based programming solutions?
• RQ4. (education) In which educational contexts are automated assessment tools applied?
• RQ5. (reliability) How reliable are LLM-based grading systems compared to human graders?

2.2. Search Strategy

The search strategy was designed to systematically identify relevant studies on automated assessment tools for programming assignments. It consists of four main components: the definition of search query, selection of digital libraries, application of inclusion and exclusion criteria, and process of data extraction and coding. The review process followed PRISMA-inspired stages of identification, screening, eligibility assessment, and inclusion to ensure a systematic and transparent study selection procedure.

2.2.1. Search Query

For the purposes of this study, search queries were defined to identify relevant scientific publications included in the SLR. The queries were constructed using selected keywords, synonyms and phrases derived from the research questions. Logical operators (AND, OR) were applied to combine search terms and capture different variations of relevant concepts, enabling the retrieval of publications related to the research topic in order to broaden or narrow the search.

After identifying the keywords from the research questions (Table 1), the query for the review of scientific papers was expanded to include their synonyms. The search combined the following keywords: “programming”, “assignment”, “automatic assessment”, “system”, “artificial intelligence”, and “machine learning”. To generate variants of these terms, we isolated them and added a wildcard. For example, “automatic” becomes “automat*” and “grading” becomes “grad*”. However, due to the common nature of our keywords, the stemming method produced numerous irrelevant results, especially for “grad*”. Therefore, we decided to use specific variants, such as “grading” and “grader”, avoiding the detection of incorrect terms (e.g., gradient), while irrelevant publications obtained due to “automat*” were discarded in the remaining phases of SLR selection.

Table 1. Keywords for Searching Scientific Paper Databases for Tools for the Automatic Assessment of Programming Tasks.

Basic Concepts	Keywords for Search
Automatic	“automatic”, “automated”
Grading	“verifier”, “grader”, “assessment”, “grading”, “marking”
System	“system”
Programming	“programming”, “code”
Assignment	“assignment”, “exercise”
Data Science	“artificial intelligence”, “machine learning”

The search query used in the scientific databases was defined as follows:

(automat*)

AND (grader OR verifier OR “assessment system” OR “grading system” OR “marking system”)

AND (code OR “programming assignment” OR “programming exercise”)

AND (”artificial intelligence” OR “machine learning”)

To improve transparency and reproducibility, of the review process, the complete database-specific search queries adapted for each scientific database are provided in Appendix A. Due to differences in database syntax and indexing mechanisms, the search queries were adapted individually for each database while maintaining consistent search concepts across all sources.

2.2.2. Databases of Scientific Papers

Defining the databases of scientific papers for searching in the SLR ensures broad coverage and provides access to reliable sources.

The databases IEEE Xplore, Scopus, and Web of Science were selected. Additionally, to obtain the full texts of the selected scientific papers, we used the Google Scholar search engine, as well as the websites ResearchGate, Springer, and the ACM Digital Library.

The majority of obtained full-text articles and book chapters are available in open access, several publications were reached via institutional login, a number of papers are found in private inventories of the authors of this study, and several papers were obtained in personal correspondence with the authors of those papers.

The search in the defined databases of scientific papers yielded the results presented in Table 2.

Table 2. The Number of Search Results in Scientific Databases.

Database of Scientific Papers	Total Number of Publications Found
IEEE Xplore	812
Scopus	76
Web of Science	98
Total	986

2.3. Inclusion and Exclusion Criteria

The analysis of the results obtained from the query in the three selected databases was carried out through a selection procedure illustrated on Figure 1, applying the study inclusion and exclusion criteria described in Table 3.

Figure 1. Workflow of Systematic Literature Review.

Table 3. Overview of Inclusion and Exclusion Criteria.

Criteria	Description
EC1 Metadata	The selection process was limited to peer-reviewed papers featuring the phrase automatic assessment tool’ or equivalent descriptors. The exported CSV datasets were used for this screening step. Keywords were analysed in title and across three specific metadata fields: Author Keywords, IEEE Terms for IEEE Xplore entries, and Index Keywords for Scopus-indexed publications.
EC2 Specific domains	Publications focused on mechanical engineering, civil engineering, blockchain, medicine, and narrowly focused engineering systems were excluded.
EC3 Focus of the literature	Publications focused on formal logic and hardware verification without educational contexts were excluded.
EC4 Lack of technical details	Publications limited to abstracts, position papers, or without a clearly defined evaluation methodology were excluded.
EC5 Obsolescence and methodological duplicates	Publications using outdated approaches already superseded by more recent work in the dataset were excluded. Moreover, duplicate records identified across multiple databases were removed.
IC1 AI automation in education	Publications employing modern AI techniques, including ML or LLM-based evaluation, were included.
IC2 High technical documentation and metrics	Publications providing detailed architecture, datasets, and evaluation metrics were included.
IC3 Block-based languages (research gap)	Publications addressing VP/BP environments were included due to their relevance to the research gap.

The selected databases also enabled the export of search results and metadata in CSV format. The exported datasets were used to organize the the records and further analyzed through keyword-based filtering of author keywords, titles, abstracts, and other metadata fields. They supported the application of the exclusion criteria during the screening and eligibility assessment procedure.

The selection of important publications that provide answers to the subject research questions was conducted in the following two phases:

• Application of the Exclusion Criteria (EC1–EC5)
• Study Selection Using Inclusion Criteria (IC1–IC3)

2.3.1. Phase 1: Application of the Exclusion Criteria

In the first phase, studies that fell outside the scope of this research were removed. Publications focused on domain-specific engineering fields such as mechanical or civil engineering, blockchain applications, or narrowly specialized technical systems were excluded. Studies addressing formal logic or hardware verification without an educational context were also removed. Furthermore, publications lacking sufficient technical detail, including abstracts, position papers, or studies without a clearly defined evaluation methodology, were excluded. Finally, outdated approaches already superseded by more recent work in the dataset were removed, and duplicate records identified across multiple databases were discarded.

2.3.2. Phase 2: Application of the Inclusion Criteria

In the second phase, the remaining studies were assessed to identify those most relevant to the research objectives. Publications employing modern AI-based approaches, including ML techniques or LLM-supported evaluation, were retained. Studies providing detailed technical descriptions, including system architecture, datasets, and evaluation metrics, were included in the review to support the analysis of the defined research questions. Publications addressing block-based programming environments, were also retained, as they contribute to understanding different approaches to automated assessment considered within the scope of this study.

2.4. Data Extraction and Coding

To ensure a structured and reproducible analysis, a predefined data extraction protocol was developed. For each selected study, key information was systematically recorded, including publication year, research objectives, grading methodology, technological approach, educational context, evaluation method, and reported results.

Particular attention was paid to identifying the underlying assessment approach used in each study, the role of AI techniques, and whether system performance was compared with human grade. Additional attributes such as feedback type, implementation platform, and target learner group were also extracted when available. This structured extraction allowed for consistent comparison of studies and supported the mapping of findings to defined research questions. The study selection process was independently conducted by two authors, and the final inclusion decisions were established through discussion and consensus to ensure consistency and objectivity.

After data extraction, each study was coded according to its dominant marking approach. The coding process followed an iterative classification strategy inspired by common taxonomies in automated assessment research. Two authors independently reviewed the studies. The results were aligned through discussion until an objective and unbiased consensus was reached. Initially, studies were grouped on the basis of methodological similarity. Through repeated refinement, six primary categories of grading approaches were defined:

• Dynamic analysis, also known as Test-based grading, evaluates programs by executing them against predefined test cases and assigning scores based on the number of tests passed. This approach is widely applicable and relies on comparing program outputs with expected results, but requires executable code and sufficiently comprehensive test cases [32]. This approach commonly involves black-box testing, which evaluates program inputs and outputs, and grey-box testing, which analyses the results of individual functions to assess overall program behavior. Dynamic analysis compares actual and expected outputs and can evaluate program functionality even without access to the source code.
• Rule/static analysis, which verify the program without executing it or taking input parameters into account. It evaluates program code without executing it by examining its structure, control flow, and data flow. In AATs, it enables the evaluation of programs even when compilation errors occur and supports deeper code analysis beyond output verification. Common techniques include style analysis [33], error detection, code metrics, keyword checking, and structural comparison with reference solutions [34]. However, the diversity of possible implementations for the same functionality can make accurate evaluation challenging, particularly for more complex programs.
• ML approaches, which use engineered features or predictive models to predict errors, provide feedback and assess code quality [35] or correctness.
• DL approaches, which learn representations of code using neural architectures [36,37].
• LLM approaches, which rely on pretrained transformer models for grading, feedback generation, or semantic interpretation.
• Hybrid approaches, combining execution-based and AI-driven methods.

When studies employed multiple assessment techniques, they were assigned to all relevant categories corresponding to the approaches explicitly implemented and evaluated in the study. Approaches that were only mentioned without practical implementation or analysis were not considered for classification. Studies were classified according to the primary automated grading mechanism used in the evaluation process. This strategy was adopted to ensure a consistent and objective categorization across the analysed publications.

To improve reliability, the coding decisions were based on explicit criteria derived from each study’s methodology section rather than inferred from keywords alone. Unknown cases were resolved by examining the system’s evaluation pipeline, ensuring that the classification reflected the actual assessment mechanism rather than the implementation details.

In conclusion, the SLR identified relevant studies that were subsequently categorized by research questions. This breakdown provides a comprehensive overview of the state-of-the-art in the field and is illustrated in Table 4. The dominance of IEEE Xplore in the final selection suggests that research on AATs is primarily published within engineering and computer science conference venues rather than in scientific journals.

Table 4. Results of Systematic Literature Review per Scientific Paper Databases.

Scientific Paper Databases	Publications	Total
IEEE Xplore	[1,2,4,5,6,7,8,10,11,12,13,14,15,18,19,21,26,29,30,38,39,40,41,42,43,44,45,46]	28
Scopus	[3,9,16,17,47,48,49,50]	8
Web of Science	[25,51,52]	3
Total		39

3. Results

3.1. Overview of Included Studies

A total of 39 primary studies were included in the final analysis after applying the defined inclusion and exclusion criteria. The selected studies span multiple publication venues, including journals, conferences, and workshops.

Table 5 presents the classification of the identified approaches for the automated assessment of programming assignments. The table further illustrates that many studies integrate multiple grading approaches.

Table 5. Classification of studies by automated grading approach.

Approach	References
Dynamic analysis	[1,2,4,5,19,50,52]
Static analysis	[6,10,12,13,25,30]
ML-based assessment	[3,6,10,11,13,19,26,30,39,42,49,51]
DL approach	[6,10,12,14,40,43,45,47]
LLM-based assessment	[7,8,9,15,16,17,18,21,29,38,41,44,46,48]
Hybrid AI systems	[9,42,47]

The temporal distribution of studies in Figure 2 reveals a clear growth trend in recent years. While early research primarily focused on traditional automated grading approaches, the number of publications increased significantly after 2021. In particular, studies published between 2021 and 2025 constitute the majority of the dataset.

Figure 2. Timeline of Included Studies by Publication Year.

3.2. Results by Research Questions

Table 6 presents the mapping of identified approaches for automated grading of programming tasks to the defined research questions (RQ1–RQ5). The rows represent categories of grading approaches, while the columns correspond to the research questions within a specific approach.

Table 6. Mapping of autograding approaches to research questions.

Approach	RQ1	RQ2	RQ3	RQ4	RQ5
Dynamic analysis	[1,4,5,19,50,52]	–	[19]	[1,2,4,52]	–
Static analysis	[6,10,12,13,25,30]	–	[25,30]	[10,12,13,25]	–
ML-based assessment	[3,6,10,11,13,19,26,30,39,42,49]	[3,10,11,13,39]	[19,30]	[3,10,13,39,49,51]	[39]
DL approach	[6,10,12,14,40,43,45,47]	[10,12,14,40,45,47]	–	[10,12,40,47]	[14]
LLM-based assessment	[7,8,9,16,17,21,29,38,44,46,48]	[7,9,15,16,17,18,29,38,41,44,46,48]	–	[7,8,16,17,18,29,38,41,44,46,48]	[7,8,9,16,17,18,29,38,46]
Hybrid AI systems	[9,42,47]	[9,42,47]	–	[15,42,47]	[9]

Furthermore, test-based autograders are primarily associated with RQ1 methods (6 papers in Table 6) and RQ4 educational contexts (4 papers in Table 6). Studies related to RQ2 and RQ5 more frequently involve AI-based approaches, including ML-, DL-, and LLM-based systems, particularly in the context of grading reliability, feedback generation, and comparison with human evaluators.

The analysed studies cover a range of educational contexts. Most systems were designed for higher education computer science courses, particularly introductory programming modules. Several studies also addressed large-scale online courses, learning management systems, and automated assessment platforms supporting remote or blended learning.

A smaller subset of studies focused on secondary education and computational thinking. Only a limited number of studies addressed specialised contexts such as visual or block-based programming environments.

Figure 3 summarizes mapping of AAT approches and shows that the majority of studies are concentrated around RQ1 and RQ2, which focus on assessment methods and the application of AI/ML techniques. However, their contribution to the evaluation of LLM grading reliability vs. human graders (RQ5) is relatively limited.

Figure 3. Coverage Of Research Questions.

Figure 4 illustrates the distribution of studies by grading approach, whereas test-based approaches constitute a smaller proportion of the reviewed studies. LLM-based approaches represent the category most studied, followed by machine learning and deep learning methods. In contrast, traditional approaches such as dynamic and static analysis are less prevalent, while hybrid AI systems remain relatively underexplored.

Figure 4. Number Of Studies Per Approach.

A summary of the knowledge obtained from the resulting papers, as well as from those reviewed during the selection process, is presented below, organized by research questions and automated assessment approaches. Additionally, the organized data supported the identification of current research trends, underexplored areas, and potential directions for future research in automated programming assessment.

3.2.1. RQ1—Methods Used in Automated Grading

Research Question 1 examines the methodological approaches for the automated assessment of programming code. The timeline of studies by approach presented in Figure 5 illustrates an evolution of assessment approaches, ranging from traditional rule-based systems toward modern AI-driven solutions.

Figure 5. Timeline Of Studies By Approach.

Early AATs relied predominantly on test-based evaluation, where student programs were assessed using predefined input–output test cases. Such approaches remain widely used due to their simplicity, transparency, and reliability in verifying functional correctness [1,4,5]. Platforms such as LMS-integrated graders and online judge systems continue to employ this method, particularly in large-scale educational environments [1,5].

Beyond functional testing, some studies introduced static and rule-based analysis, incorporating metrics such as code structure, syntactic correctness, and style compliance. These approaches aim to assess aspects of code quality in addition to correctness, but typically require manually defined rules and task-specific configurations [6].

More recent research increasingly adopts ML-based methods, where features extracted from source code—such as syntax trees, token sequences, or complexity metrics—are used to predict grades or classify solutions [10,11,39]. These models support more flexible evaluation compared to rule-based approaches and allow the analysis of larger and more diverse code datasets.

Advancements in DL have introduced representation-learning approaches in which programs are encoded using neural architectures capable of capturing structural and semantic relationships between solutions [12,14,45]. Such methods enable the identification of logically equivalent solutions even when their implementations differ syntactically.

The most recent studies explore LLM-based grading systems, which are applied to evaluate code, generate feedback, detect conceptual misunderstandings, and support interactive assessment scenarios [7,8,9,16,29,38]. These systems rely on pretrained transformer models to analyse programs in context and produce natural-language explanations of grading outcomes.

Hybrid approaches combining execution-based testing, static analysis, and AI models have also been proposed, reflecting the coexistence of multiple grading strategies within contemporary automated assessment research [6,13,47].

3.2.2. RQ2—Use of AI and ML Methods

AI-based grading methods have been applied to analyse semantic properties of code, generate automated feedback, and support scalable assessment processes [7,12,14,29]. Several studies employ ML and neural models to examine structural, syntactic, and stylistic characteristics of programs in addition to functional correctness [9,10,13].

These approaches extend automated assessment beyond execution-based testing by incorporating features derived from program representations and learned models of code behaviour [10,12,14].

Research Question 2 examines how AI techniques are applied in AATs and how these approaches differ from traditional methods.

The reviewed literature shows an increasing use of AI-driven grading methods, particularly in studies published after 2022. ML approaches are employed to model relationships between program features and grading outcomes, allowing systems to evaluate functional correctness together with structural and stylistic properties of code [10,11,39]. These models typically rely on extracted features such as token sequences, abstract syntax trees, complexity metrics, or semantic representations of programs [10,12].

DL techniques further extend this approach by learning representations directly from source code. Study [12] applies syntactic knowledge point technology for automated programming assessment and code evaluation. Neural architectures, including convolutional and transformer-based models, are used to capture relationships between program structures and behaviours, supporting the identification of semantically similar solutions [12,14,45].

More recent studies explore LLM-based grading systems. These systems are applied to analyse code, generate automated feedback, explain errors, and support interactive evaluation scenarios [7,8,9,16,29,38]. LLM-based approaches operate on contextual representations of programs and natural language descriptions, allowing them to process code together with task specifications or student explanations [29,38].

Across the reviewed studies, AI-based grading approaches are used to support several functionalities, including semantic analysis of code, automated feedback generation, similarity detection, and scalable evaluation workflows [12,13,14,29].

The literature also reports practical challenges associated with these approaches, including the need for labelled training data, variability in model outputs, and concerns related to transparency, interpretability, and fairness in automated grading systems [3,8,18,29].

3.2.3. RQ3—Automated Assessment in Visual and Block-Based Programming

Table 6 shows that only a limited number of studies address automated assessment of VP/BP solutions. Research Question 3 examines the extent to which AATs support VP/BP environments.

The reviewed literature includes only a small number of studies addressing automated assessment in VP/BP contexts [19,30]. Most automated grading research focuses on text-based programming languages.

Existing approaches for VP/BP assessment typically analyse structural or semantic properties of programs rather than relying solely on execution-based testing. Some studies employ feature extraction methods to examine program constructs, event sequences, or block usage patterns [30]. Others explore automated evaluation of higher-level aspects of student solutions, including program organization, complexity, or demonstrated understanding of computational concepts [19,30].

The reviewed systems are generally developed for specific educational contexts, most commonly introductory or K–12 computing courses [19]. These tools often focus on analysing program artefacts rather than executing test cases, reflecting the design of BP environments [30].

Overall, the number of studies addressing automated assessment for VP/BP remains limited within the reviewed literature, and existing solutions are typically designed for particular platforms or learning scenarios [19,30].

3.2.4. RQ4—Educational Contexts

Most AATs are applied in introductory programming courses and higher education settings. However, the literature also reports implementations in online learning platforms, MOOCs, and secondary education contexts.

The reviewed literature shows that AATs are most frequently applied in higher education settings, particularly in university-level programming courses [2,4,16]. For example, study [38] combines traditional unit testing with LLM-based evaluation approaches. While unit tests provide binary pass/fail results, the LLM component enables deeper assessment of code that does not pass the tests through two evaluation strategies: a Direct approach, which applies grading rubrics directly to the submitted code, and a Reverse approach, which first attempts code correction before performing rubric-based evaluation. Many studies address large undergraduate classes, where automated assessment systems are used to evaluate assignments and support continuous feedback [2,4]. In these contexts, AATs are commonly integrated into learning management systems or online programming platforms [5,52].

A smaller subset of studies focuses on secondary education and K–12 environments. These works typically investigate automated assessment in introductory programming or computational thinking courses, including visual or block-based programming contexts [19,30]. The reviewed studies cover a range of programming languages, most commonly including Java [4,5,6,42,43,45,46,49,52], Python [9,15,17,52], Scratch [25], R [17], C# [48], SQL [47], and C/C++ [4,8,10,11,13,39,45,46], reflecting the diversity of programming environments in which automated grading is applied. Furthermore, the reviewed educational contexts include a variety of programming environments and assignment types, ranging from traditional command-line programming tasks to GUI-based and visual/block-based programming activities, each requiring different evaluation criteria and assessment strategies.

In addition to classroom use, several studies examine automated grading in online and distance learning settings, including MOOCs and cloud-based platforms designed for large-scale programming education [1,2]. These systems are used to support scalable assessment and provide consistent feedback in distributed learning environments [1].

3.2.5. RQ5—Reliability of LLM-Based Grading Systems vs. Human Graders

The results of the main topic of Research Question 5 are illustrated in Figure 6. Namely, it investigates the reliability of LLM-based grading systems compared to human evaluators.

Figure 6. Studies Evaluating Reliability vs. Human Grading.

Several studies explicitly compare automated grading outcomes with human assessment, examining agreement levels, consistency, and feedback characteristics [8,17,29,38]. These works evaluate the extent to which LLM-generated scores and explanations correspond to instructor judgments.

Traditional automated grading systems, particularly those based on predefined test cases, are commonly used to verify functional correctness and produce deterministic grading results [4,5]. Such approaches focus on program execution outcomes rather than qualitative properties of solutions.

More recent studies apply ML and DL techniques to analyse structural and semantic characteristics of code, enabling comparisons between model-based evaluation and human grading [10,12,14]. In addition, research on LLM–based grading investigates agreement with human assessors and the characteristics of generated feedback [7,16,17,29].

Across the reviewed literature, studies addressing AI-based grading reliability typically measure agreement with human scores, analyse feedback similarity, or evaluate grading consistency under predefined rubrics [8,17,29,38]. Some works also explore human-in-the-loop or assistive grading scenarios, where automated systems support instructors rather than operate independently [7,16].

4. Discussion

4.1. Reflections and Limitation of the Method

This review has several limitations that should be considered when interpreting the results. First, the search strategy included AI-related keywords in order to ensure coverage of emerging intelligent assessment approaches. Although traditional automated assessment methods were also included during the screening process, the search query may have favored AI-based studies and therefore influenced the final distribution of approaches.

Moreover, although equivalent search terms were used to broaden the search process, some relevant terminology variations may not have been fully covered, which could have resulted in certain relevant studies being overlooked.

Furthermore, the study selection and classification process involved human judgment, which may introduce a degree of subjectivity despite efforts to ensure objectivity and consistency. To reduce potential bias, the reviewed studies were independently examined by two authors, and disagreements were resolved through discussion and consensus. Nevertheless, differences in interpretation and categorization remain possible, particularly in studies combining multiple assessment techniques.

In addition, ethical and fairness considerations should be acknowledged, especially in the context of AI- and LLM-based assessment systems. Issues such as transparency, explainability, potential bias, and reliability compared to human evaluation remain important challenges in the adoption of intelligent automated grading systems.

Finally, the review was limited to studies indexed in IEEE Xplore, Scopus, and Web of Science and depended on the terminology used in titles, abstracts, and metadata. Although broad search queries and multiple databases were used to improve coverage, some relevant studies may not have been retrieved due to differences in indexing practices or alternative terminology used to describe automated assessment systems.

4.2. Interpretation of Observed Trends

The results of this review illustrated in Figure 5 reveal a clear shift in automated grading research from traditional execution-based approaches toward AI-driven solutions. Earlier studies predominantly relied on deterministic testing frameworks or rule-based evaluation, focusing primarily on correctness verification [5,10]. However, more recent research increasingly explores ML, DL, and LLMs as tools for analysing program structure, semantics, and student reasoning [12,14,40,43,45,47].

This trend reflects a broader transformation in educational technology, where assessment systems are evolving from correctness-checking tools into intelligent learning support systems capable of providing feedback, diagnosing misconceptions, and modelling student performance.

4.3. Dominance of LLM-Based Approaches in Recent Research

The increasing research interest of LLM–based approaches can be explained by several factors. First, LLMs are capable of understanding code in a semantic context rather than relying solely on syntactic or behavioural testing. This enables them to evaluate solution quality, readability, and intent, which were traditionally difficult to assess automatically.

Second, LLMs can generate natural-language feedback, making them particularly attractive for formative assessment [3,15,41,49,50]. Unlike classical autograders, which typically return binary correctness results, LLM-based systems can explain errors, suggest improvements, and support reflective learning processes.

Finally, the rapid availability of pretrained models and accessible APIs has significantly lowered the technical barrier for implementing intelligent grading systems. As a result, recent research increasingly focuses on integrating LLMs into educational workflows rather than developing grading logic from scratch [7,8,17,44].

4.4. Block-Based Programming as an Underexplored Area

In contrast to textual programming assessment, automated grading of visual or block-based programming environments remains comparatively underrepresented in the literature (Figure 3). One reason is the inherent difficulty of analysing graphical program structures, which often require specialised representations beyond standard code analysis techniques.

Additionally, block-based environments are more commonly used in early education contexts, where assessment objectives may focus on creativity, problem-solving, or conceptual understanding rather than strict functional correctness [19,25,30]. These learning goals are more difficult to formalise within AATs, which may explain the limited research attention.

This gap suggests that future research should explore methods for representing visual programs in machine-readable forms and developing assessment models tailored to early programming education.

4.5. Implications for Programming Education

The increasing adoption of AI-driven grading systems has several implications for educational practice [15,25,48]. AI-based AATs have the potential to provide immediate, personalised feedback, supporting self-paced learning and improving student engagement. They may also enable instructors to focus on higher-level pedagogical tasks rather than routine grading.

However, the reliance on AI-based evaluation raises concerns regarding transparency, bias, and reliability. Educators must therefore consider how such systems are integrated into assessment processes, ensuring that automated decisions remain interpretable and pedagogically meaningful.

4.6. Implications for Research on Automated Assessment Tools

From a research perspective, the findings highlight several directions for further investigation. First, more empirical studies are needed to evaluate the reliability of AI-based grading systems in comparison to human assessors [29]. Second, research should explore hybrid grading models that combine execution-based verification with AI-driven semantic analysis [9,42,47].

Furthermore, the limited number of studies addressing VP/BP environments according to Table 6. This suggests a promising opportunity for extending automated assessment beyond traditional textual languages. Finally, future work should examine how automated grading tools influence learning outcomes, not only grading efficiency.

5. Research Gaps and Future Work

The findings of this review highlight several important gaps in the current research landscape of automated programming assessment.

First, although AI-based grading systems are increasingly proposed, relatively few studies provide rigorous empirical comparisons between automated and human grading. As a result, questions of reliability, fairness, and pedagogical validity remain insufficiently explored. Future work should therefore prioritise controlled experiments that directly evaluate grading consistency and learning outcomes.

Second, while recent studies heavily focus on LLM–based grading, many of them emphasise system design or feasibility rather than long-term educational impact. There is a lack of longitudinal studies investigating how AI-driven assessment affects student learning, motivation, and skill development over time.

Third, research on automated assessment in visual or block-based programming environments remains limited. Most existing systems target textual programming languages, leaving early programming education comparatively underrepresented. This gap suggests the need for new approaches capable of representing and analysing graphical programs and assessing conceptual understanding rather than only functional correctness.

Fourth, many studies evaluate grading accuracy but overlook explainability and transparency of automated decisions. Given the increasing use of AI in educational contexts, understanding how grading decisions are produced is essential for maintaining trust and accountability in assessment systems.

Finally, relatively few works examine the integration of AATs into real educational workflows. Issues such as instructor adoption, student acceptance, curriculum alignment, and ethical considerations remain underexplored.

Based on these findings, several promising directions for future research can be identified.

Future studies should investigate hybrid grading models that combine execution-based verification with AI-driven semantic analysis, as such systems may balance reliability with interpretability. Additionally, research should focus on improving the explainability of AI-based grading systems, ensuring that feedback is not only accurate but also pedagogically meaningful.

Another important direction involves extending automated assessment to VP/BP environments and early education contexts, where current solutions remain scarce. Advances in program representation learning and multimodal analysis may enable progress in this area.

In addition to technological performance, future research should address the practical implementation challenges of AI-based grading systems, particularly the additional effort required for prompt engineering, automated test preparation, system maintenance, and educator training.

Moreover, future work should evaluate AATs in authentic educational settings, examining their impact on teaching practices, student engagement, and learning outcomes. Such studies would provide stronger evidence for the practical adoption of intelligent autograders.

Overall, addressing these gaps will be essential for advancing AATs from experimental tools toward reliable, transparent, and pedagogically grounded components of modern programming education.

6. Conclusions

This study presented a systematic literature review of automated assessment tools for programming assignments, with particular emphasis on the role of AI, ML, and LLMs. The analysis of 39 selected studies revealed a clear evolution of automated assessment approaches, progressing from traditional execution-based grading toward intelligent systems capable of analysing code semantics and providing meaningful feedback.

The results confirm that AI-driven grading methods, especially those based on LLMs, are becoming increasingly dominant in recent research. These approaches demonstrate strong potential for supporting scalable, feedback-oriented, and adaptive programming education. At the same time, the findings highlight several limitations, including the limited empirical evaluation of grading reliability, insufficient exploration of VP/BP environments, and the need for more transparent and pedagogically grounded assessment systems.

Overall, this SLR confirms that automated grading technologies are transitioning from tools focused solely on correctness verification toward intelligent educational systems that can support both instructors and learners. This evolution is reflected in the publication trends of the reviewed studies on Figure 5, which show an increasing research focus on AI- and LLM-based approaches, as well as in the evaluation objectives of the systems themselves, which increasingly emphasize feedback generation, personalization, and educational support alongside correctness assessment. Although many studies highlighted the potential educational benefits of AI-based grading systems, only a limited number provided rigorous empirical evidence regarding their long-term impact on learning outcomes and pedagogical effectiveness. However, their successful integration into educational practice requires careful consideration of reliability, explainability, and learning impact.

Future research should therefore focus on validating AI-based grading systems in real educational settings, developing hybrid approaches that balance automation and interpretability, and extending automated assessment to diverse programming paradigms and learner populations.