Large Language Model and Fuzzy Metric Integration in Assignment Grading for Introduction to Programming Type of Courses

Abstract

The integration of large language models (LLMs) and fuzzy metrics offers new possibilities for improving automated grading in programming education. While LLMs enable efficient generation and semantic evaluation of programming assignments, traditional crisp grading schemes fail to adequately capture partial correctness and uncertainty. This paper proposes a grading framework in which LLMs assess student solutions according to predefined criteria and output fuzzy grades represented by trapezoidal membership functions. Defuzzification is performed using the centroid method, after which fuzzy distance measures and fuzzy C-means clustering are applied to correct grades based on cluster centroids corresponding to linguistic performance levels (poor, good, excellent). The approach is evaluated on several years of real course data from an introductory programming course with approximately 800 students per year called “Programski jezici i strukture podataka” in the first year of studies of multiple study programs at the Faculty of Technical Sciences, University of Novi Sad, Serbia. Experimental results show that direct fuzzy grading tends to be overly strict compared to human grading, while fuzzy metric correction significantly reduces grading deviation and improves alignment with human assessment, particularly for higher-performing students. Combining LLM-based semantic analysis with fuzzy metrics yields a more nuanced, interpretable, and adaptable grading process, with potential applicability across a wide range of educational assessment scenarios.

1. Introduction

In the rapidly evolving landscape of computer science education, programming assignments serve as a cornerstone for assessing students’ technical proficiency, problem-solving skills, and creativity. However, the assessment of these assignments places significant demands on the instructor’s time and other resources [1], posing significant challenges for educators, particularly in larger classrooms. Traditional manual grading is labor-intensive, time-consuming, and prone to inconsistencies due to subjective interpretations of code quality. In addition to the feedback that AI tools can provide, several studies have shown that one of the functionalities pursued is the automatic grading of students [2]. While automated grading tools have alleviated some of these burdens, they often rely on rigid rule-based systems that prioritize syntactic correctness over nuanced aspects such as code readability, efficiency, or algorithmic elegance. This unvarying approach fails to capture the complexity of human judgment, leaving gaps in feedback that are critical for student growth.

The objective of this paper is to design and evaluate a grading framework that integrates a large language model (LLM) with fuzzy metrics to achieve nuanced, transparent and scalable assessment of programming assignments. This paper addresses the gap between these two approaches by integrating LLM code analysis with a fuzzy grading metric. LLMs, such as GPT-4, have demonstrated remarkable capabilities in understanding and generating human-like text, including code. As mentioned in [3], in LLMs, code and reasoning reinforce each other, since code is highly structured and has strong logic, guiding reasoning in training and inference. These models can parse programming syntax, infer logical intent, and even detect subtle errors or stylistic flaws. Meanwhile, fuzzy metrics can be used as a robust mechanism to quantify subjective criteria like “clean code” or “efficient logic,” which defy binary evaluation. Fuzzy logic has been widely adopted in educational assessment to model partial correctness and linguistic evaluation, overcoming limitations of crisp grading schemes [4,5,6,7].

The main contributions of this paper are as follows:

• A fuzzy grading model for programming assignments, where each grading criterion is represented by trapezoidal membership functions corresponding to linguistic evaluation levels;
• An LLM evaluation pipeline, in which students’ solutions are assessed according to predefined criteria and given fuzzy grades rather than crisp scores as an output;
• A fuzzy metric-based correction mechanism, employing fuzzy C-means clustering and inverse fuzzy distance weighting to adjust defuzzified scores and improve alignment with human grading;
• An empirical evaluation on real course data, sampled from several years of a large introductory programming course, demonstrating the behavior and limitations of crisp, fuzzy and corrected fuzzy grading;

The motivation for this integration lies in addressing the limitations of existing automated systems. Rule-based graders lack flexibility, while pure machine-learning approaches often operate as “black boxes,” offering little transparency into scoring rationale. By contrast, the proposed system combines the explainability of fuzzy logic’s rule-based architecture with the contextual intelligence of LLMs. Furthermore, it accommodates diverse evaluation criteria, from functional correctness to pedagogical goals like encouraging experimentation or adherence to best practices.

1.1. Field Overview

An overview of the field is displayed in [8], where it is shown that most studies are performed on the academic level of undergraduate students (99 out of 138 in total). Most popular subject domains were Language Learning (17%), Computer Science (16%) and Management (14%), followed closely by Engineering (12%) and Science (10%). AI usage in education is divided into five different categories: Assessment/Evaluation, Predicting, AI Assistant, Intelligent Tutoring System and Managing Student Learning. Assessment/Evaluation is further divided into categories: Automatic Assessment, Generating Tests, Feedback, Review Online Activities and Evaluate Educational Resources. This paper is focusing on Automatic Assessment. In the research performed in [9], multiple courses were graded by AI and by teachers simultaneously. ChatGPT (GPT-5.1) was used as an AI grader, without any additional fine-tuning, so-called zero-shot training. There were differences between grading, where AI had a tendency to give scores that would form a normal distribution, while teachers mostly gave non-passing grades. Agreement on the individual question was low, so there were a lot of situations where AI gave a high score, while the teacher gave a low score and vice versa. AI did the grading multiple times to compare the scoring consistency, where the average scoring and percentage agreements were roughly the same, even though single answers were graded differently between multiple AI grading runs. AI prompts are recommended to be as small as possible, so longer answers were graded worse. Even though there are differences and flaws, the final conclusion is that AI grading could hardly be distinguished from human grading. On the other hand, AI was considered not mature enough to be used for grading at the time. In [10], AI-powered tools were provided to K-12 teachers as a help for grading. It was observed what kind of misconceptions, myths and fears were present, which were influencing the adoption of the AI grader. It gave further depth to teachers on how AI works while complementing teachers’ work. Given both synthetic and real grading data, open-ended biology questions were graded by an AI using an analytic rubric. It was shown that, in order for the AI grader to be adopted by the teachers, they had to have some insights on how it is working to develop trust to use it. To sum up, current research trends in LLMs for automatic scoring are increasingly focusing on the interplay between sophisticated general language understanding and the specific needs in educational applications [11]. Thus, there has been a noticeable increase in research on LLM-powered automated essay scoring (AES) and automated short answer grading (ASAG) in the recent literature [12].

Regarding the combination of LLMs and fuzzy logic, there are emerging references, mostly recent preprints, but early research on this topic could be found in [13]. Here, fuzzy logic is used for model evaluation, but not in direct use by LLM for grading.

1.2. Chapters Overview

The introductory chapter outlines the challenges in traditional and automated grading, the potential of LLMs and fuzzy logic as complementary tools, and the overarching vision of the proposed framework. Subsequent chapters will delve into technical implementation, case studies, and validation against human grading benchmarks. Section 2 describes the existing grading process and introduces the proposed fuzzy and LLM methodologies. Section 3 presents experimental results and comparisons with human grading. Section 4 discusses the observed benefits and limitations of the approach and Section 5 outlines directions for future research. Ultimately, this research seeks to advance educational technology by creating a grading system that is not only efficient but also pedagogically aligned, transforming assessment into a dynamic tool for learning.

2. Materials and Methods

2.1. Current Grading Process

The current state of the grading is that it is performed manually, with some minimal automation to help the grader in the process. The grader is most often a teaching assistant on the course. Students listening to the subjects are divided into groups that could contain at most 32 students. Since every group has its own term when it’s in the computer laboratory and terms rarely overlap, every group receives a different assignment. There are three assignments during the semester, with different levels of difficulty, where the assignments become harder as the semester is going. Teaching assistant that is teaching the group of students is responsible for giving the suggestion of the exam assignment, while taking into consideration guidance provided by the head teaching assistant. Every teaching assistant needs to submit their assignments to the head teaching assistant in order for them to be reviewed. The responsibility of the head teaching assistant is to provide feedback and align the assignments between different teaching assistants/groups, in order to obtain approximately the same level of difficulty for all of the groups. Once the final draft is approved by the head teaching assistant, the assignment could be used in an exam.

Students are given assignments, and their solutions are being picked up from the laboratory as an archive of file systems of all computers in the laboratory. There are some helping tools that can extract student solutions in the form of source files, but using them by a grader is not mandatory. After the preparation step, the grader starts evaluating student solutions one by one. Evaluation includes the following few steps:

• Compilation of the solution and making an executable file;
• Running some static analysis checks, checking for memory leaks;
• Testing the output of the program with some given test input values;
• Manually going through an assignment and checking if the solution contains all of the requested elements, with proper use of programming constructs and library functions of the programming language.

Some simple automation of the evaluation steps is achieved using shell scripting to invoke a compiler (gcc) and do checks with static analysis (cppcheck), memory correctness (valgrind) and testing the output (combination of expect and diff utilities). Graders are still required to go through the solution manually and do the pragmatic analysis of the solution. For every type of assignment, there are categories that are being graded. Both their meaning and maximum number of points are defined by the head teacher assistant. The final grade for the assignment represents a sum of the points for every given category.

2.2. Introducing Fuzzification in the Grading Process

To make the grading process fuzzy, the linguistic variables are introduced as follows:

• Poor, for hardly matching the grading criteria or none.
• Good, for partial fulfillment, but there is a missing part.
• Excellent, for performing everything asked.

Such linguistic variables are converted into fuzzy sets. For every criterion in Table 1, there are three trapezoidal fuzzy sets that match the linguistic variables affiliation.

Table 1. Example of the criteria for grading by categories (max. 20 points).

Criterion No.	Description	Num. of Points
1	Correct main function, including directives and successful variable declarations	2
2	Correct input of data of the requested type, input of the arrays and strings (scanf, fgets, etc.)	3
3	Correct printing of data of the requested type, writing arrays and strings (printf, puts, etc.)	2
4	Correct usage of the selection and loop statements (if, switch, for, while, do-while)	3
5	Input validation (where it makes sense)	3
6	Adequate implementation of the algorithm (point of the assignment)	7

For example, criterion number 1 from Table 2 is shown in Figure 1.

Table 2. Fuzzy set per criterion.

Criterion No.	Poor	Good	Excellent
1, 3	(0.0, 0.0, 0.2, 1.0)	(0.0, 0.8, 1.2, 2.0)	(1.0, 1.8, 2.0, 2.0)
2, 4, 5	(0.0, 0.0, 0.3, 1.5)	(0.0, 1.2, 1.8, 3.0)	(1.5, 2.7, 3.0, 3.0)
6	(0.0, 0.0, 0.7, 3.5)	(0.0, 2.8, 4.2, 7.0)	(3.5, 6.3, 7.0, 7.0)

Figure 1. Criterion 1 fuzzy set.

After grading using fuzzy criteria, defuzzification is performed using the following formula:

$$score={\displaystyle \frac{{\sum }_{L\in \{\mathrm{Poor},\mathrm{Good},\mathrm{Excellent}\}}{\mu }_L·centroi{d}_L}{{\sum }_L{\mu }_L}}$$

(1)

In the numerator, the value of the membership function ${\mu }_L$ is multiplied by the centroid value of each trapezoidal function of each linguistic variable. In the denominator, only the membership function value is summed for each of the linguistic variables. This will give a concrete student’s score value as a final result of defuzzification.

2.3. Calculating Fuzzy Distance and Clustering Method

The fuzzy distance measures used in this work are grounded in the theory of fuzzy metric spaces originally introduced by Kramosil and Michálek, further formalized by George and Veeramani [14,15]. These frameworks extend classical metric concepts by allowing uncertainty and graded proximity between elements, making them particularly suitable for modeling imprecision in educational assessment. A broader overview of distance concepts, including fuzzy and generalized distances, can be found in [16].

The distance between two trapezoidal fuzzy sets $A=(a_1,a_2,a_3,a_4)$ and $B=(b_1,b_2,b_3,b_4)$ using the Euclidean distance is calculated as

$$D(A,B)=\sqrt{{\displaystyle \frac{1}{4}}\sum _{i=1}^4{(a_i-b_i)}^2}$$

(2)

Instead of metric (2) fuzzy S-metric defined in [17] will be used as follows:

$$\mathbf{s}(A,B,t)={\displaystyle \frac{D(A,B)}{t+D(A,B)}},\text{where} t>0$$

(3)

where $D(A,B)$ will be defined as an absolute difference in two trapezoidal fuzzy sets $A=(a_1,a_2,a_3,a_4)$ and $B=(b_1,b_2,b_3,b_4)$. The parameter t has an arbitrary value, greater than 0. By choosing a larger value for t, the closer the trapezoidal fuzzy sets are, the greater the fuzzy S-metric result. That way, in algorithms, aside from standard metrics and fuzzy metrics like the distance between objects, their aggregations are also considered (see [18]), for example,

$$\mathbf{D}(A,B,t)={\displaystyle \frac{1}{4}}\sum _{i=1}^4{\displaystyle \frac{|a_i-b_i|}{t_i+|a_i-b_i|}}, \mathrm{where} t_i>0$$

(4)

Fuzzy C-means (FCM) [19] is used as a soft-clustering algorithm. Classical K-means is rigid regarding the membership to a cluster, so this is a less restraining variant, allowing for one data point to belong in multiple clusters with different levels of membership. The steps in the fuzzy C-means algorithm are as follows:

• Initialization, where the number of clusters is given and the fuzzy membership matrix is initialized using random values;
• Cluster centers calculation, where centroids of every cluster are calculated using weighted average of data points, and weights are membership values;
• Update membership matrix: belonging to certain clusters means it is updated, taking into account the distance from the data point and cluster center;
• Recalculate cluster centers: obtain new cluster centers from the updated membership matrix;
• Convergence check: repeat the previous two steps until no significant change occurs or maximum number of iterations is reached.

As input for FCM, fuzzy grades are converted using the centroid method

$$C={\displaystyle \frac{a+2b+2c+d}{6}}$$

(5)

where a, b, c and d denote the parameters of the trapezoidal membership function. In order to do a self-correcting method of grading, the fuzzy distance and C-means approach are being combined. By clustering data, cluster centers are formed, which are used to determine the average student belonging to a certain group. Let k contain an element that belongs to this fuzzy set,

$$k\in \{G_{Poor},G_{Good},G_{Excellent}\}$$

(6)

where G is a fuzzy trapezoidal function for every category. For individual student’s grade (marked with $G_i$), calculate the fuzzy distance between every element of $G_k$ and take the closest one to determine the category in which it falls,

$$category\left(i\right)=\arg \underset{k}{min}\mathbf{D}(G_i,G_k)$$

(7)

Correct the fuzzy grade using fuzzy reverse distance as a weight factor (the closer it is, the stronger the influence),

$$new\_scor{e}_i={\displaystyle \frac{{\sum }_k\frac{1}{\mathbf{D}{(G_i,G_k)}^p}·centroi{d}_k}{{\sum }_k\frac{1}{\mathbf{D}{(G_i,G_k)}^p}}}$$

(8)

where p is an exponent used to further tune results, grouping values closer to the cluster’s centers. Notice that if D is a fuzzy metric, ${\mathbf{D}}^p$ is also a fuzzy metric if $p>0$, which follows from the properties of continuous t-norms and their induced orderings (see [18,20]).

2.4. Usage of LLMs for Grading

In this paper, OpenAI’s ChatGPT, LLM version GPT-5.1, was used. With Chain of Thought enabled, the prompter was in the position to see the thinking process of the LLM. As addressed in [11], it is characterized as a sequence of intermediary reasoning steps expressed in natural language, culminating in the final output. At the moment of writing, the best commercially available GPT model, which promises a smarter, more widely useful model with deeper reasoning [21].

2.5. Calculation and Visualization of Results

To visualize and calculate additional fuzzy grades, Python version 3.10.12 was used. Package numpy version 2.2.6 is used for data arrays and precise calculations. Fuzzy C-means calculation was performed using the scikit-fuzzy package, as well as trapezoidal representation of fuzzy grades. For visualization of Fuzzy C-Means results and trapezoidal functions that were made from the centroids of the clusters, the matplotlib version 3.10.7 package was used.

3. Results

3.1. Crisp Results

As a first experiment, a randomly chosen group of students performed the same assignment. Their work was processed by an LLM (in this case ChatGPT), with the instruction to grade the assignments using crisp criteria defined in Table 1. The assignments were then graded solely by the machine.

The LLM response was a detailed explanation of the points awarded for each criterion. The report was always concluded with the addition of points given for the criteria, yielding the final grade. This result was later used for comparison between human grading. The crisp grading workflow is visualized in Figure 2.

Figure 2. Crisp grading workflow.

Table 3 displays human grades in the first column and crisp LLM in the second. In the third column, there is a calculation of how different the LLM crisp grading is compared to the human grading, expressed in percentages.

Table 3. Comparison of human and crisp grading using LLM.

Human	LLM	Grading Difference [%]
12	11	8.33
20	20	0
10	12	−20
6	5	16.67
19	16	15.79
16	9	43.75
19	19	0
18	16	11.11
4	6	−50
15	13	13.33
20	18	10
1	1	0
17	15	11.76
13	11	15.38
17	13	23.52
14	11	21.43
16	14	12.5
15	13	13.33
16	17	−6.25
20	17	15
17	17	0
18	12	33.33
18	13	27.78
20	18	10
13	13	0
15	12	20
19	16	15.79
19	17	10.53
15	12	20
17	16	5.88
9	9	0
10	12	−20

3.2. Results Using Fuzzy Grading

Table 4 shows the human grades in the first column, the fuzzy LLM grades in the second and the defuzzified values in the third. In the fourth column, there is a calculation of how different the LLM fuzzy grading is compared to the human grading, expressed in percentages.

Table 4. Comparison of human and fuzzy grading using LLM.

Human	LLM Fuzzy	LLM Defuzzified	Grading Difference [%]
12	(4.7, 6.7, 8.7, 11.8)	7.9	34.17
20	(13.8, 17.2, 19.2, 19.8)	17.73	11.35
10	(5.5, 7.5, 9.5, 11.5)	8.5	15
6	(0.0, 1.5, 2.5, 4.0)	2	66.67
19	(13.0, 15.5, 17.5, 19.0)	16.3	14.21
16	(2.5, 3.5, 4.5, 6.5)	4.17	73.94
19	(13.5, 16.0, 18.0, 19.5)	16.8	11.58
18	(11.5, 13.5, 15.5, 17.5)	14.5	19.44
4	(0.0, 0.5, 1.5, 3.0)	1.17	70.75
15	(5.0, 6.5, 8.5, 10.5)	7.6	49.33
20	(13.0, 14.5, 15.5, 17.0)	15	25
1	(0.0, 0.5, 1.0, 2.0)	0.83	17
17	(7.0, 8.5, 10.5, 12.5)	9.6	43.53
13	(4.5, 6.0, 8.0, 10.0)	7.1	45.38
17	(6.5, 8.0, 10.0, 12.0)	9.1	46.47
14	(0.5, 1.5, 2.5, 4.0)	2.1	85
16	(7.0, 9.0, 11.0, 13.0)	10	37.5
15	(4.0, 6.0, 8.0, 10.0)	7	53.33
16	(12.5, 14.5, 16.5, 18.5)	15.5	3.125
20	(12.5, 14.5, 15.5, 17.5)	15	25
17	(12.0, 14.0, 15.0, 17.0)	14.5	14.7
18	(5.5, 7.0, 9.0, 11.0)	8.1	55
18	(9.5, 11.0, 13.0, 14.5)	12	33.33
20	(13.0, 15.5, 17.0, 18.5)	16.1	19.5
13	(6.0, 7.5, 9.5, 11.0)	8, 5	34.61
15	(4.0, 6.0, 7.0, 9.0)	6.5	56.67
19	(8.5, 10.5, 12.0, 13.5)	11.2	41.05
19	(12.0, 14.0, 15.0, 17.0)	14.5	23.68
15	(8.5, 10.0, 12.0, 13.5)	11	26.67
17	(11.5, 13.0, 15.0, 16.5)	14	17.65
9	(3.5, 5.0, 6.0, 7.5)	5.5	38.89
10	(4.5, 6.0, 8.0, 10.0)	7.1	29

3.3. Fuzzy Grading Correction

Since it is imprecise by definition, there is a possibility of correcting the fuzzy grading. First, FCM is used to determine centroid values in the clusters. The algorithm is run with parameters to create three clusters: Poor, Good and Excellent. Input values are calculated as centroids for certain fuzzy grades using Equation (5). Clusters are displayed in Figure 3.

Figure 3. FCM cluster centers for provided fuzzy grading values.

Centroids can be unwrapped in trapezoid membership functions. These trapezoids are limited to a minimum of 0 and a maximum of 20 points; therefore, Poor and Excellent are rectangular trapezoids as displayed in Figure 4.

Figure 4. Trapezoid functions derived from the FCM centroids.

Then, with the given centroid trapezoids, calculate the fuzzy distance for every grade. Using the fuzzy reverse distance as a weight factor and centroid for every linguistic variable, correct the grade.

One of the advantages of using fuzzy distance (either S-metric or T-metric) is the existence of the parameter t, where the fuzzy nature of the distance is defined. Regarding the chosen value of t, it is able to obtain more expected results.

The final workflow that includes fuzzy grading by the LLM and then grading correction is displayed in Figure 5.

Figure 5. Fuzzy corrected grading workflow.

In Table 5, along the human grading column, there are three more columns for different values of parameter p introduced in Equation (8). Since the values have a greater converging step up to $p=3$, there is no need to display results for greater values of p. In the last column, there is a calculation of how different the LLM fuzzy grading is compared to the human grading, expressed in percentages. During the calculations for Table 5, different values of t were used, but the value 100 was chosen because it was giving satisfying results.

Table 5. Comparison of human and corrected fuzzy LLM grading with different weight factor p.

Human	Fuzzy Corrected LLM (p = 1)	Fuzzy Corrected LLM (p = 2)	Fuzzy Corrected LLM (p = 3)	Grading Difference (p = 3) [%]
12	9.38	9.42	9.44	21.33
20	15.42	18.59	19.62	1.9
10	9.49	9.44	9.44	5.6
6	2	0.21	0.02	99.67
19	15.08	18.27	19.46	−2.42
16	5.5	2.98	1.68	89.5
19	15.22	18.41	19.54	−2.84
18	14.32	17.34	18.83	−4.61
4	2.41	0.32	0.04	99
15	9.22	9.32	9.41	37.27
20	13.89	16.8	18.42	7.9
1	3.02	0.54	0.09	91
17	10.02	9.72	9.53	43.94
13	8.97	9.14	9.33	28.23
17	9.73	9.53	9.46	44.35
14	2.32	0.3	0.04	99.71
16	10.32	10.03	9.71	39.31
15	8.91	9.1	9.32	37.87
16	14.88	18.03	19.32	−20.75
20	14.14	17.15	18.71	6.45
17	13.98	16.9	18.48	−8.71
18	9.4	9.41	9.43	47.61
18	11.9	12.93	13.24	26.44
20	14.72	17.89	19.24	3.8
13	9.51	9.44	9.44	27.38
15	8.49	8.58	8.97	40.2
19	11.25	11.54	11.23	40.89
19	13.98	16.9	18.48	2.74
15	11.11	11.29	10.92	27.2
17	13.59	16.27	17.87	−5.12
9	7.39	6.56	6.56	27.11
10	8.97	9.14	9.33	6.7

4. Discussion

As expected, the provided results may vary. While consistency between running is pretty much achieved (small differences in grading the same assignment twice), as it was noticed in [22], some assignments were graded too harshly, especially with fuzzy grading criteria. Despite the apparent strengths of LLMs, they are not specifically designed for AES, nor is it clear how reliable they may be if employed for such purposes [23]. Similar is showing for programming assignments. When comparing the results from Table 3, deviations from the human grading are going up to 50%. Apart from that, deviations were going in both sides; sometimes LLM was more favorable than human grading. By converting grading criteria into fuzzy values and letting LLM grade with such criteria, asking back for a fuzzy value first, and then performing a defuzzification into a crisp value, two things are clearly visible: grading is much more strict, even too much and there are not any cases where LLM gives more points than a human. If we take human grading as fair, this is really unfavorable for students. By using the grading correction where FCM is used to determine the cluster center and centroids of the membership functions and inverse fuzzy distance from every category (Poor, Good, Excellent), certain corrections could be achieved in the right direction.

Taking a look at the cluster center values in Figure 3, since the centers are in smaller interval than min–max points from 0 to 20, scaling had to be performed, using the formula.

$$scaled\text{\_}scor{e}_i=(corrected\text{\_}scor{e}_i-c_{min})·{\displaystyle \frac{20}{c_{max}-c_{min}}}$$

(9)

In Table 5 there are results for different values of p. The bigger the p was, the more fair the grades were becoming for the best students. There are some cases of harshly graded assignments, where the LLM simply could not determine the real grade; in the case of fuzzy grading, it was too strict. Such cases are becoming worse by increasing the p, because they are getting closer to the center of the “Poor” cluster. Median values of different grading methods are displayed in Table 6.

Table 6. Median of the grading difference percentage for different methods of grading.

Crisp	Fuzzy	Fuzzy Corrected (p = 3)
11.44	33.75	27.155

There are a few facts, as follows, that should be taken into consideration when dealing with LLMs:

• Training data of the LLM.
• Uncertainty in response, since an identical prompt can give two different answers.
• Size of the model used; the bigger it is, the better it could comprehend the given task.
• Hallucinations, which occur when the model is unsure of the answer.

This research was performed using the zero-shot technique, where the prompt is given directly to, in this case, ChatGPT, model GPT-5. Although it could be said that it performed well while grading, there were some situations that could be handled better. As it was mentioned in [24], one promise of integrating LLMs into education lies in their capacity to provide feedback. Apart from the grade, the LLM response contained a detailed explanation of the grading by provided criteria. In some cases, the grading was harsh, which could be marked as a gross mistake. As it was stated in [25], from a comprehensive perspective, there are still issues with consistency and reliability. The reasons for this could be found in the generated answer, and the LLM could be provided with an explanation of why what it generated was a mistake. This is time-consuming, so there is another way of fine-tuning the model with a set of previously prepared data of old tests and grading so the model could be trained. According to [26], the fine-tuned models significantly outperformed the base models in accuracy and reliability.

Human grading was used as a reference, which is also imprecise. Grading is always subjective, even when the grader tries to be as objective as possible. There is always a possibility of a human-made mistake while grading, which could go under the radar if it benefits the student. This issue could be less if multiple human graders graded the same student’s assignments using the same criteria. The same is for LLMs; to obtain better results, there should be more models involved in grading.

4.1. Differences Between Crisp and Fuzzy Grading Results

Crisp grading, where the LLM outputs a single numerical score per assignment, generally follows the overall trend of human grading but exhibits notable variability at the individual assignment level. While average deviations remain moderate, crisp LLM grading occasionally overestimates or underestimates student performance, reflecting sensitivity to prompt interpretation and limited ability to represent partial correctness. This approach treats grading decisions as binary or sharply bounded, which reduces expressiveness when student solutions are only partially correct.

In contrast, fuzzy grading produces substantially different behavior. By representing grades as trapezoidal membership functions associated with linguistic categories (poor, good, excellent), fuzzy grading captures uncertainty and partial fulfillment of grading criteria more explicitly. However, the results show that direct fuzzy grading is systematically more conservative than both human and crisp LLM grading. Defuzzified fuzzy scores are consistently lower, especially for mid-range and weaker student solutions, leading to significantly larger deviations, often exceeding 30–50% from human grading.

The key observed difference, therefore, is that crisp grading prioritizes approximation to human scores but lacks nuance, whereas fuzzy grading prioritizes nuanced representation of uncertainty but introduces a strong bias toward harsher evaluation when applied without correction. This divergence highlights that fuzzification alone does not guarantee fairness or alignment with human judgment and motivates the need for fuzzy metric correction mechanisms.

4.2. Limitations of LLMs in Awarding Grades and Differences from Teacher Grading

The main limitations observed during experimentation with LLM grading are the following:

• LLMs lack contextual calibration to a specific cohort;
• LLMs exhibit sensitivity to prompt formulation and internal variability;
• LLMs apply grading criteria more rigidly than teachers;
• LLMs lack accountability and formative intent;
• LLMs do not possess a stable notion of ground truth in grading.

Human teachers implicitly normalize grading by considering the overall performance of a group, the difficulty of a particular assignment instance, and the expected learning outcomes at a given stage of the course. In contrast, LLMs evaluate each submission largely in isolation, relying only on the provided prompt and grading criteria. This absence of cohort awareness contributes to inconsistencies and systematic bias, particularly visible in fuzzy grading results, where mid-range student performance is penalized more severely than in human grading.

Identical student solutions can receive different scores across multiple grading runs, especially in crisp grading. Teachers, while subjective, tend to maintain internal consistency within a grading session and across a student group. This stochastic behavior of LLMs introduces uncertainty that is not directly observable in traditional grading practices. LLMs could be instructed with the specific prompt to be more loose or strict with grading, which can add up to the stochastic behavior of the grader. Especially, the system needs to be resilient to the prompt injection, where a student could try to put a word or sentence in a code that will be an instruction to the LLM.

Human graders often exercise pedagogical judgment, rewarding partial understanding, recognizing creative or unconventional solutions, and compensating minor technical errors when the core algorithmic idea is correct. LLMs, particularly when guided by explicit fuzzy criteria, tend to interpret missing or imperfect elements strictly, resulting in systematically lower scores. This behavior was evident in the fuzzy grading results, where defuzzified scores consistently underestimated human grades.

Teachers award grades not only to quantify performance but also to motivate students and support learning progression. Grading decisions may intentionally favor borderline cases or reflect improvement over time. LLMs, by contrast, optimize for internal consistency with the prompt rather than pedagogical impact, which can lead to grading outcomes that are technically defensible but educationally misaligned.

Human grading, despite being subjective, is anchored in institutional norms, shared experience among instructors, and tacit knowledge of course expectations. LLMs derive their judgments from training data and probabilistic inference, which may not align perfectly with local grading standards unless explicitly calibrated or fine-tuned.

4.3. Comparing with Other Automated Methods and Further Research

Unlike in [27], which uses a classifier to predict the essay grades, FCM is used to cluster the previously graded data. Since three groups were given upfront, their cluster centers are used as a base for correcting the grading formula. Assignment data does not need additional labeling in case of clustering algorithm usage, which makes it usable in this case, where no obvious labeling of the assignments could be achieved.

To compare this method with AES, the difference is that AES has primarily focused on generating a final score for student responses rather than providing detailed feedback [28]. While using ChatGPT for grading, it always gave a detailed explanation of why something was graded as it was, often with suggestions on how the assignment could be corrected. One flaw is that the current explanation is only visible to the grader, not the student. Some sort of AI integration in a web app would be required for the student to see the report as well.

Further research in the field could take grading similar to Massive open online course courses: (1) Instructor grades, (2) Peer grades and (3) ChatGPT grades [29]. Apart from it, multiple instructors or LLMs could be used to grade previously anonymized data using the crisp and/or fuzzy grading criteria. For now, pure LLM grading should be used with human supervision, in case of serious mistake, which can occur while grading. Upgrading to this could be an agentic approach, where an AI agent could choose between multiple grades of the same assignment, do additional checks and correct the grading if needed.

Regarding fuzzy metrics and fuzziness, new approaches could be explored. A great choice of different fuzzy metrics (with appropriate parameters) gives the possibility of obtaining a better description of the research results. The current approach using clustering and grade correction needs to re-scale the results because of the cluster centers that are inside the domain. Also, for the best students to receive the prize of giving them maximal points, a correction factor needed to be powered, and all of these were human decisions along the way. Also, another thing that could be handled by an AI agent in the future, if needed.

Overall, this is one new method to grade students’ programming assignments. It benefits in the way that it lowers teachers’ time spent with manually testing and grading programming assignments. Additionally, instructors have the opportunity to spend more time giving feedback and revising students’ work [30].

5. Conclusions

This paper presented a hybrid framework for grading programming assignments that incorporates fuzzy metrics in LLMs. The motivation was to overcome the limitations of both traditional manual grading and existing automated approaches, which often rely on crisp evaluation and fail to capture partial correctness, stylistic quality, and uncertainty inherent in student solutions.

The proposed approach leverages LLMs to perform semantic analysis of student code according to predefined grading criteria, producing fuzzy evaluations expressed through linguistic variables and trapezoidal membership functions. These fuzzy grades are subsequently defuzzified and refined using fuzzy distance measures and FCM clustering, enabling adaptive grade correction based on the structure of historical grading data.

Experimental results on real course data indicate that crisp LLM grading can approximate human grading but exhibits notable variability. Fuzzy grading provides a more nuanced representation of student performance; however, when applied directly, it tends to be overly strict. The introduction of fuzzy metric correction improves alignment with human grading, particularly for higher-performing students, while preserving interpretability and transparency. These findings confirm that fuzzy logic is well suited for modeling uncertainty and partial correctness in programming assessment and that LLMs offer effective semantic analysis and feedback generation capabilities.

Several limitations remain. The results depend on prompt design, model variability, and parameter selection within fuzzy metrics and clustering methods. Moreover, human grading itself is inherently subjective, which limits the notion of an absolute ground truth. Consequently, automated grading systems based on LLMs should currently be applied with human oversight, especially in high-stakes assessment contexts.

Future work will focus on fine-tuning language models using historical graded data, exploring alternative fuzzy metrics and aggregation operators, and automating parameter calibration. The proposed framework represents a step toward fairer, more transparent, and pedagogically aligned automated grading systems, with potential applicability beyond introductory programming courses to other domains requiring nuanced evaluation.