Assessing Learning Progression in a Vehicle Dynamics Course Using a Rubric-Based, GenAI-Assisted Open-Ended Approach

Abstract

This study introduces a formative assessment approach for examining changes in cognitive demand across student responses using open-ended questions and rubric-based evaluation. The method is designed for small cohorts and emphasizes instructional refinement across predefined cognitive levels. It integrates Generative AI to support rubric application, scoring consistency, and identification of response patterns without replacing instructor judgment or established assessment practices. The approach addresses challenges in specialized programs with low enrollment, where large-scale statistical designs may be impractical, particularly in rapidly evolving fields such as automotive engineering. The methodology was applied in a vehicle dynamics course with 19 students, self-assigned to 6 teams of similar size by selecting an available slot; teams were then allocated to one of two learning paths by team number, in which the same activities were delivered in reversed order in a crossover design combining simulation-based and experiential activities. Assessments were aligned with Bloom’s taxonomy and ABET outcomes. Four Generative AI models (GPT-4 Turbo, GPT o4-mini, DeepSeek, and GPT-4o-mini) were used to support scoring under rubric and prompting procedures defined by an experienced instructor. Overall, inter-model agreement was moderate (Krippendorff’s $\alpha =0.666$). This value did not meet the reference criterion of 0.7 adopted in the study and indicates meaningful model-dependent variation in rubric application. Within this limitation, task-level performance indicators showed a shift toward higher cognitive demand, particularly at the application level, with responses moving from conceptual definition toward early analysis. The approach also surfaced learning gaps that informed instructional adjustments, supporting its potential as a diagnostic tool for continuous improvement in engineering education.

1. Introduction

The emergence of Generative Artificial Intelligence (GenAI) in 2022 introduced significant challenges to academic practice, particularly in higher education (McDonald et al., 2025; Sakirin & Said, 2023; Nam & Bai, 2023; Jiang et al., 2026). In Sinha et al. (2023), the authors highlight these challenges in the context of STEM education and emphasize the need for innovative teaching approaches that embrace the opportunities presented by new technologies. They advocate for assessments that emphasize logical reasoning, deduction, inference, and the construction of abstract concepts. At the same time, the integration of emerging tools such as Generative Artificial Intelligence (GenAI) is still in its early stages within engineering pedagogy. Recent studies indicate that the application of GenAI to support instructional assessment is a novel and evolving area of research, with promising potential to enable more diverse and innovative assessment formats. Preliminary evidence suggests that it may help instructors move beyond traditional grading practices to better capture complex learning outcomes. Structured uses of GenAI, when combined with instructor oversight, have also been identified as promising approaches to support assessment literacy and scalable evaluation strategies, particularly in data-limited or resource-constrained settings (Xia et al., 2024).

In Beimel et al. (2025), the authors compare conventional instructor-led learning with learning supported by GenAI. Although GenAI-supported learning yielded better performance on moderately difficult tasks, the instructor remained a key figure for supporting complex problem-solving and the social and emotional dimensions of learning. In Salinas-Navarro et al. (2024), the potential of GenAI to enhance experiential learning and authentic assessment is studied through a thing ethnography methodology, namely, interviewing several GenAI tools about complex pedagogical concepts and analyzing their responses. However, the main limitation of this work is the lack of implementation in a real classroom setting. The integration of GenAI as an “agent to learn with” is explored in Salinas-Navarro et al. (2026), where a live case study is designed to enhance experiential learning among students in an operations management course. The researchers reported that GenAI supported each stage of Kolb’s experiential learning cycle and that students generally perceived the activity positively; however, they also identified risks associated with the uncritical use of AI and over-reliance on generic AI-generated suggestions.

Open-ended questions are a useful tool because they allow students to express reasoning, justify decisions, and connect concepts in ways that multiple choice formats do not capture, which is particularly relevant in a technical course where competence often depends on how a student articulates the relationship between concepts and system behavior. This flexibility, however, makes open-ended responses time consuming and difficult to score consistently, particularly in specialized, low enrollment programs. Generative AI was introduced to address this limitation, since recent studies have shown that large language models can evaluate open-ended responses with reasonable consistency, supporting instructors without fully removing their judgment from the process. Jauhiainen and Garagorry Guerra (2025) and Jauhiainen et al. (2026) used generative AI models to directly evaluate the quality of student written responses, comparing model generated scores against human graders. These studies, however, evaluate consistency through repeated runs of the same prompt rather than through systematic variation of prompt wording, so prompt sensitivity in the stricter sense, how changes in instruction phrasing affect scoring, remains largely unexamined in this line of work. A related study by the same authors addressed a different dimension of sensitivity and fairness, testing how grading alignment with human evaluators was affected when student responses were translated from a low resource language into English before scoring, a comparison that also speaks to whether AI assisted grading performs equitably across different language backgrounds (Jauhiainen & Garagorry Guerra, 2026). Beyond prompt sensitivity, rubric reliability itself has received uneven attention in this literature, with Krippendorff’s alpha values varying widely depending on model and rubric design (Pecuchova et al., 2025).

Various strategies have been shown to enhance the effectiveness of the teaching–learning process in engineering education. Among them, active learning methodologies stand out for their ability to develop high-level technological and professional competencies in students. These approaches aim not only to reinforce theoretical understanding but also to equip students with the skills needed to address complex problems, adapt to emerging trends, and respond to rapid changes in the industry (Nocera Alves Junior et al., 2024).

Simulation-based learning is an effective approach to introduce engineering students to realistic problem-solving scenarios. With most engineering undergraduates now owning personal computers and universities offering access to computer labs and licensed software, the integration of simulation tools into coursework has become increasingly accessible and widespread (Negahban, 2024). Among the documented benefits of simulation-based learning are increased student motivation, development of collaborative skills, and stimulation of critical thinking in complex problem-solving contexts (Campos et al., 2020). Additionally, simulations allow students to engage with real-world scenarios while minimizing the risks associated with practical applications, such as working with high-power electronics (Suthar & Joshipura, 2023), or operating in environments that are otherwise difficult to access (Alsalemi et al., 2021).

Experiential learning is a process through which learners construct knowledge, skills, and values from direct experience (Hajshirmohammadi, 2017). This approach is inherently intuitive: students engage in hands-on activities or experiments, reflect on the results, and abstract new knowledge that can be applied to future problems.

Recent studies have demonstrated the value of experiential learning in engineering disciplines. For example, Adnan et al. (2023) explores its application in robotics courses, where students developed both computational and adversarial thinking skills. The study concluded that engaging students in robotics-related activities enhances critical thinking and problem-solving abilities. Similarly, Čehovin Zajc et al. (2023) presents the design of a low-cost robotic platform, reporting successful implementation of active learning across multiple educational levels, from elementary school summer programs to undergraduate courses.

In the context of vehicle dynamics education, previous studies have shown the value of simulation, experimental platforms, and motion-based or instrumented learning environments for connecting theory with applied vehicle behavior (Hernandez-Alcantara & Morales-Menendez, 2016; Huang & Chen, 2003; Hulme et al., 2009; Lukac et al., 2021; Mehrtash, 2022; Mehrtash & Centea, 2017; Mehrtash et al., 2019; O’Brien et al., 2018; Redkar, 2012).

This study is framed by the alignment of learning outcomes, instructional activities, and assessment indicators, drawing on ABET performance indicators for outcome alignment and Anderson’s cognitive level classification for task demand (ABET, 2017; Anderson & Krathwohl, 2001). Cognitive task level is used here to classify the demand of each response, ranging from conceptual definition to application in a technical context and, when possible, analysis of system behavior. This classification is treated as a descriptive indicator of task demand within this course, not as evidence of a validated learning progression or trajectory in the formal sense used in the education research literature. Cognitive task level is operationalized as the level of performance elicited by each assessment task. Thus, comprehension is represented by the ability to define and explain a core concept, application by the ability to use that concept to interpret a technical situation, and analysis by the ability to relate observed system behavior to underlying mechanisms and cause-effect relationships.

Within this framework, GenAI is used as a support tool for applying predefined rubrics to anonymous open-ended responses, while preserving instructor judgment in the interpretation of learning outcomes.

Accordingly, this exploratory study addresses the following research questions:

• RQ1: What cohort-level patterns in rubric-based performance are observed across open-ended assessments targeting comprehension, application, and early analysis within the instructional sequence?
• RQ2: What descriptive differences in early performance and score variability are observed between the simulation-first and experimentation-first instructional sequences?
• RQ3: What levels of intra-model repeatability and inter-model agreement are observed when Generative AI models repeatedly apply the same instructor-defined rubric to the same anonymous open-ended responses?

Because responses were collected anonymously and could not be linked across assessment occasions, the analyses are conducted at the cohort level and do not represent individual longitudinal trajectories. The assessments also used different prompts and targeted different cognitive demands; comparisons across them are therefore interpreted as task-specific patterns in cohort performance rather than as direct measurements of within-student change. Comparisons between the two instructional sequences are descriptive. The AI-assisted analysis is limited to intra-model repeatability and inter-model agreement, as defined in Section 3.4.

This paper proposes a structured methodology that combines experiential and simulation-based learning activities with open-ended, non-graded assessments. Student responses are evaluated using predefined rubrics and supported by AI-assisted analysis to support consistency in diagnostic interpretation. The goal is to help instructors identify evidence of learning outcomes achieved at different cognitive levels and refine instructional strategies, even in data-constrained classroom settings. The AI-assisted evaluation procedure is characterized through an analysis of intra-model repeatability and inter-model agreement.

Brief Review of Educational Methodologies in Vehicle Dynamics

Vehicle dynamics has been part of the engineering curriculum for over a century (Jazar, 2017). Achieving satisfactory handling properties, including effective acceleration, braking, cornering, and ride comfort, depends on understanding how forces and moments generated within the vehicle influence its motion (Dukkipati et al., 2008; Gillespie, 1992). Over the last few years, the development of vehicle dynamics has moved toward modeling, analysis, and optimization of multi-body systems, among other topics. In recent years, developments in accurate positioning systems, real-time sensing, computational power, and intelligent algorithms have further expanded the scope and complexity of vehicle dynamics research and education (Jazar, 2017).

Despite the relevance of vehicle dynamics in automotive engineering education, assessing student understanding, particularly in low-enrollment groups, remains a persistent challenge. Traditional evaluation methods may lack the nuance required to identify how students progress across different levels of task demand, especially when responses involve open-ended reasoning rather than closed-form answers. The current challenges faced by the automotive industry urge vehicle dynamics instructors to go beyond the traditional theoretical foundations of classical dynamics. It is increasingly important to integrate practical tools and industry-relevant skills into the curriculum, such as simulation, communication systems, instrumentation, and data acquisition (Pacher et al., 2024).

Active learning methods have been increasingly incorporated into engineering programs to develop not only course theory but also disciplinary and transversal competencies in students (Sukackė et al., 2022). Project-based learning (PBL) has been shown to promote student engagement and critical thinking through open-ended design projects (Tharakan, 2022). In Redkar (2012), the author presents an advanced vehicle dynamics course based on the understand-crawl-walk-run approach, where students analyze and solve a practical problem using numerical multi-body software (MSC-ADAMS 2011). Similarly, Mehrtash et al. (2019) uses a virtual environment by incorporating a CARSIM laboratory to simulate the Fishhook performance test, helping students better understand rollover phenomena in commercial vehicles. Other authors have incorporated motion simulation into vehicle dynamics courses to provide experiential learning activities, concluding in general that simulations of real-life situations related to vehicle dynamics motivate students to connect experience with theory and promote a positive perception of their engineering education and their ability to solve problems in professional situations (Hulme et al., 2009; Mehrtash & Centea, 2017; O’Brien et al., 2018).

An enhancement to simulation experiences in vehicle dynamics courses is the use of didactic platforms, where students interact with a more realistic environment by collecting data from physical variables and connecting sensors. For instance, in Hernandez-Alcantara and Morales-Menendez (2016), a scaled quarter-vehicle prototype is integrated to implement a research-based learning approach analyzing the vertical dynamics of a suspension system as a way to motivate students to study beyond the topics of the vehicle dynamics course. A hydraulic motion simulator is proposed to emulate a 2 degree-of-freedom vehicle motion model, known as the bicycle model, in Huang and Chen (2003). An instrumented kart is implemented in Mehrtash (2022) to provide practice- and industry-level experimentation on handling performance for a road vehicle dynamics course. With a different aim, Lukac et al. (2021) proposes a learning platform for teaching automotive embedded software engineering in an M.Sc. course to introduce students to protocols commonly used in the automotive industry.

2. Definition of the Learning Process

A total of 19 students enrolled in the Vehicle Dynamics course of the Automotive Engineering Program participated in this study. The cohort was divided into two parallel classes under the same instructor, both following the same instructional design. The learning unit spanned six weeks in total: four weeks of conventional theory sessions (lectures, assignments, quizzes), followed by two weeks of crossover activities (simulation and experimental benchmarks).

According to the program of the course, the competence that students will acquire is: To analyze different kind of vehicles and their features in order to design and/or verify the behavior of systems or subsystems within the car, offering to the user a balance between safety, comfort and performance.

The activities proposed in this work are related with the longitudinal vehicle dynamics, including the following topics:

• Acceleration and braking performance;
• Powertrain dynamics;
• Tire dynamics.

The information used to teach the course is extracted mainly from the reference text books in the topic, such as Jazar (2017); Gillespie (1992); Dukkipati et al. (2008); Rajamani (2014), among others. Even some of the cited books include computational tools to enhance the understanding of theoretical concepts, real-world application insights are not studied in the books, such as sensors, actuators, communications, data acquisition, etc., valuable knowledge in the industrial environment.

2.1. Evaluation of the Learning Outcomes: Proposed Methodology

Despite the global relevance of the automotive industry, relatively few universities offer dedicated programs in Automotive Engineering. A review of the Accreditation Board for Engineering and Technology (ABET) programs reveals that only four institutions offer degrees directly related to automotive engineering. In contrast, there are 463 institutions offering Mechanical Engineering, 149 offering Industrial Engineering, and 407 offering Electrical Engineering programs. In the specific case of Universidad de Monterrey, the Automotive Engineering program also shows a comparatively low enrollment rate. These limitations present challenges in implementing large-scale control groups to evaluate the effectiveness of pedagogical strategies aimed at improving student learning. To address this issue, this paper proposes a formative methodology for collecting diagnostic evidence of learning outcomes in small student groups, based on open-ended questions aligned with course topics and predefined cognitive levels.

The importance of logical reasoning, deduction, inference, and the construction of abstract concepts is described in Sinha et al. (2023). The method proposed in this work aligns with this vision by using anonymous, non-graded diagnostic assessments to examine evidence of student reasoning, while relying on traditional assessments, such as reports, exams, and problem-solving tasks, for final grading. Students were informed in advance about the general nature and purpose of the activity, the voluntary nature of their participation, and the anonymous handling of their responses. However, the specific assessment format, topic, prompts, and timing were not disclosed in advance to limit targeted preparation.

The proposed assessment is intended as a formative and diagnostic procedure. Each open-ended question is designed to elicit observable evidence associated with a specific learning outcome and cognitive level. The resulting scores are therefore interpreted as indicators of task-level performance within the studied cohort. Within the rubric structure, technical accuracy and use of technical language provide the primary evidence of conceptual understanding, while clarity of expression supports the consistent interpretation of students’ reasoning.

The study included 19 students enrolled in the Automotive Engineering program, organized into 6 teams of similar size (five teams of three students and one team of four). Team formation was self-selected, with students choosing an available slot through the course learning management system, Blackboard. Teams were then allocated to one of two learning path orders based on team number: teams 1, 3, and 5 completed the simulation activity first followed by the experimental activity, while teams 2, 4, and 6 completed the experimental activity first followed by the simulation, in a crossover design. All students were enrolled under the program’s standard prerequisite structure. No formal pretest of prior knowledge was administered; the study was situated within the first instructional unit of the course, focused on longitudinal vehicle dynamics, with slip phenomena selected as the focal topic due to its complex interaction between wheel behavior and acceleration performance. All 19 students completed the full set of activities, with no missing data or attrition. Teams were physically separated during activity sessions to support closer instructor attention and reduce absences, an approach made feasible by the small cohort size typical of this specialized program.

It is important to clarify that this study does not propose an automatic system for assigning numerical grades to student performance. The instructor defines the learning outcomes, selects the assessment prompts, establishes the rubrics, supervises the learning activities, and interprets the aggregate results. GenAI is used as an assistance tool to apply predefined rubrics to anonymous student responses and to support the identification of response patterns. The consistency of AI model evaluations is assessed in Section 3.4 through a model-consistency analysis.

Algorithm 1 outlines the proposed alignment chain linking learning outcomes, instructional activities, open-ended prompts, rubric-based scoring, and instructional interpretation. It is intended to support evidence-informed refinement of learning activities in small cohorts or resource-constrained settings.

Algorithm 1 Evaluation of Learning Activities Using AI-Assisted Rubric-Based Analysis

Require: Learning outcomes, instructional activity, open-ended question Ensure: Diagnostic performance analysis and instructional improvement recommendations 1: Define learning outcomes to be assessed: 1. Specify the targeted cognitive level (e.g., comprehension, application, analysis) 2. Align the outcome with ABET indicators or Bloom’s taxonomy 2: Design an instructional activity: 1. Choose the method (e.g., simulation, experimentation, case study) 2. Ensure alignment with the intended cognitive process 3: Formulate an open-ended question that elicits observable reasoning 4: Administer a non-graded diagnostic assessment without prior disclosure of its specific format, topic, prompt, or timing, and collect responses anonymously. 5: Evaluate responses with AI-assisted rubric-based scoring: 1. Define a rubric including content-related criteria and clarity of expression 2. Use GenAI to support scoring per criterion, under instructor-defined calibration and supervision 6: Analyze results: 1. Compute descriptive statistics (mean, median, variance) 2. Categorize performance (Poor, Regular, Good, Excellent) 7: Compare with previous diagnostic assessments, when available, to identify cohort-level shifts in task-specific performance 8: Extract aggregate insights and propose instructional improvements

2.2. Learning Outcomes

The following learning outcomes were stated according to the Student Outcomes and Performance Indicators from ABET (ABET, 2017), in order to define accurately the learning objectives of the proposed activities:

• Define the fundamental concepts of longitudinal vehicle dynamics, including tire forces, traction, braking performance, and powertrain characteristics, and identify their basic influence on vehicle motion.
• Describe the relationship between tire forces, traction limits, and powertrain performance in longitudinal vehicle dynamics, and explain how these factors affect vehicle behavior during acceleration and braking.
• Apply the principles of longitudinal vehicle dynamics to interpret experimental or simulated data and assess the effect of ABS on traction and braking performance.
• Analyze the relationship between tire slip, traction forces, and powertrain efficiency by interpreting experimental data and comparing it with theoretical models to assess vehicle performance during acceleration and braking.

2.3. Learning Instruments

The experimental platform consists of a vehicle instrumented with sensors that continuously monitor variables such as temperature and velocity as shown in Figure 1. Data are extracted via a USB-to-CAN bus interface (Kvaser Leaf Light V2) connected to the OBD-II port (Rimpas et al., 2020), and communicated to Matlab through the Vehicle Network Toolbox by specifying each variable’s Parameter Identifier (PID) (MathWorks, 2024a, 2024b). Additional sensors—LVDT sensors, accelerometers, and laser profilometers are installed at strategic locations to study vertical, planar, and roll dynamics, as well as road-vehicle interaction. All sensors connect to a CompactRIO data acquisition system (National-Instruments, 2024), with signals recorded through LabView software 19.0.

Figure 1. Vehicle Dynamics Mobile Laboratory.

The simulation component is implemented in Simulink, a block-diagram environment that allows students to model multi-domain systems intuitively (MathWorks, 2025). The longitudinal dynamics block computes the friction coefficient from estimated slip, along with vehicle deceleration, speed, stopping distance, and tire torque, while a wheel-speed block models the brake system’s hydraulic mechanism to estimate wheel angular velocity and acceleration. The estimated slip is compared against a controller reference to regulate longitudinal slip and reduce stopping distance. Together, these instruments allow students to examine how controller settings, vehicle parameters, and driving conditions affect stability and braking performance.

2.4. Learning Activities

This section describes the learning activities developed for simulation and experimentation within a vehicle dynamics course. The simulation component is developed in Simulink, a block-diagram environment that allows students to model multi-domain systems and intuitively evaluate their performance. The longitudinal dynamics model computes the friction coefficient based on estimated slip, and includes calculations for vehicle deceleration, speed, stopping distance, and tire torque. It also simulates the brake system’s hydraulic mechanism to estimate wheel angular velocity and acceleration, comparing estimated slip against a control reference to regulate the system and reduce stopping distance. This platform enables students to examine how variations in controller parameters, vehicle characteristics, and driving conditions affect stability, braking time, and stopping distance.

The experimental activity, in turn, is organized into three phases: pre-laboratory preparation, data collection, and data processing. During preparation, students review theoretical foundations, extract technical specifications from a 2015 Kia Optima, and install the required software and drivers (Matlab/Simulink, Vehicle Network Toolbox, and Kvaser Leaf Light V2). During data collection, they perform controlled acceleration and braking maneuvers, including full-throttle acceleration and emergency braking tests, under instructor supervision, recording variables such as engine RPM and vehicle speed. Finally, in the ABS-focused simulation activity, students analyze how this system prevents wheel lock-up, thereby reinforcing their practical understanding of longitudinal vehicle dynamics.

Full-throttle acceleration and emergency braking tests were conducted in an open space on university grounds. The activity was authorized by the university’s security team, who also provided on-site personnel and warning signage, including traffic cones, to secure the test area from unauthorized access. The vehicle was operated by a licensed student member of the team, with a co-driver present to monitor vehicle performance during testing. Safety procedures followed an internal protocol developed by the research team, covering vehicle inspection, personal protective equipment, emergency stop procedures, and post-test checks, provided in full in Appendix A.

Results and Reporting

Students document their findings in a structured report, comparing experimental and computational results and interpreting the implications for vehicle performance, traction, and braking efficiency. The final discussion addresses real-world applications of the experiment, including its relevance to ABS, traction control, stability, etc.

2.5. Implementation of the Learning Activities

The automotive industry has a large impact in the northeast region of Mexico, where UDEM is located. The activities described in this work arose from the interaction between the automotive research group of Universidad de Monterrey and vehicle dynamics specialists working in the automotive sector. The aim of the activities was to strengthen student competencies by including the direct use of tools and protocols commonly used in the field, such as CAN bus communication, electronics, and Matlab/Simulink 2024b software, which are topics beyond the regular syllabus of the course.

In alignment with the research questions stated in the Introduction, the learning sequence was designed to examine student performance after simulation-based and experimental activities, as well as after the integration of both experiences. The activities focused on longitudinal vehicle dynamics, including the interpretation of wheel slip, braking force, control strategies, and graphical data related to acceleration and braking performance. This implementation was conceived as a classroom-based crossover sequence, where both groups ultimately completed both activities, allowing comparison of initial instructional modality while preserving equivalent exposure to the complete learning experience.

Learning Flow Proposition

This study follows a structured learning sequence designed to examine cohort-level performance patterns associated with simulation-based and experimental activities, with a specific focus on longitudinal dynamics, braking performance, and slip definition.

The pedagogical framework consists of four main phases related directly with the learning outcomes stated previously and ordered according to the performance indicators (ABET, 2017), established by ABET: knowledge, comprehension, application and analysis, supporting a progressive transition from theoretical learning to applied problem-solving.

Figure 2 shows the expected progression of the activities on the course. After four weeks of conventional classes in which theory was explored through lectures, assignments, quizzes, and exercises, students were expected to have developed sufficient background in longitudinal vehicle dynamics. This assumption was evaluated by the professor through a non-graded diagnostic test. Before participating, students were informed about the general nature and purpose of the activity, its non-graded character, and the voluntary nature of their participation. This information was provided to obtain their consent, reduce stress and other potential negative effects on performance, and encourage honest responses, as discussed in Hanssens et al. (2023); Buccafusca (2023). However, the specific assessment procedure, topic, prompts, and timing were not disclosed in advance to limit targeted preparation.

Figure 2. Learning process flow diagram.

In week 5, the six teams were assigned by the instructor to two activity sequences to permit a descriptive comparison of early performance following different initial modalities. All students completed both learning experiences by the end of the week.

The experimentation-first sequence began with the experimental activity during the first session under the supervision of the course instructor. Students conducted real-world braking tests, collected vehicle data through OBD-II and CAN bus interfaces, and compared the measured values with theoretical predictions. During the second session, they completed the simulation-based activity using the Matlab/Simulink model described in Section 2.3 and Section 2.4. The simulation allowed students to visualize how ABS prevents wheel lock-up and affects braking performance.

The simulation-first sequence completed the same activities in reversed order. Students first interacted with the computational model and subsequently performed the experimental braking tests. Because both sequences included the same learning activities and differed in their initial modality, the design permitted comparison of early cohort-level performance and score variability after students first interacted with either a computational model or a physical system.

To examine task-specific performance after the first activity, the same non-graded diagnostic test was administered to both sequences. Research in STEM education suggests that low-stakes assessments can enhance long-term retention, reduce anxiety, and encourage students to explore complex topics without fear of failure (Buccafusca, 2023; Hanssens et al., 2023). The results were analyzed descriptively to compare the simulation-first and experimentation-first sequences. Given the team-level assignment, the observed patterns provide preliminary evidence consistent with a possible initial-modality effect, although no formal causal effect was estimated.

After completing both activities, students take a second non-evaluative test, which assesses their broader understanding of braking performance and longitudinal slip. This test aims to evaluate how well they integrate insights from both activities and whether engaging in both simulation and experimentation is reflected in a broader grasp of vehicle behavior.

In the final phase, students consolidate their findings into a formal report, which serves as the summative assessment for the activity. By structuring the learning process in this way the study compares student performance after different initial activity sequences.

This crossover structure provides a comparative classroom-based design. By alternating the initial activity across groups and subsequently exposing all students to both experiences, the study compares early differences between instructional sequences and examines changes in performance after the complete learning sequence.

2.6. AI Model Calibration Process

Prior to full scale scoring, an AI calibration process was conducted to align the generative AI models with the scoring criteria and reduce potential sources of bias or misinterpretation. Three student responses were randomly selected and independently scored by the three authors according to the rubric criteria. The resulting scores were then compared and discussed as a group to resolve discrepancies and clarify the intended interpretation of each criterion. The AI models were then prompted with the same open-ended question to observe their baseline responses, followed by additional contextual information relevant to the topic, such as how ABS affects longitudinal slip, wheel performance, and lock up prevention, to ensure sufficient context within the session. The scoring rubric was then provided to the AI models, which were asked to score the same three student responses previously scored by the research team. The resulting AI scores were compared against the team’s scores, and where discrepancies arose, the research team provided explanations to the AI models to clarify the intended reasoning behind the human scoring. Following this calibration process, the AI models were used to score the complete set of student responses, with the three calibration responses embedded in their original random position among the rest of the dataset rather than scored separately.

As a final verification step, the research team confirmed that the scores assigned to these three responses during full scale scoring remained consistent with the scores obtained during the calibration process, supporting the stability of the AI’s scoring behavior once calibrated.

3. Results

To support a consistent evaluation of the performance of the students in the non-graded tests, this study employs Artificial Intelligence (AI)-based assessment techniques (Chen et al., 2020). The AI-assisted evaluation was used to apply predefined rubric criteria to student responses and support the identification of response patterns. This approach supports repeatable rubric application and provides quantitative indicators of cohort-level performance under consistent scoring conditions of the assessment after simulation-based and experimental activities. The results of the evaluation could be used as a guide in improving the activities to support future refinement of the teaching-learning process. GPT-4 Turbo was selected as the primary model for generating the results reported in this Section, as it was the model most actively used by the research team at the time the study was conceived, providing practical familiarity and consistency in prompt design and output interpretation throughout the data collection period.

A key aspect of this methodology is that the non-evaluative tests are conducted anonymously, ensuring that students can engage with the assessment without the pressure of grades or concerns about individual performance tracking (McDonald et al., 2025). The AI system does not receive any identifying information about the students, supporting full anonymity and data privacy. This was intended to reduce grade-related pressure and support more authentic responses.

By leveraging AI-driven evaluation, this study aims to identify learning trends, compare student performance after different instructional sequences, and provide data-driven insights into how simulation and experimentation are reflected in student responses (Zawacki-Richter et al., 2019). The following sections present the results obtained through this AI-based assessment, highlighting key findings related to students conceptual performance, differences in performance between the two instructional sequences, and their ability to analyze graphical data. Additionally, Section 3.4 examines the intra-model repeatability and inter-model agreement of the AI-assisted evaluation procedure.

3.1. Experimental Activity Versus Simulation Activity

This stage provides the first diagnostic comparison, where student understanding is assessed immediately after completing either the simulation or experimental activity. By comparing the simulation-first group and the experimental-first group, we obtain a cross-examined perspective on how initial modality influences conceptual gains.

After each group finalized its first activity, a test was conducted. The main object of analysis is the open-ended question: What is the definition of longitudinal slip? Evaluating an open-ended question helps to examine how students express their understanding, avoiding false positive results that could appear when a student has a correct guess in closed ended questions.

To evaluate the conceptual understanding of the students about longitudinal slip in the context of vehicle dynamics, we established a rubric composed of three criteria, according to the expected learning level reached by the students with the activity, which is comprehension level by summarizing in their own words the concept of slip. Each criterion was scored on a scale from 0 to 3, where 0 indicates a very poor response and 3 represents an excellent response. The rubric is designed to assess both technical accuracy and clarity of explanation.

The evaluation criteria are defined as follows:

• Criterion 1: Accuracy in the Definition of Longitudinal Slip

This criterion evaluates whether the student correctly identifies longitudinal slip as a relative difference between the circumferential speed of the wheels and the vehicle actual forward velocity. A complete and accurate response should describe the concept as a ratio or percentage and reference its role in vehicle motion under acceleration or braking conditions.

• 3—Precise, technically correct definition including relevant relationships and context.
• 2—Mostly accurate, but lacking specific details (e.g., omits mention of it being a ratio).
• 1—Vague or partially incorrect explanation; core concept not clearly conveyed.
• 0—Incorrect or unrelated explanation.

• Criterion 2: Use of Technical Language

This criterion assesses the use of discipline-specific terminology related to vehicle dynamics and tire-road interaction. Appropriate terms include: longitudinal slip, slip ratio, wheel speed, tractive force, vehicle speed, circumferential speed, etc.

• 3—Consistently uses accurate technical vocabulary appropriately.
• 2—Uses some relevant technical terms, though inconsistently or imprecisely.
• 1—Rare or superficial use of technical vocabulary.
• 0—No use of appropriate technical language.

• Criterion 3: Clarity (Structure and Grammar)

This criterion evaluates the clarity and coherence of the student written explanation. A high-quality response should be well-structured, logically organized, and free of grammatical or syntactic errors that hinder understanding.

• 3—Well-written, clear, logically organized, and free from grammatical or structural errors.
• 2—Generally clear and understandable, with only minor grammatical or structural issues.
• 1—Difficult to follow due to poor grammar, sentence structure, or organization.
• 0—Incomprehensible or lacks logical structure entirely.

This grading framework supports consistency in the evaluation process while emphasizing students conceptual understanding and technical communication skills.

Figure 3 shows the distribution of total scores obtained by students after completing their first learning activity, either simulation or experimentation. The boxplot shows that while the median scores are similar between the two groups, the simulation group displays a narrower interquartile range, indicating more consistent performance among students. In contrast, the experimentation group shows a wider interquartile range and a broader overall spread, reflecting greater variability in outcomes, including both stronger and weaker performances. This descriptive difference is consistent with a possible initial-modality effect, although it does not constitute a formal causal estimate.

Figure 3. Distribution of total scores obtained by students based on the initial activity performed: simulation and experimentation.

These results suggest that both instructional approaches can support high achievement, but simulation may offer a more structured and consistent foundation early in the learning process, while the greater variability observed in the experimentation group highlights the potential value of additional guidance or scaffolding during hands-on activities to support more consistent conceptual development.These patterns are interpreted here as exploratory evidence rather than as a demonstrated causal effect of instructional order.

Figure 4 presents the distribution of scores for each of the three evaluation criteria, Accuracy in Definition (Criterion 1), Use of Technical Language (Criterion 2), and Clarity (Criterion 3), grouped by the initial activity performed by the students. The boxplots show that both groups achieved their highest scores in Criterion 3, with medians close to the maximum and narrow spread, suggesting that students were generally more confident and competent in organizing their responses and expressing ideas clearly. In contrast, Criterion 2 shows lower medians and wider distributions in both groups, particularly for the experimentation-first group, where scores are more widely dispersed and include more low-end responses, indicating greater difficulty in using precise technical vocabulary when explaining the concept of longitudinal slip. More broadly, the simulation-first group shows narrower interquartile ranges and fewer extreme values across most criteria, reflecting more consistent performance, while the experimentation-first group displays greater variability, especially in Criterion 2.

Figure 4. Distribution of criteria performance based on the initial activity.

This pattern suggests that while experimentation can support strong conceptual engagement, it may require more scaffolding or follow-up to strengthen formal expression and terminology. Together, these figures highlight the complementary nature of the two instructional approaches. Simulation appears to support more stable conceptual and linguistic development early in the learning process, while experimentation provides opportunities for practical engagement but may produce more diverse outcomes.

3.2. Students Performance After the Two Activities

To evaluate the ability to apply theoretical knowledge to a real-world phenomenon, an open-ended question was presented asking them to explain the influence of the Anti-lock Braking System (ABS) on longitudinal slip during a braking or acceleration maneuver. The responses were assessed using a rubric composed of three criteria, each scored on a 0 to 3 scale, where 0 indicates an incorrect or absent response and 3 corresponds to a complete and technically sound explanation. This rubric is aligned with the application level of learning, as defined by ABET, where students are expected to use engineering principles to interpret or explain physical events.

The criteria follow the same structure introduced previously, recalibrated to match the cognitive level expected for this task. Criterion 1 evaluates the accuracy of the definition, function, and impact of ABS. Criterion 2 evaluates the use of discipline-specific technical language. Criterion 3 evaluates the clarity and coherence of the student’s written explanation.

• Criterion 1: Definition, Function, and Impact of ABS

This criterion evaluates whether the student correctly explains what the Anti-lock Braking System (ABS) is, how it works, and how it affects longitudinal slip during braking or acceleration. A complete and accurate response should mention concepts such as wheel lock-up, brake pressure modulation, the optimal slip ratio range (approximately $0.1$–$0.3$, and its role in vehicle stability.

• 3—Precise and complete explanation of ABS functionality and its impact on slip, including key terms and correct context.
• 2—Mostly accurate explanation but lacking some key details (e.g., mentions ABS prevents locking but omits reference to slip control or optimal range).
• 1—Partial or vague explanation of ABS, with limited connection to slip or technical context.
• 0—Incorrect or unrelated explanation, or does not address ABS function or slip.

• Criterion 2: Use of Technical Language

This criterion assesses the use of discipline-specific terminology related to automotive engineering and vehicle dynamics. Appropriate terms include: longitudinal slip, adhesion coefficient, braking force modulation, speed sensors, and braking distance.

• 3—Consistently uses accurate and relevant technical terminology throughout the explanation.
• 2—Uses some relevant technical terms, though inconsistently or with minor inaccuracies.
• 1—Rare use of technical language or inappropriate application of terms.
• 0—No use of relevant or discipline-specific terminology.

• Criterion 3: Clarity (Structure and Grammar)

This criterion evaluates the clarity and coherence of the student written explanation. A high-quality response should be well-structured, logically organized, and free of grammatical or syntactic errors that hinder understanding.

• 3—Well-written, clear, logically organized, and free from grammatical or structural errors.
• 2—Generally clear and understandable, with only minor grammatical or structural issues.
• 1—Difficult to follow due to poor grammar, sentence structure, or organization.
• 0—Incomprehensible or lacks logical structure entirely.

In order to clarify the use of the rubric, three representative responses are shown below, together with their assigned scores.

The first response, “Reduces the longitudinal slip by modulating the pressure of braking in each wheel, avoiding that they get blocked and maintaining the traction with the surface of the road,” shows a partial understanding of ABS function (Criterion 1 = 2): although ABS does prevent wheel lock-up, it regulates longitudinal slip toward an optimal value rather than simply reducing it. The student correctly uses discipline-specific terminology, including longitudinal slip, braking pressure, and wheel lock-up (Criterion 2 = 3), and the response is clear and well structured (Criterion 3 = 3).

The second response, “What it does is it helps so that the tire, when wanting to make the car stop, doesn’t get the tire blocked on it, since by having the ABS system it makes it so that the speed of the tire can be in a way that it’s being behaving like the speed of the car with pulsations that make it so that the tire is having the brake applied to it in pulses and not in a way that it gets applied all at once making it so that the tires get blocked,” shows a poor understanding of ABS function and effect (Criterion 1 = 1). The student uses limited technical language, and the terms included are applied imprecisely (Criterion 2 = 1). The response is lengthy and difficult to follow, reflecting weak clarity and structure (Criterion 3 = 1).

The third response, “It tries to maintain the slip between 0.1 and 0.2 regulating the pressure of the pistons of brake,” provides a precise, cause and effect explanation of ABS behavior (Criterion 1 = 3). Although few technical terms are used, those included are applied correctly (Criterion 2 = 2). The response is brief but clear and sufficiently complete (Criterion 3 = 3).

It is worth noting that the first and third responses obtain the same total score, eight out of nine points, despite reflecting distinct performance profiles. The first response demonstrates strong command of technical vocabulary and clear written expression, but reveals a conceptual gap in describing how ABS regulates, rather than simply reduces, longitudinal slip. The third response shows the opposite pattern, a precise and technically accurate causal explanation, despite using comparatively fewer technical terms. This illustrates a key advantage of the multi-criterion rubric structure over a single holistic score: it preserves diagnostic information about where a given student’s understanding is strong or weak, rather than collapsing distinct reasoning patterns into an equivalent, but uninformative, total.

Figure 5 illustrates the distribution of student scores across the three evaluation criteria after completing both learning activities, simulation and experimentation. As shown in Table 1, the highest mean score was obtained in Criterion 3, followed by Criterion 1 and then Criterion 2. Although Criterion 3 shows the strongest performance, the improvement relative to the results reported in the previous subsection is the smallest, suggesting that the activities do not directly contribute to the development of communication skills, which students may have already developed throughout their engineering studies. In contrast, the notable increase in Criterion 1 scores suggests that the combined simulation and experimentation activities were associated with an increase at cohort level in the ability to apply conceptual understanding of longitudinal dynamics and its relationship with the application of ABS systems.

Figure 5. Distribution of criteria performance obtained by students after the two activities.

Table 1. Descriptive statistics of student performance after the first activity (simulation-first and experimental-first groups) and after both activities.

Statistic/Criterion	Sim. First	Exp. First	2nd Activity
Overall Mean	5.67	5.30	6.84
Median	6.00	5.50	7.00
Standard Deviation	2.18	2.98	1.77
Minimum	2	1	4
Maximum	9	9	9
Criterion 1—Mean	1.67	1.60	2.32
Criterion 1—Median	2	2	2.00
Criterion 1—Std. Dev.	1.00	1.26	0.75
Criterion 1—Min/Max	0/3	0/3	1/3
Criterion 2—Mean	1.56	1.40	2.00
Criterion 2—Median	2	1	2.00
Criterion 2—Std. Dev.	0.88	1.26	0.82
Criterion 2—Min/Max	0/3	0/3	1/3
Criterion 3—Mean	2.44	2.30	2.53
Criterion 3—Median	2	2	3.00
Criterion 3—Std. Dev.	0.53	0.67	0.61
Criterion 3—Min/Max	2/3	1/3	1/3

The boxplots provide further insight into the distribution of student responses. Criterion 3 shows the narrowest spread, with most students scoring consistently, reinforcing the idea that clarity of expression was not a major barrier for this cohort. Criterion 2, in contrast, exhibits wider spread and lower values, with some students still struggling to consistently apply technical terminology. Criterion 1 also shows a relatively balanced distribution, suggesting that the instructional sequence reinforced students’ understanding of system dynamics, even for those who initially had difficulties.

In summary, the results suggest that while Criterion 3 reflects a skill set already well developed by students, Criteria 1 and 2 were more directly influenced by the learning activities. The structured exposure to both simulation and experimentation appears to have strengthened students’ ability to explain the role of the ABS system in slip dynamics and partially improved their use of technical language. These results emphasize the value of combining practical and computational experiences in vehicle dynamics education, particularly when aiming to deepen applied understanding among students. The previous statements are supported by the statistical analysis presented in Table 1.

The study incorporates a deliberate comparative sequence. First, the crossover structure provides evidence of how the initial modality is reflected in early performance and variability. Second, the combined results after both activities provide evidence of cohort task-level performance changes, as reflected by the shift toward higher performance categories.

As shown in Table 1, the cohort obtained higher scores on the application-level assessment administered after both activities than on the comprehension-level assessment administered after the first activity. The first two columns (“Sim. First” and “Exp. First”) correspond to the results obtained after the initial activity performed by each group, while the last column represents the combined performance after all students completed both activities. This structure provides a cross-examined view of learning progression, showing that scores improved after exposure to both the simulation and the experimental benchmarks. However, anonymous responses could not be linked across occasions; therefore, the comparison represents a cohort-level shift across task demands rather than individual longitudinal progression.

To assess whether the observed differences between the simulation-first and experimentation-first groups were statistically significant, Mann-Whitney U tests were conducted for each criterion and the total score, given the small sample size and ordinal nature of the rubric scores (Mann & Whitney, 1947). No statistically significant differences were found between groups for any measure (Total: p = 0.85; Criterion 1: p = 1.00; Criterion 2: p = 0.64; Criterion 3: p = 0.89). Rank-biserial correlation effect sizes were negligible across all comparisons (Total: r = 0.056; Criterion 1: r = 0.011; Criterion 2: r = 0.100; Criterion 3: r = 0.100), indicating that the descriptive differences discussed above reflect a very small, likely non-meaningful, effect rather than a substantive group difference, consistent with the limited statistical power associated with this sample size (Kerby, 2014).

A separate Mann-Whitney U test was conducted comparing combined first-activity scores, simulation-first and experimentation-first groups together, against second-activity, application-level scores across the full cohort, to assess whether the increase in performance following the second activity was statistically supported. This comparison also did not reach statistical significance (p = 0.107), though a moderate effect size was observed (rank-biserial r = 0.305), larger than any of the effect sizes reported above for the first-activity group comparison. This suggests a noteworthy, though not statistically confirmed, increase in performance following the second activity, which, given the small sample size, is interpreted as a descriptive, cohort-level pattern rather than a statistically confirmed improvement.

Figure 6 illustrates the categorical classification of student performance after the first and second instructional activities. The cutoffs were defined through the expert judgment of the course instructor as part of the calibration of the rubric-based evaluation. This calibration considered the three assessment criteria, each scored from 0 to 3, resulting in a total score ranging from 0 to 9. Performance was classified as Poor (below 5), Regular (5–6), Good (7–8), or Excellent (9). After the first activity, 7 students (36.8%) were classified as Poor, 5 (26.3%) as Regular, 2 (10.5%) as Good, and 5 (26.3%) as Excellent. After the second activity, 2 students (10.5%) were classified as Poor, 7 (36.8%) as Regular, 2 (10.5%) as Good, and 8 (42.1%) as Excellent. Percentages displayed in Figure 6 are rounded to the nearest whole number.

Figure 6. Categorical classification of student performance, with the student count and corresponding percentage reported for each category. (Left): Distribution after the first activity, combined across groups. (Right): Distribution after both activities. The second chart shows a cohort-level shift toward higher performance categories across the instructional sequence.

The first question, focused on defining longitudinal slip, was aligned with the ABET “comprehension” level, targeting basic conceptual understanding. In contrast, the second question, which asked students to explain the influence of ABS on slip during braking or acceleration, was designed at a higher cognitive level, “application”, requiring students to use engineering principles to interpret a real-world phenomenon. Despite the increase in complexity, cohort performance improved substantially after the second activity: the proportion of students in the “Poor” category dropped from 37% to 11%, while those in the “Excellent” category rose from 26% to 42%. This pattern reflects a cohort-level shift toward higher performance and is consistent with the possibility that experiential learning strategies such as simulation and hands-on experimentation may support deeper understanding and the ability to apply theoretical knowledge in engineering contexts, though this interpretation remains descriptive and should be confirmed through study designs with greater statistical power.

3.3. The Limits of the Learning Activity: Analysis Learning Level

To evaluate the ability of the students to perform analysis level reasoning, as defined by ABET, an open-ended question was designed based on a time-series plot of longitudinal slip during a vehicle acceleration maneuver, shown in Figure 7, the question was released after the second activity. The question prompts students to interpret a segment of the plot where slip remains nearly constant and to infer what this implies about traction conditions and the powertrain performance. This assessment targets the cognitive skill of identifying cause and effect relationships, requiring students to integrate knowledge of tire-road interaction, torque delivery, and vehicle dynamics. Unlike comprehension or application level tasks, this question challenges students to analyze dynamic behavior by relating observed system responses to underlying physical mechanisms. As such, it serves as an indicator of how students interpret vehicle behavior and explain it in terms of interconnected mechanical and control systems.

Figure 7. Slip time evolution taken from the students experiments.

The analysis level rubric follows the same three-criterion structure used in the previous rubrics, Interpretation, Technical Language, and Clarity, with Criterion 1 recalibrated to assess the interpretation of graphical data, specifically, whether students correctly identify the behavior of longitudinal slip in a given time window and draw appropriate technical conclusions, rather than define or apply a single concept as in the earlier tasks.

Figure 8 presents cohort performance across the three assessment criteria for the analysis-level task. Compared to their responses after the first activity, where the focus was on basic definition, the results in this figure show higher scores in average scores and score distribution, suggesting progress at cohort-level in the ability to engage with more complex and interpretive tasks. Notably, Criterion 3 remained comparatively strong, while Criteria 1 and 2 reflected the additional demands of graphical interpretation and cause-and-effect analysis.

Figure 8. Criteria performance obtained by students on the analysis learning level. (Left): Average score per criterion. (Right): Boxplot of criterion performance.

However, when compared to the results obtained at the application level, the overall performance in the analysis task appears slightly lower, especially in terms of the average scores for criterion 1 and criterion 2. This outcome is consistent with the increasing cognitive demand associated with the ABET learning hierarchy, as the analysis level requires not only to apply knowledge but also to interpret data, infer relationships, and justify conclusions. While students demonstrated foundational understanding and communication skills, the complexity of identifying system level dynamics and causal mechanisms presented additional challenges. These results suggest that although students are progressing in their analytical capabilities, further instructional support may be beneficial to help them confidently transition from application to analysis level reasoning.

Across the three assessment stages, definition, application, and analysis, student performance followed a pattern that offers meaningful insight into the effect of the combined instructional approach. At the comprehension level, prior to full exposure to both simulation and experimentation, scores were comparatively lower and more variable, with the simulation-first and experimentation-first groups showing standard deviations of $2.18$ and $2.98$, respectively, reflecting a wide range of initial understanding across students. After students completed both instructional activities and were asked to apply their understanding to a more demanding task, explaining the role of ABS in longitudinal slip, both the mean and consistency of performance improved: overall scores rose to a mean of 6.84 with a substantially reduced standard deviation of $1.77$. Despite the added cognitive demand of an application level task, students not only performed better but did so more uniformly, a pattern that would not be expected if increasing task difficulty alone were driving outcomes, and one that is consistent with the combined instructional approach supporting a genuine convergence in student understanding.

This consistency, however, was not simply a product of an easier or more familiar type of question. When students were subsequently asked to analyze a graphical representation of longitudinal slip and interpret its implications for traction and powertrain performance, a task requiring a still higher level of cognitive engagement, mean performance declined as expected, from $6.84$ to $5.95$, reflecting the increased difficulty of reasoning about cause and effect relationships from data. Critically, the variability in student responses remained essentially unchanged, with an overall standard deviation of $1.81$, closely matching the $1.77$ observed at the application level, and comparable standard deviations across all three individual criteria. This indicates that the assessment approach remained sensitive to genuine differences in task difficulty, evidenced by the drop in average performance, while the consistency gained through the combined instructional activities persisted even as students faced a more demanding analytical challenge. Together, these findings suggest that the integration of simulation-based and experiential learning may support not only improved performance but also more stable and reliable understanding as students progress toward higher order cognitive tasks.

Figure 9 presents the categorical classification of the students responses at the analysis learning level. Compared to the distribution obtained after the application-level task, there is a noticeable shift toward lower performance categories: the proportion of students in the “Excellent” category decreased from 42% to 21%, while those in “Poor” and “Regular” increased to a combined 63%. This outcome reflects the increased cognitive challenge posed by analysis level tasks, which require not only applying knowledge but also interpreting complex behavior and identifying cause and effect relationships. Despite this decline, the distribution remains notably better than after the first activity, suggesting continued progress even as task complexity increases.

Figure 9. Categorical classification of the analysis results.

3.4. Evaluation of AI-Assisted Grading Consistency

To examine the consistency of AI-assisted evaluations, the same anonymous student responses were evaluated using four Generative AI models: GPT-4 Turbo, GPT o4-mini, DeepSeek (https://www.deepseek.com/), and GPT-4o-mini. Each model performed ten repeated stochastic evaluation runs using the same instructor-defined rubric, scoring criteria, and prompt structure. These repeated runs were used to examine intra-model repeatability (which is important because repeated run can have different results, even with the same prompts and inputs), while comparisons across models were used to examine inter-model agreement.

All AI-assisted evaluations followed the same instructor-defined rubric, scoring criteria, and prompt structure. Student responses were anonymized before analysis, and no identifying information was provided to the models. The AI outputs were used for diagnostic and methodological analysis, not for assigning course grades. The instructor retained responsibility for defining the learning outcomes, establishing the rubric, and interpreting the aggregate results. This procedure was intended to evaluate the consistency of rubric application under controlled prompt and scoring conditions.

For the inferential comparison across GenAI models, the student response was considered the unit of analysis ($n=19$). For each response and model, scores obtained across the ten stochastic runs were averaged before conducting inferential tests; therefore, repeated runs were not treated as independent observations and did not increase the effective sample size. Because the same student responses were evaluated by all four models, differences among models were assessed using a repeated-measures ANOVA. Mauchly’s test indicated a violation of the sphericity assumption ($p=0.038$); therefore, a Greenhouse–Geisser correction was applied. The analysis showed a significant effect of GenAI model on the scores, $F(2.34,42.07)=21.72$, $p<0.001$, ${\eta }_p^2=0.547$. Tukey–Kramer-adjusted pairwise comparisons were subsequently performed.

Post hoc comparisons indicated no significant difference between GPT-4 Turbo and GPT o4-mini ($p=0.8397$), or between DeepSeek and GPT-4o-mini ($p=0.0897$). In contrast, significant differences were observed for all other pairwise comparisons ($p\le 0.0136$), as shown in Table 2. This suggests that while certain AI models demonstrate alignment in grading behavior, others introduce notable discrepancies in assessment outcomes. The mean variance across all AI-generated evaluations was 1.37 which is comparable to that observed in human-based assessments for open-ended engineering tasks. For example, previous studies report mean variances of 8.64 (Hicks & Diefes-Dux, 2017), 3.17 (Diefes-Dux et al., 2010), and 1.9 (Hicks, 2020) when considering similar settings (i.e., open-ended engineering rubric-based grading). These human-based variances are attributed to multiple factors, including grader fatigue, rubric interpretation inconsistencies, and varying levels of grader expertise (Hicks, 2020). These findings suggest that, under a predefined rubric and consistent prompt structure, AI-assisted evaluation can provide a level of scoring consistency that is comparable to reported human-based assessment variability.

Table 2. Tukey–Kramer-adjusted pairwise comparisons of mean scores across GenAI models.

p-Value	GPT O4-Mini	Deepseek	GPT 4o-Mini
GPT-4 Turbo	0.8397	0.0001	0.0001
GPT O4-mini		0.0001	0.0136
Deepseek			0.0897

Krippendorff’s alpha ($\alpha$) was calculated to quantify intra-model repeatability and inter-model agreement beyond that expected by chance. Values range from 0 to 1, with higher values representing greater agreement. The reference criterion of 0.7 used in this study was adopted from conventional human inter-rater applications in educational assessment (Krippendorff, 2004). Because no validated cutoff has been established specifically for inter-model agreement among Generative AI systems, this value is used as a contextual reference criterion.

When all four models were considered together, the overall Krippendorff’s alpha was 0.666. This value did not meet the reference criterion of 0.7 adopted in the study. In Table 3, individual models showed higher intra-model repeatability, with alpha values ranging from 0.7662 to 0.8456, while pairwise inter-model agreement ranged from 0.6254 to 0.7750. Four of the six pairwise inter-model coefficients were at or above 0.7. Thus, disagreement was not uniform across all model pairs, although the observed range indicates meaningful model-dependent variation. The models should therefore not be treated as interchangeable.

Table 3. Krippendorff’s Alpha for Intra- and Inter-Model Consistency.

Model	G4T	Go4-Mini	Deep	G4o-Mini
G4T	0.8456	0.7536	0.6489	0.7681
Go4-mini		0.7662	0.6254	0.7046
Deep			0.7935	0.7750
G4o-mini				0.8367

In comparison, human graders performing similar open-ended assessments in engineering contexts have shown considerably lower reliability, with Krippendorff’s alpha reported at 0.422 (Hicks, 2020) in a similar setup. These results indicate that some advanced AI models can achieve consistency levels comparable to those observed in well-coordinated human grading when their outputs are generated using well-defined rubrics and consistent prompting conditions.

It is worth noting that Hicks (Hicks, 2020; Hicks & Diefes-Dux, 2017) emphasizes that rubric-based human grading often suffers from inter-rater inconsistencies, even with standardized rubrics, due to varying grader interpretations and cognitive load during assessment. Similarly, Diefes-Dux et al. (Diefes-Dux et al., 2010) highlight that reliable assessment of open-ended problems requires not only well-designed rubrics but also rigorous scorer training and iterative refinement. These studies provide relevant context regarding variability in human rubric-based evaluation of open-ended responses, but they do not constitute a direct benchmark for inter-model agreement or alter the finding that the overall coefficient of 0.666 did not meet the reference criterion adopted in this study.

On the other hand, the lower inter-model alpha values, for example, 0.6254 between GPT O4-mini and Deepseek, indicate that different models may interpret and score the same student responses differently. This underscores the importance of selecting AI models carefully to ensure alignment with specific assessment goals. In practice, institutions considering AI-assisted grading should validate model behavior in context, particularly if consistent evaluation across varying tasks or student cohorts is essential. Future research may explore hybrid approaches that combine multiple models or dynamically weight their outputs to enhance both reliability and sensitivity to diverse student responses.

AI-assisted rubric scoring is characterized through intra-model repeatability and inter-model agreement. These measures describe the stability of repeated scoring under the specified model, rubric, and prompting conditions. Assessing scoring accuracy would require comparison with an independent reference assessment, typically established through evaluations by multiple qualified human raters, while validity would require broader evidence supporting the intended interpretation and use of the scores. Previously reported human inter-rater results provide relevant context because rubric-based evaluation of open-ended responses also involves interpretive variability. These cross-study comparisons are descriptive and do not establish equivalence between repeated model outputs and independent human graders. The present analysis therefore contributes evidence concerning the repeatability and model dependence of Generative AI-assisted scoring in educational assessment.

The combination of Table 1 and Figure 9 provides two complementary forms of evidence regarding the proposed methodology. First, the descriptive statistics summarize how student scores were distributed across the targeted ABET/Bloom cognitive levels. Second, the comparison between performance after a single activity and performance after both activities provides a cross-examined view of the instructional sequence. This comparison shows a consistent cohort-level shift toward higher scores and stronger categorical outcomes, suggesting that the integration of simulation and experimental activities is related with improved cohort outcomes.

4. Discussion and Further Work

Generative AI was introduced to the general public in 2022 and has since brought new challenges to academia, particularly in STEM education. From the authors’ perspective, directly using Generative AI to grade student work raises significant ethical concerns. Grading involves not only evaluating performance but also guiding learning, a process that requires empathy and pedagogical judgment, which are difficult for AI to replicate. Concerns include transparency in decision-making, potential biases in AI outputs, and accountability for grading outcomes. In this study, GenAI was therefore used as an assistance tool for rubric-based diagnostic analysis, not as an autonomous grading system. Student responses were analyzed anonymously, no identifying information was provided to the models, and AI-generated scores were not used to assign course grades.

From a STEM education perspective, the methodology proposed in this work represents a useful contribution, particularly in settings with small student cohorts where conventional statistical evaluation is often limited. By aligning learning activities and assessments with predefined cognitive levels, the approach provides diagnostic evidence of cohort level improvements across comprehension, application, and early analysis tasks. The results suggest a positive shift in student performance after exposure to simulation and experimental activities. Although performance decreased slightly at the analysis level, this behavior is consistent with the higher cognitive demand of tasks requiring interpretation of system behavior and cause–effect relationships.

The learning activities evaluated in this study were implemented within the time constraints of a semester-long course, and specifically within a five-week instructional unit (Weeks 1–5) focused on longitudinal vehicle dynamics. A sixth week was reserved for submission of a final report, graded independently by the instructor and kept anonymous relative to the open-ended assessment data analyzed here, in order to preserve the reliability of the study design. Therefore, the findings should be interpreted as exploratory evidence from a naturalistic classroom setting rather than as broad causal evidence. Within this scope, the approach enabled meaningful monitoring of task-specific cohort performance and helped identify areas where instructional refinement may be beneficial and helped identify areas where instructional refinement may be beneficial. Introducing scaffolding strategies, feedback loops, focused terminology exercises, or requiring students to use specific terms in context during activities could help strengthen outcomes without requiring significant additional time.

The results showed that GPT-4 Turbo and GPT-4o-mini achieved high intra-model repeatability (Krippendorff’s $\alpha >0.8$), indicating stable rubric application across repeated runs under the specified conditions. However, the overall Krippendorff’s $\alpha$ across models was 0.666, below the commonly cited threshold of 0.7, and the differences among models indicate that the evaluations remained model-dependent. These consistency measures do not establish scoring accuracy or validity. The results support the potential use of Generative AI for instructor-supervised application of predefined rubrics, while preserving the central role of the instructor in defining learning outcomes, interpreting results, and guiding instructional decisions.

Several alternative explanations should be considered when interpreting these findings. Because the assessments used different prompts, targeted different cognitive demands, and employed task-specific criteria, differences across assessments may reflect task difficulty and rubric variation in addition to learning. Familiarity with the assessment format may also have influenced later responses. For the initial-modality comparison, random team-level assignment and use of the same prompt and rubric reduced some sources of confounding; however, the small number of teams and the absence of linked individual baseline scores mean that residual baseline differences cannot be excluded. The involvement of the same instructor supported consistent implementation but may have introduced instructor-dependent effects. Finally, the overall inter-model agreement of $\alpha =0.666$ indicates that estimated performance patterns may also depend partly on the selected Generative AI model. The findings should therefore be interpreted as exploratory, cohort-level evidence specific to the tasks, instructor, rubric, and scoring conditions used in this study.

The approach presented here is not specific to vehicle dynamics. It can be transferred to other engineering courses by adapting prompts and rubrics to local outcomes and by using comparable instructional sequences to examine the learning progression, although individual (non-anonymous) followup would be required. From an academic perspective, the proposed methodology has the potential to evolve into a broader diagnostic tool for engineering education. By extending the approach beyond a single unit to entire courses or across multiple semesters, institutions could better monitor student learning trajectories and curriculum effectiveness. Future work could include longitudinal studies of student learning outcomes, comparative studies of AI–human hybrid evaluation systems, and the incorporation of student feedback regarding trust, fairness, and acceptance of AI-assisted assessment practices.

As noted in the introduction, assessment validity for generative AI based scoring remains an open methodological question, and a second concern, model drift, has received similarly limited attention. Model drift, meaning changes in grading behavior across model versions or extended use, has been observed in adjacent work; Zhou et al. (2026) found that continuous interaction history drove systematic drift in language model grading standards away from human expert scores in a software engineering course, raising concerns about fairness in sustained AI assisted grading. Kaldaras et al. (2024) note that no established guidelines currently exist for evaluating the validity of generative AI based assessments, and propose grounding such evaluation in the Standards for Educational and Psychological Testing. This absence of established validity guidelines is consistent with the exploratory, diagnostic framing adopted in the present study, rather than a claim of formal validation.

It is worth noting that the data collection process for this study was designed with ethical considerations in mind. All assessments consisted exclusively of technical questions aligned with course content and engineering applications, involving no personal, sensitive, or identifiable information. Student participation was voluntary and had no bearing on academic grades. To ensure confidentiality, all responses were fully anonymized prior to analysis, making it impossible to link any response to a specific individual. Prior to their participation, all students were verbally informed of the purpose of the study, the voluntary nature of their involvement, and the anonymous handling of their data.

Limitations of Our Work

This study has several limitations. The small, single-institution, single-course, single-instructor sample limits statistical power and generalizability. Anonymous data collection, while supporting honest student engagement, precluded paired analysis of individual performance across assessment stages. Reliability of AI-assisted scoring was model-dependent (Krippendorff’s $\alpha =0.666$), and, consistent with the absence of established validity guidelines for generative AI based assessment noted in the literature (Kaldaras et al., 2024), no independent human-rater benchmark was established to formally assess AI–human agreement. Exact model versions, access dates, and generation parameters were not recorded, since scoring was conducted through each model’s web interface, limiting reproducibility of the exact scoring conditions, a concern related to model drift documented elsewhere (Zhou et al., 2026).

5. Conclusions

The research questions are answered as follows.

RQ1. The assessments revealed task-specific cohort patterns, including a shift toward higher performance categories after both learning activities and greater difficulty with the analysis-level task. Because responses were anonymous these patterns do not represent individual progression.

RQ2. The simulation-first sequence showed slightly higher early performance and lower score variability than the experimentation-first sequence. This descriptive pattern is consistent with a possible initial-modality effect, but no formal causal effect was estimated.

RQ3. Intra-model repeatability ranged from $\alpha =0.7662$ to $0.8456$, while overall inter-model agreement was $\alpha =0.666$, below the reference criterion of 0.7. The results therefore indicate repeatable scoring within models but model-dependent agreement across models.

Appendix A.1. General Safety Precautions

• All participants must wear appropriate personal protective equipment, including seat belts, closed-toe shoes, and preferentially high-visibility vests if operating in an open environment.
• The test area must be clearly marked and free of obstacles, ensuring no unauthorized personnel or vehicles are present.
• A pre-test briefing should be conducted to review the procedure, emergency protocols, and roles of each team member.
• Weather conditions should be assessed before testing. Tests should not be performed in rain, fog, or extreme winds to avoid traction issues.

Appendix A.2. Vehicle Operation Safety

• Only an authorized driver with valid driver licence should operate the vehicle during the experiment.
• The co-driver must provide real-time guidance and monitor the vehicle performance.
• All acceleration and braking maneuvers must be gradual and controlled, unless performing a specific emergency stop test.
• The vehicle must be in good mechanical conditions, with brakes, tires, and steering components inspected before the test.
• Speed limits should be strictly followed, and the testing team should not exceed the predefined velocities.

Appendix A.3. Data Collection and Equipment Handling

• The laptop and data acquisition system must be securely mounted or held by a team member to prevent movement during testing.
• The CAN Bus and OBD-II connection should be checked before starting the vehicle to ensure stable communication.
• The testing team must verify data accuracy between each run to avoid unnecessary retests.

Appendix A.4. Emergency Procedures

• An emergency stop signal should be agreed upon before starting the test. If a hazardous situation arises, the test must be immediately aborted.
• A first aid kit and fire extinguisher should be readily available at the test site.
• In the event of a vehicle malfunction (e.g., brake failure, tire blowout, engine issues), the driver must gradually reduce speed and move the vehicle to a safe location.
• A designated emergency contact person should be available to assist in case of an incident.

Appendix A.5. Post-Test Safety Checks

• The vehicle should be inspected for any mechanical wear or issues after the test.
• The OBD-II and data logging devices should be safely disconnected before shutting down the system.
• Team members should review collected data in a stationary environment, avoiding distractions or analysis inside the moving vehicle.

Appendix A.6. Final Safety Reminder

All participants must be fully aware of their responsibilities and comply with all safety protocols before engaging in the test. These guidelines ensure a controlled environment, minimizing risks while allowing students to gain valuable hands-on experience in vehicle dynamics.