A Socio-Technical Systems Evaluation of Simulation-Supported Instruction in Transportation Engineering: Configuration Outcomes, Within-System Relationships, and Diagnostic Evidence for Redesign

Highlights

Please indicate how your work links to systems science via your contributions to systems practice, theory, and/or methodology.

• Simulation-supported teaching is evaluated as a bounded socio-technical instructional system.
• A three-level evaluation method links system outputs, within-system relationships, and diagnostic evidence for redesign.

What are the main findings and/or the implications of the main findings?

• The simulation-supported configuration showed higher post-instruction knowledge and scenario-based analytical performance than the conventional configuration.
• Interacting-behavior identification emerged as the clearest candidate for targeted redesign.

Abstract

Simulation platforms are often evaluated as isolated teaching tools, although classroom outcomes reflect coordinated technical and instructional elements. This study evaluated a research-team-developed CARLA–SUMO platform as part of a bounded socio-technical instructional system for teaching ramp-merging and weaving analysis. A class-level controlled quasi-experiment involved 174 undergraduates from four intact classes, with 87 students in each instructional configuration. The evaluation combined configuration-level comparisons of formal knowledge, scenario-based analytical performance, and learning-experience profiles; within-system relational analyses in the simulation-supported group; and scenario- and question-level diagnostics. The simulation-supported configuration showed higher post-instruction knowledge ($d=0.425$) and scenario-based analytical performance ($d=0.783$), together with descriptively more positive learning-experience profiles. Perceived learning support was positively associated with cognitive engagement and with learning satisfaction and behavioral intention, while cognitive engagement provided the clearest positive information about scenario performance. Question-level diagnostics identified interacting-behavior identification in Scenario 2 as the clearest candidate for targeted redesign because it combined relatively low absolute performance with a small between-configuration difference. These findings illustrate a three-level approach linking system outputs, within-system relationships, and assessment-informed redesign.

1. Introduction

Transportation engineering education increasingly requires students to reason about traffic systems whose behavior emerges from interactions among infrastructure, traffic demand, vehicles, and human decisions. Ramp-merging and freeway-weaving areas are representative examples: traffic states evolve through coupled changes in gap availability, speed coordination, lane-changing behavior, spatial competition, and responses among multiple vehicles [1,2,3]. Learning these processes therefore involves more than recalling concepts such as gap acceptance or traffic conflict. Students need to identify relevant cues, trace interactions over time, explain how local actions contribute to conflict development, and evaluate possible interventions. These requirements reflect the system-level reasoning expected in engineering education [4,5] and are becoming more important as transportation curricula address increasingly digital, connected, and intelligent systems. Recent discussions of undergraduate transportation education have highlighted the limited curricular space traditionally allocated to the field and the need for stronger preparation in emerging transportation technologies, while work on intelligent transportation systems education has similarly emphasized the integration of contemporary tools, virtual environments, simulation, and practice-oriented learning [6,7,8].

A persistent instructional difficulty is that these dynamic relationships are commonly introduced through lectures, equations, diagrams, and static cases. Such representations remain necessary for establishing fundamental concepts, but they provide limited access to the temporal coupling through which traffic interactions develop. A merging or weaving conflict is not produced by a single vehicle or isolated decision; it unfolds through mutually dependent trajectory, speed, and gap adjustments. Simulation can make these processes observable and controllable, allowing students to inspect evolving traffic states, compare alternative conditions, and revisit interactions from different perspectives. The educational question, however, is not simply whether simulation can represent a traffic process more realistically, but whether that representation is organized into an instructional system that helps students interpret what they observe and translate it into engineering reasoning.

1.1. Simulation-Supported Learning in Transportation and Engineering Education

Transportation education has a substantial, although fragmented, history of using simulation and interactive modeling. Early studies introduced simulation into transportation systems analysis and planning courses to move students beyond formula application toward interpretation, judgment, and problem solving [9,10,11]. An analytic review of transportation engineering instructional practices subsequently found considerable experimentation with simulation, visualization, and active learning, but also identified an evidence base dominated by individual institutions, limited cross-study accumulation, and frequent reliance on post-instruction assessment rather than stronger comparative designs [12]. Thus, the field established the pedagogical potential of simulation relatively early, while rigorous and integrated evaluation developed less consistently.

Recent transportation-specific studies show that this direction remains active. Liu et al. incorporated computer-based virtual simulation into a traffic-system simulation course, while Zhu et al. organized traffic-engineering learning around an integrated signal-design activity combining engineering design, programming, and immersive representation [13,14]. These studies illustrate a shift from software demonstration toward active and practice-oriented instructional arrangements. At the same time, their evidence remains largely based on individual course implementations and reported achievement or learner feedback, leaving a need for more structured evaluations that distinguish different learning outputs and clarify what part of the instructional arrangement is being evaluated.

The broader engineering-education literature reinforces this point. Meta-analytic evidence generally supports simulation and virtual laboratories as useful learning resources. Chernikova et al. reported positive effects of simulation-based learning on knowledge and complex skills [15], and more recent meta-analyses have reported positive overall effects of virtual laboratories and virtual reality on engineering outcomes and practical skills [16,17]. However, these effects are highly heterogeneous across disciplines, tasks, technologies, and instructional approaches. Yang et al., for example, found stronger practical-skill outcomes when virtual-reality activities were integrated with conventional instruction rather than treated as an isolated replacement [17]. A recent systematic review of experimental research in engineering education likewise showed that virtual and immersive technologies are implemented through diverse pedagogical designs, with active and experiential learning playing an important role [18].

The evidence is therefore not well summarized by the simple proposition that greater immersion or more advanced simulation produces better learning. Richer immersive environments have been found to increase presence while reducing learning and increasing cognitive load [19], and greater immersion does not consistently produce superior training performance [20]. Conversely, interactive and immersive environments can support engagement and learning when appropriately structured, with instructional scaffolding also shaping learning outcomes and learner behavior in simulation environments [21,22,23]. Taken together, these findings suggest that simulation can expand what learners observe and manipulate, but technological richness alone does not determine learning outcomes. How the technology is organized instructionally remains central.

1.2. A Socio-Technical Systems Perspective on Simulation-Supported Instruction

This distinction motivates a systems perspective. A system derives its behavior from relationships among its elements rather than from isolated components [24]. Socio-technical systems theory extends this principle to arrangements in which technical resources, people, tasks, organizational practices, and environmental conditions operate together [25]. In education, related work argues against separating pedagogy from technology because the instructional meaning of a technology depends on how it is embedded in goals, activities, roles, and institutional practice [26]. Constructive alignment similarly emphasizes coordination among intended outcomes, learning activities, and assessment [27].

Recent engineering-education research further supports this perspective. A scoping review of sociotechnical thinking in engineering education highlighted the roles of learner characteristics, disciplinary culture, prior knowledge, and the organization of technical and social dimensions [28]. Research on technology-supported learning likewise emphasizes relationships among technological features, pedagogy, and learners’ cognitive processes rather than treating technology as an independent educational input [29]. These perspectives support a basic methodological principle for the present study: outcomes observed after a technology has been embedded in classroom practice should not automatically be attributed to the software alone.

For simulation-supported transportation instruction, the platform, instructional activities, instructor guidance, learner participation, and assessment operate as parts of a coordinated configuration. When these elements change together, the empirical unit of comparison is the instructional configuration as a whole. Estimating the independent contribution of the platform would require its functions to be manipulated or measured separately; without such separation, a software-only interpretation risks attributing to the technical artifact outcomes that arise from the broader instructional system.

1.3. Learning Processes, Experience, and Scenario-Based Performance

A systems evaluation also requires different educational indicators to remain conceptually distinct. Simulation studies commonly assess knowledge, performance, engagement, realism or presence, satisfaction, and technology acceptance, but these measures are not interchangeable. Greater realism often improves perceived presence or user experience without necessarily improving knowledge or task performance [19,30]. Likewise, learner engagement and demonstrated performance do not always change in parallel [20,31]. Participant-reported experience should therefore complement rather than substitute for objective or performance-based outcomes.

Perceived learning support (PLS) represents students’ perceptions of whether the instructional configuration supports their understanding of traffic-conflict mechanisms, vehicle interactions, and connections between theoretical knowledge and traffic situations. PLS is therefore treated as an instructional-perception indicator rather than as an objective learning outcome.

Perceived realism and interactivity (PRI) represent the perceived representational quality of the learning environment: whether the target situation is credibly represented and whether learners can inspect or act on that representation. Research on immersive and interactive learning indicates that these features can influence presence, attention, and learner experience, but their educational consequences depend on how interaction is aligned with the learning activity [23,30]. PRI is therefore treated as an experiential indicator rather than a direct proxy for learning effectiveness.

Cognitive engagement (CE) represents a learner-process indicator. Models of immersive learning position cognitive and affective processes between technological features and observable outcomes [32], while empirical studies have associated cognitive engagement with performance in simulation-supported environments [33,34]. Li et al. further showed that task-focused and resource-management engagement patterns were associated with stronger task performance than more receptive patterns [34]. CE is therefore examined as an observed system indicator rather than as evidence of a causal mediating mechanism.

Learning satisfaction and behavioral intention (LSBI) are treated as an acceptance-oriented output rather than as evidence of objective learning. Satisfaction has been associated with intentions to continue using simulation-based learning environments and with expectations, perceived value, motivation, and perceived learning experience [35]. Recent multi-institution evidence similarly indicates that satisfaction reflects learner, course, technical, interaction, and presence-related factors [36]. LSBI therefore characterizes acceptance of the instructional configuration while remaining analytically distinct from demonstrated knowledge or performance.

This distinction also motivates the inclusion of scenario-based analytical performance. Knowledge tests can establish whether students recognize relevant concepts and principles, whereas engineering competence also requires their application under contextual constraints. Authentic-assessment research has shown that computer-supported engineering tasks can reveal decision processes not captured by conventional assessments [37], while recent computer-based authentic assessment further illustrates how complex simulated tasks can expose patterns of engineering reasoning and task organization [38]. In transportation engineering, scenario tasks can therefore examine whether students can locate conflicts, identify interacting vehicles, connect observable cues with conflict mechanisms, and transfer those principles to related traffic situations.

Aggregate performance alone, however, provides limited guidance for instructional redesign. Process-level evaluation and log-data analysis have shown that overall outcomes can conceal variation in how learners engage with simulation-supported tasks [31,39]. A small between-configuration difference does not necessarily indicate poor performance, and low absolute performance can remain an instructional concern even when the simulation-supported configuration shows a positive comparative difference. Scenario- and question-level diagnosis should therefore consider absolute performance together with between-configuration contrasts. Such diagnostics identify candidate areas for subsequent instructional modification rather than demonstrated system defects.

1.4. Research Gaps, Study Contributions, and Research Questions

Three gaps follow from this literature. First, transportation-specific simulation education has a long history and recent studies demonstrate renewed interest in virtual simulation and integrated engineering activities [12,13,14], but much of the direct evidence remains based on individual course cases, post-instruction responses, or limited sets of outcomes. Classroom evaluations that distinguish formal knowledge, applied scenario performance, and learner-reported experience remain comparatively limited.

Second, realism, interactivity, engagement, satisfaction, knowledge, and applied performance represent different parts of the learning system and need not change in the same direction [19,34,36]. Evaluating simulation through a single claim of “effectiveness” therefore provides limited information about how an implemented instructional configuration functions. A more informative evaluation should distinguish these outputs and examine theoretically relevant relationships among selected indicators without assuming a causal path.

Third, assessment is commonly used to compare overall instructional outcomes but less often to identify what should be redesigned next. Authentic and scenario-based assessments can capture forms of engineering reasoning different from those represented by conventional tests [37,38], while aggregate scores can conceal variation across scenarios and reasoning tasks. System improvement therefore requires diagnostic evidence that considers both absolute performance and between-configuration differences.

These gaps are particularly relevant to contemporary transportation education, where increasingly advanced simulation technologies are entering curricula but their technical capability does not itself establish educational value [7,8,13,14]. The more consequential question is how such technology performs after being embedded within a coordinated instructional arrangement and how assessment evidence can inform subsequent redesign.

To address these issues, the present study evaluates a research-team-developed CARLA–SUMO platform after its integration into an undergraduate transportation engineering course. Rather than treating the software as an isolated intervention, the study defines a bounded socio-technical instructional system comprising technical resources, instructional organization, learner processes, and assessment and feedback. A class-level controlled quasi-experiment involving 174 students in four intact classes is examined through three complementary evaluation levels: configuration-level comparison, within-system relational analysis, and scenario- and question-level diagnosis for subsequent redesign.

The study contributes a systems-oriented evaluation structure that keeps formal knowledge, scenario-based analytical performance, and learning-experience indicators analytically distinct while connecting overall instructional outputs, selected within-system relationships, and assessment-informed redesign. The diagnostic level further extends evaluation beyond whether one configuration performs better overall by identifying where subsequent instructional refinement may be warranted.

Accordingly, the study addresses the following research questions:

RQ1.  What configuration-level outcome pattern is observed across formal knowledge, scenario-based analytical performance, and learning-experience profiles under conventional and simulation-supported instruction?

RQ2.  What within-system relationships are observed among instructional perceptions, learner processes, and learning outcomes within the simulation-supported configuration?

RQ3.  What redesign priorities are suggested by scenario- and question-level diagnostic patterns that jointly consider absolute performance and between-configuration contrasts?

2. Methodology

2.1. System Definition and Evaluation Framework

The focal system in this study was the classroom instructional arrangement used to teach the analysis of traffic conflicts in ramp-merging and weaving situations. The empirical study began after the research team had developed the CARLA–SUMO platform and configured it for classroom use. Its system boundary extended from the introduction of the learning objectives and the assessment of students’ initial characteristics to the immediate post-instruction evaluation of knowledge, scenario-based analytical performance, and learning experience. The surrounding environment included the course curriculum, classroom organization, instructor expertise, and institutional teaching resources, while students’ prior learning and simulation-related experience were treated as learner inputs at system entry. Long-term retention, professional transfer, and the implementation and evaluation of a subsequent redesign iteration were outside the scope of the present study.

Within this boundary, conventional and simulation-supported teaching were treated as two alternative instructional configurations. The conventional configuration relied on instructor explanation, slides, diagrams, static traffic cases, and classroom discussion. The simulation-supported configuration combined dynamic traffic representation, controllable scenarios, structured analytical tasks, teacher guidance, guided observation, individual learner operation, discussion, and debriefing. The unit of comparison was therefore the instructional configuration as a whole rather than the simulation platform in isolation. Because several technical and instructional elements changed simultaneously, the observed group differences cannot be attributed to CARLA–SUMO or to any single platform function independently.

The simulation-supported configuration was conceptualized as comprising four interrelated elements: technical resources, instructional organization, learner processes, and assessment and feedback. Technical resources provided the co-simulation environment, scenario control, interactive representation, and data recording. Instructional organization connected these resources to course objectives through scenario selection, teacher prompts, guided observation, discussion, and debriefing. Learner processes included prior knowledge, interest, and relevant experience as inputs, together with attention, cognitive engagement, interpretation, and knowledge construction during instruction. Assessment captured formal knowledge, scenario-based analytical performance, and participant-reported indicators of the learning process and instructional acceptance. Their relationships are summarized in Figure 1.

Figure 1. Socio-technical instructional system boundary and three-level evaluation framework. The dashed feedback path represents proposed assessment-informed feedback for a subsequent redesign iteration rather than an empirically tested closed loop.

The framework also distinguishes implemented system features from participant-level indicators. Platform control, teacher guidance, and classroom discussion were components of the instructional design, but their separate effects were not directly measured. PLS, PRI, CE, post-instruction knowledge, scenario performance, and LSBI were used as observable indicators of selected system dimensions. The relational analysis therefore examined associations among these indicators rather than estimating the independent causal effect of every system component.

The evaluation was conducted at three complementary levels. First, configuration-level analysis compared knowledge, scenario-based performance, and learning-experience profiles across the two instructional configurations. Second, within-system analysis examined relationships among technical perceptions, instructional support, learner processes, and learning outputs within the simulation-supported group. Third, scenario- and task-question-level diagnostics integrated absolute performance with between-configuration contrasts to identify candidate priorities for subsequent instructional redesign. This structure connects system performance, internal relationships, and assessment-informed feedback while avoiding claims of a complete causal mechanism or an empirically closed feedback loop.

2.2. Study Design and Participants

A class-level controlled quasi-experimental design with a pre-test–intervention–post-test structure was used to compare the two instructional configurations. Participants came from four intact undergraduate classes within the same transportation engineering course. Existing course scheduling and classroom organization precluded the reassignment of individual students across classes. Two classes received the conventional configuration, and two received the simulation-supported configuration. Because individual students were not randomized and the allocation of the four classes was not documented as random, the study was treated as a class-level quasi-experiment rather than a fully randomized trial. Participants completed baseline assessments before instruction and immediate post-instruction assessments after their respective teaching sessions.

The study included 174 undergraduate students, with 87 receiving the conventional configuration and 87 receiving the simulation-supported configuration. Each class met for two 45-min periods per week over eight weeks, yielding 16 scheduled class periods (720 min) per class. All four classes were taught by the same instructor, addressed the same learning objectives and core content, and were conducted within this common schedule. The course content focused on traffic conflicts and vehicle interactions in ramp-merging and weaving situations. The principal difference between the configurations concerned how this content was represented and how the associated learning activities were organized, as described in the following subsections. These common conditions helped limit variation related to the instructor, topic coverage, and scheduled instructional time. However, because each configuration comprised only two intact classes, unmeasured class-level differences could not be separated fully from configuration differences.

Before instruction, participants provided information on gender, driver’s license status, driving experience, frequent prior exposure to driving-related games or simulation software, prior coursework in transportation engineering or traffic safety, and initial interest in the course content. Participants additionally completed a knowledge pre-test. These variables were collected to characterize the sample and examine observed baseline differences rather than as primary learning outcomes. All participants confirmed that they were at least 18 years old and provided informed consent before taking part. The study was reviewed and approved by the School of Modern Posts, School of Intelligent Transportation, Nanjing University of Posts and Telecommunications, before data collection. Participation was voluntary and did not affect course grades, and all participant data were analyzed and reported in anonymized or aggregated form.

2.3. Instructional System Configurations

The two instructional configurations addressed the same transportation engineering content but organized its representation and classroom enactment differently. Both configurations combined instructional materials, instructor guidance, learner activity, and classroom discussion. They differed primarily in how traffic interactions were represented, controlled, and incorporated into the analytical activities.

2.3.1. Conventional Instructional Configuration

The conventional configuration followed an instructor-led classroom format. Core concepts related to ramp merging, weaving sections, vehicle interaction, and traffic conflict formation were introduced through instructor explanation, slides, roadway diagrams, and six static traffic cases corresponding to the six scenario cases used in the simulation-supported configuration. Representative examples were used to explain how gap availability, speed coordination, lane-changing demand, and path competition contributed to the development of traffic conflicts.

Students examined the presented materials, responded to analytical questions, and participated in classroom discussion. They were asked to identify relevant vehicles and conflict locations, interpret the relationships represented in the cases, and explain the processes leading to the described traffic conditions. Temporal sequences of vehicle interaction were represented through diagrams, case descriptions, and instructor explanation rather than through continuously evolving traffic scenes.

2.3.2. Simulation-Supported Instructional Configuration

The simulation-supported configuration addressed the same concepts through six dynamic and controllable traffic scenario cases embedded in guided analytical activities. The scenarios represented lane changes, speed adjustments, limited gap availability, path competition, and conflict development under different traffic conditions. These representations allowed students to observe how local vehicle actions and surrounding traffic conditions jointly shaped the evolution of a traffic situation. The instructional focus and principal conditions of the six matched cases are summarized in Appendix A Table A1.

For each of the six instructional cases, the instructor introduced the learning objective and analytical focus, directed attention toward relevant vehicles and interaction cues, and prompted students to identify conflicts and explain the underlying processes. Students first examined the evolving scenario through guided observation and then operated it through their individual client workstations. Every student completed hands-on operation of all six scenarios. Classroom discussion and debriefing connected the observed traffic processes with the corresponding engineering concepts and analytical principles.

The simulation component was therefore embedded in an instructor-guided sequence rather than used as a stand-alone visual demonstration. Dynamic representation, scenario selection, structured analytical tasks, instructor prompts, guided observation, learner operation, discussion, and debriefing were organized as parts of the same instructional configuration. Across the simulation-supported instructional sessions, approximately one fifth of the available instructional time was devoted to instructor demonstration, three fifths to student operation, and one fifth to classroom discussion and debriefing.

2.3.3. Consistency Across Configurations

All four classes were taught by the same instructor, addressed the same learning objectives and core topics, and were conducted within the same scheduled class duration. The six instructional cases were matched one-to-one across configurations in traffic situation, conceptual focus, and analytical task; they differed in whether the case was represented statically or through a dynamic and controllable scenario. The brief platform orientation provided to the simulation-supported classes formed part of the scheduled duration and did not increase their total contact time. The same knowledge and scenario-based assessments were administered to all participants in the same sequence, followed by the learning-experience questionnaire corresponding to the assigned configuration.

The configurations differed in the form of traffic representation, the controllability of the presented situations, the opportunities for learner interaction, and the enactment of the analytical activities. These differences were interpreted jointly in the configuration-level analyses.

2.4. Technical Platform and Instructional Implementation

The technical subsystem used in the simulation-supported configuration was designed and developed by the research team for classroom use and then deployed in the participating course. It integrated CARLA version 0.9.14 and SUMO version 1.18.0 in a classroom-oriented co-simulation platform: CARLA provided the three-dimensional driving environment and ego-vehicle control, while SUMO generated background traffic and interaction vehicles. Their integration enabled ramp-merging and weaving scenarios to combine a controllable focal vehicle with parameterized surrounding traffic, allowing vehicle interactions and conflict development to be represented dynamically. The present study therefore examined the classroom deployment and educational evaluation stage of the platform lifecycle rather than the software as an isolated artifact.

Instructional cases were organized through a scenario library. Each case was linked to a traffic situation, an instructional focus, and a corresponding analytical task. Before a scenario was launched, the setup interface allowed the user to specify the initial viewpoint, weather, simulation time, vehicle type, traffic density, and data-recording location. These functions enabled the instructor to select cases and configure the traffic conditions required for a particular teaching activity.

The platform adopted a role-based client–server structure comprising one server host and an individual client workstation for each student. The server host supported scenario loading, traffic generation, and simulation control, while the student-side clients provided every participant with direct access for guided observation and hands-on scenario operation. The main console integrated classroom demonstration, student practice, data recording and replay, and system control. Students primarily operated the scenarios using a keyboard and mouse at their individual client workstations. A simulator workstation equipped with a steering wheel and pedals was also used, with all students taking turns to operate it. The same scenario organization was retained across instructor demonstration and learner practice.

Figure 2 summarizes the architecture that connected instructor-side control, the scenario library and parameter configuration, the CARLA–SUMO co-simulation engine, student-side access, and the teaching interface.

Figure 2. Architecture and instructional use of the research-team-developed CARLA–SUMO platform. Platform-generated records supported replay, discussion, and debriefing; formal learning outcomes were collected separately through the knowledge tests, scenario-based tasks, and questionnaires.

Data generated during scenario operation could be recorded and replayed to support classroom discussion and post-activity debriefing. These records served an instructional review function and were not included as participant-level learning outcomes in the present analysis. Figure 3 complements the architecture diagram by presenting the main interfaces and the classroom deployment.

Figure 3. Technical platform and classroom implementation of the simulation-supported instructional configuration. (a) Login and role access. (b) Main function console. (c) Scenario library. (d) Simulation setup interface. (e) Instructor-side CARLA–SUMO co-simulation. (f) Classroom deployment. (g) Learner driving view.

2.5. Experimental Procedure

The experiment followed a common sequence from baseline assessment to immediate post-instruction evaluation. After the pre-test, the two simulation-supported classes received a brief platform orientation at the beginning of their scheduled instructional sessions. The orientation covered role access, scenario entry, and basic operating controls. It formed part of the same scheduled class duration and did not increase the total contact time for the simulation-supported configuration. Table 1 summarizes the complete procedure.

Table 1. Sequence of the class-level quasi-experimental teaching procedure.

Phase	Step	Procedure
Preparation	1	Explain the study procedure and task requirements and confirm informed consent.
Baseline assessment	2	Collect the participant background information.
	3	Administer the knowledge pre-test.
Configuration-specific preparation	4	If the assigned configuration is simulation-supported, then provide a brief platform orientation covering role access, scenario entry, and basic operating controls.
Instruction	5	Deliver the assigned instructional configuration:
		Conventional:	Instructor explanation and discussion of six corresponding static traffic cases.
		Simulation-supported:	Guided observation and hands-on operation of all six dynamic scenarios by every student, followed by discussion and debriefing.
Immediate post-instruction assessment	6	Administer the knowledge post-test.
	7	Administer the scenario-based analytical tasks.
	8	Administer the learning-experience questionnaire corresponding to the assigned instructional configuration.
Data preparation	9	De-identify, organize, and store the participant-level records for analysis.

The post-instruction assessments were administered in the same order in all four classes. The knowledge post-test and rubric-scored scenario tasks preceded the learning-experience questionnaire so that performance-based outcomes were recorded before participants completed self-reported evaluations of the instructional configuration.

2.6. Measures

All research instruments were administered in Chinese. Complete English-language translations of the instruments, together with the answer keys and scoring criteria used in the study, are consolidated in Appendix A.

2.6.1. Learner and Contextual Inputs

The participant background form recorded gender, driver’s license status, actual driving experience, frequent prior exposure to driving-related games or simulation software, prior coursework in transportation engineering or traffic safety, and initial interest in the course content. Driving experience was recorded using four categories: none, 0–2 years, 2–5 years, and more than 5 years. Frequent prior exposure to driving-related games or simulation software and prior transportation-related coursework were recorded as binary indicators. Initial interest was measured using a five-category item coded from 0 to 4, with higher scores indicating greater initial interest.

Participant type was also recorded in the background form. However, all participants were undergraduate students, so this variable showed no analytical variation and was not retained in the subsequent analyses. The remaining background variables were used to characterize the learner context, examine observed baseline differences between the two instructional configurations, and where theoretically relevant, serve as contextual covariates. The complete background form is provided in Appendix A.2.

2.6.2. Formal Knowledge Assessment

Formal knowledge was assessed using a pre-test and a post-test covering ramp merging, weaving sections, acceptable gaps, speed coordination, multi-vehicle interaction, traffic-conflict formation, and the application of these concepts to related traffic situations. The pre-test contained five multiple-choice questions and two true/false questions, each worth one point, together with one rubric-scored short-answer question worth three points. The maximum pre-test score was therefore 10 points.

The post-test contained eight multiple-choice questions and four true/false questions, each worth one point, together with three rubric-scored short-answer questions worth three, three, and two points, respectively. The maximum post-test score was 20 points. The two assessments covered overlapping core content but were not designed as parallel forms. The pre-test emphasized baseline recognition and a brief explanation of ramp-merging, weaving, gap availability, speed coordination, and conflict mechanisms, whereas the post-test sampled these same domains more broadly and included additional mechanism comparison and knowledge-transfer demands. A test blueprint mapping the items to their principal content domains, cognitive demands, and score contributions is provided in Appendix A.5. Objective items were scored using predefined answer keys, while the short-answer questions were scored using criteria based on the identification and explanation of relevant traffic-engineering concepts and conflict-formation mechanisms.

Because the pre-test and post-test had different maximum scores, their raw totals were converted to score ratios:

$$K_{\mathrm{pre}}={\displaystyle \frac{S_{\mathrm{pre}}}{10}},K_{\mathrm{post}}={\displaystyle \frac{S_{\mathrm{post}}}{20}},\Delta K=K_{\mathrm{post}}-K_{\mathrm{pre}},$$

(1)

where $S_{\mathrm{pre}}$ and $S_{\mathrm{post}}$ denote the raw pre-test and post-test scores, $K_{\mathrm{pre}}$ and $K_{\mathrm{post}}$ denote the corresponding score ratios, and $\Delta K$ denotes the change in score ratio from pre-test to post-test. The pre-test ratio represented baseline knowledge, while the post-test ratio represented immediate post-instruction knowledge performance. The score-ratio change was retained as a supplementary indicator of within-participant change.

Converting the two totals to a common 0–1 range accounted for their different maximum scores but did not establish that the tests were psychometrically parallel, because they differed in length and item composition. Baseline-adjusted post-test performance was therefore treated as the primary formal knowledge outcome, with score-ratio change used as a supplementary outcome. The complete English-language test items, answer keys, and scoring criteria are provided in Appendix A.5.

2.6.3. Scenario-Based Analytical Assessment

Scenario-based analytical performance was assessed using three written tasks that required students to organize relevant traffic cues, explain vehicle interactions, and apply traffic-conflict concepts. Scenario 1 addressed conflict identification in a ramp-merging situation. Its four components assessed conflict-type identification, identification and explanation of the critical conflict location, analysis of contributing factors, and proposal of an appropriate improvement measure. The respective maximum scores were 2, 3, 3, and 2 points.

Scenario 2 addressed multi-vehicle interaction in a weaving section. Its four components assessed recognition of the scenario’s additional complexity, identification of the interacting vehicle behaviors, explanation of how traffic density affected conflict development, and judgment of the most important influencing factor. These components were scored on 2-, 3-, 3-, and 2-point scales, respectively.

Scenario 3 examined the application of conflict-formation principles to a closely spaced entrance–exit situation. Its four components assessed recognition and explanation of mechanisms shared with weaving sections, identification of indicators of increased conflict risk, comparison with a standard ramp-merging situation, and identification of an appropriate instructional focus. These components were scored on 3-, 3-, 2-, and 2-point scales, respectively.

Each scenario contributed a maximum of 10 points, producing an overall maximum score of 30. Responses were evaluated using predefined question-level rubrics that specified the mechanism-related elements required for partial and full credit. The three scenario totals and the overall score were used as applied learning outcomes. The 12 scenario-task question scores were additionally retained for the scenario- and task-question-level diagnostic analysis. The complete task prompts and question-level scoring rubrics are provided in Appendix A.4.

One trained rater scored all scenario-task responses. To assess inter-rater reliability, a class-stratified random subset of 52 response sets was selected, with 13 participants randomly sampled from each of the four classes. This subset represented 29.9% of the full sample. A second trained rater independently scored the selected responses while blinded to class membership, instructional configuration, and the scores assigned by the first rater. Inter-rater reliability was evaluated separately for each scenario total and for the overall 30-point score using two-way random-effects, absolute-agreement, single-measure intraclass correlation coefficients (ICC(2,1)). As shown in Table 2, all four point estimates indicated good inter-rater agreement according to the interpretive guidelines of Koo and Li [40], although the confidence intervals for Scenario 2 and the overall score extended slightly into the moderate-agreement range.

Table 2. Inter-rater reliability of the scenario-based analytical performance scores.

Assessment Component	ICC(2,1)	95% CI
Scenario 1 score	0.85	[0.78, 0.94]
Scenario 2 score	0.79	[0.73, 0.88]
Scenario 3 score	0.83	[0.76, 0.93]
Overall scenario-based score	0.81	[0.72, 0.89]

Note. ICC(2,1) denotes a two-way random-effects, absolute-agreement, single-measure intraclass correlation coefficient. CI denotes confidence interval. Estimates were based on a class-stratified random subset of 52 response sets (13 from each of four classes; 29.9% of the full sample).

2.6.4. Learning-Experience and Acceptance Indicators

Learning-experience and acceptance indicators were collected using two configuration-specific versions of a questionnaire developed for the present study. The initial item pool contained 26 items and was constructed around the course objectives, instructional activities, and four intended dimensions. The items were reviewed for content relevance by 12 faculty members with expertise in transportation engineering. Before the main study, the questionnaire was pilot-tested with 29 participants. Based on the pilot results and item-level review, six items showing weak correspondence with the intended dimensions or unsuitable wording were removed before formal data collection. The resulting questionnaire contained 20 items, with five items retained for each of the four intended dimensions.

PLS represented students’ perceptions of whether the instructional configuration supported their understanding of traffic-conflict mechanisms, vehicle interactions, and connections between theoretical knowledge and traffic situations. CE covered self-reported attention, active thinking, participation in discussion, learning interest, and effort invested in understanding vehicle interactions. PRI represented the perceived representational quality of the learning environment, combining contextual realism with the interactive and dynamic presentation of vehicle interactions. These attributes were considered jointly because both concern the extent to which the instructional representation made evolving traffic processes observable, interpretable, and experientially accessible to learners.

LSBI was treated as an acceptance-oriented output of the instructional configuration rather than as a direct measure of subsequent behavior. Its five items jointly captured satisfaction with the teaching approach, perceived practical value, willingness to participate in similar activities in other courses, perceived improvement in the overall learning experience, and willingness to recommend the approach to other students. The composite therefore represented students’ overall acceptance of the instructional configuration, including both evaluative satisfaction and stated intention toward future use.

The two questionnaire versions retained the same dimensional organization and conceptually corresponding item positions, but their wording referred to the instructional configuration experienced by each group. The simulation-supported version referred to the simulation system, dynamic scenarios, interactive activities, and post-simulation debriefing, whereas the conventional version referred to classroom instruction, diagrams, static cases, discussion, and post-instruction review. Several items in the simulation-supported form also made explicit comparisons with conventional instruction, while the corresponding conventional-form items used absolute evaluations. The resulting composite scores were therefore treated as conceptually aligned learning-experience profiles rather than as measurement-invariant versions of an identical scale [41]. Construct validity and internal consistency were therefore evaluated separately for the two questionnaire versions. Both complete questionnaire versions are provided in Appendix A.3.

2.7. Analytical Strategy

The analytical strategy followed the three evaluation levels defined in Section 2.1. Configuration-level analyses and scenario- and task-question-level diagnostic analyses used the full sample ($N=174$) to address RQ1 and RQ3, respectively. Within-system relational analyses used the 87 participants in the simulation-supported configuration to address RQ2. The final dataset contained complete records for all variables used in these analyses; therefore, no imputation was required.

Before analysis, the participant-level records were screened for duplicate identifiers, missing values, out-of-range responses, and inconsistencies between item scores and derived totals. Means and standard deviations (SDs) were used to summarize approximately continuous score variables, while frequencies and percentages were used for categorical variables. Medians and interquartile ranges were used for the five-category initial-interest item and were additionally examined for markedly asymmetric score distributions.

All analyses were conducted using Python version 3.9.13. Regression-based inference used HC3 heteroskedasticity-consistent standard errors. Bootstrap confidence intervals were estimated using the percentile method with 5000 resamples. For configuration-level and diagnostic between-configuration comparisons, participant records were resampled separately within the conventional and simulation-supported configurations, preserving 87 participants per configuration in each resample. For within-system sensitivity analyses, records were resampled separately within the two simulation-supported classes, preserving their respective sample sizes of 43 and 44. All statistical tests were two-sided, with a nominal significance level of $\alpha =0.05$. Uncertainty was reported using 95% confidence intervals, and interpretation emphasized the magnitude, direction, and precision of estimated differences and associations rather than statistical significance alone.

Construct validity of the learning-experience questionnaire was examined separately within the conventional and simulation-supported configurations because the two forms used configuration-specific wording and were not assumed to be measurement-invariant. Given the ordinal five-category item responses, exploratory factor analysis was based on polychoric correlation matrices. Sampling adequacy was evaluated using the Kaiser–Meyer–Olkin (KMO) statistic and Bartlett’s test of sphericity. The number of factors was examined using parallel analysis and the Kaiser criterion. Principal-axis factoring with oblimin rotation was then used to estimate the correlated factor structure. Factor interpretation considered primary loadings, cross-loadings, communalities, and correspondence with the four intended dimensions. Convergent validity was assessed using average variance extracted (AVE) and composite reliability (CR), and discriminant validity was examined by comparing the square root of each construct’s AVE with the corresponding inter-factor correlations. Internal consistency was additionally evaluated using Cronbach’s alpha.

2.7.1. Configuration-Level Analysis

Baseline characteristics were summarized separately for the conventional and simulation-supported configurations. The pre-test score ratio was compared using a Welch independent-samples t-test, and the five-category initial-interest item was examined using the Mann–Whitney U test. Categorical variables were examined using Pearson’s chi-square test or an exact test when expected cell frequencies were small. These comparisons were used to describe observed baseline differences and were not treated as evidence that the intact classes were statistically equivalent. Standardized mean differences, rank-biserial correlations, or Cramer’s V, as appropriate, were also examined to describe the magnitude of baseline imbalance.

The primary formal-knowledge analysis used the post-test score ratio as the dependent variable in a baseline-adjusted linear model containing instructional configuration and the pre-test score ratio. The configuration-by-pre-test interaction was examined as a diagnostic of the homogeneity-of-slopes assumption. If the interaction indicated meaningful nonparallel slopes, conditional configuration differences were reported rather than a single common adjusted difference. The unadjusted post-test difference and the between-configuration difference in score-ratio change ($\Delta K$) were reported as complementary results. Because the pre-test and post-test differed in length and item composition and were not psychometrically parallel, score-ratio change was interpreted as a supplementary outcome rather than as the primary estimate of learning.

Overall scenario-based analytical performance was compared between configurations using the 30-point total score. A complementary adjusted model included instructional configuration, the pre-test score ratio, and initial interest, all of which were measured before instruction. Post-instruction knowledge was not included as a covariate in this configuration–comparison model because it was measured after instruction and could lie downstream of the instructional configuration. For the knowledge and overall scenario outcomes, unadjusted group means, raw mean differences, and Cohen’s d values were reported. Confidence intervals for the raw mean differences and Cohen’s d values were obtained from 5000 percentile bootstrap resamples conducted separately within the two instructional configurations. Positive differences were defined as favoring the simulation-supported configuration.

The four questionnaire composites were summarized separately by instructional configuration on their original 0–4 scales. Internal consistency was evaluated using Cronbach’s alpha separately within each configuration. Between-configuration differences in PLS, CE, PRI, and LSBI were described using means, standard deviations, raw mean differences, and Cohen’s d. The 95% confidence intervals for the raw and standardized differences were estimated from 5000 percentile bootstrap resamples conducted separately within each instructional configuration. Because the two questionnaire forms used configuration-specific wording and measurement invariance was not established, these contrasts were interpreted as descriptive comparisons of learning-experience profiles rather than as inferential comparisons of strictly equivalent scale scores.

One post-test item explicitly referred to the type of instructional activity that could support understanding of traffic-conflict mechanisms. To examine whether the formal-knowledge comparison depended on this item, the post-test score ratio was recalculated after excluding it and rescaling the remaining 19 available points to a 0–1 range. The baseline-adjusted knowledge analysis was then repeated using this alternative score.

Because driving experience and gender showed the largest observed baseline imbalances, the primary knowledge and overall scenario-performance models were repeated with driving experience (four categories) and gender entered separately and jointly as sensitivity covariates.

Instructional configuration was implemented at the level of four intact classes, with two classes receiving each configuration. Because four clusters were insufficient for dependable cluster-robust or multilevel variance estimation, participant-level inferential results were not interpreted as if individual randomization had occurred. Class-specific outcome summaries and the consistency of the observed direction across the two classes within each configuration were therefore examined as a descriptive sensitivity check. This check could indicate whether a configuration contrast was disproportionately influenced by one class, but it could not eliminate class-level confounding. Accordingly, the participant-level confidence intervals and p-values should be interpreted as conditional on the four observed classes and may not capture the full uncertainty associated with class-level assignment and between-class variation.

2.7.2. Within-System Relational Analysis

RQ2 was examined only within the simulation-supported configuration ($n=87$), thereby avoiding the assumption that the two configuration-specific questionnaire forms were measurement-invariant. Pearson correlations were first used to describe the bivariate relationships among PLS, PRI, CE, post-instruction knowledge, overall scenario performance, and LSBI. Confidence intervals for the Pearson correlation coefficients were calculated using Fisher’s z transformation. Spearman correlations were examined as a rank-based sensitivity check.

Three prespecified multivariable linear regressions represented the relationships shown in the within-system component of the evaluation framework:

$$\begin{array}{cc}\hfill C{E}_i& ={\beta }_0+{\beta }_{1}PL{S}_i+{\beta }_{2}PR{I}_i+{\beta }_{3}Interes{t}_i+{\beta }_{4}Clas{s}_i+{\epsilon }_i,\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill Scenari{o}_i& ={\beta }_0+{\beta }_{1}C{E}_i+{\beta }_2{K}_{\mathrm{post},i}+{\beta }_3{K}_{\mathrm{pre},i}+{\beta }_{4}Exposur{e}_i+{\beta }_{5}Clas{s}_i+{\epsilon }_i,\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill LSB{I}_i& ={\beta }_0+{\beta }_{1}PL{S}_i+{\beta }_{2}PR{I}_i+{\beta }_{3}C{E}_i+{\beta }_4{K}_{\mathrm{post},i}+{\beta }_{5}Clas{s}_i+{\epsilon }_i.\hfill \end{array}$$

(4)

Here, $K_{\mathrm{pre},i}$ and $K_{\mathrm{post},i}$ denote the pre-test and post-test score ratios, respectively. $Clas{s}_i$ distinguishes the two simulation-supported classes, and $Exposur{e}_i$ represents frequent prior exposure to driving-related games or simulation software. The first model examined whether PLS and PRI were associated with CE after accounting for initial interest and class membership. The second examined the conditional associations of CE and post-instruction knowledge with scenario performance after accounting for baseline knowledge, prior simulation-related exposure, and class membership. The third examined the associations of PLS, PRI, CE, and post-instruction knowledge with LSBI after accounting for class membership.

For each regression, unstandardized coefficients, standardized coefficients, and 95% confidence intervals were reported. HC3-based confidence intervals provided the primary coefficient inference. Class-stratified percentile bootstrap confidence intervals based on 5000 resamples were examined as a sensitivity analysis. Linearity and residual patterns were inspected graphically, and variance inflation factors were used to assess excessive overlap among predictors. Observations with unusually large Cook’s distance, leverage, or studentized residuals were examined through influence sensitivity analyses. Such observations were not removed from the primary analysis solely because they exceeded a diagnostic guideline.

All variables in the relational analysis were observed composite scores. The regressions therefore estimated conditional associations among selected system indicators rather than latent-variable relationships or the independent causal effects of individual system components. Because PLS, PRI, CE, and LSBI were measured at the same post-instruction occasion, their temporal ordering was not empirically established. The regression system was not interpreted as a causal path model, and no causal mediation effect was estimated.

2.7.3. Scenario- and Task-Question-Level Diagnostic Analysis

RQ3 used between-configuration contrasts for the three scenario totals and the 12 rubric-scored task questions. Because the maximum available score differed across questions, each question score was also expressed as a proportion of its available score for descriptive comparison. Raw mean differences and Cohen’s d values were calculated as simulation-supported minus conventional, such that positive values indicated higher performance under the simulation-supported configuration. Corresponding 95% confidence intervals were estimated from 5000 percentile bootstrap resamples conducted separately within each instructional configuration. Two-sided Welch independent-samples t-tests were used to obtain the unadjusted p-values for the scenario- and question-level comparisons.

Multiplicity was addressed using the Holm procedure separately for the family of three scenario-level tests and the family of 12 question-level tests. Holm-adjusted p-values were reported as supplementary statistical information. Diagnostic interpretation used two complementary dimensions: (1) the normalized score level within the simulation-supported configuration as an indicator of absolute performance and (2) the magnitude and uncertainty of the between-configuration standardized difference as an indicator of comparative performance. No single numerical cutoff was used to classify an output as weak. Outputs combining comparatively low absolute performance with a small or uncertain between-configuration difference were given the strongest priority for subsequent redesign. Comparatively low absolute performance accompanied by a larger positive configuration difference was interpreted as a remaining learning challenge despite a stronger relative result under simulation-supported instruction, whereas a small configuration difference accompanied by moderate or relatively high absolute performance was interpreted as limited comparative separation rather than evidence of poor performance. All resulting redesign implications were treated as assessment-informed and provisional rather than as evidence of confirmed system defects.

3. Results and Discussion

3.1. Baseline Characteristics and System Context

The 174 participants were drawn from four intact undergraduate classes. The two classes receiving the conventional configuration included 45 and 42 participants, respectively, whereas the two simulation-supported classes included 43 and 44 participants, respectively. Class sizes were therefore closely balanced within and between the two instructional configurations. All 174 participants had complete records for the baseline variables and were included in the baseline analysis.

Table 3 summarizes the observed learner and contextual inputs. Initial interest had a median of 2 in both configurations. The Mann–Whitney test provided little evidence of a between-configuration difference in its distribution ($U=3711.5$, $p=0.821$, $r_{\mathrm{rb}}=0.019$). The mean pre-test ratio was 0.485 (SD = 0.189) in the conventional configuration and 0.505 (SD = 0.193) in the simulation-supported configuration. The corresponding difference was 0.0195 (95% CI [−0.0377, 0.0768]), $t\left(171.91\right)=0.674$, $p=0.501$, and $d=0.102$.

Table 3. Baseline characteristics by instructional configuration.

Variable	Conventional ($\mathit{n}=87$)	Simulation-Supported ($\mathit{n}=87$)	Test	p	Effect Size
Initial interest	2 [1, 2]	2 [1, 3]	$U=3711.5$	0.821	$r_{\mathrm{rb}}=0.019$
Pre-test ratio	0.485 (0.189)	0.505 (0.193)	$t\left(171.91\right)=0.674$	0.501	$d=0.102$
Gender	Male: 42 (48.3%) Female: 45 (51.7%)	Male: 53 (60.9%) Female: 34 (39.1%)	${\chi }^2\left(1\right)=2.805$	0.094	$V=0.127$
Driver’s license	No: 52 (59.8%) Yes: 35 (40.2%)	No: 45 (51.7%) Yes: 42 (48.3%)	${\chi }^2\left(1\right)=1.142$	0.285	$V=0.081$
Driving experience	None: 52 (59.8%) 0–2 years: 11 (12.6%) 2–5 years: 12 (13.8%) More than 5 years: 12 (13.8%)	None: 45 (51.7%) 0–2 years: 20 (23.0%) 2–5 years: 17 (19.5%) More than 5 years: 5 (5.7%)	${\chi }^2\left(3\right)=6.862$	0.076	$V=0.199$
Frequent simulation/game use	No: 38 (43.7%) Yes: 49 (56.3%)	No: 41 (47.1%) Yes: 46 (52.9%)	${\chi }^2\left(1\right)=0.209$	0.648	$V=0.035$
Prior transportation coursework	No: 47 (54.0%) Yes: 40 (46.0%)	No: 51 (58.6%) Yes: 36 (41.4%)	${\chi }^2\left(1\right)=0.374$	0.541	$V=0.046$

Note. Initial interest is reported as median [interquartile range]; the pre-test ratio is reported as mean (SD); categorical variables are reported as frequency (percentage). Pearson chi-square tests were calculated without continuity correction, and all expected cell counts exceeded five. $r_{\mathrm{rb}}$ denotes the rank-biserial correlation, and V denotes Cramer’s V. Positive signed effects indicate higher values in the simulation-supported configuration.

None of the observed baseline comparisons reached the nominal significance threshold. Driving-experience composition showed the largest observed imbalance ($V=0.199$), followed by gender composition ($V=0.127$); the remaining effect-size estimates were small. The absence of statistically significant baseline differences was not interpreted as evidence of baseline equivalence or as eliminating possible class-level confounding.

3.2. Configuration-Level Learning Outcomes

The configuration-level analysis addressed RQ1 by examining formal knowledge, scenario-based analytical performance, and learning-experience profiles as complementary outputs of the two instructional configurations.

3.2.1. Formal Knowledge Outcomes

The mean post-test ratio was 0.503 (SD = 0.249) in the conventional configuration and 0.610 (SD = 0.254) in the simulation-supported configuration. The unadjusted mean difference was 0.107, corresponding to $d=0.425$. The score-ratio change was also higher in the simulation-supported configuration, with means of 0.105 (SD = 0.243) and 0.018 (SD = 0.205), respectively. Because the pre-test and post-test differed in length and item composition, this change score was interpreted as a supplementary indicator rather than as the primary knowledge outcome. Table 4 summarizes the configuration-level performance outcomes.

Table 4. Configuration-level knowledge and scenario-based outcomes.

Outcome	Conventional ($\mathit{n}=87$)	Simulation-Supported ($\mathit{n}=87$)	Mean Difference [95% CI]	Cohen’s d [95% CI]
Post-test ratio	0.503 (0.249)	0.610 (0.254)	0.107 [0.030, 0.179]	0.425 [0.121, 0.740]
Score-ratio change ($\Delta K$)	0.018 (0.205)	0.105 (0.243)	0.087 [0.020, 0.155]	0.389 [0.090, 0.698]
Scenario total (0–30)	11.632 (4.796)	15.540 (5.182)	3.908 [2.506, 5.391]	0.783 [0.499, 1.104]

Note. Group values are reported as mean (SD). Differences are calculated as simulation-supported minus conventional. Confidence intervals are percentile intervals based on 5000 bootstrap resamples conducted separately within each instructional configuration.

In the primary baseline-adjusted model, the simulation-supported configuration remained positively associated with the post-test ratio after accounting for the pre-test ratio ($b=0.094$, 95% CI [0.028, 0.160], $p=0.006$). The pre-test ratio was also positively associated with post-test performance ($b=0.675$, 95% CI [0.504, 0.846], $p<0.001$). The model explained 29.6% of the variance in post-test ratios (adjusted $R^2=0.287$).

The configuration-by-pre-test interaction was not clearly supported ($b=-0.208$, 95% CI [−0.554, 0.139], $p=0.239$), providing no indication that the configuration contrast depended strongly on initial knowledge within the observed range. Removing Post-test Item 7 and rescaling the remaining 19 points produced a similar adjusted configuration coefficient ($b=0.098$, 95% CI [0.032, 0.163], $p=0.004$). The knowledge result was therefore not dependent on the instructionally explicit wording of that item.

The adjusted configuration coefficient also remained positive when driving experience was added as a four-category covariate, when gender was added, and when both were added jointly. Across these sensitivity models, the coefficient ranged from 0.077 to 0.088, and each corresponding confidence interval excluded zero. The observed knowledge contrast was therefore reduced slightly, but not eliminated, after accounting for the largest baseline imbalances.

3.2.2. Scenario-Based Analytical Performance

The mean overall scenario score was 11.632 (SD = 4.796) in the conventional configuration and 15.540 (SD = 5.182) in the simulation-supported configuration. The raw difference was 3.908 points on the 30-point assessment, corresponding to $d=0.783$. This standardized contrast was larger than the corresponding contrast in post-instruction knowledge.

After adjustment for the pre-test ratio and initial interest, the configuration coefficient remained positive ($b=3.677$, 95% CI [2.304, 5.050], $p<0.001$). Both the pre-test ratio ($b=9.506$, 95% CI [5.709, 13.303], $p<0.001$) and initial interest ($b=0.993$, 95% CI [0.332, 1.653], $p=0.003$) were positively associated with scenario performance. The adjusted model explained 30.8% of the variance in overall scenario scores (adjusted $R^2=0.296$).

Adding driving experience, gender, or both to the adjusted model produced configuration coefficients ranging from 3.701 to 3.799 points, all with $p<0.001$. The configuration contrast was therefore not explained by the observed differences in these baseline characteristics.

Class-specific descriptive statistics are reported in Table 5. Pre-test performance was broadly overlapping across the four classes. Both simulation-supported classes had higher mean post-test ratios and scenario-based analytical performance than either conventional class. This directional consistency indicates that the aggregate configuration differences were not driven solely by one class. However, with only four intact classes, these descriptive patterns cannot separate configuration differences from unmeasured class-level influences and should not be interpreted as a substitute for cluster-level inference.

Table 5. Class-specific descriptive statistics for the primary learning outcomes.

Class	Configuration	n	Pre-Test Ratio Mean (SD)	Post-Test Ratio Mean (SD)	Scenario Total Mean (SD)
C1	Conventional	45	0.460 (0.192)	0.443 (0.236)	11.156 (5.213)
C2	Conventional	42	0.512 (0.184)	0.567 (0.250)	12.143 (4.309)
S1	Simulation-supported	43	0.512 (0.195)	0.583 (0.265)	14.372 (4.981)
S2	Simulation-supported	44	0.498 (0.193)	0.636 (0.242)	16.682 (5.175)

Note: Class identifiers are anonymized. Pre-test and post-test ratios range from 0 to 1; scenario-based analytical performance ranges from 0 to 30. The class-specific summaries are descriptive and are not used as independent cluster-level inferential estimates.

3.2.3. Learning-Experience Profiles

Before the learning-experience profiles were compared, the intended factor structure was evaluated separately for the two questionnaire versions. The data were suitable for factor analysis in both configurations: the KMO values were 0.914 for the conventional version and 0.933 for the simulation-supported version, and Bartlett’s tests of sphericity were significant in both cases ($p<0.001$). Parallel analysis and the Kaiser criterion both supported retention of four factors in each configuration.

The construct validity and reliability results for both questionnaire versions are summarized in Table 6. The rotated solutions were consistent with the intended PLS, CE, PRI, and LSBI structure. Primary loadings ranged from 0.681 to 0.791 in the conventional configuration and from 0.693 to 0.794 in the simulation-supported configuration. The largest absolute cross-loadings were 0.103 and 0.109, respectively, and all 20 items loaded most strongly on their intended factors. AVE values ranged from 0.522 to 0.550 in the conventional configuration and from 0.546 to 0.559 in the simulation-supported configuration, while CR values ranged from 0.845 to 0.859 and from 0.857 to 0.864, respectively. In both versions, the square root of each construct’s AVE exceeded its correlations with the other factors, supporting the empirical distinction among the four intended dimensions. The complete rotated factor-loading matrices are provided in Appendix B.

Table 6. Construct validity and reliability of the learning-experience questionnaire.

Configuration	Construct	$\mathit{\alpha }$	Loading Range	AVE	CR
Conventional	PLS	0.735	0.684–0.779	0.525	0.847
	CE	0.778	0.687–0.791	0.546	0.857
	PRI	0.691	0.681–0.762	0.522	0.845
	LSBI	0.737	0.696–0.783	0.550	0.859
Simulation-supported	PLS	0.739	0.693–0.787	0.556	0.862
	CE	0.746	0.699–0.782	0.546	0.857
	PRI	0.762	0.708–0.794	0.559	0.864
	LSBI	0.743	0.704–0.785	0.555	0.862

Participants in the simulation-supported configuration reported numerically higher scores on all four learning-experience dimensions (Table 7). The largest standardized profile contrast was observed for PRI ($d=1.280$), followed by LSBI ($d=1.120$), PLS ($d=1.108$), and CE ($d=0.984$).

Table 7. Configuration-specific learning-experience profiles.

	Conventional (n = 87)		Simulation-Supported (n = 87)
Dimension	Mean (SD)	α	Mean (SD)	α	Mean Difference [95% CI]	Cohen’s d [95% CI]
Perceived learning support (PLS)	1.343 (0.566)	0.735	2.009 (0.636)	0.739	0.667 [0.497, 0.844]	1.108 [0.813, 1.464]
Cognitive engagement (CE)	1.338 (0.556)	0.778	1.894 (0.575)	0.746	0.556 [0.388, 0.726]	0.984 [0.679, 1.324]
Perceived realism and interactivity (PRI)	1.490 (0.470)	0.691	2.124 (0.520)	0.762	0.634 [0.487, 0.786]	1.280 [0.975, 1.624]
Learning satisfaction and behavioral intention (LSBI)	1.290 (0.570)	0.737	1.952 (0.611)	0.743	0.662 [0.485, 0.834]	1.120 [0.804, 1.487]

Note. Composite scores range from 0 to 4. $\alpha$ denotes Cronbach’s alpha calculated separately within each configuration. Differences are calculated as simulation-supported minus conventional; confidence intervals are based on 5000 configuration-stratified percentile bootstrap resamples. Because the questionnaire forms used configuration-specific wording and measurement invariance was not established, the contrasts are descriptive comparisons of learning-experience profiles rather than strictly equivalent scale scores. Their magnitude, particularly the large PRI contrast, may partly reflect wording-related measurement differences and should not be interpreted as direct evidence of differences in the underlying latent constructs or as a direct pedagogical effect.

The configuration-specific internal-consistency estimates ranged from 0.691 to 0.778. Seven of the eight coefficients exceeded 0.70, while the conventional PRI coefficient was marginally below this value ($\alpha =0.691$). The four composites therefore showed generally acceptable internal consistency within each configuration. Nevertheless, because the questionnaire forms used configuration-specific wording and measurement invariance was not established, the between-configuration contrasts were retained as descriptive learning-experience profiles.

Taken together, the configuration-level results showed a differentiated output pattern: the simulation-supported configuration had a smaller standardized contrast in formal knowledge, a larger contrast in scenario-based analytical performance, and a consistently higher participant-reported learning-experience profile. This pattern characterizes the performance of the coordinated instructional configuration across multiple output domains; it does not isolate the independent effect of the simulation platform or any other individual system component.

3.3. Within-System Relationships in the Simulation-Supported Configuration

The within-system analysis included all 87 participants in the simulation-supported configuration. Figure 4 presents the bivariate relationships among the six focal indicators. PLS and PRI were positively correlated with CE, with $r=0.318$ (95% CI [0.115, 0.496], $p=0.003$) and $r=0.275$ (95% CI [0.068, 0.459], $p=0.010$), respectively. CE and post-instruction knowledge were both positively correlated with scenario-based analytical performance, with $r=0.429$ (95% CI [0.240, 0.586], $p<0.001$) and $r=0.410$ (95% CI [0.219, 0.572], $p<0.001$), respectively.

Figure 4. Pearson correlations among the six focal indicators within the simulation-supported configuration ($n=87$). PLS, PRI, CE, and LSBI are 0–4 composite scores; post-instruction knowledge is a 0–1 score ratio; scenario-based analytical performance is scored from 0 to 30. The correlations are descriptive within-system associations and do not establish temporal or causal relationships.

PLS was also positively correlated with LSBI ($r=0.268$, 95% CI [0.061, 0.453], $p=0.012$), whereas the bivariate correlations of PRI, CE, and post-instruction knowledge with LSBI were smaller and their confidence intervals included zero. Scenario performance was positively correlated with LSBI ($r=0.325$, 95% CI [0.122, 0.501], $p=0.002$). Spearman sensitivity correlations retained the direction of all 15 Pearson correlations, and the largest absolute difference between the corresponding coefficients was 0.051.

The three prespecified regression models are summarized in Table 8. The largest variance inflation factor across the models was 1.33, providing no indication that coefficient interpretation was materially affected by overlap among the included predictors. No observations were removed from the primary analyses. Model-specific results and the corresponding sensitivity checks are reported below.

Table 8. Within-system HC3 regression models for the simulation-supported configuration.

Predictor	b	$\mathit{\beta }$	HC3 95% CI	p	Bootstrap 95% CI
Model 1: Cognitive engagement ($R^2=0.156$, adjusted $R^2=0.115$)
Intercept	0.908	–	[0.368, 10.448]	0.001	[0.406, 10.425]
PLS	0.202	0.223	[0.021, 0.382]	0.029	[0.026, 0.369]
PRI	0.220	0.199	[−0.024, 0.465]	0.077	[−0.020, 0.446]
Initial interest	0.075	0.144	[−0.034, 0.183]	0.173	[−0.028, 0.181]
Class S2	−0.027	−0.024	[−0.269, 0.214]	0.823	[−0.265, 0.202]
Model 2: Scenario-based analytical performance ($R^2=0.392$, adjusted $R^2=0.355$)
Intercept	30.515	–	[−0.073, 70.104]	0.055	[0.182, 60.818]
CE	20.763	0.306	[0.967, 40.559]	0.003	[10.124, 40.570]
Post-instruction knowledge	30.868	0.190	[−0.150, 70.887]	0.059	[0.095, 70.935]
Pre-instruction knowledge	40.008	0.150	[−20.175, 100.192]	0.201	[−10.995, 90.628]
Prior simulation/game exposure	20.584	0.250	[0.656, 40.511]	0.009	[0.741, 40.336]
Class S2	20.063	0.200	[0.217, 30.910]	0.029	[0.346, 30.791]
Model 3: Learning satisfaction and behavioral intention ($R^2=0.121$, adjusted $R^2=0.066$)
Intercept	10.089	–	[0.396, 10.782]	0.002	[0.445, 10.728]
PLS	0.242	0.252	[0.024, 0.460]	0.030	[0.029, 0.443]
PRI	0.087	0.074	[−0.168, 0.341]	0.501	[−0.148, 0.340]
CE	0.039	0.036	[−0.246, 0.323]	0.788	[−0.213, 0.318]
Post-instruction knowledge	0.003	0.001	[−0.541, 0.548]	0.991	[−0.519, 0.522]
Class S2	0.231	0.190	[−0.024, 0.486]	0.076	[−0.020, 0.463]

Note. $n=87$ in each model. b denotes the unstandardized coefficient and $\beta$ the standardized coefficient. HC3 confidence intervals and p-values provide the primary coefficient inference. Bootstrap intervals are percentile 95% confidence intervals for b, based on 5000 resamples conducted separately within the two simulation-supported classes. Class S2 is coded 1, with Class S1 as the reference. Prior simulation/game exposure is coded 1 for frequent prior exposure and 0 otherwise.

3.3.1. Conditional Associations with Cognitive Engagement

The cognitive-engagement model explained 15.6% of the observed variance (adjusted $R^2=0.115$). After initial interest and class membership were held constant, PLS had a positive conditional association with CE ($b=0.202$, $\beta =0.223$, HC3 95% CI [0.021, 0.382], $p=0.029$). The corresponding class-stratified bootstrap interval was also positive [0.026, 0.369].

The coefficient for PRI was positive but estimated less precisely ($b=0.220$, $\beta =0.199$, HC3 95% CI [−0.024, 0.465], $p=0.077$; bootstrap 95% CI [−0.020, 0.446]). Neither initial interest nor class membership showed a clearly estimated conditional association with CE. Single-case deletion checks preserved the positive directions of both the PLS and PRI coefficients. However, the PLS interval sometimes included zero after influential observations were excluded. Its relationship with CE was therefore interpreted as positive but sensitivity-qualified rather than uniformly stable across all diagnostic specifications.

3.3.2. Conditional Associations with Scenario-Based Analytical Performance

The scenario-performance model explained 39.2% of the observed variance (adjusted $R^2=0.355$). CE showed the clearest positive conditional association with scenario performance ($b=2.763$, $\beta =0.306$, HC3 95% CI [0.967, 4.559], $p=0.003$). Its class-stratified bootstrap interval was also positive [1.124, 4.570], and the coefficient remained positive and clearly estimated across the single-case deletion checks.

Post-instruction knowledge also had a positive coefficient, but its primary HC3 interval narrowly included zero ($b=3.868$, $\beta =0.190$, HC3 95% CI [−0.150, 7.887], $p=0.059$). The corresponding bootstrap interval was slightly above zero [0.095, 7.935]. Because the two inferential procedures differed only near the zero boundary, post-instruction knowledge was interpreted as showing a positive but imprecisely estimated conditional association with scenario performance. The coefficient for pre-instruction knowledge was also positive but imprecisely estimated.

Frequent prior exposure to driving-related games or simulation software ($b=2.584$, $\beta =0.250$, HC3 95% CI [0.656, 4.511], $p=0.009$) and membership in simulation-supported Class 2 ($b=2.063$, $\beta =0.200$, HC3 95% CI [0.217, 3.910], $p=0.029$) were associated with higher scenario scores after the other model variables were held constant. These results indicate that learner experience and classroom context contributed additional information about scenario performance beyond the focal post-instruction indicators. Residual inspection did not reveal a marked systematic pattern, although several observations had relatively large scenario-model residuals; these observations were retained in the primary analysis.

3.3.3. Conditional Associations with Satisfaction and Behavioral Intention

The LSBI model explained 12.1% of the observed variance (adjusted $R^2=0.066$). PLS had a positive conditional association with LSBI ($b=0.242$, $\beta =0.252$, HC3 95% CI [0.024, 0.460], $p=0.030$), and its class-stratified bootstrap interval was also positive [0.029, 0.443]. The positive direction was retained in the single-case deletion checks, although the confidence interval sometimes included zero after influential observations were excluded. This association was therefore interpreted as positive but sensitivity-qualified.

By contrast, the conditional coefficients for PRI ($b=0.087$, $\beta =0.074$), CE ($b=0.039$, $\beta =0.036$), and post-instruction knowledge ($b=0.003$, $\beta =0.001$) were small or imprecisely estimated, with all corresponding HC3 and bootstrap intervals including zero. The Class 2 coefficient was positive, but its interval also included zero. Within the specified model, PLS therefore provided the clearest positive information about LSBI, whereas CE and formal knowledge did not show clearly estimated independent relationships with this output.

Across the three models, CE showed the most consistent conditional relationship with scenario-based analytical performance. Post-instruction knowledge provided additional positive but less precise information about that output, while the PLS relationships with CE and LSBI were positive but somewhat sensitive to influential observations. These findings describe relationships among observed indicators within the implemented simulation-supported configuration and do not establish causal pathways among system components.

3.4. Scenario- and Question-Level Diagnostics for Instructional Redesign

The diagnostic analysis used the full sample ($N=174$) to examine how the configuration-level difference in the overall 30-point scenario score was distributed across individual scenarios and scenario-task questions. The overall standardized difference reported in Section 3.2 ($d=0.783$) was used as a descriptive reference rather than as a cutoff for classifying particular outputs.

3.4.1. Scenario-Level Diagnostic Pattern

As shown in Table 9, the simulation-supported configuration had higher mean scores in all three scenarios, although the magnitudes of the differences varied. Scenario 3 produced the largest scenario-specific difference ($d=0.723$, 95% CI [0.422, 1.049], Holm-adjusted $p<0.001$), followed by Scenario 1 ($d=0.688$, 95% CI [0.394, 1.018], Holm-adjusted $p<0.001$). Scenario 2 showed a smaller positive difference ($d=0.316$, 95% CI [0.030, 0.618], Holm-adjusted $p=0.039$).

Table 9. Scenario-level diagnostic comparisons by instructional configuration.

Scenario	Conventional (n = 87) Mean (SD)	Simulation-Supported (n = 87) Mean (SD)	Mean Difference [95% CI]	Cohen’s d [95% CI]	Holm p
Scenario 1: Ramp-merging conflict identification	3.655 (2.156)	5.207 (2.353)	1.552 [0.885, 2.230]	0.688 [0.394, 1.018]	<0.001
Scenario 2: Multi-vehicle interaction analysis	4.379 (2.092)	5.080 (2.344)	0.701 [0.069, 1.356]	0.316 [0.030, 0.618]	0.039
Scenario 3: Knowledge transfer and strategy judgment	3.598 (2.259)	5.253 (2.319)	1.655 [0.966, 2.322]	0.723 [0.422, 1.049]	<0.001

Note. Each scenario has a maximum score of 10. Differences are calculated as simulation-supported minus conventional. Confidence intervals are based on 5000 percentile bootstrap resamples conducted separately within each instructional configuration. Holm-adjusted p-values are based on two-sided Welch tests and were calculated within the family of three scenario-level comparisons.

The simulation-supported mean for Scenario 2 was 5.080, close to the corresponding means for Scenario 1 (5.207) and Scenario 3 (5.253). Its smaller difference partly reflected the higher conventional-configuration mean for Scenario 2 (4.379) than for Scenario 1 (3.655) or Scenario 3 (3.598). Scenario 2 therefore showed the weakest comparative contrast, but its absolute simulation-supported performance was similar to that observed for Scenarios 1 and 3. The smaller between-configuration difference was therefore not treated by itself as evidence of a scenario-level performance weakness or as sufficient grounds for redesign priority.

3.4.2. Question-Level Diagnostic Pattern and Candidate Redesign Priorities

The question-level results showed a similarly heterogeneous pattern (Table 10 and Figure 5). Seven of the 12 question-level comparisons retained Holm-adjusted p-values below 0.05. The largest standardized differences were observed for transferred-mechanism recognition and explanation in Scenario 3 ($d=0.704$), conflict-type identification in Scenario 1 ($d=0.635$), instructional-focus identification in Scenario 3 ($d=0.621$), and improvement-measure proposal in Scenario 1 ($d=0.609$). Contributing-factor analysis in Scenario 1 ($d=0.572$), conflict-source comparison in Scenario 3 ($d=0.529$), and conflict-risk indicator identification in Scenario 3 ($d=0.423$) also retained Holm-adjusted p-values below 0.05.

Table 10. Ranked question-level diagnostic comparisons.

Scenario-Task Question (Maximum)	Conventional Mean/Max.	Simulation- Supported Mean/Max.	Proportion Difference [95% CI]	Cohen’s d [95% CI]	Holm p
S3–1 Transferred-mechanism recognition and explanation (3)	0.364	0.556	0.192 [0.107, 0.268]	0.704 [0.395, 1.032]	<0.001
S1–1 Conflict-type identification (2)	0.362	0.563	0.201 [0.109, 0.287]	0.635 [0.345, 0.947]	<0.001
S3–4 Instructional-focus identification (2)	0.333	0.529	0.195 [0.103, 0.287]	0.621 [0.324, 0.949]	<0.001
S1–4 Improvement-measure proposal (2)	0.322	0.517	0.195 [0.103, 0.287]	0.609 [0.317, 0.929]	<0.001
S1–3 Contributing-factor analysis (3)	0.368	0.517	0.149 [0.073, 0.226]	0.572 [0.279, 0.891]	0.002
S3–3 Conflict-source comparison (2)	0.362	0.534	0.172 [0.075, 0.270]	0.529 [0.225, 0.864]	0.004
S3–2 Conflict-risk indicator identification (3)	0.372	0.487	0.115 [0.031, 0.192]	0.423 [0.117, 0.722]	0.035
S1–2 Critical-location identification and explanation (3)	0.395	0.498	0.103 [0.015, 0.192]	0.348 [0.050, 0.671]	0.115
S2–3 Traffic-density effect analysis (3)	0.433	0.529	0.096 [0.011, 0.184]	0.334 [0.039, 0.653]	0.116
S2–4 Critical-factor judgment (2)	0.448	0.546	0.098 [0.006, 0.190]	0.303 [0.018, 0.613]	0.141
S2–2 Interacting-behavior identification (3)	0.410	0.460	0.050 [−0.034, 0.134]	0.173 [−0.120, 0.479]	0.509
S2–1 Complexity recognition (2)	0.477	0.511	0.034 [−0.069, 0.138]	0.098 [−0.190, 0.394]	0.520

Note. Questions are ranked by Cohen’s d. Group values are mean question scores divided by the corresponding maximum score. Proportion differences are calculated as simulation-supported minus conventional. Confidence intervals for the proportion differences and Cohen’s d are based on 5000 configuration-stratified percentile bootstrap resamples. Holm-adjusted p-values are based on two-sided Welch tests and were calculated within the family of 12 question-level comparisons. The ranking is used only for relative diagnosis.

Figure 5. Ranked question-level standardized differences in scenario-based analytical performance. Points represent Cohen’s d, and horizontal bars show 95% percentile bootstrap confidence intervals. The dashed vertical line indicates the overall scenario-score difference ($d=0.783$) as a descriptive reference. Positive values favor the simulation-supported configuration. The effect-size ranking represents the comparative dimension of the diagnostic analysis; redesign priorities were determined jointly with the normalized absolute-performance levels reported in Table 10.

The two smallest between-configuration differences occurred within Scenario 2: complexity recognition (S2–1; $d=0.098$, 95% CI [−0.190, 0.394], Holm-adjusted $p=0.520$) and interacting-behavior identification (S2–2; $d=0.173$, 95% CI [−0.120, 0.479], Holm-adjusted $p=0.509$). However, their absolute performance levels led to different diagnostic interpretations. S2–2 had a simulation-supported mean proportion of 0.460, the lowest among the 12 task questions, and therefore combined relatively low absolute performance with a small comparative difference. It was consequently identified as the clearest candidate redesign priority. By contrast, the simulation-supported proportion for S2–1 was 0.511, which was not unusually low relative to the other questions. Its small effect size therefore indicated limited between-configuration separation rather than a confirmed performance weakness.

The joint diagnostic also identified two different secondary patterns. Critical-location identification and explanation in Scenario 1 (S1–2; simulation-supported proportion = 0.498, $d=0.348$) and conflict-risk indicator identification in Scenario 3 (S3–2; proportion = 0.487, $d=0.423$) showed comparatively low absolute performance despite larger positive between-configuration differences. These outputs therefore represent remaining reasoning challenges for both instructional configurations rather than evidence that simulation-supported instruction provided little comparative benefit.

Accordingly, the primary redesign emphasis is placed on helping students trace the inward and outward movements of multiple vehicles and connect simultaneous lane changes, path competition, and limited gaps to conflict development. Candidate modifications include more explicit trajectory highlighting, structured cue prompts, pause-and-replay comparisons, and debriefing questions focused on the sequence of interacting vehicle behaviors. Complexity recognition in Scenario 2 remains useful as a comparative diagnostic because of its small between-configuration difference, while critical-location explanation in Scenario 1 and conflict-risk indicator identification in Scenario 3 provide secondary targets for strengthening absolute task performance.

The configuration-level pattern is consistent with evidence that simulation can support knowledge and complex skills while effects vary across instructional designs and forms of learner support [15]. The larger contrast in scenario-based performance than in formal knowledge also accords with process- and outcome-based findings that applied performance and conventional knowledge measures need not change to the same extent [31]. The within-system results further align with models that place cognitive engagement between immersive features and learning outcomes, although the present cross-sectional regressions did not test mediation [32]. The question-level heterogeneity adds a diagnostic perspective by locating reasoning demands that may benefit from targeted redesign.

3.5. Implications for Engineering Learning Systems

The findings yield five implications for the design and evaluation of simulation-supported engineering learning systems.

• Evaluate the instructional configuration rather than the technical platform alone. The research-team-developed platform supplied the technical resources, but the observed differences characterize the coordinated instructional configuration comprising dynamic scenarios, instructor guidance, learner operation, structured tasks, discussion, and debriefing. The present design cannot isolate the contribution of CARLA–SUMO or any individual platform function. This interpretation is consistent with the socio-technical view that system outcomes arise from the organization of technical and social elements rather than from technology in isolation [25,26].
• Convert technical affordances into learning support through instructional organization. PLS was positively associated with CE and LSBI, while CE provided the clearest positive information about scenario-based performance. PRI showed a less precisely estimated conditional relationship with CE. Realism and interactivity should therefore be connected to attention-guiding prompts, structured analytical tasks, and concept-focused debriefing rather than treated as self-sufficient instructional properties. This emphasizes alignment among system capabilities, learning activities, and assessment requirements [27].
• Use hierarchical diagnostics to support targeted rather than system-wide redesign. The overall scenario-score difference was comparatively large ($d=0.783$), but the question-level diagnostic pattern was heterogeneous. Interacting-behavior identification in Scenario 2 combined the lowest absolute performance with a small between-configuration difference and therefore emerged as the clearest candidate for targeted redesign. Complexity recognition in the same scenario showed a small comparative difference without unusually low absolute performance, while critical-location explanation in Scenario 1 and conflict-risk indicator identification in Scenario 3 showed comparatively low absolute performance despite larger positive configuration differences. This pattern supports targeted rather than system-wide redesign.
• Treat learner experience and classroom context as system inputs. Prior simulation-related exposure and class membership provided additional information about scenario performance. Although the class-specific descriptive pattern favored both simulation-supported classes, the remaining contextual variation suggests that implementation should include standardized orientation, equivalent hands-on opportunities, common instructor protocols, and documentation of implementation fidelity. Introductory practice or adaptive scaffolding may also be appropriate for learners with limited prior simulation experience.
• Extend development and deployment into an empirically tested redesign cycle. The present work covered platform development, classroom deployment, and immediate multi-output evaluation. Knowledge tests, scenario-based tasks, and learning-experience indicators generated candidate redesign priorities, but the feedback loop was not empirically closed. Future iterations should implement the proposed modifications, specify the outputs expected to improve, and evaluate whether targeted gains occur without weakening other outputs. Process logs, replay records, and delayed assessments could extend evaluation to learning processes, retention, and transfer.

Together, these implications provide a transferable development-to-evaluation structure based on explicit system boundaries, coordinated technical and instructional elements, learner and contextual inputs, heterogeneous outputs, and assessment-informed redesign. The contribution is therefore an approach to developing, deploying, and evaluating simulation-supported engineering learning systems rather than a platform-specific claim about CARLA–SUMO [4,5].

3.6. Limitations

Several limitations should be acknowledged. First, instructional configuration was implemented at the class level using only four intact classes, with two classes per configuration. This design leaves potential class-level confounding and limits generalizability. Because the number of clusters was too small for dependable cluster-robust or multilevel variance estimation, the participant-level confidence intervals and p-values may not reflect the full uncertainty associated with class-level assignment. The class-specific summaries therefore serve only as a descriptive sensitivity check and do not eliminate this limitation. Second, the pre-test and post-test covered overlapping core content but differed in length, item composition, and cognitive demand and were not psychometrically parallel forms. Although the test blueprint documents their content coverage, the score-ratio change should not be interpreted as a scale-equated measure of learning gain. Baseline-adjusted post-test performance was therefore used as the primary formal-knowledge outcome, with score-ratio change retained only as supplementary information. Third, the study assessed only immediate outcomes in ramp-merging and weaving contexts, without examining long-term retention or professional transfer. Fourth, although separate construct-validity analyses supported the intended four-dimensional structure within both questionnaire versions, measurement invariance was not established across the configuration-specific forms. Their between-configuration differences therefore remain descriptive profile contrasts rather than comparisons of strictly equivalent latent measures. The cross-sectional within-system analysis ($n=87$) also constrains causal interpretation. Finally, neither the platform’s independent software-engineering performance nor the effectiveness of the proposed redesign priorities was evaluated.

4. Conclusions

This study placed a research-team-developed CARLA–SUMO platform within a bounded socio-technical instructional system and evaluated its classroom deployment rather than treating the software as an isolated technical tool. Compared with the conventional configuration, the simulation-supported configuration showed a positive standardized difference in post-instruction formal knowledge ($d=0.425$) and a larger difference in scenario-based analytical performance ($d=0.783$). Learning-experience profiles were also descriptively more positive, although these comparisons remain qualified by the configuration-specific questionnaire wording and the absence of established measurement invariance.

Within the simulation-supported configuration, PLS was positively associated with CE and LSBI, while CE provided the clearest positive information about scenario-based performance. The association between post-instruction knowledge and scenario performance was positive but imprecisely estimated. These relationships describe selected within-system associations and should not be interpreted as confirmed causal pathways.

The joint diagnostic analysis identified interacting-behavior identification in Scenario 2 as the clearest candidate for targeted redesign because it combined relatively low absolute performance with a small between-configuration difference. Complexity recognition in the same scenario showed limited between-configuration separation without unusually low absolute performance, while critical-location explanation in Scenario 1 and conflict-risk indicator identification in Scenario 3 represented secondary targets based on comparatively low absolute performance. Overall, the study provides a three-level evaluation approach linking configuration-level outputs, within-system relationships, and assessment-informed redesign. Future research should implement and evaluate the proposed modifications and examine retention and transfer across additional courses and institutions.

Appendix A.1. Instructional Cases Used in Both Configurations

The six instructional cases were matched one-to-one across the two configurations. The conventional configuration represented each case through diagrams, static traffic information, and instructor explanation, whereas the simulation-supported configuration represented the same situation dynamically and allowed every student to operate it. Table A1 records the focal traffic conditions and learning objective of each case. The platform also allowed viewpoint, weather, simulation time, vehicle type, and traffic density to be configured; these interface controls were not evaluated as separate experimental factors.

Table A1. Instructional focus and principal conditions of the six matched teaching cases.

Case	Traffic Situation	Principal Conditions Emphasized	Learning Objective
C1	Acceptable-gap selection during ramp entry	Entering vehicle, mainline stream, available gap size and duration, and traffic density	Identify acceptable gaps and explain how gap availability constrains merging opportunities.
C2	Speed coordination before merging	Merging-vehicle acceleration and the speed difference between the merging and mainline vehicles	Explain how speed matching affects gap use and the stability of the merging process.
C3	Late or forced merging near the end of the acceleration lane	Remaining acceleration-lane distance, delayed gap selection, forced entry, and the following mainline vehicle’s deceleration response	Trace the sequence of conflict development and identify the critical location and contributing factors.
C4	Simultaneous inward and outward movements in a weaving section	Entry vehicle moving inward, mainline vehicle moving toward the exit, simultaneous lane changes, and crossing paths	Identify the interacting vehicles and explain path competition in weaving.
C5	Dense weaving with limited lane-changing space	Higher traffic density, reduced gaps, spatial compression, and shortened decision time	Explain how density and limited operating space intensify multi-vehicle interaction.
C6	Closely spaced entrance–exit interaction	Short entrance–exit spacing, simultaneous entry and exit demand, uneven speeds, and late lane changing	Transfer ramp-merging and weaving concepts to a new situation and evaluate conflict-reduction priorities.

Table A1. Instructional focus and principal conditions of the six matched teaching cases.

Case	Traffic Situation	Principal Conditions Emphasized	Learning Objective
C1	Acceptable-gap selection during ramp entry	Entering vehicle, mainline stream, available gap size and duration, and traffic density	Identify acceptable gaps and explain how gap availability constrains merging opportunities.
C2	Speed coordination before merging	Merging-vehicle acceleration and the speed difference between the merging and mainline vehicles	Explain how speed matching affects gap use and the stability of the merging process.
C3	Late or forced merging near the end of the acceleration lane	Remaining acceleration-lane distance, delayed gap selection, forced entry, and the following mainline vehicle’s deceleration response	Trace the sequence of conflict development and identify the critical location and contributing factors.
C4	Simultaneous inward and outward movements in a weaving section	Entry vehicle moving inward, mainline vehicle moving toward the exit, simultaneous lane changes, and crossing paths	Identify the interacting vehicles and explain path competition in weaving.
C5	Dense weaving with limited lane-changing space	Higher traffic density, reduced gaps, spatial compression, and shortened decision time	Explain how density and limited operating space intensify multi-vehicle interaction.
C6	Closely spaced entrance–exit interaction	Short entrance–exit spacing, simultaneous entry and exit demand, uneven speeds, and late lane changing	Transfer ramp-merging and weaving concepts to a new situation and evaluate conflict-reduction priorities.

Appendix A.2. Participant Background Information

Table A2. Participant background information form.

No.	Item	Response Options
1	Gender	Male; Female
2	Participant type	Undergraduate student; Graduate student
3	Do you hold a valid driver’s license?	Yes; No
4	Do you have actual driving experience?	None; 0–2 years; 2–5 years; More than 5 years
5	Do you frequently play driving-related games or use simulation software?	Yes; No
6	Have you taken any courses related to transportation engineering or traffic safety?	Yes; No
7	What is your initial interest in this course content?	Not interested at all; Slightly interested; Neutral; Interested; Very interested

Table A2. Participant background information form.

No.	Item	Response Options
1	Gender	Male; Female
2	Participant type	Undergraduate student; Graduate student
3	Do you hold a valid driver’s license?	Yes; No
4	Do you have actual driving experience?	None; 0–2 years; 2–5 years; More than 5 years
5	Do you frequently play driving-related games or use simulation software?	Yes; No
6	Have you taken any courses related to transportation engineering or traffic safety?	Yes; No
7	What is your initial interest in this course content?	Not interested at all; Slightly interested; Neutral; Interested; Very interested

Appendix A.3. Learning Experience Questionnaires

All questionnaire items used a five-point response scale and were coded from 0 to 4 for analysis: 0 = strongly disagree, 1 = disagree, 2 = neutral, 3 = agree, and 4 = strongly agree. Two configuration-specific questionnaire versions were used. The simulation-supported version was administered to participants in the simulation-supported configuration, whereas the conventional version was administered to participants in the conventional configuration. The two versions retained the same four intended dimensions and conceptually corresponding item positions, but their wording was adapted to the instructional configuration experienced by each group. Table A3 and Table A4 present the two complete English-language versions.

Table A3. Learning-experience questionnaire: simulation-supported instruction version.

Code	Item
Section A. Perceived Learning Support (PLS)
PLS1	The simulation-based teaching system helped me better understand the mechanisms of traffic conflict formation in ramp merging scenarios.
PLS2	The system improved my understanding of vehicle interactions in weaving sections.
PLS3	The simulation-based activities helped me connect theoretical knowledge with real-world traffic scenarios.
PLS4	Compared with traditional classroom teaching, the system made complex traffic processes easier to understand.
PLS5	The post-simulation debriefing helped me more clearly identify the causes of traffic conflicts.
Section B. Cognitive Engagement (CE)
CE1	Compared with traditional lectures, I was more focused during the simulation-based learning process.
CE2	The simulation tasks encouraged me to actively think about how traffic conflicts are formed.
CE3	I was more willing to participate in discussions and analysis during the simulation-based class.
CE4	The simulation environment increased my interest in learning traffic conflict-related content.
CE5	In the simulation scenarios, I was willing to invest more effort in understanding vehicle interactions.
Section C. Perceived Realism and Interactivity (PRI)
PRI1	The ramp merging and weaving scenarios in the simulation system were realistic and supported my learning.
PRI2	The dynamic interactions between vehicles in the scenarios helped me understand how traffic conflicts are formed.
PRI3	Compared with static diagrams or instructional videos, the simulation tasks provided a stronger sense of involvement.
PRI4	The system allowed me to observe traffic processes that are difficult to understand in traditional teaching.
PRI5	The interactive experience made the learning process more intuitive.
Section D. Learning Satisfaction and Behavioral Intention (LSBI)
LSBI1	I am satisfied with the use of the simulation-based teaching system in this course.
LSBI2	I believe this teaching approach has practical value for transportation engineering courses.
LSBI3	I would like to have similar simulation-based learning activities in other transportation-related courses.
LSBI4	The system improved my overall learning experience in this topic.
LSBI5	I would recommend this teaching approach to other students.

Table A3. Learning-experience questionnaire: simulation-supported instruction version.

Code	Item
Section A. Perceived Learning Support (PLS)
PLS1	The simulation-based teaching system helped me better understand the mechanisms of traffic conflict formation in ramp merging scenarios.
PLS2	The system improved my understanding of vehicle interactions in weaving sections.
PLS3	The simulation-based activities helped me connect theoretical knowledge with real-world traffic scenarios.
PLS4	Compared with traditional classroom teaching, the system made complex traffic processes easier to understand.
PLS5	The post-simulation debriefing helped me more clearly identify the causes of traffic conflicts.
Section B. Cognitive Engagement (CE)
CE1	Compared with traditional lectures, I was more focused during the simulation-based learning process.
CE2	The simulation tasks encouraged me to actively think about how traffic conflicts are formed.
CE3	I was more willing to participate in discussions and analysis during the simulation-based class.
CE4	The simulation environment increased my interest in learning traffic conflict-related content.
CE5	In the simulation scenarios, I was willing to invest more effort in understanding vehicle interactions.
Section C. Perceived Realism and Interactivity (PRI)
PRI1	The ramp merging and weaving scenarios in the simulation system were realistic and supported my learning.
PRI2	The dynamic interactions between vehicles in the scenarios helped me understand how traffic conflicts are formed.
PRI3	Compared with static diagrams or instructional videos, the simulation tasks provided a stronger sense of involvement.
PRI4	The system allowed me to observe traffic processes that are difficult to understand in traditional teaching.
PRI5	The interactive experience made the learning process more intuitive.
Section D. Learning Satisfaction and Behavioral Intention (LSBI)
LSBI1	I am satisfied with the use of the simulation-based teaching system in this course.
LSBI2	I believe this teaching approach has practical value for transportation engineering courses.
LSBI3	I would like to have similar simulation-based learning activities in other transportation-related courses.
LSBI4	The system improved my overall learning experience in this topic.
LSBI5	I would recommend this teaching approach to other students.

Table A4. Learning-experience questionnaire: conventional instruction version.

Code	Item
Section A. Perceived Learning Support (PLS)
PLS1	The classroom instruction helped me better understand the mechanisms of traffic conflict formation in ramp merging scenarios.
PLS2	The classroom instruction improved my understanding of vehicle interactions in weaving sections.
PLS3	The classroom case analysis and teaching materials helped me connect theoretical knowledge with real-world traffic scenarios.
PLS4	The classroom instruction made complex traffic processes easier to understand.
PLS5	The post-instruction review and classroom discussion helped me more clearly identify the causes of traffic conflicts.
Section B. Cognitive Engagement (CE)
CE1	I was able to maintain a high level of focus during the classroom learning process.
CE2	The classroom case-analysis tasks encouraged me to actively think about how traffic conflicts are formed.
CE3	I was more willing to participate in discussions and analysis during the classroom session.
CE4	The classroom teaching content increased my interest in learning traffic conflict-related topics.
CE5	During classroom case analysis, I was willing to invest more effort in understanding vehicle interactions.
Section C. Perceived Realism and Interactivity (PRI)
PRI1	The ramp merging and weaving cases used in the classroom had clear contextual representation and supported my learning.
PRI2	The classroom presentation of dynamic vehicle interactions helped me understand how traffic conflicts are formed.
PRI3	The classroom diagrams, cases, and discussion activities provided a stronger sense of involvement in traffic scenarios.
PRI4	The classroom instruction helped me observe and understand traffic processes that are difficult to grasp through text-based explanation alone.
PRI5	The classroom case analysis and discussion made the learning process more intuitive.
Section D. Learning Satisfaction and Behavioral Intention (LSBI)
LSBI1	I am satisfied with the classroom teaching approach used in this course.
LSBI2	I believe this teaching approach has practical value for transportation engineering courses.
LSBI3	I would like to have similar classroom case-analysis activities in other transportation-related courses.
LSBI4	The classroom instruction improved my overall learning experience in this topic.
LSBI5	I would recommend this teaching approach to other students.

Table A4. Learning-experience questionnaire: conventional instruction version.

Code	Item
Section A. Perceived Learning Support (PLS)
PLS1	The classroom instruction helped me better understand the mechanisms of traffic conflict formation in ramp merging scenarios.
PLS2	The classroom instruction improved my understanding of vehicle interactions in weaving sections.
PLS3	The classroom case analysis and teaching materials helped me connect theoretical knowledge with real-world traffic scenarios.
PLS4	The classroom instruction made complex traffic processes easier to understand.
PLS5	The post-instruction review and classroom discussion helped me more clearly identify the causes of traffic conflicts.
Section B. Cognitive Engagement (CE)
CE1	I was able to maintain a high level of focus during the classroom learning process.
CE2	The classroom case-analysis tasks encouraged me to actively think about how traffic conflicts are formed.
CE3	I was more willing to participate in discussions and analysis during the classroom session.
CE4	The classroom teaching content increased my interest in learning traffic conflict-related topics.
CE5	During classroom case analysis, I was willing to invest more effort in understanding vehicle interactions.
Section C. Perceived Realism and Interactivity (PRI)
PRI1	The ramp merging and weaving cases used in the classroom had clear contextual representation and supported my learning.
PRI2	The classroom presentation of dynamic vehicle interactions helped me understand how traffic conflicts are formed.
PRI3	The classroom diagrams, cases, and discussion activities provided a stronger sense of involvement in traffic scenarios.
PRI4	The classroom instruction helped me observe and understand traffic processes that are difficult to grasp through text-based explanation alone.
PRI5	The classroom case analysis and discussion made the learning process more intuitive.
Section D. Learning Satisfaction and Behavioral Intention (LSBI)
LSBI1	I am satisfied with the classroom teaching approach used in this course.
LSBI2	I believe this teaching approach has practical value for transportation engineering courses.
LSBI3	I would like to have similar classroom case-analysis activities in other transportation-related courses.
LSBI4	The classroom instruction improved my overall learning experience in this topic.
LSBI5	I would recommend this teaching approach to other students.

Appendix A.4. Scenario-Based Analysis Tasks and Scoring Rubrics

The tasks were designed to evaluate conflict identification, multi-vehicle interaction analysis, knowledge transfer, and strategy judgment. Table A5 presents the complete English-language task prompts, and Table A6 presents the question-level scoring rubrics used by the raters.

Table A5. Scenario-based analysis tasks.

Code	Scenario Context and Task Question
Scenario 1: Conflict identification in ramp merging. Under medium-to-high traffic density conditions, a vehicle enters the mainline from a ramp. Mainline vehicles maintain relatively stable speeds, while the merging vehicle accelerates but fails to identify a suitable acceptable gap in time. At the end of the acceleration lane, the vehicle is forced to merge into the mainline, causing a sharp deceleration of the following mainline vehicle.
S1–1	Identify the primary type of traffic conflict in this scenario.
S1–2	Where is the most critical location for the formation of this conflict? Briefly explain your reasoning.
S1–3	What are the main factors that contribute to this conflict?
S1–4	Propose one improvement measure from either a driving behavior or traffic management perspective.
Scenario 2: Multi-vehicle interaction in a weaving section. In a weaving section, a vehicle entering from an on-ramp intends to move toward the inner lane, while a mainline vehicle plans to exit downstream. At the same time, traffic density is relatively high, and the available space for lane changing is limited. As a result, both vehicles must complete their lane changes within a short distance, leading to path crossing and mutual interference.
S2–1	Why is this scenario more complex than a standard ramp merging scenario?
S2–2	What vehicle interaction behaviors jointly contribute to the conflict risk in this scenario?
S2–3	How does high traffic density affect the development of conflicts in this scenario?
S2–4	From a traffic engineering perspective, what is the most critical influencing factor in this scenario?
Scenario 3: Knowledge transfer and strategy evaluation. Students have learned the mechanisms of traffic conflict formation in ramp merging and weaving sections. Consider a new scenario in which the distance between an upstream entrance and a downstream exit is short. Entering vehicles and exiting vehicles perform lane-changing maneuvers simultaneously within a limited road segment. In addition, traffic speeds are unevenly distributed, and some drivers tend to perform lane changes only when approaching their target position.
S3–1	Based on your prior knowledge, does this scenario exhibit conflict formation mechanisms similar to those in weaving sections? Explain your reasoning.
S3–2	What indicators suggest that the conflict risk may increase in this scenario?
S3–3	Compared with a standard ramp merging scenario, what are the key differences in the sources of conflict in this scenario?
S3–4	If this scenario were included in classroom teaching, what key instructional point should be emphasized?

Table A5. Scenario-based analysis tasks.

Code	Scenario Context and Task Question
Scenario 1: Conflict identification in ramp merging. Under medium-to-high traffic density conditions, a vehicle enters the mainline from a ramp. Mainline vehicles maintain relatively stable speeds, while the merging vehicle accelerates but fails to identify a suitable acceptable gap in time. At the end of the acceleration lane, the vehicle is forced to merge into the mainline, causing a sharp deceleration of the following mainline vehicle.
S1–1	Identify the primary type of traffic conflict in this scenario.
S1–2	Where is the most critical location for the formation of this conflict? Briefly explain your reasoning.
S1–3	What are the main factors that contribute to this conflict?
S1–4	Propose one improvement measure from either a driving behavior or traffic management perspective.
Scenario 2: Multi-vehicle interaction in a weaving section. In a weaving section, a vehicle entering from an on-ramp intends to move toward the inner lane, while a mainline vehicle plans to exit downstream. At the same time, traffic density is relatively high, and the available space for lane changing is limited. As a result, both vehicles must complete their lane changes within a short distance, leading to path crossing and mutual interference.
S2–1	Why is this scenario more complex than a standard ramp merging scenario?
S2–2	What vehicle interaction behaviors jointly contribute to the conflict risk in this scenario?
S2–3	How does high traffic density affect the development of conflicts in this scenario?
S2–4	From a traffic engineering perspective, what is the most critical influencing factor in this scenario?
Scenario 3: Knowledge transfer and strategy evaluation. Students have learned the mechanisms of traffic conflict formation in ramp merging and weaving sections. Consider a new scenario in which the distance between an upstream entrance and a downstream exit is short. Entering vehicles and exiting vehicles perform lane-changing maneuvers simultaneously within a limited road segment. In addition, traffic speeds are unevenly distributed, and some drivers tend to perform lane changes only when approaching their target position.
S3–1	Based on your prior knowledge, does this scenario exhibit conflict formation mechanisms similar to those in weaving sections? Explain your reasoning.
S3–2	What indicators suggest that the conflict risk may increase in this scenario?
S3–3	Compared with a standard ramp merging scenario, what are the key differences in the sources of conflict in this scenario?
S3–4	If this scenario were included in classroom teaching, what key instructional point should be emphasized?

Table A6. Question-level scoring rubric for the scenario-based analytical tasks.

Question	Max.	Full-Credit Content	Partial-Credit Rule
S1–1	2	Clearly identifies a ramp-merging or forced-merging conflict involving the merging vehicle and the following mainline vehicle.	1 point for a general lane-changing or merging-risk description without a clear conflict type; 0 for an incorrect or irrelevant response.
S1–2	3	Identifies the end of the acceleration lane or the forced-merging point as the critical location and links it to insufficient acceptable gaps.	2 points for the general merging area without precise localization; 1 for a vague location; 0 for an incorrect location.
S1–3	3	Identifies at least three relevant factors, such as insufficient acceptable gaps, poor speed coordination, high mainline density, late or forced merging, and limited evasive space.	2 points for two relevant factors; 1 for one factor or a vague explanation; 0 for an incorrect response.
S1–4	2	Proposes a clear and feasible behavioral or engineering measure that directly addresses the conflict mechanism, such as earlier gap selection, improved speed matching, ramp metering, or an extended acceleration lane.	1 point for a generally relevant but nonspecific suggestion; 0 for an irrelevant or unreasonable suggestion.
S2–1	2	Identifies at least two defining features: bidirectional lane-changing demand, path competition, spatial compression, or multi-vehicle interaction.	1 point for one feature or a vague description; 0 for an incorrect response.
S2–2	3	Identifies at least three interaction elements, including the entry vehicle moving inward, the mainline vehicle moving outward, simultaneous lane changes, path crossing, and limited gaps.	2 points for two elements; 1 for one element or a vague response; 0 for an incorrect response.
S2–3	3	Identifies at least three consequences of high density, such as reduced gaps, reduced operating tolerance, forced lane changes, conflict escalation, and shortened decision time.	2 points for two consequences; 1 for one consequence or a vague response; 0 for an incorrect response.
S2–4	2	Identifies a clear and interpretable core factor consistent with the scenario, such as insufficient weaving length, high density, high lane-changing demand, spatial compression, or speed-coordination problems.	1 point for a factor in the correct direction but expressed too generally; 0 for an incorrect or irrelevant factor.
S3–1	3	Correctly identifies a weaving-like mechanism and explains at least two relevant features, such as bidirectional lane changing, path competition, spatial compression, or intensive interaction within a short distance.	2 points for a correct similarity judgment supported by one mechanism; 1 for a conclusion without a mechanism or a vague explanation; 0 for an incorrect response.
S3–2	3	Identifies at least three risk indicators, such as short entrance–exit spacing, simultaneous entry and exit demand, uneven speeds, late lane changing, short decision time, or high density.	2 points for two indicators; 1 for one indicator or a vague response; 0 for an incorrect response.
S3–3	2	Identifies at least two fundamental differences from standard ramp merging, such as one-directional versus bidirectional interaction, stronger path competition, or more spatially concentrated maneuvers.	1 point for one meaningful difference or a vague comparison; 0 for an incorrect response.
S3–4	2	Identifies a clearly defined instructional focus directly related to the scenario mechanism, such as weaving-conflict formation, path competition, spatial compression, late lane changing, or speed coordination.	1 point for a broadly relevant but nonspecific teaching point; 0 for an irrelevant or unreasonable response.

Table A6. Question-level scoring rubric for the scenario-based analytical tasks.

Question	Max.	Full-Credit Content	Partial-Credit Rule
S1–1	2	Clearly identifies a ramp-merging or forced-merging conflict involving the merging vehicle and the following mainline vehicle.	1 point for a general lane-changing or merging-risk description without a clear conflict type; 0 for an incorrect or irrelevant response.
S1–2	3	Identifies the end of the acceleration lane or the forced-merging point as the critical location and links it to insufficient acceptable gaps.	2 points for the general merging area without precise localization; 1 for a vague location; 0 for an incorrect location.
S1–3	3	Identifies at least three relevant factors, such as insufficient acceptable gaps, poor speed coordination, high mainline density, late or forced merging, and limited evasive space.	2 points for two relevant factors; 1 for one factor or a vague explanation; 0 for an incorrect response.
S1–4	2	Proposes a clear and feasible behavioral or engineering measure that directly addresses the conflict mechanism, such as earlier gap selection, improved speed matching, ramp metering, or an extended acceleration lane.	1 point for a generally relevant but nonspecific suggestion; 0 for an irrelevant or unreasonable suggestion.
S2–1	2	Identifies at least two defining features: bidirectional lane-changing demand, path competition, spatial compression, or multi-vehicle interaction.	1 point for one feature or a vague description; 0 for an incorrect response.
S2–2	3	Identifies at least three interaction elements, including the entry vehicle moving inward, the mainline vehicle moving outward, simultaneous lane changes, path crossing, and limited gaps.	2 points for two elements; 1 for one element or a vague response; 0 for an incorrect response.
S2–3	3	Identifies at least three consequences of high density, such as reduced gaps, reduced operating tolerance, forced lane changes, conflict escalation, and shortened decision time.	2 points for two consequences; 1 for one consequence or a vague response; 0 for an incorrect response.
S2–4	2	Identifies a clear and interpretable core factor consistent with the scenario, such as insufficient weaving length, high density, high lane-changing demand, spatial compression, or speed-coordination problems.	1 point for a factor in the correct direction but expressed too generally; 0 for an incorrect or irrelevant factor.
S3–1	3	Correctly identifies a weaving-like mechanism and explains at least two relevant features, such as bidirectional lane changing, path competition, spatial compression, or intensive interaction within a short distance.	2 points for a correct similarity judgment supported by one mechanism; 1 for a conclusion without a mechanism or a vague explanation; 0 for an incorrect response.
S3–2	3	Identifies at least three risk indicators, such as short entrance–exit spacing, simultaneous entry and exit demand, uneven speeds, late lane changing, short decision time, or high density.	2 points for two indicators; 1 for one indicator or a vague response; 0 for an incorrect response.
S3–3	2	Identifies at least two fundamental differences from standard ramp merging, such as one-directional versus bidirectional interaction, stronger path competition, or more spatially concentrated maneuvers.	1 point for one meaningful difference or a vague comparison; 0 for an incorrect response.
S3–4	2	Identifies a clearly defined instructional focus directly related to the scenario mechanism, such as weaving-conflict formation, path competition, spatial compression, late lane changing, or speed coordination.	1 point for a broadly relevant but nonspecific teaching point; 0 for an irrelevant or unreasonable response.

Appendix A.5. Knowledge Pre-Test and Post-Test

The knowledge pre-test and post-test assessed students’ understanding of ramp merging, weaving sections, traffic-conflict mechanisms, and knowledge transfer. The pre-test was administered before instruction to measure baseline knowledge, whereas the post-test was administered after instruction to evaluate immediate learning outcomes. Table A8 and Table A10 present the complete English-language items. The corresponding answer keys and short-answer scoring criteria follow each table.

Table A7. Test blueprint for the knowledge pre-test and post-test.

Assessment	Items	Principal Content Focus	Principal Cognitive Demand	Points
Pre-test	P1, P4	Gap availability and merging opportunity	Identify and interpret conditions affecting gap selection and merging opportunities	2
	P5	Speed coordination in ramp merging	Identify an indicator of speed-matching problems	1
	P3, P6	Traffic-conflict formation in merging	Recognize conflict-triggering conditions and basic mechanism relationships	2
	P2, P7, P8	Weaving complexity and multi-vehicle interaction	Recognize and briefly explain multidirectional lane-changing demand, spatial constraints, and intensified interaction	5
Post-test	T1, T4, T13	Gap availability and speed coordination in ramp merging	Identify, integrate, and explain how gap and speed conditions jointly affect merging	5
	T2, T9, T12	Traffic-conflict formation and mechanism interpretation	Recognize conflict-producing conditions and distinguish conflict-location recognition from mechanism understanding	3
	T3, T5, T6, T10, T11, T14	Weaving, multi-vehicle interaction, path competition, and traffic density	Interpret and compare interacting mechanisms under constrained traffic conditions	8
	T8, T15	Transfer to closely spaced entrance–exit situations	Apply ramp-merging and weaving concepts to a related but different traffic situation	3
	T7	Instructional activity aligned with mechanism learning	Recognize an instructional activity intended to support analysis of dynamic vehicle interactions	1

Note: The blueprint describes the principal content and cognitive demand represented by each item. The pre-test and post-test share core traffic-engineering content but differ in length, item composition, and cognitive demand and are therefore not treated as psychometrically parallel forms. Post-test Item T7 was additionally examined through a sensitivity analysis because its wording explicitly referred to an instructional activity.

Table A7. Test blueprint for the knowledge pre-test and post-test.

Assessment	Items	Principal Content Focus	Principal Cognitive Demand	Points
Pre-test	P1, P4	Gap availability and merging opportunity	Identify and interpret conditions affecting gap selection and merging opportunities	2
	P5	Speed coordination in ramp merging	Identify an indicator of speed-matching problems	1
	P3, P6	Traffic-conflict formation in merging	Recognize conflict-triggering conditions and basic mechanism relationships	2
	P2, P7, P8	Weaving complexity and multi-vehicle interaction	Recognize and briefly explain multidirectional lane-changing demand, spatial constraints, and intensified interaction	5
Post-test	T1, T4, T13	Gap availability and speed coordination in ramp merging	Identify, integrate, and explain how gap and speed conditions jointly affect merging	5
	T2, T9, T12	Traffic-conflict formation and mechanism interpretation	Recognize conflict-producing conditions and distinguish conflict-location recognition from mechanism understanding	3
	T3, T5, T6, T10, T11, T14	Weaving, multi-vehicle interaction, path competition, and traffic density	Interpret and compare interacting mechanisms under constrained traffic conditions	8
	T8, T15	Transfer to closely spaced entrance–exit situations	Apply ramp-merging and weaving concepts to a related but different traffic situation	3
	T7	Instructional activity aligned with mechanism learning	Recognize an instructional activity intended to support analysis of dynamic vehicle interactions	1

Note: The blueprint describes the principal content and cognitive demand represented by each item. The pre-test and post-test share core traffic-engineering content but differ in length, item composition, and cognitive demand and are therefore not treated as psychometrically parallel forms. Post-test Item T7 was additionally examined through a sensitivity analysis because its wording explicitly referred to an instructional activity.

Table A8. Complete English-language knowledge pre-test (maximum score: 10 points).

Item	Question and Response Options
Multiple-choice questions (1 point each)
P1	In a freeway ramp-merging area, which of the following most directly reflects the interaction between merging vehicles and mainline vehicles?A. Number of mainline lanes; B. Acceptable gap between vehicles; C. Speed limit of the road segment; D. Length of the ramp.
P2	Compared with a standard ramp-merging section, the most distinctive feature of a weaving section is:A. Mainline vehicles always travel at lower speeds; B. Vehicles have bidirectional or multidirectional lane-changing demands within a short distance; C. Vehicles entering from the ramp are no longer affected by mainline traffic; D. There is typically no speed variation in weaving sections.
P3	In the formation of traffic conflicts, which of the following is most likely to be a direct triggering factor?A. Mainline vehicles maintain a stable speed; B. Drivers make sudden lane changes near their target position; C. The design speed of the road is relatively high; D. There are many traffic signs along the road segment.
P4	When the density of mainline traffic increases, what problem are merging vehicles more likely to encounter?A. Reduced available gaps, making it more difficult to determine a suitable merging opportunity; B. Greater ease in matching the speed of mainline traffic; C. Reduced impact of merging behavior on the mainline; D. Lane-changing demand typically disappears.
P5	Which of the following is most appropriate for describing whether speed-coordination problems exist in a merging area?A. Speed difference between mainline vehicles and merging vehicles; B. Variation in longitudinal road gradient; C. Signal timing at intersections; D. Lane-width variation.
True/false questions (1 point each)
P6	In a ramp-merging area, traffic conflicts will not occur as long as mainline vehicles do not change lanes. (True/False)
P7	Conflict risk in weaving sections is typically associated with multiple lane-changing demands within a short distance. (True/False)
Short-answer question (3 points)
P8	Briefly explain why weaving sections are generally more complex than standard ramp-merging sections.

Table A8. Complete English-language knowledge pre-test (maximum score: 10 points).

Item	Question and Response Options
Multiple-choice questions (1 point each)
P1	In a freeway ramp-merging area, which of the following most directly reflects the interaction between merging vehicles and mainline vehicles?A. Number of mainline lanes; B. Acceptable gap between vehicles; C. Speed limit of the road segment; D. Length of the ramp.
P2	Compared with a standard ramp-merging section, the most distinctive feature of a weaving section is:A. Mainline vehicles always travel at lower speeds; B. Vehicles have bidirectional or multidirectional lane-changing demands within a short distance; C. Vehicles entering from the ramp are no longer affected by mainline traffic; D. There is typically no speed variation in weaving sections.
P3	In the formation of traffic conflicts, which of the following is most likely to be a direct triggering factor?A. Mainline vehicles maintain a stable speed; B. Drivers make sudden lane changes near their target position; C. The design speed of the road is relatively high; D. There are many traffic signs along the road segment.
P4	When the density of mainline traffic increases, what problem are merging vehicles more likely to encounter?A. Reduced available gaps, making it more difficult to determine a suitable merging opportunity; B. Greater ease in matching the speed of mainline traffic; C. Reduced impact of merging behavior on the mainline; D. Lane-changing demand typically disappears.
P5	Which of the following is most appropriate for describing whether speed-coordination problems exist in a merging area?A. Speed difference between mainline vehicles and merging vehicles; B. Variation in longitudinal road gradient; C. Signal timing at intersections; D. Lane-width variation.
True/false questions (1 point each)
P6	In a ramp-merging area, traffic conflicts will not occur as long as mainline vehicles do not change lanes. (True/False)
P7	Conflict risk in weaving sections is typically associated with multiple lane-changing demands within a short distance. (True/False)
Short-answer question (3 points)
P8	Briefly explain why weaving sections are generally more complex than standard ramp-merging sections.

Pre-test objective-item answer key: P1–P7 = B, B, B, A, A, False, True.

Table A9. Scoring criteria for pre-test Item P8.

Score	Criterion
0	Irrelevant or incorrect response.
1	States only that weaving is more complex, without explaining why.
2	Identifies one or two relevant factors: multidirectional lane-changing demand, limited space or short operating distance, or stronger vehicle interaction.
3	Clearly explains the combined effects of multidirectional lane changing, spatial constraint, and intensified vehicle interaction.

Table A9. Scoring criteria for pre-test Item P8.

Score	Criterion
0	Irrelevant or incorrect response.
1	States only that weaving is more complex, without explaining why.
2	Identifies one or two relevant factors: multidirectional lane-changing demand, limited space or short operating distance, or stronger vehicle interaction.
3	Clearly explains the combined effects of multidirectional lane changing, spatial constraint, and intensified vehicle interaction.

Table A10. Complete English-language knowledge post-test (maximum score: 20 points).

Item	Question and Response Options
Multiple-choice questions (1 point each)
T1	In a ramp-merging area, which factor most directly affects whether a merging vehicle can successfully enter the mainline?A. Number of mainline lanes; B. Available gaps and their duration; C. Average speed of mainline traffic; D. Ramp design length.
T2	Which situation is most likely to cause a sharp-deceleration conflict in a ramp-merging area?A. The merging vehicle completes speed matching in advance and merges smoothly; B. The merging vehicle forces entry at the end of the acceleration lane; C. Mainline vehicles maintain large headways; D. Mainline traffic has a uniform speed distribution.
T3	One important reason why traffic conflicts are more prevalent in weaving sections than in ordinary road segments is that:A. Vehicles perform only longitudinal movements; B. There is no path competition; C. Multiple vehicles have interacting lane-changing demands within limited space; D. There is no speed variation.
T4	When analyzing a ramp-merging scenario, which condition most clearly indicates increased risk?A. Lower mainline traffic density; B. Greater speed differences combined with reduced acceptable gaps; C. No change in ramp entrance location; D. More lanes.
T5	In a high-density weaving section, which behavior is most likely to exacerbate traffic conflicts?A. Completing the desired lane change before entering the weaving section; B. Dynamically adjusting lane-change timing to surrounding traffic; C. Changing lanes only when approaching the target position; D. Maintaining the leading vehicle’s speed with fewer fluctuations.
T6	From a traffic-engineering perspective, the complexity of a weaving section mainly arises from:A. Differences in operating speed; B. Spatial path competition and crossing interactions among vehicles; C. Temporal fluctuations in traffic flow; D. Mixed vehicle types.
T7	If the instructional goal is to help students understand traffic-conflict formation mechanisms, which learning activity is most effective?A. Summarizing typical conflicts through case studies; B. Analyzing vehicle interactions using dynamic traffic scenarios; C. Reviewing definitions of traffic-conflict terms; D. Learning basic road-design parameters.
T8	Which option best demonstrates transfer of ramp-merging knowledge?A. Describing the definitions and characteristics of ramp merging; B. Explaining how conflicts vary under different traffic conditions; C. Applying merging-conflict analysis to closely spaced entrance–exit interactions; D. Reproducing examples presented in class.
True/false questions (1 point each)
T9	In a ramp-merging area, traffic conflicts will not occur as long as merging vehicles accelerate sufficiently. (True/False)
T10	Traffic conflicts in weaving sections are related not only to individual vehicle behavior but also to multi-vehicle path competition. (True/False)
T11	An increase in traffic density typically compresses the available operating space for lane changing and merging. (True/False)
T12	If a student can identify where a conflict occurs, the student has fully understood the mechanism of conflict formation. (True/False)
Short-answer questions (8 points in total)
T13	Explain why “acceptable gap” and “speed coordination” are two key concepts in ramp-merging areas. (3 points)
T14	Compare traffic-conflict formation mechanisms in standard ramp-merging sections and weaving Sections (3 points)
T15	In a road segment where an entrance and an exit are located close together, entering and exiting vehicles change lanes simultaneously within a short distance. Briefly analyze the main sources of traffic-conflict risk. (2 points)

Table A10. Complete English-language knowledge post-test (maximum score: 20 points).

Item	Question and Response Options
Multiple-choice questions (1 point each)
T1	In a ramp-merging area, which factor most directly affects whether a merging vehicle can successfully enter the mainline?A. Number of mainline lanes; B. Available gaps and their duration; C. Average speed of mainline traffic; D. Ramp design length.
T2	Which situation is most likely to cause a sharp-deceleration conflict in a ramp-merging area?A. The merging vehicle completes speed matching in advance and merges smoothly; B. The merging vehicle forces entry at the end of the acceleration lane; C. Mainline vehicles maintain large headways; D. Mainline traffic has a uniform speed distribution.
T3	One important reason why traffic conflicts are more prevalent in weaving sections than in ordinary road segments is that:A. Vehicles perform only longitudinal movements; B. There is no path competition; C. Multiple vehicles have interacting lane-changing demands within limited space; D. There is no speed variation.
T4	When analyzing a ramp-merging scenario, which condition most clearly indicates increased risk?A. Lower mainline traffic density; B. Greater speed differences combined with reduced acceptable gaps; C. No change in ramp entrance location; D. More lanes.
T5	In a high-density weaving section, which behavior is most likely to exacerbate traffic conflicts?A. Completing the desired lane change before entering the weaving section; B. Dynamically adjusting lane-change timing to surrounding traffic; C. Changing lanes only when approaching the target position; D. Maintaining the leading vehicle’s speed with fewer fluctuations.
T6	From a traffic-engineering perspective, the complexity of a weaving section mainly arises from:A. Differences in operating speed; B. Spatial path competition and crossing interactions among vehicles; C. Temporal fluctuations in traffic flow; D. Mixed vehicle types.
T7	If the instructional goal is to help students understand traffic-conflict formation mechanisms, which learning activity is most effective?A. Summarizing typical conflicts through case studies; B. Analyzing vehicle interactions using dynamic traffic scenarios; C. Reviewing definitions of traffic-conflict terms; D. Learning basic road-design parameters.
T8	Which option best demonstrates transfer of ramp-merging knowledge?A. Describing the definitions and characteristics of ramp merging; B. Explaining how conflicts vary under different traffic conditions; C. Applying merging-conflict analysis to closely spaced entrance–exit interactions; D. Reproducing examples presented in class.
True/false questions (1 point each)
T9	In a ramp-merging area, traffic conflicts will not occur as long as merging vehicles accelerate sufficiently. (True/False)
T10	Traffic conflicts in weaving sections are related not only to individual vehicle behavior but also to multi-vehicle path competition. (True/False)
T11	An increase in traffic density typically compresses the available operating space for lane changing and merging. (True/False)
T12	If a student can identify where a conflict occurs, the student has fully understood the mechanism of conflict formation. (True/False)
Short-answer questions (8 points in total)
T13	Explain why “acceptable gap” and “speed coordination” are two key concepts in ramp-merging areas. (3 points)
T14	Compare traffic-conflict formation mechanisms in standard ramp-merging sections and weaving Sections (3 points)
T15	In a road segment where an entrance and an exit are located close together, entering and exiting vehicles change lanes simultaneously within a short distance. Briefly analyze the main sources of traffic-conflict risk. (2 points)

Post-test objective-item answer key: T1–T12 = B, B, C, B, C, B, B, C, False, True, True, False.

Table A11. Scoring criteria for post-test short-answer items.

Item	Reference Content	Scoring Rule
T13	Acceptable gaps determine whether merging can be completed safely; speed coordination determines how well the merging vehicle matches mainline traffic; the two jointly influence conflict formation.	0: incorrect or blank; 1: mentions one concept; 2: mentions both with limited explanation; 3: clearly explains both concepts and their combined effect.
T14	Ramp merging mainly involves one-directional entry, whereas weaving involves bidirectional or multidirectional lane changing, spatial compression, stronger interaction, and path competition.	0: incorrect or blank; 1: only states that weaving is more complex; 2: identifies one or two differences; 3: provides a structured comparison covering interaction direction, spatial constraints, and path competition.
T15	Key risk sources include multiple lane changes within a short distance, overlapping target paths of entering and exiting vehicles, speed differences, limited space, and late lane changing.	0: incorrect or irrelevant; 1: identifies part of the risk sources; 2: identifies multiple key sources with basic reasoning.

Table A11. Scoring criteria for post-test short-answer items.

Item	Reference Content	Scoring Rule
T13	Acceptable gaps determine whether merging can be completed safely; speed coordination determines how well the merging vehicle matches mainline traffic; the two jointly influence conflict formation.	0: incorrect or blank; 1: mentions one concept; 2: mentions both with limited explanation; 3: clearly explains both concepts and their combined effect.
T14	Ramp merging mainly involves one-directional entry, whereas weaving involves bidirectional or multidirectional lane changing, spatial compression, stronger interaction, and path competition.	0: incorrect or blank; 1: only states that weaving is more complex; 2: identifies one or two differences; 3: provides a structured comparison covering interaction direction, spatial constraints, and path competition.
T15	Key risk sources include multiple lane changes within a short distance, overlapping target paths of entering and exiting vehicles, speed differences, limited space, and late lane changing.	0: incorrect or irrelevant; 1: identifies part of the risk sources; 2: identifies multiple key sources with basic reasoning.

Appendix A.6. Informed Consent Form

Study Title (as presented during data collection): Driving Simulation-Based Teaching Effectiveness Study in Transportation Engineering

Purpose of the Study: This study aims to evaluate the instructional effects of driving simulation-assisted teaching in a transportation engineering course, with particular attention to knowledge learning, scenario-based analytical ability, and learning experience.

Study Procedures: If you agree to participate, the study will collect data generated during the course learning process, including: (1) Basic background information; (2) Knowledge pre-test and post-test results; (3) Responses to scenario-based analysis tasks; (4) Learning experience questionnaire responses.

Voluntary Participation: Participation in this study is completely voluntary. You have the right to refuse to participate or withdraw from the study at any time. Refusal to participate or withdrawal from the study will not affect your course learning, course grades, teacher-student relationship, or any other rights.

Risks and Benefits: This study is a low-risk educational research project. It does not involve medical intervention, invasive procedures, or the collection of sensitive biological information. Participation in this study may help improve course teaching design and learning support.

Confidentiality: The research data will be used only for educational research and academic publication. All data will be anonymized during analysis and reporting. Your name, student ID, or any other personally identifiable information will not be disclosed. The research findings will be presented only in aggregated or anonymized form.

Consent to Publish: I agree that the researcher may use anonymized and aggregated research findings for academic journal articles, conference presentations, and other scholarly communications. I understand that no published content will contain information that can identify me personally.

Participant Statement: I confirm that I am 18 years of age or older and have the ability to make an independent decision about participation. I have understood the purpose of the study, the study procedures, the use of data, the confidentiality measures, and the voluntary nature of participation. I agree to participate in this study and agree that anonymized and aggregated data may be used for educational research and academic publication.

Participant Signature:____________

Date:_______/_______/_______

Principal Investigator:____________

Contact Information:____________