Parallel Multi-Level Simulation for Large-Scale Detailed Intelligent Transportation System Modeling

Abstract

Nowadays, the problems of traffic accidents, inefficiency, and congestion still affect transportation systems. Conventional solutions often do not resolve and can even exacerbate the problems. Intelligent transportation system (ITS) technology, including intelligent vehicles, could provide a solution for these problems. However, such technologies should be thoroughly verified and validated before their large-scale adoption. Computer simulation can be used for this task to avoid the expenses of real-world testing. Modern consumer hardware computers are not powerful enough to handle large-scale scenes with high detail. Therefore, a parallel simulation approach employing multiple computers, each processing a separate scene of limited size, is proposed. To define the requirements for a suitable simulation tool, the needs of ITS simulation and Digital Twin technology are discussed, and existing simulation environments suitable for ITS technology verification and validation are evaluated. Further, an architecture for a parallel and multi-level simulation environment for large-scale detailed ITS modeling is proposed. The proposed integrated simulation environment uses the nanoscopic CARLA and microscopic SUMO simulators to implement multi-level and parallel nanoscopic simulation by creating a large scene on the microscopic simulation level and combining the information from multiple parallelly executed nanoscopic scenes. Special handling for nanoscopic scene logic is proposed using a concept of Buffer Zones that allows traffic participants to perceive environmental information beyond the logical boundary of the scene they belong to. The proposed approaches are demonstrated in a series of experiments as a proof of concept and are integrated into the CAVISE simulation environment.

1. Introduction

The economy relies on transportation for meeting the needs of society members and various industries. However, as the population grows, so do traffic problems, such as traffic accidents, increased travel time, pollution, and energy consumption [1,2,3,4]. The development of information and communications technology (ICT) offers modern and radical ways to solve existing problems. ICT can help achieve a better understanding of transportation systems and provide an improved capability for decision-making and impacting the system state. This approach is encompassed in the development of Intelligent Transportation Systems (ITS). The concept of ITS can be applied for more efficient traffic management, relying on modern technologies [5]. Such systems can utilize Digital Twins (DT) to model real traffic, project possible developments into the future, analyze them, and predict and prevent harmful events through centralized optimal decision-making [6,7]. Connected vehicle (CV) and automated vehicle (AV) technologies are intended to become a major ITS part and promise solutions for many of the aforementioned problems [8]. However, the discussed technologies and their combinations are only emerging and require a comprehensive approach to verification and validation throughout the development process.

Computer simulation is widely used for transportation research to mitigate the expenses of real-world testing. Traffic simulators can be attributed to either one or several levels of scale—macroscopic, mesoscopic, microscopic, or nanoscopic (definitions for each level can be found in [9]). To study the network-level effects of new transportation technology and centralized traffic management systems, large-scale scenario simulation is necessary [10,11]. Existing research outlines approaches to large-scale simulation for microscopic models, proposing various combinations of parallel and distributed simulation employing spatial network decomposition as a sufficiently straightforward solution [10,12,13]. However, to test and validate the performance of complex large-scale ITSs involving AVs and CVs, highly detailed nanoscopic simulation environments are required for their accurate representation of vehicular perception and communication [14]. Existing approaches to nanoscopic large-scale simulation are limited to the combination of the nanoscopic CARLA (CAR Learning to Act) simulator [15] and the microscopic PTV Vissim simulator [16] proposed in [11]. However, the use of CARLA capabilities in this approach is limited, preventing vehicular perception modeling, and the use of PTV Vissim imposes additional restrictions as it is not an open-source tool. The full use of CARLA for nanoscopic simulation of large scenarios is currently not possible because modern computers, available to most researchers and system engineers, are not powerful enough to handle such highly detailed large-scale models of transportation systems. Additionally, CARLA itself has an internal limitation of 2 km x 2 km for map tile size that can be handled by a server for ego vehicle simulation. Considering that a large-scale transportation technology verification and validation scenario involves multiple vehicles with full behavior simulation, a different approach is warranted. A parallel approach to detailed ITS representation and simulation employing multiple computers can be taken, where each handles a separate spatially decomposed part of the system of a limited size. This way, the results of a highly detailed simulation of separate scenes can then be aggregated in an overarching large-scale simulation environment with lower detail. When implementing ITSs in the real world, the system where DT information about the transportation system is updated in real-time from distributed sources (e.g., remote sensors) can also benefit from an architecture where the real system can be emulated in a decentralized way (alleviating unnecessary communication delays). However, no instrument implementing such an approach is currently available [9,14,17]. Therefore, this paper is focused on synthesizing appropriate solutions and demonstrating their feasibility to aid in the creation of such a tool.

The use of DT technology is discussed in this article to help define the requirements for the ITS simulation environment under consideration. The existing simulation environments used for large-scale ITS verification and validation are evaluated, and their advantages, disadvantages, and notable design features are highlighted. Based on the defined requirements, a parallel and multi-level simulation environment architecture for ITS technology is proposed. It utilizes the SUMO (Simulation of Urban MObility) [18] simulator for microscopic simulation of a large-scale traffic scenario that combines information about vehicle movement from multiple scenes simulated using a nanoscopic CARLA simulator [15] that provides a 3D graphical environment for simulating ITS technologies involving sensors (e.g., cameras, LiDARs, radars).

Spatial decomposition at the microscopic simulation level requires that specific rules are used for vehicle transfer and synchronization between network partitions [12]. Border management approaches include imposing limitations on vehicle behavior at the partition border (e.g., setting a specific speed, disabling lane changes, etc.) or propagating traffic information from beyond the border and using it to replicate a car-following model [19]. However, rule-based approaches were shown to reduce simulation accuracy and are not feasible for more complex nanoscopic simulations involving vehicular perception (which requires visual information to be propagated). In [20], a simplified process for network partitioning is proposed—border edges are preserved in every partition and neighboring partitions behave in a “master-slave” fashion, controlling each other’s vehicle movement, depending on whether the border edge is an incoming edge or an outgoing edge. Although sufficient for microscopic simulation where vehicle behavior is relatively predictable, this approach cannot be directly extended to the nanoscopic level of simulation. Nanoscopic simulation considers vehicle behavior independent from traffic flow directions, ensuring capabilities for scenario diversity (e.g., a vehicle can stop and reverse). Therefore, spatial map decomposition on the nanoscopic level requires a synchronization approach that is able to propagate all available information about the upcoming road section at the partition border, but retains control of all vehicles present in the partition. A novel approach based on the concept of “Buffer Zones” is proposed to ensure that sensors in separately computed adjacent nanoscopic scenes (limited geographical areas of the overall scenario) can perceive objects beyond the horizon of the scene they belong to. A multi-level simulation environment based on CARLA and SUMO is implemented, and the possibility of using SUMO to analyze traffic parameters (road section throughput, fuel or energy consumption, emissions, etc.) based on vehicle trajectories acquired from CARLA is demonstrated. A prototype of the parallel simulation module is developed and provided as a proof of concept. The developed approaches are intended to be integrated into the CAVISE (Connected and Automated Vehicle Integrated Simulation Environment) project [17,21,22].

The remainder of this paper is structured as follows. In Section 2, an analysis of DT and additional information about ITS technologies is provided. In Section 3, existing ITS simulation tools’ architectures and operation principles are analyzed. Section 4 describes the proposed parallel and multi-level approaches and the simulation environment architecture. Section 5 provides prototype implementations of the proposed approaches to prove their feasibility. In Section 6, the results and plans for future development of the software for the CAVISE simulation environment are discussed.

2. Digital Twins in Intelligent Transportation Systems

The ITS concept aims to improve resource management, reduce operational costs, and enhance the quality of services in transportation systems [5,23]. ITSs can be described as consisting of three parts that are necessary to aid in optimizing system-wide traffic or individual actions: the local systems (including data collection and actuation systems), the connections between individual systems, and the virtual decision-making system [24].

2.1. Sensors, Actuators, and Automated Vehicles in ITS

In order to provide an accurate and comprehensive view of the traffic state for a transportation network, various traffic sensors are used to measure traffic conditions, supporting a wide range of applications for traffic control and safety. Sensors and actuators have become mandatory in vehicle manufacturing and implementation of ITSs, aimed at providing services that improve road safety and reduce traffic congestion [25]. In cases when conventional measurement systems alone are found ineffective, other sources of data (e.g., cameras, GNSS, mobile phone tracking, probe vehicles) are increasingly used to supplement the traffic information. Multiple sources may provide complementary data, and multi-source data fusion can produce a better understanding of the observed situation by decreasing the uncertainty related to the individual sources [26].

AVs can be used as both sensors and actuators for an ITS, providing additional data and allowing for increased traffic control [27]. However, this requires that AVs are also equipped with communication technology to transfer information and receive inputs from a centralized traffic control system.

2.2. Internet of Things and Connected Vehicles in ITS

ITSs and DTs are mainly used in large cities with developed infrastructure, and this trend is supported in part by its connection with the Internet of Things (IoT), as remotely accessible sensors are being built into city infrastructure. IoT allows devices to interact with each other and with cloud platforms, providing remote monitoring, control, and data collection capabilities [28]. IoT units may include light, sound, and composite (measuring multiple parameters) sensors. The collected data can be used for creating virtual models of the environment and continuously updating their state. IoT-enabled interactions between the devices can contribute to centralized traffic control system reliability and performance by reducing the risk of errors and increasing management efficiency [29].

The importance of CVs is closely linked to the development and deployment of ITSs. They can play a crucial role in enhancing resource management, implementing driving automation, contributing to sustainable development, and improving the quality of services [30]. By connecting to the IoT sensor network created by CVs and roadside infrastructure units, ITS DTs can receive and process a large amount of data in real-time. This opens up opportunities to implement smart monitoring and control systems, diagnostics, prognostics, and data analytics, making it possible to efficiently manage traffic flows, reduce travel times, and reduce the risk of traffic accidents [28]. Using CV technology along with DTs offers key possibilities:

• Real-time monitoring. CVs can actively collect and transmit data about the traffic and the environment even from remote areas unaccounted for by stationary sensors, enabling the creation of models accurately reflecting the real-world situation.
• Responsiveness. A centralized traffic control system or stakeholders can rapidly react to changes in the current state of the system and make decisions, sending commands to CVs. CVs can then aid in controlling the traffic and ensuring its safety and effectiveness [27].

Thereby, CVs are a vital component for the development of smart urban environments. Notably, the volume of the data provided by CVs can be overwhelming for human operators to analyze. Automated control systems using computer simulation play an important role in analyzing large amounts of data and identifying patterns and trends that may be difficult for humans to notice or interpret, helping prevent costly and dangerous breakdowns.

2.3. Virtual Environments, Scenario-Based Testing, and Optimization for ITS

DTs are widely used in various fields (e.g., construction, manufacturing, energy, healthcare) [7,28,31,32]. One of the significant advantages of utilizing an ITS model based on DT technology is its capability to make predictions about the future performance of the system. In this context, it is understood that transportation systems are complex, and DT technology can be used to prevent emergent negative traffic situations by simulating and analyzing a virtual copy of a real transportation system and its surroundings ahead of real-world events [29].

Virtual simulation and scenario-based validation approaches are becoming increasingly important in the modern ITS technology development process [33,34], with digital models that accurately represent real scenarios based on high-precision measurements being in focus [35]. Such virtual environments provide a convenient and safe space for various tests of ITS applications that may be hazardous in the real world. They can be used to refine and optimize new technology, allowing a wide spectrum of changes to be made and new scenarios to be designed, making improvements before real-world implementation. This enables safer, more efficient, and cost-effective development. In turn, the creation of such virtual environments for scenario testing and optimizing the performance of real systems often involves the development of DTs.

Such systems can also be used as part of a centralized decision-making and coordination system for many urban systems (including transportation systems) on various time scales [35]. This direction of development is supported in the recent reviews on DT technology, focusing on AVs, cooperative driving, and simulation [24,36,37]. To be implemented, DTs require simulation software as a basis. Many traffic simulators are currently available; however, only some are suitable for such applications [9].

3. Analysis of State-of-the-Art Large-Scale Transportation System Simulators

Prior to the analysis of simulation environments used for transportation system DT and ITS implementation, it is necessary to identify current problems, design requirements, and constraints in the development of ITS models with consideration of AV and CV behavior.

Previously defined in [14], the requirements are such that a complex urban and transportation system simulator should consider the following aspects of modeling:

• Nanoscopic Interactions and Degrees of Freedom. Nanoscopic vehicle-driver-environment models for new technology (e.g., AV perception sensors) and interactions (e.g., CV signal propagation) in a dynamic 3D environment.
• Model heterogeneity and Detailization. Generating emergent behavior heterogeneity for each traffic participant based on differences in initial input parameters provided by structured scenario description (significant for safety studies), allowing for multi-scale simulation with model consistency.
• Modeling time. Faster than real-time simulation for large-scale scenarios using parallel and distributed computing.
• Macroscopic Interactions. Modeling interactions with urban subsystems of different domains (demographics, land use, weather, energy supply, logistics, etc.).

A simulator suitable for ITS simulation and implementing a DT should be technologically aligned with these requirements to help achieve accurate traffic flow modeling, 3D environment perception, Vehicle-to-Everything (V2X) communication support, real-time data processing, and seamless integration with urban infrastructure DTs.

In this section, an analysis of existing simulators is conducted, and their advantages and disadvantages are discussed to discern their fitness for use in scenario testing and automated control of ITS technology. Although many tools have been recently reviewed in other papers [9,17], a very limited group has a close alignment with the requirements discussed above. Following the analysis of existing tools, descriptions of three projects significant to the subject of the paper (MATSim, SimMobility, and CityMoS) are provided. Each of these tools is designed for specific purposes and presents unique challenges.

3.1. MATSim

The Multi-Agent Transport Simulation (MATSim) [38,39,40] project started at ETH Zürich based on the TRansportation ANalysis and SIMulation System (TRANSIMS) project [41,42]. The MATSim team currently consists of three research groups and two companies: (1) Transport Systems Planning and Transport Telematics group at the Institute for Land and Sea Transport Systems, TU Berlin; (2) Transport Planning group at the Institute for Transport Planning and Systems, ETH Zürich; (3) Mobility and Transportation Planning group at the Future Cities Laboratory based in Singapore; (4) Senozon AG, based in Zürich, with subsidiaries in Germany and Austria; (5) Simunto GmbH, based in Zürich [39].

MATSim can simulate independent traffic participants (vehicles, pedestrians, etc.) using a multi-agent approach that employs a microscopic description of demand by following a daily activity schedule and making synthetic decisions for each traveler based on it. Utilizing a set of trip data, MATSim generates routes for traffic participants, where each agent tries to optimize its own day but is influenced by the activities and trips of other agents. Then, it performs the integral microscopic simulation of the resulting traffic flows and congestion they produce. This allows the creation of distinct results for specific conditions in each region, taking into consideration regional demographics, work locations, and local infrastructure. Various transport modes are available, including cars, transit, walking, biking, DRT (demand-responsive transport), etc. Whole days can be simulated second-by-second.

MATSim can simulate large-scale scenarios (e.g., metropolitan areas) by adopting a computationally efficient queue-based approach that combines micro- and mesoscopic elements and does not capture car-following effects. This allows for fast simulations with 107 or more entities. Multi-threading can be used to accelerate computing speeds. The development of MATSim follows a modular approach—functionality and algorithms can be extended or replaced based on user needs.

While it is a powerful tool for multi-agent simulation in transportation systems, MATSim faces limitations that impede its role as a comprehensive simulator for ITS scenario testing or as a basis for ITS DT. It lacks detail in traffic modeling, does not provide a 3D environment, or consider V2X technology, only providing single-scale, relatively low-detail simulation.

3.2. SimMobility

SimMobility [43] is developed by the Intelligent Transportation System Lab at the Massachusetts Institute of Technology (MIT) and the Singapore-MIT Alliance for Research and Technology (SMART) research center.

SimMobility is an integrated mobility simulation platform for the comprehensive exploration of future mobility scenarios through integrating long, medium, and short-term travel behavior to study the effects of various alternative technology, policy, and investment options. SimMobility is activity-based, multi-modal, and multi-scale with consistency across the levels. It can model agents (including pedestrians, drivers, cars, buses, trains, traffic lights, mobile phones, and GPS devices) on second-by-second to year-by-year time scales. Various behavioral models are integrated to predict the impact of mobility demands on transportation networks, ITS performance, or vehicular emissions. To explore uncertain future scenarios, it is important to correctly estimate changes in travel demand in response to the introduction of new technology (e.g., new infrastructure, new mode of transportation) or policy. Activity-based modeling allows the understanding of individual daily travel patterns in response to such changes. For this reason, in SimMobility, individuals are represented as agents. The decision process of each agent combines multiple trips and activities into the schedule, generating the demand for transportation systems. To support multi-modality, SimMobility considers public and private transport, pedestrian traffic, and freight transportation (not included in the open-source version). It allows agents to switch between these modes over the course of a given day. SimMobility can also simulate new modes of transportation, such as AVs, Mobility on Demand, electric vehicles, etc.

The platform is divided into three simulation modules: Short-Term [44], Middle-Term [45], and Long-Term [46]. These levels are based on the time scales in which the behavior of an urban system is considered. The Short-Term (ST) module is a multimodal microscopic agent-based simulator that can capture agent movement with a very fine resolution (up to 0.1 s). The core traffic model of SimMobility ST is based on MITSIM (Microscopic Traffic SIMulator) [47]—an open-source microscopic traffic simulator developed at MIT. The Mid-Term (MT) module handles day-to-day transportation demand for passengers and goods. It generates the agent behavior (activity and travel patterns) by combining demand-side activity-based models and supply-side day-activity scheduling by the DynaMIT traffic assignment system, also developed at MIT [48]. The Long-Term (LT) module utilizes UrbanSim [49] to capture the year-to-year land use and economic activity, predicting land use evolution and property development, and determining associated agent decisions (accounting for interactions between individuals and companies). This allows for the analysis of the effects of future mobility scenarios on residential and workplace locations, vehicle ownership, land use distribution, built environment density and value, etc. SimMobility ST, MT, and LT modules are interconnected and exchange information to obtain a consistent depiction of the urban system state. Although SimMobility can simultaneously support all three levels (agents are available at every level simultaneously), each level can work independently (shared information is only accessed according to each level’s need for updates).

SimMobility can model millions of agents across countries. It is entirely developed in C++ for Linux and uses the Boost. Threads library for parallelization. Once simulations contain a large enough number of entities, multi-threading reaches a point of diminishing returns. At this point, the simulation has to be split and run on different machines (“nodes”) using the SimMobility ST Message Passing Interface (MPI)-based distributed computing. Communication between such nodes is less flexible than multi-threading and requires that the global state of the simulation (including the road network and the trip chains) is split and allocated to corresponding nodes. The agents are allocated to specific nodes, which contain their starting trip locations, and then are transferred to other nodes as they cross a corresponding node boundary (a node boundary is defined as a line perpendicular to a select road segment dividing that segment between two nodes). The entities belonging to different nodes can sense each other via mirroring, but they cannot otherwise interact. Furthermore, entities that are not in a mirrored region cannot send each other messages unless they belong to the same node. This is necessary as it puts a limit on the number of nodes that a given node must interact with, allowing SimMobility ST to scale efficiently up to arbitrarily complex road networks and simulation workloads.

SimMobility has a fully modular architecture, with SimMobility ST being able to connect to an instance of an ns-3 network simulator [50] to provide accurate timing and packet loss information for V2X messages.

A disadvantage of SimMobility is that it lacks detailed traffic modeling and does not provide a 3D environment, omitting complex interactions that can arise between different elements of the traffic scenario.

3.3. CityMOS

The development of City Mobility Simulator (CityMoS) started in 2011 under the name Scalable Electro-Mobility Simulator (SEMSim) within a Research Programme at TUMCREATE Ltd. [51,52]. TUMCREATE is the multidisciplinary research platform of the Technical University of Munich (TUM) at the Singapore Campus for Research Excellence and Technological Enterprise (CREATE). In 2016, SEMSim was renamed to CityMoS, and its focus was expanded from electric vehicles to mobility in general. The development of the software into a feature-rich mobility simulation platform continued with the establishment of the Mobility in Virtual Environments at Scale (MoVES) Laboratory in close collaboration with Huawei since 2021.

CityMoS [52] is an agent-based microscopic mobility simulator capable of simulating transportation systems, including private, public, and commercial transport, and ITS technologies (e.g., AVs), at high resolution. CityMoS focuses not only on traffic patterns but on trip chains, activities, and mode choices of individuals. Agent-based simulation makes it possible to discover emerging effects that cannot be obtained analytically or with lower-resolution traffic simulation. The interactions between agents with different model parametrization produce a realistic heterogeneous population. CityMoS can simulate mobility at the city and state scale and time horizons ranging from minutes up to multiple days. This provides insight into how local changes can have system-wide effects.

CityMoS exists in two variants: CityMoS HPC (high-performance computing), which enables fast execution of complex simulation scenarios on high-performance hardware, and CityMoS 3D, which relies on the same underlying models while providing interactive 3D visualization for real-time inspection. CityMoS supports parallel multicore processing and is capable of simulating city or state-scale transportation systems at high resolution on a variety of computers—from consumer hardware (workstation computers and laptops) to high-performance clusters, including cloud services. Using high-performance clusters can be necessary to ensure that results for a week of a large vehicle population simulation can be ready in a few hours. Otherwise, the computationally intensive nature of high-detail city-scale simulations might be infeasible to run on workstation computers.

CityMoS has a modular architecture that simplifies model development and can be integrated with other simulation environments through bidirectional coupling using an open interface (e.g., High Level Architecture (HLA), Traffic Control Interface (TraCI)), expanding possible functionality. As V2X communication is an inherent part of the future transportation systems, CityMoS can communicate with Veins [53] (an open-source simulation framework for vehicular networks, capable of modeling physical layer communication effects and modern wireless communication standards) and similar tools (e.g., a network simulator such as ns-3 [50] and OMNeT++ [54]) using TraCI. Bidirectional coupling allows the network simulator to create and update a representation of mobile nodes based on their trajectory computed by CityMoS, and to change CV behavior according to the information it receives via the V2X channel. CityMoS can also be connected to a stream of real and even live data, and therefore can be used in traffic control systems to predict traffic situations.

This simulator is suitable for the DT implementation and scalable, but it does not consider the nanoscopic scale AV perception and decision-making processes and CV interactions, omitting a wide range of heterogeneous road situations significant for the ITS scenario exploration.

Based on the conducted analysis, none of the existing tools meet all of the requirements for large-scale and highly detailed ITS modeling. Specifically, the requirement for high model realism and detail is not fulfilled, as state-of-the-art simulators for large-scale scenarios do not take into account the effects of immediate surroundings on AV and CV technology [14,17,21]. CV simulation, for example, should consider the influence of the external environment on the signal propagation over wireless communication channels [55,56]. And AVs require a three-dimensional environment for accurate simulation of machine perception [17,57]. Therefore, there is a need for a tool to be developed, aligned with the proposed requirements, to provide a novel opportunity for scenario exploration by introducing a highly detailed environment. However, some of the concepts introduced in the tools reviewed in this section can guide the development.

4. Parallel Multi-Level Nanoscopic Simulation Environment Architecture

To address the identified challenges of existing simulators and advance the development of innovative mobility simulation that aligns with the ITS and DT principles, an architecture for a parallel multi-level simulation environment is proposed. It is suitable for modeling large-scale and highly detailed ITS scenarios and is based on the SUMO and CARLA simulators. The descriptions of a multi-level simulation approach (where the nanoscopic simulator CARLA interacts with the microscopic simulator SUMO), a general parallel multi-level nanoscopic simulation environment architecture, and an approach specifically developed to allow parallel nanoscopic simulation are provided below.

4.1. Multi-Level Simulation Approach

In transportation scenarios, individual nanoscopic actions collectively affect the overall traffic flow at higher levels of scale [14,58]. Therefore, a bottom-up approach [59] using SUMO and CARLA simulators is proposed for simulating complex transportation systems using new technology that requires interaction modeling on the nanoscopic level. Consistency between the models on the different levels of detail is crucial and can be achieved through direct exchange of relevant information [14]. The proposed approach can model the necessary aspects of AV, CV, and ITS technology on the nanoscopic level, and then move up the level of simulation, providing a stream of relevant simulation results to the next level. The microscopic level simulator SUMO, therefore, receives more accurate (due to consideration of interaction with the environment in nanoscopic simulation) resulting vehicle trajectories from the nanoscopic level simulator CARLA, and can base its calculations on this data, avoiding having to model vehicle motion itself.

CARLA uses 3D maps for simulation, and as the scale of the nanoscopic model increases, larger maps with higher numbers of agents require increasingly high computing power, which at a certain point becomes challenging even for high-end consumer hardware. The allocation of limited geographical areas of the 3D map to separate computers is proposed. To denote such separate limited areas, the concept of “scenes”—open systems consisting of objects and agents and influenced by internal and external factors—is introduced. In the proposed approach, the nanoscopic level consists of multiple interdependent scenes, which are modeled in parallel. Such areas, decreased in size compared to the full map, can be simulated in parallel to provide a larger territory coverage for highly detailed scenario simulation.

Based on the discussed approaches, an architecture is synthesized where the nanoscopic level consists of many dependent scenes, which are in turn connected to a microscopic simulator coordinating and analyzing the running scenario, as shown in Figure 1.

The CARLA-SUMO bridge uses TraCI to pass vehicle parameter values (e.g., position, speed, acceleration) at every tick of the simulation between CARLA and SUMO, which allows vehicle positions to be synchronized so that consistency between the nanoscopic and microscopic models is maintained. The CARLA-SUMO bridge can be used for vehicular control in both directions (vehicles in SUMO can be controlled by CARLA, and vehicles in CARLA can be controlled by SUMO). It is used as the basis for the implementation of the Parallel Simulation Manager (PSM) module.

To extend the capacity of the simulation environment, the following parallel simulation techniques are applied to the CARLA simulator. First, native to CARLA, the maximum number of vehicles and other agents on a single CARLA server can be increased by making multiple GPUs or client computers available to the CARLA server for load balancing. Second, based on the proposed approach, each instance of the CARLA server can contain a separate scene and be linked to the same instance of the SUMO simulator, which allows for relevant information to be transferred between the models and synchronized with the aid of the PSM. The addition of the PSM in the architecture is required for controlling the transitions of agents from one scene to another. The PSM defines the contents of the scene, such as which agents should appear, disappear, or change their state, as well as how environmental parameters change. Parallel simulation between CARLA servers can be implemented on a single device or multiple devices. The number of CARLA server instances can be configurable and depends on the requirements of a particular simulated scenario. If every CARLA scene is allocated to a separate device, the overall number of CARLA servers is theoretically only limited by SUMO’s capability to sustain the size and complexity of the corresponding microscopic scenario, and by network throughput. In this way, a complete scene made up of many nanoscopic scenes simulated by CARLA can be achieved.

Thus, the approach to modeling the highly detailed behavior of numerous vehicles on a large scale is developed. This approach makes it possible to model complex scenarios, such as road intersections with vehicles and pedestrians, as limited-scale nanoscopic-level scenes, and then transfer the results to the microscopic level, where they are aggregated into a full-size scenario. Such an approach significantly reduces computing power requirements for the individual computers and network throughput requirements for the whole system.

If the areas considered as nanoscopic scenes border each other or are geographically positioned close to each other, special considerations are required for nanoscopic simulation of ITS technology involving distant object sensors. The scene map cannot end at the same boundary where the agents it contains cease to be tracked, since there has to be an additional extension of territory from which ITS sensors belonging to the current scene can perceive objects belonging to an adjacent scene.

4.2. Parallel Nanoscopic Simulation Approach

To resolve this problem, the concept of Buffer Zones is proposed. If a vehicle enters a Buffer Zone, it is duplicated and then exists in multiple scenes at the same time, maintaining its parameters (speed, acceleration, steering angle, etc.) in all scenes where it appears, being controlled by the scene (CARLA server) it currently belongs to. Figure 2 describes the Buffer Zone concept and the proposed parallel nanoscopic simulation approach in detail, emphasizing the handling of vehicle control by CARLA and SUMO.

SUMO contains the full scenario microscopic map, and CARLA instances contain partial nanoscopic maps. SUMO tracks all vehicles through synchronization using the CARLA-SUMO bridge (i.e., vehicle trajectories in SUMO are fully determined by vehicle behavior in CARLA). The simplified example provided in Figure 2 explains the interactions between two scenes.

When a vehicle belonging to Scene 1 and controlled by CARLA Instance 1 crosses the End Zone boundary, entering the Buffer Zone, it gains control over a “ghost” of itself in the Ghost Zone of the CARLA Instance 2. The Ghost Zone of a CARLA Instance represents the same area of the scenario map that the End Zone of the adjacent CARLA Instance does (e.g., in Figure 2, the Ghost Zone of the CARLA Instance 2 represents the duplicated End Zone of the Scene 1). The “ghost” vehicles present in the Ghost Zone of any CARLA Instance are controlled by SUMO using the CARLA-SUMO bridge and therefore follow the trajectories of their representations in SUMO, which in turn are synchronized with vehicles in other CARLA instances, where the same vehicle has entered the End Zone.

When a vehicle exits the End Zone of the main scene modeled by a CARLA Instance, it simultaneously exits the Ghost Zone in the other CARLA instance, crossing the Horizon boundary between the scenes and entering another scene. Control over this vehicle is therefore transferred to the CARLA Instance responsible for the other scene (e.g., in Figure 2, vehicle 4 has left Scene 1, entering Scene 2, and its control has been assumed by CARLA Instance 2).

When any vehicle leaves the Buffer Zone entirely, entering the Inside Zone of a scene, the two-step synchronization process dedicated to the vehicle is terminated, and it is therefore deleted from the CARLA Instance where it was represented as a “ghost” (e.g., in Figure 2, the representation of vehicle 5 is not present in the CARLA Instance 1).

The synchronization of vehicles in the Buffer Zones is provided by the Zone Manager module, which is part of the PSM. Together with the CARLA-SUMO bridge, the Zone Manager allows for seamless control of both the SUMO vehicles belonging to a specific zone and the “ghost” vehicles in the Ghost Zone of another CARLA Instance. It is also responsible for the handover of control over a specific vehicle to another CARLA Instance as the vehicle crosses the Horizon and enters the main scene simulated by that instance. For vehicle control, it uses the trajectories of the CARLA-controlled vehicles in the initial CARLA Instance (e.g., blue vehicles in Scene 1 of CARLA Instance 1 control both the vehicle representations in SUMO Scene 1 and, if any are in the End Zone, yellow “ghost” vehicles in the Ghost Zone of CARLA Instance 2).

When a vehicle is transitioned to another CARLA scene, it is then synchronized with its representation in the SUMO map section responsible for the next scene, and with its ghost in the previous CARLA Instance responsible for the vehicle (e.g., blue vehicles in Scene 2 of CARLA Instance 2 control both the vehicle representations in SUMO Scene 2 and, if any are in the End Zone, yellow “ghost” vehicles in the Ghost Zone of CARLA Instance 1).

The described logic applies to any number of adjacent scenes. CARLA-controlled vehicles are therefore never duplicated. If scenes are configured as same-size squares, SUMO-controlled “ghost” vehicles can have a maximum of three duplicates (at the intersection of four scenes), and each scene can have a maximum of eight Buffer Zones (one for every edge of the square and the corners).

The size of the Buffer Zone (the total distance from the Horizon line between scenes to the nearest zone boundary in both directions) should be established according to the characteristics (e.g., perception range, object detection range) of the ITS technology sensors used in the modeled scenario. The common long-range sensors used in the AV and ITS technology are cameras, LiDARs, and radars [60]. Radar sensors create a narrow conical view of the surroundings, translating it into a 2D map of objects and their velocities (using the Doppler effect). It is used in vehicles to detect obstacles up to a distance of 250 m [61]. LiDAR sensors create 3D point clouds representing the surrounding environment with high precision. The working range of LiDAR sensors is up to 100–200 m [61]. Cameras provide artificial vision, with some solutions promising over 1 km perception ranges [62].

Based on the possible maximum sensor ranges, the minimum size of the End Zone is set at 500 m, and the whole Buffer Zone, therefore, at 1000 m. If the available computational resources demand smaller scene sizes, and the scenes are densely located, the Inside Zone size can be reduced to the point of non-existence (with Buffer Zones of different scene pairs bordering each other). The division of a large map into limited-size scenes makes the large-scale nanoscopic simulation computationally feasible and more suitable for real-world systems, such as decentralized ITS DT implementations, because otherwise the information propagation time can become a concern for a centralized ITS DT system.

For the purpose of consistent coordinate tracking, the synchronous simulation mode and a fixed time-step are used in all CARLA instances, with the processes in all simulator instances being executed on the same tick. SUMO, therefore, can track all the vehicles controlled by CARLA instances and analyze their overall impact on the transportation system. For example, the average speed or vehicle throughput at a certain road section, fuel or energy consumption, and harmful emission estimates can be given for the simulated large-scale scenarios with consideration of highly detailed interactions of ITS technology.

5. Implementation of the Proposed Simulation Approaches

In this section, the implementation of both multi-level and parallel nanoscopic simulation approaches is demonstrated to prove their feasibility.

5.1. Multi-Level Simulation

Multi-level simulation using the nanoscopic simulator CARLA and the microscopic simulator SUMO allows tracking the trajectories of the vehicles simulated by CARLA with high detail in the 3D environment. These trajectories can then be analyzed in SUMO to study the effects of intelligent vehicle technology on the higher-level transportation system parameters (average speed, throughput, delays, fuel and energy consumption, harmful emissions, or others available in SUMO or its extensions). Figure 3 demonstrates the implementation of this simulation approach for the CARLA Town06 map. In the lower right corner of Figure 3, a zoomed-in CARLA simulator window is visible following a specific vehicle (blue vehicle controlled by CARLA in the middle), and in the background, the same scenario is shown from a zoomed-out point of view in the SUMO simulator. Such integration of CARLA and SUMO simulators provides the ability to analyze high-level performance indicators of transportation scenarios, while modeling ITS technologies with high detail, verifying the proposed multi-level simulation approach.

5.2. Parallel Nanoscopic Simulation

To prove the feasibility of the proposed parallel nanoscopic simulation approach, two solutions should be demonstrated: first, the capability for the transition of vehicles between two CARLA instances, and second, the capability of SUMO to control this transition, which is required to implement the Buffer Zone logic.

The CARLA simulator requires a 3D map to create a scenario. Three-dimensional maps can be chosen from the collection provided with the simulator, created with the aid of special software based on the OpenDRIVE (.xodr) description format [63], or exported from OpenStreetMap [64], based on a section of a real-world map. The specialized software for creating maps based on the OpenDRIVE format includes RoadRunner [65].

Figure 4 shows an example of two CARLA instances simulated on the same computer in parallel, implementing scenes with simple maps defined using the OpenDRIVE format. When a vehicle is transitioned from one scene to another, its speed, acceleration, and direction remain unchanged. Two scenes are used in this example, but the number can be increased. This implementation confirms the feasibility of using CARLA for parallel simulation with vehicle transitions between the simulated areas.

The CARLA simulator has network settings that must be differentiated if multiple servers are running on the same device. The ports through which each instance of the simulator is accessed should be specified with a different number for the -carla-rpc-port flag. The -opengl flag should be used when launching the CARLA scenarios to bypass the standard Unreal Engine graphics output system, avoiding conflicts during rendering, which is important with limited graphics resources available.

To implement parallel simulation with SUMO overseeing the scenario and managing vehicle transitions between the scenes, at least three maps should be created: the full scenario SUMO map, and two maps representing the CARLA scenes. There should be a common area on both CARLA maps representing the Buffer Zone. To create the scenario for both simulators, different format maps are required. To use .xodr maps created with RoadRunner for SUMO, they have to be converted into the native SUMO .net.xml format. The built-in SUMO ‘netconvert’ tool [66] can be used to create the road network topology based on an OpenDRIVE file. However, the .xodr format uses an application-specific coordinate system. When converting the map to the .net.xml format, the coordinates are shifted because SUMO uses a distinct coordinate system. As a result, there is no direct correspondence between the OpenDRIVE and .net.xml coordinates. To restore it and make simulator communication possible, an additional tool that converts the OpenDRIVE coordinates to the .net.xml format based on the information about the original and transformed map borders is developed.

To implement the parallel simulation with two CARLA and one SUMO instances, it is necessary to modify the existing synchronization architecture, including the CARLA-SUMO bridge, by expanding its capabilities for managing a list of CARLA objects. In the initial version of the program used to implement the prototype, only part of the CARLA-SUMO bridge capabilities is used. The bridge’s code is extended, with a method that encompasses the main simulation loop performing the sumo.tick(), filtering vehicle representations in SUMO by their belonging to a specific map zone, and then sequentially calling the carla.tick() for each CARLA instance. This way, each CARLA instance receives control over those agents that are within its area of responsibility. Otherwise, general synchronization logic remains the same as in the original implementation of the CARLA-SUMO bridge. This architecture allows scaling the nanoscopic simulation by performing it for several map sections in parallel, and at the same time maintaining the centralized oversight by SUMO. The demonstration of an operational implemented solution is shown in Figure 5a,b.

The demonstrated implementation has reduced-size Buffer Zones to make the scenario legible. It is executed on a single computer, serving as a proof of concept, and making performance comparisons not possible at this stage of development. The code for the parallel simulation environment is available at https://github.com/CAVISE (accessed on 8 October 2025).

6. Conclusions

The existing simulation environments used for validating the ITS technology, including AVs and CVs, do not consider the necessary complexity (nanoscopic level of detail) of interaction between the intelligent technology, traffic participants, and the environment. At the same time, nanoscopic simulators demand a level of computational resources unrealistic for consumer hardware when it comes to highly populated, large-scale scenario simulation.

In this paper, the requirements for ITS technology simulation and DT implementation suitability were presented. In this regard, the large-scale transportation scenario simulators MATSim, SimMobility, and CityMoS were analyzed, and their limitations and useful features were identified. Based on the identified deficiencies, a novel approach to large-scale, highly detailed urban transportation simulation with consideration of intelligent vehicles was developed. An architecture for the parallel and multi-level ITS integrated simulation environment was proposed. It develops the idea of partitioning a transportation scenario map into nanoscopic scenes modeled by the CARLA simulator, taking into account the interactions between the technology and the 3D environment, and monitoring their parallel execution using the microscopic SUMO simulator. Data collected through SUMO allows a subsequent analysis of the influence of the ITS technology on the transportation system.

Two distinct approaches useful for large-scale, detailed simulation were described, and a preliminary implementation of both was provided. First, a multi-level simulation approach using the CARLA simulator for AV simulation and the SUMO simulator for analyzing transportation system performance measures was tested. Second, a parallel nanoscopic simulation approach and the concept of Buffer Zones were described, and a prototype of a simulation environment using this approach was implemented to prove its feasibility. The obtained results are intended to be applied in transportation planning research and the development of projects focusing on AV and CV integration into urban transportation systems, aiding the development of more sustainable, safe, and effective solutions. Potential integration of the proposed approaches into ITS DTs in real systems is also possible.

Directions for future work are further outlined:

• Testing the Buffer Zone concept with a larger scenario using at least two individual computers is the next step in the parallel multi-level simulation development. It should allow the performance of the parallel simulation approach to be measured and compared with the standard approach using a single computer without spatial decomposition.
• Further work is also required to make the proposed approach compatible with any maps that include the objects surrounding the road network, and to incorporate multimodal traffic participants (pedestrians, bicycles, etc.). Developing capabilities for the automated segmentation of existing CARLA and .xodr maps into scenes or for segmented map generation during the automated map creation process is another task for future work.
• Approaches to load balancing for CARLA servers used to handle nanoscopic scenes and CARLA clients controlling the agents in the traffic scenarios should be studied. Notably, it is assumed that traffic can change significantly for each partition of the map at the nanoscopic level of simulation. Therefore, load balancing cannot be based on network size and traffic intensity as in microscopic simulations.
• The parallel simulation environment is to be integrated with a V2X simulator. Theoretically, this integration should be straightforward, as it would be done through SUMO. Nevertheless, it requires further investigation into simulator performance and interplay.
• Another current gap is the lack of macroscopic-level simulation in the proposed multi-level environment, which is available in other tools. These shortcomings are intended to be overcome in further development.