MCS-Sim: A Photo-Realistic Simulator for Multi-Camera UAV Visual Perception Research

Abstract

Multi-camera systems (MCSs) are pivotal in aviation surveillance and autonomous navigation due to their wide coverage and high-resolution sensing. However, challenges such as complex setup, time-consuming data acquisition, and costly testing hinder research progress. To address these, we introduce MCS-Sim, a photo-realistic MCS simulator for UAV visual perception research. MCS-Sim integrates vision sensor configurations, vehicle dynamics, and dynamic scenes, enabling rapid virtual prototyping and multi-task dataset generation. It supports dense flow estimation, 3D reconstruction, visual simultaneous localization and mapping, object detection, and tracking. With a hardware-in-loop interface, MCS-Sim facilitates closed-loop simulation for system validation. Experiments demonstrate its effectiveness in synthetic dataset generation, visual perception algorithm testing, and closed-loop simulation. Here we show that MCS-Sim significantly advances multi-camera UAV visual perception research, offering a versatile platform for future innovations.

1. Introduction

The intelligent autonomy of unmanned aerial vehicles (UAVs) represents a leading trend in modern technology, enabling tasks such as reconnaissance, surveillance, disaster relief, transportation delivery, and virtual reality (VR) applications. Within VR ecosystems, UAV-mounted imaging systems enable dynamic viewpoint control through several key technologies [1]. Specifically, trajectory planning with optimized flight paths ensures stable image acquisition while avoiding occlusions [2]. Additionally, multi-UAV coordination allows synchronized drone swarms to provide panoramic coverage through distributed camera arrays [3]. These implementations address the core requirements of VR systems: broad field-of-view (FOV) coverage, high resolution, and motion consistency. Beyond VR applications, effective UAV operation across various domains fundamentally relies on robust environmental perception capabilities, including accurate self-localization and environmental mapping [4]. This foundation enables advanced capabilities such as navigation, obstacle avoidance, and path planning. Vision sensors dominate scene perception tasks due to their low cost and high information density. However, enhanced perception demands broader FOV, higher resolution, and precise sensing.

Monocular and binocular stereo cameras dominate current vision sensors, yet their FOV is inherently limited. While fisheye cameras expand FOV, they face trade-offs between effective resolution and FOV because of the impact of diffraction limits. To address demands for high resolution, wide FOV, scale depth perception, and motion sensitivity, multi-aperture vision systems emerged in the early 20th century [5]. These systems, termed artificial compound eye systems (ACESs) and light field cameras, are classified as multi-ocular sensors alongside unstructured multi-camera systems. Applications span computational imaging [6], simultaneous localization and mapping (SLAM) [7,8,9], and autonomous driving [10]. Except for two types of light field camera with unique light field acquisition models [5], most multi-ocular sensors can be considered as multi-camera systems (MCSs).

These MCSs integrate three or more cameras with tilted optical axes. Their non-parallel alignment expands the FOV; meanwhile, the baseline distances enable precise depth measurement for nearby objects. Additionally, the multi-view design counters two major limitations of traditional monocular and stereo cameras: lighting fluctuations and occlusions. This enhancement improves UAVs’ 3D perception and boosts operational performance in complex, wide-area scenarios.

Currently, research on MCSs faces some challenges focused on three levels: (1) Physical level: The process of configuration design and entity preparation is time-consuming and expensive, necessitating simulation tools for validation. (2) Algorithm level: Algorithm research relies on the input of physical information, but dataset scarcity limits progress. (3) Application level: Intricate system testing introduces safety hazards and high expenses. To overcome these barriers and accelerate research for multi-camera UAV visual perception, it is crucial to explore simulation tools that can compensate for the deficiency of physical systems.

In the field of embodied intelligence, using simulators to synthesize datasets for large-scale algorithm training, or providing closed-loop interfaces for online training and testing, has become a widely adopted practice. Similar principles extend to computer vision tasks, where algorithms increasingly leverage large-scale and diverse synthetic datasets to achieve robust generalization to real-world scenarios. For instance, DROID-SLAM [11] demonstrated exceptional accuracy and robustness in real-world testing after being trained exclusively on the synthetic dataset TartanAir [12]. The foundation model Depth Anything V2 [13] further exemplifies this paradigm: it first employs synthetic data to train a high-capacity teacher model; then, it generates pseudo-labels for unlabeled real images to enable knowledge distillation for efficient student models, ultimately achieving state-of-the-art zero-shot generalization. These successes suggest the viability of applying analogous approaches to multi-camera perception in UAV applications.

In this work, we introduce MCS-Sim, a novel photo-realistic simulator designed for multi-camera UAV visual perception research. The introduced framework integrates six key modules to address critical requirements in autonomous perception development: (1) MCS simulation, (2) vehicle dynamics modeling, (3) movable object simulation, (4) structured synthetic dataset generation, (5) diversified scene configuration, and (6) closed-loop testing capabilities. Our contributions include the following:

• A modular simulator architecture is introduced that extends conventional monocular simulation paradigms by supporting both geometrically regular ACESs and arbitrary multi-camera configurations. This design enables rapid virtual prototyping of MCSs.
• A structured synthetic dataset generation pipeline is integrated to output multi-camera sequences with multi-modal pixel-wise ground truth (e.g., depth, semantic labels, optical flow, and surface normal). This facilitates visual perception research including dense motion flow estimation, 3D reconstruction, visual SLAM, object recognition, and tracking.
• A hardware-in-the-loop (HIL) interface is developed to enable end-to-end system testing. This capability is demonstrated through a representative case: real-time ground target tracking using UAV electro-optical payloads, where simulated sensor feedback directly interfaces with embedded perception algorithms for closed-loop control validation.

While AI-driven autonomous agents and more advanced features could enhance the simulator, time and cost constraints necessitate deferring these enhancements to future work, which is beyond the scope of this study.

This paper is organized as follows. In Section 2, prior work related to our approach is reviewed. Section 3 provides a detailed introduction of the simulator. In Section 4, key functions mentioned above are verified through simulation experiments. Finally, Section 5 concludes our work and proposes future directions.

2. Current Research and Projects in Open-Source UAV Simulators

In this section, related work is divided into four distinct categories: (1) vision-oriented simulators, (2) physics-based simulators, (3) neural closed-loop simulators, and (4) generative closed-loop simulators. Each category is detailed below.

• Vision-oriented simulators. Within the computer vision community, there are numerous simulators that utilize 3D engines [14] to create synthetic datasets for researchers’ specific needs. The core of vision-oriented simulators lies in the generation of photo-realistic image data, coupled with precise ground truth annotations, tailored for various visual tasks. CARLA [15] and AirSim [16] are open-source simulation toolkits developed on the Unreal Engine 4 (UE4), providing high-fidelity physical and visual simulations. They facilitate the cost-effective creation of vast amounts of training data, inclusive of depth maps and semantic masks, which in turn accelerates the development of AI models in simulations involving autonomous vehicle maneuvers or UAV flights. UnrealCV [17], a UE4 plugin, simplifies virtual world creation for vision researchers. It automates the generation of RGB images, object masks, depth maps, and surface normals. However, using UnrealCV requires significant programming effort in both vision tasks and UE4 environment configuration. Sim4CV [18] streamlines this process by eliminating the need for complex UE4 game development. It provides an integrated solution for vision-based tracking and driving simulations. Another UE4-based tool, UnrealGT [19], automatically augments scenes with contextually placed objects to enhance training data variability. More recently, GRADE [20], built on Nvidia Isaac Sim, introduced dynamic elements like pedestrians and animals into simulations. This enables the generation of dynamic SLAM datasets. For vision-oriented simulators, while constructing high-fidelity scenes remains labor-intensive, the resultant data is kept free from geometric errors, ensuring ground truth annotations with perfect accuracy.
• Physics-based simulators. Physics-based simulators prioritize realistic physical interactions between agents and environments, driving research in embodied AI and robotics. These simulators have sparked extensive research in various domains, including navigation (e.g., Habitat [21] and iGibson [22]), robotic manipulation (e.g., SAPIEN [23]), and embodied learning (e.g., AI2-THOR [24], TDW [25]). Recent advancements include LEGENT [26] and HAZARD [27], which integrate Large Language Models (LLMs) and Large Multimodal Models (LMMs), alongside multi-agent simulators like Habitat 3.0 [28] and UnrealZoo [29]. Notably, Genesis [30] introduces generative physics engines for ultra-fast, differentiable simulations with physic-realistic accuracy. However, traditional simulators rooted in geometric modeling still exhibit greater robustness.
• Neural closed-loop simulators. Neural closed-loop simulators, inspired by Neural Radiance Fields (NeRFs) [31] and 3D Gaussian Splatting (3DGS) [32], reconstruct real-world environments for autonomous driving applications. These novel simulators leverage NeRF (e.g., MARS [33], UniSim [34], NeuRAD [35], and NeuroNCAP [36]) and 3DGS (e.g., HUGSIM [37]) to achieve efficient scene construction with high fidelity in textures and lighting compared to those found in real-world scenario. However, challenges persist under extreme behavioral deviations or viewpoint changes, where synthesized views exhibit artifacts like blurred shadows, reducing performance relative to traditional simulators based on 3D engines.
• Generative closed-loop simulators. Since 2024, the rapid evolution of video diffusion models, such as GAIA-1 [38], DriveDreamer [39], Vista [40], and GenAD [41], has shown great promise in generating photo-realistic driving videos. Generative closed-loop simulators, exemplified by DriveArena [42], belong to the category of world models [43] and represent the evolutionary frontier of simulation technology. However, generative closed-loop simulators face limitations in spatial and temporal consistencies, as well as computational demands.

Each existing simulator has distinct advantages and drawbacks, as shown in

Table 1. Comparison of typical simulators.

Simulator	Vision Sensor	Image Quality	Base Engine	Scalability	Pixel-Wise Annotation	Hardware Requirement	Frame Rate
MCS-Sim (ours)	S|C|A	P|C	UNIGINE	✓	D|S|N|F	L	⩾30FPS
CARLA [15]	M		UE	✓	D|S
AirSim [16]			UE	✓	D|S
Sim4CV [18]			UE	✓	D|S
GRADE [20]			NIS	✓	D|S|N|F
Habitat [21]	M	S|C	Habitat-Sim	✓	D|S	L	⩾30FPS
TDW [25]		P|C	Unity	✓	-	L
LEGENT [26]		S|C	Unity	✓	-	S
HAZARD [27]		S|C	Unity	✓	D|S	W
MARS [33]	S	(H)|C	(NeRF)	-	(D)|(S)	W	<30FPS
NeuRAD [35]			(NeRF)	-	(D)		<30FPS
NeuNCAP [36]			(NeRF)	-	-		<30FPS
HUGSIM [37]			(3DGS)	-	(D)|(S)|(F)		⩾30FPS
DriveArena [42]	S	(H)|IC	LimSim|(SD)	-	-	W	⩾30FPS

■: Vision-oriented simulator; ■: Physics-based simulator; ■: Neural closed-loop simulator; ■: Generative closed-loop simulator. Vision sensor: M: Monocular camera; S: Surround MCS; C: ACES; A: Arbitrary MCS. Image quality: S: Simple; P: Photo-realistic; H: High-fidelity; C: Spatio-temporal consistency; IC: Spatio-temporal inconsistency; ( ) denotes that it contains visual aliasing. Base Engine: UE: Unreal Engine; NIS: Nvidia Isaac Sim; SD: Stable Diffusion; ( ) denotes algorithm implementation. Scalability: The check mark denotes support for secondary development and the addition of more sensor types. Pixel-wise annotation: D: Depth map; S: Semantic segmentation; N: Surface normal; F: Optical flow; ( ) denotes that it contains errors. Hardware requirement: L: Laptop with a consumer-grade GPU; S: Connected to remote servers for training and deployment; W: Workstation with advanced GPUs. Frame rate: Average frame rate of integrated graphics rendering and physics computation.

. Vision-oriented and physics-based simulators leverage mature graphical pipelines to achieve geometrically precise rendering, maintaining spatio-temporal consistency while providing ground-truth pixel-wise annotations. These simulators typically operate in real time using consumer-grade GPU-equipped laptops. In contrast, neural and generative closed-loop simulators, which have emerged only in the past two years, produce high-fidelity image quality through volume rendering techniques. However, they suffer from inherent geometric inaccuracies leading to rendering artifacts and spatio-temporal inconsistencies, with compromised pixel-wise annotation accuracy. Moreover, their computational demands necessitate high-performance workstations, often struggling to maintain real-time operation due to hardware limitations.

The proliferation of MCS configurations has exposed a critical research gap: the absence of specialized simulators for multi-camera UAV visual perception research. To advance this field, such a simulator must satisfy dual requirements. From an economic perspective, it should enable easy deployment and efficient operation. Technically, it must generate photo-realistic images with error-free annotations and provide closed-loop testing capabilities. The ground truth annotations should encompass multi-modal pixel-wise outputs to support multiple vision tasks, including dense pixel-wise matching, 3D reconstruction, visual SLAM, object recognition, and tracking. Therefore, MCS-Sim is introduced to address these requirements. Ours follows the design paradigm of vision-oriented simulators, with functional and performance comparisons presented in Table 1. It extends support from monocular vision to arbitrary MCS. Additionally, it integrates a multi-modal dataset generation pipeline with enhanced annotation capabilities and provides a HIL closed-loop interface for validation.

3. System Design of MCS-Sim

MCS-Sim adopts a video game-like design for easy installation without extra dependencies, ensuring user-friendly operation. The simulator features an intuitive graphical user interface (GUI) that simplifies adjustment of all critical settings. Researchers can modify configuration files directly or via C++ scripting for advanced customization. Flight environments are expansive and detailed, containing multiple aerial vehicles (two UAVs, a small aircraft, and a freely mobile object) across a 50 km × 50 km map. This map includes diverse scenarios: towns, airports, warehouses, grasslands, forests, mountains, coastlines, oceans, and underwater zones. The closed-loop interfaces, which encompasses image injection and serial communication, enable bidirectional data stream between external devices or programs and the simulator.

3.1. Overview of Simulator Architecture

The overview of the simulator’s architecture is presented in Figure 1. The core functionality of the software lies in its ability to customize MCS configurations, orchestrate vehicle movements, set up dynamic objects, and generate and record ground truth annotations for vision perception research. It supports closed-loop simulation for photoelectric data processing and flight control via a HIL interface. Developed utilizing a 3D engine, the simulator comprises various components, including a simulation setting module, an MCS model module, a UAV control module, a third-person visual display, a first-person view display, an image data generation module, a data recording module, a visual model library, and communication interfaces.

• 3D engine. The 3D engine offers a realistic graphics rendering pipeline and an integrated real-time physics engine. MCS-Sim employs a commercial 3D engine, UNIGINE 2.7.2 Sim, known for its 64-bit coordinate precision and advanced rendering capabilities. While popular 3D engines [14] like Unreal Engine 4 and Unity 2019.3 utilize 32-bit systems, UNIGINE’s 64-bit architecture enables superior coordinate accuracy and wide-area visualization, which is critical for aerospace simulations. This engine integrates a physics-based rendering pipeline that produces realistic visuals while maintaining precise spatial computations, outperforming 32-bit alternatives in large-scale scene modeling. The selection of UNIGINE aligns with recent advancements in 3D simulation technology, where high-performance engines now support complex multisensory applications through integrated physics systems and visual fidelity.
• Simulation setting module. The simulation settings module provides a user interface for configuring simulation parameters. It allows for the specification of vehicle types and motion modes, MCS configuration, data recording options, coordinate systems, and environmental effects. Developed with Qt Creator 4.0.3, this module integrates a GUI design that simplifies parameter input while establishing real-time communication between the interface and 3D engine core through Socket protocols.
• MCS model module. The MCS model module streamlines the MCS configuration through three customizable input methods: preset templates, GUI-based adjustments, or Excel file imports. These settings are automatically converted into a unified model containing critical parameters, including MCS type, camera quantity, and both extrinsic and intrinsic parameters for each camera. Then, the corresponding light field acquisition model is constructed. This modular design ensures flexibility for researchers while maintaining parameter consistency across simulation workflows.
• UAV control module. The UAV control module enables movement simulation via two modes: (1) direct manual operation using keyboard and mouse control, and (2) automated control through real-time motion control data received from external programs or devices, which drives a UAV and updates its pose state accordingly.
• Third-person visual and first-person view display. The third-person visual display tracks vehicle movements using a spectator camera controlled via the PlayerPersecutor programming interface in UNIGINE. For the first-person view display, the simulator renders real-time image feeds captured by individual cameras. Each image feed is generated through the WidgetSpriteViewport programming interface in UNIGINE and displayed in dedicated interface zones, providing simultaneous multi-view monitoring.
• Image data generation module. The image data generation module is used to generate RGB images tailored for MCS, alongside depth maps, segmentation masks, surface normals, optical flow, and other ground truth data. This module is mainly implemented by parsing the rendering pipeline.
• Data recording module. The data recording module generates synthetic datasets for MCS-based 3D vision tasks, including single-shot images, videos, and metadata such as vehicle poses, dynamic object trajectories, and MCS calibration parameters.
• Visual model library. The visual model library contains scene models with adjustable environmental effects for rendering, plus vehicle and object models. Scene models and their environmental effects are created using the scene editor of UNIGINE, with parameters modifiable through script adjustments. Vehicle and object models can also be designed in third-party software like 3ds MAX and imported into the simulator.
• Communication interfaces. The communication interface enables two-way interaction between the simulator and external programs or physical devices through (1) image injection, which inputs image data into the photoelectric data processing device, and (2) serial communication, transmitting real-time simulation status and flight control parameters.

The basic workflow of the simulation process comprises the following steps: (1) configuration of simulation settings and MCS modeling; (2) 3D engine execution for visual rendering; (3) UAV control or closed-loop simulation; (4) configuration modification; (5) dataset generation and recording.

Hereinafter, several key design strategies, essential to our methodology, are introduced.

3.2. Multi-Camera System Modeling

As introduced in Section 1, MCSs are classified into two categories: ACESs and unstructured MCSs. Our prior work [44] developed simulation software for ACES, supporting preset templates, GUI-based adjustments, and Excel file imports. This simulator now extends its capabilities to accommodate unstructured MCS configurations.

Table 2. Relevant parameters pertaining to the configuration settings of MCS.

Type	Configuration	Variables for Rendering	Ground-Truth Recording
Number of cameras	${\mathrm{N}}_{\mathrm{c}}$
Optical settings for camera c	${\mathrm{f}}_c$, ${\mathrm{d}}_c$, $\gamma$, ${\mathrm{N}}_{\mathrm{u},c}$, ${\mathrm{N}}_{\mathrm{v},c}$ (${\sigma }_{f_c}$, ${\mathbf{w}}_{optical\_errors}$)	${\mathrm{f}}_c+\delta {\mathrm{f}}_c$, ${\mathrm{d}}_c$, $\gamma$, ${\mathrm{N}}_{\mathrm{u},c}$, ${\mathrm{N}}_{\mathrm{v},c}$ (${\mathbf{w}}_{optical\_errors}$)	${\mathbf{K}}_c$
Pose settings for camera c	${\mathrm{n}}_{\mathrm{x},c}$, ${\mathrm{n}}_{\mathrm{y},c}$, ${\mathrm{n}}_{\mathrm{z},c}$, ${\mathrm{X}}_c^{\mathrm{s}}$, ${\mathrm{Y}}_c^{\mathrm{s}}$, ${\mathrm{Z}}_c^{\mathrm{s}}$ (${\sigma }_{{\phi }_c^{\mathrm{s}}}$, ${\sigma }_{{\theta }_c^{\mathrm{s}}}$, ${\sigma }_{{\varphi }_c^{\mathrm{s}}}$, ${\sigma }_{{\mathrm{X}}_c^{\mathrm{s}}}$, ${\sigma }_{{\mathrm{Y}}_c^{\mathrm{s}}}$,${\sigma }_{{\mathrm{Z}}_c^{\mathrm{s}}}$)	${\phi }_c^{\mathrm{s}}+\delta {\phi }_c^{\mathrm{s}}$, ${\theta }_c^{\mathrm{s}}+\delta {\theta }_c^{\mathrm{s}}$, ${\varphi }_c^{\mathrm{s}}+\delta {\varphi }_c^{\mathrm{s}}$, ${\mathrm{X}}_c^{\mathrm{s}}+\delta {\mathrm{X}}_c^{\mathrm{s}}$, ${\mathrm{Y}}_c^{\mathrm{s}}+\delta {\mathrm{Y}}_c^{\mathrm{s}}$, ${\mathrm{Z}}_c^{\mathrm{s}}+\delta {\mathrm{Z}}_c^{\mathrm{s}}$	${\mathbf{T}}_c^{\mathrm{s}}$

Configuration: These parameters are used for the rapid entry of configuration. Variables for rendering: These variables are used for graphical rendering execution via application programming interfaces. Ground-truth recording: These parameters are recorded as ground truth. ${\mathrm{f}}_c$, ${\sigma }_{{\mathrm{f}}_c}$: focal length and its error variance; ${\mathrm{d}}_c$: image plane width; $\gamma$: pixel aspect ratio; ${\mathrm{N}}_{\mathrm{u},c},{\mathrm{N}}_{\mathrm{v},c}$: image resolution; ${\mathbf{w}}_{optical\_errors}$: optical error parameter vector; ${\mathbf{K}}_c$: intrinsic matrix. ${\mathrm{n}}_{\mathrm{x},c}$, ${\mathrm{n}}_{\mathrm{y},c}$, ${\mathrm{n}}_{\mathrm{z},c}$, ${\phi }_c^{\mathrm{s}}$, ${\theta }_c^{\mathrm{s}}$, ${\varphi }_c^{\mathrm{s}}$: rotation parameters (axis-angle and Euler angles); ${\mathrm{X}}_c^{\mathrm{s}}$, ${\mathrm{Y}}_c^{\mathrm{s}}$, ${\mathrm{Z}}_c^{\mathrm{s}}$: translational deviations; ${\sigma }_{{\phi }_c^{\mathrm{s}}}$, ${\sigma }_{{\theta }_c^{\mathrm{s}}}$, ${\sigma }_{{\varphi }_c^{\mathrm{s}}}$, ${\sigma }_{{\mathrm{X}}_c^{\mathrm{s}}}$, ${\sigma }_{{\mathrm{Y}}_c^{\mathrm{s}}}$,${\sigma }_{{\mathrm{Z}}_c^{\mathrm{s}}}$: error variances; ${\mathbf{T}}_c^{\mathrm{s}}$: extrinsic matrix. ( ) denotes optional error terms; $\delta *$ represents random perturbation generated from variance $\sigma *$.

summarizes the relevant parameters’ MCS modeling, including both configuration variables and the ground-truth intrinsic/extrinsic parameters of individual cameras.

For modeling the imaging parameters of individual cameras, all cameras adopt the pinhole model characterized by four physical parameters: focal length ${\mathrm{f}}_c$ with its error variance ${\sigma }_{{\mathrm{f}}_c}$, image plane width ${\mathrm{d}}_c$, pixel aspect ratio $\gamma$, and image resolution ${\mathrm{N}}_{\mathrm{u},c}\times {\mathrm{N}}_{\mathrm{v},c}$, where $c\in \left\{0,1,\dots ,{\mathrm{N}}_{\mathrm{c}}-1\right\}$ represents the camera index. Given the optical parameters $\left(\mathrm{f},\mathrm{d},\gamma ,{\mathrm{N}}_{\mathrm{u}},{\mathrm{N}}_{\mathrm{v}}\right)$ for camera c, its intrinsic matrix is recorded as follows:

$$\mathbf{K}=\left[\begin{array}{ccc}{\mathrm{f}}_{\mathrm{u}}& 0& {\mathrm{o}}_{\mathrm{u}}\\ 0& {\mathrm{f}}_{\mathrm{v}}& {\mathrm{o}}_{\mathrm{v}}\\ 0& 0& 1\end{array}\right],$$

(1)

where ${\mathrm{f}}_{\mathrm{u}}=\mathrm{f}·{\mathrm{N}}_{\mathrm{u}}/\mathrm{d}$ and ${\mathrm{f}}_{\mathrm{v}}=\mathrm{f}·{\mathrm{N}}_{\mathrm{u}}/(\mathrm{d}·\gamma )$ define pixel focal lengths, while ${\mathrm{o}}_{\mathrm{u}}=\left({\mathrm{N}}_{\mathrm{u}}-1\right)/2$ and ${\mathrm{o}}_{\mathrm{v}}=\left({\mathrm{N}}_{\mathrm{v}}-1\right)/2$ specify the principal point coordinates. The configured optical parameters $\left(\mathrm{f}+\delta {\mathrm{f}}_c,\mathrm{d},\gamma ,{\mathrm{N}}_{\mathrm{u}},{\mathrm{N}}_{\mathrm{v}}\right)$ are passed to the UNIGINE API for automated image generation, augmented by an optical error parameter vector ${\mathbf{w}}_{optical\_errors}$ encompassing color saturation, color gamma, lens smudges, exposure adaptation, depth-of-field effects, and motion blur artifacts, which are dynamically applied through dedicated API invocations to achieve physically plausible rendering. It should be noted that the graphics engine simulates optical phenomena without strict adherence to physical laws. Geometric distortion, while requiring synchronous pixel-wise ground-truth annotation generation, is implemented as an offline post-processing step to avoid real-time rendering frame rate degradation.

The relevant coordinate systems for MCS modeling are defined as Figure 2 shows. The MCS coordinate is usually centered at its geometric origin and denoted as ${\mathcal{F}}_{\mathrm{s}}$. This establishes coordinate system conversions between the world coordinates ${\mathcal{F}}_{\mathrm{w}}$, MCS coordinates ${\mathcal{F}}_{\mathrm{s}}$, and camera coordinates ${\mathcal{F}}_c$ via the equation below:

$${\tilde{\mathit{X}}}_{\mathrm{w}}={\mathit{T}}_{\mathrm{s}}^{\mathrm{w}}·{\mathbf{T}}_c^{\mathrm{s}}·{\tilde{\mathit{X}}}_c,$$

(2)

where ${\tilde{\mathit{X}}}_{\mathrm{w}}$ and ${\tilde{\mathit{X}}}_c$ represent homogeneous world and camera coordinates. The transformation matrix ${\mathit{T}}_{\mathrm{s}}^{\mathrm{w}}$ denotes the pose of MCS, and the extrinsic matrix ${\mathbf{T}}_c^{\mathrm{s}}$ of camera c combines a rotation matrix ${\mathbf{R}}_c^{\mathrm{s}}\in \mathbb{SO}\left(3\right)$ and a translation vector ${\mathbf{t}}_c^{\mathrm{s}}={\left[{\mathrm{X}}_c^{\mathrm{s}},{\mathrm{Y}}_c^{\mathrm{s}},{\mathrm{Z}}_c^{\mathrm{s}}\right]}^{\top }$ from offsets.

When configuring the MCS setup settings, the orientation of camera c is represented by the axis-angle $\left({\mathrm{n}}_{\mathrm{x},c},{\mathrm{n}}_{\mathrm{y},c},{\mathrm{n}}_{\mathrm{z},c}\right)$, while translational deviations are denoted as $\left({\mathrm{X}}_c^{\mathrm{s}},{\mathrm{Y}}_c^{\mathrm{s}},{\mathrm{Z}}_c^{\mathrm{s}}\right)$. The error variances of these setting parameters are ${\sigma }_{{\phi }_c^{\mathrm{s}}}$, ${\sigma }_{{\theta }_c^{\mathrm{s}}}$, ${\sigma }_{{\varphi }_c^{\mathrm{s}}}$, ${\sigma }_{{\mathrm{X}}_c^{\mathrm{s}}}$, ${\sigma }_{{\mathrm{Y}}_c^{\mathrm{s}}}$, and ${\sigma }_{{\mathrm{Z}}_c^{\mathrm{s}}}$, respectively. The extrinsic matrix ${\mathbf{T}}_c^{\mathrm{s}}$ for each camera is derived from $\left({\mathrm{n}}_{\mathrm{x},c},{\mathrm{n}}_{\mathrm{y},c},{\mathrm{n}}_{\mathrm{z},c},{\mathrm{X}}_c^{\mathrm{s}},{\mathrm{Y}}_c^{\mathrm{s}},{\mathrm{Z}}_c^{\mathrm{s}}\right)$ and subsequently recorded together with ${\mathbf{T}}_{\mathrm{s}}^{\mathrm{w}}$ during simulation. If calibration error parameters (${\sigma }_{{\phi }_c^{\mathrm{s}}}$, ${\sigma }_{{\theta }_c^{\mathrm{s}}}$, ${\sigma }_{{\varphi }_c^{\mathrm{s}}}$, ${\sigma }_{{\mathrm{X}}_c^{\mathrm{s}}}$, ${\sigma }_{{\mathrm{Y}}_c^{\mathrm{s}}}$, ${\sigma }_{{\mathrm{Z}}_c^{\mathrm{s}}}$) are configured, the axis-angle representation is converted to Euler angles (where ${\phi }_c^{\mathrm{s}}$, ${\theta }_c^{\mathrm{s}}$, and ${\varphi }_c^{\mathrm{s}}$ correspond to azimuth, pitch, and roll, respectively), yielding the perturbed angles $\left({\phi }_c^{\mathrm{s}}+\delta {\phi }_c^{\mathrm{s}},{\theta }_c^{\mathrm{s}}+\delta {\theta }_c^{\mathrm{s}},{\varphi }_c^{\mathrm{s}}+\delta {\varphi }_c^{\mathrm{s}}\right)$. Similarly, the translational deviations are updated to $\left({\mathrm{X}}_c^{\mathrm{s}}+\delta {\mathrm{X}}_c^{\mathrm{s}},{\mathrm{Y}}_c^{\mathrm{s}}+\delta {\mathrm{Y}}_c^{\mathrm{s}},{\mathrm{Z}}_c^{\mathrm{s}}+\delta {\mathrm{Z}}_c^{\mathrm{s}}\right)$. Finally, the corresponding application programming interfaces (APIs) in UNIGINE are invoked to execute graphical rendering operations according to $\left({\phi }_c^{\mathrm{s}}+\delta {\phi }_c^{\mathrm{s}},{\theta }_c^{\mathrm{s}}+\delta {\theta }_c^{\mathrm{s}},{\varphi }_c^{\mathrm{s}}+\delta {\varphi }_c^{\mathrm{s}}\right)$ and $\left({\mathrm{X}}_c^{\mathrm{s}}+\delta {\mathrm{X}}_c^{\mathrm{s}},{\mathrm{Y}}_c^{\mathrm{s}}+\delta {\mathrm{Y}}_c^{\mathrm{s}},{\mathrm{Z}}_c^{\mathrm{s}}+\delta {\mathrm{Z}}_c^{\mathrm{s}}\right)$.

3.3. Image Data Generation and Recording

To generate MCS-related datasets for vision perception research, three key methodologies are addressed: dynamic scene construction, graphic rendering pipeline parsing, and synchronous data recording.

The dynamic scene is built using the scene editor of UNIGINE, integrating terrain, vegetation, architecture, roads, water systems, sky, movable vehicles, various objects, and a spectrum of environmental effects (e.g., lighting, rain, clouds, wind, water motion) as outlined in [44]. Environmental effects are scripted for dynamic control. Dynamic objects employ three modeling approaches: 3D software-animated effects (e.g., helicopter rotors), editor-defined motion paths, and script-driven route management with pose updates. For segmentation annotations, each object’s auxiliary pass color is predefined in the scene editor. As an illustrative case, only moving cars are individually segmented based on their distinct color attributes.

In UNIGINE, cameras are created via Camera and viewport programming interfaces. Following the MCS model, camera poses in the world coordinate system are calculated through coordinate transformations, combining the vehicle’s pose and MCS installation parameters. Each viewport is manipulated by a corresponding Camera to render image data using the Unified UNIGINE Shader Language (UUSL) scripted pipeline, extracting depth (from the native depth buffer and converted to linear value), world-aligned normal (from the normal buffer), optical flow (from the velocity buffer) from the deferred G-buffer pass, and segmentation mask from the auxiliary pass. The final RGB images are captured via the same interfaces as the last results.

Synchronous data recording follows the pipeline in Algorithm  1. Raw data, including G-buffers (native depth, normal, velocity), auxiliary passes, and RGB images, are first captured from the GPU to RAM. Processed data is then saved to disk via a parallel thread, bypassing RAM-to-disk latency. Vehicle and object poses are recorded at each simulation step, alongside MCS configurations, automating dataset generation for visual perception research. During simulation, the rendering frame rate dynamically changes with the scale of geometry in the scene. For each simulation snapshot, all captured camera images are precisely synchronized in a rendering step, and the corresponding physical timestamp is recorded. If temporal alignment errors need to be simulated, only a single offline processing step is required. By setting equal sampling intervals to generate simulation timestamps, the temporal alignment error is defined as the deviation between these timestamps and the physical timestamps.

Algorithm 1 Image data generation and synchronous recording process.

1: while True do 2:     Update MCS Pose; 3:     Display first-person view using WidgetSpriteViewport API; 4:     if data recording is required then 5:         Parse rendering pipeline using Camera and Viewport APIs; 6:         Write UUSL scripts to 7:            (1) Extract native depth buffer and convert it to linear depth; 8:            (2) Extract normal buffer and unify surface normal coordinate system; 9:            (3) Extract velocity buffer and convert velocity map to optical flow; 10:           (4) Extract auxiliary render pass and generate segmentation mask; 11:        Transfer these image data from GPU into RAM using Camera and Viewport APIs; 12:        if image datasets in RAM are not empty then 13:            Start a standalone thread; 14:            while thread is active do 15:                Save image data from RAM to hard disk; 16:            end while 17:         else 18:            Wait for standalone thread cleanup; 19:        end if 20:        Record poses of vehicle and dynamic objects; 21:        if recording termination is requested then 22:            Record MCS configuration settings; 23:        else 24:            continue 25:        end if 26:     else 27:         continue 28:     end if 29: end while

3.4. UAV Motion Control and Closed-Loop Simulation

UAV motion control is achieved through two modes: (1) manual human–machine interaction through keyboard and mouse interfaces, and (2) external data-driven kinematic propulsion. For manual operation, flight commands are configured via designated keys, with aerial vehicle dynamics (fixed-wing and quadrotor models) integrated into the simulator. Real-time kinematic state propagation is computed using UNIGINE’s native physics engine, which processes user inputs through a force-feedback loop mechanism that accounts for inertial properties and environmental interactions. Alternatively, the simulator supports external data injection protocols where motion profiles (e.g., from JSBSim flight dynamics models or MATLAB/Simulink control algorithms) override internal kinematic calculations.

A critical challenge in achieving closed-loop simulation lies in coordinating data flow among software components, external programs, and hardware entities [45]. Efficient communication is ensured via RS422 serial ports for bidirectional hardware-simulator exchange, Socket interfaces for UNIGINE-Qt data transfer, and a dedicated image injection interface [45] for photoelectric data processing device. Robust communication protocols standardize data frames, composed of header, length, type, content, and checksum, to maintain data integrity. Temporal synchronization is enforced using the Network Time Protocol [46], guaranteeing all simulation components operating in accordance with the logical clock.

4. Functional Validation Through Simulation

In this section, the main functions of MCS-Sim are validated through three simulation approaches: (1) synthetic dataset generation using a customized MCS configuration; (2) visual perception algorithms testing; (3) closed-loop simulation through HIL.

For visual perception algorithms testing, the generated dataset samples are used to benchmark some representative visual perception algorithms (e.g., optical flow, scene flow, depth estimation, visual SLAM). The purpose is on demonstrating the practicality of the simulator, not introducing new algorithms.

For closed-loop simulation, firstly, the image injection function is verified by simulating multichannel image capture from an ACES and performing real-time stitching on the photoelectric data processing device. Secondly, end-to-end system verification is demonstrated via a practical HIL case: real-time ground target tracking using UAV electro-optical payloads. Simulated sensor feedback directly interfaces with embedded perception algorithms for closed-loop control validation.

4.1. Synthetic Dataset Generation

During image data generation, RGB images and multi-modal ground truth data (depth maps, segmentation masks, surface normals, optical flow) are extracted by parsing the rendering pipeline and post-processing. Camera and dynamic object poses are synchronously recorded. These raw data form the basis for deriving task-specific annotations for visual perception. For example, the stereo disparity is calculated using depth maps and camera calibration parameters. LiDAR point clouds are created by sampling depth values from virtual cameras with synthetic noise. Inertial measurement unit (IMU) data is generated from pose changes, incorporating noise models. Occlusion masks are defined using geometric constraints. While similar approaches exist in synthetic datasets (e.g., TartanAir [12], Spring [47]), producing these annotations significantly increases development effort. MCS-Sim prioritizes adaptability: its modular design allows for rapid reconfiguration of MCS and other simulation settings to match experimental requirements. To maintain focus, raw unprocessed datasets generated through our pipeline are enough for testing some representative visual perception algorithms.

Generally, an unstructured MCS comprising four cameras is customized as depicted in Figure 3, with the detailed configuration settings outlined in

Table 3. Detailed configuration settings of the customized MCS.

Camera	Focal Length (mm)	Image Plane Width (mm)	Image Resolution	Orientation (Axis-Angle)	Position Offset (mm)
0	20	20	$720\times 640$	$\left(0,0,0\right)$	$\left(0,0,300\right)$
1				$\left(0,-0.524,0\right)$	$\left(-150,0,259.808\right)$
2				$\left(0,-1.047,0\right)$	$\left(-259.808,0,150\right)$
3				$\left(0,0,0\right)$	$\left(500,0,300\right)$

. These cameras lie on a single plane, with cameras 0 through 2 forming a simple cylinder ACES with a radius of 30 cm. Besides these, cameras 0 and 3 act as a binocular stereo camera with a baseline of 50 cm. The combination of the two types of structures enables this MCS configuration to cover most autonomous driving visual perception and stereo vision applications. Following the convention established by other synthetic datasets, we omit calibration and temporal misalignment errors in this configuration. This approach ensures test results can be accurately compared against ground-truth data.

According to the virtual MCS, a sample dataset with six sequences is generated using MCS-Sim. This dataset encompasses three scenarios: uptown, warehouse, and forest environments, as depicted in Figure 4. Each scenario includes sequences with forward and loop motion patterns. The lighting changes in these scenes are slightly different, and a rainy day environment (Warehouse2) is also set up. The demo dataset is available for download at https://github.com/qqm-666/MCS-Sim (accessed on 1 September 2025). A more extensive dataset will be released at the same repository in the future.

4.2. Visual Perception Algorithms Testing

In this subsection, the dataset samples generated as described in Section 4.1 are used to benchmark a selection of representative visual perception algorithms. For the purpose of general applicability, within the domains of optical flow or scene flow, depth estimation, and visual SLAM, only two typical methods from each domain are subjected to zero-shot evaluation. This evaluation aims to demonstrate the practical utility of the simulator, rather than focusing on introducing novel algorithms.

4.2.1. Optical Flow and Scene Flow Estimation

Optical flow represents 2D pixel motion between consecutive image frames, while scene flow describes 3D displacement of points in spatial scenes. Both are vital for tasks like object detection, object tracking, action recognition, facial expression analysis, among others. Dense motion flow estimation has been applied in autonomous driving and intelligent robot systems. Two representative methods, RAFT [48] (Optical flow) and RAFT-3D [49] (Scene flow), were evaluated using the generated dataset.

For RAFT [48], the authors’ provided fine-tuned model (raft-kitti.pth) was used to evaluate it. Metrics computed included End Point Error (EPE or EPE2D) and F1-all, with results shown in

Table 4. Experimental results of RAFT and RAFT-3D.

Sequence	RAFT (Optical Flow)		RAFT-3D (Scene Flow)
	EPE2D	F1-All	EPE2D	F1-All	EPE3D	${\mathbf{\delta }}_{3\mathbf{D}}\mathbf{<}$ 5 m	${\mathbf{\delta }}_{3\mathbf{D}}\mathbf{<}$ 10 m
Uptown1	0.595	2.265%	0.590	1.323%	15.126	86.341%	91.232%
Uptown2	1.210	3.716%	1.252	3.454%	9.424	88.079%	92.366%
Warehouse1	1.688	2.491%	0.687	2.223%	2.334	94.284%	98.557%
Warehouse2	1.525	4.757%	1.453	4.097%	3.130	95.479%	98.171%
Forest1	3.182	21.281%	1.929	8.464%	27.215	70.993%	76.922%
Forest2	0.924	4.935%	1.062	5.989%	5.506	75.326%	86.422%

The projected optical flow of RAFT-3D is compared with that of RAFT. Bold font highlights the optimal term under the same evaluation metric. The unit of EPE_2D is pixel, and the unit of EPE_3D is meters.

. During testing, the number of iterations was set to 24 for the recurrent unit, and only the optical flow for camera 0 was estimated. Figure 5 demonstrates qualitative optical flow predictions from our dataset.

The model demonstrates strong generalization to unseen synthetic scenarios despite lacking dataset-specific training. For the Forest1 test sequence, performance drops (EPE is 3.182, F1-all is 21.281%) due to dense tree proximity, which creates substantial pixel displacement and matching ambiguity. Other sequences show better results. Weather adaptability is evident in the rainy condition of Warehouse2, maintaining consistent metrics with Warehouse1. However, optical flow estimation at object boundaries proves problematic in wooded environments (Forest1 and Forest2), caused by repetitive leaf textures and absence of dataset fine-tuning.

RAFT-3D [49] is configured with bi-Laplacian smoothing, 16 iterations for the recurrent unit, and a fine-tuned model (raft3d_kitti.pth) as recommended by the authors. Cameras 0 and 3 are selected as binocular stereo inputs, separated by a 50 cm baseline. Stereo matching is performed via GA-Net [50] to generate disparity maps, followed by RAFT-3D inference. Valid pixels are defined as those with depth less than 250 m and optical flow less than 250 pixels. Evaluation metrics align with RAFT, including EPE2D and F1-all, with an added EPE3D. Threshold metrics ${\delta }_{3\mathrm{D}}<$ 5 m and ${\delta }_{3\mathrm{D}}<$ 10 m measure pixels within specified error bounds. Results are summarized in Table 4, with qualitative comparisons in Figure 5.

Table 4 shows that RAFT-3D outperforms RAFT in optical flow accuracy across most sequences, as indicated by bold values. This improvement stems primarily from incorporating binocular disparity, except in the Forest2 sequence. The most pronounced gains occur in Forest1, where low-altitude forest traversal creates texture-poor RGB imagery that challenges optical flow algorithms. For RAFT-3D, depth information enhances structural awareness, thereby improving accuracy. However, Figure 5 reveals significant sky-region flow errors in RAFT-3D, though these pixels are excluded from evaluation. This arises from the inaccurate sky matching of GA-Net [50], traceable to pre-training datasets (Flyingchairs [51], Flyingthings3D [52]) lacking sky content. Fine-tuning on KITTI [53,54] datasets, which also lack sky annotations, fails to resolve this limitation.

4.2.2. Depth Estimation

Vision-based 3D perception offers a cost-effective alternative to expensive depth sensors like LiDAR by leveraging rich semantic information. Depth estimation acts as a critical link between 2D image and 3D environmental understanding, driving progress in downstream applications. MCS-based egocentric depth estimation particularly benefits 3D reconstruction and autonomous driving sensing. Two surround MCS methods are evaluated using the generated dataset: SurroundDepth [55] and R3D3 [56].

SurroundDepth [55] is evaluated using the source code with identical settings as the DDAD dataset [57] implementation. Fine-tuned models (ddad_scale/encoder.pth and ddad_scale/depth.pth) are directly applied. During inference, the batch size is set to 4, processing four synchronized camera inputs per frame.

Table 5. Experimental results of SurroundDepth and R3D3.

Sequence	AbsRel		SqRel		RMSE		${\mathit{\delta }}_{1.25}$
	SD	R3D3	SD	R3D3	SD	R3D3	SD	R3D3
Uptown1	0.288	0.295	4.147	4.173	14.654	11.443	0.496	0.495
Uptown2	0.259	0.431	4.549	10.286	15.916	17.376	0.568	0.361
Warehouse1	0.225	0.437	3.718	12.699	14.770	20.153	0.597	0.397
Warehouse2	0.243	0.440	2.778	9.038	10.942	13.064	0.585	0.426
Forest1	0.303	0.424	4.677	9.303	15.017	12.561	0.521	0.404
Forest2	0.203	0.479	3.230	15.439	13.687	25.950	0.600	0.257

SurroundDepth is abbreviated as SD. Bold font highlights the optimal term under the same evaluation metric. The unit of RMSE is meters.

presents quantitative results within a 100 m range, evaluated via Absolute Relative Error (AbsRel), Squared Relative Error (SqRel), Root Mean Squared Error (RMSE), and Accuracy with Threshold ${\delta }_{1.25}$. Qualitative comparisons on our dataset are shown in Figure 6.

For R3D3 [56], the fine-tuned models offered by authors (completion_ddad.ckpt and r3d3_finetuned.ckpt) are used with all other parameters fixed. Each sequence starts with 3 warmup frames, applying a mean flow threshold of 1.75 for filtering. The co-visibility graph is configured with $\Delta t_{intra}=3$, $r_{intra}=2$, $\Delta t_{inter}=2$, and $r_{inter}=2$. Evaluation follows the same metrics as DDAD, with scale-aware results within 100 m shown in Table 5. Qualitative comparisons appear in Figure 6.

SurroundDepth demonstrates better overall accuracy than R3D3. However, both methods exhibit reduced precision when benchmarked against original paper results, revealing limited generalization. Figure 6 shows weakened edge perception for these two methods, particularly in repetitive-texture vegetation areas. This stems from their self-supervised training relying on sparse LiDAR data rather than the dense pixel-wise annotations as MCS-Sim. The training datasets, DDAD [57] and nuScenes [58], focus on road scenes, differing significantly from the diverse objects and overhead squint perspectives of the generated dataset. These domain discrepancies contribute to the observed performance degradation.

4.2.3. Visual SLAM

SLAM aims to construct a map of an unknown environment and precisely locate sensors within this map, with a heightened emphasis on real-time functionality. Visual SLAM solutions, in which the main sensors are cameras, hold significant appeal due to the low cost of cameras and the relative sensitivity to texture information. This facilitates robust and precise self-positioning, as well as the perception of a 3D structure. For the generated dataset, two representative methods, ORB-SLAM2 [59] and Stereo DSO [60], are selected as the subjects for this testing. Additionally, the localization implementation of the multi-camera algorithm R3D3 [56] is similar to that of a visual odometry system, except that it lacks real-time capability. Its trajectory is also included in the evaluation.

ORB-SLAM2 [59] is an open-source visual SLAM framework supporting monocular, stereo, and RGB-D cameras. It includes loop closure, relocalization, and map reuse, requiring descriptor computation for feature points with re-projection error optimization. In contrast, Stereo DSO [60] optimizes photometric errors, eliminating descriptor calculations for faster processing. Both methods represent conventional visual SLAM approaches. In our implementation, a stereoscopic camera using cameras 0 and 3 (baseline 50 cm) is implemented, adopting default parameters from the source code. The performance evaluation employs Absolute Trajectory Error (ATE) and Relative Pose Error (RPE). Both ATE and RPE use only the translation component, and their units are in meters. Quantitative results appear in

Table 6. Evaluation results of ORB-SLAM2, Stereo DSO, and R3D3.

Sequence	ATE			RPE
	ORB-SLAM2	Stereo DSO	R3D3	ORB-SLAM2	Stereo DSO	R3D3
Uptown1	0.180	0.142	9.657	0.012	0.007	0.085
Uptown2	0.369	1.690	7.250	0.022	0.144	0.165
Warehouse1	0.124	0.019	6.935	0.047	0.005	0.154
Warehouse2	(1.786)	(4.748)	1.835	(0.182)	(0.731)	0.095
Forest1	(0.116)	(0.517)	3.615	(0.152)	(0.185)	0.048
Forest2	0.336	0.086	7.676	0.035	0.032	0.082

Parentheses ( ) denote an unexpected termination of the tracking process, with only valid frames being considered for evaluation. Bold font highlights the optimal term, underlined font indicates the suboptimal term, and a continuous, uninterrupted tracking is preferred over a tracking that is disrupted.

, with trajectory visualizations in Figure 7.

Under conventional motion patterns with small viewpoint changes (Uptown1, Warehouse1, Uptown2, and Forest2), all three algorithms can track the trajectory completely. Among traditional methods, ORB-SLAM2 [59] generally achieves higher accuracy than Stereo DSO [60] in nonlinear trajectory Uptown2 due to its loop-closure and relocation modules. An exception occurs in Forest2, where repetitive vegetation textures lack distinct features, causing ORB-SLAM2 to fail. Here, Stereo DSO [60] provides significantly better accuracy when loop-closure mechanisms are ineffective (Uptown1, Warehouse1, and Forest2). Notably, Stereo DSO requires a longer initialization period than ORB-SLAM2, as observed in repetitive textures and rainy scenes (Forest1, Forest2, and Warehouse2). R3D3 [56] exhibits larger-scale errors and fails to leverage the advantages of MCS.

Traditional binocular SLAM methods outperform R3D3 [56] in new scenes, primarily due to the generalization limits of deep learning models and the restricted FOV overlap in our MCS configuration. However, in challenging environments, such as rainy conditions (Warehouse2) and dense forests (Forest1), R3D3 [56] demonstrates greater robustness, maintaining positioning accuracy and full trajectory tracking. This advantage emerges when R3D3 [56] uses dense flow fields to refine inter-frame and intra-frame correspondences, compensating for sparse feature limitations in traditional SLAM. Additionally, the training data includes complex environments and diverse motion patterns with significant view changes.

4.3. Closed-Loop Simulation

To implement closed-loop simulation for visual perception algorithms through HIL, the image injection interface is a crucial element. As a means of functional verification, wide FOV imaging is simulated by capturing multichannel images from a virtual ACES that performs real-time stitching on the photoelectric data processing device. Furthermore, closed-loop system testing is demonstrated through a paradigmatic example of real-time ground target tracking using UAV electro-optical payloads. Simulated vision sensor feedback interfaces directly with embedded perception algorithms for closed-loop control validation. The two simulation examples are presented to demonstrate functionality validation, rather than to evaluate specific algorithms or software implementations.

Taking the closed-loop simulation of a UAV system as an example, its key information flow is illustrated in Figure 8. In the figure, solid lines represent the data stream between hardware entities, solid-arrow dashed lines denote the serial communication data stream between the simulator and hardware entities, and hollow-arrow dashed lines indicate the image injection data stream. The simulated visual sensor injects rendered images in real time into the photoelectric data processing device. Simultaneously, it sends status parameters to both the photoelectric data processing device and the flight control device. After calculations by the embedded visual perception algorithm, the photoelectric data processing device feeds the processed information back to the flight control device and the ground station. It also transmits the video stream back to the ground station for display. The ground station sends operational control commands to the flight control device. Subsequently, the flight control device performs flight control calculations based on the control commands from the ground station, the measured position, attitude, and velocity of the UAV body, as well as the status of the visual sensor. It then outputs a control message to the simulator to update the pose of the UAV and its vision sensor. Additionally, the photoelectric data processing device sends a control message to the simulator to update the status of the vision sensor.

4.3.1. Functional Verification of Image Injection

MCS-based computational imaging has emerged as a critical method for wide FOV, high-resolution image reconstruction in photoelectric systems [61]. This approach overcomes the inherent limitation of the space bandwidth product in monocular imaging systems and demonstrates proven effectiveness in aerial applications. As an illustrative example, the calibration of both intrinsic and extrinsic parameters for a spherical ACES (Figure 9a) is undertaken using a random noise calibration pattern [62]. The ACES configuration (

Table 7. Detailed configuration settings of the spherical ACES.

Camera	Focal Length (mm)	Image Plane Width (mm)	Image Resolution	Roll Angle (°)	Pitch Angle (°)	Radius (mm)
0	$3.240$	$5.568$	$640\times 360$	$0.0$	$0.0$	$170.0$
1				$0.0$	$56.2$
2				$41.93$	$55.0$
3				$88.87$	$53.5$
4				$132.52$	$53.0$
5				$183.82$	$53.0$
6				$223.16$	$52.0$
7				$268.42$	$54.5$
8				$314.32$	$55.0$

) is then input into MCS-Sim for simulation.

In this example, the objective is to verify the image injection function. The ACES-specific image stitching algorithm is pre-installed on the hardware entity. The simulator generates ACES image data and combines them into a single image with a resolution of 1920 × 1080 in real time (Figure 9b). This combined image is then immediately injected into the hardware entity. Each image set undergoes seamless stitching on-chip, producing a wide FOV, high-resolution composite image, as depicted in Figure 9c. The wide FOV image is then sent back to the ground station for display.

4.3.2. Closed-Loop System Testing

Closed-loop simulation creates a real-time feedback loop between virtual environments and tested systems. To validate the closed-loop simulation capabilities of the simulator, we conducted a comprehensive simulation of a small fixed-wing UAV equipped with a monocular camera, performing target recognition, tracking, and motion following.

Prior to testing, the photoelectric data processing device is loaded with the following vision algorithms: YOLOv3 [63] for target detection, SORT [64] for data association, and GOTURN [65] for single target tracking. Initially, YOLOv3 and SORT work in tandem to detect multiple objects, with SORT correlating YOLOv3 detection results frame by frame. Subsequently, GOTURN is employed to track the target that SORT identifies with the highest correlation confidence.

The flight control is managed on a hardware entity, which adjusts the flight path based on the output from the photoelectric data processing device and inputs the control instructions into the simulator for further simulation. During the simulation, the UAV operates at an altitude of 200 m with a flight speed of 25 m/s. Upon recognizing the ground target (using an aircraft as an example), it initiates target tracking and modifies its flight path to follow the target accordingly, as shown in Figure 10.

5. Conclusions and Future Work

We introduce a photo-realistic simulator, MCS-Sim, designed to address challenges in multi-camera UAV visual perception research. This simulator integrates multi-camera system simulation, UAV motion simulation, dynamic scene simulation, synthetic dataset generation, and a closed-loop testing interface into a unified platform. Consequently, it supports the prototype design of multi-camera systems, the preparation of synthetic datasets, and the closed-loop verification of the system. The functionality of MCS-Sim was verified through simulation experiments, demonstrating its significance in advancing multi-camera UAV visual perception research.

The future work will focus on three key directions to establish MCS-Sim as a versatile platform for next-generation AI research. (1) Enhancing existing functionalities: We will incorporate supported capabilities, such as dense flow estimation, 3D reconstruction, visual SLAM, object detection, and tracking, into the simulator to better facilitate algorithm verification processes. Additionally, we will expand the vehicle models, scene libraries, physical simulation fidelity, and software optimization to enhance the simulator’s capabilities and expand its application scope. (2) Developing novel capabilities: We will introduce support for autonomous movable entities and embodied AI systems, along with automated dataset generation pipelines. These additions will broaden the applicability of MCS-Sim to a wider range of research scenarios. (3) Integrating cutting-edge techniques: We will integrate advanced techniques such as neural rendering (e.g., NeRF, 3DGS) and comprehensive world models to advance the simulator’s capabilities. However, these advancements will require addressing challenges related to computational costs.