UGV Swarm Multi-View Fusion Under Occlusion: A Graph-Based Calibration-Free Framework

Highlights

What are the main findings?

• A calibration-free, end-to-end graph-based framework is proposed for joint camera and subject registration in UGV swarms, operating robustly under severe occlusion and low inter-view co-visibility.
• A graph-based pose propagation module (GPPM) is proposed that enables global alignment via BFS-guided pose propagation along local co-visibility links, eliminating the need for full co-visibility with a root node or pre-calibrated extrinsics.

What are the implications of the main findings?

• Despite being a purely vision-based solution, the method enables infrastructure-free perception in GPS-denied or complex environments by decoupling multi-view fusion from explicit agent pose estimation, therefore supporting dynamic deployment of UGV swarms with a central node for robust collaborative perception tasks.
• The method enables robust BEV scene representation and collective situational awareness for UGV swarms, even when inter-camera overlap is minimal and occlusions are severe.

Abstract

In unmanned ground vehicle (UGV) swarm systems, comprehensive environmental awareness is critical for coordinated operations. Yet they are frequently deployed in occlusion-rich, constrained environments where multi-agent visual fusion is essential. However, existing methods are critically limited by offline-calibrated extrinsic parameters, hindering flexible deployment, and by a strong co-visibility assumption, which fails under severe occlusion. To overcome these constraints, we introduce an end-to-end, calibration-free framework for the joint registration of cameras and subjects. Our approach begins with a single-view module that estimates subjects’ poses and appearance features. Subsequently, a novel graph-based pose propagation module (GPPM) treats UGVs’ cameras as nodes in a graph, connecting them with edges when they share co-visible subjects identified via appearance matching. Breadth-first search (BFS) then finds the shortest registration path from any camera to a designated root camera, enabling pose propagation via local co-visibility links and global alignment of all subjects into a unified bird’s-eye-view (BEV) space. This strategy relaxes the stringent requirement of full co-visibility with the root node. A multi-task loss function is proposed to jointly optimize pose estimation and feature matching. Trained and evaluated on a synthetic dataset with occlusions (CSRD-O) collected by a UGV swarm system, our framework achieves mean camera pose errors of 1.57 m/8.70° and mean subject pose errors of 1.40 m/9.14°. Furthermore, we demonstrate a scene monitoring task using a UGV swarm system. Experiments show that the proposed method generates robust BEV estimates even under severe occlusion and low inter-view overlap. This work presents a purely visual, self-calibrating multi-view fusion perception scheme, demonstrating its potential to support cooperative perception, task-oriented monitoring, and collective situational awareness in UGV swarm systems.

1. Introduction

In the field of unmanned swarm systems, achieving a comprehensive understanding of the surrounding environment is paramount for coordinated operations. Unmanned aerial vehicle (UAV) systems often benefit from an aerial perspective, rendering them relatively less susceptible to occlusions caused by ground-level obstacles. Conversely, unmanned ground vehicle (UGV) systems are frequently deployed in complex environments where their field of view is predominantly constrained by occlusions. Fusing visual information from multiple agents is highly valuable, as it overcomes the limitations of single-view perception, such as occlusion and a narrow field of view, which are prevalent in complex real-world scenarios. By integrating subject-level data across multiple views, a unified bird’s-eye-view (BEV) representation can be constructed. This unified representation, where each subject’s location and orientation are precisely annotated, serves as the foundational “shared brain” for swarm systems. It enables critical downstream tasks, including formation control and collaborative decision-making. Consequently, robust multi-view fusion has shown great promise in diverse domains such as search-and-rescue operations, urban surveillance, and autonomous driving [1,2,3,4,5,6,7].

Multi-view subject detection has emerged as a critical capability for enhancing situational awareness in unmanned swarm systems [8,9,10,11]. Effective collaboration within such systems relies on the fusion of visual information from multiple agents, enabling coherent, large-scale environmental perception. A common deployment strategy involves agents distributed around the perimeter of an operational area, each contributing a local view toward a shared region of interest. In these scenarios, conventional approaches typically assume that the relative poses among cameras are static or known a priori. These extrinsic parameters are usually obtained through offline calibration or external localization systems, which facilitate feature alignment and cross-view association. However, this reliance on precise pose priors introduces significant limitations, as it couples system performance to the robustness of external localization modules and hinders rapid deployment in GPS-denied or structurally ambiguous environments.

Most existing multi-view subject detection methods critically rely on precisely calibrated extrinsic camera parameters obtained through offline procedures. Consequently, they exhibit significant performance degradation when camera poses change due to unmanned swarm deployment shifts or dynamic operations. Frequent recalibration is often impractical in real-world scenarios, limiting the application of pure vision fusion in such systems [12,13]. A calibration-free solution that jointly estimates camera and subject poses directly from raw images to produce BEV representations is thus highly desirable. The existing method [14] addresses this by using monocular depth estimation and appearance-based matching to identify co-visible subjects as pseudo-calibration objects, leveraging 3D pose uniqueness for joint registration. Typically, one camera is designated as the ‘root node’ to define the global BEV frame. However, this method relies on a strong co-visibility assumption: every non-root camera must share co-visible subjects with the root to enable alignment. This requirement becomes overly restrictive in occluded scenes or when fields of view are limited, as it is often impractical to guarantee that all agents observe common targets. In typical operation scenarios for unmanned systems, such as indoor or confined environments, obstacles frequently break this co-visibility condition, causing the fusion scheme to fail.

To address this limitation, we propose an end-to-end, calibration-free framework for joint camera and subject registration under occlusion. Our key aim is to formulate UGVs’ camera pose registration as a graph-based information propagation problem. First, a single-view module extracts subject poses and appearance features in each UGV’s local camera frame. Then, a graph-based pose propagation module (GPPM) treats registration as a graph optimization task and constructs a graph dynamically where cameras serve as nodes and edges are established based on co-visible subjects between views. By performing breadth-first search (BFS) from a designated root node, the GPPM identifies the shortest path for propagating pose information, enabling chained, local-to-global alignment through sequential coordinate transformations. This allows global registration to be achieved even when only localized co-visibility exists between neighboring UGVs. Both synthetic and real-world occlusion scene datasets with multi-UGV views are constructed to train and validate the framework. An efficient multi-task loss is designed to jointly optimize pose estimation and appearance matching. Our method only requires local co-visibility between neighboring views to achieve global registration, thereby relaxing the high co-visibility constraint in the existing method. Crucially, our method decouples multi-view fusion from explicit agent pose estimation, enabling cooperative perception without prior knowledge of agent positions, orientations, or inter-robot spatial relationships. This reduces reliance on external infrastructure and enhances adaptability for UGV systems in dynamic, GPS-denied, or structurally ambiguous environments.

The primary contributions are enumerated below:

• A calibration-free, end-to-end joint camera and subject registration framework is proposed, which obviates the reliance on pre-calibrated extrinsic parameters and operates robustly under severe occlusion and low inter-camera co-visibility.
• A graph-based camera propagation mechanism is proposed to achieve global alignment, where cameras serve as nodes and are connected via co-visible appearance-matched subjects to form a graph. BFS-guided shortest-path propagation from a root camera enables chained pose alignment under local co-visibility, relaxing the full co-visibility assumption of prior work and suppressing error accumulation.
• A synthetic and a real scene dataset collected using a UGV swarm system are presented, both featuring multi-view occluded scenes with precise annotations for camera and subject poses. A multi-task loss that jointly considers camera pose regression, appearance similarity, cross-view consistency, and subject pose regression is proposed, enabling end-to-end optimization of our joint camera and subject registration framework.

2. Related Work

In multi-view fusion perception, most existing methods are based on CNNs or Transformer architectures [15,16,17]. These approaches typically extract object features from each view, apply perspective transformations at the feature level to map them onto a unified BEV plane, and then perform regression under supervision from ground-truth annotations. For example, MVDet [18] proposes an end-to-end trainable multi-view detection model. It introduces anchor-free view feature aggregation and a convolutional spatial aggregation mechanism based on large convolutional kernels, enabling effective pedestrian detection in crowded scenes. MVDetr [19] employs a novel Shadow Transformer-based multi-view detector. It replaces conventional convolutions with a position- and view-adaptive attention mechanism for multi-view feature aggregation. In addition, it incorporates a data augmentation strategy that preserves cross-view consistency, leading to improved pedestrian detection performance. SHOT [20] establishes 3D correspondences from multi-view images to a BEV occupancy map by approximating 3D projections using a learnable stack of homographies, thereby enhancing the accuracy of 3D coordinate estimation. 3DROM [21] addresses the issue of data insufficiency by generating random 3D cylindrical occluders on the ground plane for data augmentation and fuses height information by projecting multi-view features onto various height planes using homography transformations. This approach effectively mitigates overfitting caused by limited annotated data. MVFlow [22] introduces a novel paradigm that directly predicts spatiotemporally consistent motion trajectories to infer pedestrian positions within frames, replacing the traditional detect-then-track approach. M-MVOT [23] proposes a Mahalanobis distance-based multi-view optimal transport loss function. By integrating line-of-sight direction-guided ellipse costs, depth-aware weighting, and nearest-view fusion strategies, it enhances the accuracy of crowd localization in crowded scenes. EarlyBird [24] simplifies cross-view association to pure temporal association and learns robust re-identification features, achieving multi-view multi-target tracking in BEV. TrackTacular [25] presents a unified framework that merges multi-view and multi-temporal features, considering appearance and motion cues, and can predict both pedestrians and vehicles. TMVD [26] infers pedestrian locations using CNNs by performing weighted embedding of multi-view features in each cell of a discretized ground plane. MVUDA [27] improves model generalization across different camera configurations through an enhanced mean teacher self-training framework and pseudo-labeling strategies tailored for multi-view pedestrian detection. MSASW [28] introduces a multi-view pedestrian detection architecture based on scale-aware features and shared-weight supervision, which adaptively encodes contextual scale information and efficiently fuses single-view supervisory signals. DSM [29] achieves superior multi-target cross-camera association performance in both synthetic and real-world scenarios by establishing geometric consistency constraints between multiple views. However, these methods all rely on known camera extrinsic parameters to project 3D coordinates onto 2D planes, facilitating feature mapping from image space to a unified BEV space. This reliance makes them less adaptable to flexible deployment scenarios involving multiple cameras with varying poses.

In unmanned systems, BEV perception provides an intuitive representation of spatial structures, making it well-suited as a common reference frame for integrating data from multiple sensors. This is beneficial for scene understanding, path planning, and control in unmanned system operation algorithms. BEV fusion stands out as one of the most reliable perception methods in the field of unmanned systems [30]. For instance, the notable LSS [31] lifts each image individually, transforms it into a feature frustum corresponding to the camera’s viewpoint, and subsequently splats all frustums onto a rasterized BEV grid. BEVFormer [32,33,34] introduces a unified BEV representation framework based on transformers, effectively merging the spatiotemporal information from multi-camera images through spatial cross-attention and temporal self-attention mechanisms, thereby enhancing awareness performance. BEV-TS [35] improves the detection accuracy for small targets by employing multi-scale feature enhancement and causal inference mechanisms. Ref [36] compensates BEV perception with LiDAR features, improving object detection performance across multiple views by integrating image and LiDAR voxel and point cloud features. BevDrive [37] merges camera and LiDAR information, proposing an end-to-end framework for unmanned systems that integrates BEV perception with planning. DenseBEV [38] enhances object perception capabilities via multi-view multi-scale cross-attention mechanisms and contrastive denoising training strategies. While these methods have significantly advanced BEV perception in unmanned systems, they share a common limitation. Reliable BEV perception often relies on sensor fusion, most commonly between cameras and LiDAR, and increasingly incorporates additional modalities to improve robustness. However, such multi-sensor setups incur high hardware and integration costs [39]. Purely vision-based approaches offer a lower-cost alternative but remain critically dependent on pre-calibrated extrinsic parameters to project features from multiple views into a unified BEV space. Moreover, any change in camera configuration necessitates repeated calibration to maintain performance, which further increases operational complexity and deployment cost.

In camera pose registration, traditional approaches rely on high-precision calibration boards and offline calibration procedures. Recently, significant progress has been made in online calibration. For instance, BEVSOC [40] proposes a 3D object detection method that eliminates the need for explicit camera calibration. It implicitly learns the mapping from multi-view images to BEV space through a group-equivariant convolutional network and a contrastive learning-driven projection module, thereby removing dependence on extrinsic parameters and enhancing long-term system stability. Ref [41] introduces an automatic online multi-camera calibration and refinement method. It addresses the problem of local minima caused by large initial extrinsic errors using a coarse-to-fine, offset-enabled stochastic search strategy. Calibration accuracy is further improved by integrating adaptive image binarization weighting and a road segmentation module. Meta-Calib [42] achieves high-precision, general-purpose calibration for arbitrary multi-camera setups by combining a specially designed calibration target with a two-stage deep learning-based detection and ellipse fitting pipeline, followed by an iterative optimization strategy. Ref [43] presents a spatiotemporal joint online multi-camera calibration approach. It separately estimates each camera’s pose using spatial and temporal cues, then fuses both estimates to refine the extrinsic parameter, enabling high-accuracy alignment without calibration boards, even across non-overlapping fields of view. Ref [44] dynamically estimates the transformation between cameras and the road under non-rigid motion by performing coarse-to-fine geospatial feature extraction and incorporating wheel odometry information within a nonlinear optimization framework. Ref [45] employs a neural network to predict camera extrinsics in real time. By integrating multi-level, multi-scale image encoding, dense temporal fusion, and an efficient BEV encoder, this method demonstrates robustness to mounting misalignments and calibration inaccuracies. However, these approaches typically rely either on structured scenes with rich texture and spatiotemporal cues to enable implicit projection learning or on auxiliary sensors to assist calibration. Moreover, they generally lack end-to-end co-optimization with downstream perception tasks, limiting their adaptability and overall system efficiency.

3. Methodology

As shown in Figure 1, the proposed framework for joint camera and subject registration is based on monocular pose estimation and appearance feature matching, eliminating the need for prior calibration of inter-camera extrinsic parameters. The framework consists of four main components: (1) a single-view module extracts subject poses and appearance feature vectors in each camera’s local coordinate system; (2) GPPM performs camera graph construction and BFS-guided camera pose propagation by first identifying co-visible subjects between cameras through appearance feature matching and then using these subjects to search for the shortest registration path from any camera to the root node; (3) a camera and subject pose registration strategy aligns all observations into a common global frame; and (4) a multi-task loss function enables end-to-end joint optimization of the entire pipeline.

3.1. Single-View Module

The single-view module generates predictions of subject location and orientation, along with appearance features, for each pedestrian in camera view. Specifically, PifPaf [46] is first applied to detect human poses, yielding 17 keypoints and a bounding box for each pedestrian. PifPaf was selected due to its high accuracy in human pose estimation, a capability attributed to its training on both the COCO and CrowdPose datasets. The bounding boxes are cropped and fed into a ResNet backbone to extract the appearance feature vector $f_c^i\in {ℝ}^d$ for the i-th subject in view c. ResNet was adopted due to its proven robustness and widespread adoption as a standard feature extractor in computer vision. Monocular pose estimation is performed using LocoNet [14], producing the subject pose $p_c^i=\left(x_c^i,y_c^i,r_c^i\right)$ in the camera’s local coordinate system, which includes the 2D ground-plane coordinates and a heading angle. LocoNet is a deep neural network designed for pedestrian pose regression. It is well-suited to take the 2D keypoints generated by PifPaf as input and output the pedestrian’s planar position and orientation. The network employs multilayer perceptrons (MLPs) combined with residual structures to enhance feature extraction. It comprises fully connected layers, batch normalization, ReLU activation functions, and dropout layers. The final layer employs a simple fully connected head to directly regress the pedestrian pose. The structure of LocoNet is illustrated in Figure 2. All outputs are stored as lists $\left\{{\mathcal{P}}_c\right\}$, $\left\{{\mathcal{F}}_c\right\}$ for c = 1, …, 5.

3.2. Graph-Based Pose Propagation Module

To achieve global consistency under low co-visibility, we dynamically construct the graph topology based on appearance matching. Specifically, the spatial uniqueness of subject pose allows co-visible subjects across two views to serve as natural calibration targets, enabling relative camera pose estimation via a 2D rigid transformation. In a multi-camera system, one camera is designated as the global reference and referred to as the “root node” in graph-theoretic terms. Prior work has computed pairwise relative poses from each non-root camera to the root based on co-visible subjects, aligning all pose estimates into the global coordinate system. This works under high inter-camera field overlap and no occlusion. However, with obstacles or limited fields of view, a camera may share no co-visible subjects with the root, making joint registration infeasible.

As we have observed, 2D pose transformations are transitive—relative poses between cameras can be computed chain-wise through intermediate views. To relax the stringent high co-visibility requirement, we propose GPPM, comprising two components: camera graph construction and BFS-guided camera pose propagation. First, co-visibility between cameras is determined by matching appearance feature vectors of detected subjects across views. If co-visible subjects exist, a transformable relationship is established, and an edge is added between the corresponding camera nodes. Each camera serves as a node, and an edge connects two nodes if they share at least one co-visible subject. We then apply BFS to find the shortest pose transformation path from any non-root camera to the root. Under high co-visibility, this yields a direct transformation. Under low co-visibility, BFS identifies an indirect path through one or more intermediate cameras to enable pose alignment. By performing a level-by-level search, BFS ensures the path is shortest, thereby minimizing error accumulation from chained transformations. This process is illustrated in Figure 3. Our core theoretical hypothesis posits that inter-camera geometric connectivity can be inferred directly from the co-visible subjects. The camera graph is constructed dynamically based on the detection of co-visible subjects across camera views, rather than being predefined by sensor data or spatial proximity.

The multi-view images should be time-synchronized. This synchronization is essential to guarantee the spatial uniqueness of subject poses across different views. Without time synchronization, co-visible pedestrians could not serve as reliable natural calibration targets between cameras. We concatenate the appearance feature vectors $\left\{{\mathcal{F}}_c\right\}$ along the instance dimension to form a matrix $F=\left[{\mathcal{F}}_1;\dots ;{\mathcal{F}}_5\right]\in {ℝ}^{N\times d}$, and then compute its self-similarity matrix. The similarity is calculated as follows:

$$M_{row,col}=1-\sigma \left(D_{row,col}\right)$$

(1)

where $\sigma \left(\cdot \right)$ denotes the sigmoid function and $D^{\mathrm{sim}\_\mathrm{pred}}=\left[D_{row,col}\right]\in {ℝ}^{N\times N}$ is the feature distance matrix.

Similarities greater than a threshold ${\tau }_{\mathrm{sim}}$ are considered indicative of co-visible subjects between two cameras. We apply binarization to obtain a similarity mask $M^{\mathrm{sim}\_\mathrm{mask}}$. If $M_{c_a,c_b}^{\mathrm{sim}\_\mathrm{pred}}=1$, an edge $\left(c_a,c_b\right)$ is added to the camera graph. Then a graph structure $\mathcal{G}=\left(\mathcal{V},\mathcal{E}\right)$ representing the registrable relationships between cameras is established.

The pose relationship between non-root cameras and the root node (with camera c₁ designated as the root node) can be expressed as follows:

$$T_{c_1\leftarrow c_j}=\left\{\begin{array}{cc}{T}_{c_1\leftarrow c_j}& \mathrm{direct} \mathrm{transformation}\\ T_{c_1\leftarrow c_k}\circ \dots \circ T_{c_k\leftarrow c_j}& \mathrm{chained} \mathrm{transformation}\end{array}\right.$$

(2)

Camera poses are computed from subject poses predicted by the single-view module, making the process differentiable. Although GPPM operates discretely by constructing the graph structure and searching for pose transformation paths, it does not participate in numerical computation and thus does not affect differentiability. The established graph represents the propagative relationships of camera poses and serves as a coordinate transformation tool, thereby establishing a globally consistent coordinate system. It is worth noting that the camera graph is constructed separately for each input and is not fixed.

3.3. Camera and Subject Registration

Due to the spatial uniqueness of subject poses, pose estimates of co-visible subjects from two cameras can be aligned to solve for their relative camera pose. In the 2D plane, all cameras are assumed to be upright and facing forward, perpendicular to the ground, with no pitch or roll. Thus, the relative pose between cameras c_k and c_j consists of a translation Δx, Δy, and a rotation Δθ:

$$T_{c_k\leftarrow c_j}=\left(\Delta x_{kj},\Delta y_{kj},\Delta {\theta }_{kj}\right)$$

(3)

The pose predictions for the co-visible subject from cameras c_k and c_j are denoted as $p_{c_k}=\left(x_{c_k},y_{c_k},r_{c_k}\right)$ and $p_{c_j}=\left(x_{c_j},y_{c_j},r_{c_j}\right)$, respectively. The relative pose from camera c_j to c_k can then be computed as follows:

$$\Delta x_{kj}=x_{c_k}-x_{c_j}\cos \Delta {\theta }_{kj}+y_{c_j}\sin \Delta {\theta }_{kj}$$

(4)

$$\Delta y_{kj}=y_{c_k}-x_{c_j}\sin \Delta {\theta }_{kj}-y_{c_j}\cos \Delta {\theta }_{kj}$$

(5)

$$\Delta {\theta }_{kj}=r_{c_k}-r_{c_j}$$

(6)

After registering each camera’s pose to the root node, all subject pose estimates can be transformed into the BEV global coordinate system. The transformation is given by

$$\left[\begin{array}{c}{x}_{c_1}\\ y_{c_1}\\ 1\end{array}\right]=\left[\begin{array}{ccc}\cos \Delta {\theta }_{1j}& -\sin \Delta {\theta }_{1j}& \Delta x_{1j}\\ \sin \Delta {\theta }_{1j}& \cos \Delta {\theta }_{1j}& \Delta y_{1j}\\ 0& 0& 1\end{array}\right]\cdot \left[\begin{array}{c}{x}_{c_j}\\ y_{c_j}\\ 1\end{array}\right]$$

(7)

$$r_{c_1}=\Delta {\theta }_{1j}+r_{c_j}$$

(8)

Subjects that are aligned in the unified coordinate system and satisfy empirically determined thresholds on similarity, distance, and orientation are regarded as the same identity. Specifically, a match is established when the appearance similarity exceeds a set value, while the spatial distance and orientation difference remain below their respective limits. A union-find data structure is employed to cluster such subjects into subgraphs, with self-matches and intra-view matches excluded. Duplicate detections are resolved using a nearest-neighbor strategy. Notably, the union-find operation is used only for clustering decisions during inference and does not interfere with the end-to-end training process.

3.4. Multi-Task Loss Design

To improve pose estimation and cross-view matching under occlusion, we introduce a multi-task loss: a camera pose regression loss, an appearance similarity loss, a cross-view consistency loss, and a subject pose regression loss. These components enable multi-task joint optimization through explicit pose supervision and implicit consistency constraints.

The camera pose regression loss is calculated by the ground truth and the prediction of camera poses, $\left(\Delta x_j^{\mathrm{gt}},\Delta y_j^{\mathrm{gt}},\Delta {\theta }_j^{\mathrm{gt}}\right)$ and $\left(\Delta x_j,\Delta y_j,\Delta {\theta }_j\right)$, respectively, can be expressed as

$$L_{\mathrm{cam}}={\displaystyle \sum _c\left(‖\left(\Delta x_j,\Delta y_j\right)-\left(\Delta x_j^{\mathrm{gt}},\Delta y_j^{\mathrm{gt}}\right)‖+\left|\Delta {\theta }_j-\Delta {\theta }_j^{\mathrm{gt}}\right|\right)}$$

(9)

In the appearance similarity loss, we incorporate the ground-truth similarity matrix and use a weighted combination of the normalized subject distance and orientation difference matrix, ${\overline{M}}_{\mathrm{dis}}$ and ${\overline{M}}_{\mathrm{ang}}$, and the ground-truth similarity matrix $M_{\mathrm{gt}}$. This strengthens the ResNet’s ability to identify the same pedestrian across different views. The loss can be expressed as

$$L_{\mathrm{app}}=‖M^{\mathrm{sim}\_\mathrm{pred}}-M^{\mathrm{sim}\_\mathrm{label}}‖$$

(10)

where the supervision signal is defined as

$$M^{\mathrm{sim}\_\mathrm{label}}=\left(1-{\alpha }_{\mathrm{gt}}\right)\left({\alpha }_{\mathrm{dis}}{\overline{M}}_{\mathrm{dis}}+{\alpha }_{\mathrm{ang}}{\overline{M}}_{\mathrm{ang}}\right)+{\alpha }_{\mathrm{gt}}{M}_{\mathrm{gt}}$$

(11)

where ${\alpha }_{\mathrm{dis}}$, ${\alpha }_{\mathrm{ang}}$, and ${\alpha }_{\mathrm{gt}}$ are scalar weights.

The cross-view consistency loss uses the average pose of a subject $\left({\overline{x}}_{c_1}^i,{\overline{y}}_{c_1}^i,{\overline{r}}_{c_1}^i\right)$ after aggregation as the supervision signal to enforce consistency among pose predictions from different views, thereby improving the accuracy and robustness of subject registration. It is formulated as

$$L_{\mathrm{consis}}={\displaystyle \sum _i\left(‖\left(x_{c_1}^i,y_{c_1}^i\right)-\left({\overline{x}}_{c_1}^i,{\overline{y}}_{c_1}^i\right)‖+\left|r_{c_1}^i-{\overline{r}}_{c_1}^i\right|\right)}$$

(12)

The subject pose regression loss supervises the pose predictions using ground-truth BEV poses $\left(x_{\mathrm{gt}}^i,y_{\mathrm{gt}}^i,r_{\mathrm{gt}}^i\right)$ and is expressed as

$$L_{\mathrm{subject}}={\displaystyle \sum _i\left(‖\left({\widehat{x}}_{c_1}^i,{\widehat{y}}_{c_1}^i\right)-\left(x_{\mathrm{gt}}^i,y_{\mathrm{gt}}^i\right)‖+\left|{\widehat{r}}_{c_1}^i-r_{\mathrm{gt}}^i\right|\right)}$$

(13)

The total loss function is given by

$$L_{\mathrm{total}}={\lambda }_{\mathrm{cam}}{L}_{\mathrm{cam}}+{\lambda }_{\mathrm{app}}{L}_{\mathrm{app}}+{\lambda }_{\mathrm{consis}}{L}_{\mathrm{consis}}+{\lambda }_{\mathrm{subject}}{L}_{\mathrm{subject}}$$

(14)

where ${\lambda }_{\mathrm{cam}}$, ${\lambda }_{\mathrm{app}}$, ${\lambda }_{\mathrm{consis}}$, and ${\lambda }_{\mathrm{subject}}$ are loss weights.

4. Experiments and Analysis

This section presents comprehensive experiments to qualitatively and quantitatively evaluate our proposed method. We first introduce the datasets used in our study. Then, we compare our approach against state-of-the-art methods on these datasets, reporting both quantitative metrics and qualitative results. Furthermore, we perform extensive ablation studies by systematically removing each key component of our method, thereby validating the effectiveness and necessity of the individual design choices. We perform thorough experiments to demonstrate the applicability of our method in a real-world scene.

4.1. Dataset

To the best of our knowledge, no existing dataset directly supports multi-view perception under low co-visibility and occluded conditions, especially one capturing multi-view perspectives from a UGV swarm system. The precise annotations of both camera and subject poses, including heading angles, along with strict image synchronization, further complicate dataset construction. Thus, we first build a synthetic scene using Unreal Engine 5, placing obstacles and pedestrians within a designated area. Surrounding the area, a UGV swarm system equipped with cameras captures synchronized multi-view images, while ground-truth poses of cameras and subjects are obtained via the simulation environment’s API. Time synchronization of the multi-view images is achieved through a message-triggered mechanism. Specifically, a callback function is executed to save the multi-view images only when the timestamps of images from all views fall within a specified time window, thereby ensuring approximate time alignment. We construct a five-view dataset consisting of 550 frames of images, named Camera Subject Registration Datasets-Occlusion (CSRD-O). The dataset covers an area of approximately 22 m × 12 m. We evaluate our method on the CSRD-O dataset. Additionally, we perform the same validation on the CSRD dataset introduced in prior work [14]. The CSRD dataset represents an open-area scenario featuring densely co-visible subjects.

4.2. Implementation Details

In our single-view module, the LocoNet is pretrained on subject poses using a loss that combines Euclidean distance and orientation difference to align its pose estimation scale with our dataset. The ResNet-50 is initialized with pretrained weights [47]. The framework is implemented in PyTorch 1.8.1 and runs on an NVIDIA RTX 4090 GPU. In the GPPM, when multiple co-visible subjects exist between two cameras, only the pair with the highest appearance similarity is used. This helps suppress incorrect subject appearance matches. In subject registration, the appearance similarity threshold, distance, and orientation thresholds need to strike a balance between insufficient matching and incorrect matching of different subjects. Here, a parameter sweep experiment is conducted to identify thresholds that achieve favorable performance. In our method, the appearance similarity threshold is set to 0.24, the distance threshold to 2.0 m, and the orientation threshold to 25°. Furthermore, the weights associated with the loss functions are also determined through parameter sweep experiments, achieving a comprehensively optimal performance under the proposed hyperparameter configuration. The hyperparameters are listed in

Table 1. Hyperparameter settings.

Hyperparameter	Value	Hyperparameter	Value
${\lambda }_{\mathrm{cam}}$	0.2	${\alpha }_{\mathrm{dis}}$	0.5
${\lambda }_{\mathrm{app}}$	1	${\alpha }_{\mathrm{ang}}$	0.5
${\lambda }_{\mathrm{consis}}$	0.1	${\alpha }_{\mathrm{gt}}$	0.5
${\lambda }_{\mathrm{subject}}$	0.2	Max epoch	200
Optimizer	Adam	Learning rate	1 × 10−6

.

4.3. Results

We evaluate the accuracy and reliability of camera and subject pose estimation using the average localization and orientation error and the proportion of predictions that fall within predefined thresholds. Experiments are conducted on both the CSRD dataset and our proposed CSRD-O dataset, with comparisons against some state-of-the-art methods, including MVDet [18], MVDeTr [19], MonoLoco++ [48], and BEVSee [14]. The results are summarized in

Table 2. Results of camera and subject pose estimation on CSRD and CSRD-O datasets.

Method	Dataset	Loc. Error	% < 0.5 m	% < 1.0 m	% < 1.5 m	Ori. Error	% < 5°	% < 10°	% < 15°
Camera
MonoLoco++	CSRD	2.57 m	7.4%	24.7%	40.9%	18.87°	17.7%	34.9%	50.7%
BEVSee	CSRD	1.11 m	29.8%	62.4%	78.6%	7.14°	47.0%	78.5%	91.9%
Ours	CSRD	1.54 m	22.9%	50.5%	65.5%	8.74°	44.8%	76.1%	87.5%
MonoLoco++	CSRD-O	21.22 m	0.0%	0.0%	0.0%	86.55°	1.0%	2.5%	5.0%
BEVSee	CSRD-O	7.31 m	5.0%	16.0%	21.5%	41.45°	18.0%	29.0%	34.5%
Ours	CSRD-O	1.57 m	25.0%	44.5%	68.5%	8.70°	43.5%	70.0%	83.5%
Subject
MVDet	CSRD	3.54 m	5.1%	19.6%	31.4%	/	/	/	/
MVDetr	CSRD	3.47 m	5.1%	19.7%	31.8%	/	/	/	/
MonoLoco++	CSRD	2.01 m	16.3%	40.6%	54.8%	28.19°	11.8%	23.7%	35.6%
BEVSee	CSRD	0.91 m	36.8%	73.4%	86.5%	13.07°	34.3%	59.8%	74.4%
Ours	CSRD	0.94 m	35.1%	73.9%	88.0%	12.85°	41.5%	67.2%	81.3%
MVDet	CSRD-O	3.32 m	3.2%	15.5%	31.7%	/	/	/	/
MVDetr	CSRD-O	3.29 m	3.3%	15.9%	32.0%	/	/	/	/
MonoLoco++	CSRD-O	15.70 m	0.0%	0.5%	5.6%	82.67°	4.1%	7.1%	10.8%
BEVSee	CSRD-O	3.19 m	8.5%	19.9%	30.5%	43.78°	9.4%	18.2%	23.8%
Ours	CSRD-O	1.40 m	12.7%	35.2%	64.8%	9.14°	48.2%	70.4%	81.8%

.

The results show that, although multi-view fusion is more challenging than pairwise fusion, our method achieves performance on the unoccluded CSRD dataset comparable to BEVSee in both camera and subject pose estimation. This is particularly noteworthy because BEVSee performs registration only between pairs of cameras, whereas our approach jointly integrates observations from all five cameras, which is more challenging. Nevertheless, in conventional open-area scenes with sufficient co-visible subjects, our method effectively fuses multi-view information to achieve accurate registration of both camera and subject poses.

On the occluded CSRD-O dataset, however, occlusions severely limit co-visibility between camera pairs, especially between non-root cameras and the root camera. This makes direct pairwise registration approaches ineffective. The resulting failure in camera pose estimation subsequently leads to large errors in subject pose prediction. In contrast, our method leverages the GPPM to enable indirect inter-camera pose registration, thereby reducing the adverse impact of occlusion. Our approach achieves impressive accuracy on CSRD-O. The mean pose errors for camera and subject achieve 1.57 m/8.70° and 1.40 m/9.14°, respectively. Moreover, 68.5% of camera localization errors and 64.8% of subject localization errors are below 1.5 m, while 83.5% of camera heading errors and 81.8% of subject heading errors fall within 15°. Despite being a purely vision-based solution, these results demonstrate the promising potential of our method for the dynamic deployment of a UGV swarm system, enabling robust multi-view perception under occlusion without reliance on GPS, pre-calibration, or external localization systems. It is important to note, however, that this approach currently requires a central node within the swarm system to aggregate and merge data from multiple perspectives.

4.4. Ablation Study

We conduct ablation studies on CSRD and CSRD-O datasets by removing key components from our full framework: the camera pose regression loss, the normalized pedestrian distance and the orientation difference matrix ${\overline{M}}_{\mathrm{dis}}$ and ${\overline{M}}_{\mathrm{ang}}$, the ground-truth similarity matrix $M_{\mathrm{gt}}$, the cross-view consistency loss, the subject pose regression loss, and the GPPM. The results are shown in

Table 3. Ablation study.

$L_{\mathrm{cam}}$	${\overline{M}}_{\mathrm{dis}}$$\mathrm{and} {\overline{M}}_{\mathrm{ang}}$	$M_{\mathrm{gt}}$	$L_{\mathrm{consis}}$	$L_{\mathrm{subject}}$	GPPM	CSRD
						Camera		Subject
						Loc. Error	Ori. Error	Loc. Error	Ori. Error
√	√	√	√	√	√	1.54 m	8.74°	0.94 m	12.85°
×	√	√	√	√	√	1.74 m	10.35°	1.05 m	14.15°
√	×	√	√	√	√	1.72 m	9.70°	0.96 m	12.78°
√	√	×	√	√	√	5.85 m	41.62°	3.00 m	38.80°
√	√	√	×	√	√	1.65 m	9.42°	1.02 m	13.36°
√	√	√	√	×	√	1.63 m	8.85°	1.03 m	13.04°
√	√	√	√	√	×	2.42 m	15.14°	1.44 m	17.61°
$L_{\mathrm{cam}}$	${\overline{M}}_{\mathrm{dis}}$$\mathrm{and} {\overline{M}}_{\mathrm{ang}}$	$M_{\mathrm{gt}}$	$L_{\mathrm{consis}}$	$L_{\mathrm{subject}}$	GPPM	CSRD-O
						Camera		Subject
						Loc. Error	Ori. Error	Loc. Error	Ori. Error
√	√	√	√	√	√	1.57 m	8.70°	1.40 m	9.14°
×	√	√	√	√	√	2.66 m	14.91°	1.19 m	9.72°
√	×	√	√	√	√	1.62 m	8.70°	1.42 m	9.79°
√	√	×	√	√	√	4.95 m	28.01°	2.30 m	33.88°
√	√	√	×	√	√	2.14 m	11.21°	1.43 m	12.04°
√	√	√	√	×	√	2.18 m	11.77°	1.82 m	15.60°
√	√	√	√	√	×	2.21 m	9.62°	1.57 m	11.06°

. Notably, completely removing GPPM renders our method non-functional; thus, we replace its BFS with depth-first search (DFS) for comparison, as DFS results in a longer registration path.

The results show that removing any component of our proposed framework leads to a noticeable degradation in performance. Removing the camera pose regression loss and the subject pose regression loss both directly lead to a drop in the estimation accuracy of the camera and the subject. This occurs because our camera registration relies on accurate subject pose estimates, and the accuracy of camera registration, in turn, affects the precision of multi-view subject aggregation. This also reflects the joint optimization of calibration and perception in our framework. Since our method relies on identifying co-visible subjects across all five camera views, removing the ground-truth similarity matrix prevents the algorithm from reliably matching the same subject across different views. This leads to errors in the chained registration among multiple cameras and results in a significant performance drop. Removing the normalized pedestrian distance and orientation angle difference matrices also causes performance degradation, as it eliminates the spatial cues necessary for identifying the same pedestrian across different views. Removing the cross-view consistency loss weakens the constraint that enforces coherent subject pose predictions across different views, leading to increased estimation errors. BFS ensures the shortest camera pose transformation path, and replacing it with DFS leads to longer and suboptimal paths. This induces error accumulation during pose propagation, ultimately reducing overall accuracy.

4.5. Influence of Registration Path Length on Pose Accuracy

To further validate the rationale of using BFS to find the shortest camera registration path, we evaluated how different registration path lengths affect pose estimation accuracy. As shown in Figure 4, both the mean errors in location and orientation and their spread increase with longer registration paths, which results from error accumulation. Crucially, local pose discrepancies (both translation and orientation) between adjacent cameras do not remain isolated. Instead, they propagate and accumulate along the registration chain, causing the total estimation error to scale with the path length. The optimal case is path length = 1, meaning registration is most accurate when two cameras perform pose transformation directly.

This finding offers practical guidance for deploying a UGV system in multi-view cooperative observation. Although multiple UGVs may be needed to overcome occlusions, it is preferable to maximize the number of cameras sharing co-visible subjects with the root camera to enhance prediction accuracy.

4.6. Influence of View Number on Prediction

We compare the impact of varying numbers of views on fusion and prediction. As shown in Figure 5, both camera and subject pose estimation errors increase noticeably as the number of views grows. Adding more views expands the system’s field of view but also increases fusion difficulty. This stems partly from longer camera registration paths and partly from the challenge of identifying the same subject across multiple views.

Figure 6 shows the BEV results from different numbers of views. Although prediction accuracy is affected by the number of views, the visual quality of the BEV remains consistent. Reducing the number of input views only causes subjects visible in specific views to be lost, which is expected. Subjects still within the field of view maintain good prediction accuracy. This demonstrates that our algorithm adapts well to varying numbers of views. This scalability and graceful degradation are essential for real-world UGV swarm deployment, where agents may join, leave, or temporarily lose visibility due to occlusion or communication constraints, enabling robust, coordination-aware perception without requiring fixed topology or predefined swarm size.

4.7. Qualitative Results

To intuitively illustrate the effectiveness of our method, we visualize predictions on the CSRD and CSRD-O datasets in Figure 7 and Figure 8, respectively, including the input images and the predicted BEV output with ground truth.

For each dataset, three cases are presented. On the CSRD dataset, despite the dense crowd, there are numerous co-visible subjects across all camera views, which ensures robust camera registration. The generated virtual BEV robustly reflects the true distribution of cameras and subjects. In the CSRD-O dataset, pedestrians are partially obstructed by randomly placed walls within the observation area, preventing any single camera from directly observing all subjects and leading to incomplete pairwise co-visibility among cameras. However, our framework can establish a camera graph structure based on available co-visible subjects and identify the correct camera registration path, achieving joint calibration of cameras and subjects. In contrast, BEVSee, which heavily relies on the presence of co-visible subjects between non-root cameras and the root camera, generates incorrect BEVs in these three cases. By deploying the UGV swarm system around the periphery to observe the central region, our framework can overcome occlusions caused by obstacles and robustly generate BEVs that reflect the true distribution of subjects, despite minor offsets caused by monocular depth estimation errors.

4.8. Real-World Scene Experiment

To validate the feasibility of our algorithm in real-world scenarios, we constructed a multi-view dataset named Camera Subject Registration Dataset Occlusion Real (CSRD-OR). Within an area of approximately 20 m × 10 m, we arranged obstacles that completely occluded the line of sight. Specifically, an L-shaped screen was placed in the center of the area to serve as the primary occlusion source. We deployed four UGVs to simulate a scene monitoring task. The UGVs were positioned around the perimeter of the monitored area, with their cameras oriented inward toward the center to create a view overlap. This configuration resulted in significant field of view (FOV) overlap among UGVs 1–3, while UGV 4 overlapped primarily with UGV 3, forming a specific camera graph structure for algorithm validation. The onboard cameras were activated to monitor pedestrians appearing in the scene, and synchronized images were collected using these cameras. Time synchronization for the multi-view images was achieved via post-processing alignment. Prior to data collection, the system clocks of all UGVs were synchronized. Each UGV independently recorded image streams with embedded timestamps into ROS bag files. In the post-processing stage, multi-view images with matching timestamps were extracted from these recordings to construct the CSRD-OR dataset. Figure 9 illustrates the configuration of our experimental setup and the UGV platforms employed. It should be noted that because the top camera was located directly above UGV 1, UGV 1 cannot be seen directly.

Due to the inherent differences between synthetic and real environments, the algorithm was retrained. As shown in Figure 10, we present two case studies conducted in real scenarios. Notably, in Cam 4 of Case 1, the system operator was captured in the frame. We manually excluded the network’s predictions corresponding to the operator, thereby strictly confining the perception range within the designated observation area. Due to the obstruction, no single camera can directly see all the subjects. The results demonstrate that our algorithm can effectively achieve multi-view fusion and predict the spatial relationships between cameras and subjects in practical settings, demonstrating its potential to support collaborative perception in a UGV swarm system and enabling task-oriented surveillance and collective situational awareness without a central calibration reference or global positioning infrastructure.

It is worth noting that the UGVs were maintained in a static state during data collection to simulate the scene monitoring task, ensuring stable coverage of the designated area and avoiding motion blur. However, this does not imply that our algorithm is incapable of handling dynamic scenarios. As detailed in Section 3.2 and Section 3.3, the construction of the camera graph and the camera and subject registration are performed independently for each input frame. Consequently, UGV movement does not incur additional computational overhead, provided that inter-view connectivity is maintained. Furthermore, we evaluated the computational cost on an NVIDIA RTX 4090 GPU. During inference, the model requires approximately 2.1 GB of VRAM. For a single multi-view frame (using four cameras as an example), the average processing time is 0.12 s.

5. Discussion

Given that real-world unmanned swarm systems typically operate in complex environments requiring flexible deployment, our method’s calibration-free nature makes it particularly suitable for such scenarios, provided that a central node can be used to aggregate and fuse multi-view data.

However, the aggregation of subjects across multiple views relies on fixed thresholds, which occasionally leads to aggregation failures. As illustrated in the synthetic scene case of Figure 11, Sub 1 failed to aggregate successfully, resulting in a duplicate appearance in the BEV output. Furthermore, as our method depends a lot on monocular depth estimation and feature matching between views, there remains room for improvement in accuracy. In the real scene case shown in Figure 11, compared to the ground-truth BEV, the predicted subjects in the algorithm’s BEV are clustered closer together, and the registration position of Cam 4 exhibits a significant offset. Despite these individual errors in camera and subject estimation, the generated BEV maintains a similar topological structure to the ground truth. Specifically, key relative spatial relationships (e.g., “Subject 1 is in front of Camera 1” or “Camera 4 is to the left of Camera 1”) remain consistent with the ground truth. This preservation of topological correctness is valuable for downstream tasks such as coarse localization and situational awareness, where correct relative ordering often outweighs the need for precise metric coordinates. Notably, this consistency holds for the majority of cameras and subjects, demonstrating the robustness of our method even under imperfect estimation conditions.

Regarding data scope and limitations, the CSRD-O dataset comprises diverse regional configurations, varying obstacle shapes, and pedestrian distributions to facilitate the construction of varied camera graph topologies. Its primary purpose is to validate the core logic of our algorithm through these critical scenarios. In contrast, the CSRD-OR dataset focuses on a single, specifically designed real-world scenario to verify the algorithm’s practical effectiveness under real sensor noise. Both datasets were constructed to address the core challenge of fusion under occlusion. They feature specific indoor and semi-indoor settings with precise annotations for camera and subject positions and orientations. Crucially, they provide scarce, controllable topological structures of camera graphs under occlusion, which are essential for evaluating graph-based fusion methods. However, the datasets currently lack extreme weather conditions and dynamic lighting variations. Furthermore, as the UGVs were kept stationary to simulate a stable scene monitoring task, the direct applicability of our current models to high-speed dynamic scenarios or highly complex, unstructured environments is limited without further adaptation. Additionally, due to the inherent limitations of the Unreal Engine 5 platform in simulating realistic sensor characteristics, the performance metrics reported on CSRD-O are not affected by noise factors such as motion blur or varied sensor noise. These factors, however, must be carefully considered when transferring the algorithm from simulation to real-world scenarios, as they can adversely impact front-end pedestrian detection and feature extraction. There is a distribution gap between the synthetic and real-world datasets. Consequently, models trained on a single domain exhibit performance degradation when directly transferred to the other, highlighting the need for future work in domain adaptation. Despite these limitations in scene diversity, our proposed method is fundamentally topology-agnostic. As the camera graph is constructed dynamically and separately for each input based on co-visibility rather than fixed spatial rules, the method is theoretically applicable to any scenario satisfying the local co-visibility condition, provided it undergoes appropriate supervised training for the target domain.

It should also be noted that multi-view synchronization was easily achieved in our synthetic dataset through simulation. However, in real-world applications, we had to rely on adding timestamps and manual alignment to ensure synchronization. In practical deployment, constrained by limited bandwidth and data transmission rates, guaranteeing temporal synchronization of image data across multiple views remains a significant challenge, particularly as the swarm scale increases. Software or hardware synchronization mechanisms are essential prerequisites for the practical application of our method. Furthermore, our fusion strategy requires a central node or server to aggregate information. This reliance on a central entity inherently limits the algorithm’s direct application in fully distributed systems. This inspires us that performing distributed optimization of the algorithm could potentially resolve these issues. Critically, by achieving joint camera and subject registration using visual information alone, our method inherently decouples pose estimation from GPS and other external positioning technologies and potentially serves as a vision-based local pose reference for unmanned swarm-level localization pipelines.

6. Conclusions

In this paper, we propose an end-to-end joint camera and subject registration framework for challenging scenarios with low co-visibility and severe occlusions commonly encountered in UGV swarm applications within complex environments. With a single-view module extracting subject poses and appearance features, our method formulates camera pose registration as a graph optimization problem. The GPPM enables chained pose transformation and global alignment, effectively unifying both camera and subject poses into a common global coordinate system. A multi-task loss function is designed to support end-to-end joint optimization, achieving performance comparable to state-of-the-art methods on the occlusion-free CSRD dataset and yielding stronger results on the heavily occluded CSRD-O dataset. Moreover, we demonstrate the effectiveness of our framework in a real-world UGV swarm monitoring task. Our approach provides a self-calibrating solution for multi-view fusion perception under occlusion. It offers a feasible, purely visual approach for collaborative perception in UGV swarm systems in occluded scenes. The self-calibrating framework is particularly well-suited for deployment across diverse scenarios without external positioning systems, as long as a central node or server is provided to aggregate and fuse multi-view data. It enables rapid, infrastructure-free deployment in GPS-denied environments and supports collaborative perception without global pose priors. In future work, we plan to integrate our framework with local positioning techniques for unmanned systems and consider multi-sensor fusion to further enhance collaborative perception robustness under extreme occlusion conditions.