Robust Occluded Object Detection in Multimodal Autonomous Driving: A Fusion-Aware Learning Framework

Abstract

Reliable occluded object detection remains a persistent core challenge for autonomous driving perception systems, particularly in complex urban scenarios where targets are predominantly partially or fully obscured by static obstacles or dynamic agents. Conventional single-modality detectors often fail to capture adequate discriminative cues for robust recognition, while existing multimodal fusion strategies typically lack explicit occlusion modeling and effective feature completion mechanisms, ultimately degrading performance in safety-critical operating conditions. To address these limitations, we propose a novel Fusion-Aware Occlusion Detection (FAOD) framework that integrates explicit visibility reasoning with implicit cross-modal feature reconstruction. Specifically, FAOD leverages synchronized red–green–blue (RGB), light detection and ranging (LiDAR), and optional radar/infrared inputs, employs a visibility-aware attention mechanism to infer target occlusion states, and embeds a cross-modality completion module to reconstruct missing object features via complementary non-occluded modal information; it further incorporates an occlusion-aware data augmentation and annotation strategy to enhance model generalization across diverse occlusion patterns. Extensive evaluations on four benchmark datasets demonstrate that FAOD achieves state-of-the-art performance, including a +8.75% occlusion-level mean average precision (OL-mAP) improvement over existing methods on heavily occluded objects $O=2$ in the nuScenes dataset, while maintaining real-time efficiency. These findings confirm FAOD’s potential to advance reliable multimodal perception for next-generation autonomous driving systems in safety-critical environments.

1. Introduction

Occlusion has long been considered one of the most critical challenges in object detection for autonomous driving, especially in dense and dynamic urban environments [1]. Target objects may be partially obscured by static obstacles such as vehicles, poles, or roadside infrastructure; completely invisible due to severe full occlusion; or involved in complex interactive occlusions where multiple dynamic agents overlap. Such scenarios substantially deteriorate perception performance by distorting object appearances, blurring spatial boundaries, and introducing high levels of uncertainty in both localization and classification.

Recent advances in multimodal perception have opened promising avenues to mitigate these issues. By integrating complementary sensory modalities—RGB cameras, LiDAR sensors, and radar/infrared (IR) systems—perception systems can leverage distinct advantages: RGB imagery provides fine-grained texture and semantic cues; LiDAR offers precise 3D geometric measurements; and radar/IR sensors ensure resilience under adverse weather or low-illumination conditions [2]. This synergy allows for the construction of more comprehensive and robust object representations, particularly in scenarios characterized by severe occlusion or degraded visibility.

1.1. Motivation and Problem

Despite these benefits, existing multimodal fusion methods remain insufficient for reliable occlusion handling. First, while middle-fusion transformers such as BEVFormer can effectively align multimodal features in a shared bird’s-eye-view (BEV) space, they lack a dedicated mechanism to explicitly identify occluded regions and orchestrate targeted, directional information flow from unoccluded modalities for recovering missing object cues [3]. Second, feature alignment and cross-modal completion strategies are often inadequate, leading to fragmented or inconsistent representations across modalities, and recent BEV-space fusion frameworks such as BEVFusion primarily optimize geometric consistency and efficiency rather than explicit occlusion reasoning [4]. Third, most state-of-the-art fusion pipelines suffer from high computational complexity and limited scalability, hindering their deployment in real-time, safety-critical autonomous driving systems [1]. These limitations underscore the need for new frameworks that can explicitly reason about occlusion while efficiently exploiting multimodal complementarities.

We address these gaps by coupling (i) explicit visibility estimation, (ii) geometry-aware cross-modal attentive completion, and (iii) occlusion-adaptive fusion and calibration within a single trainable objective, designed to preserve efficiency for deployment.

1.2. Approach Overview

At a high level, FAOD introduces a visibility-guided multimodal detector. It first estimates multi-granular visibility cues, then uses geometry-aware cross-modal attention to complete features for occluded regions, and finally performs occlusion-adaptive fusion and calibrated post-processing. All components are trained end-to-end under a unified objective; the architectural details are provided in Section 6.

1.3. Contributions

In this work, we propose a novel framework termed Fusion-Aware Occlusion Detection (FAOD), which tightly integrates explicit occlusion modeling with implicit cross-modal feature reconstruction. The main contributions of this study are summarized as follows:

• Explicit visibility reasoning for occlusion-aware BEV detection: We propose FAOD as a unified multimodal detection framework that explicitly models occlusion/visibility as learnable variables, including an instance-level occlusion classification and a region-level visibility map. These signals are supervised by occlusion-aware objectives and geometric consistency constraints, and are further used to guide downstream feature completion, fusion, and confidence scoring, rather than relying on implicit BEV aggregation.
• Visibility-guided directed cross-modal attention (CMA) for alignment and feature completion: We design a geometry-aware CMA module that performs asymmetric, visibility-driven information transfer (donor → recipient): when a target modality is heavily occluded, complementary less-occluded modalities are selectively attended to reconstruct missing BEV features and align cross-modal representations. This goes beyond symmetric BEV fusion, enabling targeted restoration of occluded object regions.
• Occlusion-aware dynamic fusion and score calibration at inference: FAOD couples visibility estimation with adaptive modality weighting and occlusion-aware post-processing. Fusion weights are adjusted conditioned on occlusion severity and modality reliability, while occlusion-aware Soft-NMS and confidence calibration mitigate false suppression of heavily occluded objects, improving detection stability under partial and complete occlusions.
• Occlusion-oriented augmentation/labeling and comprehensive benchmarking with deployment considerations: To evaluate FAOD under controlled occlusion levels, we develop an occlusion-centric augmentation and labeling pipeline that explicitly accounts for different visibility regimes. Extensive experiments on four representative datasets (nuScenes, KITTI-MOD, DENSE, and JRDB) show consistent gains over strong baselines, and we additionally adopt streamlined fusion components to maintain practical efficiency for real-time safety-critical deployment.

2. Related Work

2.1. Single-Modality Occluded Detection

Under occlusion, purely image-based detectors rely on spatial context, part/instance completion, and occlusion-aware heads to stabilize recall in crowded scenes. CityPersons highlighted occlusion as a first-class challenge for pedestrian detection and catalyzed research on modeling visible parts and suppressing inter-instance interference [5]. Representative lines include explicit occlusion reasoning within detectors (e.g., Occlusion-Aware Region-based Convolutional Neural Network (R-CNN)) and repulsive/overlap-aware losses that discourage proposals from drifting toward neighboring instances in crowds [6,7]. Several single-modality image detectors have revisited occlusion from the perspective of feature design and loss shaping. For road-scene RGB detection, RE-YOLOv5 enhances the receptive field and introduces deformable feature extraction to better localize heavily occluded vehicles and pedestrians in dense traffic [8], while monocular 3D pipelines with thermodynamic-inspired object losses have reported improved accuracy on distant and partially occluded targets [9].

Meanwhile, LiDAR-based detectors benefit from accurate 3D geometry and can be comparatively robust to appearance changes, yet suffer at long range and under self-occlusion where returns become sparse. Milestones range from voxelized or pillarized backbones (SECOND, PointPillars) to modern center-based heads (CenterPoint) and hybrid point–voxel designs (PV-RCNN) [10,11,12,13]. Occlusion-aware formulations further leverage the notion of “measurable” geometry; for instance, WYSIWYG constrains learning to what is physically observable, decoupling prediction from unmeasurable, fully occluded regions [14]. Sparsity-aware backbones such as SPBA-Net emphasize robust aggregation under extremely sparse returns, and depth-guided monocular detectors like MonoDFNet refine regression for partially visible objects [15,16]. Despite these advances on both RGB and LiDAR fronts, these approaches remain fundamentally single-modality and cannot exploit complementary sensing when one modality is severely degraded, which motivates the multimodal, occlusion-aware design adopted in this work.

2.2. Multimodal Fusion Strategies

To overcome the limitations of single sensors, numerous multimodal fusion strategies have been proposed. Early fusion methods operate at the data level. PointPainting injects per-pixel semantics into points before 3D detection, yielding a simple pipeline but making the system sensitive to semantic noise and camera–LiDAR projection errors [17].

Feature-level or middle-fusion methods, MVX-Net couples image and LiDAR streams at the feature level; 3D-CVF performs cross-view projection/gating to align modalities in 3D/BEV; and more recent UVTR/UniTR families unify features in a shared BEV/voxel space and exchange information via attention, improving geometric consistency and efficiency [18,19,20,21,22]. Recent LiDAR–camera fusion architectures further refine BEV-space multimodal detection. Zhao et al. propose SimpleBEV, an improved LiDAR–camera fusion backbone that jointly reasons over LiDAR BEV and image features, achieving gains on KITTI and nuScenes compared with earlier middle-fusion designs [23]. MSAFusion introduces a multi-sensor adaptive BEV fusion scheme that adjusts modality contributions according to scene context and improves detection robustness in dense traffic [24]. SpaRC integrates sparse 4D imaging radar with cameras via spatially aligned fusion and attention, showing clear benefits under adverse weather and low-visibility conditions where radar complements RGB [25]. Samfusion employs a sensor-adaptive multimodal fusion mechanism that explicitly downweights degraded modalities under rain, fog, and low light [26].

Decision-level or late-fusion strategies, candidate-level fusion (e.g., CLOCs) combines per-modal proposals/scores with weaker sensitivity to alignment noise but provides limited capacity to recover fine-grained details in heavy occlusion [27]. TransFusion and BEVFormer aggregate multi-view/multimodal evidence with queries anchored in the object or BEV space, capturing long-range dependencies while preserving cross-view geometry, and have become a dominant paradigm for robust fusion [3,4,28]. However, these fusion pipelines still lack a dedicated, learnable visibility variable that explicitly drives cross-modal completion and occlusion-aware fusion, which is the focus of FAOD.

2.3. Occlusion Modeling

Explicit supervision of occlusion masks/levels or depth ordering improves reliability under partial and full occlusion by exposing the detector to visibility as a separate variable that gates features and calibrates scores [6,14,29].

In monocular 3D detection, modeling geometric/pose uncertainty in projection helps recover partially occluded instances (e.g., GUPNet), while cross-view attention in BEVFormer-style architectures maintains spatial coherence despite missing observations [3,30].

RadarOcc predicts a dense 3D occupancy grid from 4D imaging radar (optionally combined with LiDAR), explicitly modeling occluded free space and demonstrating robustness in fog and rain [31]. GS-Occ3D builds on the Occ3D benchmark by introducing visibility-aware labels and a geometry-guided network, enabling more accurate reconstruction of occupied, free, and occluded voxels from multi-view images [32]. LinkOcc extends occupancy prediction temporally by linking LiDAR–camera features across frames for 4D panoptic occupancy, which helps recover occluded regions in crowded scenes [33]. More recently, a BEV occlusion mitigation framework has been proposed to minimize the occlusion effect on multi-view camera perception by combining multi-sensor cues and layered depth reasoning [34]. While these works explicitly model visibility or occupancy, they typically do not couple visibility estimation directly with cross-modal completion and the detection head in a single objective, as FAOD does.

2.4. Feature Alignment and Completion

Middle-level fusion must handle parallax, time misalignment, and calibration errors. Cross-view mapping with gating (3D-CVF) and unification in BEV/voxel space (UVTR/UniTR/BEV-style fusion) reduce reprojection artifacts and enable stable geometry-aware interaction [3,18,20,21,22].

Methods such as EPNet and PointAugmenting inject image semantics into point features to increase separability in sparse regions; DeepInteraction and Cross-Modal Transformer variants deploy attention to pass information bidirectionally across modalities and to align latent spaces under occlusion/missing-data, improving robustness in $O=1/2$ regimes [35,36,37,38,39].

Beyond early BEVFusion-style alignment, several recent works focus on more precise cross-modal feature alignment and completion. High-performance sparse fusion frameworks project image features only at sparse LiDAR locations and refine them with attention in BEV, reducing computation while preserving alignment quality [40]. FusionFormer combines BEV-space LiDAR–camera fusion with a temporal transformer, enforcing temporal consistency of multi-sensory features across frames [41]. Wang et al. introduce a BEV fusion method based on mutual deformable attention and temporal aggregation, explicitly aligning LiDAR and camera features across depth and time [42]. These methods primarily target alignment quality and temporal smoothing, whereas FAOD ties cross-modal attention and completion directly to an explicit occlusion signal and learns them jointly with the detection losses.

2.5. Datasets and Metrics

KITTI provides occlusion/truncation tags and remains a foundational benchmark for 2D/3D detection; nuScenes offers a rich multi-sensor suite (6 cameras + LiDAR + radar) and a 10-class evaluation set; JRDB focuses on mobile robotics with 360° cameras and 3D LiDAR in crowded indoor/outdoor scenes; DENSE targets adverse weather and low-visibility conditions with additional sensing (e.g., radar/thermal) [43,44,45,46].

The standard mean average precision (mAP) and the nuScenes Detection Score (NDS) are widely adopted. For occlusion, occlusion-level mAP (OL-mAP)—computed over subsets partitioned by the visibility ratio or official occlusion flags—has emerged as a sensitive measure of robustness under $O=1/2$ conditions [44].

Overall, prior BEV and multimodal detectors either (a) lack explicit visibility modeling, (b) fuse modalities without reliability- or occlusion-aware weighting, (c) decouple completion from detection, or (d) rely on calibration-sensitive alignment without safeguards to misalignment. FAOD addresses these limitations by unifying visibility estimation, directed cross-modal completion with geometric bias, and occlusion-adaptive fusion/calibration within one objective.

3. Problem Formulation

3.1. Sensor Inputs and Metadata

The multimodal input includes synchronized RGB images, LiDAR point clouds, and IR maps, together with calibration parameters and temporal alignment and deskew pre-processing. The RGB image, denoted as $I_{\mathrm{RGB}}\in {\mathbb{R}}^{H\times W\times 3}$, is accompanied by intrinsic parameters $\mathbf{K}$, extrinsic transform ${\mathbf{T}}_{E\leftarrow \mathrm{Cam}}\in \mathrm{SE}\left(3\right)$, and a time stamp t. The LiDAR point cloud is represented as ${\mathcal{P}}_{\mathrm{LiDAR}}={\{{\mathbf{p}}_i=(x_i,y_i,z_i,[{\mathrm{int}}_i,{\mathrm{ring}}_i,t_i^{\mathrm{rel}}])\}}_{i=1}^N$. We keep the intensity $\mathrm{int}$, the laser $\mathrm{ring}$, and the relative sample time $t^{\mathrm{rel}}$. Record ${\mathbf{T}}_{E\leftarrow \mathrm{LiDAR}}$, scan duration $\mathsf{\Delta }t$, and whether multi-sweep accumulation is used. The IR map is expressed as $T_{\mathrm{IR}}\in {\mathbb{R}}^{H^{\prime }\times W^{\prime }[\times C^{\prime }]}$. Its metadata comprise intrinsic parameters and resolution, extrinsics ${\mathbf{T}}_{E\leftarrow \mathrm{IR}}$, sampling rate, and beam model for mapping range–velocity.

For temporal alignment and deskewing, let E denote the unified world frame. We align all sensor timestamps to the LiDAR mid-scan time $t_0$ and deskew LiDAR by continuous poses $\mathbf{T}\left(t\right)$:

$${\tilde{\mathbf{p}}}_i=\mathbf{T}\left(t_0\right)\mathbf{T}{\left(t_i\right)}^{-1}{\mathbf{p}}_i.$$

(1)

3.2. Frames, Projections, and Gridding

All modalities are geometrically aligned with a unified coordinate frame E or a common BEV representation for spatial consistency. For the camera projection (LiDAR → image), a 3D point $\mathbf{p}$ in the LiDAR frame is first transformed into the camera coordinate system as $\mathbf{X}={\mathbf{T}}_{\mathrm{Cam}\leftarrow \mathrm{LiDAR}}[\mathbf{p};1]$. The homogeneous pixel $\tilde{\mathbf{u}}=\mathbf{K}\left[\mathbf{R}\right|\mathbf{t}]\mathbf{X}$ is normalized to $\mathbf{u}=(u,v)=(\tilde{u}/\tilde{w},\tilde{v}/\tilde{w})$. For BEV mapping (points/voxels → BEV), the ground plane is discretized according to $(i,j)=(\lfloor (x-x_{min})/s\rfloor ,\lfloor (y-y_{min})/s\rfloor )$, and features are pooled/encoded along the vertical dimension z to obtain ${\mathbf{F}}_{\mathrm{BEV}}$.

3.3. Instance-Level Annotations and Visibility

Each object instance is associated with a semantic category label $c\in \mathcal{C}$, where $\mathcal{C}$ denotes the predefined set of target classes (e.g., pedestrian, cyclist, passenger vehicle, large vehicle, traffic facility, etc.). The category label characterizes the semantic attributes of an object and serves as one of the fundamental prediction variables in multimodal detection tasks. Depending on the dataset configuration, the cardinality $\left|\mathcal{C}\right|$ can range from a small set of classes (e.g., three in KITTI) to a richer taxonomy (e.g., ten in nuScenes) and may be extended to support additional categories in more complex scenarios. To ensure cross-modality consistency, the category labels are defined and indexed in a unified manner across RGB images, LiDAR point clouds, and IR/radar annotations, enabling the detection model to align and share a common semantic space among heterogeneous sensors. Moreover, in order to evaluate robustness under occlusion, the category labels are further combined with visibility levels and bounding box annotations, which facilitates fine-grained performance analysis under varying occlusion conditions.

In addition to the semantic category label, each object is also described by a 3D bounding box $b=(x,y,z,w,h,l,\theta )$ with optional velocity $\mathbf{v}$. The occlusion level is defined $O\in \{0,1,2\}$ and unified via a visible ratio:

$$\begin{array}{cc}\hfill v_{\mathrm{img}}& ={\displaystyle \frac{\#\mathrm{visible}\mathrm{pixels}}{\#\mathrm{box}\mathrm{pixels}}},\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill v_{\mathrm{pc}}& ={\displaystyle \frac{\#\mathrm{valid}\text{in-box}\mathrm{points}}{\#\mathrm{expected}\mathrm{points}}},\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill v_{\mathrm{ratio}}& =\gamma v_{\mathrm{img}}+(1-\gamma )v_{\mathrm{pc}},\gamma \in [0,1],\hfill \end{array}$$

(4)

with thresholds

$$O=\left\{\begin{array}{cc}0,\hfill & v_{\mathrm{ratio}}\ge 0.75,\hfill \\ 1,\hfill & 0.25\le v_{\mathrm{ratio}}<0.75,\hfill \\ 2,\hfill & v_{\mathrm{ratio}}<0.25.\hfill \end{array}\right.$$

Here, $v_{\mathrm{img}}$ denotes the fraction of visible pixels within the 2D bounding box in the image plane, $v_{\mathrm{pc}}$ denotes the fraction of valid LiDAR points inside the 3D box relative to the expected number of points, and $v_{\mathrm{ratio}}$ is a weighted combination of the two, with $\gamma$ controlling the relative contribution of image and point-cloud visibility. The thresholds above assign $O=0$ to mostly visible objects, $O=1$ to partially occluded objects, and $O=2$ to heavily or fully occluded ones.

The cutoffs at $0.75$ and $0.25$ follow common three-level occlusion protocols in driving benchmarks, where roughly three quarters of the object area being visible corresponds to “non-occluded”, and less than one quarter corresponds to “heavily occluded”. The intermediate band $[0.25,0.75)$ provides a sufficiently wide regime of partially occluded samples for learning while keeping the semantic interpretation of each level clear. Since $v_{\mathrm{ratio}}$ is a convex combination of $v_{\mathrm{img}}$ and $v_{\mathrm{pc}}$, increasing $\gamma$ shifts the occlusion decision towards image-based visibility, whereas decreasing $\gamma$ emphasizes LiDAR-based visibility. The sensitivity of $v_{\mathrm{ratio}}$ to $\gamma$ is bounded by $|v_{\mathrm{img}}-v_{\mathrm{pc}}|$; when the two modalities broadly agree, moderate changes of $\gamma$ do not alter the assigned occlusion level O, and only strong disagreements lead to boundary cases.

Optional fine-grained labels include a pixel/BEV visibility mask ${\mathbf{M}}_{\mathrm{vis}}\in [0,1]$, a depth-order/occlusion graph $\mathcal{G}$ (edges “occluder → occluded”), and sensor-availability flags.

3.4. Sample Organization and Occlusion-Aware Sampling

Under heavy occlusion, temporal windows with motion compensation are employed to increase visibility and maintain spatiotemporal continuity across frames.

To further address dataset imbalance, stratified sampling is applied to balance samples with different occlusion levels ($O=0/1/2$ or to oversample partially and fully occluded instances ($O\in \{1,2\}$)), preventing domination by non-occluded samples.

For the assignment, both anchor-free and anchor-based strategies are adapted to handle occluded samples. In the anchor-free setting, a top-k dynamic assignment with center/distance priors is commonly used. For anchor-based methods, intersection over union (IoU) thresholds are relaxed for high-O samples, and additional center biases are introduced. Define a composite cost (anchor-free example):

$$\mathcal{C}={\lambda }_{\mathrm{cls}}{\mathcal{C}}_{\mathrm{cls}}+{\lambda }_{\mathrm{box}}{\mathcal{C}}_{\mathrm{box}}+{\lambda }_{\mathrm{occ}}\varphi \left(O\right),$$

(5)

where $\varphi \left(O\right)$ downweights penalties for highly occluded instances.

4. Task Objective and Subtasks

The learning objective follows the detection mapping introduced in Section 6 (Equation (9)), which transforms multimodal sensory inputs into instance-level predictions of category, 3D geometry, and occlusion state.

For readability, we describe the learning objective through three coupled subtasks rather than listing all loss terms in the main text: (i) visibility estimation that predicts instance-level occlusion and a region-level visibility cue, (ii) visibility-guided cross-modal completion that reconstructs missing BEV evidence using complementary modalities, and (iii) modality-aware detection that dynamically fuses completed features and decodes $(c,b)$ with calibrated confidence. The full formulations of Subtask A/B/C (including all losses and constraints) are provided in Appendix A.

5. Overall Optimization Objective and Training Strategy

The visibility, completion, and fusion modules are trained jointly so the network learns not only to detect objects, but also to estimate occlusion and recover missing information in a coordinated manner. This section summarizes the global objective and the strategies used to emphasize low-visibility cases while keeping training stable.

5.1. Global Objective

We jointly optimize detection, multi-granular visibility estimation, and cross-modal completion using

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\det }+{\lambda }_A{\mathcal{L}}_A+{\lambda }_B{\mathcal{L}}_B,$$

(6)

where ${\mathcal{L}}_{\det }$ is the detection objective (classification + box regression), ${\mathcal{L}}_A$ is the visibility-estimation objective (including occlusion classification and region-level visibility constraints), and ${\mathcal{L}}_B$ is the completion objective. The complete definitions of ${\mathcal{L}}_{\det }$, ${\mathcal{L}}_A$, and ${\mathcal{L}}_B$ and all constituent losses are given in Appendix A.

5.2. Strategy I: Occlusion-Aware Reweighting

This strategy upweights hard samples so partially and fully occluded instances contribute more strongly during training. Concretely, for difficult cases ($O\in \{1,2\}$), we amplify the occlusion-related objectives and the completion consistency:

$${\lambda }_{\mathrm{occ}}\left(O\right)={\lambda }_{\mathrm{occ}}^0\left(1+{\beta }_{\mathrm{occ}}·⊮[O\in \{1,2\}]\right),$$

(7)

$${\lambda }_{\mathrm{rec}}\left(O\right)={\lambda }_{\mathrm{rec}}^0\left(1+{\beta }_{\mathrm{rec}}·⊮[O=2]\right),$$

(8)

with stronger amplification for $O=2$ to enforce completion consistency under full occlusion.

In addition, we optionally employ robust multi-task balancing (e.g., homoscedastic uncertainty weighting) and gradient normalization techniques (e.g., GradNorm/PCGrad) to prevent a single objective (typically classification) from dominating optimization. The explicit formulation used in our implementation is provided in Appendix B.

5.3. Strategy II: Spatiotemporal Consistency (Stable Completion with Multi-Frame Aggregation)

To stabilize completion across time, we enforce that point clouds and feature maps corresponding to the same physical object remain consistent across neighboring frames. This reduces flicker and overfitting to single-frame noise, which becomes noticeable when visibility is low.

Given a temporal window $\mathcal{T}=\{t-K,\dots ,t+K\}$ and poses $\mathbf{T}(·)$, we incorporate (i) point-level consistency under ego-motion and (ii) feature-level consistency via geometric warping. The complete equations (including the Chamfer-like point loss and feature warping loss) are reported in Appendix B. In practice, we first converge a single-frame model, and then introduce the temporal consistency terms together with multi-frame aggregation.

5.4. Strategy III: Post-Processing (Occlusion-Aware NMS and Calibration)

Beyond the core network, we apply occlusion-aware post-processing to avoid suppressing hard occluded true positives and to calibrate confidence scores. The key idea is to soften suppression and adjust score calibration when a hypothesis is predicted as heavily occluded, because occlusion increases localization uncertainty and reduces IoU overlap.

We adopt occlusion-aware Soft-NMS and occlusion-conditioned temperature scaling (and, optionally, uncertainty-aware NMS). To keep the main text lightweight, the full post-processing equations are given in Appendix B.

5.5. Training Pipeline and Curriculum

We recommend a staged training pipeline. Training proceeds as follows: (i) train ${\mathcal{L}}_{\det }$ to stability; (ii) enable ${\mathcal{L}}_A$ for visibility estimation; and (iii) activate ${\mathcal{L}}_B$ together with temporal consistency (if used). The occlusion curriculum increases the synthetic occlusion strength $\rho$ from ${\rho }_0$ to ${\rho }_{max}$ (linear/cosine schedule) while ramping up the completion weight. To improve robustness to missing sensors, we randomly drop modalities (guided by sensor-availability flags) so the completion and fusion modules generalize across sensor degradation. Unless otherwise stated, all weights and thresholds are treated as learnable or scheduled hyperparameters, supporting reproducibility and systematic ablations across datasets and sensor configurations.

6. Method

6.1. Overall Architecture

FAOD is an occlusion-robust multimodal detector that links modality-specific encoding → occlusion-aware representation → cross-modal attentive completion → multi-task detection in an end-to-end pipeline. To make the data flow easier to follow at first glance, Figure 1 provides a conceptual view of how visibility estimation, cross-modal completion, and occlusion-aware fusion operate together. Figure 2 then presents the detailed module design and training/inference signals.

Formally, given synchronized sensory inputs from RGB cameras ${\mathcal{I}}_{\mathrm{RGB}}$, LiDAR point clouds ${\mathcal{P}}_{\mathrm{LiDAR}}$, and optionally radar/IR maps ${\mathcal{T}}_{\mathrm{IR}}$, the objective is to learn a detection function:

$$f:({\mathcal{I}}_{\mathrm{RGB}},{\mathcal{P}}_{\mathrm{LiDAR}},{\mathcal{T}}_{\mathrm{IR}})\mapsto \left\{(c,b,O)\right\}$$

(9)

where $c\in \mathcal{C}$ denotes the semantic object category, $b=(x,y,z,w,h,l,\theta )$ defines the 3D bounding box that includes spatial position, dimensions, and orientation, and $O\in \{0,1,2\}$ represents the occlusion state corresponding to no occlusion, partial occlusion, and full occlusion. FAOD comprises (i) modality-specific encoders for RGB, LiDAR, and IR/radar; (ii) an occlusion-aware feature extractor producing multi-granular visibility signals; (iii) CMA for selective fusion and completion; and (iv) a multi-task head that predicts $(c,b,O)$ with occlusion-adaptive fusion and decoding (see Figure 2).

6.2. Feature Extraction Modules

With a ResNet/Swin backbone and an FPN, the image encoder produces multi-scale features ${\left\{{\mathbf{F}}_l^{\mathrm{img}}\right\}}_{l=1}^L$, where ${\mathbf{F}}_l^{\mathrm{img}}\in {\mathbb{R}}^{H_l\times W_l\times C_l}$. For alignment with BEV/point features, a perspective or learnable view transform is applied:

$${\tilde{\mathbf{F}}}_l^{\mathrm{img}}={\mathcal{T}}_{\mathrm{view}}\left({\mathbf{F}}_l^{\mathrm{img}};\mathbf{K},{\mathbf{T}}_{E\leftarrow \mathrm{Cam}}\right).$$

(10)

For LiDAR, the voxel pathway (VoxelNet/SECOND) builds a tensor $\mathbf{V}\in {\mathbb{R}}^{D\times H\times W\times C}$ and yields BEV features ${\mathbf{F}}^{\mathrm{BEV}}\in {\mathbb{R}}^{H_b\times W_b\times C_b}$ via 3D/2D convolutions. The point pathway (PointNet++) aggregates raw points $\mathcal{P}=\left\{{\mathbf{p}}_i\right\}$ to ${\mathbf{F}}^{\mathrm{pc}}\in {\mathbb{R}}^{N\times C_p}$, then pools to BEV with a voxel/grid operator $\mathsf{\Gamma }(·)$:

$${\tilde{\mathbf{F}}}^{\mathrm{pc}}=\mathsf{\Gamma }\left(\mathcal{P},{\mathbf{F}}^{\mathrm{pc}}\right)\in {\mathbb{R}}^{H_b\times W_b\times {\tilde{C}}_b}.$$

(11)

For IR/Radar, a lightweight CNN/Transformer produces ${\mathbf{F}}^{\mathrm{ir}}\in {\mathbb{R}}^{H^{\prime }\times W^{\prime }\times C^{\prime }}$; geometric calibration maps it to the unified view:

$${\tilde{\mathbf{F}}}^{\mathrm{ir}}=\Phi \left({\mathbf{F}}^{\mathrm{ir}};{\mathbf{T}}_{E\leftarrow \mathrm{IR}}\right).$$

(12)

The aligned main-scale maps ${\mathbf{F}}_{\mathrm{RGB}},{\mathbf{F}}_{\mathrm{LiDAR}},{\mathbf{F}}_{\mathrm{IR}}\in {\mathbb{R}}^{H_b\times W_b\times C}$ are then used by subsequent modules in a common BEV/grid domain.

6.3. Occlusion-Aware Submodules

FAOD augments the backbone with auxiliary occlusion branches that provide explicit visibility cues for downstream completion and fusion. The goal of these submodules is to estimate, for each candidate and region in the scene, how strongly it is occluded, so that later stages can selectively trust or discount modality evidence.

At each candidate (instance or BEV grid), the occlusion branch outputs an instance probability $p_O=(p_0,p_1,p_2)$ for $O\in \{0,1,2\}$ and a region-level visibility map ${\mathbf{A}}_{\mathrm{vis}}\in {[0,1]}^{H_b\times W_b}$. Instance-level occlusion is trained with class-balanced cross-entropy or Focal loss, and the visibility map is supervised by BCE with total-variation (TV) regularization, as defined in Subtask A.

We obtain a semantic-guided visibility map by concatenating RGB and LiDAR features and projecting to a single channel:

$${\mathbf{A}}_{\mathrm{vis}}=\sigma \left({\mathrm{Conv}}_{1\times 1}\left([{\mathbf{F}}_{\mathrm{RGB}}\parallel {\mathbf{F}}_{\mathrm{LiDAR}}]\right)\right),$$

(13)

where $[·\parallel ·]$ denotes channel-wise concatenation.

Given a coarse fused map ${\mathbf{F}}_{\mathrm{fusion}}$, multi-head self-attention with positional encoding $\mathbf{P}$ and geometric bias ${\mathbf{B}}_{\mathrm{geo}}$ is applied using the following geometry-aware attention operator:

$$\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{softmax}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\top }+{\mathbf{B}}_{\mathrm{geo}}}{\sqrt{d}}}\right)\mathbf{V}.$$

(14)

6.4. Cross-Modal Attention and Completion

Once visibility has been estimated, FAOD uses cross-modal attention to transfer information from less occluded “donor” modalities to more occluded “target” modalities. Intuitively, this module aims to complete or refine target features in regions where they are unreliable, by borrowing geometry-consistent evidence from other sensors.

Given a target modality ${\mathbf{F}}_a$ and a donor modality ${\mathbf{F}}_b$, queries, keys, and values are obtained by linear projections:

$$\mathbf{Q}={\mathbf{W}}_Q{\mathbf{F}}_a,\mathbf{K}={\mathbf{W}}_K{\mathbf{F}}_b,\mathbf{V}={\mathbf{W}}_V{\mathbf{F}}_b.$$

(15)

The attended target features are then computed by the operator in Equation (14):

$${\hat{\mathbf{F}}}_a=\mathrm{Attn}(\mathbf{Q},\mathbf{K},\mathbf{V}).$$

(16)

For occlusion-gated mixing, let ${\mathbf{M}}_{\mathrm{occ}}=1-{\mathbf{A}}_{\mathrm{vis}}$. A modality reliability score $r_m\in (0,1)$ (estimated from density/SNR/texture/motion blur) yields a donor weight

$${\omega }_b={\displaystyle \frac{exp\left\{\kappa (1-{\mathbf{A}}_{\mathrm{vis}})r_b\right\}}{{\sum }_{m\in \{\mathrm{RGB},\mathrm{LiDAR},\mathrm{IR}\}}exp\left\{\kappa (1-{\mathbf{A}}_{\mathrm{vis}})r_m\right\}}},$$

(17)

and the completed target features are updated by

$${\mathbf{F}}_a^{\mathrm{comp}}={\mathbf{F}}_a+{\omega }_b{\mathbf{M}}_{\mathrm{occ}}\odot ({\hat{\mathbf{F}}}_a-{\mathbf{F}}_a).$$

(18)

6.5. Detection Head and Occlusion-Aware Fusion

The final detection stage converts the completed multimodal features into class, box, and occlusion predictions, while adaptively weighting each modality according to visibility and reliability. This head ties together the preceding modules and determines how much each sensor contributes to the final decision at each spatial location.

On the fused representation ${\mathbf{F}}_{\mathrm{final}}$, a multi-task head predicts class, box, and occlusion:

$$\{p_c,\hat{b},\hat{O}\}=\mathrm{Head}\left({\mathbf{F}}_{\mathrm{final}}\right).$$

(19)

The detection objective follows Subtask C, with ${\mathcal{L}}_{\det }={\lambda }_{\mathrm{cls}}{\mathcal{L}}_{\mathrm{cls}}+{\lambda }_{\mathrm{box}}{\mathcal{L}}_{\mathrm{box}}+{\lambda }_{\mathrm{occ}}{\mathcal{L}}_{\mathrm{occ}}$; the formulations of ${\mathcal{L}}_{\mathrm{cls}}$, ${\mathcal{L}}_{\mathrm{box}}$ (IoU/DIoU + ${\ell }_1$ with periodic angle), and ${\mathcal{L}}_{\mathrm{occ}}$ are defined there and not repeated here.

Occlusion-aware dynamic fusion computes, at each location $\mathbf{x}$, per-modality weights via a learnable gate $g_m$ (e.g., a two-layer MLP over $[{\mathbf{M}}_{\mathrm{occ}}\left(\mathbf{x}\right),r_m\left(\mathbf{x}\right),\mathrm{GAP}\left({\mathbf{F}}_m\right)]$); the resulting logits are normalized to weights:

$$\begin{array}{cc}\hfill g_m\left(\mathbf{x}\right)& ={\mathrm{MLP}}_m\left([{\mathbf{M}}_{\mathrm{occ}}\left(\mathbf{x}\right),r_m\left(\mathbf{x}\right),\mathrm{GAP}\left({\mathbf{F}}_m\right)]\right),\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill {\alpha }_m\left(\mathbf{x}\right)& ={\displaystyle \frac{exp\left\{g_m\left(\mathbf{x}\right)\right\}}{{\sum }_{m^{\prime }\in \{\mathrm{RGB},\mathrm{LiDAR},\mathrm{IR}\}}exp\left\{g_{m^{\prime }}\left(\mathbf{x}\right)\right\}}},\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill {\mathbf{F}}_{\mathrm{final}}\left(\mathbf{x}\right)& =\sum _{m\in \{\mathrm{RGB},\mathrm{LiDAR},\mathrm{IR}\}}{\alpha }_m\left(\mathbf{x}\right){\mathbf{F}}_m^{\left[\mathrm{comp}\right]}\left(\mathbf{x}\right).\hfill \end{array}$$

(22)

Here, ${\mathbf{M}}_{\mathrm{occ}}$ is the occlusion mask defined earlier, $r_m\in (0,1)$ denotes modality reliability, and $\mathrm{GAP}(·)$ is global average pooling. Higher occlusion (larger ${\mathbf{M}}_{\mathrm{occ}}$) and higher reliability $r_m$ increase ${\alpha }_m$, prioritizing robust modalities (e.g., LiDAR/IR) when needed.

6.6. Training and Implementation Notes

All modalities are aligned to a unified BEV/grid; for multi-frame inputs, pose-based registration and LiDAR deskew are applied. Synthetic occlusion ${\mathcal{T}}_{\mathrm{occ}}\left(\rho \right)$ is applied with strength $\rho$ gradually increased, and ${\lambda }_{\mathrm{rec}}$ is ramped up in tandem to stabilize completion learning.

The overall objective follows the global formulation in Section 5 (Equation (6)). For samples with $O\in \{1,2\}$, ${\lambda }_{\mathrm{occ}}$ and ${\lambda }_{\mathrm{rec}}$ are increased; homoscedastic uncertainty weights $\left\{{\sigma }_k\right\}$ may be used for task balancing. During inference, occlusion-aware Soft-NMS (weaker suppression for $O=2$), temperature scaling, and variable IoU thresholds are used to reduce misses and over-suppression under heavy occlusion.

The pipeline forms a causal loop from visibility estimation → cross-modal completion → dynamic fusion. The heatmap ${\mathbf{A}}_{\mathrm{vis}}$ localizes occlusions, CMA performs geometry-consistent targeted recovery, and ${\alpha }_m$ assigns modality weights based on occlusion and reliability. The multi-task head jointly optimizes these components, yielding robust 3D detection under partial and full occlusion.

7. Experiments

7.1. Overall Performance and Stratified Analysis

Across four benchmarks, FAOD consistently outperforms multimodal and occlusion-specialized baselines. Averaged over three independent runs (distinct random seeds), the absolute improvements are typically $+3\%$ to $+7\%$ in mAP and $+2$ to $+5$ in NDS, with statistical significance verified by bootstrap testing $(p<0.05)$.

To assess robustness under occlusion, results are partitioned by the unified visibility thresholds into $O\in \{0,1,2\}$ (non-occluded, partially occluded, and severely/near-completely occluded) and reported as OL-mAP per subset. For non-occluded cases $(O=0)$, FAOD matches or slightly exceeds strong baselines by $+0.5\%$ to $+2\%$, indicating no loss of upper-bound accuracy. For partially occluded cases $(O=1)$, gains of $+4\%$ to $+9\%$ are observed, attributable to visibility guidance and cross-modal completion that mitigate weak texture and sparse points. For severely/near-completely occluded cases $(O=2)$, the gains are most pronounced $(+8\%$ to $+15\%)$; recall increases more than precision, consistent with CMA and dynamic fusion recovering detectability under low-information conditions.

Category-wise and scale-wise analyses indicate especially notable improvements in small/distant classes (e.g., pedestrian, cyclist). Scale binning shows larger OL-mAP gains for small-to-medium objects, consistent with FAOD’s ability to compensate for sparse LiDAR returns and weak image cues. This trend is also aligned with typical road scenes, where small agents are the first to disappear behind occluders and the last to provide clean geometry.

The region-level visibility map ${\mathbf{A}}_{\mathrm{vis}}$ attains higher IoU and lower cross-entropy against ground-truth ${\mathbf{M}}_{\mathrm{vis}}$ than unsupervised baselines. The Spearman correlation between ${\mathbf{A}}_{\mathrm{vis}}$ and classification confidence is also higher, supporting its use in score calibration. In practice, this correlation matters more for $(O=1,2)$: the score needs to reflect “how much evidence is really there”, otherwise the post-processing step tends to discard the hard positives.

On DENSE nighttime/low-light/rain–snow subsets, FAOD’s advantage widens (OL-mAP gains of about $+6\%$ at $O=1$ and $+12\%$ at $O=2$). On JRDB’s crowded indoor scenes with long occlusion chains, FAOD maintains stable recall. These are the cases where symmetric fusion is easily confused by missing or noisy cues; the visibility-gated completion and occlusion-aware calibration are simply more forgiving, and the improvements show up consistently in the occlusion-stratified metrics.

7.2. Baseline Comparison and Component Contributions

Compared with multimodal baselines (e.g., PointPainting, MVX-Net, UVTR), PointPainting is vulnerable to noisy image semantics, particularly at $O=2$; FAOD suppresses unreliable channels via ${\mathbf{A}}_{\mathrm{vis}}$ and $r_m$, reducing false positives. MVX-Net/UVTR degrade under alignment errors or missing modalities; FAOD’s geometric bias ${\mathbf{B}}_{\mathrm{geo}}$ and gated fusion ${\alpha }_m$ show greater robustness.

Versus occlusion-specialized baselines (e.g., GUPNet, ORN, DetZero), single-modality/view occlusion reasoning is limited under complete occlusion; FAOD’s cross-modal completion transfers information directionally (donor→recipient), reconstructing features for invisible recipients. In dense crowds, ORN/DetZero’s reliance on ordering/logic graphs is less robust to annotation noise; FAOD with ${\mathcal{L}}_{\mathrm{rank}}$ yields smoother behavior.

Ablations show consistent trends. Removing the occlusion branch $({\mathcal{L}}_{\mathrm{occ}}+{\mathcal{L}}_{\mathrm{vis}})$ notably degrades OL-mAP at $O=2$ and weakens ${\mathbf{A}}_{\mathrm{vis}}$, limiting CMA completion. Removing the geometric bias ${\mathbf{B}}_{\mathrm{geo}}$ hurts more in high-parallax camera–radar/camera–LiDAR settings. Removing reliability gating $r_m$ increases mis-fusion in low-light/sparse segments and reduces the variance of ${\alpha }_m$. Disabling consistency/contrastive losses $({\mathcal{L}}_{\mathrm{rec}},{\mathcal{L}}_{\mathrm{align}})$ during the occlusion curriculum leads to over-completion or local overfitting with larger per-subset variance. Temporal consistency (optional) further improves $O=2$ recall at modest latency cost.

7.3. Performance Analysis: Efficiency, Resources, and Deployability

We use three model scales—FAOD-S (small), FAOD-M (medium), and FAOD-L (large). Unless otherwise stated, the latency breakdown reports the large scale (FAOD-L). Under a common protocol (nuScenes, single GPU, FP16, batch = 1), we report latency and key resource metrics in

Table 1. Latency breakdown (nuScenes, single GPU, FP16, batch $=1$).

Component	Latency (ms)	Share (%)
Image backbone + FPN	38.4	41.0
LiDAR voxel/BEV	21.7	23.2
IR/Radar branch	4.8	5.1
CMA (with ${\mathbf{B}}_{\mathrm{geo}}$)	17.6	18.8
Head + Occl-SoftNMS	12.2	13.0
Total (FAOD-L)	94.7	100

and

Table 2. Key resource metrics (same protocol).

Metric	Value
FPS (=1000/total latency)	10.6
Params (M)	79.2
FLOPs (G)	321.5
Peak memory (GB)	9.3

. CMA and the image backbone dominate compute; reducing image resolution/backbone width and triggering CMA sparsely (e.g., ROI-based) provide the largest speedups.

Speed–accuracy trade-offs are shown in

Table 3. Pareto trade-off across model scales (same protocol).

Config	Img Short	BEV Step	Sweeps	mAP	NDS	OL-mAP ($\mathit{O}=2$)	Lat. (ms)	FPS
FAOD-L	960	0.20	10	66.0	68.1	52.8	94.7	10.6
FAOD-M	800	0.25	5	64.3	66.7	50.0	58.1	17.2
FAOD-S	640	0.30	3	62.5	64.9	47.2	41.0	24.4

. FAOD-M reduces latency by ≈39% vs. FAOD-L while losing ≈2.8 pts on $O=2$, suitable for online use; FAOD-L favors offline high-accuracy.

In this context, FAOD-S can be regarded as the lightweight variant targeting resource-constrained or embedded deployments. Compared with FAOD-L, it reduces latency from $94.7$ ms to $41.0$ ms (about a $57\%$ decrease) and peak memory (see Table 2) at the cost of several points in mAP and OL-mAP $(O=2)$. Such a trade-off is acceptable for many automotive ECUs where on-board compute and memory are limited. On automotive-grade SoCs, additional gains are expected from TensorRT/ONNX engines, mixed precision, and moderate backbone width scaling; a full evaluation of FAOD-S on embedded hardware is left for future work.

Engine-level optimizations reduce memory and improve throughput (

Table 4. Memory and throughput with/without engine optimizations.

Variant	Peak Mem (GB)	TRT/ONNX	Throughput (FPS)
FAOD-L (native)	9.3	No	10.6
FAOD-L + TensorRT	7.8	Yes	12.9
FAOD-M (native)	6.1	No	17.2
FAOD-S (native)	4.7	No	24.4

); e.g., TensorRT yields 20–$25\%$ throughput gains.

Calibration and post-processing analyses on nuScenes val are given in

Table 5. Effect of calibration and occlusion-aware NMS (lower is better for ECE/Brier; higher is better for mAP/OL-mAP).

Setting	ECE ↓	Brier ↓	mAP ↑	OL-mAP ($\mathit{O}=2$) ↑
No temperature scaling and standard NMS	4.6%	0.158	65.3	50.9
+Temperature scaling ($\tau$ constant)	2.9%	0.141	65.8	51.6
+Occl-SoftNMS ($\sigma \left(O\right)$ adaptive)	2.8%	0.140	66.0	52.8

. Temperature scaling improves calibration (ECE/Brier), and occlusion-aware Soft-NMS further improves detection under heavy occlusion ($O=2$; higher OL-mAP) together with overall mAP.

Robustness to modality dropout at inference is summarized in

Table 6. Robustness to modality dropout at inference (nuScenes val).

Dropped Modality	ΔmAP	ΔOL-mAP (O = 2)
IR/Radar off	$-0.6$	$-1.1$
RGB off	$-3.9$	$-6.4$
LiDAR off	$-12.7$	$-18.9$

. LiDAR is critical under strong occlusion; RGB/IR remain complementary in low light and sparse-point regimes.

Efficiency impacts of key components are shown in

Table 7. Ablations on efficiency and occlusion accuracy (deltas vs. full FAOD).

Variant	ΔmAP	ΔOL-mAP (O = 2)	ΔLatency (ms)
w/o CMA	$-2.8$	$-6.9$	$-13.7$
w/o ${\mathbf{B}}_{\mathrm{geo}}$	$-1.3$	$-3.2$	$-2.1$
w/o $r_m$/gating	$-1.7$	$-4.1$	$-0.8$

. CMA yields the largest accuracy gains with moderate cost; ${\mathbf{B}}_{\mathrm{geo}}$ and $r_m$ are highly cost-effective for high-occlusion accuracy.

FAOD delivers statistically significant gains on aggregate and occlusion-stratified metrics across four benchmarks, with the largest improvements at $O=2$ due to cross-modal completion and adaptive gating. Efficiency-wise, FAOD traces a clear Pareto frontier via image/BEV resolution and sparse attention, enabling both offline and online deployments. Interpretability (${\mathbf{A}}_{\mathrm{vis}}$, ${\alpha }_m$ maps) and better calibration (temperature scaling) support practical deployment and safety analyses.

7.4. Occlusion-Stratified Results on nuScenes

To rigorously assess robustness under varying degrees of occlusion, the nuScenes validation set is stratified into three visibility tiers—non-occluded $(O=0)$, partially occluded $(O=1)$, and heavily occluded $(O=2)$—and OL-mAP is reported for each tier. Overall, FAOD-L attains the best or tied-best performance across all tiers and exhibits a smaller degradation as occlusion increases than both multimodal and occlusion-specialized baselines (Figure 3).

For $(O=0)$, FAOD-L achieves an OL-mAP of $71.0$, exceeding the mean of four baselines (69.25) by $+1.75$ percentage points (pp) and outperforming the best baseline (70.0) by $+1.0$ pp. This suggests that introducing explicit visibility reasoning does not come at the cost of peak accuracy: when observations are clean, the model largely behaves like a strong BEV fusion detector rather than “over-correcting” what is already reliable. For $(O=1)$, FAOD-L reaches $63.0$, improving over the baseline mean (56.5) by $+6.5$ pp and over the strongest baseline (58.0) by $+5.0$ pp. In many nuScenes scenes, partial occlusion is the more common and also the more confusing case: one modality may still carry a usable fragment (e.g., a contour in RGB), while another becomes sparse or locally corrupted (e.g., missing returns in LiDAR). The visibility heatmap helps here by damping unreliable regions and letting the fusion focus on the parts that are still trustworthy; CMA then supplies complementary cues where the target stream is weak, instead of mixing all modalities symmetrically in BEV. For $(O=2)$, FAOD-L attains $53.0$, surpassing the baseline mean (44.25) by $+8.75$ pp and the strongest baseline (46.0) by $+7.0$ pp. The improvement is strongest under $(O=2)$ and is driven mainly by recall: in these cases, the detector often needs to work with very limited evidence (a few points, a small edge fragment, or intermittent responses). Visibility-gated CMA, together with reliability weighting, reconstructs discriminative features only where information is genuinely missing, making the remaining cues usable without spreading artifacts across the scene. This also makes post-processing less brittle, because a hard true positive under severe occlusion may not achieve the “nice” overlap pattern that standard suppression heuristics assume.

The tiered results also hint at what kind of situations in nuScenes FAOD benefits from. Under $(O=1)$, the gain tends to come from cases that are partially blocked but still geometrically consistent—for instance, an agent visible in one stream while partially missing in another due to occluders or viewpoint. The directed (donor → recipient) completion is especially useful in this regime: it transfers information from the less-occluded donor stream to the occluded target stream, which is a different behavior from symmetric BEV aggregation. Under $(O=2)$, detections are closer to the decision boundary. Here, the visibility gating prevents the completion module from “guessing everywhere”, and the occlusion-aware calibration/NMS helps avoid over-suppressing these low-IoU, low-confidence but correct hypotheses. In short, $(O=1)$ benefits more from selective restoration, while $(O=2)$ benefits from both restoration and a more forgiving confidence/suppression policy.

Figure 4, Figure 5 and Figure 6 provide qualitative comparisons between the baseline fusion model (BEVFormer) and FAOD under heavy occlusion. For each scene, the top image shows the baseline result, while the bottom image shows FAOD. In scenarios where target objects are largely invisible in RGB and only sparsely observed in LiDAR, the baseline often fails to form meaningful responses, leading to missed detections or fragmented hypotheses. In contrast, FAOD produces more coherent BEV activations and more stable object predictions. The visibility cues highlight occluded regions, while cross-modal attention selectively transfers complementary geometric information from less-occluded modalities, resulting in more complete object representations.

Degradation with occlusion is quantified by ${\mathsf{\Delta }}_{\mathrm{occ}}=\text{OL-mAP}(O=0)-\text{OL-mAP}(O=2)$. PointPainting: 26 pp; MVX-Net: 25 pp; UVTR: 26 pp; DetZero: 23 pp; FAOD-L: 18 pp. Relative to the best baseline(DetZero, 23 pp), FAOD-L reduces the penalty by 5 pp (relative reduction $\approx 21.7\%$), yielding a flatter performance–occlusion curve and stronger cross-tier consistency.

8. Discussion

8.1. Generalization Capability

The proposed FAOD framework demonstrates strong generalization ability when deployed in previously unseen environments and across novel object categories. By leveraging multimodal feature representations and explicit occlusion reasoning, the model is less reliant on dataset-specific appearance patterns, thereby enhancing robustness in diverse urban scenarios. Experimental results across four heterogeneous benchmarks confirm that FAOD can effectively adapt to varying sensor configurations and scene geometries without significant performance degradation.

8.2. Robustness to Occlusion Types

A key strength of our approach lies in its robustness against different types of occlusion. In addition to handling static and partial occlusions, FAOD exhibits stable performance in highly dynamic conditions where occlusions are caused by moving vehicles, pedestrians, or other agents. The explicit visibility reasoning module enables reliable estimation of occlusion levels, while the cross-modal feature completion mechanism recovers object representations even when large portions are visually obscured.

8.3. Computational Efficiency

Practical deployment in autonomous driving requires a balance between accuracy and efficiency. In this work, all runtime and resource measurements are obtained on a single NVIDIA RTX 3090 GPU under the evaluation protocol described in the Experiments section (FP16, batch $=1$, with image resolution and LiDAR sweeps as specified for each FAOD-S/M/L configuration). The reference implementation has a compact model size of about 110 MB of learnable parameters, which fits comfortably within the memory budgets of current GPU and automotive SoC platforms.

Under this protocol, the three model scales trace a clear accuracy–latency frontier: FAOD-L targets offline or high-compute settings, FAOD-M offers a favorable trade-off between accuracy and speed, and FAOD-S is explicitly designed as a lightweight variant for resource-constrained or embedded deployments, using lower image resolution, fewer LiDAR sweeps, and narrower backbones while preserving most of the occlusion-stratified gains. Engine-level optimizations such as TensorRT/ONNX conversion, mixed-precision execution, operator fusion, and sparsified (e.g., ROI-triggered) attention further reduce latency and memory footprint. A detailed quantitative evaluation on specific automotive-grade embedded hardware is left for future work.

8.4. Limitations and Future Directions

Despite its effectiveness, the current framework has several limitations. First, the present FAOD implementation assumes fixed and precise extrinsic calibration between cameras, LiDAR, and IR/radar. The geometric bias term ${\mathbf{B}}_{\mathrm{geo}}$ and the BEV projections are computed directly from these calibration parameters. In practice, LiDAR misalignment (e.g., due to mechanical tolerances, thermal drift, or mounting vibrations) can distort cross-modal attention and reliability gating, and we do not yet explicitly model or correct such effects. Future variants could incorporate calibration-robust feature encodings, online refinement of extrinsics, or uncertainty-aware fusion that downweights modalities suspected to be misaligned.

FAOD is evaluated in a single-frame setting and does not yet include explicit temporal modeling. Without temporal aggregation, the method cannot fully exploit motion cues and cross-frame visibility to stabilize occlusion estimates or recover objects that are only intermittently visible under heavy occlusion. A natural extension is to aggregate BEV features over short, pose-compensated temporal windows and apply a lightweight temporal attention module on top of the existing BEV representation, together with temporal consistency losses to regularize completion in heavily occluded scenes.

The current study focuses on passive sensing with a fixed sensor layout. We do not consider active strategies such as view planning, adaptive sensor scheduling, or dynamic exposure control, which may further mitigate severe occlusion and adverse-weather degradation. Exploring these directions, together with temporal reasoning and calibration-aware fusion, is left for future work.

9. Conclusions

In this work, we proposed FAOD, a novel Fusion-Aware Occlusion Detection framework designed to address the persistent challenge of object detection under occlusion in autonomous driving systems. By integrating explicit visibility reasoning with implicit cross-modal feature completion, FAOD is capable of reconstructing object representations even in highly cluttered and visually degraded scenarios. A central innovation of our approach lies in the attention-guided multimodal fusion mechanism, which dynamically aligns heterogeneous features from RGB, LiDAR, and infrared/radar modalities to maximize complementary strengths while mitigating occlusion-induced information loss.

Extensive experiments on four representative autonomous driving benchmarks demonstrate that FAOD achieves state-of-the-art performance across a wide range of occlusion conditions, including partial and full occlusions, static and dynamic obstacles, and diverse sensor configurations. Notably, the framework maintains both high accuracy and computational efficiency, reaching real-time inference rates with a compact model size, which highlights its potential for practical deployment in safety-critical driving environments.

Beyond empirical performance, FAOD contributes a methodological foundation that can generalize to multimodal perception research. Its explicit occlusion modeling, modality-aware feature reconstruction, and attention-driven alignment are not confined to detection; they could also support occlusion-aware tracking, improve the reliability of motion forecasting, and refine occupancy prediction, in addition to aiding cooperative multi-agent perception. In each of these tasks, the same principle applies: reasoning about which signals are missing and selectively completing them with information from other modalities can make the system more robust. More broadly, FAOD exemplifies a practical paradigm for dealing with incomplete multimodal data, offering a transferable approach that extends beyond autonomous driving and remains relevant wherever sensor degradation or partial observability pose challenges.

Looking ahead, future research directions include incorporating temporal reasoning to leverage motion dynamics across video sequences, as well as exploring active perception strategies that adapt sensor utilization to occlusion severity. By advancing towards these goals, FAOD can serve as a stepping stone for the development of next-generation robust, reliable, and intelligent perception systems for autonomous driving and broader real-world applications.