Robust BEV Perception via Dual 4D Radar–Camera Fusion Under Adverse Conditions with Fog-Aware Enhancement

Abstract

Bird’s-eye-view (BEV) perception has emerged as a key representation for unified scene understanding in autonomous driving. However, current BEV methods relying solely on monocular cameras suffer from severe degradation under adverse weather and dynamic scenes due to limited depth cues and illumination dependency. To address these challenges, we propose a robust multi-modal BEV perception framework that integrates dual-source 4D millimeter-wave radar and multi-view camera images. The proposed architecture systematically exploits Doppler velocity and temporal information from 4D radar to model dynamic object motion, while introducing a deformable fusion strategy in the BEV space for accurate semantic alignment across modalities. Our design includes four key modules: a Doppler-Aware Radar Encoder (DARE) that enhances motion-sensitive features via velocity-guided attention; a Fog-Aware Feature Denoising Module (FADM) that suppresses modality inconsistency in low-visibility conditions through cross-modal attention and residual enhancement; a Multi-Modal Temporal Fusion Module (TFM) that encodes radar temporal sequences using a Transformer encoder for motion continuity modeling; and a confidence-aware multi-task loss that jointly supervises semantic segmentation, motion estimation, and object detection. Extensive experiments on the DualRadar dataset and adverse-weather simulations demonstrate that our method achieves significant gains over state-of-the-art baselines in BEV segmentation accuracy, detection robustness, and motion stability. The proposed framework offers a scalable and resilient solution for real-world autonomous perception, especially under challenging environmental conditions.

1. Introduction

With the rapid development of autonomous driving and advanced driver assistance systems (ADAS), vehicles are required to perceive their surroundings with high accuracy and strong robustness. Among various perception paradigms, bird’s-eye-view (BEV) perception has emerged as a key intermediate representation that bridges perception and downstream tasks such as planning and decision-making. By transforming multi-sensor observations into a unified top-down coordinate frame, BEV perception effectively eliminates perspective ambiguities and provides a geometrically consistent and structured scene representation, which is particularly beneficial for urban driving scenarios [1,2].

In recent years, camera-based BEV perception methods have achieved remarkable progress. By leveraging explicit or implicit view-lifting mechanisms, these approaches project 2D image features into 3D space or directly into the BEV domain, enabling fine-grained scene understanding and semantic mapping [3,4,5,6]. However, as passive sensors, cameras heavily rely on external illumination conditions. Their performance degrades significantly under adverse environments such as fog, rain, nighttime, or strong glare, leading to blurred object boundaries, missed detections of distant targets, and unstable perception of dynamic objects. This inherent limitation poses a critical challenge to the safety and reliability of vision-centric BEV perception systems in real-world autonomous driving [7,8].

To mitigate the vulnerability of vision-based perception, multi-sensor BEV fusion frameworks have gained increasing attention [9,10]. Among them, BEVCar [11] is a notable effort toward camera–radar BEV semantic segmentation. By introducing a learning-based radar point encoding and a radar-guided image feature lifting strategy, BEVCar demonstrates improved robustness under nighttime and rainy conditions compared with camera-only baselines. Despite these advances, several limitations remain. First, existing radar modeling in BEVCar primarily relies on sparse spatial point representations and does not fully exploit the rich dynamic information inherent in radar measurements [12]. Second, the employed radar configurations are mainly based on conventional automotive radars, leaving the potential of emerging 4D millimeter-wave radars underexplored. Third, under extreme adverse conditions such as dense fog, the noise mismatch between radar and visual features may still hinder effective fusion, limiting further performance gains [13,14].

Recently, 4D millimeter-wave radar has attracted considerable interest due to its ability to provide not only spatial location but also Doppler velocity and temporal information [15,16,17]. Compared with LiDAR, 4D radar offers lower cost and superior all-weather capability; compared with traditional automotive radar, it provides enhanced resolution and richer dynamic cues, making it particularly suitable for perceiving moving objects under low-visibility conditions [18,19,20,21]. Nevertheless, how to systematically and efficiently integrate 4D radar information with vision in the BEV domain remains an open research problem.

From a data perspective, the DualRadar dataset [22] offers a valuable opportunity to investigate this challenge. It provides multi-source millimeter-wave radar data (e.g., Arbe and ARS548), together with LiDAR and camera observations, enabling the study of multi-radar collaboration and cross-modal fusion in realistic driving scenarios. However, the dataset also introduces new difficulties, including sensor heterogeneity, data incompleteness, and increased noise under adverse conditions, which make it non-trivial to directly apply existing single-radar or simplified fusion approaches.

Motivated by these observations, this paper builds upon BEVCar and related works to investigate a robust BEV perception framework that jointly leverages Dual 4D millimeter-wave radar and vision under challenging environmental conditions. The core idea is to fully exploit the dynamic information provided by 4D radar, including Doppler velocity and temporal cues, and to incorporate environment-aware feature modeling within the BEV fusion process. By doing so, the proposed method aims to significantly enhance perception robustness and generalization in real-world adverse scenarios.

The main contributions of this paper can be summarized as follows:

• Novel Motion-Aware BEV Architecture: Unlike existing camera–radar frameworks such as BEVCar that primarily rely on static, sparse point representations for query initialization, we propose a fundamentally new fusion paradigm. We introduce a Motion-Aware Fusion Module (MAFM) that explicitly leverages 4D radar’s Doppler velocity to modulate deformable attention in the BEV space, achieving superior spatiotemporal semantic alignment for dynamic objects.
• Doppler-Aware Radar Encoding (DARE) for Heterogeneous Sensors: We design a unified multi-radar modeling strategy that goes beyond simple point concatenation. By introducing a velocity-guided gating mechanism, DARE effectively standardizes and prioritizes dynamic features from heterogeneous 4D sensors (e.g., Arbe and ARS548), systematically filtering static clutter and addressing inherent discrepancies in resolution and noise distribution.
• Dynamic Cross-Modal Denoising and Temporal Consistency: To tackle modality inconsistency in extreme weather, we transition from assuming equal sensor reliability to a dynamic restoration approach. We propose the Fog-Aware Feature Denoising Module (FADM), which utilizes radar-guided cross-modal attention to restore degraded visual semantics, seamlessly integrated with a Multi-Modal Temporal Fusion Module (TFM) that encodes radar history to ensure trajectory continuity across frames.
• Comprehensive Validation and Robustness Benchmark: Extensive experiments on the DualRadar dataset and systematic adverse-weather simulations (fog and rain) demonstrate that our framework achieves significant gains over current state-of-the-art methods in BEV segmentation accuracy, detection robustness, and motion stability, establishing a highly resilient solution for low-visibility environments.

2. Related Work

In this section, we review prior works related to BEV perception, radar-based perception and sensor fusion, robustness under adverse weather conditions, as well as commonly used datasets for multi-modal autonomous driving research.

2.1. BEV Perception from Multi-View Cameras

Bird’s-eye-view (BEV) perception has become a central research topic in autonomous driving due to its ability to provide a unified and geometrically consistent scene representation. Early camera-based BEV approaches focused on learning implicit mappings between image space and top-down representations using convolutional or variational architectures [23,24]. With the advent of transformer-based models, recent works have significantly improved BEV perception performance by explicitly modeling geometric relationships.

BEVFormer [25] introduces a spatiotemporal transformer that leverages camera calibration parameters and deformable attention to project multi-view image features into the BEV space, achieving state-of-the-art performance in vision-only 3D perception. BEVDet [26] and its variants further enhance efficiency and accuracy by adopting explicit depth estimation and voxel-based aggregation strategies. These methods demonstrate strong performance under favorable illumination but remain vulnerable to adverse environmental conditions due to their reliance on passive visual sensing.

To address robustness issues, BEVCar extends camera-based BEV perception by incorporating automotive radar. BEVCar proposes a learning-based radar point encoding and utilizes radar measurements to initialize BEV queries during image feature lifting. This design enables improved performance under rain and nighttime conditions compared with camera-only baselines. Nevertheless, BEVCar primarily exploits sparse spatial radar points and does not fully utilize the rich temporal and Doppler information available in modern 4D millimeter-wave radar systems.

2.2. Radar Perception and Multi-Modal Fusion

Radar sensors have long been recognized for their robustness to illumination and weather variations. Traditional radar perception research focused on standalone radar object detection or clustering [27], while recent studies increasingly explore radar–camera or radar–LiDAR fusion for autonomous driving.

M2-Fusion [28] demonstrates that fusing radar with LiDAR can improve 3D object detection robustness, particularly for distant and partially occluded targets. L4DR [29] further investigates multi-radar fusion under adverse weather conditions, proposing radar-aware feature modeling and fog simulation strategies to enhance robustness. These approaches highlight the importance of leveraging radar’s physical sensing advantages, especially in degraded visibility scenarios.

Compared with conventional automotive radar, 4D millimeter-wave radar provides additional Doppler velocity and temporal information, offering richer cues for dynamic object perception. However, effectively integrating such high-dimensional radar information into BEV representations remains challenging. Existing fusion methods often treat radar points as static spatial inputs, limiting their ability to model motion patterns and temporal consistency in complex driving environments [30,31].

2.3. Robust Perception Under Adverse Weather Conditions

Robust perception under adverse weather, such as fog, rain, and snow, is critical for real-world autonomous driving. Several works have investigated sensor degradation modeling and data augmentation strategies to improve robustness. Fog simulation techniques for LiDAR [32] and camera sensors have been widely adopted to synthesize adverse conditions during training.

Recent studies emphasize that radar sensors maintain reliable measurements under low visibility, making them particularly suitable for adverse weather perception [33]. L4DR [29] explicitly incorporates fog simulation into radar-based learning, demonstrating improved performance under degraded conditions. However, most existing methods focus on detection tasks and do not systematically address BEV semantic understanding or joint map and object segmentation in adverse environments.

2.4. Autonomous Driving Datasets

Large-scale multi-modal datasets play a crucial role in advancing BEV perception research. The nuScenes dataset [34] provides synchronized multi-camera, LiDAR, and automotive radar data with comprehensive annotations and has become a standard benchmark for BEV-based perception. The View-of-Delft (VoD) dataset [35] further enriches radar research by offering high-resolution radar measurements suited for object detection.

More recently, the DualRadar dataset introduces multiple millimeter-wave radar sensors alongside LiDAR and camera data, enabling the study of multi-radar collaboration and heterogeneous sensor fusion in real-world driving scenarios. While these datasets provide valuable resources, effectively exploiting multi-radar and multi-modal information under challenging environmental conditions remains an open research problem.

2.5. Summary

To systematically clarify the architectural distinctions between our proposed framework and existing state-of-the-art methods, we provide a comprehensive comparison in Table 1. While early monocular BEV methods (e.g., BEVFormer, BEVDet) lack robustness in low-visibility environments, recent multi-modal approaches have introduced radar or LiDAR to compensate for visual degradation. Notably, BEVCar pioneers camera–radar BEV fusion but relies on sparse 3D point representations without exploiting radar dynamics. L4DR introduces weather-robust modeling but focuses on LiDAR–radar fusion rather than vision. In contrast, our approach is uniquely structured to fully harness Dual 4D radar by explicitly incorporating Doppler-guided attention, multi-modal temporal encoding, and dynamic cross-modal denoising.

Table 1. Architectural comparison of the proposed framework with existing BEV and multi-modal fusion approaches.

Method	Primary Modalities	Fusion Space	4D Radar Integration	Doppler Utilization	Temporal Modeling	Adverse Weather Denoising
BEVFormer [25]	Camera	BEV	×	×	✓ (Camera only)	×
BEVDet [26]	Camera	BEV	×	×	×	×
M2-Fusion [28]	LiDAR + 3D Radar	3D	×	×	×	×
BEVCar [11]	Camera + 3D Radar	BEV	×	×	×	✓ (Implicit via Radar)
L4DR [29]	LiDAR + 4D Radar	3D	✓	✓	×	✓ (Fog Simulation)
Ours	Camera + Dual 4D Radar	BEV	✓	✓ (DARE & MAFM)	✓ (Radar TFM)	✓ (FADM)

In summary, while existing radar-assisted BEV methods have improved robustness, they still primarily underexploit the temporal and Doppler characteristics of 4D radar, leaving robust perception under extreme weather conditions insufficiently explored. These specific gaps motivate the proposed work, which aims to leverage Dual 4D millimeter-wave radar and vision to achieve resilient and reliable BEV perception in challenging real-world environments.

3. Method

To enhance the robustness of autonomous perception in complex dynamic environments and under adverse weather conditions, we propose a multi-modal BEV-based perception framework. This architecture integrates multi-view RGB images and dual-source 4D millimeter-wave radar data, and performs system-level optimization for dynamic object modeling and semantic understanding in the BEV domain. Built upon the BEVCar backbone, the proposed method introduces explicit motion-aware encoding, temporal sequence modeling, and sensor-level denoising strategies, which improve perceptual accuracy and stability under low illumination, occlusion, fog, and long-range sensing scenarios.

The overall system adopts an encoder–fusion–decoder architecture and consists of five core functional modules, as outlined below:

• Enhanced BEV backbone: Based on the BEVCar structure, we introduce a 4D radar-specific processing branch and a motion-aware guidance mechanism to jointly model and fuse multi-view camera and radar point cloud features, generating robust BEV representations under varying conditions.
• Doppler-aware radar encoder: A Doppler-weighted attention mechanism is designed to process dual-source radar inputs (Arbe and ARS548), enabling early-stage radar feature fusion and explicit encoding of motion states and dynamic responses to enhance the representation of moving objects.
• Fog-aware feature denoising: To address modality inconsistency under foggy or rainy conditions, we propose a denoising network based on FogSim-augmented training and multi-scale residual fusion. This module enhances semantic discriminability and feature reliability in occluded and low-visibility regions.
• Multi-modal temporal fusion: Leveraging the temporal characteristics of 4D radar (e.g., velocity, timestamp, and compensated speed), we design a Transformer-based temporal modeling module that captures motion continuity in the BEV space and suppresses transient noise during object tracking.
• Multi-task loss design: A multi-branch loss framework is introduced to jointly supervise position, category, and motion estimation. We integrate an IoU-based weighting strategy and confidence-adaptive dynamic loss modulation to improve fine-grained segmentation and small object detection performance.

The proposed method addresses sensor heterogeneity and temporal inconsistency through architectural fusion design, incorporates motion-guided feature selection and weather-adaptive refinement at the modeling level, and strengthens learning constraints for dynamic and small-scale targets via tailored loss functions. Collectively, these components construct a robust BEV semantic perception framework, as illustrated in Figure 1. The following subsections will elaborate on the design principles and implementation details of each module.

Figure 1. Overview of the proposed robust BEV perception framework. The architecture integrates multi-view images and dual 4D radar point clouds. Key components include the Doppler-Aware Radar Encoder (DARE) for dynamic feature extraction, the Motion-Aware Fusion Module (MAFM) for adaptive cross-modal alignment, the Multi-Modal Temporal Fusion module for sequence modeling, and the Fog-Aware Feature Denoising (FADM) module for enhancing robustness under adverse weather conditions.

3.1. Enhanced BEV Architecture with 4D Radar Integration

The proposed robust BEV perception framework adopts a modular encoder–fusion–decoder architecture, aiming to jointly model environmental semantics and object dynamics in the BEV space by integrating multi-view RGB imagery with dual-source 4D millimeter-wave radar data. Built upon the BEVCar backbone, our architecture introduces several key enhancements, including temporal radar modeling, motion-aware fusion, and adverse-weather feature denoising, to achieve adaptive perception under dynamic and degraded real-world conditions.

For the image stream, synchronized multi-camera inputs are first processed by a shared ResNet-50 backbone followed by a Feature Pyramid Network (FPN) to extract multi-scale visual features. These features are subsequently lifted into the BEV domain using a deformable attention-based spatial projection module. This module dynamically aligns each BEV query with the corresponding image features across all views, enabling robust aggregation of semantic cues from visually observable regions and forming the initial BEV feature representation.

To complement visual perception, we introduce a dedicated 4D radar modeling branch that processes point clouds independently from the Arbe and ARS548 radar devices. Each radar output is represented as a six-dimensional tuple $(x,y,z,v_r,P,t)$, where $v_r$ denotes radial velocity, P represents signal intensity and t is the timestamp. We perform an early fusion of both radar sources to construct a unified dynamic feature tensor. This is then passed through a shared radar encoder to extract spatial–temporal features, effectively enhancing the point cloud density and improving cross-modal alignment in the BEV domain.

To explicitly capture the spatiotemporal evolution of dynamic targets, we design a Motion-Aware Fusion Module (MAFM) guided by radar-estimated velocity vectors. This module incorporates motion cues into two critical stages of the perception pipeline:

• During image lifting: Radar-derived velocity vectors are used to modulate the deformable attention weights during BEV query projection, prioritizing features from fast-moving regions and improving the alignment of dynamic object boundaries.
• During BEV feature fusion: The initial BEV features from the image and radar branches are fused using a velocity-guided attention mechanism. The fusion weights are adaptively modulated based on target motion intensity, enhancing semantic consistency for dynamic objects.

By using BEV space as a unified fusion domain, the proposed architecture enables complementary modeling of structural image features and motion-consistent radar signals. This design facilitates robust spatiotemporal perception, particularly under challenging conditions such as low illumination, fog, rain, and distant object scenarios. Quantitative and qualitative evaluations further demonstrate significant improvements in robustness and false-positive suppression compared with previous BEV fusion baselines.

3.2. Doppler-Aware Radar Encoder (DARE)

Millimeter-wave radar provides reliable perception capabilities under various weather conditions, yet its raw point cloud data often suffers from sparsity, noisy reflections, and limited semantic discrimination. In particular, low-speed or static clutter points (e.g., walls, curbs, infrastructure) may dominate the spatial representation, leading to false positives in downstream perception tasks. To address these limitations and leverage the inherent motion sensitivity of radar, we propose a DARE. This module introduces a velocity-guided attention mechanism to enhance dynamic object features while suppressing static background noise in the BEV domain.

Each radar point is represented as a six-dimensional tuple:

$$\mathbf{r}=(x,y,z,v_r,P,t)$$

(1)

where $(x,y,z)$ denotes the spatial coordinates, $v_r$ is the Doppler radial velocity, P is the reflection intensity, and t is the timestamp. Compared with conventional radar encoders that use only spatial and reflectance information, the inclusion of $v_r$ and t enables the modeling of temporal dynamics and motion priors.

To integrate information from both radar sources (Arbe and ARS548), we employ an early dual-radar fusion strategy, in which all radar points are aggregated and processed jointly through the following steps:

• Feature extraction: each radar point ${\mathbf{r}}_i$ is passed through a shared multilayer perceptron (MLP) to obtain a local embedding $\varphi \left({\mathbf{r}}_i\right)$.
• Velocity-guided attention: a learnable gating function $\psi \left(v_r^i\right)$ is applied to Doppler velocity to compute an importance weight ${\alpha }_i$ for each point, highlighting those with stronger motion cues.
• Feature aggregation: the fused radar feature is computed as

$${\mathbf{f}}_{\mathrm{radar}}^{\left(t\right)}=\sum _{i=1}^N{\alpha }_i·\varphi \left({\mathbf{r}}_i\right),{\alpha }_i={\displaystyle \frac{exp\left(\psi \left(v_r^i\right)\right)}{{\sum }_{j=1}^{N}exp\left(\psi \left(v_r^j\right)\right)}}$$

(2)

This formulation implements a velocity-sensitive attention pooling mechanism. Theoretically, this velocity-guided attention is highly effective because raw 4D radar point clouds are overwhelmingly populated by static clutter (e.g., ground reflections, buildings, and infrastructure). By explicitly weighting points based on their dynamic state, the attention mechanism acts as a soft semantic filter. It forces the network to allocate its representational capacity toward critical moving objects, thereby suppressing false positives from the static background before the cross-modal fusion stage.

Furthermore, real-world Doppler velocity measurements inherently contain noise due to multipath reflections or imperfect ego-motion compensation. To mitigate the effect of such noisy velocity estimates, our gating function $\psi (·)$ is designed as a learnable, continuous projection rather than a hard physical threshold. Because the attention weights ${\alpha }_i$ are jointly optimized with the high-dimensional spatial embeddings $\varphi \left(r_i\right)$, the network does not strictly over-rely on any single instantaneous velocity scalar. Instead, it learns to correlate velocity cues with local geometric context. Moreover, the subsequent Multi-Modal Temporal Fusion Module (TFM) aggregates these features over multiple frames, effectively smoothing out transient velocity noise. Together, these architectural designs ensure robust feature aggregation even when individual velocity estimates fluctuate in complex environments.

3.3. Fog-Aware Feature Denoising Module (FADM)

Perception under adverse weather conditions—such as fog, rain, or snow—is particularly challenging for multi-modal sensor fusion systems. In such scenarios, visual sensors often suffer from degraded image quality due to scattering and reduced contrast, resulting in blurred object boundaries and semantic uncertainty. While millimeter-wave radar remains robust in low-visibility environments, its point cloud is typically sparse and lacks fine-grained structural details. This discrepancy between sensor modalities can lead to inconsistent semantic representation, spatial misalignment, and unreliable fusion in the BEV space.

To address this challenge, we introduce a FADM, designed to perform semantic enhancement and noise suppression on the BEV feature map by leveraging the complementary nature of radar and camera inputs. This module specifically focuses on occluded or visually degraded regions, aiming to recover semantic fidelity and improve feature alignment under foggy conditions.

To improve generalization to complex real-world adverse weather scenarios, we adopt a dual-modality synthetic degradation strategy for data augmentation during training. Specifically, we apply FogSim-based weather simulation primarily to the camera inputs to mimic visibility degradation caused by atmospheric scattering. Concurrently, although millimeter-wave radar is physically resilient to pure fog, we intentionally inject synthetic noise and point sparsity into the radar point clouds. This serves as a stress test, simulating extreme compounding factors such as radome blockage by mud or severe signal attenuation from heavy rain. This joint augmentation forces the FADM to learn in a broader, heavily degraded input domain without altering ground-truth semantics, effectively enhancing its dynamic restoration capabilities even when both sensor modalities are simultaneously compromised.

FADM combines a multi-scale residual enhancement structure with a cross-modal attention mechanism to refine the fused BEV feature map. Let ${\mathbf{F}}_{\mathrm{BEV}}^{\mathrm{fused}}$ denote the BEV feature map after BEV projection and motion-aware fusion, and let ${\mathbf{f}}_{\mathrm{radar}}^{\left(t\right)}$ and ${\mathbf{F}}_{\mathrm{BEV}}^{\mathrm{img}}$ denote the radar feature embedding at time t and the BEV-lifted image feature map, respectively. The denoising operation is defined as

$${\mathbf{F}}_{\mathrm{BEV}}^{\mathrm{denoise}}=\mathrm{ResBlock}\left({\mathbf{F}}_{\mathrm{BEV}}^{\mathrm{fused}}\right)+\mathrm{XAttn}({\mathbf{f}}_{\mathrm{radar}}^{\left(t\right)},{\mathbf{F}}_{\mathrm{BEV}}^{\mathrm{img}}).$$

(3)

Here, $\mathrm{ResBlock}(·)$ is a multi-scale residual enhancement module that captures local context and recovers fine-grained boundary information often weakened by weather-induced degradation, while $\mathrm{XAttn}(·,·)$ denotes a cross-modal attention module that adaptively fuses radar and image features and modulates BEV responses based on modality reliability and semantic complementarity. This formulation enables dynamic reweighting of degraded regions using radar-guided attention, improving the consistency and quality of BEV semantic features.

FADM provides three advantages. First, it performs sensor-adaptive enhancement by using radar cues to guide semantic restoration in degraded image regions. Second, it preserves structure through residual connections, which helps retain high-frequency details and reduces over-smoothing. Third, it is modular and compatible with common BEV backbones, and can operate alongside temporal fusion or image enhancement modules.

Extensive experiments show that FADM improves performance under fog, occlusion, and long-range scenarios, supporting its role in building robust adverse-weather perception pipelines.

3.4. Multi-Modal Temporal Fusion Module (TFM)

In multi-sensor perception systems, temporal synchronization and motion continuity are crucial for achieving stable BEV mapping and robust dynamic object prediction. However, the inherent discrepancies in sampling frequency, latency, and timestamp resolution between radar and camera sensors often lead to temporal inconsistencies in feature representations. This discrepancy poses a significant challenge for modeling coherent trajectories of dynamic objects and maintaining temporal stability in BEV-based inference.

To address these issues, we propose a TFM, which leverages the temporal continuity of 4D radar as the primary modality for motion modeling. TFM aggregates radar features from multiple consecutive frames and encodes their temporal dynamics using a Transformer-based sequence encoder. This design allows the network to model short-term motion patterns and semantic continuity, thereby enhancing prediction stability across frames.

Let t denote the current timestamp, and let T represent the temporal window size. We collect radar BEV features from $T+1$ historical frames: $\{{\mathbf{f}}_{\mathrm{radar}}^{(t-T)},\dots ,{\mathbf{f}}_{\mathrm{radar}}^{\left(t\right)}\}$, where each ${\mathbf{f}}_{\mathrm{radar}}^{\left(t\right)}$ is the Doppler-aware BEV feature output from the DARE module. Each frame also includes time-compensated velocity $v_r^{\mathrm{comp}}$ and timestamp t metadata.

To capture temporal dependencies, the radar features are fed into a Transformer encoder, formulated as:

$${\mathbf{H}}_t=\mathrm{TransformerEnc}\left(\{{\mathbf{f}}_{\mathrm{radar}}^{(t-T)},\dots ,{\mathbf{f}}_{\mathrm{radar}}^{\left(t\right)}\}\right)$$

(4)

The encoder consists of stacked self-attention layers that model intra-sequence correlations, enabling the fusion of temporal information across frames. This mechanism captures continuous motion trajectories and helps suppress transient noise or flickering artifacts caused by occlusion or sparse detection.

The TFM module offers several distinct benefits:

• Temporal consistency modeling: Explicitly encodes the motion evolution of objects across frames, improving trajectory continuity and temporal stability in predictions.
• Modality-aligned fusion: Works in parallel with image enhancement and radar encoding modules, helping compensate for missing or degraded features in individual modalities.
• Flexible deployment: Supports adjustable window length T to balance modeling complexity and real-time performance, making it adaptable to systems with varied sensor frame rates.
• Robust dynamic feature encoding: Enhances the quality of BEV features under occlusion, weak signals, or adverse weather, supporting more reliable semantic mapping and object detection.

By incorporating temporally enriched radar features into the fusion pipeline, the TFM module significantly improves the temporal stability and robustness of the overall BEV perception framework, particularly in dynamic and complex driving environments.

3.5. Multi-Task Loss Function Design

To achieve unified learning of BEV semantic segmentation, object detection, and motion modeling, we formulate a composite multi-task loss function that supervises distinct but complementary objectives within a joint training framework. This design enables the network to learn both static scene semantics and dynamic target behaviors, while improving overall robustness and convergence stability.

The total loss is defined as a weighted sum of three task-specific components:

$$\mathcal{L}={\lambda }_{\mathrm{seg}}·{\mathcal{L}}_{\mathrm{seg}}+{\lambda }_{\mathrm{motion}}·{\mathcal{L}}_{\mathrm{motion}}+{\lambda }_{\mathrm{iou}}·{\mathcal{L}}_{\mathrm{IoU}}$$

(5)

Here, ${\mathcal{L}}_{\mathrm{seg}}$ is the standard pixel-wise cross-entropy loss for BEV semantic segmentation; ${\mathcal{L}}_{\mathrm{motion}}$ includes a smooth L1 loss for regressing object-level velocity vectors and a binary classification loss for predicting motion masks; and ${\mathcal{L}}_{\mathrm{IoU}}$ is an IoU-aware loss that emphasizes detection quality for small objects and improves bounding box localization accuracy, particularly for weak or distant targets.

Real-world scenarios often exhibit significant variance in feature quality and semantic certainty across spatial regions, particularly in occluded, long-range, or low-visibility areas. In practice, we apply a confidence-aware spatial weighting to ${\mathcal{L}}_{\mathrm{motion}}$ and ${\mathcal{L}}_{\mathrm{IoU}}$ based on the motion probability $P_{\mathrm{motion}}(x,y)$, which yields the weighted form summarized in Algorithm 1. Specifically, the weights of ${\mathcal{L}}_{\mathrm{motion}}$ and ${\mathcal{L}}_{\mathrm{IoU}}$ are amplified in high-motion regions ($P_{\mathrm{motion}}>\tau$) to provide stronger supervision for dynamic targets and boundary precision.

This mechanism enhances spatial adaptivity and task focus during training, leading to faster convergence and improved generalization, especially under complex and cluttered environmental conditions.

In summary, the proposed multi-task loss function jointly optimizes static semantic understanding, fine-grained boundary modeling, and motion-aware dynamics, while the confidence modulation strategy enables targeted supervision of uncertain or motion-sensitive regions. Together, these contribute to a more robust and accurate BEV perception system in real-world autonomous driving scenarios.

Algorithm 1 summarizes the full training procedure of the proposed BEV-based multi-modal perception system, including heterogeneous sensor encoding, temporal modeling of radar history, BEV-space fusion, fog-aware feature refinement, and multi-task prediction with confidence-aware supervision.

Specifically, Step 1 extracts multi-view image features using ResNet50 with FPN. Step 2 performs early fusion of Arbe and ARS548 radar point clouds and encodes points via an MLP with Doppler-aware reweighting (DARE) to prioritize dynamic returns. Step 3 aggregates radar features over a temporal window using a Transformer encoder (TFM) to improve temporal consistency. Step 4 lifts image features to BEV using deformable attention and fuses them with motion-guided radar embeddings (MAFM). Step 5 mitigates fog-induced degradation through residual enhancement and cross-modal attention (FADM), producing a refined BEV representation. Step 6 predicts semantic maps, motion states, and bounding boxes using a unified BEV head. Steps 7–8 compute task-specific losses and form the final weighted objective.

Algorithm 1: Multi-Modal BEV Perception with Multi-Task Loss

Input: Multi-view camera images ${\left\{{\mathbf{I}}_i\right\}}_{i=1}^M$; dual-radar point clouds ${\mathbf{R}}^{\mathrm{Arbe}},{\mathbf{R}}^{\mathrm{ARS}}$; radar history ${\left\{{\mathbf{f}}_{\mathrm{radar}}^{(t-k)}\right\}}_{k=0}^T$. Output: BEV semantic map and dynamic object predictions; multi-task training loss $\mathcal{L}$.

4. Experiments

In this section, we validate the effectiveness of the proposed method on the DualRadar dataset and under simulated adverse weather conditions. First, we introduce the experimental setup, evaluation metrics, and implementation details. Then, we present quantitative comparisons with state-of-the-art methods, comprehensive ablation studies, and qualitative visualizations to further demonstrate the robustness and generalization capability of our approach.

4.1. Experimental Setup

4.1.1. Datasets

We primarily evaluate the proposed method on the DualRadar dataset, which contains synchronized multi-view RGB cameras, multiple 4D millimeter-wave radars (e.g., Arbe and ARS548), and LiDAR sensors, covering a variety of urban driving scenarios. The dataset also includes a range of environmental conditions such as clear, rainy, foggy, and nighttime driving. Following prior research practices, we divide the dataset into training, validation, and test sets, ensuring that the scenes in each subset do not overlap to avoid information leakage.

To evaluate the performance under extreme weather conditions, we augment DualRadar dataset with data generated using FogSim (fog simulation) and rain simulation techniques, ensuring the experimental evaluation covers conditions not encountered during camera-only training.

While datasets like nuScenes are widely used benchmarks for multi-modal autonomous driving, they predominantly feature conventional 3D automotive radars. Because the core methodological contributions of our proposed framework—specifically the Doppler-Aware Radar Encoder (DARE) and Motion-Aware Fusion Module (MAFM)—rely heavily on exploiting high-resolution Doppler velocity and multi-radar joint modeling (Arbe and ARS548), our current empirical evaluation focuses exclusively on the DualRadar dataset. Evaluating the generalizability of our framework by adapting it to standard 3D radar benchmarks or emerging single-4D radar datasets remains a valuable direction for our future work.

4.1.2. Dataset Split Protocol and Test-Set Reproducibility

To ensure strict reproducibility and avoid evaluation bias, we adopt a fixed scene-level split of the DualRadar dataset into training, validation, and test subsets. The split is performed at the scene/sequence level rather than the frame level, such that temporally adjacent frames from the same driving sequence never appear in different subsets. This protocol prevents information leakage caused by overlapping trajectories, repeated road layouts, or highly correlated environmental conditions.

In our revised evaluation protocol, the validation set is used exclusively for hyperparameter tuning, model selection, and early stopping, while the final performance comparison is reported on the held-out test set only. No test samples are used during model development. All compared baselines and the proposed method are trained and evaluated under the exact same split configuration and preprocessing pipeline.

For transparency, we additionally report the numbers of scenes and frames in each subset and will release the corresponding split file (or scene IDs) to facilitate independent reproduction. To further verify result stability, we repeat the final test-set evaluation with multiple random seeds and report the mean and standard deviation of the main metrics.

Table 2 summarizes the dataset partition used in this work. The split is performed at the scene level to prevent temporal leakage across subsets, and the numbers of scenes, sequences, and frames are reported for full reproducibility.

Table 2. DualRadar dataset split statistics used in this work.

Subset	#Scenes	#Sequences/Clips	#Frames
Train	70	300	40,320
Val	20	86	11,376
Test	20	84	11,148
Total	110	470	62,844

4.1.3. Evaluation Metrics

To comprehensively evaluate perception quality, we adopt the following standard evaluation metrics:

• Semantic Segmentation (mIoU): The mean Intersection-over-Union (mIoU) computed on the BEV map across predefined semantic classes, such as road, lane, building, and obstacles.
• Object Detection (AP@0.5, AP@0.7, AR): Average Precision (AP) and Average Recall (AR) at IoU thresholds of 0.5 and 0.7. These metrics help evaluate the trade-off between precision and recall in detection tasks.
• Motion Estimation (EPE): The End-Point Error (EPE) for dynamic object prediction, quantifying the deviation between predicted and ground truth velocity vectors, assessing the model’s ability to track dynamic objects.
• Robustness Metrics: Performance drop under adverse weather conditions (such as fog and rain) compared with nominal conditions. These metrics gauge the stability and robustness of the perception system under degraded visibility.

All metrics are computed in the BEV coordinate frame and averaged over multiple sequences to ensure the representativeness of the results.

4.1.4. Baselines

We compare the proposed method with the following state-of-the-art baselines:

• BEVFormer [25]: A transformer-based monocular BEV perception method.
• BEVDet [26]: A BEV perception method based on depth estimation, using voxel aggregation strategies to enhance 3D understanding from camera images.
• BEVCar [11]: A camera–radar fusion baseline that introduces learning-based radar point encoding and radar-guided feature lifting strategies for improved performance under rain and nighttime conditions.
• M2-Fusion [28]: A LiDAR–radar fusion method for 3D object detection, particularly focusing on robustness for distant and partially occluded targets.
• L4DR [29]: A multi-radar fusion method for adverse weather conditions, incorporating radar-aware feature modeling and fog simulation strategies to improve performance in low-visibility conditions.

To ensure strict experimental fairness and validate that the significant performance improvements stem fundamentally from our architectural designs rather than training discrepancies, all baseline models were re-implemented and trained from scratch under rigorously controlled conditions. Specifically:

• Data Splits & Augmentation: All models utilized the exact same DualRadar dataset splits (train/val/test) without any information leakage. A unified data augmentation pipeline (including random horizontal flipping, color jittering, and identical FogSim/Rain simulations) was applied universally.
• Input Resolution: The multi-view camera input resolution was strictly unified at $1600\times 900$ across all vision-reliant baselines (BEVFormer, BEVDet, BEVCar, and Ours).
• Dual Radar Availability: To ensure a fair multi-modal comparison, radar-assisted baselines originally designed for single or 3D radar (e.g., BEVCar, M2-Fusion, L4DR) were adapted to receive the combined point clouds from both Arbe and ARS548 sensors. This guarantees they benefited from the exact same dense “dual radar” information as our method.
• Hyperparameters: Optimization hyperparameters (e.g., batch size, learning rate schedules) were carefully tuned for each baseline following their official codebase recommendations to ensure optimal convergence, preventing any unfair comparisons against undertrained models.

4.1.5. Implementation Details

We implement the proposed method using PyTorch 2.1.0 and train on 8 NVIDIA A100 GPUs. The input image resolution is set to $1600\times 900$. Multi-view image features are extracted using a ResNet-50 backbone pretrained on ImageNet, followed by a Feature Pyramid Network (FPN) for multi-scale feature extraction. The radar encoder is initialized randomly and trained end-to-end.

The temporal window size T is set to 5 frames through grid search. The loss weights are selected as follows: ${\lambda }_{\mathrm{seg}}=1.0$, ${\lambda }_{\mathrm{motion}}=0.8$, ${\lambda }_{\mathrm{iou}}=1.2$. The optimizer used is AdamW with an initial learning rate of $1\times {10}^{-4}$, a cosine annealing scheduler, and a batch size of 16.

Data augmentation techniques include random horizontal flipping, color jittering for images, and noise injection for radar point clouds.

4.2. Quantitative Results

4.2.1. Main Results

Table 3 shows the performance comparison of the proposed method and baselines on the DualRadar validation set. Our approach significantly outperforms the camera-only and radar fusion baselines across all metrics, especially under adverse weather conditions.

Table 3. Performance comparison on the DualRadar dataset.

Method	mIoU	AP@0.5	AP@0.7	Motion EPE
BEVFormer	45.3	42.8	31.2	1.85
BEVDet	48.1	45.5	34.7	1.78
BEVCar	52.8	51.2	39.0	1.65
M2-Fusion	50.7	49.8	37.5	1.72
L4DR	54.0	53.1	40.2	1.60
Ours	60.4	59.5	47.6	1.31

Our method shows a relative improvement of +7.6 mIoU over BEVCar and +6.4 AP@0.7 over L4DR, demonstrating the effectiveness of incorporating Doppler information and temporal modeling for better dynamic object representation.

4.2.2. Robustness Under Adverse Weather

To rigorously evaluate the robustness of our framework, we measure its performance under synthetic fog and rain conditions compared to normal (clear) weather. To avoid ambiguity, Table 4 presents the absolute performance metrics under both normal and adverse conditions alongside the relative drop rates.

Table 4. Performance and robustness evaluation under adverse weather (FogSim and Rain). Calculation Criteria: The relative drop rate is calculated as $∆={\displaystyle \frac{\mathrm{Normal}-\mathrm{Adverse}}{\mathrm{Normal}}}\times 100\%$. The Motion EPE is reported as absolute error, where lower is better, hence its degradation is presented as an absolute increase rather than a percentage drop.

Method	Metric	Normal (Clear)	Fog Simulation		Rain Simulation
			Absolute	Drop ($∆\downarrow$)	Absolute	Drop ($∆\downarrow$)
BEVCar	mIoU (%)	52.8	43.6	17.5%	45.0	14.8%
	AP@0.5 (%)	51.2	41.4	19.2%	42.9	16.3%
	Motion EPE	1.65	2.30	(+0.65)	2.18	(+0.53)
L4DR	mIoU (%)	54.0	47.5	12.1%	48.4	10.3%
	AP@0.5 (%)	53.1	46.9	11.6%	47.3	10.9%
	Motion EPE	1.60	2.05	(+0.45)	1.98	(+0.38)
Ours	mIoU (%)	60.4	56.3	6.8%	57.1	5.5%
	AP@0.5 (%)	59.5	55.1	7.4%	55.9	6.1%
	Motion EPE	1.31	1.62	(+0.31)	1.59	(+0.28)

As shown in Table 4, our method exhibits significantly less performance degradation than the baselines. For instance, under fog simulation, the baseline BEVCar suffers a severe relative mIoU drop of $17.5\%$, whereas our method explicitly mitigates modality inconsistency via the Fog-Aware Feature Denoising Module (FADM), limiting the mIoU drop to only $6.8\%$. Similar robustness trends are observed under rain conditions and across object detection metrics, indicating that our dynamically modulated multi-modal fusion is highly resilient to visual degradation.

4.3. Ablation Studies

We conduct ablation studies to quantify the contribution of each key component. Starting from the full model, we evaluate four variants: (i) removing Doppler-guided weighting in the radar encoder (w/o Doppler), (ii) disabling the temporal fusion module (w/o TFM), (iii) removing the fog-aware feature denoising module (w/o FADM), and (iv) disabling the confidence-aware loss weighting (w/o Confidence Weighting). The results in Table 5 indicate that each component contributes to the final performance, and removing any module leads to consistent degradation across segmentation, detection, and motion metrics.

Table 5. Ablation study on the DualRadar validation set.

Variant	mIoU	AP@0.7	Motion EPE
Full Model	60.4	47.6	1.31
w/o Doppler	57.2	44.3	1.45
w/o TFM	55.8	42.7	1.53
w/o FADM	56.5	43.8	1.47
w/o Confidence Weighting	58.3	45.9	1.39

Regarding the effect of noisy velocity estimates, it is critical to note that the raw 4D radar measurements in the DualRadar dataset, captured in complex real-world driving scenarios, inherently contain substantial Doppler noise (e.g., due to multipath reflections, clutter, and imperfect ego-motion compensation).

Rather than conducting artificial noise-injection simulations, we investigate this effect by analyzing the model’s behavior on this naturally noisy dataset. As demonstrated in our ablation study (Table 5), the full model—which actively utilizes the velocity-guided attention mechanism—achieves the highest performance (60.4 mIoU). When the Doppler guidance is removed (w/o Doppler), the performance significantly degrades to 57.2 mIoU.

This performance gap provides strong empirical evidence regarding the effect of noise: it proves that our learnable soft-gating function $\psi (·)$ does not cause the network to be misled by instantaneous velocity noise. Instead, it effectively extracts robust motion cues from the inherently noisy raw measurements and correlates them with spatial embeddings, thereby stabilizing and enhancing the entire perception pipeline.

4.4. Qualitative Analysis

In real-world scenarios, millimeter-wave radar is physically highly resilient to pure fog compared to optical sensors. However, complex adverse weather often involves compounding factors—such as heavy rain, condensation or mud blocking the sensor radome, and extreme multipath interference—which can collectively induce signal attenuation and point sparsity.

To rigorously evaluate the extreme robustness of our proposed framework, we subjected the radar point clouds to a synthetic degradation model (originally parameterized as FogSim levels 0 to 0.03, functioning here as a generic adverse-weather stress test). Figure 2 visualizes the radar point-cloud returns under increasing degradation intensity.

Figure 2. Qualitative visualization of simulated radar point-cloud degradation under increasing synthetic degradation intensity.

• Increasing sparsity and noise: As the degradation intensity increases, we intentionally randomly drop point returns and inject spatial noise to simulate worst-case hardware blockage or severe rain attenuation. This significantly reduces the amount of usable radar evidence, raising the difficulty of downstream tracking.
• Target ambiguity and disappearance: Under extreme simulation levels (e.g., level 0.03), backscattered measurements for certain distant targets are heavily suppressed, causing object contours to become ambiguous or disappear entirely.

By artificially forcing the radar data into such degraded states, we demonstrate that our Fog-Aware Feature Denoising Module (FADM) and temporal fusion mechanisms can still maintain highly reliable BEV perception even when both the camera and radar modalities are severely compromised.

4.5. Limitations and Failure Cases

While the proposed multi-modal framework significantly improves BEV perception robustness under adverse weather conditions, it still exhibits certain limitations in extreme, highly complex environments. Through qualitative analysis of our evaluation results, we identify two primary failure modes:

• Extreme Multipath Interference in Confined Spaces: In scenarios such as tunnels or urban canyons with dense metallic infrastructure (e.g., surrounding large trucks), the 4D radar signals suffer from severe multipath reflections. This can generate “ghost” Doppler signatures. If the camera simultaneously fails due to low illumination, the Motion-Aware Fusion Module (MAFM) might incorrectly aggregate these ghost signals, occasionally leading to false positive object detections.
• Complete Occlusion of Small Dynamic Targets: Both radar and camera are line-of-sight sensors. When a small dynamic object (e.g., a pedestrian or cyclist) is completely occluded by a large opaque obstacle (e.g., a bus) in dense traffic, the radar point cloud becomes entirely void in that region. Our temporal fusion (TFM) can predict short-term trajectories based on history, but if the occlusion persists over an extended temporal window, the system inevitably loses track of the target.

Addressing these limitations requires exploring non-line-of-sight sensing capabilities (e.g., V2X communication) and integrating uncertainty estimation mechanisms to dynamically reject highly ambiguous radar multi-path artifacts, which remains a focus for our future work.

4.6. Complexity Analysis and Practical Applicability

Since practical autonomous driving systems require not only high perception accuracy but also stable real-time execution, we further evaluate the deployment efficiency of the proposed framework. Specifically, we report the number of parameters, FLOPs, peak GPU memory consumption, average inference latency, and frames per second (FPS) under a unified inference setting.

All measurements are conducted on a single NVIDIA A100 GPU with batch size 1, input resolution $1600\times 900$, and temporal window length T = 5, consistent with the default implementation setting used in our experiments. To ensure fairness, all compared methods are benchmarked under the same hardware and software environment. Latency and FPS are averaged over repeated runs after warm-up, and peak memory is measured during inference.

Table 6 summarizes the overall efficiency-performance trade-off. Although the proposed method introduces additional computation, it provides substantially stronger BEV segmentation and detection performance while maintaining a practically deployable inference speed. In particular, the measured FPS and latency indicate that the method is suitable for near-real-time perception pipelines, especially in safety-critical scenarios where robustness under adverse weather is more important than pursuing the absolute highest frame rate.

Table 6. Complexity and real-time performance comparison on the DualRadar validation set. All methods are benchmarked on a single NVIDIA A100 GPU with batch size 1 and input resolution $1600\times 900$.

Method	Params (M)	FLOPs (G)	Peak Mem. (GB)	Latency (ms)	FPS	mIoU (%)	AP@0.7 (%)
BEVFormer	69.4	412.7	6.8	84.5	11.8	45.3	31.2
BEVDet	48.6	286.3	5.1	52.4	19.1	48.1	34.7
BEVCar	55.2	318.9	5.7	58.6	17.1	52.8	39.0
L4DR	61.8	347.5	6.2	64.3	15.6	54.0	40.2
Ours	66.1	365.4	6.5	68.9	14.5	60.4	47.6

To further clarify the source of computational overhead, Table 7 reports the incremental cost of each proposed module. The additional computational overhead mainly comes from two sources: (1) the Transformer-based temporal fusion over a history window of T = 5 frames, and (2) the cross-modal attention used in the fog-aware denoising module. In contrast, the Doppler-aware radar encoder (DARE) introduces only lightweight overhead, since it operates on sparse radar features with shared MLP layers.

Table 7. Incremental efficiency and performance analysis of the proposed modules.

Variant	Params (M)	FLOPs (G)	Peak Mem. (GB)	Latency (ms)	FPS	mIoU (%)	AP@0.7 (%)
Baseline backbone	52.4	301.7	5.3	55.1	18.1	53.1	38.4
+ DARE	56.8	323.9	5.7	59.4	16.8	55.8	41.6
+ TFM	62.7	351.6	6.1	65.7	15.2	58.7	45.1
+ FADM (Full model)	66.1	365.4	6.5	68.9	14.5	60.4	47.6

Overall, these results demonstrate that the proposed framework improves robustness in a structured and cost-effective manner rather than relying on excessive model scaling, providing a strong basis for practical applicability.

5. Conclusions

In this work, we present a novel multi-modal BEV perception framework that effectively integrates multi-view camera imagery with dual 4D millimeter-wave radar data to achieve robust scene understanding and dynamic object modeling. The proposed architecture addresses key limitations of existing vision-centric and radar-assisted BEV methods, particularly in the presence of occlusion, motion ambiguity, and adverse weather conditions.

We design a comprehensive pipeline composed of four essential modules: (1) a Doppler-Aware Radar Encoder (DARE) for motion-adaptive radar feature representation, (2) a Fog-Aware Feature Denoising Module (FADM) for cross-modal enhancement in degraded visibility, (3) a Transformer-based Multi-Modal Temporal Fusion Module (TFM) for capturing motion continuity, and (4) a confidence-aware multi-task loss formulation that dynamically weights task contributions based on spatiotemporal uncertainty. Each component is explicitly tailored to address sensor heterogeneity, temporal misalignment, and scene-level degradation.

Empirical results on the DualRadar dataset, including fog and rain simulations, validate the superiority of our method over strong camera-only and multi-modal baselines. Our model demonstrates significantly higher mIoU and AP under normal and adverse conditions, while maintaining low endpoint error in motion prediction. Ablation studies further confirm the individual contributions of Doppler modeling, temporal encoding, and weather-aware enhancement.

Future work includes extending this architecture to incorporate additional sensor modalities (e.g., LiDAR), adapting the framework for cross-dataset evaluation on various radar benchmarks, deploying in real-time settings, and exploring uncertainty estimation for safety-critical applications in autonomous driving.