Semantic-Enhanced Bidirectional Multimodal Fusion for 3D Object Detection Under Adverse Weather

Abstract

Multimodal fusion methods leveraging various sensors provide strong support for 3D object detection. However, under adverse weather conditions such as rain, fog, snow, and intense glare, complex environmental factors can degrade sensor data quality, leading to increased false positives and missed detections. In addition, sensor modalities (e.g., LiDAR and cameras) inherently vary in information density, and directly fusing them can cause critical details in high-density data to be diluted by low-density data, thereby increasing errors. To address these issues, we propose a Semantic-Enhanced Bidirectional Multimodal Fusion (SeBFusion) framework. By introducing a semantic enhancement mechanism and a bidirectional fusion strategy, SeBFusion mitigates the impact of noise under adverse weather and alleviates information dilution in multimodal fusion. Specifically, SeBFusion first employs a virtual point generation and camera semantic injection module to selectively map image semantic features into 3D space, producing semantically enhanced LiDAR features to compensate for the sparsity of the raw LiDAR point cloud. Then, during cross-modal interaction, we design a bidirectional cross-attention fusion module. This module estimates the confidence of each modality and adaptively reweights the bidirectional information flow, thereby reducing the risk of noise propagation across modalities and improving the robustness and accuracy of 3D object detection in complex environments. Experiments on adverse-weather versions of datasets such as KITTI-C and nuScenes-C validate the effectiveness and superiority of the proposed method. On the nuScenes-C dataset, it achieves 66.2% mAP and 66.6% mAP under fog and snow conditions, respectively.

1. Introduction

With the rapid development of autonomous driving and intelligent perception technologies, multi-sensor-fusion-based 3D object detection methods have become an important research direction in the field of environmental perception [1,2]. By fusing data from LiDAR, cameras, and other sensors, multimodal approaches can exploit the complementary strengths of heterogeneous sensors, thereby improving detection accuracy and robustness in complex scenarios. However, real-world driving often involves adverse weather (e.g., rain, fog, and snow) and extreme lighting conditions, which can degrade sensor data quality and compromise system reliability and safety [3,4]. Therefore, how to reliably leverage complementary multimodal information in adverse weather to improve detection robustness remains a critical open challenge.

To address the degradation in perception performance caused by adverse weather and environmental interference, existing studies have mainly explored two directions. The first line of work focuses on improving the data quality of each modality by suppressing noise at the input or feature level. This includes image restoration techniques (e.g., dehazing, deraining, and low-light enhancement) [5,6,7] as well as robust training strategies (e.g., data augmentation, domain adaptation, and adversarial training) to mitigate distribution shifts between training and testing conditions [8,9], thereby improving the stability of modality-specific features. The second line of work aims to improve multimodal fusion mechanisms to enhance a model’s ability to select and exploit cross-modal information. Typical approaches introduce attention mechanisms, gating units, or dynamic weight assignment strategies to enable finer-grained cross-modal interactions in BEV space or at the point-cloud feature level [10], thereby better balancing the contributions of different modalities during fusion, as shown in Figure 1. However, these fusion strategies remain challenged under complex noise conditions. In particular, in fusion paradigms dominated by a single modality (e.g., the camera or LiDAR modality), when that modality becomes globally or locally unreliable, the fusion network often fails to suppress error propagation effectively, leading to unstable fusion results and degraded performance on downstream tasks.

In addition, differences in feature density between LiDAR and cameras can lead to instability in multimodal fusion in practice. Specifically, LiDAR point clouds are inherently sparse in 3D space, and this sparsity becomes more pronounced as target distance increases. At longer ranges, fewer returns are observed and spatial coverage becomes more discontinuous, making it difficult for point- or voxel-based 3D features to be sufficiently informative. In contrast, cameras provide continuous, dense semantic features on the image plane, offering richer texture and category cues that are especially important for distinguishing small and distant objects. However, most mainstream fusion paradigms require aligning image features to 3D representations (e.g., projecting them onto point clouds, voxels, or BEV grids) to enable cross-modal interaction. During this process, alignment and aggregation are often constrained by the underlying 3D representation. When the point cloud is extremely sparse in distant regions, there are not enough 3D support points to enable reliable alignment and matching, so dense image semantics can only be sampled or aggregated at a limited number of locations. This forces image features to enter the 3D space in a sparse form, causing a large amount of potentially useful semantic information to be underutilized or discarded during fusion.

To address the above issues, we propose a semantic-enhanced bidirectional multimodal fusion framework (SeBFusion) to simultaneously mitigate weather-induced noise and information dilution during multimodal fusion. SeBFusion first leverages reliable image semantics to selectively enrich 3D features, improving the semantic richness and density of point-cloud representations. Next, during cross-modal interaction, it explicitly models the confidence of each modality and suppresses cross-modal noise propagation via adaptive reweighting of bidirectional information flow, thereby obtaining a more robust fused representation, as shown in Figure 1. Specifically, SeBFusion first introduces a virtual point generation and camera semantic injection module. By selectively mapping semantic features from images into 3D space, it generates semantically enhanced LiDAR features while preserving geometric consistency, compensating for the sparsity of raw point clouds in long-range regions. Then, at the cross-modal interaction stage, we propose a bidirectional cross-attention fusion module that performs information exchange in both directions and estimates the confidence of each modality in the current scene, enabling adaptive reweighting of the bidirectional information flow. This strategy effectively reduces the risk that low-quality modalities propagate noise into the other modality under corruptions such as rain, fog, and snow, while also preventing the strong modality from being overly disturbed by the weak modality during fusion, thereby improving detection accuracy and stability in complex environments. We validate the effectiveness of our method on adverse-weather benchmarks such as KITTI-C and nuScenes-C, and achieve consistent improvements under corruptions including fog and snow. The results show that SeBFusion can more reliably exploit multimodal complementary information, reducing false positives and missed detections caused by improper fusion, and demonstrating strong robustness to adverse weather. The main contributions of this paper are summarized as follows:

1.

We propose SeBFusion to address adverse-weather noise and multimodal information dilution via semantic enhancement and bidirectional fusion.

2.

We design a virtual point generation and camera semantic injection mechanism to selectively map image semantics into 3D space, enhancing point-cloud feature representations.

3.

We propose a confidence-aware bidirectional cross-attention fusion module that adaptively regulates bidirectional information flow to suppress noise propagation and improve fusion robustness.

The remainder of this paper is organized as follows: Section 2 reviews related work on multimodal 3D detection under adverse weather. Section 3 presents the proposed SeBFusion framework, including semantic enhancement via virtual point generation and confidence-aware bidirectional cross-attention fusion. Section 4 reports experimental settings and results on KITTI-C and nuScenes-C, with comparisons and ablations. Section 5 concludes the paper and discusses future directions.

2. Related Work

Adverse weather degrades multi-sensor observations in modality-specific and spatially non-uniform ways: camera images suffer from reduced contrast, streaks, and occlusions, while LiDAR point clouds exhibit scattering-induced outliers, attenuation, and echo dropouts that increase sparsity. To mitigate the resulting performance drop in 3D object detection, existing research largely follows two technical routes: improving per-modality data quality by suppressing noise at the input or feature level, and improving multimodal fusion mechanisms to better select and exploit cross-modal information.

2.1. Enhancing Data Quality Under Adverse Weather

A first line of work improves the quality and stability of each modality before (or alongside) detection. On the image side, adverse-weather restoration aims to recover informative textures and semantics via dedicated enhancement networks, including Transformer-based restoration and other high-capacity restoration models [11,12]. Such restoration-then-detection pipelines can strengthen cross-weather visual representations and benefit downstream perception, but they often introduce additional computational overhead and may propagate restoration artifacts to detection when the weather corruption is severe or spatially heterogeneous [13].

On the LiDAR side, many studies model weather-induced degradations and synthesize corruptions to augment training. Representative works simulate fog attenuation on real LiDAR scans and snowfall interference with explicit dropout or outlier modeling, enabling training and evaluation under controlled adverse conditions [14,15]. LISA further provides physics-based light scattering augmentation to generate adverse-weather point-cloud perturbations [16]. Beyond simulation, targeted denoising methods attempt to remove specific noise patterns using intensity and spatio-temporal cues [17]. Meanwhile, reconstruction-oriented approaches address the sparse or hollow far-field issue by filling missing points and densifying distant objects. For example, EquiDetect reconstructs point clouds via equirectangular projection using distance-constrained denoising and object-centric ray generation, targeting adverse-weather noise and long-range sparsity [7].

In addition, robust learning strategies tackle distribution shifts between training and testing environments. Unsupervised domain adaptation (UDA) via self-training and cross-modal adversarial alignment have been explored to reduce domain gaps for 3D detection [18,19]. Weather-based out-of-distribution (OOD) robustness has also been improved with adversarial training schemes tailored to LiDAR perception under unseen adverse weather [9]. Knowledge distillation across weather conditions provides another direction to transfer clean-weather knowledge to corrupted settings, while robust multimodal designs have been investigated in combination with large foundation models [20,21,22]. Robustness benchmarks for 3D detection under common corruptions further standardize evaluation and highlight failure modes under realistic degradations [23,24]. However, denoising and restoration-based methods are often brittle. Errors or artifacts introduced in sequential pre-processing can be amplified by downstream fusion and detection, especially under spatially non-uniform corruptions.

2.2. Improving Multimodal Fusion Mechanisms for Robust 3D Detection

A second line of work focuses on improving fusion to better exploit complementary sensor cues. Early LiDAR–camera fusion methods integrate image semantics into 3D representations, such as MVX-Net and EPNet, which enhance voxel or point features with image-derived cues [25,26]. Later efforts introduce stronger cross-modal interaction modules and Transformer-style fusion, including TransFusion, which refines LiDAR proposals with image features, and BEVFusion, which aligns multiple sensors into a unified BEV representation for end-to-end fusion and multi-task learning [27,28]. Other works emphasize interaction design and sparse multimodal representations, such as DeepInteraction and SparseFusion, to improve information exchange and efficiency [29,30]. Semantic injection strategies (e.g., FusionPainting) further explore injecting image semantics into the 3D detection pipeline via adaptive attention [31]. Beyond camera–LiDAR, radar-involved fusion improves robustness in adverse weather where radar is less affected by fog and rain, albeit with lower spatial resolution and more clutter [3,32].

More recently, sensor-adaptive multimodal fusion under adverse weather has been studied in several works. For example, Bijelic et al. [33] proposed an entropy-driven adaptive deep fusion method for handling unseen adverse weather conditions under asymmetric sensor distortions, thereby improving generalization to previously unseen weather conditions without requiring large amounts of annotated adverse-weather data. In addition, SAMFusion [34] uses depth information to guide cross-modal interaction, facilitating feature transformation and fusion. In contrast, our method exploits depth priors for multi-depth candidate generation and virtual-point semantic lifting. Under different depth hypotheses, image semantics are lifted into multiple plausible 3D locations and then aggregated into a semantic BEV representation. Thus, our method focuses on semantic lifting and uncertainty modeling before fusion.

While improved fusion architectures generally boost accuracy on clean data and provide gains under moderate corruption, two core issues remain. When a modality becomes globally or locally unreliable, fusion modules may fail to suppress erroneous information flow, allowing corrupted cues to contaminate the other modality and destabilize the fused representation. Moreover, the density mismatch between dense image semantics and sparse far-field LiDAR support can force image features to enter 3D in an overly sparse manner, leading to underutilization and dilution, while aggressive semantic injection can cause semantic contamination under misalignment.

These limitations provide a clear motivation for explicit, controllable fusion decisions: selectively enriching 3D features with camera semantics only when alignment is trustworthy, and regulating bidirectional cross-modal interaction based on estimated per-modality confidence to prevent noise propagation under adverse weather.

3. Method

3.1. Overall Framework

Adverse weather induces multimodal degradation, as LiDAR echoes become sparser and noisier due to rain droplets, snow particle scattering, and far-field attenuation, while camera images experience reduced contrast and texture loss under fog, rain streaks, and occlusion. More critically, multimodal degradation is spatially non-uniform and modality-dependent: within a single frame, LiDAR and camera reliability can vary by region. As a result, naïve fusion may propagate corrupted cues across modalities and dilute high-density geometric evidence with low-density or noisy features, causing misalignment, false positives, and missed detections. To address these challenges, we propose SeBFusion, a Semantic-Enhanced Bidirectional Multimodal Fusion framework for robust 3D object detection under adverse weather. Figure 2 illustrates the overall architecture of SeBFusion.

To mitigate weather-induced noise and alleviate information dilution in multimodal fusion, SeBFusion introduces a two-stage design. First, it performs semantic enhancement by generating virtual points and selectively injecting camera semantics into BEV space to obtain semantically enhanced LiDAR features, compensating for point-cloud sparsity in long-range regions. Second, it conducts confidence-aware bidirectional fusion: the model explicitly estimates per-modality confidence and adaptively reweights the bidirectional information flow, reducing cross-modal noise propagation and alleviating information dilution under corruptions. SeBFusion makes multimodal fusion more robust under adverse weather by integrating semantic enhancement and confidence-aware bidirectional fusion, which jointly suppress noise propagation and mitigate information dilution during cross-modal interaction.

3.2. Semantic LiDAR Feature Generation Module

The semantic LiDAR feature generation module aims to preserve the geometric stability of LiDAR BEV features while selectively injecting discriminative camera semantics into BEV space, so as to compensate for long-range sparsity and avoid diluting reliable LiDAR geometry with unreliable image cues under adverse weather. The module comprises two components: virtual-point generation with semantic BEV aggregation, and alignment assessment with dynamic injection. The core idea is to leverage camera-derived semantic information to reinforce LiDAR features while restricting the injection process to a guided, alignment-aware regime rather than unconditional concatenation or addition. This controlled, region-aware fusion reduces the propagation of cross-modal misalignments under harsh weather.

3.2.1. Virtual Point Generation

In multimodal 3D detection, translating camera semantic information into 3D-space representations is a critical step. Directly lifting image features into 3D with a single depth hypothesis is fragile under adverse weather: LiDAR becomes sparse and noisy, image semantics are dense but depth-ambiguous, and sparse 3D supports in the far field can cause dense semantics to enter 3D in an overly sparse form, leading to underutilization and information dilution. To address this, we introduce a virtual-point mechanism with multi-depth candidates to improve geometric tolerance and semantic coverage. For each pixel, the corresponding semantic is elevated into several plausible 3D points across different depths. This yields multiple camera-derived 3D hypotheses that inherit stronger geometric tolerance, enabling more reliable and comprehensive semantic injection into the BEV space even in harsh weather.

First, we run a lightweight 2D detector on each camera view to obtain high-confidence foreground proposals [35], which are then used to select seed pixels for virtual point generation:

$$B^v={\left\{\left({\mathrm{box}}_n^v,{\mathrm{score}}_n^v,{\mathrm{class}}_n^v\right)\right\}}_{n=1}^{N_v},$$

(1)

where $B^v$ denotes the set of 2D detection results on the v-th camera view, with n indexing the detections and $N_v$ the number of results kept after non-maximum suppression. The term ${\mathrm{box}}_n^v$ represents the 2D bounding box parameters of the n-th detection in view v, ${\mathrm{score}}_n^v$ the confidence score of that 2D bounding box, and ${\mathrm{class}}_n^v$ the class information of the bounding box. To reduce background noise and occlusion artefacts under adverse weather, we restrict seed sampling to high-confidence 2D proposals instead of performing depth search over the entire image.

Then, we construct the reference depth and the multi-depth candidates, and project the LiDAR points $x_i\in \mathcal{P}$ onto the camera plane to obtain the reference set:

$$(u_i,v_i,d_i)={\pi }_{\nu }\left(x_i\right),R^v=\left\{(u_i,v_i,d_i)\right\},$$

(2)

where $x_i$ is the coordinate vector of the i-th LiDAR point in the 3D coordinate system, ${\pi }_v(·)$ is the projection function from the 3D point to the v-th camera view, $(u_i,v_i)$ are the pixel coordinates of the projection of $x_i$ onto the image from the v-th camera view (horizontal axis u, vertical axis v), and $d_i$ is the depth value of $x_i$ in that camera view. For each seed $(u_m,v_m)$, perform a KNN search in the pixel space over $R^{\prime }$ to obtain K neighboring reference depths as candidates:

$$D\left((u_m,v_m)\right)={\left\{d_k^K\right\}}_{k=1}^K,$$

(3)

where $d_m^k$ denotes the k-th candidate depth value, which comes from the reference set $R^{\nu }$ obtained by projecting LiDAR points onto the v-th camera and performing a nearest-neighbor search in the pixel space for $(u_m,v_m)$. For each seed, the number of depth candidates kept is K. The index m denotes the seed index. Multiple depth candidates are used to explicitly model depth ambiguity. When depth is unreliable due to rain and fog occlusion or sparse point clouds, multiple candidates can improve the probability that the semantic is assigned to the correct BEV cell, thus reducing the negative impact of single-depth incorrect projections.

Subsequently, for each $\left((u_m,v_m),d_m^k\right)$, project through the camera model in reverse and transform to the ego coordinate system, as follows:

$$x_m^k=T_{ego\leftarrow \mathrm{cam}}^v\left(d_m^k{\left(K^v\right)}^{-1}{\overline{U}}_m\right),{\overline{U}}_m={[u_m,v_m,1]}^{\top },$$

(4)

where $x_m^k$ denotes the 3D coordinates of the virtual point generated for the m-th seed under the k-th depth candidate. $T_{\mathrm{ego}\leftarrow \mathrm{cam}}^v$ denotes the rigid-body transformation from the v-th camera coordinate system to the ego coordinate system. ${\overline{U}}_m$ denotes the homogeneous pixel coordinates of the seed pixel, and $u_m,v_m$ denote the pixel coordinates of the seed in the image. From the camera feature map $U_m$, sampling yields $f_{\mathrm{img}}\left(U_m\right)$, and it is fused with the depth encoding $PE\left(d\right)$ to generate depth-aware semantics:

$$f_{vp}^k=\mathrm{MLP}\left(\left[f_{\mathrm{img}}\left(U_m\right)\Vert \mathrm{PE}\left(d_m^k\right)\right]\right),$$

(5)

where $f_{vp}^k$ denotes the semantic feature vector of the virtual point generated for the m-th seed under the k-th depth candidate. The operation $\mathrm{MLP}(·)$ is a multilayer perceptron used to map the concatenated vector to the target feature dimension. The symbol ‖ denotes feature concatenation, and $\mathrm{PE}(·)$ is the positional encoding function for the depth values.

Finally, scatter the virtual points onto the BEV grid, and for the input features, their BEV cell indices are as follows:

$$(i,j)=\left(\left[{\displaystyle \frac{x-x_{min}}{r_x}},{\displaystyle \frac{y-y_{min}}{r_y}}\right]\right),$$

(6)

where x and y denote the coordinates of a virtual point on the BEV plane; $x_{min}$ and $y_{min}$ denote the minimum boundaries of the BEV detection region along the x and y directions (the ROI origin), used to translate continuous coordinates to a grid coordinate system starting from zero. $r_x$ and $r_y$ denote the grid cell sizes of the BEV grid in the x and y directions, and $(i,j)$ denote the corresponding BEV grid indices. For the virtual point features $\left\{f_{vp}\right\}$ within the same cell, aggregation yields the camera fusion feature $F_C^{\mathrm{fuse}}$.

3.2.2. Camera Semantic Injection

Under adverse weather, cross-modal misalignment is more likely. We therefore adopt alignment-guided injection: a BEV-level semantic-consistency score is computed to gate camera-to-LiDAR semantic injection, preventing corrupted or misaligned image semantics from contaminating reliable LiDAR geometry. First, as shown below, align the channel dimension of the LiDAR features and the camera fusion features $F^{\mathrm{fuse}}$:

$${\tilde{F}}_L=W_L\left(F_L\right),{\tilde{F}}_C=W_C\left(F_C\right),{\tilde{F}}_L,{\tilde{F}}_C\in {\mathbb{R}}^{H\times W\times D},$$

(7)

Then, as shown in the formula, construct alignment scores using cosine similarity and score them using Sigmoid mapping:

$$S(x,y)=\sigma \left(\alpha ·{\displaystyle \frac{\left[{\tilde{F}}_L(x,y),{\tilde{F}}_C(x,y)\right]}{\parallel {\tilde{F}}_L(x,y)\parallel \parallel {\tilde{F}}_C(x,y)\parallel }}+\beta \right),S\in {[0,1]}^{H\times W\times 1},$$

(8)

where $\alpha$ and $\beta$ are learnable parameters, and $\sigma (·)$ denotes the Sigmoid function. Next, as indicated by the formula, perform cell-level gating on the camera BEV features to address the reliability-selection problem in the spatial dimension, significantly reducing semantic contamination in misaligned regions.

$${\widehat{F}}_C^{\mathrm{fuse}}=S·F_C^{\mathrm{fuse}},$$

(9)

We concatenate the LiDAR feature $F_L$ with the camera fused feature ${\widehat{F}}_C^{\mathrm{fuse}}$ and perform local convolution fusion; the detailed process is shown below:

$$F^{\mathrm{static}}=\mathrm{Conv}\left(\left[F_L\Vert {\widehat{F}}_C^{\mathrm{fuse}}\right]\right)\in {\mathbb{R}}^{H\times W\times C},$$

(10)

Finally, as shown in the formula, we use a channel attention mechanism to model the contributions of different channels, and inject the camera semantic information into the LiDAR features according to their contributions, yielding the LiDAR semantic feature $F_L^{\mathrm{sem}}$ with LiDAR geometric stability and camera semantic discriminability:

$$F_L^{\mathrm{sem}}=\sigma \left(\mathrm{Conv}\left(\mathrm{GAP}\left(F^{\mathrm{static}}\right)\right)\right)·F^{\mathrm{static}},$$

(11)

where $\mathrm{GAP}(·)$ denotes the channel-wise global average pooling operation, and $\sigma (·)$ denotes the Sigmoid function. In the process of injecting camera semantic information into LiDAR features, the semantic-consistency score can control spatial reliability, and channel-wise modeling reveals which fusion channels are more beneficial for robust detection under adverse weather, enabling selective semantic enrichment at both spatial and channel levels, which improves robustness by reducing semantic contamination and mitigating information dilution during fusion.

3.3. Bidirectional Cross-Attention Fusion Module

Semantic LiDAR feature $F_L^{\mathrm{sem}}$ possesses geometric stability and semantic discriminability, while the camera BEV feature $F_c^{\mathrm{bev}}$ obtained by aggregating from the camera viewpoint has cross-view context and dense semantics. Although both are aligned in BEV space to the same coordinate system, two typical difficulties still arise under adverse weather. First, modality degradation can be modality-dependent and spatially non-uniform, causing a modality to be unreliable in parts of a frame; second, cross-attention during information interaction may explicitly propagate the noise of the degraded modality to the other modality, leading to false positives and overfitting. To address this, we introduce a confidence-aware bidirectional cross-attention module that first estimates modality confidence and then performs confidence-weighted bidirectional interaction, adaptively allocating cross-modal contributions while suppressing unreliable information flow.

We define the confidence used in the proposed bidirectional fusion module. For each modality $m\in \{\mathrm{LiDAR},\mathrm{Camera}\}$, confidence is a sample-level modality reliability scalar $C_m\in [0,1]$, predicted by an auxiliary head. Unlike relying solely on fusion weights learned implicitly from the detection loss, we explicitly predict the confidence coefficients for the two modalities, $C_{\mathrm{LiDAR}}$ and $C_{\mathrm{Camera}}$, and use them to control the strength of cross-modal information interaction. First, we fuse the semantic LiDAR feature $F_L^{\mathrm{sem}}$ and the camera BEV feature $F_C^{\mathrm{bev}}$ globally and map them into a shared embedding space, as shown by the equation:

$$Z_L={⌀}_L\left(\mathrm{GAP}\left(F_L^{\mathrm{sem}}\right)\right),Z_C={⌀}_C\left(\mathrm{GAP}\left(F_C^{\mathrm{bev}}\right)\right),$$

(12)

where ${\varphi }_L^0$ and ${\varphi }_C^0$ as lightweight MLP networks, which are used to encourage semantic alignment between LiDAR and Camera features at the same time, and to push apart non-matching samples. We adopt the formula shown below to constrain embedding consistency:

$$L_{\mathrm{con}}=-log\left({\displaystyle \frac{exp\left(\mathrm{sim}(z_L,z_C)/\tau \right)}{{\sum }_{i=1}^{K}exp\left(\mathrm{sim}(z_L,z_C^i)/\tau \right)}}\right),$$

(13)

where $\mathrm{sim}(·,·)$ is the cosine similarity, $\tau$ as the temperature parameter, and K as the number of candidates in a batch. When a modality degrades severely, its embedding becomes harder to align with the other modality, thus providing a more discriminative feature basis for subsequent confidence estimation. Next, we predict modality-level confidences $C_{\mathrm{LiDAR}}$ and $C_{\mathrm{Camera}}$ on the embeddings; the concrete procedure is shown below:

$$C_{\mathrm{LiDAR}}=\sigma \left({\omega }_L{Z}_L+b_L\right),C_{\mathrm{Camera}}=\sigma \left({\omega }_C{Z}_C+b_C\right),$$

(14)

where $\sigma (·)$ is the Sigmoid function, and ${\omega }_L$ and ${\omega }_C$ as the linear-layer weight vectors. During training, we apply stochastic modality degradation augmentation to generate supervisory signals, with labels $y_m\in \{0,1\}$, where 1 indicates clean and 0 degraded, and we compute the confidence loss via binary cross-entropy:

$$L_{\mathrm{conf}}=BCE\left(C_{\mathrm{LiDAR}},y_{\mathrm{LiDAR}}\right)+BCE\left(C_{\mathrm{Camera}},y_{\mathrm{Camera}}\right),$$

(15)

Ultimately, the reliability loss is:

$$L_{\mathrm{rel}}={\lambda }_{\mathrm{con}}{L}_{\mathrm{con}}+{\lambda }_{\mathrm{conf}}{L}_{\mathrm{conf}},$$

(16)

where ${\lambda }_{\mathrm{con}}$ and ${\lambda }_{\mathrm{conf}}$ is the weighting coefficients used to balance the influence of $L_{\mathrm{con}}$ and $L_{\mathrm{conf}}$ on training. Then, as shown in Figure 3, we construct bidirectional interactions at the BEV-token level; the linear projections of LiDAR and camera features are shown below:

$$Q_{\mathrm{LiDAR}}=X_{\mathrm{LiDAR}}{W}_{\mathrm{LiDAR}}^Q,K_{\mathrm{LiDAR}}=X_{\mathrm{LiDAR}}{W}_{\mathrm{LiDAR}}^K,V_{\mathrm{LiDAR}}=X_{\mathrm{LiDAR}}{W}_{\mathrm{LiDAR}}^V,$$

(17)

$$Q_{\mathrm{Camera}}=X_{\mathrm{Camera}}{W}_{\mathrm{Camera}}^Q,K_{\mathrm{Camera}}=X_{\mathrm{Camera}}{W}_{\mathrm{Camera}}^K,V_{\mathrm{Camera}}=X_{\mathrm{Camera}}{W}_{\mathrm{Camera}}^V,$$

(18)

where $X_{\mathrm{LiDAR}}$ and $X_{\mathrm{Camera}}$ are the flattened BEV token sequences derived from the semantically enhanced LiDAR feature map $F_L^{\mathrm{sem}}$ and the camera BEV feature map $F_C^{\mathrm{bev}}$, respectively.

For the LiDAR → Camera branch, using the camera token as the query, we aggregate the structural information from LiDAR tokens and multiply by $C_{\mathrm{LiDAR}}$:

$$X_{L\to C}=C_{\mathrm{LiDAR}}·Softmax\left({\displaystyle \frac{Q_{Camera}{\left(K_{LiDAR}\right)}^{\top }}{\sqrt{d_k}}}\right)V_{LiDAR},$$

(19)

When LiDAR degrades, $C_{\mathrm{LiDAR}}$ decreases, reducing the injection strength in the LiDAR-to-Camera branch and avoiding unreliable cross-branch contextual modeling in the Camera branch. For the Camera-to-LiDAR branch, using the LiDAR token as the query, semantic information is aggregated from the Camera BEV tokens and multiplied by $C_{\mathrm{Camera}}$:

$$X_{C\to L}=C_{\mathrm{Camera}}·Softmax\left({\displaystyle \frac{Q_{LiDAR}{\left(K_{Camera}\right)}^{\top }}{\sqrt{d_k}}}\right)V_{Camera},$$

(20)

When rain and fog degrade imaging contrast and texture, $C_{\mathrm{Camera}}$ suppresses semantic propagation in this direction, reducing false detections caused by camera noise. Finally, as shown in below, the outputs of the two branches are fused to obtain the final fused feature:

$$F_{\mathrm{fuse}}=X_{L\to C}+X_{C\to L},$$

(21)

After the semantic enhancement and bidirectional cross-modal fusion, we obtain the final fused BEV feature $F_{\mathrm{fuse}}$. We then feed $F_{\mathrm{fuse}}$ into the 3D detection head, which predicts the classification scores and regresses the 3D bounding box parameters based on this fused representation, thereby enabling 3D object detection under adverse weather conditions. Since the fused feature integrates LiDAR geometry with camera semantics while suppressing low-confidence modality noise, the detection head can produce more stable and reliable detection results.

4. Experiments

4.1. Dataset

We evaluate SeBFusion on the adverse-weather variants of the KITTI, Seeing Through Fog and nuScenes datasets. KITTI is a large-scale autonomous-driving dataset collected mainly under daytime and favorable weather conditions, with 7481 training samples and 7518 test samples. nuScenes comprises 1000 sequences, each about 20 seconds long, covering a range of urban and suburban driving scenes. Following prior work, we apply established weather-simulation techniques to generate fog, rain, snow, and strong illumination conditions [14,15,16,36], forming KITTI-C and nuScenes-C to validate the model’s generalizability. Seeing Through Fog is a real-world multimodal adverse-weather benchmark collected over more than 10,000 km of driving in northern Europe [33]. The dataset provides synchronized multimodal measurements, including RGB cameras, LiDAR, radar, and gated sensors, together with fine-grained weather annotations under clear and adverse conditions. Compared with synthetic corruption benchmarks, SeeingThroughFog contains real sensor degradations caused by fog and precipitation, making it more suitable for evaluating the robustness of multimodal 3D detection methods in practical adverse-weather scenarios. Following a distance-aware evaluation protocol, we report performance in three distance ranges, i.e., 0–30 m, 30–50 m, and 50–80 m, to further analyze the robustness of different fusion methods from near to long range.

4.2. Implementation Details

We implement our model within the MMDetection3D framework and train it on four NVIDIA GeForce RTX 4090 GPUs with a per-GPU batch size of 2 (effective batch size = 8). Training follows a two-stage protocol. We first pretrain the image branch to provide stable semantic cues and the LiDAR branch to learn geometry-aware BEV features. We then train SeBFusion end-to-end with both branches initialized, enabling selective semantic injection and confidence-aware bidirectional fusion under corruptions. The fusion training uses AdamW with an initial learning rate of 1$\times$10⁻⁴ and runs for 12 epochs. $\alpha$ and $\beta$ are initialized as $\alpha$ = 2.0 and $\beta$ = −0.5, respectively. To supervise confidence prediction, we apply stochastic modality degradation during training by randomly corrupting either LiDAR or images, and use the applied corruption state as the binary label for $L_{conf}$.

4.3. Compare with Other Methods

We compare SeBFusion with representative LiDAR-only, camera-only, and LiDAR + camera fusion 3D detectors on KITTI-C and nuScenes-C under Clean, Fog, Rain, Snow, and Strong Light corruptions, and further evaluate camera–LiDAR fusion methods on SeeingThroughFog for pedestrian detection under real adverse-weather conditions.

Table 1. Performance results of different methods on the KITTI-C dataset.

Methods	Modality	Clean	Fog	Rain	Snow	Light
3DSSD [37]	LiDAR	80.18	46.26	28.31	28.33	25.14
PV-RCNN [38]	LiDAR	85.17	79.22	52.21	53.12	80.55
SECOND [39]	LiDAR	81.15	74.63	52.12	55.81	77.21
PointRCNN [40]	LiDAR	82.23	77.15	51.02	52.64	62.08
PointPillars [41]	LiDAR	78.41	64.28	36.18	36.47	62.28
Part-A2 [42]	LiDAR	82.45	71.61	41.63	42.70	76.45
SMOKE [43]	Camera	7.09	5.63	3.94	2.47	6.00
PGD [44]	Camera	8.10	0.87	3.06	0.63	7.07
ImVoxelNet [45]	Camera	11.49	1.34	1.24	0.22	10.08
EPNets [26]	LiDAR + Camera	85.13	44.16	40.12	34.71	69.12
Focals Conv [46]	LiDAR + Camera	84.16	44.15	40.12	35.23	80.75
AWARDistill [21]	LiDAR + Camera	88.62	82.74	70.92	65.74	80.19
Ours	LiDAR + Camera	89.05	83.60	74.20	69.10	82.10

reports the performance of different detectors on KITTI-C under clean and four adverse conditions. A clear trend is that camera-only methods are the most fragile, even on clean data, SMOKE, PGD, and ImVoxelNet remain at 7.09, 8.10, and 11.49, and their performance collapses once visibility and texture cues are corrupted. For example, ImVoxelNet drops to 1.34 in fog and 0.22 in snow. This is consistent with the fact that monocular 3D detection relies heavily on appearance cues and implicit depth reasoning, both of which become unreliable when contrast is reduced or when rain or snow introduces structured artifacts and occlusions.

LiDAR-only methods are notably more stable than camera-only baselines, but they still suffer substantial degradation under severe corruptions. For example, although PV-RCNN reaches 85.17 on clean data, it falls to 52.21 in rain and 53.12 in snow, reflecting that scattering and weather-induced outliers can significantly reduce point cloud quality. Simply adopting multimodal fusion does not automatically guarantee robustness either: EPNets and Focals Conv perform well on clean KITTI-C, yet drop sharply in fog, suggesting that when the degradation is spatially non-uniform, unconditional cross-modal interaction can amplify misalignment, propagate corrupted cues across modalities, and dilute reliable LiDAR geometry with low-quality image features, especially under spatially non-uniform degradations.

In contrast, our method achieves the best results across all KITTI-C conditions, reaching 89.05, 83.60, 74.20, 69.10 and 82.10 on Clean, Fog, Rain, Snow, Light. Compared with AWARDistill, which is the strongest baseline among fusion methods in Table, our gains are consistent and become more pronounced in harder weather. The larger improvements in rain and snow indicate that explicitly controlling what to fuse and how strongly to fuse is particularly beneficial when one modality becomes locally unreliable and when cross-modal alignment errors are more likely to appear. This is also reflected in corruption sensitivity: relative to clean performance, our drops are smaller than AWARDistill in every corruption type, showing that the benefit is not only higher peak accuracy but also improved stability.

We observe the same pattern on nuScenes-C in

Table 2. Performance results of different methods on the nuScenes-C dataset.

Methods	Modality	Clean	Fog	Rain	Snow	Light
PointPillars [41]	LiDAR	27.69	24.49	27.71	27.57	23.71
SSN [47]	LiDAR	46.65	41.64	46.50	46.38	40.28
CenterPoint [48]	LiDAR	59.28	43.78	56.08	55.90	54.20
FCOS3D [49]	Camera	23.86	13.53	13.00	2.01	17.20
PGD [44]	Camera	23.19	12.83	13.51	2.30	22.77
DETR3D [50]	Camera	34.71	27.89	20.39	5.08	34.66
BEVFormer [51]	Camera	41.65	32.76	24.97	5.73	41.68
FUTR3D [52]	LiDAR + Camera	64.17	53.19	58.40	52.73	57.70
TransFusion [27]	LiDAR + Camera	66.38	53.67	65.35	63.30	55.14
BEVFusion [28]	LiDAR + Camera	68.45	54.10	66.13	62.84	64.42
RoboFusion-B [22]	LiDAR + Camera	69.40	65.54	67.01	66.07	66.71
AWARDistill [21]	LiDAR + Camera	68.11	60.11	66.93	66.03	62.92
Ours	LiDAR + Camera	69.90	66.20	67.80	66.60	67.30

. Camera-only methods again degrade severely under adverse weather, especially snow, while LiDAR-only methods are more resilient but still affected. For example, CenterPoint drops from 59.28 clean to 43.78 in fog. Fusion-based approaches provide the strongest overall performance, and our method ranks first under all five conditions, achieving 69.90, 66.20, 67.80, 66.60, 67.30 on Clean, Fog, Rain, Snow, and Strong Light. Compared with RoboFusion-B, our improvements are consistent across conditions, and we also obtain slightly smaller drops from clean to corrupted settings, indicating better robustness when the corruption type changes. Figure 4 shows the visualization results of our method compared to other methods. Overall, the results on both KITTI-C and nuScenes-C support our motivation that making fusion decisions explicit—by gating semantic injection with alignment cues and modulating bidirectional interaction with reliability—helps prevent degraded-modality noise from propagating and enables more stable exploitation of multimodal complementarity under adverse weather.

To further verify the effectiveness of SeBFusion on real adverse-weather data, we additionally evaluate it on the SeeingThroughFog dataset for pedestrian detection. The results are reported in

Table 3. Pedestrian detection AP on the Seeing Through Fog dataset under snow and fog at different distance ranges.

Method	Modality	Snow			Fog
		0–30 m	30–50 m	50–80 m	0–30 m	30–50 m	50–80 m
MVXNet [25]	Camera + LiDAR	76.23	59.73	25.83	73.89	50.98	16.73
BEVFusion [28]	Camera + LiDAR	71.12	62.61	10.01	76.24	58.04	8.61
SparseFusion [30]	Camera + LiDAR	73.33	66.84	19.87	79.25	58.39	17.05
Ours	Camera + LiDAR	80.64	74.63	38.59	81.79	63.37	26.51

. Our method achieves the best performance across all weather and distance settings. Under snow, SeBFusion obtains 80.64, 74.63, and 38.59 AP in the 0–30 m, 30–50 m, and 50–80 m ranges, respectively, surpassing the strongest baseline by 4.41, 7.79, and 12.76 points. Under fog, it achieves 81.79, 63.37, and 26.51 AP, improving over the best competing method by 2.54, 4.98, and 9.46 points, respectively. Notably, the gains become more pronounced at longer distances, where adverse weather further aggravates point-cloud sparsity and modality imbalance. These results indicate that the proposed semantic enhancement and confidence-aware bidirectional fusion can more effectively exploit complementary camera–LiDAR information and maintain robust pedestrian detection performance in real adverse-weather environments.

Although SeBFusion achieves strong overall robustness under adverse weather, it still exhibits failure cases when modality degradation becomes extremely severe. Figure 5 presents several representative failure cases of SeBFusion under severe adverse-weather corruption. When the degradation simultaneously affects both camera and LiDAR observations, the model may still produce missed detections, false positives, or bounding-box misalignment. In particular, dense fog weakens image contrast and shortens the effective sensing range of LiDAR, making distant pedestrians harder to detect reliably. Rain and snow can introduce strong scattering noise and irregular sparse patterns, which further disturb multimodal correspondence. Similarly, strong illumination may locally impair image semantics and reduce the quality of cross-modal fusion. These results suggest that, while SeBFusion improves robustness by suppressing unreliable information flow, it is still fundamentally constrained by the quality of the available sensor observations in extremely degraded scenarios.

4.4. Ablation Study

We conduct a module-wise ablation study on nuScenes-C using mAP as the evaluation metric, and report results under five conditions: Clean, Fog, Rain, Snow, and Light. Our baseline adopts a simple fusion strategy. We then progressively add MDC, CSI, and BCAF to quantify the contribution of each component. The results are summarized in

Table 4. Ablation results on nuScenes-C. MDC denotes multi-depth candidate virtual-point generation; CSI denotes camera semantic injection; BCAF denotes bidirectional cross-attention fusion.

Setting	MDC	CSI	BCAF	Clean	Fog	Rain	Snow	Light
Baseline				62.15	53.08	58.11	51.96	57.52
+MDC	✓			64.95	58.08	61.61	57.46	61.12
+CSI	✓	✓		67.25	62.08	64.61	61.96	64.12
+BCAF (Full)	✓	✓	✓	69.90	66.20	67.80	66.60	67.30

.

Adding MDC on top of the baseline yields consistent gains across all conditions, $+5.00$ mAP on Fog and $+5.50$ mAP on Snow, indicating that MDC provides richer candidate information and mitigates fusion uncertainty under corruptions, thereby improving robustness. Further introducing CSI upon MDC brings additional improvements. This suggests that alignment-guided injection reduces cross-modal mismatch in corrupted scenarios and stabilizes the fusion process, with more pronounced benefits under severe conditions such as Fog and Snow.

Finally, incorporating BCAF into MDC + CSI gives the full model, achieving 69.90, 66.20, 67.80, 66.60 and 67.30 mAP across the five conditions. Relative to the previous setting (MDC + CSI), the improvements are particularly notable on Fog and Snow, which implies that BCAF suppresses negative transfer from low-confidence features while enhancing complementary information exchange via reliability-aware interactions, further boosting robustness under corruptions.

5. Conclusions

Adverse weather simultaneously causes point-cloud sparsity and noise as well as degradation in image contrast and texture, and can further lead to cross-modal misalignment and noise propagation. To address these issues, this paper proposes a robust LiDAR–camera multimodal 3D object detection framework for adverse weather. In the semantic enhancement stage, the proposed method robustly lifts camera semantics into the BEV space via virtual point generation and semantic BEV aggregation, and explicitly characterizes depth uncertainty using multi-depth candidates (MDC), thereby improving geometric tolerance and semantic coverage. Meanwhile, it introduces camera semantic injection (CSI), which gates and selectively injects camera semantics at the BEV grid level according to semantic consistency. Furthermore, in the cross-modal interaction stage, we propose bidirectional cross-attention fusion (BCAF), which explicitly estimates modality-level confidence to modulate the strength of bidirectional information flow, reducing the risk of noise from a degraded modality propagating to the other. Although our method performs well in most cases, its performance may still degrade when both modalities suffer severe degradation simultaneously. Future work may further target more complex real-world degradation mechanisms and distribution shifts, exploring stable fusion with temporal information, finer-grained reliability modeling, and adaptive mechanisms for engineering factors such as extrinsic calibration drift and sensor asynchrony, to enhance reliability in practical deployment.