RMP: Robust Multi-Modal Perception Under Missing Condition

Abstract

Multi-modal perception is a core technology for edge devices to achieve safe and reliable environmental understanding in autonomous driving scenarios. In recent years, most approaches have focused on integrating complementary signals from diverse sensors, including cameras and LiDAR, to improve scene understanding in complex traffic environments, thereby attracting significant attention. However, in real-world applications, sensor failures frequently occur; for instance, cameras may malfunction in scenarios with poor illumination, which severely reduces the accuracy of perception models. To overcome this issue, we propose a robust multi-modal perception pipeline designed to improve model performance under missing modality conditions. Specifically, we design a missing feature reconstruction mechanism to reconstruct absent features by leveraging intra-modal common clues. Furthermore, we introduce a multi-modal adaptive fusion strategy to facilitate adaptive multi-modal integration through inter-modal feature interactions. Extensive experiments on the nuScenes benchmark demonstrate that our method achieves SOTA-level performance under missing-modality conditions.

1. Introduction

3D perception is a core pillar of modern autonomous driving systems, allowing edge devices to achieve reliable environmental understanding and decision-making in complex traffic scenarios [1,2]. Among various three-dimensional perception tasks, semantic segmentation plays a crucial role. By assigning semantic labels to each pixel or point, it enables fine-grained scene comprehension, which further supports high-level reasoning and navigation in autonomous systems [3,4]. Traditional perception systems typically rely on single-modality sensors, such as LiDAR-based [5,6] or camera-based [7,8] approaches, to construct spatial awareness of the environment. However, single-modal methods face constraints that are rooted in the physical characteristics of their corresponding sensors. LiDAR provides accurate geometric and structural information but lacks rich semantic and texture cues, while cameras capture detailed visual appearance yet remain vulnerable to illumination changes, occlusion, and depth ambiguity [9]. These inherent limitations have motivated increasing research interest in multi-modal perception, which aims to fuse information across different sensing sources to achieve a better understanding of the environment [10].

Multi-modal perception involves using information from various heterogeneous sensors, such as cameras and LiDAR, to construct a more reliable understanding of the environment [11]. Unlike single-modality perception, multi-modal methods can exploit the complementary strengths of different sensors, enabling richer semantic understanding and more accurate geometric reasoning [10]. Furthermore, multi-modal methods can achieve more stable perception capabilities under challenging conditions such as low light, occlusions, and adverse weather, thereby improving model robustness. Beyond conventional visual and LiDAR sensors, research in wireless imaging suggests that non-visual modalities can offer additional environmental information, particularly under adverse conditions [12]. Although such modalities are not considered in this work, they highlight a promising avenue for improving the robustness of multi-modal perception systems. Therefore, multi-modal perception has become a core foundation of modern autonomous driving systems.

Existing perception methods that utilize multiple sensing sources are typically grouped into early fusion [13], middle fusion [11], and late fusion [14]. Early fusion integrates information directly at its initial data stage, middle fusion learns joint representations from processed features, while late fusion combines task-specific outputs at the decision level. Although these fusion strategies have significantly improved perception accuracy and robustness, they still encounter substantial challenges in real-world autonomous driving scenarios. In L3/L4 autonomous driving systems, multi-modal perception models are typically deployed on edge computing platforms that must satisfy strict real-time and safety constraints. However, factors such as sensor malfunction, occlusion, or adverse weather conditions may cause partial camera inputs to become unavailable, thereby disrupting cross-modal feature alignment and degrading model robustness. Consequently, missing camera modalities have emerged as one of the critical issues that must be addressed in the development of reliable multi-modal perception research.

To address the aforementioned challenges, several recent studies have explored robust fusion strategies for multi-modal perception [10,15]. For instance, MSeg3D [15] proposes a joint framework that performs both intra-modal feature extraction and inter-modal feature fusion, effectively mitigating the multi-sensor alignment issues outside the field of view (FOV) and maintaining reasonable performance under missing-modality conditions. Similarly, MetaBEV [16] enhances fusion robustness by introducing modality-specific layers within cross-modal attention blocks, which alleviates the severe performance degradation caused by sensor corruption or partial failure. Although these approaches are capable of handling incomplete modalities to some extent, they fundamentally operate on an incomplete feature space. Since no mechanism explicitly infers the representation of the absent view, the fused multi-modal feature becomes biased toward the remaining modalities, causing unstable cross-modal interactions. As a result, robust fusion alone cannot recover the semantic information that would have been contributed by the missing viewpoint. Consequently, an increase in the number of absent camera inputs leads to a notable drop in the model’s accuracy. As illustrated in Figure 1, when all camera inputs are missing, MSeg3D achieves an mIoU of only 74.5%, representing a 4.7% drop compared to the complete-modality scenario. In contrast, our proposed method attains an 80.8% mIoU, with a notably smaller performance decline of only 3.2%, demonstrating stronger robustness under severe missing conditions.

To this end, we present the Robust Multi-modal Perception (RMP) method, which explicitly models missing features using intra-modal information to generate effective replacement values, thereby improving the robustness of the perception model under missing-view conditions. Specifically, the RMP comprises two principal modules: the Missing Feature Reconstruction Module (MFR) and the Multi-modal Adaptive Fusion Module (MAF). The MFR leverages intra-modal correlations to reconstruct the representations of missing modalities, thereby preserving the semantic consistency of the feature space even when certain camera views are unavailable. The MAF, on the other hand, learns dynamic weighting factors to adaptively balance and fuse information from different modalities, achieving reliable integration between the reconstructed image features and LiDAR representations. By jointly optimizing these two modules, RMP effectively maintains robust perception capability in challenging scenarios with incomplete sensor inputs.

The main contributions of our work can be summarized as follows:

• We present RMP, a robust multi-modal perception framework designed to address missing-modality scenarios.
• We design a missing feature reconstruction mechanism that exploits intra-modal feature correlations to recover missing camera representations, thereby alleviating performance degradation. Furthermore, we introduce a cross-modal adaptive fusion strategy that learns adjustable weights to fuse image and LiDAR features, improving the efficiency and reliability of cross-modal information interaction.
• Experiments on nuScenes [17] show SOTA-level performance and strong robustness under various camera view missing conditions.

2. Related Work

2.1. Single-Modal Perception

Methods of single-modal perception mainly rely on information from visual images or LiDAR points to enable edge devices to perceive and understand the driving environment [18]. LiDAR-based approaches exploit accurate 3D geometric structures to achieve high-precision scene perception. For instance, PointNet [19] and its variants design networks that operate on point cloud inputs while preserving their spatial structures. VoxelNet [20] introduces a trainable end-to-end framework that converts irregular point sets to regular voxel grids, enabling 3D convolutions to effectively learn features. Although LiDAR-based methods exhibit strong robustness against illumination and weather variations, they often lack the ability to capture rich semantic and texture information [21]. In contrast, camera-based approaches rely on image data with abundant texture details to achieve fine-grained recognition of scene appearance and object categories. For example, PRET [22] integrates 2D visual features with 3D spatial coordinates to generate position-aware representations, facilitating the detection of 3D objects. However, since cameras are highly sensitive to illumination changes and occlusions, the performance of such methods tends to degrade under complex environmental conditions. In summary, single-modal perception approaches are inherently limited by the weaknesses of individual sensors and struggle to maintain completeness under challenging autonomous driving environments.

2.2. Multi-Modal Perception

To better leverage the unique strengths provided by various sensing modalities, an increasing number of studies have focused on multi-modal perception in autonomous driving [10]. According to the stage of information interaction, existing fusion methods are typically divided into early fusion, middle fusion, and late fusion. In early fusion, raw data from different modalities are first projected into a unified coordinate system before feature extraction. For example, AutoAlignV2 [13] introduces a deformable feature aggregation module that enables the model to adaptively align LiDAR points with image pixels according to the geometric structure of the scene. Middle fusion approaches, on the other hand, perform modality-specific feature encoding followed by feature-level integration. BEVFusion [23] maps the outputs of different sensors into a shared BEV (Bird’s-Eye-View) representation space, allowing joint cross-modal encoding and spatially consistent fusion. In late fusion, the integration occurs at the decision level, where the emphasis lies in ensuring consistency across the outputs of different modality-specific detectors. SparseFusion [24] aggregates the detection outputs from LiDAR and camera modalities as candidate points and performs decision-level fusion, achieving efficient and complementary multi-sensor integration. Although multi-modal fusion has significantly improved perception accuracy and environmental understanding, its performance still deteriorates when partial sensor data are missing. This remains one of the major bottlenecks for deploying multi-modal perception systems in real-world autonomous driving scenarios. In addition, the works on wireless imaging by channel state information using ray tracing for next-generation 6G offer promising non-visual cues that remain robust under darkness or fog, potentially serving as complementary modalities for future multi-modal perception frameworks.

2.3. Missing Modality Perception

In response to the challenge of missing modalities, a growing body of research has recently emerged [15,16,25], aiming to enhance the robustness of multi-modal perception systems under incomplete input conditions. MSeg3D [15] introduces a unified projection-based cross-modal alignment framework that maintains effective perception capability even when some camera views are unavailable. MMFusion [26] adopts an attention-weighted strategy to dynamically modify the contribution of each modality, emphasizing reliable features while suppressing the influence of missing or noisy modalities. Although these methods have achieved notable progress in improving robustness against missing inputs, most of them primarily focus on robust fusion rather than explicitly modeling or reconstructing the missing information. Consequently, as the degree of modality absence increases, the overall perception performance still degrades significantly. Recent studies have also explored available unannotated data and incomplete data for learning. NJIMA et al. [27] employ a weighting-based semi-supervised deep learning strategy, together with an alternative localization framework that relies on adversarial generative modeling, to mitigate the challenge posed by limited labeled samples during model training. This method aims to alleviate the problem of insufficient training samples. Our work focuses on the reconstruction problem when camera views are missing. Although the challenges faced by these two problems are different, the underlying ideas are complementary. Based on the above, our work explores a missing-feature reconstruction strategy from the perspective of intra-modal feature correlations, aiming to achieve more robust and informative multi-modal representation learning.

3. Method

3.1. Overall Framework

Figure 2 outlines the pipeline of RMP, which is structured around three modules: feature extraction, missing feature reconstruction, and multi-modal fusion. Firstly, each of the LiDAR and remaining camera modalities is processed by its respective backbone, extracting its initial features (voxel features for LiDAR, image features for available views). The features of the LiDAR voxel are then projected to the point domain through a voxel-to-point transformation. Second, for the missing camera views, the corresponding image features are reconstructed by leveraging intra-modal correlations from neighboring camera perspectives, after which the reconstructed image features are also converted to the point level. Finally, point-wise features from both modalities are jointly aggregated through the multi-modal fusion module, producing the fused multi-modal representations that serve as the final perception features.

3.2. Feature Extraction

Given a set of N raw LiDAR points $P\in {\mathbb{R}}^{N\times C_{3D}^{in}}$ with input dimension $C_{3D}^{in}$, the 3D backbone extracts voxel features $V\in {\mathbb{R}}^{N_v\times C_v}$. Similar to [28], a voxel-to-point mapping $M^{v\to p}$ is applied to obtain point-wise features $F_p\in {\mathbb{R}}^{N\times C_{3D}}$. For multi-view images captured by $N_c$ cameras, the 2D backbone extracts image features from the available views, denoted as $I_r\in {\mathbb{R}}^{N_c\times C_c\times H_c\times W_c}$, where $C_c$, $H_c$, and $W_c$ represent the channel dimension, height, and width of the feature maps, respectively. To simulate real-world missing-view conditions, one or more camera images are randomly masked during training, and their corresponding original image features are recorded as $I_m$.

3.3. Missing Feature Reconstruction

This module aims to reconstruct the image features of missing camera views based on the information available from existing views. Adjacent camera views often share overlapping regions, which provide valuable intra-modal contextual cues. Building on this observation, we design a Missing Feature Reconstruction (MFR) mechanism that employs the shared information among neighboring views to infer and reconstruct the features corresponding to the missing perspectives.

For clarity of explanation, we assume that the front-view image is missing in this section. To handle this case, we first replace the missing front-view features with shared and learnable mask tokens, and the replaced features are denoted as $I_t$. The learnable mask token is initialized with a zero-mean Gaussian distribution. Let $I_{left}$ and $I_{right}$ represent the image features extracted by the neighboring cameras positioned at the front-left and front-right sides, respectively. As illustrated in Figure 2, the area to the left of the front camera is highly correlated with the right region of the front-left camera, while the area in the right region is highly correlated with the left region of the front-right camera. Based on this observation, we introduce a cropping ratio $\eta$ to divide the image features along the width dimension into three parts: left, with column indices [0, $\eta W_c$); middle, with column indices [$\eta W_c$, ($1-\eta$)$W_c$); and right, with column indices [($1-\eta$)$W_c$, $W_c$). (1) For the left part, we reference the right-region features of the front-left view, denoted as $I_{left}\left[right\right]$; (2) for the right part, we reference the left-region features of the front-right view, denoted as $I_{right}\left[left\right]$; (3) for the middle part, since no adjacent view can provide reliable guidance, we retain the corresponding masked tokens of the front view. Therefore, the reconstructed front-view feature consists of the following three parts:

$$I_{mask}=\mathcal{C}[I_{left}\left[right\right],I_t\left[mid\right],I_{right}\left[left\right]]$$

(1)

where $\mathcal{C}$ denotes the concatenation operation. The merged feature representation $I_{mask}$ is processed by a Transformer-based decoder to produce the reconstructed feature of the missing view, denoted as $I_{reg}$.

$$I_{reg}=Decoder\left(I_{mask}\right)$$

(2)

After reconstructing the missing-view features using intra-modal information from adjacent perspectives, the disparity between the reconstructed features $I_{reg}$ and the original features $I_m$ is quantified using the Mean Squared Error (MSE) metric.

$${\mathcal{L}}_{MFR}=MSE(I_m,I_{reg})$$

(3)

The core of MFR relies on the relative adjacency relationship between cameras, rather than the absolute field-of-view settings. For any missing viewpoint, its left and right adjacent cameras are selected based on the camera positions, and the cropping region division remains consistent with the description in the paper. For boundary viewpoints with only one adjacent camera, the unavailable side is filled with the learnable mask tokens. If the left adjacent camera is missing, the reconstructed features can be represented as $I_{mask}=\mathcal{C}[I_t\left[right\right],I_t\left[mid\right],I_{right}\left[left\right]]$. The subsequent decoding process remains unchanged.

3.4. Multi-Modal Fusion

The reconstructed front-view image features $I_{reg}$ are combined with the remaining camera-view features $I_r$ to form the reconstructed multi-view image feature I. Similar to the LiDAR point cloud, a point-to-pixel mapping $M^{p\to c}$ is applied to obtain 2D point-wise features $F_I\in {\mathbb{R}}^{N\times C_{2D}}$. Both $F_I$ and $F_p$ are then processed within the multi-modal fusion module, as shown in Figure 2. Specifically, a linear projection operation is first applied to map both $F_I$ and $F_p$ into a common feature space with a unified dimension of $N\times C_u$. Then, $1\times 1$ convolutional layers are used to expand the feature channels of $F_I$ and $F_p$ to appropriate dimensions. The expanded features are summed element-wise, and a convolution together with a tanh activation generates the feature weighting map w. The tanh function normalizes the cross-modal interaction into the range [−1, 1], allowing the network to express both positive enhancement and negative suppression when combining features. The formula can be expressed as follows:

$$F_I^{\prime }=Linear\left(F_I\right),F_p^{\prime }=Linear\left(F_p\right)$$

(4)

$$w=\sigma \left(Conv(tanh(Conv\left(F_p^{\prime }\right)+Conv\left(F_I^{\prime }\right)))\right)$$

(5)

where $\sigma$ converts the above results into a probabilistic confidence weight $w\in (0,1)$, representing the relative reliability of LiDAR and image features for each point. Next, the 3D point-wise features and 2D point-wise features are adaptively fused according to the learned feature weights w, and the fused representation is denoted as $F_{fu}$. The fusion process can be formulated as follows:

$$F_{fu}=Cat\left(w{F}_p^{\prime },(1-w)F_I^{\prime }\right)$$

(6)

Finally, the fused feature $F_{fu}$ is passed through an MLP-based segmentation head to produce the final segmentation prediction $\widehat{Y}$:

$$\widehat{Y}=MLP\left(F_{fu}\right)$$

(7)

Given the model’s segmentation output $\widehat{Y}$ and the ground-truth labels Y, the model is supervised using a composite segmentation loss composed of the Lovasz term ${\mathcal{L}}_l$ and the cross-entropy term ${\mathcal{L}}_c$:

$${\mathcal{L}}_{Seg}={\mathcal{L}}_l(\widehat{Y},Y)+{\mathcal{L}}_c(\widehat{Y},Y)$$

(8)

The final training objective integrates the reconstruction loss and segmentation loss with weighting coefficients ${\lambda }_1$ and ${\lambda }_2$, defined as follows:

$$\mathcal{L}={\lambda }_1{\mathcal{L}}_{MFR}+{\lambda }_2{\mathcal{L}}_{Seg}$$

(9)

4. Experiment

4.1. Dataset

We assess the performance and robustness of our method using the nuScenes dataset [17]. This dataset contains synchronized multi-sensor recordings captured in complex urban environments, incorporating six camera views, a LiDAR scanner, and multiple radars. It consists of 1000 driving scenes, each lasting about 20 s, split into 700 scenes for training, 150 for validation, and 150 for testing. Each scene is annotated with semantic labels, ego-vehicle poses, and other metadata, making the dataset suitable for various 3D scene understanding tasks.

4.2. Metrics

Following previous studies [5], the mean Intersection-over-Union (mIoU) is employed as the metric for evaluating the performance of segmentation. For each semantic class C, IoU is computed as $Io{U}_C={\displaystyle \frac{T{P}_C}{T{P}_C+F{P}_C+F{N}_C}}$, where $T{P}_C$, $F{P}_C$, and $F{N}_C$ denote the number of true positive, false positive, and false negative predictions for the C-th category. mIoU is obtained by averaging IoUs over all valid classes. All results are reported on the nuScenes official class definitions (class 17).

4.3. Implementation Details

Following [15], the LiDAR branch is constructed using a 3D U-Net [29], while the camera branch utilizes HRNet18 [30]. Unless otherwise specified, one camera view is randomly masked during both training and inference to simulate missing-view scenarios. Optimization is performed with Adam [31] with a learning rate of 0.001. Training is carried out with a batch size of 2, 48 epochs, and the cropping ratio is set to 0.2. To accelerate training, each image is cropped to 40% of its original resolution. The Transformer decoder uses a four-layer architecture, with 4 heads for multi-head attention, a hidden dimension of 512, and a dropout setting of 0.2. The weights for ${\mathcal{L}}_{Seg}$ and ${\mathcal{L}}_{MFR}$ are assigned values of 1 and 2. Similar to [15], we used data augmentation techniques such as LiDAR rotation (rotation about the Z axis with angles sampled from $[-{\displaystyle \frac{\pi }{4}},{\displaystyle \frac{\pi }{4}}]$) and image scaling (scaling using a stochastic factor drawn from the range [1.0, 1.5]) to avoid overfitting. Experiments are run on four NVIDIA RTX 4090 GPUs, and no Test-Time Augmentation (TTA) is employed during either training or inference.

4.4. SOTA Comparison

nuScenes testing. 

Table 1. Quantitative results of different methods on the nuScenes testing set. ’L’ is LiDAR and ’C’ is camera. ^★ denotes that the training and inference environments are identical to those used in this paper. The bold numbers indicate the best results.

Method	Input	mIoU	Barrier	Bicycle	Bus	Car	Construction	Motorcycle	Pedestrian	Traffic_CONE	Trailer	Truck	Driveable	Other_FLAT	Sidewalk	Terrain	Manmade	Vegetation
PolarNet [32]	L	69.4	72.2	16.8	77.0	86.5	51.1	69.7	64.8	54.1	69.7	63.5	96.6	67.1	77.7	72.1	87.1	84.5
JS3C-Net [33]		73.6	80.1	26.2	87.8	84.5	55.2	72.6	71.3	66.3	66.8	71.2	96.8	64.5	76.9	74.1	87.5	86.1
SPVNAS [34]		77.4	80.0	30.0	91.9	90.8	64.7	79.0	75.6	70.9	81.0	74.6	97.4	69.2	80.0	76.1	89.3	87.1
Cylinder3D [6]		77.2	82.8	29.8	84.3	89.4	63.0	79.3	77.2	73.4	84.6	69.1	97.7	70.2	80.3	75.5	90.4	87.6
AF2S3Net [35]		78.3	78.9	52.2	89.9	84.2	77.4	74.3	77.3	72.0	83.9	73.8	97.1	66.5	77.5	74.0	87.7	86.8
SphereFormer [18]		78.1	81.5	39.7	93.4	87.5	66.4	75.7	77.2	70.6	85.6	73.6	97.6	64.8	79.8	75.0	92.2	89.0
LaCRange [36]	LC	75.3	78.0	32.6	88.3	84.5	63.9	81.5	75.6	72.5	64.7	68.0	96.6	65.9	78.6	75.0	90.4	88.3
PMF-ResNet50 [37]		77.0	82.1	40.3	80.9	86.4	63.7	79.2	79.8	75.9	81.2	67.1	97.3	67.7	78.1	74.5	90.0	88.5
APPFNet [38]		78.1	77.2	52.2	90.9	93.6	54.2	79.2	80.7	71.2	64.1	84.2	97.5	73.9	77.2	75.2	91.1	87.9
PMF [37]		75.5	80.1	35.7	79.7	86.0	62.4	76.3	76.9	73.6	78.5	66.9	97.1	65.3	77.6	74.4	89.5	88.5
EPMF [39]		79.2	76.9	39.8	90.3	87.8	72.0	86.4	79.6	76.6	84.1	74.9	97.7	66.4	79.5	76.4	91.1	87.9
MSeg3D-H48 [15]		81.1	83.1	42.5	94.9	92.0	67.1	78.6	85.7	80.5	87.5	77.3	97.7	69.8	81.2	77.8	92.4	90.1
Mseg3D ★ [15]		72.6	71.5	39.7	92.5	87.8	45.0	79.5	76.9	61.2	51.0	83.0	95.7	69.5	69.3	70.3	85.3	84.1
Ours	LC	79.8	82.5	44.9	92.7	91.0	71.5	73.9	82.5	76.1	85.7	75.5	97.4	69.1	79.9	76.3	90.3	87.4

reports the performance of different methods on the nuScenes testing set. During both training and inference, our method assumes that one random camera view is missing. As shown in the table, multi-modal approaches generally outperform single-modality ones, demonstrating the advantage of combining heterogeneous data to mitigate sensor-specific weaknesses and enhance perception accuracy in complex driving scenarios. Among the multi-modal methods, our approach achieves competitive performance, only 1.3% mIoU lower than MSeg3D-H48 [15]. We attribute this small gap to two main factors: (1) MSeg3D-H48 adopts a more powerful image backbone, HRNet48, whereas we employ the lighter HRNet18 to meet the efficiency requirements of edge devices; and (2) MSeg3D-H48 is trained and evaluated under fully available modalities, while our method is designed for random missing-view conditions in both training and inference. To ensure fairness, we re-trained Mseg3D using HRNet18 under the same settings as our method. Results show that our method surpasses Mseg3D by 7.2% mIoU, which further validates the robustness and effectiveness of our proposed framework under incomplete modality scenarios.

nuScenes validation. We also conducted experiments on the nuScenes validation set, as presented in

Table 2. The results of different approaches on the nuScenes validation set. ^★ denotes that the training and inference environments are identical to those used in this paper. The bold numbers indicate the best results.

Method	Input	mIoU	Barrier	Bicycle	Bus	Car	Construction	Motorcycle	Pedestrian	Traffic_CONE	Trailer	Truck	Driveable	Other_FLAT	Sidewalk	Terrain	Manmade	Vegetation
AF2S3Net [35]	L	62.2	60.3	12.6	82.9	80.0	20.1	62.0	59.0	49.0	42.2	67.4	94.2	68.0	64.1	68.6	82.9	82.4
PolarNet [32]		71.0	74.7	28.2	85.3	90.9	35.1	77.5	71.3	58.8	57.4	76.1	96.5	71.1	74.7	74.0	87.3	85.7
Cylinder3D [6]		76.1	76.4	40.3	91.2	93.8	51.3	78.0	78.9	64.9	62.1	84.4	96.8	71.6	76.4	75.4	90.5	87.4
2DPASS [5]		76.4	74.4	44.3	93.6	92.0	54.0	79.7	78.9	57.2	72.5	85.7	96.2	72.7	74.1	74.5	87.5	85.4
SphereFormer [18]		78.4	77.7	43.8	94.5	93.1	52.4	86.9	81.2	65.4	73.4	85.3	97.0	73.4	75.4	75.0	91.0	89.2
SDSeg3D [40]		77.7	77.5	49.4	93.9	92.5	54.9	86.7	80.1	67.8	65.7	86.0	96.4	74.0	74.9	74.5	86.0	82.8
APPFNet [38]	LC	78.1	77.2	52.2	90.9	93.6	54.2	79.2	80.7	71.2	64.1	84.2	97.5	73.9	77.2	75.2	91.1	87.9
PMF-ResNet50 [37]		79.0	74.9	55.4	91.0	93.0	60.5	80.3	83.2	73.6	67.2	84.5	95.9	75.1	74.6	75.5	90.3	89.0
2D3DNet [41]		79.0	78.3	55.1	95.4	87.7	59.4	79.3	80.7	70.2	68.2	86.6	96.1	74.9	75.7	75.1	91.4	89.9
MSeg3D-H48 [15]		80.0	79.2	59.8	96.1	89.4	54.1	89.3	82.2	72.8	70.4	86.0	96.7	73.6	76.1	75.6	89.3	88.3
MSeg3D ★ [15]		74.7	72.4	48.1	93.8	88.4	47.0	82.6	79.7	64.6	56.1	84.1	96.0	69.5	70.2	71.0	87.0	86.2
MSeg3D ★ $+MFR$  [15]		79.0	78.4	55.4	95.3	89.4	57.3	88.4	83.4	68.8	66.5	86.1	96.4	74.9	74.3	73.0	89.0	87.1
Ours		80.3	79.2	56.8	96.1	89.7	56.7	89.5	84.4	72.6	71.1	87.7	96.9	75.9	76.5	76.1	89.3	87.4

. The experimental protocol mirrors that used for the testing benchmark. As shown in the results, multi-modal methods consistently outperform single-modality ones, highlighting the advantages of leveraging complementary information across different sensors. Notably, our approach attains an mIoU of 80.3%, which stands as the highest score among all evaluated models. Similar to the testing set, we also re-trained MSeg3D under the same experimental configuration for a fair comparison. The result of MSeg3D is surpassed by our method with a 5.6% mIoU advantage, primarily due to the proposed missing-feature completion strategy. Furthermore, we incorporated the MFR module into MSeg ^★, and the results show a performance improvement of 4.3% mIoU, demonstrating that this performance gain primarily stems from the MFR module, rather than differences in backbone capabilities. The MFR module explicitly reconstructs semantic features from missing views, providing a more complete feature space for the fusion network. This effectively alleviates the feature imbalance caused by missing modalities, making the final multi-modal fusion more stable. These observations further validate the robustness and effectiveness of our work under missing-modality conditions.

Different crop ratio & missing camera views. Figure 3 (on the left) evaluates how different crop ratios influence segmentation performance. As the crop ratio increases from 0.16 to 0.20, the mIoU steadily improves and reaches its peak at 0.20. When the ratio continues to rise, the performance slightly declines, suggesting that excessive cropping may introduce noise or remove useful contextual information. A moderate crop ratio therefore contributes to a more improved model performance. The right part of Figure 3 plots the result of our method and MSeg3D under varying numbers of missing cameras. As the number of available cameras decreases from six to zero, MSeg3D exhibits a significant drop in mIoU, while our method experiences only a minor decrease. The relative improvement over MSeg3D grows from 5.1% to 7.8%, demonstrating the strong robustness of our approach under missing-modality conditions. Furthermore, when the number of missing cameras exceeds three, the RMP’s performance decline slows. We attribute this phenomenon to the limits of adjacent information used by MFR. Once most neighboring views are absent, losing additional views provides limited new contextual information. These results confirm that the proposed modality completion and adaptive fusion strategies can effectively compensate for the absence of visual inputs, ensuring stable perception performance in complex autonomous driving scenarios.

4.5. Ablation Study

Module ablation & Different learning rate. Figure 4 shows the performance variation of the model on the nuScenes validation set under different settings. To verify the effectiveness of the MFR module, we removed it and replaced the missing features with zero values, as shown in Figure 4a. The results indicate a degradation of 2.4% mIoU without MFR, demonstrating that explicit modeling has a significant advantage over the simple imputation strategy. Figure 4b compares three fusion strategies. Compared to average fusion and fixed-weight fusion (LiDAR accounting for 60%), the proposed adaptive fusion achieves gains of 1.1% mIoU and 1.8% mIoU, respectively. This highlights that dynamically adjusting the weights of different modal features can achieve superior perceptual performance. Figure 4c and Figure 5 report how different learning rates affect model performance. When the learning rate is set to 0.001, the model achieves optimal performance. In our experiments, this setting yielded more stable results compared to larger learning rates.

Efficiency and Performance Analysis. Figure 6 presents a comprehensive assessment of different methods based on efficiency and performance. As shown, our approach achieves the optimal balance between efficiency and performance compared to other competitors. Specifically, our model attains an mIoU of 80.3% with an inference time of 207 ms. In contrast, MSeg3D-H48, which achieves similar accuracy, incurs increased inference time (207 ms versus 445 ms) and model size (30 M versus 87.3 M). Compared with MSeg3D, which has a parameter scale similar to ours (30 M versus 31.5 M), our method delivers an additional 1.1% percentage-point gain in mIoU. When contrasting MSeg3D-H48 and MSeg3D, although selecting HRNet-48 yields a 0.7% performance boost, it significantly increases both model parameters and inference time. Considering the resource constraints of edge devices, our method thus selects HRNet-18 as the backbone for the image branch.

Visualization. We visualized our approach and MSeg3D on the nuScenes validation set, as illustrated in Figure 7. The figure presents results from three different viewpoints and compares them with the ground truth. As shown, our model produces noticeably cleaner and more reliable segmentation than MSeg3D. Specifically, for dynamic categories such as trailer, our approach produces clearer and more complete boundaries (improved mIoU by 1.7%). Moreover, for large-scale static elements like vegetation, it exhibits superior regional consistency (improved mIoU by 0.5%). In addition, our method successfully corrects segmentation errors made by MSeg3D, such as misclassifying pedestrians near trailer (improved mIoU by 0.4%). Overall, the visual comparisons provide strong evidence of the effectiveness and robustness of our proposed method.

5. Conclusions

In this study, we proposed RMP, a robust multi-modal perception model designed to handle missing modality scenarios in autonomous driving. RMP integrates a missing feature reconstruction module and a cross-modal adaptive fusion strategy to enhance perception robustness under an incomplete sensor environment. A series of experiments on the nuScenes benchmark demonstrates that the proposed method effectively mitigates performance degradation caused by missing camera views and produces results comparable to those of the most advanced contemporary methods. Future work will focus more on practical deployment scenarios and conduct more comprehensive efficiency analyses, such as the peak memory consumption on edge devices with different hardware configurations.