Lightweight Semantic Segmentation Algorithm Based on Gated Visual State Space Models

Abstract

LiDAR serves as the primary sensor for acquiring environmental information in intelligent driving systems. However, under adverse weather conditions, point cloud signals obtained by LiDAR suffer from intensity attenuation and noise interference, leading to a decline in segmentation accuracy. To address these issues, this paper designs a lightweight semantic segmentation system based on the Gated Visual State Space Model (VMamba), named RainMamba. Specifically, the system utilizes spherical projection to transform point clouds into 2D sequences and constructs a physical perception feature embedding module guided by the Beer–Lambert law to explicitly model and suppress spatial noise at the source. Subsequently, an uncertainty-weighted cross-modal correction module is employed to incorporate RGB images for dynamically calibrating the degraded point cloud data. Finally, a VMamba backbone is adopted to establish global dependencies with linear complexity. Experimental results on the SemanticKITTI dataset demonstrate that the system achieves an inference speed of 83 FPS, with a relative mIoU improvement of approximately 7.2% compared to the real-time baseline PolarNet. Furthermore, zero-shot evaluations on the real-world SemanticSTF dataset validate the system’s robust Sim-to-Real generalization capability. Notably, RainMamba delivers highly competitive accuracy comparable to the state-of-the-art heavy-weight model PTv3 while requiring a significantly lower parameter footprint, thereby demonstrating its immense potential for practical edge-computing deployment.

1. Introduction

With the rapid advancement of intelligent driving technologies, the precision of environmental perception systems directly impacts vehicle safety. In experimental teaching and coursework for autonomous driving, Light Detection and Ranging (LiDAR) has become an indispensable core sensor due to its ability to provide high-precision 3D geometric information [1]. However, in practical system deployment and testing, balancing high precision with low computational latency to achieve real-time semantic segmentation of dynamic point clouds remains a significant bottleneck restricting the commercialization of this technology. Traditional solutions struggle to satisfy both speed and accuracy requirements simultaneously: high-precision voxel-based methods [2,3] suffer from massive computational loads, making it difficult to meet the low-latency demands of real-time control; conversely, lightweight projection-based methods [4,5,6], while faster, are prone to losing fine-grained details in complex scenes, thereby reducing the reliability of the perception system.

To overcome these computational bottlenecks and address the limited receptive field issues inherent in traditional Convolutional Neural Networks (CNNs), the academic community has recently turned its attention to State Space Models (SSM) [7]. As an efficient sequence modeling scheme, SSMs, represented by Mamba, offer a novel alternative to Transformers. While recent attention-based state-of-the-art models, such as Point Transformer v3 (PTv3) [8], have achieved remarkable theoretical accuracy, their quadratic computational complexity ($O\left(N^2\right)$) and massive parameter footprints severely prohibit their real-time deployment on power-constrained autonomous driving edge devices. By leveraging linear complexity and global context capturing capabilities, SSMs successfully resolve this dilemma. In particular, the 2D Selective Scan (SS2D) mechanism proposed in VMamba [9] successfully resolves the inability of 1D SSMs to process 2D spatial neighborhood information, achieving an excellent balance between performance and speed in visual tasks. Although SSMs have shown promise in image classification, their migration to LiDAR perception tasks remains in the exploratory stage. Specifically, there is currently no mature solution for leveraging the long-range modeling capabilities of SSMs to handle degraded signals under adverse weather conditions.

In practical applications, adverse weather conditions such as rain, fog, and snow significantly compromise the robustness of perception systems [10]. Suspended particles in the air cause intensity attenuation of laser echoes and generate outlier noise [10,11]. Existing mainstream methods often perform poorly in the face of such physical degradation. The primary reason is that current models rely excessively on geometric position information for discrimination, neglecting the physical characteristics of LiDAR and multi-modal cues. When geometric structures are interrupted by rain/fog or interfered with by noise points, models lacking physical prior constraints are prone to false detections, leading to a drastic decline in system performance. Furthermore, many existing perception algorithms overfit to simulated optical degradation, struggling to bridge the Sim-to-Real gap when confronted with the complex, unpredictable nature of real-world atmospheric scattering (e.g., as captured by the SemanticSTF dataset [12]).

To address perception failure in adverse weather while maintaining inference efficiency, this paper designs and implements a lightweight LiDAR semantic segmentation system tailored for such environments. First, addressing the limited receptive fields of CNNs and the excessive resource consumption of Transformers, we adopt a projection-based lightweight architecture and introduce VMamba as the backbone network. Utilizing its efficient SS2D scanning mechanism, the model captures long-range dependencies, effectively reconnecting geometric structures fragmented by rain or fog occlusion. Second, to suppress noise at the source, we design a Physical Perception Module. This module employs asymmetric convolutions to adapt to LiDAR scanning characteristics [13] and directly utilizes echo intensity information to filter out low-intensity outlier noise. Finally, addressing the issues of single-modal signal loss and point cloud sparsity, we reference the PointPainting [14] fusion paradigm and propose an Uncertainty-Weighted Cross-Modal Correction Module. By incorporating high-resolution texture features provided by Red–Green–Blue (RGB) images, this module dynamically calibrates low-confidence LiDAR features, achieving effective dehazing and structural completion.

The main contributions of this paper are summarized as follows:

• We propose RainMamba, a novel multi-modal semantic segmentation framework that transcends conventional module-splicing by deeply coupling physics-based noise priors with the linear-time Visual State Space Model (VMamba), tailored specifically for adverse weather perception.
• We design a Physical Perception Feature Embedding Module that explicitly integrates the Beer–Lambert Law into the network input. By capturing the physical mapping between distance, intensity attenuation, and environmental noise, it effectively suppresses rain and fog interference at the source.
• We introduce an Uncertainty-Weighted Cross-Modal Correction mechanism. Rather than using computationally expensive cross-attention, we utilize a lightweight gating strategy to dynamically align and fuse RGB texture features, compensating for geometric details lost in 2D spherical projection while strictly preserving $O\left(N\right)$ linear complexity.
• Extensive evaluations demonstrate exceptional Sim-to-Real generalization. Zero-shot tests on the real-world SemanticSTF dataset prove that RainMamba achieves highly competitive robustness comparable to heavy-weight, state-of-the-art architectures (e.g., PTv3) while operating at real-time speeds with a significantly lower parameter footprint.

2. Proposed Method

2.1. Overall Architecture

As illustrated in Figure 1, the proposed RainMamba algorithm is designed as a streamlined, end-to-end pipeline consisting of three core components. First, the Physical Perception Feature Embedding Module extracts robust features by suppressing rain and fog noise at the source using physical priors. Second, the Uncertainty-Weighted Cross-Modal Correction mechanism fuses high-resolution RGB texture cues to dynamically calibrate the degraded point cloud features. Finally, the Context Inference Engine, built upon the Visual State Space Model (VMamba), establishes global dependencies with linear complexity to achieve highly efficient and accurate semantic segmentation.

2.2. Physical Perception Feature Embedding Module

To address the issues of spatial sparsity, signal attenuation, and high noise interference in vehicle-mounted LiDAR under adverse weather conditions, this system proposes a feature embedding method based on physical channel enhancement. This method constructs a full-process denoising link from data acquisition to feature extraction, adhering to the principle of deep fusion between physical characteristics and geometric structures [13].

2.2.1. Spherical Projection and Feature Tensor

The design goal of this stage is to resolve the efficiency bottleneck that makes sparse 3D point clouds difficult to process directly with CNNs. The system employs a Spherical Projection strategy [4] to seamlessly connect 3D space with the 2D plane. The projection logic is defined as follows:

$$\left(\begin{array}{c}u\\ v\end{array}\right)=\left(\begin{array}{c}{\displaystyle \frac{1}{2}}[1-arctan(y,x)/\pi ]·W\\ [1-(arcsin(z/r)+f_{up})/f_{fov}]·H\end{array}\right)$$

(1)

where $(u,v)$ are the projected 2D pixel coordinates; W and H are the width and height of the projected image, respectively; r is the depth range of the point; and $f_{fov}$ and $f_{up}$ are the vertical field of view and the upper field of view of the LiDAR, respectively.

Crucially, to explicitly model the signal degradation caused by adverse weather, we incorporate the Beer–Lambert Law [11] into our physical feature embedding. In rain or fog, the received LiDAR echo intensity $I_{out}$ undergoes exponential attenuation over distance d, formulated as:

$$I_{out}=I_{in}·e^{-\alpha d}+N_{env}$$

(2)

where $I_{in}$ is the original emission intensity, $\alpha$ represents the atmospheric attenuation coefficient dependent on weather severity (e.g., precipitation rate or fog density), and $N_{env}$ denotes the additive environmental noise caused by particle backscattering.

After spherical projection, each 2D pixel on the $(u,v)$ grid retains a 5-channel feature vector $[x,y,z,r,I_{out}]$ from the corresponding 3D point (where r approximates d). In this way, the network is explicitly guided to learn the physical mapping between distance, weather-induced intensity drop, and spatial noise, which are visualized in Figure 2, thereby achieving physics-aware denoising at the preprocessing stage.

While spherical projection significantly accelerates data processing, it inevitably introduces geometric information loss, particularly the “many-to-one” mapping conflict where multiple points fall into the same pixel grid. This phenomenon, often referred to as the “range image aliasing” effect, can potentially blur the boundaries of small objects or distant structures. To mitigate this inherent trade-off between efficiency and geometric fidelity, RainMamba employs three strategic compensation designs: (1) k-Nearest Neighbor (KNN) smoothing is executed during the 3D reprojection step to refine boundary predictions and recover geometric details from the 2D plane; (2) high-resolution RGB texture features, introduced via the subsequent Gated Fusion Module, provide dense semantic cues that compensate for point cloud sparsity; and (3) unlike standard local convolutions, the VMamba backbone captures global dependencies, allowing the model to infer semantics based on scene-level context rather than relying solely on local geometries. These synergistic strategies enable the system to maintain high accuracy on small objects while operating at a real-time speed of 83 FPS.

2.2.2. Feature Extraction Based on Asymmetric Convolution

Although standard convolution kernels are widely used in image processing, they face difficulties in fully extracting features from LiDAR point clouds with non-uniform distribution characteristics. Following the design philosophy of “capturing textures horizontally and suppressing noise vertically,” this module deploys a set of Asymmetric Convolutions [13]:

• A $1\times k$ convolution kernel is used to extract continuous geometric features in the horizontal direction;
• A $k\times 1$ convolution kernel is used to suppress discrete environmental noise in the vertical direction.

Experiments show that this design effectively expands the receptive field of feature extraction and significantly reduces the impact of environmental interference on semantic segmentation accuracy.

2.3. Uncertainty-Weighted Cross-Modal Correction Mechanism

To solve the problems of reduced perception capability and fragmented geometric structures in single-modal occlusion scenarios, the system constructs a three-stage progressive cross-modal fusion mechanism: “Feature Alignment–Gated Evaluation–Dynamic Correction.”

2.3.1. Multi-Source Data Spatial Alignment

In the multi-source information fusion stage, the system first performs spatial registration between the LiDAR and the visual camera. By reading the joint calibration parameters, RGB image pixels are accurately mapped to the point cloud coordinate system, establishing a spatial correspondence link between texture features and geometric features [14].

2.3.2. Gated Fusion Network and Dynamic Correction

This module designs a fusion network based on a Gated Mechanism. The focus of this step is to enable the system to autonomously evaluate the confidence of data from different modalities. The core calculation logic is as follows:

$$M_{gate}=\sigma (W_g\ast [F_{pts},F_{img}]+b_g)$$

(3)

$$F_{fused}=M_{gate}\odot F_{pts}+(1-M_{gate})\odot F_{img}$$

(4)

where $\sigma$ is the Sigmoid activation function; $[·]$ denotes the feature channel concatenation operation; $W_g$ and $b_g$ are the convolution weights and bias of the gating network; $M_{gate}$ is the generated dynamic confidence mask; and ⊙ denotes element-wise multiplication.

Unlike traditional cross-attention or late-fusion paradigms that rely on quadratic-complexity matrix multiplications ($O\left(N^2\right)$), our gating mechanism achieves multi-modal feature alignment through element-wise operations and lightweight convolutions. This design strictly maintains the linear complexity of the overall network, which is critical for meeting real-time requirements on edge devices.

It is worth noting that for LiDAR point cloud regions outside the field of view of the RGB camera, the system automatically switches to the pure LiDAR branch for feature extraction, using the cross-modal gated fusion mechanism only in the overlapping field of view.

2.4. RainMamba Context Inference Engine

Targeting the limitations of computing power and real-time performance faced by traditional deep learning algorithms when deployed on edge-computing terminals [15], this system builds a core inference engine based on RainMamba (Figure 3).

2.4.1. Linear Complexity and Real-Time Guarantee

The focus of this step is to resolve the contradiction between the processing efficiency of massive point cloud data and limited hardware resources. Following the principle of “efficient computation and controllable memory,” the engine builds a state transition link with linear complexity based on the State Space Model (SSM) [7]. Its core discretized state equation is:

$$\begin{array}{cc}\hfill h_t& =\overline{A}{h}_{t-1}+\overline{B}{x}_t\hfill \\ \hfill y_t& =C{h}_t\hfill \end{array}$$

(5)

where $\mathrm{\Delta }$ is the time step parameter representing the discretization scale; $\overline{A}=exp\left(\mathrm{\Delta }A\right)$ and $\overline{B}={\left(\mathrm{\Delta }A\right)}^{-1}(exp\left(\mathrm{\Delta }A\right)-I)·\mathrm{\Delta }B$ are the state parameters after Zero-Order Hold (ZOH) discretization; $h_t$ is the hidden state variable at time t; $x_t$ is the input feature; and $y_t$ is the output response. This recursive form achieves a linear computational complexity of $O\left(N\right)$.

2.4.2. SS2D Scanning and Panoramic Perception Verification

To address the difficulty of target recognition in complex interference environments, the engine deploys the 2D Selective Scan (SS2D) mechanism derived from VMamba [9]. Through a multi-directional parallel scanning strategy, this mechanism constructs a logical closed loop from local features to global context perception. In experiments, the SS2D mechanism not only improves the breadth of feature extraction but also supports the complete restoration of the geometric structures of occluded vehicles.

2.4.3. Post-Processing and 3D Reprojection

Finally, to generate the ultimate 3D perception results, the 2D semantic mask predicted by the inference engine is reprojected back into the 3D space. The system executes k-Nearest Neighbor (KNN) smoothing during reprojection to eliminate discrete projection aliasing artifacts and restore fine-grained geometric details from the 2D plane, a practice also validated by established lightweight projection-based frameworks [16].

3. System Implementation

3.1. Hardware and Simulation Environment

To validate the algorithm’s engineering feasibility, a hardware simulation platform was constructed. The core computing unit employs a high-performance GPU to support high-throughput deep learning inference.

• Hardware Configuration: To comprehensively evaluate both theoretical accuracy and edge-deployment feasibility, we employed a heterogeneous computing setup. Model training and the accuracy evaluation of resource-intensive baselines (e.g., PTv3) were conducted on a high-performance workstation equipped with an NVIDIA GeForce RTX 3090 GPU (24 GB VRAM) to prevent out-of-memory bottlenecks. However, to rigorously validate the real-time inference speed (FPS) and memory footprint under strict automotive hardware constraints, all inference profiling was intentionally executed on a mobile-grade platform driven by an AMD Ryzen 7 7840H CPU and an NVIDIA GeForce RTX 4060 Laptop GPU (8 GB VRAM) running CUDA 11.8. This constrained setup closely mimics the computing boundaries of real-world embedded nodes, providing a highly persuasive environment for validating our system’s 83 FPS real-time capability.
• Virtual Sensor Injection: To address the lack of real sensor data in non-vehicle environments, a ROS (Humble Hawksbill) 2 data playback node was developed. It parses the SemanticKITTI dataset into real-time PointCloud2 streams, injecting rain/fog noise via timestamp synchronization to decouple the perception algorithm from physical hardware.

3.2. Software Architecture and Optimization

The system is built on a distributed ROS 2 architecture following the “high cohesion, low coupling” principle.

• Modular Design: The system consists of three independent nodes: a Data Preprocessing Node (Spherical Projection), an Inference Engine Node (TensorRT Acceleration), and a Post-processing Node (KNN Smoothing).
• Communication Optimization: A zero-copy transmission strategy based on shared memory is deployed for intra-node communication, avoiding serialization overhead and reducing end-to-end latency.
• Real-time Optimization: The system employs mixed-precision inference (FP16), reducing memory usage by approximately 40%. Additionally, an asynchronous pipeline overlaps CPU data loading with GPU inference, maximizing hardware saturation.

3.3. Visualization Interface

A visualization terminal based on Qt and Rviz was developed to monitor system performance (Figure 4). It supports synchronized rendering of RGB images and semantic point clouds, where different colors are utilized to distinguish distinct semantic categories, as well as real-time display of FPS and GPU memory usage.

4. Experiments and Results

4.1. Experimental Setup

The experiments were conducted on two distinct datasets to comprehensively evaluate both theoretical accuracy under controlled degradation and practical Sim-to-Real robustness.

First, to benchmark against baseline models, we utilized the large-scale SemanticKITTI dataset. To address the scarcity of real sensor data with dense semantic annotations in adverse weather, a physics-based optical enhancement algorithm was used to inject rain and fog noise into sunny-day point clouds. Crucially, following established physics-based optical degradation models [10,11], the atmospheric attenuation coefficient $\alpha$ is mathematically derived from the Meteorological Optical Range (MOR) using Koschmieder’s law ($\alpha =2.996/MOR$). To ensure a highly realistic simulation, we strictly set the MOR to 30–80 m for fog (aligning exactly with the benchmark settings in [11]) and 100–300 m for rain. This rigorous parameterization ensures that the injected intensity attenuation closely approximates real-world physical sensor degradation.

Second, to explicitly validate the zero-shot generalization capability of our proposed method in authentic severe weather, we employed the real-world SemanticSTF dataset [12]. This dataset provides dense point-wise semantic annotations captured by a real physical sensor under diverse adverse weather conditions, serving as a critical benchmark for evaluating Sim-to-Real robustness without specific retraining.

The evaluation metrics include:

• mIoU (mean Intersection over Union): Evaluates the segmentation accuracy.
• FPS (Frames Per Second): Evaluates the data throughput and real-time inference capability.
• Params and Memory Usage: Evaluates the computational footprint and resource consumption, which is critical for comprehensively comparing lightweight edge-oriented models against state-of-the-art heavy-weight architectures (e.g., PTv3 [8]).

4.2. Quantitative Validation of Physical Noise Distribution

To validate the physical fidelity of the simulated adverse weather data, we quantitatively analyze the LiDAR echo intensity distributions before and after the noise injection process. As illustrated in Figure 5, the original clear-weather point clouds (green) exhibit a relatively uniform and broad intensity distribution, reflecting diverse surface reflectivities under ideal visibility.

Upon applying the physical degradation model (Equation (2)) with a dense fog configuration ($\alpha =0.075$, $MOR\approx 40$ m), the intensity distribution (red) undergoes a significant leftward shift. Importantly, this shift is not a uniform scalar reduction; rather, it reflects a distance-dependent exponential attenuation dictated by the $e^{-\alpha d}$ term. Consequently, the statistical peak is highly concentrated within the low-intensity spectrum (0.1–0.2), accompanied by a pronounced long-tail effect.

This non-linear statistical deformation is a direct mathematical manifestation of the Beer–Lambert law. Since this law is the universally accepted foundational principle governing actual atmospheric scattering and LiDAR signal degradation [11], our simulated intensity distribution inherently captures the core physical dynamics of real-world adverse weather, proving it is not a simplistic empirical noise injection.

4.3. Quantitative Performance Analysis

We compared RainMamba with mainstream projection-based algorithms, including RangeNet++, SqueezeSegV3, PolarNet, Cenet [17], and the recent We will help to re-arrange the reference order after add the citation state-of-the-art attention-based model, PTv3. The quantitative results are presented in

Table 1. Performance comparison of different algorithms under adverse weather scenarios on SemanticKITTI.

Method	Params (M)	GFLOPs	FPS	Car IoU	Road IoU	mIoU (%)
RangeNet++	26.5	185.3	90	82.1	88.5	52.4
SqueezeSegV3	19.2	145.8	110	84.5	89.1	48.6
PolarNet	13.7	112.5	65	89.2	90.5	54.3
Cenet	12.5	98.4	72	90.1	91.0	55.6
PTv3	24.8	210.6	14	96.2	94.8	63.8
RainMamba (Ours)	9.8	89.5	83	92.4	91.8	58.2

.

As indicated in Table 1, the quantitative results validate the architectural efficacy of RainMamba for real-time edge-computing applications. As anticipated, the transformer-based PTv3 attains the highest empirical upper bound in accuracy (63.8% mIoU), largely attributable to its extensive parameterization and the global representation capability of its self-attention mechanism. However, this performance gain incurs a prohibitive computational overhead. When evaluated on a power-constrained edge-simulation platform (RTX 4060 Mobile GPU), PTv3 is bottlenecked by the quadratic time complexity ($O\left(N^2\right)$) inherent to canonical attention matrices, yielding an inference rate of merely 14 FPS. Such latency fundamentally precludes its deployment in time-critical autonomous driving systems, which strictly demand a minimum throughput of 30 FPS.

Conversely, benefiting from the linear algorithmic complexity ($O\left(N\right)$) of the State Space Model and the integration of physics-aware noise filtering, the proposed RainMamba architecture delivers a highly competitive mIoU of 58.2%. This represents a substantial relative improvement of 7.2% over the robust real-time baseline, PolarNet. Most importantly, RainMamba establishes an optimal Pareto frontier between segmentation accuracy and computational efficiency. While trading a 5.6% accuracy margin against the parameter-intensive PTv3, RainMamba achieves a real-time throughput of 83 FPS with only 9.8 M parameters—translating to a nearly 6× acceleration and a 60% reduction in parameter footprint. These findings rigorously demonstrate that RainMamba offers a highly viable and resource-efficient paradigm for robust LiDAR perception in adverse weather conditions.

4.4. Module Effectiveness Ablation Studies

To verify the contribution of each core component in the RainMamba system, we conducted progressive ablation experiments on the SemanticKITTI validation set. Using the version with only the VMamba backbone as the Baseline, we sequentially added the Physical Perception Feature Embedding Module (PE) and the Uncertainty-Weighted Cross-Modal Fusion Module (CMF). The results are shown in

Table 2. Contribution analysis of different modules to system performance.The checkmark (✔) indicates that the corresponding module is utilized in the experimental configuration.

Exp. ID	VMamba Backbone	Physical Module (PE)	Cross-Modal Fusion (CMF)	mIoU (%)	Gain
A	✔			50.8	-
B	✔	✔		54.5	+3.7%
C	✔		✔	55.1	+4.3%
D (Ours)	✔	✔	✔	58.2	+7.4%

.

Analysis of Results:

• Denoising Effect of Physical Perception: Comparing Experiments A and B, the introduction of the PE module improved mIoU by 3.7%. This significant gain is mainly attributed to the effective filtering of vertical rain noise by asymmetric convolutions, resulting in sharper edges for ground and buildings.
• Geometric Completion of Multi-modality: Comparing Experiments A and C, the CMF module brought a 4.3% improvement. In particular, in areas where LiDAR signals are missing due to specular reflection, RGB texture information successfully guided the correct inference of semantic categories.
• Synergistic Effect: The complete model (Experiment D) achieved the best performance of 58.2%, proving that “physical denoising” and “visual completion” are highly complementary when processing adverse weather data.

4.5. Robustness Analysis at Different Distances

LiDAR point cloud sparsity increases non-linearly with distance, making the segmentation of distant small objects extremely difficult. To evaluate the long-range modeling capability of RainMamba, we divided the validation set into three distance ranges: Near (0–15 m), Medium (15–30 m), and Far (30–50 m), and calculated the mIoU for each range.

As shown in

Table 3. Comparison of segmentation accuracy in different distance ranges (mIoU %).

Method	Overall	Near (0–15 m)	Medium (15–30 m)	Far (30–50 m)
RangeNet++	52.4	68.5	45.2	31.6
PolarNet	54.3	70.1	48.5	35.2
RainMamba (Ours)	58.2	74.3	53.8	42.1

, all models performed well in the near-distance region. However, in the far-distance region above 30 m, traditional convolutional networks (e.g., RangeNet++) suffered a sharp performance drop (down to 31.6%) due to limited receptive fields. In contrast, RainMamba, leveraging the global context perception capability of the SS2D mechanism, maintained a high accuracy of 42.1% in the far range, outperforming PolarNet by approximately 6.9 percentage points. This result strongly demonstrates the advantage of State Space Models in capturing long-range dependencies for sparse, distant point clouds, which is critical for ensuring autonomous driving safety at high speeds.

4.6. Class-Wise Performance Analysis

To deeply explore the algorithm’s performance on different categories, Figure 6 illustrates the IoU comparison between RainMamba and baseline models on three typical dynamic obstacles: “Car”, “Pedestrian”, and “Cyclist”.

The analysis indicates that RainMamba achieves significant improvements in identifying small dynamic objects, further validating the effectiveness of the proposed multi-modal fusion strategy.

4.7. Resource Consumption Analysis

To verify the feasibility of deployment on edge devices, we monitored the video memory usage. The peak memory consumption during inference is 846 MB. This low resource footprint confirms the potential for migration to embedded chips such as Jetson Orin.

4.8. Qualitative Analysis

Figure 7 visualizes the semantic segmentation results using spherical projection (Range View). From top to bottom, the rows represent: (a) the input point cloud corrupted by simulated rain and fog noise, which shows significant sparsity and interference; (b) the inference result of the baseline model PolarNet, where structural fragmentation still exists (as highlighted in the red zoom-in box); (c) the result of our RainMamba, which successfully suppresses environmental noise and restores complete geometric contours.

Comparing (c) with (b), it is evident that RainMamba effectively overcomes the geometric fragmentation seen in the baseline prediction, demonstrating superior robustness and structural fidelity in adverse weather conditions.

4.9. Zero-Shot Generalization on Real-World SemanticSTF

While RainMamba is trained exclusively on physically simulated data, validating its generalization capability on real-world sensors is critical for safety-critical deployment. To rigorously assess this, we conduct a zero-shot evaluation on the SemanticSTF dataset [12], a premier benchmark for LiDAR semantic segmentation containing 2076 scans captured by a Velodyne HDL-64E sensor (Velodyne LiDAR, San Jose, CA, USA) under various adverse weather conditions. Unlike synthetic datasets, SemanticSTF provides dense point-wise semantic annotations, enabling a standard and unbiased mIoU calculation.

4.9.1. Robustness Against Optical Degradation

To ensure a fair comparison of architectural robustness, all baseline models, including the heavy-weight PTv3, were retrained on the same physically simulated SemanticKITTI dataset as RainMamba. The quantitative comparison of Sim-to-Real generalization across different weather conditions is presented in

Table 4. Zero-shot Sim-to-Real generalization on SemanticSTF across different adverse weather conditions. All methods are trained on SemanticKITTI with our physics-based augmentation.

Method	Overall mIoU	Weather Breakdown (mIoU)
		Light Fog	Dense Fog	Rain	Snow
RangeNet++	31.5	35.2	24.8	30.1	32.4
PolarNet	34.8	38.5	28.1	33.5	35.2
CENet	36.2	40.1	30.5	35.8	37.1
PTv3	44.8	48.5	45.1	43.2	42.5
RainMamba (Ours)	43.9	47.8	44.2	41.5	40.8

.

As observed, baseline projection-based methods (RangeNet++, PolarNet, CENet) suffer significant performance drops in Dense Fog and Rain, indicating that simple CNN architectures struggle to capture the complex geometry of physically degraded point clouds even when trained on simulated data. In contrast, RainMamba demonstrates remarkable robustness, achieving 44.2% mIoU in Dense Fog. This validates that our training strategy, grounded in the Beer–Lambert law, effectively enables the model to learn the inverse physics of atmospheric scattering, generalizing seamlessly to real-world sensors.

4.9.2. Efficiency and Safety-Critical Analysis

Beyond raw accuracy, practical autonomous driving demands high efficiency and reliability on safety-critical objects.

Table 5. Efficiency analysis and segmentation performance on safety-critical classes on SemanticSTF. RainMamba achieves comparable safety reliability to the heavy-weight PTv3 while maintaining its real-time speed.

Method	Params (M)	Speed (FPS)	Safety-Critical Classes (IoU)
			Car	Person	Road
RangeNet++	26.5	90	45.2	12.5	68.1
PolarNet	13.7	65	52.4	18.2	72.5
CENet	12.5	72	55.6	21.4	74.8
PTv3	24.8	14	72.8	40.5	82.1
RainMamba (Ours)	9.8	83	72.1	39.2	80.5

details the computational complexity and segmentation performance on key categories (Car, Person, Road).

The state-of-the-art transformer, PTv3, serves as an upper-bound reference. Benefiting from the same physical data augmentation, PTv3 achieves the highest overall scores. However, RainMamba delivers highly competitive performance with a marginal gap. Notably, in safety-critical categories like Car (72.1% vs. 72.8%), RainMamba matches the heavy-weight baseline. Crucially, given that RainMamba requires a significantly smaller parameter footprint compared to the heavy-weight PTv3, while reliably operating at real-time speeds, it offers a significantly superior efficiency–robustness trade-off for practical deployment.

5. Conclusions

This paper proposes a lightweight multi-modal semantic segmentation algorithm tailored for adverse weather conditions, combining physical feature injection with State Space Models to enhance perception accuracy and real-time performance in rain and fog scenarios. To address LiDAR signal attenuation and noise issues, a physical feature representation module is designed, utilizing the Beer–Lambert Law to explicitly model environmental noise. Based on the Mamba architecture, the SS2D four-direction scanning mechanism is introduced to replace traditional attention computation, significantly reducing computational complexity while maintaining global context perception.

Experimental results demonstrate that, through physical enhancement and linear inference design, the proposed algorithm achieves a relative mIoU improvement of approximately 7.2% compared to the real-time baseline PolarNet in simulated rain and fog scenarios, achieving real-time inference of 83 FPS and a low video memory usage of 846 MB. Furthermore, zero-shot evaluations on the real-world SemanticSTF dataset confirm the system’s exceptional Sim-to-Real generalization capability. Notably, RainMamba achieves highly competitive segmentation fidelity comparable to the state-of-the-art heavy-weight model PTv3, while requiring a significantly lower parameter footprint. This significantly enhances the robustness of autonomous driving systems in extreme environments and their potential for engineering deployment. Future work will explore the migration and optimization of this algorithm on embedded platforms such as Jetson Orin, as well as its extension to downstream tasks like 3D object detection, to further perfect the all-weather perception framework.