MRD: A Linear-Complexity Encoder for Real-Time Vehicle Detection

Abstract

Vehicle detection algorithms constitute a fundamental pillar in intelligent driving systems and smart transportation infrastructure. Nevertheless, the inherent complexity and dynamic variability of traffic scenarios present substantial technical barriers to robust vehicle detection. While visual transformer-based detection architectures have demonstrated performance breakthroughs through enhanced perceptual capabilities, establishing themselves as the dominant paradigm in this domain, their practical implementation faces critical challenges due to the quadratic computational complexity inherent in the self-attention mechanism, which imposes prohibitive computational overhead. To address these limitations, this study introduces Mamba RT-DETR (MRD), an optimized architecture featuring three principal innovations: (1) We devise an efficient vehicle detection Mamba (EVDMamba) network that strategically integrates a linear-complexity state space model (SSM) to substantially mitigate computational overhead while preserving feature extraction efficacy. (2) To counteract the constrained receptive fields and suboptimal spatial localization associated with conventional SSM sequence modeling, we implement a multi-branch collaborative learning framework that synergistically optimizes channel dimension processing, thereby augmenting the model’s capacity to capture critical spatial dependencies. (3) Comprehensive evaluations on the BDD100K benchmark demonstrate that MRD architecture achieves a 3.1% enhancement in mean average precision (mAP) relative to state-of-the-art RT-DETR variants, while concurrently reducing parameter count by 55.7%—a dual optimization of accuracy and efficiency.

1. Introduction

In recent years, deep learning has undergone rapid advancements, particularly within the domain of computer vision. A series of groundbreaking architectures—from early convolutional neural networks (CNNs) [1,2,3] to vision transformers (ViTs) [4,5]—have demonstrated remarkable efficacy in addressing complex visual tasks. In object detection, one of the core challenges in downstream applications, two primary methodologies dominate: convolution-based frameworks and transformer structures. While CNNs and their variants excel in balancing accuracy with computational efficiency [6,7], they inherently struggle to capture long-range spatial correlations within images, which limits their contextual understanding capabilities. To mitigate this limitation, researchers have integrated self-attention mechanisms from vision transformers into object detection pipelines, as exemplified by the DETR series of models [8,9]. Despite early concerns over the computational and memory overhead associated with transformer architectures, advancements in hardware infrastructure have alleviated these constraints to some extent. However, recent studies have shifted focus toward re-examining the architectural design of CNNs to further enhance computational efficiency [10,11], driven by growing dissatisfaction with the quadratic complexity inherent to pure transformer structures. This has spurred the development of hybrid architectures that blend convolutional and transformer components—such as MobileViT [12], EdgeViT [13], and EfficientFormer [14]—to optimize performance while reducing computational burden. Notably, hybrid models often face trade-offs between efficiency and accuracy, with many implementations experiencing non-trivial performance degradation. Consequently, achieving an optimal balance between model efficacy and inference speed remains a central challenge in the field. Recent breakthroughs leveraging state space models (SSMs), such as Mamba [15], have introduced promising alternatives by capitalizing on their ability to model long-range dependencies while maintaining linear computational complexity—a critical advantage for large-scale visual tasks.

This paper presents a novel vehicle detection model named Mamba RT-DETR, which applies the visual Mamba model to vehicle detection and designs an efficient vehicle detection feature extraction network called EVDMamba. Unlike classification tasks, object detection often deals with images of higher resolution and pixel density. Existing visual state space models (SSMs), designed mainly for text sequence modeling, are not fully capable of utilizing the channel and spatial information in images. To address this, we introduce a Residual Channel–Spatial Gating (RSCG) module and use a selective 2D scanning module with high-dimensional dot products to enhance channel and spatial feature extraction. Experiments on the BDD100K dataset show that Mamba RT-DETR is highly competitive for vehicle detection. The main contributions of this paper are as follows:

• A Mamba RT-DETR model based on SSM is proposed. By leveraging the linear complexity of SSM, this model improves vehicle detection performance while reducing the number of model parameters.
• An EVDMamba module is designed. It introduces a dilated scanning mechanism and optimizes the module structure to enhance the performance of SSM in image feature extraction.
• The idea of gated aggregation is combined with effective convolutions and residual connections, and an RSCG module is introduced to capture local dependencies and enhance model robustness.

2. Related Work

2.1. Object Detection

Object detection, a key computer vision technology, focuses on identifying and locating objects in images. Current detection architectures mainly fall into two categories: those based on convolutional neural networks [7,16,17] and those using transformer architectures [18,19,20]. In recent years, detection methods have evolved from two-stage detectors (e.g., RCNN series [7,21]) to one-stage detectors (e.g., SSD and YOLO series [22,23,24]). Detection frameworks also split into anchor-based and anchor-free approaches [25]. These advances have driven the field forward, improving both algorithm speed and detection accuracy.

However, conventional object detection algorithms face a common bottleneck: they often require numerous hand-designed components, such as non-maximum suppression (NMS). The introduction of transformer-based detectors [9] changed this paradigm by directly outputting detection results through a sequential prediction mechanism, eliminating traditional prior designs and achieving an end-to-end detection framework. This innovation simplified algorithms and unified the mathematical formulation of detection tasks. Since the debut of DETR, researchers have made systematic improvements to address its slow convergence, low computational efficiency, and poor small object detection performance. Deformable DETR [8] replaced the standard multi-head attention module with a deformable attention mechanism, using dynamic sparse sampling to reduce attention computation complexity from quadratic to linear. This improved training convergence speed by over 10 times without sacrificing detection accuracy. Anchor DETR [26] incorporated anchor priors into the transformer query vector generation process, using spatial position encoding and scale-aware anchor initialization to boost small object detection AP on the COCO dataset by 4.2 percentage points. Efficient DETR [27] focused on computational efficiency, reconstructing the encoder module with a cross-scale feature fusion architecture and using knowledge distillation and hierarchical pruning. This achieved a 2.3× inference speed increase with a 45% reduction in parameters. RT-DETR [28], designed for real-time detection, used lightweight depthwise separable self-attention modules and a dynamic convolution-enhanced detection head, combined with progressive distillation training. It maintained 68.9% mAP, surpassing YOLOv7 in both speed and accuracy by 32% and 2.4 percentage points, respectively. These advancements reflect the growing application of transformers in object detection and push algorithms toward greater efficiency, intelligence, and practicality.

2.2. Visual State Space Models

Research on state space models shows Mamba models have linear computational complexity in long-sequence modeling, solving the efficiency bottleneck of transformers. In vision backbones, Vision Mamba [29], the first pure vision backbone based on a selective state space mechanism, marked Mamba’s entry into vision computing. VMamba [30] enhanced 2D image scanning with a cross-scan module, achieving 87.6% top-1 accuracy on ImageNet. LocalMamba [31] focused on window scanning for vision spatial modeling, improving local dependency capture via dynamic scanning and boosting mIoU on ADE20K over ViT-B by 4.3 points. Inspired by breakthroughs of Mamba models in visual tasks [29,30,31], this paper presents Mamba RT-DETR, a new state space-driven object detection algorithm. It features an innovative global-local state space collaboration mechanism for global receptive field modeling.

3. Materials and Methods

3.1. Overall Architecture

The design architecture of MRD is illustrated in Figure 1. The model consists of a backbone, a neck, and a decoder part. The backbone is composed of a Stem module, an EVDMamba module, and a Visual Feature Fusion module. It starts with the Stem module for downsampling, producing a 2D feature map with a resolution of H⁄4 and W⁄4. Subsequently, all models consist of EVDMamba and Vision Clue Merge modules for further downsampling. In the neck design, the PAFPN method is adopted, using the EVDMamba module instead of cf2 to capture a richer flow of gradient information.

3.2. Mamba

The state space sequence models S4 [32] and Mamba [15] are both built upon the framework of state space models, originating from a continuous dynamical system. This system establishes a mapping between an input sequence $x\left(t\right)\in \mathbb{R}$ and output sequence $y\left(t\right)$ through an implicit latent representation $h\left(t\right)\in {\mathbb{R}}^N$. By leveraging this hidden state mechanism, the architecture simultaneously captures temporal dependencies while maintaining explicit input–output relationships in sequence modeling tasks. The system can be mathematically defined as follows:

$$h^{\prime }\left(t\right)=Ah\left(t\right)+Bx\left(t\right),$$

(1)

$$y\left(t\right)=Ch\left(t\right),$$

(2)

In Equation (1), the state transition matrix $A\in {\mathbb{R}}^{N\times N}$ defines the temporal evolution mechanism of the latent state vector h(t), while the input mapping matrix $B\in {\mathbb{R}}^{N\times 1}$ establishes the coupling between external inputs and the hidden state dynamics. Complementing these components, the observation matrix $C\in {\mathbb{R}}^{N\times 1}$ serves as the projection operator that translates the internal state representation into observable outputs. To adapt this continuous-time formulation for discrete sequence processing in Mamba, the model employs a fixed discretization strategy that converts the continuous parameters A and B into their discrete-time equivalents $\overline{A}$ and $\overline{B}$ through established numerical methods. Among various discretization techniques, the Zero-Order Hold (ZOH) approach is frequently adopted due to its computational efficiency and stability properties. The discretized version can be defined as follows:

$$\overline{A}=\mathrm{e}\mathrm{x}\mathrm{p}(∆A),$$

(3)

$$\overline{B}={\left(∆A\right)}^{-1}\left(\exp \left(∆A\right)-I\right)∆B,$$

(4)

In Equation (4), ∆ functions as a temporal scaling factor that modulates the model’s temporal resolution, while the discrete-time parameters ∆A and ∆B represent the transformed counterparts of their continuous-domain equivalents within a specified time interval. Here, $I$ denotes the identity matrix of appropriate dimensionality. Following this discretization process, the system dynamics are implemented through linear recurrence relations, which can be formally expressed as follows:

$$h^{\prime }\left(t\right)=\overline{A}{h}_{t-1}+\overline{B}{x}_t,$$

(5)

$$y_t=C{h}_t,$$

(6)

The complete sequence transformation can be equivalently expressed through a convolutional representation, mathematically characterized by the following formulation:

$$\overline{K}=\left(C\overline{B},C\overline{AB},\cdots ,C{\overline{A}}^{L-1}\overline{B}\right),$$

(7)

$$y=x\ast \overline{K},$$

(8)

The structured convolutional kernel $K\in {\mathbb{R}}^L$ serves as the fundamental operation for sequence transformation, where $L$ specifies the dimensionality of the input sequence. In the proposed architectural design, the model leverages a convolutional formulation to enable parallelizable training processes while simultaneously adopting a linear recurrence relation to optimize the efficiency of autoregressive inference.

3.3. EVDMamba

In the field of computer vision, both convolutional neural networks and visual transformers typically use convolutional operations for downsampling. However, this traditional convolutional downsampling method can interfere with the selective operations of SS2D [30] at different stages of information flow. Based on previous studies [30,33], this paper introduces a more concise and efficient approach. Firstly, the computation pipeline is simplified by removing normalization layers. Then, the feature map is segmented to make the processing more precise and effective. For redundant feature maps, they are appended to the channel dimension, allowing the full utilization of this feature information. Finally, a 4× compressed pointwise convolution is used for downsampling instead of the conventional stride-2 3 × 3 convolution. This design effectively preserves the feature maps selected by the previous SS2D layer, thus better serving the training of the model.

This paper introduces the ES2D [34] module in the design of EVDMamba, as shown in Figure 2, serving as the core module for MRD and thereby initiating its data processing with a convolutional module. It performs deep feature extraction on the input feature $x^{l-3}\in {\mathbb{R}}^{C\times H\times W}$ and utilizes the SiLU non-linear activation function σ(∙) to achieve non-linear mapping in the feature space. This module refers to the layer normalization mechanism and residual connection design of the transformer module, effectively supporting deep stacked training through cross-layer gradient propagation. In EVDMamba, ES2D employs a selective state space model (S6) to achieve long-range spatial dependency modeling via a hidden state evolution equation, outputting intermediate states $X^{l-1}$. On the other hand, RSCG enhances the feature output from ES2D through parallel pathways that facilitate cross-channel interaction and local spatial focusing by utilizing dynamic channel gating and spatial attention mechanisms.

$$X^{l-2}=\sigma \left(BatchNorm\left(Conv\right(X^{l-3}\left)\right)\right),$$

(9)

$$X^{l-1}=ES2D\left(LayerNorm\left(X^{l-2}\right)\right)+X^{l-2},$$

(10)

$$X^l=RSCG\left(LayerNorm\left(X^{l-1}\right)\right)+X^{l-1},$$

(11)

Recently, some works [33,35,36] have found that gating mechanisms demonstrate strong performance in both natural language processing and visual processing. Therefore, this paper proposes a Residual-based Spatial–Channel Attention Gating module (RSCG), aimed at improving the model’s performance without significantly increasing the computational cost.

As show in Figure 3, the RSCG module employs a dual-branch structure to extract channel and spatial information from the inputs $f_A^{\prime }$ and $f_B^{\prime }$, and it utilizes gating mechanisms to control the flow of information, enabling the model to more effectively transmit important features.

$$Z_{channel}^{l-1}={\mathcal{T}}_{channel}^{l-1}\left(f_A^{\prime }\right),$$

(12)

$$Z_{spatial}^{l-1}={\mathcal{T}}_{spatial}^{l-1}\left(f_B^{\prime }\right),$$

(13)

On the $Z_{spatial}^{l-1}$ branch, depthwise separable convolutions are used as the position embedding model. By cascading residuals, gradients are more effectively backpropagated during training, thereby preserving and utilizing the spatial structural information of the image while maintaining a low computational cost, significantly enhancing performance. The RSCG module uses the non-linear GeLU activation function to control the flow of information across layers. After processing through the spatial branch, local information is fused with the globally refined information that has passed through a linear layer.

$${\gamma }_{spatial}\left(x\right)=\mathsf{\Phi }\left(\mathrm{F}\mathrm{C}\left(GAP\left(x\right)\right)\right),$$

(14)

$$Z_{channel}^l=Z_{channel}^{l-1}⨀ {\gamma }_{spatial}\left(Z_{channel}^{l-1}\right),$$

(15)

The feature information obtained from the dual branches is added to the original input information through residual connections, allowing the RSCG module to capture more global and local features without a significant increase in computational cost.

$$Z_{fusion}^l=Z_{channel}^l⨀Z_{spatial}^l,$$

(16)

$$f_{RSCG}={\mathcal{T}}_{fusion}^l\left(Z_{fusion}^l\right)⨁f_A^{\prime },$$

(17)

4. Results

4.1. Datasets

This study primarily focuses on vehicle detection in complex road conditions under various weather scenarios, hence the selection of the BDD100K dataset. BDD100K is a large-scale and diverse public driving dataset that includes images from different times of day (e.g., day and night), various scenes (highways, urban streets, and residential areas), and different weather conditions (sunny, rainy, and snowy). The BDD100K dataset comprises 69,434 training images, 9919 validation images, and 20,000 test images, with an image resolution of 1280 × 720.

4.2. Training Details and Evaluation Metrics

During the training process in this paper, the model optimizer chosen was AdamW with a momentum parameter of 0.9, a weight decay parameter of 0.001, an initial learning rate of 0.0001, a training epoch setting of 300, and a batch size of 16. The configuration of the experimental host is as shown in

Table 1. The details of the training platform.

Name	Configuration
CPU	Intel Xeon CPU E5-2686 v4
RAM	92G
GPU	NVIDIA RTX 3090 24G
System	Linux
Pytorch	2.0.0
CUDA	11.8

.

To test the performance of the proposed algorithm, mean average precision (mAP@50) and the number of parameters are used as the main evaluation metrics. The mean average precision is calculated as follows:

$$mAP@50={\displaystyle \frac{1}{n}}{\sum }_{i=1}^{n}P\left(R\right)dR,$$

(18)

In the above equation, P and R represent precision and recall, respectively.

$$P={\displaystyle \frac{TP}{TP+FP}},$$

(19)

$$R={\displaystyle \frac{TP}{TP+FN}},$$

(20)

The number of parameters refers to the total number of trainable parameters in the model, which is an important metric for measuring the complexity of deep learning models and evaluating computational requirements.

Floating-point operations (Flops) represent a fundamental measure of the computational complexity of a machine learning (ML) model, particularly in deep learning. It quantifies the total number of floating-point calculations (like additions, subtractions, multiplications, and divisions) required for a single forward pass of the model, typically during inference.

4.3. Performance Comparison on the BDD100K Dataset

Currently, there are many deep learning-based object detection algorithms. To demonstrate the advantages of our proposed algorithm, we compared it with existing classic object detection models on the BDD100K dataset. The compared algorithms include convolutional neural network-based object detection algorithms (YOLOv5m, YOLOv6s, and YOLO8m) and transformer-based object detection algorithms (RT-DETR-r18, RT-DETR-r34, and RT-DETR-r50). The specific results are shown in

Table 2. Performance comparison on the BDD100K dataset.

Model	Parameter	Flops	mAP@50	mAP@95
RT-DETR-r18	19.9	57.0	0.50	0.276
RT-DETR-r34	30.0	87.2	0.511	0.28
RT-DETR-r50	42.0	129.6	0.525	0.29
YOLOv5m	20.1	48.0	0.479	0.265
YOLOv6s	17.2	21.89	0.467	0.259
YOLOv8m	25.8	49.9	0.527	0.292
MRD	9.1	18.5	0.535	0.302

. It can be seen that on the challenging BDD100K dataset, compared to both convolutional neural network-based algorithms and transformer algorithms, our studied algorithm achieves the highest mAP@50, and the number of parameters is significantly lower than other object detection algorithms. The proposed MRD achieves a 1.9% increase in mAP@50 with only 9.1M parameters, realizing a lightweight algorithm without sacrificing high recognition accuracy in complex traffic scenes.

4.4. Performance Comparison on the KITTI Dataset

To further validate the effectiveness of our algorithm, we compare it with a range of detection algorithms on the KITTI dataset, as shown in

Table 3. Performance comparison on the KITTI dataset.

Model	Parameter	Flops	mAP@50	mAP@95
RT-DETR-r18	19.9	57.0	0.743	0.522
YOLOv5m	20.1	48.0	0.695	0.509
YOLOv6s	17.2	21.89	0.689	0.504
YOLOv8m	25.8	49.9	0.731	0.514
MRD	9.1	18.5	0.752	0.530

. It can be seen that our algorithm’s mAP@50 outperforms other detection algorithms on the KITTI dataset. For example, compared to RT-DETR-r18, YOLOv5m, YOLOv6s, and YOLOv8m, our algorithm achieves an increase in mAP@50 by 1.2%, 8.2%, 9.1%, and 2.8%, respectively. Therefore, the overall performance of the proposed algorithm is further validated on the KITTI dataset.

4.5. Ablation Study

Ablation experiments (

Table 4. Ablation experiment.

Backbone		Neck		Metrics
ResNet34	EVDMamba	HybridEncoder	Enhanced FPAFN	Parameters	mAP@50	mAP@50:95
√		√		30.0	0.511	0.28
√			√	22.8	0.508	0.274
	√	√		9.8	0.526	0.287
	√		√	9.1	0.535	0.302

) quantitatively verify the impact of the EVDMamba module and the enhanced FPAFN module on detection accuracy and model lightweighting by comparing the performance of different backbone and neck combinations: The baseline model (ResNet34 + HybridEncoder) has 30.0M parameters and a mAP@50 of 0.511. When only modifying the neck to the enhanced FPAFN, the parameters decrease to 22.8M (a reduction of 24%), but the mAP@50 drops by 0.6% to 0.508, indicating that optimizing the neck alone may lead to accuracy loss due to the lack of global modeling capability in ResNet34. When only replacing the backbone with EVDMamba, the parameters are significantly reduced to 9.8M (a reduction of 67.3%), and the mAP@50 increases by 3.0% to 0.526, demonstrating the effectiveness of EVDMamba’s linear-complexity SSM and RSCG modules in lightweighting and global feature modeling. With full module optimization (EVDMamba + enhanced FPAFN), the parameters are further reduced to 9.1M (a 70% reduction compared to the baseline), the mAP@50 increases to 0.535 (an improvement of 4.7%), and the mAP@50:95 increases to 0.302 (an improvement of 7.9%). This highlights the synergistic benefits of the backbone and neck modules—EVDMamba compensates for the lack of global features, while the enhanced FPAFN optimizes cross-layer feature fusion, ultimately achieving dual optimization of parameters and accuracy.

4.6. Visualization

To clearly demonstrate the performance of the improved algorithm in object detection across diverse traffic scenarios, this study extracts a subset of scene views from the BDD100K dataset based on attribute labels. The selected scenes encompass four typical driving scenario types (urban streets, highways, residential areas, and tunnels), three time periods (daytime, nighttime, and dawn/dusk), and four weather conditions (overcast, clear, rainy, and snowy). Detection results for these extracted scenes are visualized to provide intuitive comparisons. Additionally, output features from the model’s encoding phase are extracted and visualized as heatmaps to analyze spatial attention patterns.

The above visualization results systematically validate the vehicle detection capability of the MRD model across multiple scenarios, complex weather conditions, and varying lighting environments. In typical scenarios such as urban streets, highways, residential areas, and tunnels (Figure 4), the MRD model demonstrates excellent detection performance even under challenging conditions such as heavily occluded vehicles and distant targets. For low-light environments (Figure 5), the model exhibits outstanding nighttime detection ability, successfully identifying black vehicles that are highly blended with the surroundings. As shown in Figure 6, during rainy conditions with interferences such as raindrop occlusion and strong light glare, the model still maintains stable detection outputs, fully demonstrating its strong robustness and anti-interference capabilities.

As show in Figure 7, in the sunny scenario, the model shows strong heat responses to vehicle locations on the road, indicating its ability to effectively capture and accurately locate vehicles under favorable lighting conditions, thus demonstrating excellent detection performance under ideal illumination. In the morning scenario, where lighting conditions are more challenging—such as backlighting and low-angle sunlight—the heatmap indicates a more scattered focus on vehicles. While the model can still recognize general vehicle locations, showing a certain level of robustness, feature extraction and localization accuracy are somewhat compromised. In the underground parking scenario with dim lighting, the model focuses on key vehicle parts such as the body outline, suggesting it possesses some degree of adaptability to low-light environments by relying on critical features for target localization. However, the heatmap quality is noticeably inferior compared to that in the sunny scenario.

In summary, the model maintains detection capability across varying lighting conditions. Nonetheless, there remains room for improvement in scenarios involving complex or low-light illumination. Future work could focus on optimizing illumination normalization or enhancing feature extraction modules for scenes with challenging morning lighting. Additionally, for low-light environments such as underground parking, integrating infrared imaging or employing advanced image enhancement algorithms in preprocessing could further improve detection performance and heatmap quality.

5. Discussion

The MRD model achieves a dual breakthrough in detection accuracy and computational efficiency by introducing a linear-complexity state space model (SSM) and a multi-branch collaborative framework for vehicle detection tasks. On the BDD100K and KITTI datasets, MRD achieves 0.535 (BDD100K) and 0.752 (KITTI) mAP@50 with only 9.1M parameters, representing 3.1% and 1.2% improvements, respectively, compared to mainstream RT-DETR variants, while reducing parameter count by 55.7%. CNNs like YOLOv8m struggle with long-range dependencies, evident in their 1.6% lower mAP@50 than MRD on BDD100K. The multi-branch framework fuses global context (via SSM) and local details (via convolutions), resolving this limitation.

Although MRD demonstrates strong performance, it still faces technical limitations: Firstly, as an emerging architecture, the lightweight deployment strategies for SSM remain immature, particularly lacking hardware-adapted optimization for edge computing devices, which hinders inference efficiency. Secondly, in scenarios involving small objects (pixel occupancy < 1%) and heavy occlusions, detection confidence drops below 0.7, highlighting limitations in modeling low-resolution features and complex occlusion relationships.

Future research will focus on the following three directions: (1) Model Compression and Edge Adaptation: By implementing dynamic sparsification of the SSM state matrix, mixed-precision quantization, and edge-device-specific compilation strategies, we aim to reduce FLOPs to below 10G and achieve end-to-end real-time detection on edge devices. (2) Multimodal Fusion and Scene Generalization: Integrating LiDAR and infrared imagery, we will design cross-modal feature alignment modules and employ generative data augmentation to simulate extreme scenarios. This will enhance robustness for small objects and adverse weather conditions. (3) Architecture Optimization and Mechanism Exploration: Developing hierarchical state space models to reduce neck parameters, combined with neural radiance fields (NeRF) to improve 3D scene understanding, will further refine target localization accuracy. These efforts will advance MRD toward high robustness and low energy consumption, enabling its practical deployment in intelligent driving and smart transportation systems.