D3SSTrack: Center-Focused State-Space Modeling for Monocular 3D Multi-Object Tracking

Abstract

Monocular 3D multi-object tracking (3D MOT) remains challenging because it is hard to model how objects move over time and to keep correct identities without explicit depth information. In this context, we introduce D3SSTrack, a novel tracking-by-detection framework that integrates Mamba state-space modeling into the 3D tracking pipeline. At its core is the Solid State Track (SST) block, which extends the original Mamba block with dropout regularization and an additional projection layer to improve feature integration before temporal fusion. This design enables efficient modeling of long-range temporal dependencies while maintaining real-time performance at 38 FPS on a single GPU. The proposed tracker combines structured sequence modeling with effective temporal association, improving robustness against occlusions and abrupt motion changes. On the KITTI benchmark, D3SSTrack achieves the best sAMOTA (97.12%) and AMOTA (49.95%) among recent monocular 3D MOT methods, outperforming the best model S3MOT by 0.16% and 0.22%, respectively. Our results highlight the potential of state space-based architectures for real-world monocular 3D MOT applications.

1. Introduction

Monocular 3D multi-object tracking (MOT) is an essential task in computer vision used mainly in autonomous driving, robotics, and intelligent surveillance systems. This cutting-edge area of research focuses on determining the three-dimensional position and orientation of objects over time from a single RGB camera stream. Notable advancements in this field have emerged through the application of deep learning techniques, which have enabled more robust and efficient tracking methods where resource constraints rule out expensive sensor setups like LiDAR or stereo vision.

The core difficulty of monocular 3D MOT stems from the lack of depth cues [1,2,3,4]. Unlike LiDAR systems, which capture spatial distance directly, monocular approaches must infer depth, complicating accurate 3D localization and tracking. Researchers have investigated different approaches over the years that can be broadly classified into separate trackers, which detect objects before tracking them [5,6,7], and joint trackers, which perform detection and tracking simultaneously [8,9,10,11]. Among these, center-based trackers [12,13] have demonstrated superiority through their streamlined workflow of regressing object centers and estimating inter-frame offsets. However, the integration of neural networks raises concerns about computational efficiency versus tracking performance, particularly for resource-constrained platforms such as autonomous vehicles [2,9].

While CNNs remain foundational for 3D MOT, Transformers have emerged as the dominant architecture across a wide range of domains, including computer vision, NLP, and robotics, due to their attention mechanism [14,15,16], with their biggest negative being the quadratic time complexity based on input sequence length. Looking to solve this limitation, advancements have been made in state-space models (SSMs) [17,18,19], introducing a novel selection mechanism that allows for input-dependent and hardware-efficient processing of long sequences in linear time. This linear scaling enables processing of longer temporal windows without memory explosion, which is a critical advantage for autonomous vehicles operating under strict power budgets, where reduced memory bandwidth and FLOPs directly benefit embedded deployment. Building on these advantages, Mamba has gained significant attention in the field of computer vision, becoming the new hot topic. However, adapting it to vision tasks requires architectural modifications since, unlike language modeling, where token order is critical, vision tasks demand global spatial context. Vision Mamba [20] introduces a bidirectional SSM but with processing latency drawbacks, VMamba [21] decomposes 2D images into four primary directions processed independently to capture global understanding, and MambaVision [22] creates a hybrid architecture combining SSM and Transformer blocks that capture both short- and long-range dependencies more effectively.

Existing SSM trackers present in recent works, such as TrackSSM [23] and S3MOT [24], have demonstrated the promise of SSM-based temporal modeling for tracking. They primarily leverage Mamba as a drop-in replacement for attention mechanisms without addressing the inherent tension between temporal smoothing and identity discrimination. We argue that it this necessary but not sufficient; you also need explicit identity supervision and you need to preserve the local spatial detail that pure SSMs degrade. Our method brings temporal reasoning into the field of monocular 3D object tracking, extending the CenterNet [25] framework by replacing its traditional CNN backbone with Mamba’s SSM architecture. Introducing SST, a hybrid block that preserves local spatial fidelity through parallel convolutional gating, and coupling with a supervised contrastive loss that explicitly enforces cross-frame identity consistency—capabilities that remain underexplored in prior SSM-based trackers—we claim to take an important step towards solving the 3D tracking paradigm. The main contributions of the paper are as follows:

• We propose SST (Solid State Track), a dual-branch temporal modeling block that separates state-space dynamics from spatial gating, enabling robust tracking and efficient spatial–temporal modeling from monocular video input.
• We demonstrate that Mamba sequence modeling improves temporal consistency and identity preservation in 3D tracking tasks, especially in cases of occlusion or fast motion, reducing the total number of identity switches by 9.75% and track fragmentation by 6.6% for the Car category on the KITTI dataset.
• We introduce a contrastive embedding loss that explicitly enforces temporal consistency by pulling together embeddings of the same object across frames while pushing apart distractors and background clutter.
• We evaluate our model on the KITTI 3D tracking benchmark, where it achieves state-of-the-art performance in key metrics, outperforming the best model in sAMOTA by 0.16% and AMOTA by 0.22%, validating the effectiveness of our design.

2. Related Work

2.1. 3D Tracking Across Sensing Modalities

Multi-dimensional multi-object detection and tracking has evolved along distinct sensor-driven trajectories, with techniques developed for LiDAR and camera modalities increasingly informing one another. LiDAR-based systems [26,27,28] benefit from direct geometric access through point cloud data, enabling explicit shape reasoning and straightforward depth estimation. CenterPoint [29] demonstrates that center-based representations simplify detection and tracking in point cloud domains, while SimpleTrack [13] decomposes 3D MOT into modular components, identifying motion modeling as a critical performance bottleneck. Sensor fusion approaches such as EagerMOT [30] combine camera and LiDAR strengths but introduce a system complexity, relating to sensor synchronization, calibration, and coupled failure modes, that is prohibitive for resource-constrained deployments. Similarly, stereo vision provides depth through triangulation but requires precise calibration and doubles hardware costs [31]. These considerations motivate high-performance monocular 3D tracking not as a compromise, but as a deliberate choice for cost-sensitive platforms demanding sensor independence. The center-based paradigm, proven effective in CenterTrack’s [12] monocular adaptation, traditionally relies on Kalman filters or constant velocity assumptions that struggle with complex motion patterns. To directly address the motion modeling bottleneck identified in prior LiDAR studies, our work replaces handcrafted motion priors with learned state-space dynamics that capture long-range dependencies in linear time.

2.2. Spatial Object Perception and Tracking

With 2D object detection and segmentation largely solved, research has shifted towards understanding 3D scenes, a considerably more challenging problem. Camera sensors being more cost effective has driven efforts to infer depth directly from 2D RGB images, with the ultimate goal of achieving accurate 3D perception using only monocular input for applications like autonomous driving [32]. Early CNN-based methods directly regressed 3D bounding box parameters from single images [33], while later approaches incorporated auxiliary depth supervision through pseudo-LiDAR representations [34] or depth-aware attention mechanisms [35]. Traditionally, detection and tracking operated as separate modules, first detecting objects, then associating bounding boxes using Kalman filters or learned variants [34,36]. However, this pipeline suffered from error propagation in complex scenarios, motivating joint detection and tracking frameworks that unify both tasks for improved temporal consistency [37,38]. These methods introduced correlation-based motion modeling, instance embeddings via triplet losses [39], and video-level cues such as optical flow warping to reduce inference time [40]. Despite their progress, most remain anchored to Faster R-CNN architectures that depend heavily on spatial bounding box continuity across frames.

2.3. Monocular 3D Multi-Object Tracking

In contrast to LiDAR-based approaches, which directly regress 3D centers and velocities from point cloud bird’s-eye-view representations, monocular methods must infer depth from learned priors and geometric constraints, introducing fundamental ambiguity for object localization in 3D space. Early approaches SORT [41] and DeepSORT [25] often adapted 2D tracking-by-detection frameworks to the 3D domain by extending bounding boxes, but this led to poor motion modeling. Notably, a more hybrid combination between deep learning monocular detection and a probabilistic multi-Bernoulli mixture (PMBM) filter established better tracking [42]. In more recent works, the QD-3DT [43] and DEFT [44] implementations used joint learning to simultaneously estimate the 3D state and reidentification features of objects by leveraging LSTMs, while CenterTrack [12] represented objects by a single point at the center of their bounding box. Further advancement introduced the MoMA-M3T [11] motion Transformer module, which influences temporal and spatial encoders to jointly model object appearance, motion, and interaction over time. As the trend dictates, traditional Kalman filters, LSTMs, and Transformers are replaced with state-space models. MambaTrack [45] replaces handcrafted motion models with an SSM architecture tailored to 2D tracking for accurate nonlinear trajectories, TrackSSM [24] further builds on this with a Flow-SSM module, enhancing tracking robustness in low-frame-rate or fast-motion scenarios, and S3MOT [35] integrates three-way SSM-dependent modules for better global data association and enhanced temporal velocity dependencies. While these methods demonstrate the effectiveness of SSM-based temporal modeling, they share a common limitation: SSM modules are applied without explicit mechanisms to preserve local spatial information, which can be degraded through temporal recursion.

Despite steady progress, monocular 3D multi-object tracking remains a fundamentally in-progress challenge, and models continue to struggle with reliably associating noisy and uncertain predictions over time, especially in dynamic environments with fast-moving objects or occlusions. This underscores the need for lightweight, unified frameworks that can jointly reason in relation to detection and tracking, adapt to complex motion patterns, and robustly fuse spatial–temporal cues.

3. Proposed Approach

In this section, we present D3SSTrack, a novel monocular 3D multi-object tracking architecture that achieves state-of-the-art performance on the KITTI benchmark with respect to the MOTA, AMOTA, and sAMOTA metrics [45]. As illustrated in Figure 1, our method begins with an RGB image processed using a Deep Layer Aggregation (DLA) [46] feature extraction backbone built within the CenterNet [25] detection. Further, we integrate a new temporal modeling module named Solid State Track (SST) (Section 3.2), inspired by recent advances in state-space models (SSMs) and specifically adapted from the Mamba architecture [18]. To reinforce temporal consistency and object identity preservation across frames, we introduce a dedicated contrastive embedding loss (Section 3.3), which encourages embeddings of the same object across time to remain close in feature space.

3.1. Preliminaries

3.1.1. Centers of Objects

Our approach builds upon the CenterNet detector [25], which formulates object detection as keypoint estimation by predicting the center of each object in an image. Starting with an input RGB image $I\in {\mathbb{R}}^{W\times H\times 3}$, it produces a set of class-wise object detections ${\left\{\left(c{p}_i,b{b}_i\right)\right\}}_{i=0}^{N-1}$ for each category $c\in \left\{0,\dots ,C-1\right\}$, where $c{p}_i\in {\mathbb{R}}^2$ denotes the center point of the object and $b{b}_i\in {\mathbb{R}}^2$ represents the object’s bounding box size (height and width). To achieve this, CenterNet outputs a low-resolution heatmap $\widehat{H}\in {\left[0,1\right]}^{{\displaystyle \frac{W}{R}}\times {\displaystyle \frac{H}{R}}\times C}$ and a size map $\widehat{S}\in {\mathbb{R}}^{{\displaystyle \frac{W}{R}}\times {\displaystyle \frac{H}{R}}\times 2}$, with R being the downsampling factor. Each local maximum $\widehat{cp}\in {\mathbb{R}}^2$ in the heatmap $\widehat{H}$ (defined as a peak in a $3\times 3$ neighborhood) corresponds to an object center. The detection confidence is given by $\widehat{\omega }={\widehat{H}}_{\widehat{cp}}$, and the corresponding object size is $\widehat{s}={\widehat{S}}_{\widehat{cp}}$. During training, a set of annotated object centers $\left\{c{p}_0,c{p}_1,\dots \right\}$ is rendered as a Gaussian-shaped heatmap $H\in {\left[0,1\right]}^{{\displaystyle \frac{W}{R}}\times {\displaystyle \frac{H}{R}}\times C}$ using a rendering function per class. Specifically, for a location $q\in {\mathbb{R}}^2$, the rendered value is

$$R_q\left(\left\{c{p}_i\right\}\right)=maxexp\left(-{\displaystyle \frac{{\left(c{p}_i-q\right)}^2}{2{\sigma }_i^2}}\right),$$

where ${\sigma }_i$ is a Gaussian radius depending on the object size [42]. The heatmap is supervised using a focal loss [42,48] defined as

$${\mathcal{L}}_k={\displaystyle \frac{1}{N}}\sum \left\{\begin{array}{cc}{\left(1-{\widehat{H}}_{xyc}\right)}^{\alpha }log\left({\widehat{H}}_{xyc}\right),\hfill & \mathrm{if}{\widehat{H}}_{xyc}=1\hfill \\ {\left(1-H_{xyc}\right)}^{\beta }{\widehat{H}}_{xyc}^{\alpha }log\left(1-{\widehat{H}}_{xyc}\right),\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$$

(1)

where N is the number of objects and $\alpha ,\beta$ are focal loss hyperparameters.

3.1.2. Solid State Blocks

To increase CenterNet’s temporal modeling capabilities, we integrate Mamba sequence modeling blocks into the detection backbone [18]. Starting with a normal one-dimensional sequence $x\left(t\right)$, Mamba evolves a hidden state $h\left(t\right)\in {\mathbb{R}}^N$ using trainable parameters $A\in {\mathbb{R}}^{N\times N},B\in {\mathbb{R}}^{N\times 1}$ and $C\in {\mathbb{R}}^{1\times N}$:

$$\begin{array}{c}h\left(t\right)=\overline{A}h\left(t-1\right)+\overline{B}x\left(t\right)\hfill \\ y\left(t\right)=\overline{C}h\left(t\right)\hfill \end{array},$$

(2)

where $\overline{A},\overline{B},\overline{C}$ are discrete versions of the continuous operators, derived via matrix exponentiation and zero-order hold:

$$\begin{array}{c}\overline{A}=exp\left(\Delta A\right)\hfill \\ \overline{B}={\left(\Delta A\right)}^{-1}\left(exp\left(\Delta A\right)-I\right)·\left(\Delta B\right)\hfill \\ \overline{C}=C\hfill \end{array}.$$

For a full sequence of length T, the model computes the output as a global convolution over time with a learned kernel K, allowing fast global interaction:

$$\begin{array}{c}\overline{K}=\left(C\overline{B},C\overline{AB},\dots ,C{\overline{A}}^{T-1}B\right)\hfill \\ y=x\ast \overline{K}\hfill \end{array},$$

where ∗ denotes 1D convolution. The distinguished feature of Mamba is its selectivity mechanism, where parameters $B,C,\Delta$ are dynamically conditioned on the input, enabling adaptive filtering of temporal signals, focusing on important dynamics in the sequence.

3.2. Solid State Track

Existing SSM-based trackers (MambaTrack [45], TrackSSM [24], S3MOT [49]) share a common architectural pattern: SSM modules are applied as sequential processors of temporal features without explicit mechanisms to preserve fine-grained spatial information. We employ a different perspective by retaining DLA’s multi-scale aggregation [46] through SST (Solid State Track) blocks. Using a parallel dual-branch design, this architectural choice incorporates temporal state-space dynamics at intermediate feature stages, enabling the network to model object persistence. At its core, the SST block is a hybrid token mixing block that leverages both state-space models (SSMs) and parallel convolutional gating to capture rich temporal and spatial information in vision features. Asymmetric DropPath regularization is applied only on one branch ($Z_2$, visible in Figure 2) relying on local features, forcing the SSM branch to learn robust temporal representations capable of maintaining identity through occlusion.

Given a token sequence $X_{in}\in {\mathbb{R}}^{T\times C}$, the SST block performs a projection into two reduced dimensional branches:

$$Z_{1,2}=Linea{r}_{C\to {\displaystyle \frac{C}{2}}}\left(X_{in}\right)\in {\mathbb{R}}^{T\times {\displaystyle \frac{C}{2}}}.$$

Further, we treat each dimension differently: First, $Z_1$ applies a learnable 1D convolution, then an SSM captures long-range dependencies via the dynamics using (2), while $Z_2$ provides a spatially aware gate that helps recover local content that could be degraded due to temporal recursions in the SSM and encourages feature diversity during training by randomly dropping residual paths:

$$\begin{array}{c}{Z}_1=SSM\left(\sigma \left(Conv1D\left(Z_1\right)\right)\right)\hfill \\ Z_2=DropPath\left(Linear\left(\sigma \left(Conv1D\left(Z_2\right)\right)\right)\right)\hfill \end{array},$$

(3)

where $DropPath\left(Z_i\right)$ is the stochastic regularization operator applied to an input tensor and defined as

$$DropPath\left(Z_i\right)\left\{\begin{array}{cc}{\displaystyle \frac{Z_i}{p}},\hfill & \mathrm{with}\mathrm{probability}p\hfill \\ 0,\hfill & \mathrm{with}\mathrm{probability}1-p\hfill \end{array}\right..$$

(4)

This operation randomly drops the layer output during training with probability $1-p$ and scales the remaining paths by ${\displaystyle \frac{1}{p}}$ to preserve the expected value of the output during training. At inference time, $DropPath$ becomes the identity function. Applying it exclusively to the spatial gating branch ($Z_2$) rather than the SSM branch ($Z_1$) reflects the asymmetric roles of temporal and spatial processing in tracking. The SSM branch must reliably propagate object state across frames; dropping it would break temporal continuity. During training, stochastic removal of $Z_2$ simulates occlusion scenarios where local appearance is unavailable, forcing the SSM to maintain identity through state evolution alone.

The output of both branches is concatenated along the channel dimension and projected back to the original embedding size C, incorporating temporal filtering and content-aware gating in a unified representation:

$$X_{out}=Linea{r}_{{\displaystyle \frac{C}{2}}\to C}\left(Concat\left(Z_1,Z_2\right)\right).$$

(5)

After integrating the proposed Solid State Track (SST) blocks into the CenterNet backbone at intermediate depth, each SST block processes normalized activation and is paired with a feedforward MLP head in a residual-style formulation:

$$\begin{array}{c}{\widehat{X}}^{\left(n\right)}=SST\left(LayerNorm\left(X^{(n-1)}\right)\right)+X^{(n-1)}\hfill \\ X^{\left(n\right)}=MLP\left(LayerNorm\left({\widehat{X}}^{\left(n\right)}\right)\right)+{\widehat{X}}^{\left(n\right)}\hfill \end{array}.$$

(6)

3.3. Contrastive Embedding Loss

The SST block produces spatio-temporal embeddings that represent objects over time. These embeddings are intended to be both descriptive of the object’s identity and robust to viewpoint, occlusion, and appearance variations. Traditional losses that preserve identity in multi-object tracking use triplet formulations, which require careful mining of training samples and provide weak learning signals. Classification approaches assume a fixed number of identities, which breaks down in tracking scenarios where objects vary across sequences. We adopt a contrastive formulation that scales naturally with multiple negatives, avoids manual hyperparameter tuning, and handles varying object instances without assuming a predefined identity set.

To enhance the capabilities of the Solid State Track blocks, we incorporate an auxiliary contrastive embedding loss ${\mathcal{L}}_{SST}$ that encourages embeddings of the same object across frames to cluster together while pushing apart embeddings of different objects or background clutter. Our approach builds on InfoNCE [50] but adapts it specifically for tracking; unlike self-supervised methods (MoCo [39], SimCLR [51]) that create positive pairs through image augmentations, we define positives using ground truth annotations where embeddings of the same object at consecutive frames form positive pairs, while negatives come from other objects in the same frame (spatial distractions) and objects from different frames (temporal confusion candidates).

Let $Z_t\in {\mathbb{R}}^C$ be the SST output embedding for an object at time step t, with $Z_t^{+}$ being the corresponding positive embedding (the same object tracked in another frame $t+1$) and $\left\{Z_0^{-},Z_1^{-},\dots ,Z_k^{-}\right\}$ a set of negative embeddings, either from different objects within the same frame or from objects in other frames. Then, the contrastive embedding loss is defined as follows:

$${\mathcal{L}}_{SST}=-log{\displaystyle \frac{exp\left(sim\left(Z_t,Z_t^{+}\right)\right)}{{\sum }_{k}exp\left(sim\left(Z_t,Z_k^{-}\right)\right)}},$$

(7)

where $sim(u,v)$ is the cosine similarity between vectors u and v. This formulation is a form of Noise-Contrastive Estimation [50], used in self-supervised contrastive learning, adapted to a supervised tracking context, where object identities are known.

D3SSTrack incorporates the ${\mathcal{L}}_{SST}$ contrastive loss to encourage temporal consistency in object representations across frames while also disentangling object identities via the SST embedding space. The overall loss is a weighted sum of ${\mathcal{L}}_{SST}$ and the original CenterTrack [12] components: focal loss for heatmap prediction (1), L1 size regression loss, and local offset regression loss.

4. Experiments

4.1. Setups

4.1.1. Dataset

We conduct all experiments using the KITTI 3D tracking benchmark [52], which offers real-world urban driving scenarios captured at 10 frames per second from a moving vehicle equipped with a stereo camera setup. The dataset comprises 21 training sequences and 29 testing sequences, with 3D annotations available for several object classes, including cars, pedestrians, and cyclists. As there is no official evaluation protocol for 3D multi-object tracking on this dataset, we adopt the widely used data split strategy for fair comparison. Specifically, we divide the 21 training sequences into 13 for training and 8 for validation, following an established precedent [43]. All reported results are computed on the validation subset. Our experiments focus on the Car category, consistent with the scope of most existing 3D MOT works.

4.1.2. Metrics

To evaluate the performance of our 3D tracking system, we adopt the standard set of metrics introduced by AB3DMOT [45], which are widely used in the context of the KITTI 3D tracking benchmark. These include the Average Multi-Object Tracking Accuracy (AMOTA), Scaled AMOTA (sAMOTA), and Average Multi-Object Tracking Precision (AMOTP), which collectively provide a robust assessment of tracking performance in 3D space. These metrics account for common tracking challenges such as missed detections, false positives, and identity switches. A true positive match between predicted and ground truth objects is determined based on a center distance threshold of 2 m on the ground plane. In addition to the primary metrics, we also report auxiliary statistics such as the number of identity switches (IDSW), fragmentation events (FRAG), and object tracking ratios like Mostly Tracked (MT), Partially Tracked (PT), Mostly Lost (ML), offering further insight into temporal consistency and object permanence across sequences. This evaluation setup ensures a comprehensive and fair comparison with other state-of-the-art approaches on the KITTI dataset.

4.1.3. Implementation Details

The model was trained using an NVIDIA GeForce RTX 4070 GPU with a batch size of 8. Optimization is performed using the Adam optimizer [49] with an initial learning rate of $1.25·{10}^{-4}$. The learning rate is reduced by a factor of 10 after 60 epochs and training is conducted for a total of 90 epochs. All models are trained using monocular RGB input at a fixed original resolution of $1280\times 384$ with data augmentation techniques such as horizontal flipping and color jittering employed to improve generalization. During training, for good ${\mathcal{L}}_{SST}$ computation, both positive and negative embeddings are ${\ell }_2$-normalized to ensure that cosine similarity operates in a stable range $\left[-1,1\right]$. During inference, we apply non-maximum suppression with a confidence threshold of $0.5$ on the center heatmap outputs to filter low-confidence detections. Only tracklets with confidence scores above $0.4$ are retained for evaluation. Our implementation is developed in PyTorch 1.10.0, and experiments are conducted on a Linux-based workstation equipped with an Intel Core i7 processor. Runtime performance is affected by the number of detected objects per frame and the image resolution, with our system processing KITTI sequences in near real time. All components, including Mamba temporal modeling and 3D lifting, are executed jointly in a single forward pass during inference.

4.2. Main Results

To evaluate the performance of D3SSTrack, we conduct experiments on the KITTI validation set against recent state-of-the-art monocular 3D multi-object tracking approaches following the approach and metrics from AB3DMOT. As shown in

Table 1. Results for D3SSTrack on the KITTI validation set are reported, evaluated using the AB3DMOT protocol [45] for the Car category at an IoU threshold of 0.25. Bold numbers represent the best performance. Following S3MOT [35], we report results grouped by detector: PointRCNN (†) uses LiDAR input; DD3D (‖) uses depth-pre-trained monocular detection; unmarked methods (including D3SSTrack) use standard monocular detection without depth pre-training. For FPS, * results report inference without 3D detection. ↑ indicates bigger is better. Best results are highlighed in bold.

Method	sAMOTA (%) ↑	AMOTA (%) ↑	AMOTP (%) ↑	MOTP (%) ↑	MOTA (%) ↑	FPS ↑
CenterTrack [12] (ECCV 2020)	-	-	-	-	88.7	22
DEFT [44] (CVPRw 2021)	-	-	-	-	88.1	13
ine QD-3DT † [2] (PAMI 2022)	75.94	33.69	61.87	66.61	-	6
AB3DMOT † [45] (ECCVw 2020)	93.28	45.43	77.41	78.43	86.24	207 *
PC3T † [53]	87.08	40.50	75.18	80.84	-	700 *
DeepFusionMOT † [43]	90.51	44.87	79.47	79.83	-	104
AB3DMOT ‖ [45] (ECCVw 2020)	95.78	48.74	76.22	74.56	-	207 *
PC3T ‖ [53]	96.52	49.22	75.23	73.39	-	700 *
DeepFusionMOT ‖ [43]	96.68	49.49	76.39	74.66	88.2	104
S3MOT ‖ [35]	96.96	49.73	77.25	73.25	86.93	31
D3SSTrack (Ours)	97.12	49.95	77.63	77.9	89.06	38

, D3SSTrack achieves the highest sAMOTA (97.12%) and AMOTA (49.95%) scores among all methods, demonstrating superior robustness and tracking accuracy across varying confidence thresholds. In comparison with S3MOT, D3SSTrack improves sAMOTA by 0.16% and AMOTA by 0.22%, confirming the effectiveness of our Mamba-based Solid State Track (SST) module (Section 3.2) in modeling temporal dependencies and refining object permanence across frames. Although DeepFusionMOT and PC3T achieve slightly higher AMOTP (79.47%) and MOTP (80.84%) scores, respectively, D3SSTrack delivers the best balance between tracking consistency and localization precision in a purely monocular setting, attaining a strong AMOTP (77.63%) and MOTP (77.9%) without multimodal or LiDAR-based enhancements.

D3SSTrack achieves 38 FPS on an NVIDIA RTX 4070 GPU with 46 M parameters and peak memory under 12 GB, exceeding the 30 FPS threshold for real-time autonomous driving. This reflects complete end-to-end inference including 3D detection. The high-FPS reports like AB3DMOT (207 FPS) and PC3T (700 FPS) present only tracking runtime, without 3D detections. Among end-to-end methods, D3SSTrack provides a 1.73× speedup over CenterTrack (22 FPS) and 2.92× over Transformer-based DEFT (13 FPS), validating the SSM efficiency advantages for temporal modeling. Compared to S3MOT (31 FPS), the closest SSM-based method, D3SSTrack achieves 1.23× higher throughput while improving sAMOTA (+0.16%) and AMOTA (+0.22%).

Overall, even against methods that utilize different and often more powerful detectors, D3SSTrack consistently establishes a new benchmark across core tracking metrics, showcasing its robustness and scalability in challenging 3D MOT scenarios. Notably, the close performance between D3SSTrack and S3MOT suggests that, while SSM-based designs are effective, there remains untapped potential in further refining sequential modeling blocks to push the boundaries of 3D MOT performance.

4.3. Qualitative Results

Figure 3 presents the monocular 3D MOT results of D3SSTrack on three challenging KITTI sequences, with 3D bounding boxes projected onto the image plane for visual clarity. In the residential scene (Figure 3a), the system precisely detects and separates closely parked vehicles on both sides of the street, maintaining consistent IDs without drift. In the highway scenario (Figure 3b), where vehicles are far apart and frames are sampled at wider intervals, D3SSTrack preserves identity assignments flawlessly, even for fast-moving targets at long range. In the inner-city intersection (Figure 3c), the tracker handles multiple cars entering from different directions under heavy occlusion, keeping their 3D boxes stable and identities intact as they cross paths. These examples highlight our tracker’s resilience in heavy and occluded environments, effectively associating current detections with predicted states to ensure continuous and accurate tracking across diverse urban driving conditions.

5. Ablation Study

5.1. Ablation SST DropPath Probability

To evaluate the impact of stochastic regularization within the SST module, we perform an ablation study varying the DropPath probability parameter $p\in \left\{0.75,0.85,0.95\right\}$. DropPath disables branches of the network during training, encouraging robustness and improved generalization, by randomly zeroing the output branch during training with probability $1-p$, scaling the remaining paths by ${\displaystyle \frac{1}{p}}$ to maintain the expected activation magnitude. We assess how this parameter influences performance on the KITTI validation set across three key metrics: sAMOTA, AMOTA, and AMOTP. Based on the results in

Table 2. Analysis of different DropPath probability values on the KITTI validation set. We evaluate the performance in the same conditions, and the highest values are highlighted in bold. ↑ indicates bigger is better.

Probability	sAMOTA (%) ↑	AMOTA (%) ↑	AMOTP (%) ↑
0.75	93.43	43.18	72.84
0.85	97.12	49.95	77.63
0.95	96.88	49.81	77.98

, we observe the following:

• Lowering DropPath’s keep probability by $25\%$ begins to hurt performance across all metrics. While regularization is stronger, it appears to overregularize the temporal filtering paths, reducing both tracking accuracy and matching quality.
• At $p=0.85$, we obtain the best overall tracking performance, with the highest sAMOTA ($97.12\%$) and AMOTA ($49.95\%$), indicating an optimal balance between information flow and regularization.
• Losing only $0.05\%$ paths yields the highest localization accuracy (AMOTP = $77.98\%$) due to more consistent feature propagation. However, it leads to slightly lower tracking accuracy, suggesting that regularization is not strong enough to prevent overfitting.

These results confirm that DropPath plays a crucial role in stabilizing training and improving generalization, particularly within the sequential structure of the SST. The optimal configuration of $p=0.85$ ensures enough regularization to prevent overfitting while preserving the capacity of the model to maintain accurate and temporally consistent object representations.

5.2. Ablation SST Block Depth

The number of SST blocks integrated into the backbone directly affects the model’s capacity for temporal reasoning and its computational efficiency. Too few blocks may provide insufficient representational capacity for capturing complex temporal dynamics across frames, while excessive depth risks overfitting on limited tracking data. We investigate this trade-off by varying the block configuration across backbone stages, specifically, modulating the depth of Stages 2 and 3, which contain the Mamba-Attention hybrid blocks.

We evaluate three configurations that systematically vary the number of SST blocks while preserving the architectural structure of early convolutional stages. The configurations are (1) shallow, totaling 9 blocks with reduced Mamba-Attention capacity; (2) normal, i.e., the default configuration with 16 blocks; and (3) deep, an expanded configuration with 20 blocks concentrating additional capacity in Stage 2. Stage 2 operates at ${\displaystyle \frac{1}{8}}$ resolution and is particularly critical for D3SSTrack’s dense prediction heads, motivating our focus on varying its depth. All configurations are trained under identical conditions on the KITTI training set and evaluated on the validation set.

The results in

Table 3. Ablation study on SST block depth configuration. Depths indicate blocks per stage $[S_0,S_1,S_2,S_3]$. Stage 2 contains the primary Mamba-Attention blocks for spatio-temporal modeling. ↑ indicates bigger is better. Best results are highlighted in bold.

Config	Depths	sAMOTA (%) ↑	AMOTA (%) ↑	AMOTP (%) ↑	FPS ↑
Shallow	[1, 2, 4, 2]	96.22	48.63	74.41	45
Normal	[1, 3, 8, 4]	97.06	49.5	77.02	38
Deep	[1, 3, 12, 4]	96.89	48.73	77.19	25

reveal a clear trade-off between model depth and tracking performance:

• Shallow [1, 2, 4, 2]: Achieves the highest inference speed (45 FPS) but lowest tracking accuracy, indicating insufficient capacity for temporal modeling when processing concatenated frame features.
• Normal [1, 3, 8, 4]: Delivers the best tracking metrics (sAMOTA = 97.06%, AMOTA = 49.5%) at a practical inference speed (38 FPS), with eight Mamba-Attention blocks providing optimal capacity for KITTI tracking.
• Deep [1, 3, 12, 4]: Shows marginal localization improvement (AMOTP = 77.19%) but significant speed degradation (25 FPS), suggesting diminishing returns due to overfitting on the relatively small KITTI dataset.

The substantial gap between shallow and normal configurations validates that sufficient SST depth is necessary to capture temporal dynamics in multi-object tracking. However, the marginal gains from the deep configuration do not justify its inference cost. The normal configuration provides the optimal balance between representational capacity and computational efficiency for real-time deployment.

5.3. Ablation Contrastive Embedding Loss

The contrastive embedding loss ${\mathcal{L}}_{SST}$ explicitly supervises SST embeddings for identity discrimination. To validate its necessity, we compare the full model against a variant trained without ${\mathcal{L}}_{SST}$ using only detection losses. Both variants share identical architecture (depths [1, 3, 8, 4], DropPath $p=0.85$). The results can be seen in

Table 4. Ablation study on contrastive embedding loss. IDSW = identity switches, Frag = track fragmentations. Lower is better for IDSW and Frag. ↑ indicates bigger is better. ↓ indicates smaller is better. Best results are highlighted in bold.

Model	sAMOTA (%) ↑	AMOTA (%) ↑	AMOTP (%) ↑	IDSW ↓	Frag ↓
w/o ${\mathcal{L}}_{SST}$	95.94	48.47	76.23	46	212
Full (w/ ${\mathcal{L}}_{SST}$)	97.12	49.95	77.63	37	198

.

Removing the contrastive loss degrades all metrics, with identity-sensitive metrics being most affected. Identity switches increase from 37 to 43 (14% increase), and fragmentations rise from 198 to 212 (7.1% increase). This confirms that detection losses alone do not produce identity (discriminative embedding) and the network learns to localize objects but fails to distinguish them across frames. The SST blocks provide temporal modeling capacity, but, without ${\mathcal{L}}_{SST}$, this capacity is not directed toward identity preservation.

This ablation validates the core argument of our approach: temporal modeling (SST) and identity supervision (${\mathcal{L}}_{SST}$) are complementary necessities. The SST architecture enables learning of temporally coherent features; the contrastive loss ensures these features discriminate between object identities. Neither component alone achieves full tracking performance; their combination yields the best results.

6. Limitations and Future Direction

While D3SSTrack achieves state-of-the-art performance on the KITTI 3D tracking benchmark, several limitations warrant explicit discussion. These constraints arise from the dataset characteristics, evaluation scope, and inherent challenges in monocular 3D perception. Acknowledging these limitations generalizes the context of our results and identifies concrete directions for future investigation.

6.1. Dataset and Class Scope

6.1.1. Evaluation Scope

D3SSTrack is evaluated on the KITTI tracking benchmark, a widely adopted dataset representing moderate-density urban and suburban driving scenarios. While KITTI provides a standardized evaluation framework, its relatively small scale (21 sequences, ∼8000 frames) and limited scenario diversity may not fully exercise the temporal modeling capabilities of SST blocks. Contemporary benchmarks such as nuScenes, Waymo, and Argoverse 2 offer a substantially larger scale and diverse geographic contexts that would further validate generalization. Our architectural contributions are dataset independent and should transfer to other benchmarks without modification.

6.1.2. Single-Class Focus

Evaluation is restricted to the Car category due to significant class imbalance in KITTI, where Cars comprise approximately 70% of annotations. Pedestrians and Cyclists present distinct tracking challenges: smaller spatial extent, non-rigid motion, and higher maneuverability. While our single-class focus ensures robust evaluation on well-represented data, multi-class tracking, particularly for vulnerable road users, remains an important direction for future validation.

6.2. Challenging Scenarios

D3SSTrack’s performance under heavy occlusion and high object density is not systematically characterized. While our contrastive embedding loss is designed to maintain identity through temporary occlusion by learning discriminative features, extended occlusion or dense object clustering may exceed the model’s association capacity. In crowded scenarios with 15+ simultaneously visible vehicles, potential failure modes include merged heatmap peaks when objects cluster, association ambiguity during close-proximity motion, and increased identity switches during multi-object occlusion events. KITTI’s moderate density (typically 5–15 vehicles) does not stress-test these scenarios. Systematic evaluation on dense intersections, highway merging zones, and parking lots would characterize scaling behavior.

KITTI sequences were captured under favorable daylight conditions with minimal motion blur or adverse weather. D3SSTrack’s robustness to degraded imaging, including low-light, night-time, motion blur, and other weather scenarios, remains unevaluated. Monocular depth estimation, already ill-posed under ideal conditions, becomes substantially more challenging when appearance cues are degraded. Validation on nuScenes night sequences, BDD100K, and synthetically corrupted images following ImageNet-C protocols would characterize robustness boundaries and define the future direction. If performance degrades, domain adaptation or architectural modifications may be necessary. Notably, the temporal context from SST blocks could potentially mitigate transient image degradation—a hypothesis warranting empirical investigation.

7. Conclusions

In this work, we introduced D3SSTrack, a monocular 3D multi-object tracking framework that incorporates a Mamba SSM block-inspired design that integrates state-space modeling, temporal filtering, and stochastic depth regularization. By enriching the backbone with the state-space tracker, the model learns with temporal consistency and preserves identity embeddings, enabling more robust data association across challenging scenarios. Furthermore, the proposed contrastive embedding loss enhances the temporal coherence in object representations. Experiments on the KITTI dataset demonstrate that D3SSTrack achieves competitive or superior performance across key metrics, validating the advantages of combining Mamba SSM processing with lightweight, yet effective, temporal modeling. In the future, we plan to extend our evaluation to larger and more diverse benchmarks to better assess cross-dataset generalization beyond the limited scene diversity offered by KITTI, to systematically study multi-class 3D MOT in the same framework and to investigate real-time and embedded deployment aspects of D3SSTrack on platforms which are resource constrained. We believe this work underscores the potential of structured sequence modeling in 3D MOT and paves the way for further exploration of efficient state space-based designs in real-world tracking systems.