KalmanFormer: Integrating a Deep Motion Model into SORT for Video Multi-Object Tracking

Abstract

This paper presents the study of integrating a deep motion model into simple online and real-time tracking for video multi-object tracking. The tracking-by-detection paradigm faces significant challenges in handling nonlinear motion and occlusions. Although conventional Kalman-filter-based methods such as the SORT are efficient, they suffer from error accumulation because of their linear motion assumption. We propose KalmanFormer, a novel framework that enhances Kalman-filter-based tracking through adaptive motion modeling for video sequences. KalmanFormer consists of three key components. First, the inner-trajectory motion corrector, built upon the transformer architecture, refines Kalman filter predictions by learning nonlinear residuals from historical trajectories, thereby improving adaptability to complex motion patterns in videos. Second, the cross-trajectory attention module captures interobject motion correlations, significantly boosting object association under occlusions. Third, a pseudo-observation generator is integrated to provide neural-based predictions when detections are missing, stabilizing the Kalman filter update process. To validate our approach, we conduct comprehensive evaluations on the video benchmarks DanceTrack, MOT17, and MOT20 to demonstrate its effectiveness in handling complex motion and occlusion. The experimental results on the DanceTrack, MOT17, and MOT20 benchmarks demonstrate that KalmanFormer achieves competitive performance, with HOTA scores of 66.6 on MOT17 and 63.2 on MOT20, and strong identity preservation, IDF1: 82.0% and 80.1%, respectively.

1. Introduction

Video multi-object tracking (video MOT) is fundamental to dynamic scene understanding, supporting applications such as autonomous navigation, video surveillance, and crowd analysis. Owing to their simplicity and efficiency, the dominant tracking-by-detection paradigm frequently employs Kalman-filter-based models [1,2,3,4,5,6,7]. However, these methods assume linear motion, making them ill suited for real-world scenarios characterized by nonlinear trajectories [8] and frequent occlusions [9]. As a result, issues such as identity switches, track fragmentation, and target loss become prominent, particularly in crowded scenes [10]. Although many improvements have been proposed, state-of-the-art KF-based trackers such as simple online and real-time tracking (SORT) [1] and OC-SORT [9] remain limited by their constant-velocity motion assumption [3]. This makes them inadequate for handling abrupt trajectory changes common in applications such as sports analytics [11] and pedestrian tracking [12]. Nonlinear variants, such as the extended Kalman filter [13] and the unscented Kalman filter [14], attempt to address this gap but are constrained by Gaussian noise assumptions and increased computational cost [15], limiting their real-time applicability [16,17,18,19].

Another challenge arises from the short-term nature of traditional KF updates, which rely solely on the latest observations and neglect long-term motion dependencies. This limitation is particularly detrimental under occlusion, where the absence of historical context degrades tracking robustness [20]. While transformer-based models have succeeded in sequence modeling [21], their high computational overhead hinders real-time deployment [22]. Moreover, KF-based trackers suffer from error accumulation during prolonged occlusions, as predictions proceed without external corrections [9]. Although particle filters offer a more flexible framework, their computational complexity makes them impractical for large-scale tracking [23,24].

To address the aforementioned limitations, we propose KalmanFormer, a hybrid multi-object tracking (MOT) framework that integrates deep motion modeling into a traditional Kalman-filter-based pipeline. Unlike existing approaches that rely solely on linear motion assumptions, KalmanFormer introduces transformer-based components to enhance motion estimation and association under complex dynamics and occlusion. Specifically, the inner-trajectory motion corrector (ITMC) leverages a transformer pretrained on individual trajectories to learn nonlinear motion residuals, mitigating the limitations of the KF’s linear prediction. The cross-trajectory attention module (CTAM) captures interactions across object trajectories, improving the association performance during occlusions. Furthermore, a pseudo-observation generator (POG) synthesizes surrogate observations when detections are missing, preventing drift and error accumulation. In contrast to prior transformer-based MOT methods that process raw image features [25,26,27,28], KalmanFormer operates directly on sequences of bounding boxes via lightweight attention mechanisms. Extensive experiments on the MOT17, MOT20, and DanceTrack benchmarks demonstrate that KalmanFormer achieves state-of-the-art performance while maintaining real-time efficiency, confirming its effectiveness in complex environments.

Table 1. Comparison of different tracking approaches. The table summarizes the key characteristics and limitations of existing tracking methods compared with our proposed KalmanFormer approach.

Method Type	Representative Works	Key Characteristics & Limitations
Traditional TBD (Tracking-by-Detection)	SORT [1], DeepSORT [2], ByteTrack [29], OC-SORT [9]	Efficient computation, real-time performance; poor handling of nonlinear motion, identity switches during occlusions
End-to-End Transformer (Image-to-Track)	TransTrack [25], TrackFormer [21], MOTR [30], MeMOTR [31]	Strong feature representation, joint detection and tracking; high computational overhead, slow inference speed
KalmanFormer (Hybrid TBD + Transformer)	Ours	Combines KF efficiency with transformer’s nonlinear modeling, handles occlusions via pseudo-observations; relies on geometric information

provides a comparative overview of existing tracking methods, highlighting their key characteristics and limitations compared with our proposed KalmanFormer approach.

Our main contributions are summarized as follows:

• We propose KalmanFormer, a hybrid tracking framework that integrates transformer-based modules with the Kalman filter. It incorporates an inner-trajectory motion corrector (ITMC) and a cross-trajectory attention module (CTAM) for modeling nonlinear motion and interobject interactions.
• We introduce a pseudo-observation generator that addresses detection failures by producing reliable surrogate observations, enhancing robustness under occlusions and missing detections.
• We conduct comprehensive experiments on the MOT17, MOT20, and DanceTrack benchmarks. Compared with existing SORT variants, the KalmanFormer achieves state-of-the-art performance and significantly reduces the number of identity switches, particularly in occlusion-heavy scenarios.

2. Related Work

2.1. Single-Object Tracking

Single-object tracking (SOT) involves tracking a target across a video, facing challenges such as occlusion, scale changes, and background clutter. Early methods, such as correlation filters [16,17,23,32,33], achieved success in terms of computational efficiency. Some approaches have extended this to multimodal tracking by combining RGB and thermal data, which has improved performance under low-light conditions [24,27,28,34,35,36,37]. Particle filter methods [19,38,39,40] were introduced to address noisy environments, but they can be slow and not suitable for real-time tracking [18,29,41,42].

With the rise of deep learning, trackers such as GOTURN [43] and SiamFC [44] improved speed and accuracy by using convolutional neural networks (CNNs) for feature extraction [4,5,6,45,46,47]. Some newer methods, such as SCOOT [48], use self-supervised learning and the fusion of multiple data sources to handle more complex situations [7,49,50,51]. Despite their progress, appearance-based trackers are still vulnerable to occlusions and large changes in scale, pose, or lighting. To address this, methods such as ATOM [52] add motion models for better robustness [53,54,55], whereas SiamRPN++ [56] helps with scale changes by combining region proposal networks [57,58,59].

More recently, transformer-based models, such as TransT [60], have been used for SOT [33,39], but they often face the challenge of high computational cost, limiting their use in real-time applications [61,62,63,64,65]. These methods could be improved through model compression techniques, sparse attention mechanisms, or hardware-specific optimizations to reduce their computational overhead while maintaining their strong representational capabilities [66,67,68,69].

2.2. Multi-Object Tracking

Multi-object tracking (MOT) extends SOT by tracking multiple targets simultaneously, which adds complexity, especially in dense or occlusion-heavy scenes. Some MOT trackers, such as CenterTrack [70], focus only on motion and predict positions on the basis of minimal information but struggle to recover occluded targets [71,72,73]. Other methods focus on maintaining object identity during occlusions but rely on learned heuristics [33].

SORT-based methods [1,2,9,29,41,42,45] use Kalman filters for motion prediction and the Hungarian algorithm for data association. These methods are efficient but fail in situations where motion is nonlinear or when objects are occluded for long periods. DeepSORT [2] and ByteTrack [29] improve performance by adding appearance cues but still suffer from the linear motion assumption of Kalman filters. Transformer-based MOT methods, such as TransTrack [25,49,50,51], TrackFormer [21,53,54,55], and MOTR [30,57,58,59], attempt to overcome these problems by using attention mechanisms to model motion and temporal dependencies more effectively [37,61,62,63].

Recent advances in deep metric learning and cross-modal tracking [64,65,66,67] have also contributed to improved performance in challenging scenarios [68,69,71,72,73].

3. Method

We propose KalmanFormer, a hybrid multi-object tracking framework that integrates the strengths of Kalman filters and transformers for robust motion modeling and occlusion handling. As shown in Figure 1, the framework consists of three key modules: the inner-trajectory motion corrector (ITMC), which captures object-specific nonlinear motion patterns from historical trajectories; the cross-trajectory attention module (CTAM), which models interactions between nearby objects to refine predictions in crowded environments; and the pseudo-observation generator (POG), which provides surrogate measurements when detections are missing. These modules are integrated into a unified Kalman-filter-based framework, where ITMC and CTAM refine motion estimates, and the pseudo-observation mechanism ensures continuity during occlusions.

3.1. Notation and State Definition

We adopt an absolute bounding-box state representation for each object at time t as ${\mathbf{x}}_t = {[c_x,c_y,a,r,v_x,v_y,v_a]}^{\top }$, where $(c_x,c_y)$ denotes box center coordinates in pixels, a denotes the box area, r is the aspect ratio, and $(v_x,v_y,v_a)$ represents velocities. The standard constant-velocity Kalman model is used with process noise ${\mathit{\epsilon }}_t\sim \mathcal{N}(0,Q)$ and measurement noise ${\mathit{\eta }}_t\sim \mathcal{N}(0,R)$. Throughout, ${\mathbf{x}}_t^{\mathrm{inner}}$ denotes the ITMC prediction (absolute coordinates), ${\widehat{\mathbf{x}}}_t^{\mathrm{hybrid}}$ denotes the hybrid motion prediction before Kalman correction, and ${\mathbf{x}}_t^{\mathrm{final}}$ denotes the posterior state after the Kalman update. Unless otherwise stated, all coordinates are absolute in the image frame. When we refer to residuals, these are additive residuals in the absolute state space. The same MLP used in Equation (1) is reused after CTAM to map interaction-aware embeddings to the absolute state space.

3.2. Hybrid Motion Prediction and Transformer Integration

Kalman filters assume linear dynamics, which often fail in real-world tracking scenarios involving acceleration or abrupt turns. To overcome this, we introduce a hybrid motion prediction model that augments the Kalman filter’s output with a nonlinear residual learned by a transformer:

$${\widehat{\mathbf{x}}}_t^{\mathrm{hybrid}}=\underset{\mathrm{Linear}\mathrm{prediction}}{\underbrace{\mathbf{F}{\mathbf{x}}_{t-1}}}+\underset{\mathrm{Nonlinear}\mathrm{residual}}{\underbrace{\mathrm{Transformer}\left({\mathbf{x}}_{t-N:t-1}\right)}},$$

(1)

where the residual is applied in absolute state space; process noise is handled by the Kalman filter via ${\mathit{\epsilon }}_t$.

The core innovation of KalmanFormer lies in the deep integration of a transformer architecture within the recursive Bayesian estimation framework of the Kalman filter. This hybrid approach is designed to leverage the transformer’s superior ability to model complex, nonlinear dynamics and long-range dependencies from data while retaining the Kalman filter’s efficiency and probabilistic rigor for state estimation. The integration is realized through three specialized components that augment the classical Kalman filter cycle: a trajectory corrector, an interobject attention module, and a pseudo-observation generator.

3.2.1. Mathematical Formulation of Integration

The standard Kalman filter consists of a prediction step and an update step. KalmanFormer enhances both steps by integrating transformer-based components that learn complex motion patterns and interobject interactions from data. The integration is achieved through two key components:

$${\mathbf{x}}_t^{\mathrm{corrected}}=\underset{\mathrm{KF}\mathrm{prediction}}{\underbrace{\mathbf{F}{\mathbf{x}}_{t-1}}}+\underset{\mathrm{Nonlinear}\mathrm{correction}}{\underbrace{\mathrm{ITMC}\left({\mathbf{x}}_{t-N:t-1}\right)}}$$

(2)

where ITMC processes the trajectory history to predict nonlinear motion patterns and:

$${\mathbf{K}}_t={\mathbf{K}}_t^{\mathrm{base}}·\mathrm{CTAM}\left({\left\{{\mathbf{x}}_t^i\right\}}_{i=1}^M\right)$$

(3)

where CTAM models interobject interactions to adaptively adjust the Kalman gain. The specific architectures and implementations of ITMC and CTAM are detailed in their respective sections.

3.2.2. Limitations of the Integrated Model

Despite its strong performance, the KalmanFormer architecture has several inherent limitations. The interobject attention module has a computational complexity of $O\left(N^2\right)$, where N is the number of objects being tracked, which can become a bottleneck in scenarios with a very large number of objects, potentially hindering real-time performance. As a data-driven model, the performance of the transformer components is heavily dependent on the diversity and representativeness of the training data, and the model may struggle to generalize to motion patterns or object interaction behaviors that are drastically different from those seen during training. While the model is designed for nonlinear movements, its predictive capabilities may degrade when tracking objects with extremely erratic, aperiodic, or random motion profiles that lack any learnable underlying pattern, where the learned priors of the transformer might introduce bias. Additionally, the sequential processing of the transformer modules within the Kalman filter loop introduces additional latency compared with a standard Kalman filter, which must be considered for applications with stringent real-time constraints.

3.3. Inner-Trajectory Motion Corrector (ITMC)

The ITMC explicitly models long-term, nonlinear dynamics via a transformer encoder. As detailed in Figure 2, it takes the past N states of an object as input and encodes them into a latent representation:

$${\mathbf{Z}}_t=\mathrm{TransformerEncoder}\left(\mathrm{Embed}\left({\mathbf{x}}_{t-N:t-1}\right)\right),$$

(4)

where $\mathrm{Embed}(·)$ maps each input state to a high-dimensional vector and ${\mathbf{Z}}_t\in {\mathbb{R}}^{N\times d^{\prime }}$ denotes the encoded sequence. We extract the last token (or a [CLS]-style representation) ${\mathbf{Z}}_t^{\left[\mathrm{last}\right]}\in {\mathbb{R}}^{d^{\prime }}$ and produce a prediction via a multilayer perceptron (MLP):

$${\widehat{\mathbf{x}}}_t^{\mathrm{inner}}=\mathrm{MLP}\left({\mathbf{Z}}_t^{\left[\mathrm{last}\right]}\right).$$

(5)

This nonlinear prediction is fused with the Kalman linear output in a hybrid estimate, improving robustness under curved, abrupt, and stop-and-go motions.

3.4. Cross-Trajectory Attention Module (CTAM)

CTAM models interactions among multiple targets via attention, as illustrated in Figure 3, yielding interaction-aware embeddings for refined prediction and association. Given that M targets at time t with embeddings $\{{\mathbf{Z}}_t^1,\dots ,{\mathbf{Z}}_t^M\}$, each ${\mathbf{Z}}_t^i$ is projected to queries (${\mathbf{Q}}_i$), keys (${\mathbf{K}}_j$), and values (${\mathbf{V}}_j$) via learned weight matrices ${\mathbf{W}}_q,{\mathbf{W}}_k,{\mathbf{W}}_v$, as shown in Equation (6):

$${\mathbf{Q}}_i={\mathbf{W}}_q{\mathbf{Z}}_t^i,{\mathbf{K}}_j={\mathbf{W}}_k{\mathbf{Z}}_t^j,{\mathbf{V}}_j={\mathbf{W}}_v{\mathbf{Z}}_t^j.$$

(6)

The attention score ${\alpha }_{ij}$ between object i and object j is defined as the scaled dot product, as shown in Equation (7):

$${\alpha }_{ij}={\displaystyle \frac{{\mathbf{Q}}_i·{\mathbf{K}}_j^{\top }}{\sqrt{d_a}}},$$

(7)

where $d_a$ is the dimension of the attention space. We adopt a top-K sparse attention mechanism to retain only the most relevant K neighbors for each target, as shown in Equation (8):

$${\mathcal{N}}_K\left(i\right)=\mathrm{TopK}\left({\left\{{\alpha }_{ij}\right\}}_{j=1}^M\right),{\tilde{\alpha }}_{ij}=\left\{\begin{array}{cc}{\alpha }_{ij},\hfill & j\in {\mathcal{N}}_K\left(i\right)\hfill \\ -\infty ,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(8)

The interaction-enhanced embedding is computed as a weighted sum of the values, as shown in Equation (9):

$${\widehat{\mathbf{Z}}}_t^i=\sum _{j\in {\mathcal{N}}_K\left(i\right)}\mathrm{Softmax}\left({\tilde{\alpha }}_{ij}\right)·{\mathbf{V}}_j,$$

(9)

where $\mathrm{Softmax}$ normalizes the attention scores to create a probability distribution over the selected neighbors. This embedding is then mapped back to the absolute state space for hybrid prediction and association.

3.5. Pseudo-Observation Generation

When detections are available, the hybrid prediction serves as the motion prior to the Kalman prediction step. When detections are missing or unreliable, we substitute a pseudo-observation to avoid error accumulation under occlusion, as defined in Equation (10):

$${\mathbf{o}}_t={\widehat{\mathbf{x}}}_t^{\mathrm{hybrid}}$$

(10)

where ${\mathbf{o}}_t$ is the generated pseudo-observation at time t. The Kalman update then proceeds as usual, as shown in Equation (11):

$${\mathbf{x}}_t={\mathbf{x}}_{t|t-1}+{\mathbf{K}}_t\left({\mathbf{o}}_t-\mathbf{H}{\mathbf{x}}_{t|t-1}\right)$$

(11)

where ${\mathbf{K}}_t$ is the Kalman gain at time t and where $\mathbf{H}$ is the observation matrix. The key insight is that our pseudo-observation is not merely a simple linear state extrapolation. Instead, it is the direct output of our full hybrid motion model, as defined in Equation (1). This generated observation is therefore informed by both the object’s own learned nonlinear dynamics via the ITMC and the contextual influence of its neighbors via the CTAM. This makes it a significantly more robust estimate than what a standard Kalman filter could produce on its own. In terms of accuracy under strong occlusions, the performance of the POG degrades gradually. For short-term occlusions (1–5 frames), the recent trajectory provides a strong prior, and the generated pseudo-observation is highly accurate. For longer-term occlusions, while the uncertainty of the prediction naturally increases over time, the learned deep motion model provides a much more physically plausible trajectory than does a constant-velocity assumption. This prevents catastrophic drift and increases the likelihood of correct reidentification when the object’s detection becomes available again. The training targets for ITMC/CTAM are absolute states derived from the annotated boxes; the residual learner minimizes the discrepancy between ${\widehat{\mathbf{x}}}_t^{\mathrm{hybrid}}$ and the ground-truth absolute state. This keeps the notation consistent: ${\mathbf{x}}_t^{\mathrm{inner}}$ and ${\widehat{\mathbf{x}}}_t^{\mathrm{hybrid}}$ are in the same coordinate space, and ${\mathbf{x}}_t^{\mathrm{final}}$ is the KF posterior. We refer to this component as the pseudo-observation generator (POG).

3.6. Complete Tracking Pipeline

To synthesize these components into a complete tracking pipeline, we present Algorithm 1, which integrates the Kalman filter with our transformer-based components. The algorithm begins by predicting the next state for all existing tracklets via the standard Kalman filter prediction step. The ITMC module then refines these predictions by incorporating nonlinear motion patterns learned from historical trajectories, followed by the CTAM module, which further enhances the predictions by modeling interobject interactions. The resulting motion-aware states are matched with incoming detections via the Hungarian algorithm. The matched pairs are updated directly with the detections, whereas the unmatched tracklets are updated via our pseudo-observation mechanism to maintain track continuity during occlusions. Finally, unmatched detections are used to initialize new tracklets. This integrated approach ensures robust tracking even in challenging scenarios with nonlinear motion and frequent occlusions.

Algorithm 1 KalmanFormer Tracking Pipeline

1: Input: Detections ${\mathcal{D}}_t$, existing tracklets ${\mathcal{T}}_{t-1}$ 2: Output: Updated tracklets ${\mathcal{T}}_t$ 3: for each ${\tau }_i\in {\mathcal{T}}_{t-1}$ do 4:  Predict ${\widehat{\tau }}_i^t$ via Kalman filter 5:  Generate pseudo-observation ${\tilde{o}}_i^t$ if needed 6: end for 7: Fuse ${\widehat{\tau }}^t$ and ${\mathcal{D}}_t$ to form candidate set ${\mathcal{X}}^t$ 8: ${\mathcal{X}}^{\prime }\leftarrow$ Inner-trajectory correction via ITMC 9: ${\mathcal{X}}^{″}\leftarrow$ Cross-trajectory refinement via CTAM 10: Compute the similarity matrix S between ${\mathcal{X}}^{″}$ and ${\mathcal{D}}_t$ 11: Apply matching (e.g., Hungarian) on S 12: for each matched pair $({\tau }_i,d_j)$ do 13:  Update ${\tau }_i$ with $d_j$ 14: end for 15: for each unmatched ${\tau }_i$ do 16:  if it is missing for N frames then 17:   Discard 18:  else 19:   Update with pseudo-observation 20:  end if 21: end for 22: for each unmatched $d_j$ do 23:  Initialize a new tracklet 24: end for 25: Return:${\mathcal{T}}_t$

3.7. Training Strategy

We train the inner-trajectory and intertrajectory modules via trajectory ground truths from public MOT datasets. The training process consists of two stages:

• Stage 1 (Pretraining): The inner trajectory module is trained to minimize the L2 distance between the hybrid prediction ${\widehat{\mathbf{x}}}_t^{\mathrm{hybrid}}$ and the ground truth ${\mathbf{x}}_t$ via single-object tracks.
• Stage 2 (Joint training): The intertrajectory attention module is introduced, and the model is fine-tuned on multi-object sequences. The loss includes both trajectory prediction loss and consistency loss for correct interactions.

The total loss consists of:

• Inner trajectory loss:

  $${\mathcal{L}}_{\mathrm{inner}}={∥\mathrm{MLP}\left({\mathbf{x}}_t^{\mathrm{inner}}\right)-{\mathbf{x}}_t∥}_2^2$$

  (12)
• Intertrajectory loss:

  $${\mathcal{L}}_{\mathrm{inter}}={∥\mathrm{CrossAttn}\left({\mathbf{x}}_t^{\mathrm{inner}}\right)-{\mathbf{x}}_t∥}_2^2$$

  (13)
• Hybrid output loss:

  $${\mathcal{L}}_{\mathrm{hybrid}}={∥{\mathbf{x}}_t^{\mathrm{final}}-{\mathbf{x}}_t∥}_2^2$$

  (14)

Additionally, we simulate realistic tracking challenges during training: random frame drop, target switching, and target loss simulation by masking.

3.8. Comparison with Transformer-Based Methods

Several recent MOT methods, such as TransTrack, TrackFormer, and MOTR, have successfully employed transformers. It is important to distinguish KalmanFormer from these approaches.

The primary difference lies in the input domain. Methods such as TransTrack and MOTR are end-to-end models that operate directly on image features extracted by a CNN backbone. While powerful, this makes them computationally expensive and tightly couples the tracking logic to a specific visual detector. In contrast, KalmanFormer operates purely on geometric and kinematic data (i.e., sequences of bounding box coordinates). This is a deliberate design choice that decouples our motion model from the detection process and makes our framework significantly more lightweight and efficient, allowing for easier integration with any detector and better real-time performance. Our strong results on multiple benchmarks demonstrate that a sophisticated motion model can rival or even surpass feature-based methods, especially in motion-centric challenges.

3.9. Computational Complexity and Efficiency

KalmanFormer operates purely on geometry without image feature extraction. ITMC and CTAM adopt sparse top-K attention, yielding $\mathcal{O}(K·M)$ complexity per frame for M tracked objects. This design is suitable for real-time or near real-time settings on modern GPUs/CPUs.

To assess the scalability of KalmanFormer, we analyze its theoretical computational complexity with respect to the number of tracked objects, M. The model’s total complexity per frame is the sum of its main components: the Kalman filter updates, the Inner-Trajectory Motion Corrector (ITMC), and the Cross-Trajectory Attention Module (CTAM).

• Kalman Filter: The prediction and update steps for M objects have a complexity of $O(M·d^2)$, where d is the dimension of the state vector. The calculations for each object are independent, making this component highly parallelizable.
• ITMC: The ITMC processes a fixed-length history of N states for each of the M objects. While the total computational load is linear in M, with a complexity of $O(M·N^2·d^{\prime })$ where $d^{\prime }$ is the embedding dimension, the key architectural advantage is its parallel nature. The processing of each trajectory is independent of the others. On modern hardware with sufficient parallel processing units (e.g., GPUs), the wall-clock time required for this step is largely determined by the computation for a single trajectory, i.e., approximately $O(N^2·d^{\prime })$, and is therefore nearly constant for a wide range of M.
• CTAM: This module is the most critical for scalability. A naive, full self-attention mechanism would incur a prohibitive $O(M^2·d^{\prime })$ complexity, rendering the model impractical for scenes with many objects. However, as described in our method, we employ a top-K sparse attention mechanism. Each object attends only to its K most relevant neighbors, where K is a small constant ($K\ll M$). This crucial design choice reduces the complexity of CTAM to $O(M·K·d^{\prime })$, re-establishing a linear and manageable relationship with the number of objects. This operation is also efficiently implemented via parallel batch matrix multiplication on GPUs.

Consequently, the overall theoretical complexity of KalmanFormer is approximately $O(M·(d^2+N^2·d^{\prime }+K·d^{\prime }))$. The model scales linearly with the number of tracked objects M, and its parallel-friendly design ensures that its practical runtime remains efficient, avoiding the quadratic bottleneck of standard transformer architectures. This makes it a highly scalable solution for real-world MOT tasks.

As established in our complexity analysis, KalmanFormer is designed for efficiency. By operating on low-dimensional geometric data instead of high-dimensional image features, its computational overhead is minimal. The model’s parameters are fixed after offline training, meaning that there is no online adaptation or fine-tuning required during inference. This results in consistent and predictable latency, making it well suited for real-time applications where a stable frame rate is necessary. Our use of sparse attention in the CTAM is critical to ensuring this real-time capability even as the number of objects increases.

3.10. Adaptation to Dynamic Scenes

Handling objects with high speeds and accelerations is a primary motivation for this work. Standard Kalman filters fail in these scenarios because their linear motion model cannot account for abrupt changes. Our ITMC module directly addresses this. By training a transformer on vast amounts of trajectory data, it learns a sophisticated model of object dynamics, including acceleration, deceleration, and sharp turns. When tracking a fast-moving object, the ITMC can predict its future position far more accurately than a linear model, which would significantly lag behind. This inherent ability to model nonlinear dynamics makes KalmanFormer naturally adaptive to challenging, high-speed scenarios without requiring any explicit parameter tuning during runtime.

4. Experiment

4.1. Experiment Settings

We evaluated our method on three well-known benchmark datasets: DanceTrack [10], MOT17 [74], and MOT20 [75]. For evaluation, we use standard tracking metrics: HOTA [76], AssA/DetA [76], IDF1 [74], MOTA [74], and IDS [77]. We train our motion predictor using only the training data from these datasets, without additional samples. For object detection, we use YOLOX [78] detector weights (ByteTrack [29]). The transformer encoder has 6 layers with 8 heads and an embedding dimension of 512. We use Adam [79] with lr = 1 × 10⁻⁴ for 50 epochs and a batch size of 64. The history length is $T=10$, and the masking probability is $p=0.1$.

Unless otherwise specified, we report HOTA and IDF1 as primary metrics and include AssA/DetA, MOTA, and IDS for completeness.

We primarily report results under the private-detection protocol. Unless otherwise noted, baselines are taken from their respective papers with the authors’ recommended detectors. Where detectors differ, we avoid direct claims on detection-driven metrics. We group trackers into appearance-, attention-, correlation/graph-, IoU-, and motion-based families following common practices. Notably, ByteTrack does not exploit appearance features and is treated as IoU-based, and methods such as MOTR/MeMOTR are not categorized as appearance+motion.

4.2. Comparison with State-of-the-Art Methods

We compare KalmanFormer with state-of-the-art trackers on MOT17, DanceTrack and MOT20. As shown in

Table 2. Comparison of KalmanFormer with other methods on the MOT17 test set. The arrows ($\uparrow ,\downarrow$) indicate whether higher or lower values are better. HOTA and MOTA are measured as percentages (%), IDF1 as a percentage (%), and IDS as a count (the lower percentage is better). Red indicates the best-performing method, and blue indicates the second-best.

Tracker	HOTA↑	MOTA↑	IDF1↑	IDS↓
Appearance-based:
DAN [80]	39.3	52.4	49.5	8431
QuasiDense [81]	53.9	68.7	66.3	3378
Semi-TCL [82]	59.8	73.3	73.2	2790
FairMOT [11]	59.3	73.7	72.3	3303
CSTrack [83]	59.3	74.9	72.6	3567
ReMOT [84]	59.7	77.0	72.0	2853
StrongSORT [85]	64.4	79.6	79.5	1194
Attention-based:
CTracker [86]	49.0	66.6	57.4	5529
TransCenter [87]	54.5	73.2	62.2	4614
RelationTrack [88]	61.0	73.8	74.7	1374
TransTrack [25]	54.1	75.2	63.5	3603
MOTRv2 [89]	62.0	78.6	75.0	2619
Graph/Correlation-based:
GSDT [90]	55.2	73.2	66.5	3891
FUFET [91]	57.9	76.2	68.0	3237
CorrTracker [92]	60.7	76.5	73.6	3369
TransMOT [93]	61.7	76.7	75.1	2346
IoU-based:
ByteTrack [29]	63.1	80.3	77.3	2196
BoT-SORT [94]	64.6	80.6	79.5	1257
Motion-based:
TubeTK [95]	48.0	63.0	58.6	4137
CenterTrack [70]	52.2	67.8	64.7	3039
TraDes [96]	52.7	69.1	63.9	3555
MAT [97]	53.8	69.5	63.1	2844
OC-SORT [9]	63.2	78.0	77.5	1950
MotionTrack [98]	65.1	81.1	80.1	1140
KalmanFormer (Ours)	66.6	80.7	82.0	1080

,

Table 3. Tracking results on the DanceTrack set. The arrows ($\uparrow ,\downarrow$) indicate whether higher or lower values are better. All metrics (HOTA, IDF1, MOTA, AssA, and DetA) are measured as percentages (%). Red indicates the best-performing method, and blue indicates the second-best.

Tracker	Appear.	Motion	HOTA↑	IDF1↑	MOTA↑	AssA↑	DetA↑
Appearance-based:
DeepSORT [2]	✓	✓	45.6	47.9	87.8	29.7	71.0
ByteTrack [29]	✓	✓	47.3	52.5	89.5	31.4	71.6
MOTR [30]	✓	✓	54.2	51.5	79.7	40.2	73.5
MeMOTR [31]	✓	✓	68.5	71.2	89.9	58.4	80.5
Attention-based:
TransTrack [25]	✓		45.5	45.2	88.4	27.5	75.9
GTR [99]	✓		48.0	50.3	84.7	31.9	72.5
QuasiDense [81]	✓		54.2	50.4	87.7	36.8	80.1
GHOST [100]	✓		56.7	57.7	91.3	39.8	81.1
Motion-based:
CenterTrack [70]		✓	41.8	35.7	86.8	22.6	78.1
TraDes [96]		✓	43.3	41.2	86.2	25.4	74.5
SORT [1]		✓	47.9	50.8	91.8	31.2	72.0
OC-SORT [9]		✓	54.6	54.6	89.6	40.2	80.4
KalmanFormer (Ours)		✓	51.44	56.74	88.79	36.90	71.92

and

Table 4. Tracking results on the MOT20 test set under private detection protocols. The arrows ($\uparrow ,\downarrow$) indicate whether higher or lower values are better. HOTA and MOTA are measured as percentages (%), IDF1 as a percentage (%), and IDS as a count (the lower percentage is better). Red indicates the best-performing method, and blue indicates the second-best.

Tracker	HOTA↑	MOTA↑	IDF1↑	IDS↓
Appearance-based:
FairMOT [11]	54.6	61.8	67.3	5243
Semi-TCL [82]	55.3	65.2	70.1	4139
CSTrack [83]	54.0	66.6	68.6	3196
UTM [101]	62.5	78.2	76.9	1228
Attention-based:
TransTrack [25]	48.5	65.0	59.4	3608
RelationTrack [88]	56.5	67.2	70.5	4243
MOTR [30]	57.8	73.4	68.6	2439
Graph/Correlation-based:
MLT [102]	43.2	48.9	54.6	2187
GSDT [90]	53.6	67.1	67.5	3131
Motion-based:
MotionTrack [98]	62.8	78.0	76.5	1165
ByteTrack [29]	61.3	77.8	75.2	1223
KalmanFormer (Ours)	63.2	78.5	80.1	1930

, KalmanFormer consistently outperforms the strong motion- and IoU-based methods on MOT17 and MOT20 in HOTA and IDF1. On DanceTrack, which exhibits diverse, nonlinear group motions, KalmanFormer remains competitive via robust geometric modeling. Appearance-based trackers often reduce the IDS by leveraging reidentification; our geometry-only design emphasizes motion fidelity and CTAM-based association, narrowing the IDS gap without image features.

To visually compare the two datasets, we provide a line–plot comparison, as shown in Figure 4.

While quantitative metrics demonstrate the overall performance, a qualitative analysis of specific challenging scenarios can better highlight the contributions of our model’s core components. Our model shows significant advantages in several key scenarios. In dense crowds with frequent interobject occlusions, such as those found in the MOT20 dataset, the CTAM module models interobject interactions to maintain correct track hypotheses when objects reappear after being occluded, significantly reducing identity switches compared with methods that treat objects in isolation. For highly nonlinear motion, particularly evident in the DanceTrack dataset, where dancers make rapid, unpredictable movements, the ITMC module excels by learning a rich motion model that anticipates turns, accelerations, and decelerations, following complex trajectories more accurately than linear models do. During long-term occlusion and detection failures, our POG module generates reliable pseudo-observations on the basis of learned motion patterns, maintaining track continuity and reducing fragmentation.

We also present qualitative visualizations from MOT17 and MOT20 in Figure 5, illustrating the performance under severe occlusions and dense crowds. The figure is placed near the conclusion for better reading flow.

4.3. Ablation Study

To understand the contribution of each component in KalmanFormer, we conduct an ablation study on the MOT17 validation set. As shown in

Table 5. Ablation study on the MOT17 validation set. Bold indicates the best-performing result for each metric.

Components				Metrics
Kalman Filter	POG	ITMC	CTAM	HOTA ↑	IDF1 ↑	MOTA ↑
✓				67.0	77.8	73.5
✓	✓			67.6	79.2	74.6
✓	✓	✓		65.0	78.1	74.2
✓	✓	✓	✓	69.14	82.7	76.2

, each component contributes complementary gains. ITMC emphasizes long-horizon intratrajectory dynamics, whereas CTAM refines short-horizon interobject interactions; using ITMC alone may slightly reduce association stability when interaction cues are absent, whereas combining CTAM consistently improves all metrics, indicating complementary roles. Regarding hyperparameters,

Table 6. Impact of historical trajectory embedding length T. The arrows ($\uparrow ,\downarrow$) indicate whether higher or lower values are better. Bold indicates the parameter value that achieves the best performance.

	IDF1↑
T	5	10	20	30	40	50
KalmanFormer (Ours)	75.2	78.1	82.7	80.4	77.6	77.8

suggests that $T=20$ can be slightly better than $T=10$ for IDF1 (we choose $T=10$ as a latency/accuracy trade-off), and

Table 7. Impact of masked tokens at varying probabilities p. The arrows ($\uparrow ,\downarrow$) indicate whether higher or lower values are better. Bold indicates the parameter value that achieves the best performance.

	IDF1↑
p	0	0.05	0.1	0.2	0.3	0.4
KalmanFormer (Ours)	76.1	78.2	80.4	82.7	73.6	40.2

shows that moderate masking ($p\in [0.1,0.2]$) improves robustness, whereas large masking harms performance.

5. Conclusions

In this study, a hybrid multi-object tracking framework called KalmanFormer was developed by integrating a deep motion model into the SORT tracking pipeline. Critical limitations in existing tracking methods include three key innovations: (1) an inner-trajectory motion corrector (ITMC) that captures nonlinear motion patterns from historical trajectories, (2) a cross-trajectory attention module (CTAM) that models interobject interactions to improve associations under occlusions, and (3) a pseudo-observation generator (POG) that provides reliable neural-based predictions when detections are missing.

Comprehensive experiments were conducted on the MOT17, MOT20, and DanceTrack benchmarks. The results demonstrate that KalmanFormer achieves state-of-the-art performance, with HOTA scores of 66.6% on MOT17 and 63.2% on MOT20. The model is particularly effective at reducing identity switches, achieving an IDF1 score of 82.0% on MOT17, which represents a significant improvement over previous approaches. These results validate the hypothesis that integrating transformer-based components with traditional Kalman filtering can substantially enhance tracking performance in challenging scenarios.

Several important insights were revealed through the experimental results. First, the combination of ITMC and CTAM provides complementary benefits, with ITMC handling nonlinear motion and CTAM addressing interobject interactions. Second, the sparse attention mechanism ensures that the computational complexity scales linearly with the number of objects, making the approach suitable for real-time applications. Third, the pseudo-observation mechanism significantly improves tracking continuity during occlusions, preventing the drift that typically affects Kalman-filter-based trackers.

Despite these advantages, the approach has limitations that warrant further investigation. The primary limitation is the reliance on geometric information alone, which can be insufficient for reidentification after very long occlusions or in cases of extreme visual ambiguity between two objects. An appearance-based method such as StrongSORT may be more effective at reidentifying a person who disappears for hundreds of frames and reappears elsewhere in the scene. Additionally, while the model generalizes well to the tested datasets, its performance on highly specialized domains with unique motion patterns might require domain-specific fine-tuning.

Several promising directions for future work emerge from this research:

• Appearance Integration: A hybrid model that combines this motion-based approach with lightweight appearance features could further improve reidentification after prolonged occlusions. KalmanFormer’s advanced motion predictions can be fused with a lightweight, efficient appearance feature extractor (e.g., MobileNet). The motion model would handle short-term tracking and occlusions, whereas the appearance features would only be invoked for resolving ambiguities or for reidentification after prolonged track loss. This preserves most of the model’s efficiency while adding another layer of robustness.
• Online Adaptation: Mechanisms for online adaptation of the transformer components could be incorporated to enhance performance in domains with evolving motion patterns.
• Multimodal Fusion: The framework could be extended to incorporate multiple sensor modalities (e.g., LiDAR and radar) to improve robustness under adverse environmental conditions.
• Extended Dataset Evaluation: While the current work focuses on the MOT17, MOT20, and DanceTrack datasets, future work should evaluate the approach on a wider range of datasets, such as SportsMOT and BDD100K, to better understand the model’s performance across diverse scenarios, including sports analytics and autonomous driving applications.

In conclusion, the integration of classical filtering techniques with deep learning approaches for multi-object tracking has been demonstrated through the KalmanFormer framework. The system handles motion patterns and occlusions while maintaining computational efficiency, offering potential benefits for various tracking applications.