OcclusionTrack: Multi-Object Tracking in Dense Scenes

Abstract

This paper presents OcclusionTrack (OCCTrack), a robust multi-object tracker designed to address occlusion challenges in dense scenes. Occlusion remains a critical issue in multi-object tracking; despite significant advancements in current tracking methods, dense scenes and frequent occlusions continue to pose formidable challenges for existing tracking-by-detection trackers. Therefore, four key improvements are integrated into a tracking-by-detection paradigm: (1) a confidence-based Kalman filter (CBKF) that dynamically adapts measurement noise to handle partial occlusions; (2) camera motion compensation (CMC) for inter-frame alignment to stabilize predictions; (3) a depth–cascade-matching (DCM) algorithm that uses relative depth to resolve association ambiguities among overlapping objects; and (4) a CMC-detection-based trajectory Re-activate method to recover and correct tracks after complete occlusion. Despite relying solely on IoU matching, OCCTrack achieves highly competitive performance on MOT17 (HOTA 64.9, MOTA 80.9, IDF1 79.7), MOT20 (HOTA 63.2, MOTA 76.9, IDF1 77.5), and DanceTrack (HOTA 57.5, MOTA 91.4, IDF1 58.4). The primary contribution of this work lies in the cohesive integration of these modules into a unified, real-time pipeline that systematically mitigates both partial and complete occlusion effects, offering a practical and reproducible framework for complex real-world tracking scenarios.

1. Introduction

The multi-object tracking (MOT) task aims to obtain the spatial–temporal trajectories of multiple objects in videos. The tracking targets in multi-object tracking encompass various types: pedestrians, vehicles, animals, and more. Currently, there are three mainstream paradigms for multi-object tracking: (1) tracking-by-detection (TBD); (2) joint detection and tracking (JDT); and (3) transformer-based tracking. Through experiments, it has been demonstrated that TBD is currently the strongest paradigm in MOT [1,2,3].

TBD includes two main stages: detecting and tracking. The working process of TBD is to input video frames into the detector to generate detection bounding boxes, then run the tracking algorithm to obtain the final tracking results. The tracking stage has two main parts. (1) The motion model, which is used for predicting the states of trajectories at the current frame to generate bounding boxes. Most TBD paradigms employ the Kalman filter [4] as the motion model. (2) Associating new detections with the current set of tracks. The associating task is mainly to compute pairwise costs between trajectories and detections, then solve the cost matrix using the Hungarian algorithm to obtain the final matching results. There are two mainstream methods for cost calculation. (a) Localizing the objects through the intersection-over-union (IoU) between the predicted bounding boxes and the detection bounding boxes [1,3]. (b) The appearance model [2,5,6], which compute the similarity between their appearance features. Both methods are quantified into distances and used for solving the association task as a global assignment problem.

While simple and effective, mainstream TBD methods like SORT [1], DeepSORT [2], and ByteTrack [3] exhibit significant limitations in dense and occluded scenes, primarily stemming from their core components:

(1)

Vulnerability to Partial Occlusions: Occlusions pose a major challenge. Partial occlusions can degrade detection quality, but the KF’s update stage typically uses a uniform of measurement noise, failing to account for this variability.

(2)

Sensitivity to Camera Motion: The performance of IoU-based tracking is highly dependent on the accuracy of the Kalman filter’s predicted boxes. Camera motion can cause coordinate shifts between frames, leading to prediction offsets and subsequent association failures.

(3)

Ineffective Association under Occlusion: During the association stage, occlusion poses significant challenges to matching methods. Because occlusion arises from overlapping targets, the positions and sizes of the bounding boxes for these overlapping targets become extremely close, and conventional matching algorithms fail to deliver satisfactory performance.

(4)

Vulnerability to Complete Occlusions: Complete occlusions cause tracks to be lost, meaning that the detector cannot detect that trajectory in the image, during which the KF continues to predict without correction, accumulating significant error and hindering the target’s tracking performance after rematching.

To address these interconnected challenges, we propose a comprehensive framework that enhances the TBD pipeline at multiple stages:

• We introduce a confidence-based Kalman filter (CBKF) that dynamically adjusts the measurement noise based on detection confidence, improving robustness to partial occlusions.
• We incorporate camera motion compensation (CMC) [5] to align frames, thereby rectifying the prediction offsets in the KF and boosting the reliability of IoU-based association.
• We integrate depth–cascade-matching (DCM) [7] into the association phase. By leveraging relative depth information to partition targets and perform matching within the same depth level, DCM effectively resolves ambiguities caused by inter-object occlusions.
• For tracks recovering from complete occlusion, we employ the CMC-detection-based Re-activate method to correct accumulated errors in the track state and KF parameters, which ensures that the target can be successfully associated in subsequent frames.

2. Related Works

The key characteristics and limitations of different classic tracking methods are shown in Table 1.

Table 1. The key characteristics and limitations of different tracking methods.

Method	Key Characteristics and Limitations
SORT	SORT is the pioneer of the tracking-by-detection model, providing a powerful baseline model for subsequent research. However, it did not incorporate specific modules designed to address occlusion.
DeepSORT	DeepSORT incorporates an appearance feature extraction module on top of SORT, introducing appearance feature similarity into the association stage. However, in scenes with severe occlusions, the appearance feature extraction module may fail to extract accurate appearance features, potentially introducing additional errors into the association stage.
ByteTrack	ByteTrack employs confidence scores during the association stage: manually set confidence thresholds categorize detection results into high-score and low-score detections, with high-score detections used for the first association and low-score detections for the second association. This two-stage association method effectively reduces the interference of occlusions during the association stage. However, ByteTrack does not address camera motion or the impact of occlusions on other components.

Motion model. In recent years, motion models for multi-object tracking TBD paradigms have primarily fallen into two categories: (1) the Kalman filter [4], based on constant-velocity assumptions [1,2,3,5,6,7,8,9]; and (2) motion prediction grounded in deep learning [10,11]. The traditional Kalman filter offers fast computation and performs well in pedestrian tracking, while deep learning-based motion prediction excels at handling complex non-linear motions. Concurrently, to address camera motion in complex scenes, numerous researchers employ image registration for frame alignment—specifically through methods like maximizing the enhanced correlation coefficient (ECC) or feature-matching techniques such as ORB.

Re-identification. The appearance-based re-identification (Re-ID) technique has also garnered significant attention [2,6,8]. By training an appearance feature extraction network to capture the appearance features of objects within detection bounding boxes, the system calculates the similarities between extracted features and historical appearance features from trajectories. These similarities are then integrated into a cost matrix to achieve superior association performance. However, in scenarios with severe occlusions, the detector performance degrades, causing detection bounding box shifts that make it difficult for the appearance feature extraction network to accurately capture features. Additionally, since the detector and appearance feature extraction network are two separate networks, the computational cost is high. Therefore, researchers have proposed the joint-detection-embedding (JDE) paradigm, which integrates the detector and appearance feature extraction network to reduce computational cost.

Association. Data association is a crucial component of multi-object tracking, and effective data association significantly enhances tracker performance. DeepSORT [2] integrates appearance information into the cost matrix, employing cascaded matching that prioritizes newer trajectories to prevent occluded targets from being incorrectly associated with new detections. ByteTrack [3] divides the detection set into high-score detections and low-score detections based on confidence scores and performs matching from highest to lowest. SparseTrack [7] designed the DCM algorithm, which divides both the trajectory set and the detection set into an equal number of subsets based on depth information. Matching is then performed sequentially. BoTSORT [5] uses the exponential moving average to dynamically update visual characteristics. Most algorithms using appearance features set a hyperparameter to adjust the weighting between motion features and appearance features. Most tracking-by-detection trackers use the Hungarian algorithm to solve the cost matrix to obtain the final matching results.

3. Methods

We present four improvements in this paper, and we use ByteTrack [3] as the baseline model. Our tracker only uses IoU matching for data association. The framework of our tracker is shown in Figure 1.

Figure 1. The overall framework of our tracker. CMC is included in the CBKF prediction in this figure.

3.1. Confidence-Based Kalman Filter

In recent years, there have been two mainstream motion models in tracking-by-detection: (1) the Kalman filter [4]; and (2) the deep learning motion model. The core task of the Kalman filter is state estimation. Given past state information of a target, it predicts the target’s potential location at the next frame based on a constant-velocity kinematic model. This predicted state is then refined using the current detection bounding box to produce the final result. Deep learning models are primarily data-driven, relying on large training datasets and appropriate training methods. They often demonstrate strong performance in scenarios with highly complex motion patterns. However, in multi-pedestrian tracking scenarios, target motion patterns tend to be relatively simple, and the frame intervals are short, and pedestrian movements over short durations generally align well with the Kalman filter’s prior assumptions about target motion patterns. Consequently, the Kalman filter can demonstrate sufficiently robust performance in such scenarios. Deep learning models, due to their data-driven nature, require large amounts of high-quality training samples for pedestrian tracking scenarios and carry the risk of overfitting. Furthermore, compared to the Kalman filter, deep learning models have a vast number of parameters, increasing computational costs. Therefore, we selected the Kalman filter as the motion model.

The Kalman filter [4] is employed in multi-object tracking for motion state prediction and information fusion after matching. In recent years, the Kalman filter state vector has been defined as an eight-dimensional vector $x=[x_c,y_c,s,a,\dot{x_c},\dot{y_c},\dot{s}{]}^{\mathrm{\top }}$. However, experiments of BoTSORT [5] demonstrated that directly predicting the bounding box width yields better results than predicting the aspect ratio. Therefore, we adopt the eight-dimensional state vector from BoTSORT as detailed in Equation (1). The measurement vector is defined in Equation (2).

$$x_k=\left[x_c\right(k),y_c(k),w(k),h(k),{\dot{x}}_c(k),{\dot{y}}_c(k),\dot{w}(k),\dot{h}(k){]}^{\mathrm{\top }}$$

(1)

$$z_k={\left[\begin{array}{c}{z}_{x_c}\left(k\right),z_{y_c}\left(k\right),z_w\left(k\right),z_h\left(k\right)\end{array}\right]}^{\mathrm{\top }}$$

(2)

Here, $x_c\left(k\right)$ and $y_c\left(k\right)$ denote the horizontal and vertical coordinates of the center point of the bounding box, while $w\left(k\right)$ and $h\left(k\right)$ represent the width and height of the bounding box.

The Kalman filter performs two functions: prediction in Equations (3) and (4) and update in Equations (5)–(7). $\mathit{F}$ and $\mathit{H}$ are in Equations (8) and (9). In multi-object tracking, prediction generates the predicted state of trajectories, while update combines the detection boxes and predicted boxes to produce the final tracking results.

$${\widehat{x}}_{k|k-1}=F_k{\widehat{x}}_{k-1}+B_k{u}_k$$

(3)

$$P_{k|k-1}=F_k{P}_{k-1|k-1}{F}_k^T+Q_k$$

(4)

$$K_k=P_{k|k-1}{H}_k^T{\left(H_k{P}_{k|k-1}{H}_k^T+R_k\right)}^{-1}$$

(5)

$${\widehat{x}}_{k|k}={\widehat{x}}_{k|k-1}+K_k\left(z_k-H_k{\widehat{x}}_{k|k-1}\right)$$

(6)

$$P_{k|k}=(I-K_k{H}_k)P_{k|k-1}$$

(7)

$$\mathit{F}=\left[\begin{array}{cccccccc}1& 0& 0& 0& 1& 0& 0& 0\\ 0& 1& 0& 0& 0& 1& 0& 0\\ 0& 0& 1& 0& 0& 0& 1& 0\\ 0& 0& 0& 1& 0& 0& 0& 1\\ 0& 0& 0& 0& 1& 0& 0& 0\\ 0& 0& 0& 0& 0& 1& 0& 0\\ 0& 0& 0& 0& 0& 0& 1& 0\\ 0& 0& 0& 0& 0& 0& 0& 1\end{array}\right]$$

(8)

$$\mathit{H}=\left[\begin{array}{cccccccc}1& 0& 0& 0& 0& 0& 0& 0\\ 0& 1& 0& 0& 0& 0& 0& 0\\ 0& 0& 1& 0& 0& 0& 0& 0\\ 0& 0& 0& 1& 0& 0& 0& 0\end{array}\right]$$

(9)

However, through experiments and some studies, we have discovered that there is a pressing issue that needs to be addressed for the traditional Kalman filter. In dense scenes, occlusions frequently occur, posing significant challenges to detector performance. Severe occlusions will degrade detection results. However, in such cases, the traditional Kalman filter still calculates measurement noise using the same method regardless of detection quality, causing trackers to experience substantial performance degradation when confronting occlusion challenges. Therefore, based on detection confidence, an intuitive metric for quantifying the quality of detection results, we designed a non-linearly varying coefficient to scale measurement noise $R\left(\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{f}\right)$ in Equation (10).

$$R\left(\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{f}\right)=R\cdot e^{-\beta \cdot (\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{f}-c_0)}$$

(10)

Among these, $\beta$ is an adjustable hyperparameter that controls the rate of change in the scaling factor. $c_0$ is the confidence threshold, representing the boundary line determining whether a detection result is considered reliable. According to the properties of exponential functions, within the interval ($c_0$, 1], as the confidence level increases, the function curve flattens out, and the scaling factor ultimately approaches zero. Within the interval [0, $c_0$], as the confidence level decreases, the function curve becomes steeper, and the scaling factor tends toward a very large value. The curve is shown in Figure 2. This indicates that when confidence is high, the tracker fully trusts the detection result, while when confidence is low, it places greater trust in the predicted state. The improved Kalman filter update functions are in Equations (11)–(13).

Figure 2. The figure above shows the confidence-based scaling factor $e^{ - \beta (\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{f} - c_0)}$ function curve in the range from 0 to 1. The figure below shows the confidence-based scaling factor $e^{ - \beta (\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{f} - c_0)}$ function curve in the range from 0.7 to 1. The horizontal axis represents the confidence score.

Furthermore, to evaluate the performance of different types of scaling factors, we conduct a comparative experiment on the MOT17 validation set. The results are shown in Table 2.

$$K_k=P_{k|k-1}{H}_k^T{\left(H_k{P}_{k|k-1}{H}_k^T+R\left(\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{f}\right)\right)}^{-1}$$

(11)

$${\widehat{x}}_{k|k}={\widehat{x}}_{k|k-1}+K_k(z_k-H_k{\widehat{x}}_{k|k-1})$$

(12)

$$P_{k|k}=(I-K_k{H}_k)P_{k|k-1}$$

(13)

Table 2. The impact of different scale factors on MOT17.

Methods	HOTA	MOTA	IDF1
Baseline	67.9	77.8	79.9
Linear	68.0	77.8	79.8
Sigmoid	67.6	77.8	79.4
CBKF	68.1	77.8	80.3

3.2. Camera Motion Compensation

Tracking-by-detection trackers that rely solely on IoU distance place significant emphasis on the quality of prediction boxes. However, in complex scenarios where the camera position is not fixed, camera motion causes shifts in the reference coordinate system between consecutive frames. In such cases, the Kalman filter [4] continues to output predicted states based on the previous frame’s coordinate system, resulting in prediction boxes exhibiting substantial bias. Therefore, in recent years, some trackers have begun to apply camera motion compensation to achieve inter-frame coordinate system registration [5,6]. We introduced the global motion compensation (GMC) technique [5] from OpenCV [12]. This registration method is suitable for revealing the background motion of adjacent frames. First, image key-points are extracted [13], followed by feature point tracking using sparse optical flow. Then, based on the corresponding feature points, a system of equations is constructed. The RANSAC algorithm [14] is used to solve this system of equations and obtain the affine matrix in Equation (14).

$$A=\left[\begin{array}{c}{s}_{2x2}|z_{2x1}\end{array}\right]=\left[\begin{array}{c}{a}_{11}{a}_{12}{a}_{13}\\ a_{21}{a}_{22}{a}_{23}\end{array}\right]$$

(14)

$s\in {\mathbb{R}}^{2\times 2}$ represents the rotation and scaling components within the affine matrix, while T represents the translation component within the affine matrix. Since the predicted state x is an eight-dimensional vector, each dimension requires transformation. Therefore, the $S$ $\in {\mathbb{R}}^{8\times 8}$ and $\mathrm{Z}$ $\in {\mathbb{R}}^8$ that satisfy the tracking requirements are as follows in Equation (15). Then, apply $S$ and $Z$ to transform the predicted state and the predicted covariance matrix in Equations (16) and (17).

$$S=\left[\begin{array}{cccc}s& 0& 0& 0\\ 0& s& 0& 0\\ 0& 0& s& 0\\ 0& 0& 0& s\end{array}\right], Z=\left[\begin{array}{c}{a}_{13}\\ a_{23}\\ 0\\ 0\\ ⋮\\ 0\end{array}\right]$$

(15)

$${\widehat{x}}_k=S{\widehat{x}}_{k-1}+\mathrm{Z}$$

(16)

$$P_k=S{P}_{k|k-1}\mathrm{S}$$

(17)

where ${\widehat{x}}_{k - 1}$ represents the predicted state before transformation. $P_{k|k - 1}$ represents the predicted covariance matrix before transformation. ${\widehat{x}}_k$ represents the predicted state after transformation. $P_k$ represents the predicted state after transformation. After CMC processing, the predicted state is based on the current frame coordinate system. The results are shown in Figure 3.

Figure 3. The left image is the predicted state before CMC. The right image is the predicted state after CMC.

3.3. Depth–Cascade-Matching

Occlusion refers to the overlapping of multiple objects. In dense scenes, due to frequent occlusions, multiple targets are highly prone to overlapping. In such cases, both their prediction boxes and detection boxes become extremely close, making conventional matching methods susceptible to causing ID switches. Therefore, we introduced the depth–cascade-matching (DCM) algorithm [7], shown in Figure 4, to address occlusion issues during the matching phase. The DCM algorithm consists of two main stages: calculating the pseudo-depth value and the cascade-matching bases on the obtained pseudo-depth value.

Figure 4. The framework of DCM is shown in this figure.

For calculating the pseudo-depth value, we first obtain the connection point $A$ between the target and the ground in three-dimensional space. This connection point $A$is then projected onto the camera plane, with the projection point denoted as $B$.

Secondly, a perpendicular line is drawn from point $B$ to the bottom edge of the image, with the intersection point denoted as $C$. The Euclidean distance between $B$ and $C$ is calculated to serve as the pseudo-depth value. The pseudo-depth value, in a certain sense, represents the occlusion relationships between objects. The computational process is shown in Figure 5.

Figure 5. The illustration of depth value.

After obtaining the pseudo-depth value, the tracker performs cascade matching. For the detection set, we divided the depth values of detections into k intervals between the maximum depth value${ S}_{max}$ and the minimum depth value $S_{min}$ using a linear method. Based on the depth values, the detection results are assigned to corresponding intervals, forming k subsets $\{D_0,D_1,\dots ,D_{k-1}\}$. k signifies the highest depth level. Then use the same operation to split the track set into k subsets $\left\{T_0,T_1,\dots ,T_{k-1}\right\}.$

Trajectories and detections at the same depth level undergo matching. The matching process proceeds from lower to higher depth levels. Our tracker only uses IoU distance to match, solving the cost matrix through the Hungarian algorithm. Trajectories and detections that remain unmatched at the current level are passed to the next level for matching after the data association of each depth level.

3.4. CMC-Detection-Based Re-Activate Method

Severe occlusions can cause long-term target lost. During the period of the long-term target loss, both the Kalman filter and deep learning models predict motion state based on the target’s historical trajectory. However, according to the analysis of OC-SORT [9], this prediction method will lead to substantial error accumulation in the trajectory state and the Kalman filter’s parameter during the lost period, meaning that even if the target is rematched, it remains at risk of being lost again in subsequent phases. The accumulation of error mainly stems from the absence of detection in the lost period. Therefore, we propose a Re-activate method based on CMC and detection for generating virtual detections to address the issue of error accumulation. This re-activate method is applied after the trajectory has been rematched.

Firstly, our tracker records the information of the last matched detection before becoming lost, including bounding boxes $z_L$, detection confidence $c_L$, and its frame $T_L$. When a trajectory is marked as lost, our tracker records its trajectory state $x_L$. Meanwhile, the tracker maintains a CMC list whose length matches the maximum age of the trace to contain the CMC affine matrix at the beginning of tracking. When the trajectory is rematched, the newly matched detection bounding box, confidence score, and frame are denoted as $z_{new}$, $c_{new}$, and $T_{new}$ separately.

Secondly, the tracker performs the Re-activate operation by applying the accumulated CMC from $T_L$ to $T_{new}$ to $z_L$ in Equation (18). Due to $x_L$ and $z_L$ being recorded at different frames, the tracker applies the accumulated CMC from $T_L+1$ to $T_{new}$ to $x_L$ in Equation (19). For generating a series of virtual detection bounding boxes $z_k$, the tracker uses linear interpolation between the newly matched detection bounding box $z_{new}$and the last matched detection bounding box $z$ after CMC in Equation (20), shown in Figure 6. Then the tracker generates a series of virtual detection confidence scores $c_k$ using linear interpolation between the newly matched detection confidence $c_{new}$and the last matched detection confidence $c_L$in Equation (21). After completing the aforementioned data preparation tasks, the tracker backtracks the trajectory state to the recorded state $x$ after CMC, reruns CBKF for the lost period from $T_L+1$ to $T_{new}$ and obtains the final trajectory state for the current frame. The operational process of Re-activate is shown in Figure 7.

$$x=CM{C}_{T_{L+1}}^{T_{new}}(x_L)$$

(18)

$$z=CM{C}_{T_L}^{T_{new}}(z_L)$$

(19)

$$z_k=z+(T_k-T_L){\displaystyle \frac{Z_{new}-z}{\mathsf{\Delta }t}}$$

(20)

$$c_k=c_L+(T_k-T_L){\displaystyle \frac{C_{new}-C_L}{\mathsf{\Delta }t}}$$

(21)

where $x$ represents the trajectory state after cumulative CMC transformation, $z$ denotes the last detection box before becoming lost after cumulative CMC transformation, and $T_k$ indicates the frame number corresponding to the generated virtual detection.

Figure 6. The left image shows the position of the pedestrian in the last frame before becoming lost. The right image shows the position of the pedestrian in the re-associated frame. The dashed rectangles represent the generated virtual detections.

Figure 7. The operational process of Re-activate.

It is worth noting that, initially, we only intended to use detection bounding boxes and the same confidence score to generate virtual detections. However, experiments reveal that this approach performs poorly in certain scenarios. Therefore, we introduce CMC to help the tracker generate more robust virtual detections; furthermore, linear interpolation is also applied to the confidence score to generate a more reliable confidence level. Additionally, we evaluate both approaches on the MOT17-14 video sequence. The results are shown in Table 3 and Figure 8.

Table 3. The comparison between the two approaches on MOT17-14.

Methods	HOTA	MOTA	IDF1
BB and same conf	49.9	56.8	68.3
Improved method	51.1	59.0	69.1

Figure 8. The left image was obtained using CMC-detection-based Re-activate. The right image was obtained using the method of detection bounding boxes and the same confidence score.

From Table 2 and Figure 8, we can conclude that the improved Re-activate method indeed demonstrates superior performance in certain scenarios. In the MOT17-14 video sequence scenarios, the camera is in constant motion. Under such conditions, if trackers do not use CMC to calibrate the last detection $z_L$ and $x_L$ coordinate systems, this would introduce additional errors into the target trajectory. Additionally, when we perform linear interpolation between two detection results with different confidence levels to generate virtual detection bounding boxes, accurately evaluating these generated boxes becomes challenging. Therefore, we opt to linearly interpolate the confidence scores to generate virtual confidence scores. These virtual confidence scores gradually decrease from the higher confidence side toward the lower confidence side, indicating that the tracker places greater trust in virtual detection boxes closer to the higher confidence side.

3.5. Tracking Pipeline

To more clearly demonstrate the pipeline of OcclusionTrack and enhance reproducibility, we present Algorithm 1, which integrates ByteTrack with our four improvements.

Step 1: Use the YOLOX detector to detect bounding boxes and confidence scores from the current frame image. Classify these detections into high-score and low-score detections based on confidence scores.

Step 2: Calculate the CMC parameter using the current frame image and the previous frame image. Apply Kalman filtering to predict the trajectory state for the current frame, then correct it using CMC.

Step 3: Perform the first association: match the predicted trajectory state with high-scoring detections using DCM to obtain association results.

Step 4: Perform the second association: match the unmatched trajectory from the first association with low-score detections using DCM to obtain association results.

Step 5: Merge the association results from the first two steps.

Step 6: Update the matched results: if the trajectory state is activated, perform CBKF update; if the trajectory state is lost, perform Re-activate.

Step 7: Process unmatched tracks by incrementing their lost frame count by one. If the lost frame count exceeds the max age, remove the track. If within the max age, set the track state to lost.

Step 8: Initialize unmatched detections as new tracks.

Step 9: Output the current frame tracking results.

Algorithm 1: Proposed OcclusionTrack Pipeline for Frame t.

Input: Current frame I_t, Previous tracks T_{t-1}, Previous image I_{t-1} Output: Updated tracks T_t # Step 1: Object Detection and Classification 1: D_all = detector(I_t) // Get all detections {bbox, score} 2: D_high = {d ∈ D_all | d.score ≥ τ_high} // High-score detections 3: D_low = {d ∈ D_all | d.score < τ_high ∧ d.score ≥ τ_low} // Low-score detections # Step 2: Kalman Filter Prediction with CMC 4: H = estimate(I_{t-1}, I_t) // Camera motion compensation 5: for each track τ in T_{t-1} do 6:   if τ.state ∈ {“active”, “lost”} then 7:     τ.pred_box = KalmanPredict(τ) // Standard prediction 8:     τ.pred_box = compensate(H, τ.pred_box) // Apply CMC 9:   end if 10: end for # Step 3: First Association (High-score detections) 11: T_candidates = {τ ∈ T_{t-1} | τ.state ∈ {“active”, “lost”}} // All tracks 12: matches_high, unmatched_tracks_1, unmatched_dets_high = DCM_Association(D_high, T_candidates) # Step 4: Second Association (Low-score detections) 13: matches_low, unmatched_tracks_2, unmatched_dets_low = DCM_Association(D_low, unmatched_tracks_1) # Step 5: Merge Matching Results 14: all_matches = matches_high ∪ matches_low 15: unmatched_tracks = unmatched_tracks_2 16: unmatched_dets = unmatched_dets_high ∪ unmatched_dets_low # Step 6: Track Update and Management 17: for each match (τ, d) in all_matches do 18:   if τ.state == “lost” then 19:     τ = Re_activate(τ, d, CMC) // Re-activate for lost tracks 20:   end if 21:    22:   # CBKF update with non-linear measurement noise 23:   R = compute_measurement_noise(d.score, R) // Measurement noise based on confidence 24:   KalmanUpdate(τ, d.bbox, R) 25:   τ.state = “active” 26:   τ.lost_count = 0 // Reset lost counter 27: end for # Step 7: Handle Unmatched Tracks 28: for each track τ in unmatched_tracks do 29:   τ.lost_count += 1 30:   if τ.lost_count > max_age then 31:     τ.state = “removed” // Delete track 32:   else 33:     τ.state = “lost” 34:   end if 35: end for # Step 8: Initialize New Tracks 36: for each detection d in unmatched_dets do 37:   if d.score ≥ τ_init then 38:     create_new_track(d) 39:   end if 40: end for # Step 9: Output Results 41: T_t = {τ ∈ T_{t-1} ∪ new_tracks | τ.state ≠ “removed”} 42: return T_t

4. Experiments

4.1. Datasets

We evaluate the performance of our tracker on the MOT17 [15], MOT20 [16], and DanceTrack [17] datasets. Tracking is performed on the test sets of MOT17 and MOT20, and tracking results are submitted to MOTchallenge to obtain tracking metrics. Based on ByteTrack [3], we split the MOT17 and MOT20 training sets in half to form validation sets and evaluate the performance of each component on the validation sets.

MOT17. The MOT17 [15] dataset is one of the most influential and widely used benchmark datasets in the field of multi-object tracking (MOT). It aims to provide a unified, rigorous, and challenging platform for evaluating and comparing multi-object tracking algorithms. The MOT17 dataset comprises 14 video sequences primarily featuring pedestrian tracking scenarios across both indoor and outdoor environments. It encompasses multiple key challenges: (1) occlusion; (2) lighting variations; (3) camera motion; (4) scale changes; and (5) dense crowds.

MOT20. The MOT20 [16] dataset is a benchmark dataset proposed in the field of multi-object tracking (MOT) to address scenarios involving extremely dense crowds. The core objective of MOT20 is to evaluate the robustness and accuracy of tracking algorithms in environments characterized by exceptionally high crowd density and severe occlusions. MOT20 comprises eight video sequences primarily captured at large public gatherings, train stations, and plazas. The key challenges of MOT20 are extremely high target density and severe and persistent occlusions.

DanceTrack. The DanceTrack [17] dataset exhibits the following characteristics: tracking targets share similar visual features, their motion patterns are highly complex, and occlusion occurs frequently. Dancetrack comprises a total of 100 video sequences, including 40 training sequences, 20 validation sequences, and 40 test sequences. Furthermore, similar to the MOT series datasets, test set metrics require uploading results to the official website to obtain.

4.2. Metrics

We employ the general CLEAR metrics [18], HOTA [19], MOTA, IDF1 [20], ID Switch, FP, and FN to evaluate the performance of our tracker. MOTA emphasizes detector performance. ID Switch represents the number of target ID switches, reflecting the identity retention capability of a tracker. Additionally, IDF1 emphasizes association performance. HOTA serves as a comprehensive metric to evaluate the overall effectiveness of detection and association.

4.3. Implementation Details

During the confidence-based Kalman filter update and Re-activate phases, we set different $\beta$ values based on distinct datasets. For MOT17, the default value for $\beta$ is 8. For MOT20, the default value is set to 12. For DanceTrack, we set the default value of $\beta$ at 12. Additionally, the confidence threshold $c_0$ is set to 0.7 in MOT17, MOT20, and DanceTrack. During the DCM association stage, for MOT17, the default depth levels are set to 3, for MOT20, the default depth levels are set to 8, and for DanceTrack, the default depth levels are set to 12. The high-score and low-score detection threshold is set to 0.6 and 0.1 by default. The down scale of CMC is set to 4 by default. The method of CMC is the sparse optical flow method. All our tracking results rely solely on IoU distance.

In the comparison experiments, to ensure fair comparison, we employed the pre-trained YOLOX detector from ByteTrack [3], utilizing the same weights and NMS threshold as ByteTrack [3]. Additionally, all experiments are performed on the NVIDIA GeForce RTX3060 GPU (NVIDIA, Santa Clara, CA, USA).

4.4. Comparison on MOT17, MOT20, and DanceTrack

We evaluate our tracker on the MOT17, MOT20, and DanceTrack test sets in comparison with other state-of-the-art trackers. The best results are shown in Table 4, Table 5 and Table 6.

Table 4. The comparison of our tracker with other state-of-the-art methods under the “private detector” protocol on the MOT17 test set.

Tracker	HOTA↑	MOTA↑	IDF1↑	ID Switch↓	FP↓	FN↓	FPS↑
Tube_TK [21]	48.0	63.0	58.6	4137	27,060	177,483	3.0
CenterTrack [22]	52.2	67.8	64.7	3039	18,498	160,332	17.5
TransCenter [23]	54.5	73.2	62.2	4614	23,112	123,738	1.0
FairMOT [24]	59.3	73.7	72.3	3303	27,507	117,477	25.9
TransMOT [25]	61.7	76.7	75.1	2346	36,231	93,150	9.6
MAATrack [26]	62.0	79.4	75.9	1452	37,320	77,661	189.1
ByteTrack [3]	63.1	80.3	77.3	2196	25,491	83,721	29.6
OC-SORT [9]	63.2	78.0	77.5	1950	15,129	107,055	29.0
Hybrid-SORT [27]	63.6	79.3	78.4	\	\	\	\
UCMCTrack [28]	64.3	79.0	79.0	\	\	\	\
StrongSORT [8]	64.4	79.6	79.5	1194	27,876	86,205	7.1
DiffMOT [10]	64.5	79.8	79.3	\	\	\	\
BoT-SORT [5]	64.6	80.6	79.5	1257	22,524	85,398	6.6
Deep OC-SORT [6]	64.9	79.4	80.6	1023	16,572	98,796	28.1
OCCTrack	64.9	80.9	79.7	1269	23,421	83,199	12.5

Table 5. The comparison of our tracker with other state-of-the-art methods under the “private detector” protocol on the MOT20 test set.

Tracker	HOTA↑	MOTA↑	IDF1↑	ID Switch↓	FP↓	FN↓	FPS↑
MLT [29]	43.2	48.9	54.6	2187	45,660	216,803	3.7
TransTrack [30]	48.5	65.0	59.4	3608	27,197	150,197	7.2
FairMOT [24]	54.6	61.8	67.3	5243	10,3440	88,901	13.2
Semi-TCL [31]	55.3	65.2	70.1	4139	61,209	114,709	\
RelationTrack [32]	56.5	67.2	70.5	4243	61,134	104,597	2.7
MAATrack [26]	57.3	73.9	71.2	1331	24,942	108,744	14.7
ByteTrack [3]	61.3	77.8	75.2	1223	26,249	87,594	17.5
OC-SORT [9]	62.4	75.7	76.3	942	19,067	105,894	18.7
Hybrid-SORT [27]	62.5	76.4	76.2	\	\	\	\
StrongSORT [8]	62.6	73.8	77.0	770	16,632	117,920	1.4
BoT-SORT [5]	62.6	77.7	76.3	1212	22,521	86,037	6.6
MotionTrack [33]	62.8	78.0	76.5	1165	28,629	84,152	9.0
UCMCTrack [28]	62.8	75.6	77.4	1335	28,678	96,198	44.8
OCCTrack	63.2	76.9	77.5	916	23,042	95,412	8.5

Table 6. The comparison of our tracker with other state-of-the-art methods under the “private detector” protocol on the DanceTrack test set.

Tracker	HOTA↑	MOTA↑	IDF1↑	AssA↑	DetA↑
FairMOT [24]	39.7	82.2	40.8	23.8	66.7
CenterTrack [22]	41.8	86.8	35.7	22.6	78.1
TraDes [34]	43.3	86.2	41.2	25.4	74.5
TransTrack [30]	45.5	88.4	45.2	27.5	75.9
ByteTrack [3]	47.7	89.6	53.9	32.1	71.0
MotionTrack [33]	52.9	91.3	53.8	34.7	80.9
MOTR [35]	54.2	79.7	51.5	40.2	73.5
BoT-SORT [5]	54.7	91.3	56.0	37.8	79.6
OC-SORT [9]	55.1	89.4	54.2	38.0	80.3
SparseTrack [7]	55.5	91.3	58.3	39.1	78.9
StrongSORT [8]	55.6	91.1	55.2	38.6	80.7
OCCTrack	57.5	91.4	58.4	40.9	81.0

MOT17. Through the comparison with other state-of-the-art methods, our tracker has demonstrated strong performance on the MOT17 test set. Compared to the baseline ByteTrack [3] with the same pre-trained detector YOLOX, our tracker achieves gains of +1.8 HOTA, +0.6 MOTA, and +2.4 IDF1, and ID Switch reduced to almost half that of ByteTrack’s.

MOT20. Through the comparison with other state-of-the-art methods, our tracker has demonstrated that it still maintains robust tracking performance in scenes with severe occlusion on the MOT20 test set. Compared to the baseline (ByteTrack) with the same pre-trained detector YOLOX, our tracker achieves gains of +1.9 HOTA and +2.3 IDF1, and ID switch has been reduced by twenty-five percent.

DanceTrack. Through the comparison with other state-of-the-art methods, our tracker has demonstrated strong performance on the DanceTrack test set. Compared to the baseline with the pre-trained YOLOX detector from OC-SORT, our tracker achieves gains of +9.8 HOTA, +1.8 MOTA, and +4.5 IDF1.

4.5. Ablation Studies of Different Components

We evaluate our tracker on the validation sets of MOT17 and MOT20 to observe the impact of different components on tracking performance. The results are shown in Table 7 and Table 8.

Table 7. Ablation studies on the MOT17 validation set.

Method	CBKF	DCM	CMC	Re-Activate	HOTA	MOTA	IDF1
Baseline					68.1	77.7	80.0
Baseline + 1	√				68.1	77.8	80.3
Baseline + 2		√	√		67.8	77.6	79.5
Baseline + 3					68.4	78.1	80.9
Baseline + 1–2	√	√			68.1	77.5	80.3
Baseline + 1–3	√	√	√		69.1	77.9	82.1
Baseline + 1–4	√	√	√	√	69.1	78.0	82.1

Table 8. Ablation studies on the MOT20 validation set.

Method	CBKF	DCM	CMC	Re-Activate	HOTA	MOTA	IDF1
Baseline					69.5	87.9	84.4
Baseline + 1	√				69.6	87.9	84.7
Baseline + 2		√	√		69.6	87.8	84.7
Baseline + 3					69.4	87.9	84.4
Baseline + 1–2	√	√			69.6	87.8	85.1
Baseline + 1–3	√	√	√		69.7	87.8	85.3
Baseline + 1–4	√	√	√	√	69.8	87.8	85.3

MOT17. On the MOT17 validation set, we set $\beta$ to 8 and $c_0$ to 0.7. The depth level is set to 3. We use sparse optical flow as the CMC method. We use the pre-trained detector YOLOX from ByteTrack [3] as the detector. Through the experiments, CMC achieves primary gains of +0.9 IDF1, CBKF achieves gains of +0.3 IDF1, and Re-activate achieves gains of +0.1 MOTA.

MOT20. On the MOT20 validation set, we set $\beta$ to 15 and $c_0$ to 0.7. The depth level is set to 8. We use sparse optical flow as the CMC method. We use the pre-trained detector YOLOX from SparseTrack [7] as the detector. Through the experiments, CBKF and DCM both achieve gains of +0.3 IDF1 and Re-activate achieves gains of +0.1 HOTA.

4.6. The Impact of Different $\beta$ Values

We evaluate our tracker on the MOT17 validation set to obtain the impact of different $\beta$ values on tracking performance. The results are shown in Table 9.

Table 9. The performance of the OcclusionTrack when using different $\beta$ values.

$\mathit{\beta }$ Values	HOTA	MOTA	IDF1
2	68.9	77.9	81.9
4	69.0	77.9	82.0
6	69.1	78.0	82.1
8	69.1	78.0	82.1
10	69.1	77.9	82.0

We set the value to 2, 4, 6, 8, and 10. As $\beta$ values increase, the tracking performance gradually improves. When $\beta$ values reach 6 and 8, the tracking performance achieves the best scores. When we continue to increase $\beta$ to 10, tracking performance begins to decline. This indicates that a larger $\beta$ value facilitates Kalman filter update. However, when it becomes excessively large, performance actually deteriorates. We think that an overly large $\beta$ value may cause measurement noise to fluctuate too rapidly, thereby interfering with detection results with confidence between 0.4 and 0.6.

4.7. The Impact of the Max Age

Severe occlusions can cause the long-term lost of trajectories. The max age represents the maximum number of frames a trajectory remains lost for before being removed. Both excessively low and excessively high max age values can degrade tracking performance. When the max age is set too low, even if a trajectory is rematched in subsequent frames, it cannot recover its original ID because it has already been removed by the tracker. Instead, it is assigned a new ID. When the max age is set too high, the target may have disappeared from the frame, yet the tracker continues to predict its state. This not only increases computational loads but also risks erroneous matches. Therefore, we evaluate the tracking performance of different max age settings on the MOT17 and MOT20 validation sets to observe the impact of the max age on tracking performance. The results are shown in Figure 9 and Figure 10.

Figure 9. The impact of different max age settings on the MOT17 validation set.

Figure 10. The impact of different max age settings on the MOT20 validation set.

4.8. The Impact of Different Detectors

Tracking-by-detection (TBD) trackers rely heavily on the detection quality of the detector. It can be said that TBD’s current powerful capabilities are inseparable from the continuous advancement of detectors. Therefore, we evaluate our tracker’s performance with different detectors to obtain the impact of the detector on tracking performance. The results are shown in Table 10.

Table 10. The performance of the OcclusionTrack when using different detectors.

Detector	HOTA	MOTA	IDF1
YOLOX-X	79.9	90.3	89.0
YOLOX-Ablation	69.1	78.0	82.1

We can observe that detectors with enhanced performance significantly improve the tracking capabilities of trackers. This indicates that we can enhance tracking performance by designing detectors with superior capabilities.

To observe the generalization capabilities of each module when using different detectors, we replaced the detector with a more powerful one and conducted ablation experiments on each module. The results are shown in Table 11. Through the above two experiments, we observed that OcclusionTrack is highly sensitive to detector performance; poor-quality detectors significantly limit the tracker’s capabilities. This situation may be due to the fact that each module within OcclusionTrack relies heavily on the data provided by the detector.

Table 11. Performance analysis of each module when using a better detector on the MOT17 validation set.

Method	CBKF	DCM	CMC	Re-Activate	HOTA	MOTA	IDF1
Baseline					77.9	90.1	86.4
Baseline + 1	√				78.4	90.2	87.0
Baseline + 2		√			78.2	89.6	86.8
Baseline + 3			√		78.9	90.5	87.5
Baseline + 1–4	√	√	√	√	79.9	90.3	89.0

4.9. Visualization

To better evaluate the tracking performance of our tracker and observe the specific performance of our tracker, we conduct visualization experiments on the MOT20 and MOT17 test sets. The visualization results are shown in Figure 11 and Figure 12.

Figure 11. These two images are taken from two consecutive frames within the same MOT20 video sequence.

Figure 12. These two images are taken from two consecutive frames within the same MOT17 video sequence.

This MOT20 video sequence represents a scene with severe occlusion. The scene depicted in the MOT17 video sequence is a street between two buildings. Certain areas within the sequence feature relatively dense targets and moderate occlusion.

5. Conclusions

A robust tracking-by-detection framework named OcclusionTrack has been proposed to address key challenges in dense multi-object tracking, particularly those arising from occlusions and camera motion. The framework integrates four core improvements into a cohesive pipeline: (1) a confidence-based Kalman filter (CBKF) for adaptive state update under partial occlusion; (2) camera motion compensation (CMC) for inter-frame alignment; (3) a depth–cascade-matching (DCM) algorithm for association in overlapping scenarios; and (4) a CMC-detection-based trajectory Re-activate method for recovering from complete occlusion.

The effectiveness of the proposed method has been validated through comprehensive experiments on the MOT17, MOT20, and DanceTrack benchmarks. OcclusionTrack achieves highly competitive performances, notably improving IDF1 by 2.4 and reducing ID switches by nearly 50% over the ByteTrack baseline on the MOT17 test set, improving IDF1 by 2.3 and reducing ID switches by nearly 25% over the ByteTrack baseline on the MOT20 test set, and improving IDF1 and HOTA by 4.5 and 9.8 over the ByteTrack baseline on the DanceTrack test set.

The experimental results demonstrate that all the predefined research objectives have been met to a certain extent, with quantitative evidence supporting each contribution:

(1)

Objective 1 (Handling Partial Occlusion Challenges during Kalman Filter Update): To mitigate the impact of detection quality degradation during partial occlusion, the CBKF module was designed. Its efficacy is confirmed through experiments, where its introduction leads to the +0.3 IDF1 improvement on both the MOT17 and MOT20 validation sets. This gain is attributed to the module’s ability to dynamically change the weight of detections during the Kalman update, preventing trajectory drift.

(2)

Objective 2 (Compensation for Camera Motion): To stabilize the prediction base for IoU matching, a CMC module was incorporated. Results indicate a +0.9 IDF1 increase on the MOT17 validation set. It is the component with the greatest improvement among the four. More importantly, its application is a prerequisite for the effective functioning of the reactivation mechanism, showcasing its systemic value beyond a direct performance bump.

(3)

Objective 3 (Robust Association under Occlusion): To resolve ambiguities in dense, overlapping scenes, the DCM algorithm was integrated. DCM contributes significant gains, boosting IDF1 by +0.4 on the MOT20 validation set. This confirms that depth-level partitioning effectively addresses complex inter-object interactions.

(4)

Objective 4 (Correction for Complete Occlusion): To address error accumulation in lost tracks, a Re-activate method was proposed. While its isolated impact is smaller (+0.1 HOTA) on the MOT20 validation set, its role in maintaining long-term identity consistency is critical.

While demonstrating strong performance, the portability and effectiveness of OcclusionTrack are subject to certain conditions which define its scope of applicability:

(1)

Dependence on Detector: Experiments demonstrate that TBD trackers are highly dependent on detector performance, with higher-quality detectors significantly improving tracking accuracy. Additionally, the CBKF module relies on the confidence scores from the detector being reasonably calibrated to detection quality. Performance may degrade if confidence scores are not predictive of localization accuracy.

(2)

Assumption for Camera Motion Scenario: The CMC module employs a global 2D model, which assumes dominant, low-frequency camera motion. Its effectiveness may diminish in scenarios with highly dynamic, non-planar backgrounds or very high-frequency jitter. Moreover, it cannot meet the requirements of 3D scenes.

(3)

Scene-Type Specificity: The current motion model retains a linear constant-velocity assumption. Consequently, tracking performance can decline in sequences with extremely low frame rates or highly non-linear, agile object motion. Furthermore, the framework is primarily designed and tested on pedestrians, and its extension to objects with highly deformable shapes may require adjustments.

Future research will focus on enhancing generalizability by (1) exploring the joint optimization of the detector and tracker components to reduce dependency on the performance of the detections; (2) investigating adaptive, scene-aware CMC strategies that can handle more diverse camera dynamics; and (3) developing deep, data-driven motion models to replace the linear Kalman predictor for handling complex motions.