Occlusion-Robust Multi-Target Tracking and Segmentation Framework with Mask Enhancement

Abstract

Multi-object tracking stands as one of the most prominent domains in Computer Vision and has significant research value and practical importance. However, due to the complexity of scenarios in the real world, especially in crowded environments with frequent target occlusion, existing MOT frameworks often struggle to achieve precise tracking results. To enhance the trajectory association accuracy of MOT frameworks in occluded scenarios, this paper proposes a mask-enhanced occlusion-robust multi-target tracking and segmentation framework. Our method first introduces a mask-conditional feature fusion network and an occlusion-aware mask propagation network. The former network integrates a mask-guided attention mechanism with a spatial–temporal feature aggregation sub-network to improve tracking robustness in crowded scenes, and the latter network prevents the contamination of online tracking templates from noise inputs by perceiving a target occlusion state. The framework merges the mask-based methods above into a mask-integrated multi-hypothesis tracking algorithm, achieves superior adaptability in occluded scenarios, and enhances the robustness of MOTS tasks. Our framework achieves the best performance on the MOTSA (84.4%), MT, and FN metrics, with a 6.1% reduction in FN compared to the state-of-the-art method. Our method achieves significant improvements in both accuracy and precision and is validated on public datasets.

1. Introduction

Multiple-object tracking (MOT) is one of the prominent tasks in the field of Computer Vision. The process of MOT involves associating multiple targets across different frames in a video sequence frame-by-frame and generating motion trajectories of the tracked targets. MOT serves as the foundation of action recognition and behavior analysis of targets [1].

The Tracking By Detection (TBD) framework is the mainstream approach in MOT methods. This framework divides MOT into two stages: detection and data association. In the detection stage, a pretrained detector is used to obtain targets’ detection results in the video sequence. The detection process transforms the target tracking problem into a data association problem across consecutive frames. During the data association stage, targets in consecutive frames are associated so as to form complete trajectories. This stage outputs the positional information of different targets at different times. Recent years have witnessed various solutions proposed to optimize trajectory association in MOT tasks. They have addressed complex interactions such as inter-object occlusion and dynamic target counts in different scenarios [2,3,4,5,6,7,8].

Existing MOT trackers often perform well in simple scenarios. However, when applied to crowded scenes, their performance significantly deteriorates, since target occlusion interferes with the target detection and the data association processes. In practical applications, not all observations reflect target information correctly, since they suffer from background noise, motion blur, or occlusion [9,10]. The lack of state evaluation mechanisms makes traditional approaches vulnerable to noise inputs and ineffective for long-term tracking in occluded scenarios.

To address these challenges, Voigtlaender et al. [1] integrated instance segmentation techniques with MOT algorithms. They constructed a multiple-object tracking and segmentation (MOTS) framework, which can achieve pixel-level tracking on multiple targets.

Compared to traditional bounding box detections, instance segmentation provides pixel-level detection, assigning a unique mask to each target. Image instance segmentation aims to classify each pixel in an image, predicting both target categories and pixel-level segmentation masks to accomplish simultaneous object detection and semantic segmentation [11,12]. Additionally, the non-overlapping nature of binary instance masks resolves the ambiguity involved in distinguishing target identities during occlusion. Approaches like Mask R-CNN [13] have demonstrated that pixel-level detection based on segmentation can effectively handle complex scenarios with occlusion by eliminating errors from traditional bounding boxes.

Therefore, merging instance segmentation into MOT tasks can mitigate issues caused by traditional bounding box detections and enhance tracking performance in occluded scenarios.

The main contributions of our paper can be summarized as follows:

• We propose a mask-conditional feature fusion network that integrates a mask-guided attention mechanism and a spatial–temporal feature aggregation sub-network. The network improves tracking robustness under crowded scenes. The network leverages temporal information from short trajectories and learns to generate occlusion-robust spatial–temporal aggregation features under the guidance of masks.
• We present an occlusion-aware mask propagation network. The network prevents the contamination of online tracking templates from noise inputs by perceiving the occlusion states of targets. It completes the association of missing trajectory segments through a constructed mask propagation network.
• We propose a mask-enhanced multi-object tracking and segmentation framework. This framework combines mask-based methods with a mask-integrated multi-hypothesis tracking algorithm to achieve adaptability in occluded scenarios and enhance the robustness of the MOTS framework.

The rest of this paper is organized as follows. Related work is discussed in Section 2. The details of the MOTS framework methodology are described in Section 3. Experimental results are shown and analyzed in Section 4. Section 5 discusses the benefits, as well as the limitations, of the method, and it is followed by the conclusions in Section 6.

2. Related Works

2.1. Trajectory Association Algorithms

In recent years, researchers have proposed various solutions to optimize trajectory association in MOT tasks, as shown in

Table 1. Summary of trajectory association algorithms.

Method	Technical Features	Advantages
Online methods [2,3,4,5,6,7,8]	Sequential frame processing	Real-time performance
Filter-based [9,10]	Motion/appearance modeling	Fast computation
MHT [5,14]	Multi-Hypothesis Trees	Comprehensive association
Kim et al. [15]	CNN features with MHT	Improved appearance modeling
Zhang et al. [16]	LSTM-enhanced MHT	Long-term dependency capture
Chu et al. [17]	Spatial–temporal attention	Occlusion handling
Sadeghian et al. [18]	LSTM networks	Temporal dependency modeling
Peng et al. [8]	Deep residual networks	Frame-to-frame similarity
Yi et al. [19]	CenterNet	Background bias reduction

. Online trajectory association methods process current frames using present and past data without future information. Existing online methods [2,3,4,5,6,7,8] process video frames sequentially to generate tracking results but often cause trajectory loss or identity drift due to occlusion or detection errors.

Filter-based tracking [9,10] based on target motion and appearance generates predictive data for trajectory association. These algorithms demonstrate fast computational speeds but suffer from background noise and motion blur.

The Multiple-Hypothesis Tracking (MHT) algorithm [5,14] constructs multi-layer hypothesis trees for each target. However, the computational complexity grows exponentially with hypothesis expansion.

Kim et al. [15] integrated deep neural network-derived appearance features into MHT frameworks. Zhang et al. [16] further optimized the hypothesis, updating it through LSTM networks. Progress was also made in deep-learning based methods. Chu et al. [17] employed spatial–temporal attention to address partial occlusion. Sadeghian et al. [18] proposed LSTM networks for modeling temporal dependencies. Peng et al. [8] utilized deep residual networks to compute inter-frame target similarity. Yi et al. [19] developed an anchor-free tracking approach based on CenterNet [20].

2.2. Instance Segmentation-Based Multi-Object Tracking

With the development of object detection and instance segmentation, the accuracy of MOT algorithms has significantly improved. Voigtlaender et al. introduced the MOTS task based on MOT datasets [21]. MOTS integrates detection, segmentation, and tracking to achieve pixel-level multi-object tracking.

Mask R-CNN [13] became a standard framework: it performed well on datasets like COCO [22], Cityscapes [23], and MVD [24]. It featured three branches for regression, classification, and segmentation. The core principle of the architectures involves adding a mask prediction branch to existing Fast R-CNN [25] or Faster R-CNN [26] frameworks, and the mask prediction branch utilizes an encoder–decoder Fully Convolutional Network (FCN) [27] structure.

Video instance segmentation (VIS) methods include propagation-based approaches [13,28,29,30,31,32] that refine masks using temporal information. In this field, Li et al. [30] integrated a pedestrian re-identification module into a video instance segmentation system. Wang et al. [31] improved mask propagation through clustering. For crowded scenes, Liu et al. [4] used spatial–temporal trees, while others focused on occlusion detection: Sun et al. [33] compared target features, Dong et al. [34] maintained feature pools, and Stadler et al. [35] scored occlusion patterns.

2.3. Multi-Object Tracking and Segmentation Frameworks

Yang et al. [36] first integrated instance segmentation with tracking tasks. Subsequently, to improve the accuracy of existing multi-object tracking, Voigtlaender et al. [1] constrained the tracking targets in VIS to pedestrians and vehicles.

They also developed the TrackR-CNN framework that combined segmentation instances with tracking algorithms. Following the tracking-by-detection paradigm, TrackR-CNN extended Mask R-CNN [13] by adding a tracking branch. It employed 3D convolutions to capture temporal and appearance information in videos, computed target embedding vectors and IoU predictions from optical flow, and incorporated both as weights in a bipartite graph matching algorithm for data association.

3. Methodology

We proposed a novel MOTS framework based on masks and an enhanced multi-hypothesis tracking algorithm (MHT). The framework integrated a mask-conditioned feature fusion network and a occlusion-aware mask propagation network in congestion scenes. Our framework first extracted spatial–temporal features from video sequences via mask-guided foreground-background separation. Foreground features captured the target geometry such as contour and area and background features encoded the scene context. These features were then processed by a hierarchical hypothesis tree that dynamically weighted appearance, motion, and IoU based on the real-time occlusion state. Eventually, a memory-enhanced decoder synthesized the masks of the occlusion targets by correlating historical non-occlusion features with the current frame query to achieve pixel-accurate trajectory recovery.

3.1. Mask-Conditioned Feature Fusion Network

Traditional feature extractors perform poorly when processing occlusion and acquiring temporal information. Therefore, we proposed a mask-conditional feature fusion network to solve the limitations. The network integrated a mask-guided attention mechanism and a spatial–temporal feature aggregation sub-network so as to improve tracking robustness in crowded scenes. Its main sub-modules aew shown in Figure 1 below.

3.1.1. Mask-Attentive Dual-Stream Encoder

Conventional single-frame feature extraction methods have limited performance in occlusion scenarios since they fail to capture target motion patterns. Furthermore, in re-identification, background removal often degrades performance [37,38,39,40] due to segmentation errors. Tian et al. [40] showed that background removal reduces performance despite creating biases.

To solve these problems, we proposed a dual-stream encoder architecture based on short tracklet pairs. Given a tracklet of K frames as the encoder’s input, the data in each frame contained an RGB patch clipped from the bounding box detection and a corresponding instance mask (RLE encoding). The encoding process began with two parallel paths.

The foreground branch processed the mask-cropped region to extract geometric features. In the foreground branch, element-by-element multiplication was performed between the input image $I_t$ and binary mask $M_t$ to obtain the foreground region $I_{fg}$. Features were then extracted through a ResNet network and the spatial attention mechanism was applied to enhance feature responses in the target region. The background branch analyzed the entire detection patch to capture contextual meaning. It directly processed the original detection box image $I_{bg}=I_t$ using the same ResNet network so as to extract contextual features.

This dual-stream design preserved both geometric target features and valuable contextual scene information. To increase robustness to segmentation noise, dynamic mask expansion was introduced during training. The method simulated real-world mask imprecision with random morphological operations. This allowed the model to prioritize geometric attributes invariant to occlusion. Both branches employed a shared-weight ResNet50 backbone to ensure feature compatibility. The ResNet50 backbone extracted features from both streams, with the foreground branch receiving additional mask-derived attention:

$$F_{fg}^t=\mathrm{ResNet}(I\odot M)·\mathrm{Sigmoid}\left(\mathrm{MaskCNN}\left(M\right)\right)$$

(1)

where M is the binary mask and ⊙ is the element-by-element multiplication. Eventually, this part employed a global average pooling (GAP) layer to reduce the computational complexity through dimensionality reduction. The method output the following two sets of features: foreground features $F_{fg}^t$ and background features $F_{bg}^t$.

3.1.2. Adaptive Feature Fusion Sub-Module

Studies [37,40] also show that directly removing the background with a mask not only degrades person re-identification’s performance but also tends to degrade MOT tasks’ performances entirely, since directing the network’s attention with incorrect segmentation instances will impact tracking results negatively.

The Adaptive Feature Fusion Sub-Module is an intelligent component that combines foreground and background features and dynamically adapts to masks’ quality changing conditions. Based on the dual-stream features $F_{fg}^t,F_{bg}^t$ generated from the aforementioned encoder, this module implemented a fusion mechanism through the following steps.

The process began with a quality evaluation of the mask. For foreground feature $F_{fg}^t$ in each frame, the module provided a quality scoring mechanism through a lightweight evaluation network. The output was a quality-scoring ${\alpha }_t\in [0,1]$ according to the following:

$${\alpha }_t=\sigma \left(W_2\mathrm{ReLU}\left(W_1[\mathrm{GAP}\left(F_{fg}^t\right)\oplus \mathcal{Q}\left(M_t\right)]\right)\right)$$

(2)

where $\mathcal{Q}\left(M_t\right)$ represents a mask quality index that includes boundary sharpness and area consistency between frames, while $W_1$ and $W_2$ are the learnable weights. The sigmoidal activation $\sigma (·)$ guaranteed a normalized quality score. The quality scores led to an adaptive fusion of foreground and background features through a gating mechanism as follows:

$$F_{fused}^t={\alpha }_t·F_{fg}^t+(1-{\alpha }_t)·\mathrm{SEBlock}\left(F_{bg}^t\right)$$

(3)

The Squeeze and Excitation Block (SEBlock) plays a crucial role in the fusion process. It performs per-channel feature recalibration. This design allows the module to emphasize reliable features and suppresses noise components. When the quality of the mask is high (${\alpha }_t\to 1$), foreground features are dominant. When the mask is unreliable (${\alpha }_t\to 0$), enhanced background features are dominant. The output $F_{fused}^t$ of this module maintains the same dimension as the input features. It achieves temporal consistency through frame-by-frame adaptive fusion.

3.1.3. Spatial–Temporal Aggregation Sub-Network

The spatial–temporal aggregation sub-network constitutes the final processing stage that transforms frame-level fusion features into a comprehensive tracklet representation. The network operates on the quality-weighted features $F_{fused}^t$ from the Adaptive Feature Fusion Sub-Module. It establishes spatial–temporal relationships through a graph neural network. The aggregation process initiates a spatial–temporal graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$. Each node $v_t\in \mathcal{V}$ corresponds to the frame’s feature $F_{fused}^t$, and each edge $e_{ij}\in \mathcal{E}$ encodes pairwise relationships between frames. We utilized a learned affinity function to compute the edge weights:

$${\mathcal{A}}_{ij}=\mathrm{softmax}\left({\displaystyle \frac{W_q{F}_{fused}^i{\left(W_k{F}_{fused}^j\right)}^{\top }}{\sqrt{d}}}+\varphi (\Delta t_{ij})\right)$$

(4)

where $W_q$ and $W_k$ are learnable projection matrices, d is the feature dimension, and $\varphi (·)$ encodes the time interval as a position embedding. This formulation captures similarity in appearance and temporal proximity simultaneously. For frames with low quality scores, the network activates the memory augmentation mechanism. It retrieves historically reliable features from the memory bank $\mathcal{M}$ through content-based addressing. The expanded features are then processed in the transformer encoder layer to produce a tracklet representation:

$$F_{memory}^t=\sum _{k=1}^K\mathrm{softmax}\left(\langle F_{fused}^t,{\mathcal{M}}_k\rangle \right)·{\mathcal{M}}_k$$

(5)

$$F_{transformer}^t=\mathrm{Transformer}({F_{fused}^t+F_{memory}^t·\mathbb{I}({\alpha }_t<\tau )}_{t=1}^N+\mathrm{PE}\left(t\right))$$

(6)

where $\mathrm{PE}\left(t\right)$ is the standard sinusoidal position coding and $\mathbb{I}(·)$ is the indicator function. The transformer encoder employs multi-head self-attention and layer normalization to model long-range dependencies across tracklets effectively.

Eventually, output features are dimensionally reduced by mean pooling and fully connected layers. The final output encompasses both the pattern of occlusion and the temporal motion characteristics of the target.

3.2. Occlusion-Aware Mask Propagation Network

Existing occlusion perception methods rely primarily on traditional bounding box modeling [1]. These approaches use techniques such as IoU, confidence scores, and regression models to detect occlusion states and integrate target states into the appearance model in order to optimize tracking performance. However, these approaches require manually designed templates and hyperparameters such as thresholds and result in complex models that lack robustness.

To solve these issues, this section details an occlusion-aware mask propagation network that overcomes the three main sources of noise in congestion tracking: occlusion between objects, occlusion between objects and scenes, and mutual interference between targets.

3.2.1. Occlusion State Perception with Mask IoU

The network proposes an occlusion state perception method. It defines four different occlusion states in crowded scenes. For frame t, we defined target A’s bounding box as

$${\mathrm{Box}}_t^A=(x_t^{A1},y_t^{A1},x_t^{A2},y_t^{A2},c_t^A),A\in {\mathcal{O}}_t$$

(7)

where ${\mathcal{O}}_t$ is the target set, $(x^1,y^1)$ is target A’s top-left coordinate, $(x^2,y^2)$ is target A’s bottom-right coordinate, and c is the confidence. We defined target A’s mask as

$$M_t^A=(m_t^A,c_t^A),m_t^A\in {\{0,1\}}^{H\times W}$$

(8)

where H and W denote frame height and width, respectively. The method defines IoU between target A and target B in the same frame t as ${\alpha }_t^{AB}$:

$${\alpha }_t^{AB}=\mathrm{IoU}({\mathrm{Box}}_t^A,{\mathrm{Box}}_t^B)={\displaystyle \frac{{\mathrm{Box}}_t^A\cap {\mathrm{Box}}_t^B}{{\mathrm{Box}}_t^A\cup {\mathrm{Box}}_t^B}}$$

(9)

The method then defines the mask area of target A as follows, where ${\beta }_t^A$ stands for the mask area of target A in frame t:

$${\beta }_t^A=\mathrm{area}\left(M_t^A\right)=\mathrm{sum}\left(m_t^A\right)$$

(10)

With the definitions above, we introduce the occlusion state of target A in frame t as $o_t^A\in \{0,1,2,3\}$. $o_t^A=0$ corresponds to “no occlusion”, $o_t^A=1$ corresponds to “occluded by target”, $o_t^A=2$ corresponds to “occluding other targets”, and $o_t^A=3$ corresponds to “occluded by non-target objects”. The occlusion state from frame 1 to frame t of target A can be represented as $o_{1:t}^A$, and the mask area from frame 1 to frame t of target A can be represented as ${\beta }_{1:t}^A$:

$$o_{1:t}^A=\{o_1^A,\dots ,o_t^A\}$$

(11)

$${\beta }_{1:t}^A=\{{\beta }_1^A,\dots ,{\beta }_t^A\}$$

(12)

Eventually, we build the crowded perception model $M_{oa}$ based on the least squares method:

$$o_t^A=M_{oa}\left(\underset{i\in \mathcal{O}t}{max}\left({\alpha }_t^{Ai}\right),{\beta }_{t-T:t}^A\right)$$

(13)

$$o_t^A=M_{oa}\left({\alpha }_t^{A,i},W_k\right),$$

(14)

where ${\alpha }_t^{A,i}$ represents the IoU between target A and other targets, and $W_k$ is a weight matrix learned from the mask area. During the tracking process, ${\alpha }_t^{A,i}$ is input to $M_{oa}$ to determine the occlusion state of the target along with the mask area of previous T frames. The output is the occlusion perception state.

The operation of $M_{oa}$ consists of two stages: trend fitting and state determination. In the trend fitting stage, for each target A in frame t, the input matrix is defined as

$$X_{t-T:t}={\left[t-T,t-T+1,\dots ,t\right]}^T,$$

(15)

And the response matrix is defined as

$$V_{t-T:t}^A=\mathrm{matrix}{\left({\beta }_{t-T:t}^A\right)}^T={\left[{\beta }_{t-T}^A,{\beta }_{t-T+1}^A,\dots ,{\beta }_t^A\right]}^T,$$

(16)

The model learns the weight matrices $W_k\in {\mathbb{R}}^{1\times 1}$ and $W_b\in {\mathbb{R}}^{1\times 1}$ to perceive the crowding state of the target. Weight $W_k$ is computed by minimizing the following objective function:

$$\underset{W_k}{min}\sum _{q=t-T}^t\parallel X_q{W}_k-{\beta }_q^A{\parallel }_2^2+\lambda {\parallel W_k\parallel }_2^2$$

(17)

where ${\parallel ·\parallel }_F^2$ is the Frobenius norm. Therefore, $W_k$ can be solved with the linear regression equation as follows:

$$\left[\sum _{q=t-T}^t\left(X_q^T{X}_q\right)+\lambda I\right]W_k=\sum _{q=t-T}^t\left(X_q^T{V}_q\right)$$

(18)

With the least squares method, $W_k$ can be computed as

$$\widehat{W_k}={\displaystyle \frac{{\sum }_{q=t-T}^t\left(x_q-\overline{x}\right)\left(v_q-\overline{v}\right)}{{\sum }_{q=t-T}^t{\left(x_q-\overline{x}\right)}^2}}$$

(19)

In the state determination stage, we designed a occlusion states classification strategy based on the IoU in current frame. The strategy utilized the following strategy to calculate $o_t^A$:

$$o_t^A=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}\underset{i\in O_t}{max}\left({\alpha }_t^{Ai}\right)<{\alpha }_{gt}\mathrm{and}\left|W_k\right|\ge {\theta }_k\hfill \\ 1,\hfill & \mathrm{if}\underset{i\in O_t}{max}\left({\alpha }_t^{Ai}\right)\ge {\alpha }_{gt}\mathrm{and}\left|W_k\right|<{\theta }_k\hfill \\ 2,\hfill & \mathrm{if}\underset{i\in O_t}{max}\left({\alpha }_t^{Ai}\right)\ge {\alpha }_{gt}\mathrm{and}\left|W_k\right|\ge {\theta }_k\hfill \\ 3,\hfill & \mathrm{if}\underset{i\in O_t}{max}\left({\alpha }_t^{Ai}\right)<{\alpha }_{gt}\mathrm{and}\left|W_k\right|<{\theta }_k\hfill \end{array}\right.$$

(20)

where ${\alpha }_{gt}$ is the minimum value of the IoU. This determines if the result is the same as labeling. ${\theta }_k$ is the threshold used to determine if $W_k$ decays too fast, and $\underset{i\in O_t}{max}\left({\alpha }_t^{Ai}\right)$ represents the maximum value of IoU between target A and other targets in frame t:

• If there exists a target i such that ${\alpha }_t^{Ai}\ge {\alpha }_{gt}$, this indicates that target A overlaps with another target. To determine whether A is the occluder or the occluded, $W_k$ is used:

  –

  If A is smaller than $W_k$, this indicates a rapid decrease in the mask area ${\beta }_{t-T:t}^A$, which means that A is an occluder ($o_t^A$ = 1);

  –

  If A is larger than $W_k$, this indicates no decreasing trend in the mask area, which means that A is the occluder ($o_t^A$ = 2).

• If there is no target i that satisfies ${\alpha }_t^{Ai}\ge {\alpha }_{gt}$, the occlusion state is also determined by $W_k$:

  –

  If A is larger than $W_k$, this indicates no occlusion ($o_t^A$ = 0);

  –

  If A is smaller than $W_k$, this indicates occlusion by non-target objects ($o_t^A$ = 3).

3.2.2. Robust Mask Propagation Network

To face the challenge of data association in congestion scenarios, we propose a novel approach for robust data association in such scenarios. The Robust Mask Propagation Network utilizes a constrained time window and captures the movement context effectively. The implementation steps of the method are as follows:

(1) When the target is not occluded, the correlation between trajectories is realized based on the multi-hypothesis tracking algorithm.

(2) When the occlusion sensor detects that a target is occluded, the first T frame of the current frame is input into the memory encoder, and the mask information of the network learning target is propagated based on the mask.

(3) When the occlusion sensor detects that a target is occluded, the current image is input into the query encoder as a query after the memory encoder is constructed.

(4) The mask propagation network builds a mask for the input query construction target, and the mask with the same size as the original query is obtained through decoder processing. Steps (3) and (4) are repeated for each frame where the occluded target is located.

(5) When the target ends the occlusion stage, the correlation between trajectories is continued based on the multi-hypothesis tracking algorithm.

Based on the steps above, the trajectory of the occluded target is generated by the mask propagation network. The relevant definition and network structure of the mask propagation network will be introduced in detail, and the workflow of mask propagation is shown in Figure 2.

We defined memory encoder as $E{n}_M$, query encoder as $E{n}_Q$, and mask decoder as $De$. We also defined $S_t^A$ as the set of masks of target A in the previous T frames before frame t, where trajectories are not occluded:

$$S_t^A=\{m_{t-T-1}^A,m_{t-T}^A,\dots ,m_{t-1}^A\}$$

(21)

We took a random target in frame t as an example to explain the mask propagation process. The input of the network was divided into two parts. In order to learn and memorize the coding pattern, the memory encoder $E{n}_M$ took the historical frames $I_j^{t-T-1:t-1}$ and $S_t^A$ as inputs, outputting $F_t^{E{n}_M}$, $K_t^{E{n}_M}$, and $V_t^{E{n}_M}$. $F_t^{E{n}_M}$, $K_t^{E{n}_M}$, $V_t^{E{n}_M}$ are the key-value pair set, key set, and value set corresponding to all the detections after being encoded. For a single frame input, the encoder $E{n}_M$ generated $f_t^{E{n}_M}$, $k_t^{E{n}_M}$, and $v_t^{E{n}_M}$. $f_t^{E{n}_M}$, $k_t^{E{n}_M}$, and $v_t^{E{n}_M}$ are the key-value pairs, keys, and values corresponding to the detections in frame t after being encoded.

In order to generate new masks, the query encoder $E{n}_Q$ processed the current frame $I_j^t$ and generated $f_t^{E{n}_Q}$, $k_t^{E{n}_Q}$, and $v_t^{E{n}_Q}$. In order to search based on the query key, the dot product was used between $k_t^{E{n}_Q}$ and $k_t^{E{n}_M}$. The method used normalization to obtain the similarity matrix $k_t^{QM}$:

$$k_t^{QM}=\mathrm{softmax}(k_t^{E{n}_Q}·k_t^{E{n}_M})$$

(22)

The similarity matrix $k_t^{QM}$ was multiplied by the memory values $v_t^{E{n}_M}$ to obtain the memory retrieval result matrix $f_t^{QM}$. By concatenating $f_t^{QM}$ with $v_t^{E{n}_Q}$, we obtained the mask feature matrix $f_t^{concat}$. We then inputted $f_t^{concat}$ in $De$ for mask refinement.

Eventually, the module restored $f_t^{concat}$ to the original image size.

3.3. Mask-Enhanced Multi-Object Tracking and Segmentation Framework

Despite recent progress in MOT tasks, existing frameworks struggle with persistent occlusion scenarios. Traditional MHT algorithms [5,14] rely on delayed decisions and cannot handle model occlusions dynamically. Bounding box-based representations lack accuracy for robust pixel-level associations.

Therefore, we introduced a Mask-Enhanced MHT-based Framework. The framework integrated mask geometry and hypothesis management based on the mask-conditional feature fusion network and the occlusion-aware propagation network. The components in the framework are shown in Figure 3.

Our method addressed three critical shortcomings of existing MHT: the occlusion-blind association, geometric inaccuracy of bounding boxes, and inefficiency of mask-based operations.

3.3.1. Mask-Integrated Multi-Hypothesis Tracking

The mask-integrated MHT method redefines traditional hypothesis management by incorporating mask shape and occlusion dynamics. The core of the method operates with a short tracklet $\mathcal{T}$, where each node in the hypothesis tree represents a sequence of mask-annotated detections rather than an isolated bounding box.The hypothesis generation process is governed by geometric validation criteria that evaluate the spatial–temporal compatibility of candidate tracklets. For any given pair of tracklets ${\mathcal{T}}_p$ and ${\mathcal{T}}_q$, their associated validity is determined by a mask crossing constraint:

$$\mathrm{valid}({\mathcal{T}}_p,\mathcal{T}q)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\displaystyle \frac{1}{L}}\sum _{k=1}^L\mathrm{maskIoU}(M_k^p,M_k^q)\ge {\tau }_g\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(23)

where ${\tau }_g$ is the threshold that ensures physical plausible associations. This constraint filters out conflicting hypotheses and allows legitimate occlusions, which is an improvement over bounding box based methods that often fail to distinguish overlapping targets. Temporal coherence is modeled by an exponential smoothing mechanism. Each hypothetical branch maintains a coherence score ${\psi }_t$, which is calculated as follows:

$${\psi }_t=\gamma {\psi }_{t-1}+(1-\gamma )\left({\displaystyle \frac{1}{L}}\sum _{k=t-L+1}^t{|f_k-{\overline{f}}_t|}_2\right)$$

(24)

Scores are adaptively weighted by the deviation ($f_k$) of recent features from the moving average ${\overline{f}}_t$, where the smoothing rate is controlled by $\gamma$. Branches that show sudden coherence drops are truncated as potential tracking failures. To handle dynamic occlusions, the framework integrates the occlusion state classification $o_t^A$ from Section 3.2. The hypothesis scoring function $\mathcal{S}\left(H_i^j\right)$ is augmented with an occlusion-aware term:

$$\mathcal{S}\left(H_i^j\right)={\mathcal{S}}_{base}+\lambda \left(o_t^A\right)·{\mathcal{S}}_{occ}$$

(25)

The ${\mathcal{S}}_{occ}$ term incorporates mask continuity metrics for occluded targets ($o_t^A=1$), motion consistency for occluded targets ($o_t^A=2$), and scene context similarity for scene occluded targets ($o_t^A=3$). This adaptive scoring ensures that the hypothesis is consistent with the observed occlusion patterns.

3.3.2. Mask-Enhanced Multi-Hypothesis Tracking Framework

The proposed framework integrates three core components to achieve MOTS in occluded scenarios. The proposed framework also establishes an integrated pipeline that performs MOTS with five interlinked processing stages.

The framework workflow follows the steps below:

• Short tracklet generation. Match background tracklets with foreground tracklets and generate short tracklet pairs. The tracklets to be matched come from detections generated by object detection and instance segmentation. These tracklet pairs have one-to-one correspondence with targets and are used for subsequent feature extraction and track association.
• Short tracklet representation. Based on the Mask-Conditioned Feature Fusion Network, extract the spatial–temporal features of target tracklet pairs to generate more robust features suitable for occluded scenarios. They are used for subsequent data association.
• Track association with multi-hypothesis tracking algorithm. For target A in frame t, if $o_t^A=0$ or $o_t^A=2$, update the tracker’s internal template using the MHT algorithm. Compute similarity between adjacent nodes through mask association evaluation and perform track association.
• Occlusion state perception. Use the crowded perception model to detect target occlusion states. If $o_t^A=0$ or $o_t^A=2$, perform data association as described in step (3). If $o_t^A=1$ or $o_t^A=3$, suspend template updates in the multi-hypothesis tracker and proceed to step (5) for occluded track completion.
• Mask propagation. Generate target masks in occluded regions with the mask propagation network from Section 3.2 to complete missing track segments. When the target disappears or $o_t^A=0$, return to step (3) for tracking with the MHT algorithm and complete the association of all track segments.

4. Experiments

The experiment was conducted on the public dataset MOTS to evaluate the effectiveness of the Mask-Enhanced Multi-Hypothesis Tracking Framework.

4.1. MOTS Dataset

MOTS [1] extends the MOT benchmark MOT17 [41] to pixel-level accuracy by providing instance mask annotation for pedestrians based on the MOT17 video sequences. MOTS uses frame-by-frame segmentation masks to provide more accurate annotation. The dataset contains four training videos and four test videos, for a total of 5906 frames. The dataset contains 556 tracked pedestrians, 59,163 are detected, and instance segmentation results are obtained for both the training and test sets. MOTS is a pixel-level multi-object tracking and segmentation performance evaluation dataset.

4.2. Implementation Details

The framework incorporates the mask-conditioned feature fusion network, the mask-integrated multi-hypothesis tracking algorithm, and the occlusion-aware mask propagation network, which are trained, respectively, following the experimental configurations below. The training data for the mask-conditioned feature fusion network comes from the MOT17 and KITTI [42] datasets. Foreground extraction and related preprocessing are performed using the Mask R-CNN instance segmentation detection method. The method is pretrained on the COCO trainval35k dataset [22].

The mask-integrated multi-hypothesis tracking algorithm is configured with the following parameters:

• Short trajectory pair length: 3.
• Average confidence filtering threshold for short trajectory pairs: 0.7.
• Pruning backtracking layers: 5.
• Target disappearance window length: 20.

The occlusion-aware mask propagation network uses a residual network [43] as the backbone of the encoder and is trained on the YouTube-VOS [36] and DAVIS-2017 [44] datasets.

4.3. Performance Metrics

This subsection introduces the main performance metrics which are used in the experiments below.

The evaluation of MOTS performance employs several key metrics. False negatives (FNs) quantify missed detections where true targets are not identified, and false positives (FPs) measure spurious detections of non-existent targets. Lower values for both metrics indicate better detection accuracy. Identity switches (IDSs) count occurrences where target identities are incorrectly swapped, directly reflecting tracking consistency. The MOT accuracy (MOTA) integrates the three factors above as a comprehensive accuracy measure. However, since MOTA has limited sensitivity to identity preservation, researchers typically complement it with the identity F1-score (IDF1) to specifically evaluate ID maintenance capability, which is important in crowded scenarios.

For segmentation-specific evaluation, MOTS accuracy (MOTSA) and MOTS precision (MOTSP) extend these metrics: MOTSA adapts the MOTA framework using mask areas for error calculation, and MOTSP focuses on geometric precision of segmentation boundaries through mask IoU. To further evaluate segmentation performance, the segmented MOT accuracy (sMOTSA) metric extends MOTSA by incorporating mask quality assessment. Specifically, sMOTSA computes accuracy based on correctly segmented pixels rather than entire masks, so it can provide finer-grained evaluation of segmentation quality.

All metrics follow ingthe convention that higher values indicate better performance, though their complementary nature must be noted. Low MOTA with high IDF1 may suggest excellent long-term tracking despite short-term detection deficiencies. These metrics collectively enable thorough evaluation across different aspects of tracking and segmentation quality.

4.4. Ablation Experiment

We conducted ablation experiments on the Occlusion-aware Mask Propagation Network using the MOTS training set. To investigate the optimal performance of linear interpolation, we analyzed the impact of different mask IoU thresholds on the method.

Since the MOTS20-05 and MOTS20-09 video sequences among the MOTS dataset were captured by a moving camera, the displacement of the same target across consecutive frames is relatively large. In contrast, for the remaining GT data, the mask IoU ratios of the same target in consecutive frames are mostly clustered around 0.9.

To determine the optimal value for the mask IoU threshold $\alpha$, we tested the impacts of different $\alpha$ values on the final results using the MOTS-02 sequence. The experimental results are presented in

Table 2. Tracking results with different mask IoU thresholds.

Mask IoU Threshold ($\mathit{\alpha }$)	MOTSA ↑	sMOTSA ↑	(FP + FN) ↓
$\alpha =0.8$	63.5	45.5	2509
$\alpha =0.85$	64.9	46.8	2410
$\alpha =0.9$	65.0	47.9	2392
$\alpha =0.95$	64.8	46.3	2452

.

It can be observed that when $\alpha$ = 0.9, the MOTSA, sMOTSA, and (FN + FP) metrics achieve relatively optimal results. Thus, we set the mask IoU threshold $\alpha$ of the linear interpolation algorithm to 0.9.

We conducted the ablation experiment on the MOTS training set, with the results shown in

Table 3. Ablation results on the MOTS training set.

Method	MOTSA ↑	MOTSP ↑	MT ↑
Baseline	74.9	86.7	42
+LI	74.7	86.5	110
+LI+OA	75.3	86.6	105
+OA+MP	75.4	87.0	110

. We define LI as the linear interpolation method, OA as the occlusion-aware sub-module, and MP as the mask propagation network. The comparative results between linear interpolation and mask propagation fully demonstrate the effectiveness of mask propagation in completing trajectories.

MT stands for Mostly Track targets. It is observed that after adding only the linear interpolation method, the MOTSA and other metrics of the baseline decreased, which means worse tracking performance. However, with the further addition of the occlusion-aware sub-module that identifies the occluded states of targets, the +LI+OA approach shows improvements in both MOTSA and MT metrics compared to the baseline. This proves the effectiveness of the occlusion-aware sub-module. Furthermore, after integrating OA with our proposed Robust Mask Propagation Network, which uses the mask propagation network to complete missing trajectories based on the occlusion-aware module, this method achieved a 0.5% improvement in MOTSA over the baseline and a significant 138% increase in MT. This indicates that the framework produces more complete target trajectories. Additionally, MOTSP improved by 0.3% and 0.4% compared to the baseline and +LI+OA, respectively, further enhancing the accuracy of our method.

4.5. Experiment Results

This subsection evaluates the proposed MOTS framework on the MOTS test set. Table 4 presents a comparison with state-of-the-art methods in multi-target tracking and segmentation, where our approach outperforms in MOTSA, MT, and FN metrics.

For the multi-object tracking and segmentation trajectory association algorithms compared in Table 4, the principles of the trackers are analyzed as follows.

ReMOTS_ [45] enhances tracking components through detection fusion and adaptive thresholding. EMNT [46] introduces a general recurrent tracking unit with multi-hypothesis framework for association modeling. GMPHD_MAF [10] incorporates pedestrian re-identification classifiers to filter outliers before trajectory association. MPNTrackSeg [47] employs differentiable message-passing networks based on network flow models. TrackRCNN [1] extends Mask R-CNN [32] with tracking branches using bipartite graph matching.

Table 4. Comparison results for the MOTS test set.

Tracker	MOTSA↑	IDF1↑	sMOTSA↑	MOTSP↑	MT↑	TP ↓	FP ↓	FN ↓
TrackRCNN [1]	55.2	42.4	40.6	76.1	127	19,628	1261	12,641
MPNTrackSeg [47]	73.7	68.8	58.6	80.6	207	25,036	1059	7233
GMPHD_MAF [10]	83.3	66.4	69.4	84.2	249	28,284	935	3985
EMNT [46]	83.7	77.0	70.0	84.1	234	27,943	666	4326
ReMOTS_ [45]	84.4	75.0	70.4	84.0	248	28,270	819	3999
Our Method	84.4	75.1	70.2	84.3	254	28,504	1043	3765

Table 4. Comparison results for the MOTS test set.

Tracker	MOTSA↑	IDF1↑	sMOTSA↑	MOTSP↑	MT↑	TP ↓	FP ↓	FN ↓
TrackRCNN [1]	55.2	42.4	40.6	76.1	127	19,628	1261	12,641
MPNTrackSeg [47]	73.7	68.8	58.6	80.6	207	25,036	1059	7233
GMPHD_MAF [10]	83.3	66.4	69.4	84.2	249	28,284	935	3985
EMNT [46]	83.7	77.0	70.0	84.1	234	27,943	666	4326
ReMOTS_ [45]	84.4	75.0	70.4	84.0	248	28,270	819	3999
Our Method	84.4	75.1	70.2	84.3	254	28,504	1043	3765

Our method detects occluded states by analyzing regression trends, extracts spatial–temporal features through mask propagation network, and achieves multi-target tracking and segmentation using a masked-enhanced MHT algorithm.

Notably, our method achieves a 0.7% higher MOTSA (84.4%) compared to the second-best tracker and reduces false negatives (FNs) by 6.1% (3765 vs. 3999). The results prove the framework’s superior accuracy in occluded scenarios. This improvement stems from the two key innovations: the mask propagation network’s ability to recover occluded targets by leveraging temporal coherence in spatial features and the fusion of mask-based association metrics with motion cues. The significant reduction in FN demonstrates the effectiveness of our mask propagation network, as the framework bridges gaps caused by transient occlusions successfully.

EMNT achieves higher IDF1 compared to our method (77.0% vs. our 75.1%), but its reliance on sequential feature updates leads to 15% higher FN (4326) in long-term occlusions, underscoring the advantage of our parallel mask-and-motion fusion design. Our method exhibits a trade-off in FP, primarily due to the sensitivity of mask propagation to abrupt motion changes in dynamic scenes. This limitation arises from the linear interpolation assumption in occluded regions, which may fail under rapid camera or target movements. Future work could integrate adaptive motion modeling to mitigate this issue.

The tracking framework proposed in these experiments shows that it can associate masks across frames accurately and achieves target tracking and segmentation while adapting to various scenarios, which contains severe occlusion scenarios by design. Our competitive sMOTSA (70.2%) and IDF1 (75.1%) scores, approaching those of SOTA performance (ReMOTS_: 70.4%; sMOTSA, 75.0% IDF1), validate the robustness of our method under heavy occlusion conditions. Tests conducted on public datasets have verified the effectiveness of this mask-enhanced framework.

To demonstrate the effectiveness of the proposed multi-object tracking and segmentation framework directly, Figure 4 shows the partial tracking results of our method on the MOTS dataset. In Figure 4a, for horizontally moving pedestrians under static camera, most of the mask IDs remain unchanged after mutual occlusion. In Figure 4b, the movement of the camera and pedestrians causes complete and partial occlusions jointly, resulting in fierce bounding box variations and mask variations within the detections, but our method still keeps the occluded targets’ mask IDs correctly.

Specifically, in Figure 4a, three targets under trees mutually occlude each other, yet our method maintains their mask IDs without failure. The tracker only loses track when one target crosses the camera view, causing prolonged full occlusion. This aligns with the quantitative results where our method reduces FN by 5.9% compared to ReMOTS_. Notably, even when severe occlusions occur between targets, our method successfully maintains the mask ID throughout the frame sequences. Figure 4b further highlights this capability: the green-masked target (initially heavily occluded by the red-masked one) is accurately segmented and tracked despite drastic mask deformation during occlusion. The ID remains stable after the red-masked target moves away, and the segmentation becomes more precise. Similarly, the yellow-masked target (initially non-occluded) retains the correct mask ID when later obscured by the red-masked person, proving our method’s adaptability to dynamic occlusion severity.

The performance data and visualization results shown above propose that MHT framework can associate masks across frames and achieve target tracking with segmentation, as well as that it is more adaptive to severe occluded scenarios.

5. Discussion

False positive, false negative, and target identity drift are often encountered in traditional MOT algorithms. One of the main reasons for this situation is the occurrence of occlusion. In this paper, we propose an optimization method for the error problem of trajectory association under occlusion scenarios with a mask-based MOTS framework, which achieves a better result for the multi-object tracking problem, as shown in Section 4.

However, the proposed method also has certain limitations. Several challenges still remain in multi-object tracking research. Although our experiments demonstrate that the mask-enhanced network improves trajectory association in occluded scenarios, the current feature network does not utilize inter-object interaction. Since most occlusion scenarios result from object interactions, incorporating interaction information with a feature extraction network is a promising research direction. Additionally, our framework employs several manually designed hyperparameter thresholds. Current scene-adaptive parameter trackers lack designs for occluded scenarios. The development of parameter adaptation methods for mask-based MOTS frameworks remains an essential yet understudied research direction.

6. Conclusions

This paper focuses on multi-object trajectory association in occluded scenarios. It uses instance segmentation to optimize trajectory association methods under occlusion conditions. We propose a mask-integrated multi-hypothesis tracking algorithm that achieves trajectory association through short tracklet pairs. The framework incorporates a mask-conditioned feature fusion network and an occlusion-aware mask propagation network that enable better trajectory association in occluded scenarios and enhance the robustness of the MOTS framework. The proposed mask-based framework is tested on public datasets, and the effectiveness is proved scientifically.