Dual-Linear Attention Network for Multi-Object Tracking and Segmentation

Abstract

Multi-object tracking and segmentation (MOTS) is a critical task in video analysis with applications spanning autonomous driving, robot navigation, and scene understanding. MOTS has made significant progress but still faces persistent challenges, such as crowded scenes, abnormal illumination, and small objects. Several trackers have implemented attention mechanisms to overcome these difficulties. However, many attention mechanisms have quadratic computational complexity and use little spatio-temporal information. This paper proposes a Dual-Linear Attention Network (DLAN), a novel approach that effectively integrates both appearance and spatio-temporal information while maintaining linear attention complexity. DLAN employs recursive linear self-attention to strengthen the appearance representation and prototypical linear cross-attention to condense rich spatio-temporal information, which can compensate for missing pixel information. DLAN optimizes both image features and segmentation, with the refined segmentation guiding frame-level memory updates to improve instance consistency. Extensive experiments on BDD100K MOT, BDD100K MOTS, and KITTI MOTS datasets demonstrate the following: (1) The three main challenges of object occlusion, illumination variation, and distant objects have been successfully mitigated by integrating DLAN. (2) DLAN has achieved an overall competitive performance when compared to state-of-the-art trackers, with a 26% reduction in identity switches (IDS) when compared to QDTrack-mots-fix.

1. Introduction

Multi-object tracking and segmentation (MOTS) is a derivative of multi-object tracking that combines detection, segmentation, and tracking into a unified task. It has great potential and can address important challenges in various real-world applications. Applications of MOTS are extensive and include, but are not limited to, intelligent transportation, autonomous driving, robot navigation, and scene understanding.

Popular tracking-by-detection methods rely on object-detection bounding boxes and perform well in object tracking. However, they face significant difficulties in crowded scenes with frequent occlusion. In such situations, bounding boxes from different instances may overlap considerably [1], leading to ambiguous target localization, and interference from occluding objects can obscure essential target information, ultimately degrading tracking performance. Given these obstacles, TrackR-CNN has emerged as a pioneering MOTS approach based on instance segmentation [2]. Unlike traditional tracking-by-detection methods, instance segmentation generates precise object contours, effectively mitigates occlusion effects, and provides more accurate delineation of object regions. Although TrackR-CNN pioneers the integration of instance segmentation into MOTS, its reliance on 3D convolutions introduces a computational overhead and limits its scalability in real-time applications.

Recently, attention-based methods have been successfully applied to address the three main challenges of object occlusion, illumination variation, and distant objects in MOTS. Sheng et al. [3] integrated the attention mechanism and feature aggregation sub-network to generate occlusion-robust features. SCATT [4] employed a cross-attention mechanism to fuse features between the original image and the search region, thus enhancing the tracker’s adaptability to intricate scenarios. LUXFormer [5] utilized illumination-guided sparse attention to improve performance under low-light conditions. Most of these methods are based on traditional attention mechanisms with task-specific adaptations to alleviate the tracking errors in complex environments. Fundamentally, these three main challenges can be attributed to the loss or degradation of the target pixel information. Occlusion and abnormal illumination remove or distort local target details, whereas increasing the target distance reduces the spatial resolution and pixel density. Richer spatio-temporal context and historical instance data have great potential to compensate for missing target information in the current frame, thereby mitigating the aforementioned challenges. Unfortunately, traditional attention fails to fully use historical spatio-temporal information and has quadratic complexity, which makes it difficult for the model to work with long videos [6,7].

Recognizing the crucial role of spatio-temporal information, many studies have extended the Transformer to better handle long videos. STARK [8] introduced a dynamically updated template obtained from a Transformer decoder. After obtaining high-quality short-term frame features using a temporally rich spatial feature extractor, LoViT [9] used a multi-scale temporal aggregator to capture long-term dependencies. STMT [10] employed 3D convolution to simulate the spatial and temporal interactions of objects, yielding comprehensive features in spatio-temporal data. These approaches gradually dilute the information from distant historical frames because they mainly rely on the network to learn the fusion weights between the current and previous frames. By contrast, clustering explicitly preserves early information and long-term dependencies during tracking. PCAN [11] aggregated similar object representations from historical frames into prototypes, with each prototype summarizing the distribution of all the previous frames. However, prototypes are easily contaminated by background noise [12], weakening their semantic integrity and degrading tracking performance, particularly in crowded or cluttered scenes.

Various linear-attention mechanisms have been proposed to improve efficiency. Linformer [13] achieved linear complexity by employing low-rank linear projections to approximate the traditional self-attention. However, visual features usually exhibit high-rank distributions, and low-rank approximation tends to compress discriminative spatial details, leading to reduced tracking performance. To address the low-rank limitation of linear attention, RAVLT [14] proposed a rank-augmented linear attention module that enhances the representational rank of feature matrices without increasing computational complexity. While RAVLT improves representational expressiveness with linear complexity, it still falls short in modeling comprehensive spatio-temporal correlations, especially in challenging multi-object tracking scenarios. To overcome these limitations, we propose a novel Dual-Linear Attention Network (DLAN) that combines appearance and spatio-temporal features. As shown in Figure 1, the dual-linear attention mechanism contains two key components: recursive linear self-attention (RLSA) and prototypical linear cross-attention (PLCA). The RLSA modifies the computational order of the key, value, and query to maintain linear attention complexity while strengthening the appearance feature within the key memory of the PLCA. The prototypes, derived from Gaussian components via EM-based clustering, represent a compact abstraction of spatio-temporal features in past frames. PLCA condenses reference features into prototypes to achieve linear complexity, whereas the memory clustering algorithm and the update mechanism are utilized to concentrate rich spatio-temporal information. Therefore, the DLAN can compensate for critical details that are missing in the current view but are available in past frames. This design alleviates the challenges posed by complicated scenarios, particularly when the target transitions from full visibility to partial occlusion, from normal to abnormal illumination, or from near to distant objects.

The DLAN integrates two key modules: the frame attention module (FAM) for feature reconstruction and the instance attention module (IAM) for segmentation optimization. FAM applies RLSA to enhance the appearance feature and uses PLCA to perform clustering on the historical frame image features. The prototypes generated through clustering, along with the current image feature, undergo cross-attention to produce a new feature that integrates both appearance and spatio-temporal information. Meanwhile, IAM performs clustering on both foreground and background features from historical and current frames, and uses online learning to achieve accurate instance segmentation predictions. Precise segmentation delineates the object boundaries more clearly, enabling the tracker to better distinguish targets from cluttered backgrounds and overlapping instances. Additionally, the IAM-refined segmentation is leveraged by a memory update mechanism to guide FAM such that only the historical features corresponding to foreground regions are stored. This prevents prototypes from being contaminated by background noise, preserves semantic consistency, and facilitates more reliable data association, ultimately mitigating challenges posed by complex scenes.

Evaluation experiments are conducted on three autonomous driving datasets: BDD100K MOT, BDD100K MOTS [15], and KITTI MOTS [2]. The results illustrate that DLAN demonstrates superior performance compared to the state-of-the-art methods. The primary contributions of the proposed method are summarized as follows.

• DLAN optimizes image features and refines the segmentation results while maintaining the linear complexity. The joint optimization of features and segmentation enables more discriminative representations that significantly enhance the robustness of multi-object association.
• Unlike existing linear-attention methods that struggle to preserve long-term historical context effectively, DLAN strengthens spatio-temporal appearance modeling by applying PLCA on RLSA-enhanced prototypes and updating cluster centers and memory. This design enables the network to compactly encode past spatio-temporal features and recover missing details in challenging scenarios.
• In contrast to prior prototype-based approaches, DLAN introduces a segmentation-guided memory update mechanism that filters noisy regions and prevents prototype contamination. The instance-to-frame reverse loop allows memory to store only valid object entities, improving data association and reducing identity switches in MOTS.

The remainder of this paper is structured as follows: Section 2 introduces related works and main theoretical innovations. Section 3 provides a detailed description of the proposed modules and network. Section 4 presents extensive experiments. Finally, Section 5 presents the conclusions of this study.

2. Related Works

This section reviews the current developments related to the fields and the main innovations involved in DLAN. Leveraging the power of deep learning, MOTS has recently achieved significant advancements in two primary processing features and segmentation approaches: feature fusion models and attention-based methods.

2.1. Multiple Object Tracking and Segmentation

Currently, MOTS adopts a two-stage framework. In the first stage, the algorithm performs object detection and instance segmentation, identifying and segmenting the objects of interest within the scene. The second stage focuses on inter-frame association and tracking, linking segmentations across frames to maintain a consistent object identity. The performance of MOTS relies heavily on the accuracy of the instance segmentation.

TrackR-CNN [2] integrated temporal information into the Mask R-CNN framework using two 3D convolutional layers. PointTrack++ [16] employed focal loss to train a semantic segmentation decoder and data augmentation strategies for instance segmentation. In contrast, SiamMask [17] enhanced the popular fully convolutional Siamese network with a binary segmentation loss. By integrating pixel-level memory with instance-level memory, MaskTrack [18] achieved remarkable stability for long-term video object segmentation.

Compared with these methods, DLAN’s online dynamic updating of various prototypes can more comprehensively utilize spatio-temporal information in all historical frames and enhance the generalization ability of the model.

2.2. Feature Fusion Models

The MOTS task requires multi-scale features and effective feature fusion to use information across layers. Feature fusion is the key to the deep utilization of complementary information across networks. This detailed integration of different feature representations improves the accuracy of object detection and segmentation and enhances the performance of MOTS.

The development of multimodal feature fusion techniques that utilize 2D pixels and 3D point cloud spatial relationships [19] has led to an increase in system complexity. On the other hand, single-modality feature fusion techniques focus on enhancing representations within a single feature space. For example, FFFN [20] achieved effective frame fusion by iteratively refining low-level representations with high-level information using a frame-by-frame feedback fusion mechanism. SiamMaskAttn [21] combined inverted residual structures with channel and spatial attention mechanisms. FFFN and SiamMaskAttn both provide limited modeling of long-range temporal interactions. To further incorporate temporal context, STCTrack [22] integrated the historical spatio-temporal information of the target through a spatio-temporal context network based on a multilayer residual ConvLSTM, but does not explicitly maintain global instance-level representations over long video sequences. To address the insufficient long-term temporal modeling, PCAN [11] used the Gaussian Mixture Model (GMM) to obtain various prototypes to condense complex spatio-temporal features. However, prototypes are susceptible to contamination from background noise, which undermines the semantic consistency of instances.

In contrast, DLAN supports feature fusion at both the frame and the instance levels by jointly modeling appearance and spatio-temporal information. Moreover, DLAN adopts a segmentation-guided memory update mechanism, where refined instance masks regulate the memory update process. By ensuring the memory retains only valid object entities, this instance-to-frame reverse pathway prevents background noise from contaminating prototypes.

2.3. Attention-Based Methods

The human brain prioritizes key areas with high information content to process all the available information. In recent years, attention mechanisms have been widely applied in deep-learning-based MOTS owing to their ability to highlight important features from the global receptive field.

TransTrack [23] set up a novel joint-detection-and-tracking paradigm that employs a Transformer architecture with dual decoders, aiming to leverage the query-key mechanism and reduce the risk of missing new objects. UnionTrack [24] introduced a scale-aware attention module to optimize the detection and embedding effects by applying channel and spatial attention mechanisms. EANTrack [25] strengthened tracking performance by introducing a feature-aware attention module within the prediction head, effectively boosting feature representation. RMOTS [26] presented a concise Transformer-based approach and performed feature fusion at both the decoder and encoder to fully utilize the benefits of the Transformer architecture. The deformable attention and encode-decode communication module designed in EAPT [27] can guarantee information communication between patches, thus efficiently extracting attention information from a global perspective.

However, the attention mechanisms of these methods often face challenges in handling long sequences owing to the quadratic computational complexity of dot-product matrices. In contrast, DLAN’s RLSA and PLCA reduce computational complexity while effectively leveraging spatio-temporal information to better address the challenges posed by object occlusion, illumination variation, and distant objects.

3. Method

As shown in Figure 2, DLAN combines RLSA and PLCA, which rely solely on the current frame and historical frames. Similarly to many advanced trackers, the DLAN employs Faster R-CNN [28] with FPN [29] for bounding box extraction and Mask R-CNN [30] for initial mask generation. Building upon this foundation, FAM and IAM are introduced to refine the frame and segmentation results. The refined segmentation result from IAM can be used to guide the memory update of FAM. Finally, the bi-softmax [31] association is utilized to generate the final tracking results.

The details of the dual-linear attention mechanism are elaborated in Section 3.1. The details of FAM, IAM, the update mechanism, and the loss function are further depicted in Section 3.2.

3.1. Dual-Linear Attention Mechanism

This study explores the application of the dual-linear attention mechanism to extract deep features from image data. As illustrated in Figure 3, the core of this mechanism lies in the synergy between RLSA and PLCA. RLSA improves the quality of the appearance features that are built into the key prototypes for PCAN. On the other hand, PLCA efficiently leverages spatio-temporal data by using both the current and previous features. The coordinated operation of RLSA and PLCA is therefore essential for achieving robust and accurate tracking performance.

Assume that the feature tensor has $B\times C\times H\times W$ dimensions, where $B$, $C$, $H$, and $W$ represent the batch size, number of channels, and height and width of the images. These features are first passed through fully connected layers to generate key vectors $\mathbf{K}$, value vectors $\mathbf{V}$, and query vectors $\mathbf{Q}$. Attention weights are then computed by applying softmax normalization to the dot product similarity between $\mathbf{Q}$ and $\mathbf{K}$. Finally, attention weights are used to compute the weighted sum of the $\mathbf{V}$, and obtain the updated feature representations. These formulas are as follows.

$$\mathbf{A}\left(\mathbf{Q},\mathbf{K},\mathbf{V}\right)=softmax\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\mathrm{T}}}{\sqrt{d}}}\right)\mathbf{V}$$

(1)

In this context, $\sqrt{d}$ represents the scaling factor, and $d$ denotes the model dimensions.

Traditional attention mechanisms have proven to be highly effective in many applications, particularly with Transformers [32,33,34]. The computational complexity of the dot-product is $O\left(C{N}^2\right)$, where $N=H\times W$. This quadratic complexity presents a challenge for segmentation tasks that require a high resolution and precision.

To reduce computational complexity and improve appearance details, we employ RLSA, a recursive linearization method that improves the traditional self-attention mechanism.

First, we aim to find a non-negative function $f\left(x\right)$ such that

$$\mathit{exp}\left({\displaystyle \frac{{\mathbf{Q}}_{\mathbf{i}}·{\mathbf{K}}_{\mathbf{j}}}{\sqrt{d}}}\right)\approx f\left({\mathbf{Q}}_{\mathbf{i}}\right)f^T\left({\mathbf{K}}_{\mathbf{j}}\right)$$

(2)

Substituting the approximation from Equation (2) into Equation (1) and applying the commutative property of matrix multiplication, we get the following formula:

$$\mathbf{S}{\mathbf{A}}_{\mathbf{i}}^{\mathbf{\prime }}={\displaystyle \frac{f\left({\mathbf{Q}}_{\mathbf{i}}\right){\sum }_j{f}^T\left({\mathbf{K}}_{\mathbf{j}}\right){\mathbf{V}}_{\mathbf{j}}}{f\left({\mathbf{Q}}_{\mathbf{i}}\right){\sum }_j{f}^T\left({\mathbf{K}}_{\mathbf{j}}\right)}}$$

(3)

Thus, the computational complexity is reduced to $O\left(N{D}_{\mathrm{K}}{D}_{\mathrm{Q}}\right)$, where $D_{\mathrm{K}}$ and $D_{\mathrm{Q}}$ denote the dimensions of the key vectors $\mathbf{K}$ and query vectors $\mathbf{Q}$. Because $D_{\mathrm{K}}$ and $D_{\mathrm{Q}}$ are significantly smaller than N, this results in a more efficient computation.

In this study, we employ the exponential linear unit ($\mathrm{E}\mathrm{L}\mathrm{U}$) activation function [35] to define the desired $f\left(x\right)$, where the hyperparameter $\alpha$ is set to 1 with the following specific form:

$$\mathrm{E}\mathrm{L}\mathrm{U}\left(x\right)=\left\{\begin{array}{cc}x,\hfill & \mathrm{if} x\ge 0\hfill \\ \alpha \left(\mathit{exp}\left(x\right)-1\right),\hfill & \mathrm{if} x<0\hfill \end{array}\right.$$

(4)

$$f\left(x\right)=\mathrm{E}\mathrm{L}\mathrm{U}\left(x\right)+1$$

(5)

A previous study [36] has shown that such functions have minimal impact on attention results. Equation (3) is rewritten as recursive Equation (6) to optimize the performance by reducing redundant computations.

$$\left\{\begin{array}{c}\begin{array}{c}{\mathbf{r}}_{\mathbf{0}}=0,{\mathbf{s}}_{\mathbf{0}}=0\hfill \\ {\mathbf{r}}_{\mathbf{j}}={\mathbf{r}}_{\mathbf{j}-\mathbf{1}}+f^T\left({\mathbf{K}}_{\mathbf{j}}\right){\mathbf{V}}_{\mathbf{j}}\hfill \\ {\mathbf{s}}_{\mathbf{j}}={\mathbf{s}}_{\mathbf{j}-\mathbf{1}}+f^T\left({\mathbf{K}}_{\mathbf{j}}\right)\hfill \\ {\mathbf{S}\mathbf{A}}_{\mathbf{i}}^{\mathbf{\prime }}={\displaystyle \frac{f\left({\mathbf{Q}}_{\mathbf{i}}\right){\mathbf{r}}_{\mathbf{j}}}{f\left({\mathbf{Q}}_{\mathbf{i}}\right){\mathbf{s}}_{\mathbf{j}}}}\hfill \end{array}\end{array}\right.$$

(6)

Next, we describe the PLCA module, which integrates appearance and spatio-temporal information based on GMM-based soft clustering. PLCA stores spatio-temporal features in the reference image memory.

To ensure continuous differentiability in the computation, this study employs the GMM to cluster both the value prototypical memory and the key prototypical memory.

First, assume that both ${\mathbf{K}}^{\mathbf{M}}$ and ${\mathbf{V}}^{\mathbf{M}}$ are composed of $N_c$ Gaussian distributions of $D$ dimensions, with a covariance matrix $\mathsf{\Sigma }={diag\left({\sigma }^2,{\sigma }^2,\dots ,{\sigma }^2\right)}_{D\times D}$, and that the cluster assignment probabilities are equal, that is, $p\left(z=c\right)={\displaystyle \frac{1}{N}},\left(c=1,\dots ,N_c\right)$.

The probability function of ${\mathbf{K}}^{\mathbf{M}}$ under each Gaussian distribution in the clusters is:

$$p\left({\mathbf{K}}^{\mathbf{M}}|z=c\right)={\displaystyle \frac{1}{{\left(2\pi {\sigma }^2\right)}^{\frac{D}{2}}}}exp\left(-{\displaystyle \frac{1}{2{\mathsf{\sigma }}^2}}{‖{\mathbf{K}}^{\mathbf{M}}-{\mathbf{k}}_{\mathbf{c}}^{\mathsf{\mu }}‖}^2\right)$$

(7)

in this context, ${\mathbf{k}}_{\mathbf{c}}^{\mathsf{\mu }}$ represents the $\mathrm{c}$-th cluster center, which is obtained using the EM algorithm.

Moreover, the GMM supports soft clustering by utilizing posterior probabilities to estimate the likelihood of ${\mathbf{K}}^{\mathbf{M}}$ belonging to various clusters, as defined in Equation (8), when the probability functions of each Gaussian distribution are known.

$$p\left(z=c|{\mathbf{K}}^{\mathbf{M}};{\mathbf{k}}_{\mathbf{c}}^{\mathsf{\mu }}\right)={\displaystyle \frac{exp\left(-\frac{1}{2{\sigma }^2}{‖{\mathbf{K}}^{\mathbf{M}}-{\mathbf{k}}_{\mathbf{c}}^{\mathsf{\mu }}‖}^2\right)}{\sum _{j=1}^{N_c}exp\left(-\frac{1}{2{\sigma }^2}{‖{\mathbf{K}}^{\mathbf{M}}-{\mathbf{k}}_{\mathbf{j}}^{\mathsf{\mu }}‖}^2\right)}}$$

(8)

Once the assignment probabilities of ${\mathbf{K}}^{\mathbf{M}}$ to each cluster are known, the key prototypes ${\mathbf{K}}^{\mathbf{p}}=\left\{{\mathbf{k}}_{\mathbf{c}}^{\mathbf{p}}\right\}$ can be computed using a weighted average as shown in Equation (9). Similarly, the value prototypes ${\mathbf{V}}^{\mathbf{p}}$ can be derived. By contrast, query vector $\mathbf{Q}$ presents the pixel-level features obtained after the unfolding of the current image features.

$${\mathbf{k}}_{\mathbf{c}}^{\mathbf{p}}=\sum _{j=1}^{N}p\left(z=c|{\mathbf{k}}_{\mathbf{j}}^{\mathbf{M}}\right){\mathbf{k}}_{\mathbf{j}}^{\mathbf{M}}$$

(9)

By transforming Equation (8), we see that $p\left(z=c|\mathbf{Q};{\mathbf{k}}_{\mathbf{c}}^{\mathbf{p}}\right)$ represents the cross-attention weight scores, calculated using SoftMax normalization on the squared Euclidean distance between the query vector $\mathbf{Q}$ and the key prototypes ${\mathbf{K}}^{\mathbf{p}}$, scaled by $-2{\mathsf{\sigma }}^2$. Thus, the PLCA output features ${\mathbf{C}\mathbf{A}}^{\prime }$ can be rewritten based on standard cross-attention as

$${\mathbf{C}\mathbf{A}}_{\mathbf{i}}^{\mathbf{\prime }}=\sum _{c=1}^{N_c}p\left(z=c|{\mathbf{q}}_{\mathbf{i}};{\mathbf{k}}_{\mathbf{c}}^{\mathbf{p}}\right){\mathbf{v}}_{\mathbf{c}}^{\mathbf{p}}=\sum _{c=1}^{N_c}{softmax}_c(-{\displaystyle \frac{1}{2{\sigma }^2}}{‖{\mathbf{q}}_{\mathbf{i}}-{\mathbf{k}}_{\mathbf{c}}^{\mathbf{p}}‖}^2){\mathbf{v}}_{\mathbf{c}}^{\mathbf{p}}$$

(10)

Overall, PLCA reduces the cross-attention complexity from quadratic $O\left(C{N}^2\right)$ to $O\left(CN{N}_c\right)$, since $N_c\ll N$. In addition, while traditional cross-attention must consider the entire sequence, our cross-attention only needs to focus on a limited number of prototypes.

The dual-linear attention mechanism utilizes kernel functions and prototypes to accomplish linear attention.

Table 1. Comparison of computational complexity and practical FLOPs of the with RLSA, PLCA, and traditional self-attention.

Method	Computational Complexity	N	FLOPs (B)
RLSA	$O\left(C^{2}N\right)$	64 $\times$ 64	1.21
		128 $\times$ 128	4.84
		256 $\times$ 256	19.37
PLCA	$O\left(CN{N}_c\right)$	64 $\times$ 64	0.29
		128 $\times$ 128	1.17
		256 $\times$ 256	4.70
Traditional Self-Attention	$O\left({CN}^2\right)$	64 $\times$ 64	19.41
		128 $\times$ 128	285.13
		256 $\times$ 256	4458.21

illustrates the computational complexity and practical FLOPs associated with RLSA, PLCA, and traditional self-attention. It is assumed that $D_{\mathrm{K}}$, $D_{\mathrm{Q}}$, and $C$ are all equal, which is a standard assumption found in the literature regarding attention. For the FLOPs calculation, $C$ is set to 256, which is a common channel count in the field of computer vision. $N_c$ is set to 35, based on the findings from the hyperparameter experiments outlined in Section 4.4. This comparison demonstrates that RLSA and PLCA excel in reducing computational complexity, with their FLOPs increasing linearly. As the sequence length $N$ grows, both methods use far fewer FLOPs than traditional self-attention.

3.2. Dual-Linear Attention Network

In this study, we propose a DLAN for the MOTS. The DLAN consists of two key modules: the FAM and IAM, which together process the frame features and optimize the instance segmentation.

3.2.1. Frame Attention Module

The FAM consists of the RLSA and PLCA. Specifically, RLSA enhances the historical appearance information in the key prototypical memory of PLCA. PLCA performs cross-attention between the current frame and the reinforced reference frame prototypes and subsequently updates the memory and clusters online. When the target undergoes occlusion, illumination degradation, or scale reduction, the FAM retrieves complementary evidence from historical frames, which often have a more complete and high-quality target appearance. By leveraging the synergy between RLSA and PLCA, the FAM integrates high-fidelity appearance features with a spatio-temporal context, yielding more robust feature representations and improving tracking stability.

As illustrated in Figure 4, the FAM extracts the key memory ${\mathbf{K}}^{\mathbf{M}}$ and value memory ${\mathbf{V}}^{\mathbf{M}}$ from the reference image. The key prototypes that undergo RLSA before the EM operation include detailed appearance features. The PLCA output feature is enhanced with spatio-temporal information by updating the cluster centers and prototypical memory. For the reference image in the (t − 1)-th frame, we apply the RLSA mechanism to construct the ${\mathbf{K}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathbf{M}}$. It then performs EM clustering on the ${\mathbf{V}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathbf{M}}$ and ${\mathbf{K}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathbf{M}}$ to generate value prototypes ${\mathbf{V}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathbf{p}}$ and key prototypes ${\mathbf{K}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathbf{p}}$. Frame-level PLCA is used for query embedding, ${\mathbf{K}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathbf{p}}$, and ${\mathbf{V}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathbf{p}}$ to generate new frame features. Finally, the new features and prototypes are updated in the memory and cluster centers. Finally, the new frame feature ${\mathbf{Y}}^{\prime }$ is defined based on the PLCA output result ${\mathbf{C}\mathbf{A}}^{\prime }$ and the original current frame feature, with a rate ${\lambda }_Y$, as follows:

$$\mathbf{Y}\mathit{’}={\lambda }_Y{\mathbf{C}\mathbf{A}}^{\mathbf{\prime }}+\left(1-{\lambda }_Y\right)\mathbf{Y}$$

(11)

3.2.2. Instance Attention Module

As shown in Figure 5, we use the IAM to refine the instance segmentation. The key branch of the PLCA module first undergoes RLSA to enhance its appearance representation and then performs cross-attention with the query feature, explicitly guiding the current frame to focus on the salient information in the reference frame. Unlike FAM, which works with image features, IAM focuses on two key prototypes for segmentation tasks: the key foreground prototypes ${\mathbf{K}}^{\mathbf{p}+}$ and the key background prototypes ${\mathbf{K}}^{\mathbf{p}-}$. Foreground prototypes enhance segmentation accuracy by focusing on object details and local regions, whereas background prototypes reduce interference from complex backgrounds. This combination improves the precision of foreground segmentation and yields sharper target boundary delineation, thereby increasing the data association accuracy and enhancing the tracker’s robustness in challenging environments.

We generate foreground and background representations via element-wise multiplication of the reference image features, current image features, and binary masks. As shown in Equations (12)–(14), the sigmoid function is used to obtain the binary masks in this study.

$$sigmoid\left(x\right)={\displaystyle \frac{1}{1+e^{-x}}}$$

(12)

$${\mathbf{m}\mathbf{a}\mathbf{s}\mathbf{k}}^{\mathbf{\prime }}=\left\{\begin{array}{c}1, \mathrm{if} sigmoid\left(\mathbf{m}\mathbf{a}\mathbf{s}\mathbf{k}\right)\ge 0.5\\ 0, \mathrm{if} sigmoid\left(\mathbf{m}\mathbf{a}\mathbf{s}\mathbf{k}\right)<0.5\end{array}\right.$$

(13)

$${\mathbf{f}\mathbf{e}\mathbf{a}\mathbf{t}}^{+}=\mathbf{f}\mathbf{e}\mathbf{a}\mathbf{t}\times {\mathbf{m}\mathbf{a}\mathbf{s}\mathbf{k}}^{\mathbf{\prime }}, {\mathbf{f}\mathbf{e}\mathbf{a}\mathbf{t}}^{-}=\mathbf{f}\mathbf{e}\mathbf{a}\mathbf{t}\times (1-{\mathbf{m}\mathbf{a}\mathbf{s}\mathbf{k}}^{\mathbf{\prime }})$$

(14)

Here, ${\mathbf{m}\mathbf{a}\mathbf{s}\mathbf{k}}^{\prime }$ represents binary masks, where 1 indicates that the original masks correspond to the foreground and 0 indicates the background. The ${\mathbf{f}\mathbf{e}\mathbf{a}\mathbf{t}}^{+}$ and ${\mathbf{f}\mathbf{e}\mathbf{a}\mathbf{t}}^{-}$ represent the foreground and background features, respectively.

In this operation, the reference foreground features are treated as the value source, the foreground and background features are treated as the key sources, and the current image features are treated as the query. Similarly to the FAM, the EM algorithm is performed sequentially on the value source to generate the value foreground prototypes ${\mathbf{V}}^{\mathbf{p}+}$. The foreground and background features of the reference frame are processed using the RLSA and EM algorithms to obtain the centers. These centers are then used by the next EM algorithm to generate the ${\mathbf{K}}^{\mathbf{p}+}$ and ${\mathbf{K}}^{\mathbf{p}-}$. According to Equation (10), the query combines with the ${\mathbf{K}}^{\mathbf{p}+}$ and ${\mathbf{K}}^{\mathbf{p}-}$ to calculate the foreground cross-attention weights ${\mathbf{Z}}^{+}$ and background cross-attention weights ${\mathbf{Z}}^{-}$. Finally, ${\mathbf{V}}^{\mathbf{p}+}$, the original current feature, ${\mathbf{Z}}^{+}$, ${\mathbf{Z}}^{-}$ and the original reference mask are fed into the temporal segmentation module (TSM) [11], producing the final instance segmentation correction prediction ${\mathbf{S}}^{\prime }$.

3.2.3. Update Mechanism

DLAN initializes the cluster centers only for the first frame of a sequence. For all subsequent frames, the clustering process does not restart from scratch. Instead, the cluster centers from the previous frame are used as the initialization for the current update. The initial cluster centers are sampled from a random Gaussian distribution, which serves as the prior for the following frames.

To ensure that the prototypes incorporate long-term spatio-temporal information, an update mechanism for cluster centers is required. In FAM, the cluster centers of the current frame are computed as a weighted combination of the previous frame’s prototypes and cluster centers. This design enables each frame’s cluster centers to progressively accumulate and encode all historical prototype information, as expressed by the following equation:

$${\mathbf{K}}_{\mathbf{t}}^{\mathsf{\mu }}=\lambda {\mathbf{K}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathbf{p}}+\left(1-\lambda \right){\mathbf{K}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathsf{\mu }},{\mathbf{V}}_{\mathbf{t}}^{\mathsf{\mu }}=\lambda {\mathbf{V}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathbf{p}}+\left(1-\lambda \right){\mathbf{V}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathsf{\mu }}$$

(15)

Similarly to FAM, the foreground and background cluster centers in IAM also adopt an update mechanism. The key difference is that IAM updates the current frame’s cluster centers using the previous frame’s average prototypes. The update process is formulated as follows:

$$\begin{gathered} {\mathbf{K}}_{\mathbf{t}}^{\mathsf{\mu }+}={\lambda \widehat{\mathbf{K}}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathbf{p}+}+\left(1-\lambda \right){\mathbf{K}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathsf{\mu }+},\hspace{1em}\hspace{1em}{\mathbf{K}}_{\mathbf{t}}^{\mathsf{\mu }-}=\lambda {\widehat{\mathbf{K}}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathbf{p}-}+\left(1-\lambda \right){\mathbf{K}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathsf{\mu }-}, \\ {\mathbf{V}}_{\mathbf{t}}^{\mathsf{\mu }+}=\lambda {\widehat{\mathbf{V}}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathbf{p}+}+\left(1-\lambda \right){\mathbf{V}}_{\mathbf{t}\mathbf{-}\mathbf{1}}^{\mathsf{\mu }+} \end{gathered}$$

(16)

during each iteration, the average prototype ${\widehat{\mathbf{K}}}^{\mathbf{p}}$ is used to update the key’s cluster centers ${\mathbf{K}}^{\mathsf{\mu }}$. Similarly, the average prototypes ${\widehat{\mathbf{V}}}^{\mathbf{p}+}$ are used to update the cluster centers ${\mathbf{V}}^{\mathsf{\mu }+}$. All cluster centers are updated at each iteration.

We observe that the optimized segmentation $\mathbf{S}\mathit{’}$ from the IAM provides reliable guidance for foreground-background separation. This is critical because prototypes, which are reconstructed from memory, often suffer from performance degradation due to background noise accumulation. Addressing this, we introduce a segmentation-guided memory update mechanism shown in Equation (17), where ${\mathbf{S}}^{\mathbf{\prime }}$ is used to filter the memory update process. This selective approach isolates foreground features, effectively shielding the prototypes from background disturbances.

$$\begin{gathered} {\mathbf{K}}_{\mathbf{t},\left(\mathbf{i},\mathbf{j}\right)}^{\mathbf{M}}=\lambda {{\mathbf{Y}}^{\mathbf{\prime }}}_{\mathbf{t}\mathbf{-}\mathbf{1},\left(\mathbf{i},\mathbf{j}\right)}{{\mathbf{S}}^{\mathbf{\prime }}}_{\mathbf{t}\mathbf{-}\mathbf{1},\left(\mathbf{i},\mathbf{j}\right)}+\left(1-\lambda \right){\mathbf{K}}_{\mathbf{t}\mathbf{-}\mathbf{1},\left(\mathbf{i},\mathbf{j}\right)}^{\mathbf{M}}, \\ {\mathbf{V}}_{\mathbf{t},(\mathbf{i},\mathbf{j})}^{\mathbf{M}}=\lambda {{\mathbf{Y}}^{\mathbf{\prime }}}_{\mathbf{t}\mathbf{-}\mathbf{1},\left(\mathbf{i},\mathbf{j}\right)}{{\mathbf{S}}^{\mathbf{\prime }}}_{\mathbf{t}\mathbf{-}\mathbf{1},\left(\mathbf{i},\mathbf{j}\right)}+\left(1-\lambda \right){\mathbf{V}}_{\mathbf{t}\mathbf{-}\mathbf{1},(\mathbf{i},\mathbf{j})}^{\mathbf{M}} \end{gathered}$$

(17)

3.2.4. Loss Function

The loss function of our tracker is a linear sum of three terms:

$$\mathcal{L}={\mathcal{L}}_{det}+{\mathcal{L}}_{mask}+{\mathcal{L}}_{track}$$

(18)

In our framework, detection is performed using Faster R-CNN [28] with an FPN backbone. As a two-stage detector, Faster R-CNN employs a Region Proposal Network (RPN) to generate candidate regions. The overall detection loss ${\mathcal{L}}_{det}$ can be expressed as:

$${\mathcal{L}}_{det}={\mathcal{L}}_{rpn}+{\alpha }_1{\mathcal{L}}_{cls}+{\beta }_1{\mathcal{L}}_{reg}$$

(19)

where ${\mathcal{L}}_{rpn}$ denotes the RPN loss, and ${\mathcal{L}}_{cls}$ and ${\mathcal{L}}_{reg}$ correspond to the classification and bounding-box regression losses, respectively. All components follow the original definitions [28], with the loss weights ${\alpha }_1$ and ${\beta }_1$ set to 1.0.

The IAM is designed to refine the mask predictions obtained from Mask R-CNN [30]. Therefore, the mask loss ${\mathcal{L}}_{mask}$ follows the same formulation as in Mask R-CNN, using an average binary cross-entropy loss with a per-pixel sigmoid activation.

Finally, in the track head, we adopt quasi-dense similarity learning [31] to obtain more discriminative instance embeddings for reliable cross-frame association. The tracking loss ${\mathcal{L}}_{track}$ is formulated as:

$${\mathcal{L}}_{track}={\alpha }_2{\mathcal{L}}_{embed}+{\beta }_2{\mathcal{L}}_{aux}$$

(20)

the embedding loss ${\mathcal{L}}_{embed}$ is computed using a non-parametric SoftMax with cross-entropy over feature similarities and averaged across all training samples. The auxiliary loss ${\mathcal{L}}_{aux}$ adopts L2 loss. The loss weights ${\alpha }_2$ and ${\beta }_2$ are set to 0.25 and 1.0.

The DLAN and the EM algorithm are summarized in Algorithms 1 and 2.

Algorithm 1: Inference Pipeline of Dual-Linear Attention Network

Inputs:

$\mathbf{X}$: image metadata;

$N_c$: number of prototypes and clusters;

${\mathbf{K}}^{\mathsf{\mu }}$, ${\mathbf{V}}^{\mathsf{\mu }}$: cluster centers for the frame-level EM algorithm;

$\mathsf{\lambda }$: update rates of the memory and cluster centers;

${\lambda }_Y$: update rate of the feature;

${\mathbf{K}}^{\mathsf{\mu }+}$$, {\mathbf{K}}^{\mathsf{\mu }-}$$, {\mathbf{V}}^{\mathsf{\mu }+}$: cluster centers for the Instance-level EM algorithm;

Outputs:

$\mathbf{D}$: identity results;

$\mathbf{S}$: refined segmentation result;

$\mathbf{C}$: category results.

1. for (frame = 1 to last) do

2.  Using ResNet50 to extract the original frame features $\mathbf{Y}$$\mathrm{of} \mathbf{X}$;

Frame attention module

3.  If (frame =1) then

4.  Initialize the value memory ${\mathbf{V}}^{\mathbf{M}}$$\mathrm{as} \mathbf{Y}$;

5.  Initialize the key memory ${\mathbf{K}}^{\mathbf{M}}$ based on RLSA (${\mathbf{Y}}_{\mathbf{t}}$) using Equation (6);

6.  Initialize the cluster centers ${\mathbf{K}}^{\mathsf{\mu }}$$, {\mathbf{V}}^{\mathsf{\mu }}$$, {\mathbf{K}}^{\mathsf{\mu }+}$$, {\mathbf{K}}^{\mathsf{\mu }-}$$, {\mathbf{V}}^{\mathsf{\mu }+}$: 7.    for each center $i$$= 1 \mathrm{to} N_c$ do 8.    Sample center vector $i\leftarrow N(0, {\displaystyle \frac{2}{N_c}})$; 9.    Normalize $i$ to unit L2norm; 10.  end for

11. end if

12. The key prototypes ${\mathbf{K}}^{\mathbf{p}}=EM({\mathbf{K}}^{\mathbf{M}},{\mathbf{K}}^{\mathsf{\mu }})$, the value prototypes ${\mathbf{V}}^{\mathbf{p}}\leftarrow EM({\mathbf{V}}^{\mathbf{M}},{\mathbf{V}}^{\mathsf{\mu }})$;

13. The PLCA output features ${\mathbf{C}\mathbf{A}}^{\mathbf{\prime }}\leftarrow SoftMax\left(\mathbf{Y}{·\mathbf{K}}^{\mathbf{p}},\mathrm{d}\mathrm{i}\mathrm{m}=N_c\right){\mathbf{V}}^{\mathbf{p}};$

14. Obtain new frame-level features ${\mathbf{Y}}^{\mathbf{\prime }}⟵{\lambda }_Y{\mathbf{C}\mathbf{A}}^{\mathbf{\prime }}+(1-{\lambda }_Y\mathbf{Y})$;

15. Update cluster centers ${\mathbf{K}}^{\mathsf{\mu }}$$, {\mathbf{V}}^{\mathsf{\mu }}$ using Equation (15);

Object detection and original segmentation

16. Based on ${\mathbf{Y}}^{\mathbf{\prime }}$, utilize Faster R-CNN with FPN for object detection and employ Mask R-CNN for segmentation to obtain $\mathbf{C}$, the soft mask $\mathbf{S}$, and their feature $\mathbf{F}$;

Instance attention module

17. Compute reference and current binary masks with Equation (13);

18. Get the foreground and background features of images using Equation (14);

19. Obtain value prototypes ${\mathbf{V}}^{\mathbf{p}+}=EM({\mathbf{F}}_{\mathbf{r}\mathbf{e}\mathbf{f}}^{+},{\mathbf{V}}^{\mathsf{\mu }+})$;

20. The key foreground prototypes ${\mathbf{K}}^{\mathbf{p}+}\leftarrow EM({\mathbf{F}}^{+}, EM(RLSA\left({\mathbf{F}}_{\mathbf{r}\mathbf{e}\mathbf{f}}^{+}\right),{\mathbf{K}}^{\mathsf{\mu }+}\left)\right)$; 21. The key background prototypes ${\mathbf{K}}^{\mathbf{p}-}\leftarrow EM({\mathbf{F}}^{-}, EM(RLSA\left({\mathbf{F}}_{\mathbf{r}\mathbf{e}\mathbf{f}}^{-}\right),{\mathbf{K}}^{\mathsf{\mu }-}\left)\right)$;

22. Calculate the foreground and background cross-attention weights: 23. ${\mathbf{Z}}^{+}\leftarrow SoftMax\left({\mathbf{F}}^{+}·{\mathbf{K}}^{\mathbf{p}+},\mathrm{d}\mathrm{i}\mathrm{m}=N_c\right)$$, {\mathbf{Z}}^{-}\leftarrow SoftMax\left({\mathbf{F}}^{-}·{\mathbf{K}}^{\mathbf{p}-},\mathrm{d}\mathrm{i}\mathrm{m}=N_c\right)$;

24. Execute the TSM on the reference mask, ${\mathbf{Z}}^{+}$$, {\mathbf{Z}}^{-}$, current features, and ${\mathbf{V}}^{\mathbf{p}+}$ to obtain the refined segmentation results ${\mathbf{S}}^{\mathbf{\prime }}$;

25. Update cluster centers ${\mathbf{K}}^{\mathsf{\mu }+}$, ${\mathbf{K}}^{\mathsf{\mu }-}$, ${\mathbf{V}}^{\mathsf{\mu }+}$ for the next frame according to Equation (16);

Memory Update Mechanism 26. for each spatial location ($i, j$) do 27.  ${\mathbf{K}}_{(\mathbf{i},\mathbf{j})}^{\mathbf{M}}\leftarrow \lambda {\mathbf{Y}}_{(\mathbf{i},\mathbf{j})}^{\mathbf{\prime }}{{\mathbf{S}}^{\mathbf{\prime }}}_{\left(\mathbf{i},\mathbf{j}\right)}+\left(1-\lambda \right){\mathbf{K}}_{\left(\mathbf{i},\mathbf{j}\right)}^{\mathbf{M}}$; 28.  ${\mathbf{V}}_{(\mathbf{i},\mathbf{j})}^{\mathbf{M}}\leftarrow \lambda {\mathbf{Y}}_{(\mathbf{i},\mathbf{j})}^{\mathbf{\prime }}{{\mathbf{S}}^{\mathbf{\prime }}}_{\left(\mathbf{i},\mathbf{j}\right)}+\left(1-\lambda \right){\mathbf{V}}_{\left(\mathbf{i},\mathbf{j}\right)}^{\mathbf{M}}$; 29. end for

Association and tracking

30. Based on ${\mathbf{S}}^{\mathbf{\prime }}$, and $\mathbf{C}$, the tracker associates data and returns $\mathbf{D}$ as the output;

31. end for

Algorithm 2: EM Algorithm

Inputs:

$\mathbf{F}\in R^{C\times N}$: feature input;

$\mathsf{\mu }\in R^{C\times N_c}$: cluster centers;

$\mathrm{T}$: number of EM iterations;

Outputs:

$\mathrm{P}$: prototypes.

1. for (t = 1 to T) do

2.  E-Step: Compute Assignment Probabilities

3.  $W\leftarrow Softmax(\mathbf{F}·\mathsf{\mu },\mathrm{d}\mathrm{i}\mathrm{m}=N_c)$;

4.  $W\leftarrow L1Normalize(W,\mathrm{d}\mathrm{i}\mathrm{m}=N)$;

5.  M-step: Update Prototypes

6.  $\mathrm{P}\leftarrow L2Normalize(\mathbf{F}\mathbf{W},\mathrm{d}\mathrm{i}\mathrm{m}=C)$; 7. end for

4. Experiments

This section presents the implementation details, qualitative results, state-of-the-art comparison, and an ablation study of DLAN on the BDD100K MOT, BDD100K MOTS, and KITTI MOTS datasets. These three video datasets present significant computational challenges owing to their extended temporal sequences that contain numerous occlusion scenarios. The complex interaction patterns between objects and frequent environmental obstructions make them suitable for evaluating long-term tracking robustness, particularly when targets experience prolonged disappearance or undergo substantial appearance transformations. To quantitatively assess the multi-object tracking and segmentation performance under these conditions, we adopt MOTS metrics, which comprehensively measure the tracker performance.

4.1. Experiment Set Up

To ensure a fair evaluation, the parameters of the trackers are maintained in their original settings, as provided in their official versions. All the tests are conducted on the same platform.

4.1.1. Datasets

The BDD100K dataset is a large-scale, open-source resource widely used in autonomous driving research. It includes BDD100K MOT and BDD100K MOTS datasets. The MOT dataset consists of 1400 videos for training, 400 videos for testing, and 200 videos for validation. The MOTS dataset consists of 154 videos and 30,971 images for training, 37 videos and 7521 images for testing, and 32 videos and 6507 images for validation. The BDD100K dataset covers a variety of driving environments, including different times of day, weather conditions, and road types. It also provides annotations for eight object categories, including pedestrians, riders, and cars.

The KITTI MOTS dataset is an essential resource for computer vision and autonomous driving, particularly for testing and improving the MOTS algorithms. Building on the bounding box annotations from the KITTI MOTS tracking dataset, it includes instance segmentation annotations and divides the data into training and validation sets. The KITTI MOTS dataset contains 12 videos and 5039 images for training, 29 videos and 11,124 images for testing, and 9 videos and 2990 images for validation.

4.1.2. Evaluation Criteria

A confusion matrix from the MOTS domain is employed to investigate the relationship between the segmentation tracking results and the ground truth masks. A mask detected by the tracker is classified as a true positive ($\mathrm{T}\mathrm{P}$) if it overlaps the ground truth. Masks with no overlap are classified as false positives ($\mathrm{F}\mathrm{P}$s), whereas masks present in the ground truth but not detected by the tracker are considered false negatives ($\mathrm{F}\mathrm{N}$s). Masks that are neither part of the ground truth nor detected by the tracker are classified as true negatives ($\mathrm{T}\mathrm{N}$s). Two key MOTS performance metrics are derived from this confusion matrix: MOTS accuracy ($\mathrm{M}\mathrm{O}\mathrm{T}\mathrm{S}\mathrm{A}$) and MOTS precision ($\mathrm{M}\mathrm{O}\mathrm{T}\mathrm{S}\mathrm{P}$). When tracking with multiple classes, we calculate the MOTA for each class separately and then average the results over the total number of classes (mMOTA).

$$\mathrm{M}\mathrm{O}\mathrm{T}\mathrm{S}\mathrm{P}={\displaystyle \frac{\widetilde{\mathrm{T}\mathrm{P}}}{\left|\mathrm{T}\mathrm{P}\right|}}$$

(21)

$$\mathrm{M}\mathrm{O}\mathrm{T}\mathrm{S}\mathrm{A}=1-{\displaystyle \frac{\left|\mathrm{F}\mathrm{N}\right|+\left|\mathrm{F}\mathrm{P}\right|+\left|\mathrm{I}\mathrm{D}\mathrm{S}\right|}{\left|\mathrm{M}\right|}}$$

(22)

In this context, $\widetilde{\mathrm{T}\mathrm{P}}$ denotes the sum of the IoU values for all objects in $\mathrm{T}\mathrm{P}$, $\mathrm{M}$ denotes the total number of ground truth masks, and $\mathrm{I}\mathrm{D}\mathrm{S}$ denotes the number of identity switches.

In MOTS tasks, in addition to using $\mathrm{I}\mathrm{D}\mathrm{S}$ to count the number of identity switches, the $\mathrm{I}\mathrm{D}\mathrm{F}1$ score can also be employed to measure an algorithm’s ability to maintain identity consistency. $\mathrm{I}\mathrm{D}\mathrm{F}1$ is the harmonic mean of the identity accuracy and identity recall, providing a comprehensive assessment of how well the algorithm preserves object identities during tracking. Additionally, floating-point operations (FLOPs) and frames per second (FPS) are adopted as the primary metric for assessing computational efficiency.

To perform fine-tuning of the hyperparameters, we adopt standard object detection and segmentation metrics. mAP is calculated as the average of the precision scores across different recall levels, providing a comprehensive measure of the model’s overall performance. mAP50 and mAP75 represent the mean precision at stricter thresholds of 0.5 and 0.75 for intersection over union, respectively. These metrics are commonly used in object tracking and segmentation tasks to assess the accuracy of detected objects with varying levels of localization precision.

4.1.3. Implementation Details

Training Configuration. We train our network in an Ubuntu 20.04 environment (CUDA 11.0, Python 3.8, PyTorch 1.7) with one NVIDIA RTX A6000 GPU (48 GB). The fixed random seed is set to 166. For the feature extraction, we utilize a ResNet-50 with FPN as the backbone and begin the training process by applying ImageNet-1K pre-trained weights to the base model. The detection head is based on Faster R-CNN with FPN and initialized with the QDTrack model pre-trained on the BDD100K object detection dataset, while the segmentation head utilizes Mask R-CNN. The tracking head employs a quasi-dense matching head with a bi-softmax matching strategy [31].

Optimizer and Scheduler. We optimize the network for 12 epochs and 2 batch sizes using SGD with an initial learning rate of 10⁻², momentum of 0.9, and weight decay of 10⁻⁴. In addition, we adopt gradient clipping (max-norm 0.5) to ensure stable optimization. A step learning rate schedule is adopted with linear warm-up for the first 10³ iterations, and the learning rate is decayed at epochs 8 and 11.

Datasets. On KITTI MOTS and BDD100K, we train the model on the respective dataset unless otherwise noted, following the official data splits. In addition, all annotations are converted to the COCO format for unified training and evaluation. For data preprocessing, the image sizes of BDD100K and KITTI MOTS are resized to 1296 × 720 and 1248 × 384. All images are then normalized and padded to ensure that both spatial dimensions are divisible by 32. For data augmentation, we only apply random horizontal flipping.

Model Settings. The channel number of feature maps is set to 256. The EM algorithm used in both FAM and IAM is based on EMA [37] with 10 iterations. The number of frame-level prototypes, as well as the instance-level foreground and background prototypes, is set to 35 for each. The memory and center update rates are set to 0.5. The feature update rate is set to 0.25.

4.2. Qualitative Results

Object occlusion, illumination variation, and distant objects have long been the main challenges in the MOTS field. The following section discusses the tracking performance of DLAN in addressing these three challenges compared with QDTrack.

Object Occlusion. MOTS tasks are invariably challenged by occlusion scenes on busy streets. The tracker is more likely to experience IDS errors when the object to be tracked is partially or completely obscured by other objects or interference. These scenarios are best analyzed using the BDD100K MOTS datasets. The DLAN dramatically improves the regions where IDS errors occur for both vehicles and pedestrians, as shown in Figure 6, which eventually enhances the tracking performance.

Illumination Variation. Different lighting conditions can be produced in daily driving situations using elements such as sunlight or shadows. As shown in Figure 7, QDTrack generates a sizable number of missed object detections when abrupt direct sunlight during the day causes overexposed areas. Whether on open roads or busy streets at night, QDTrack frequently confuses the background with an object. However, the DLAN’s benefit of extracting background and foreground prototypes can help solve these problems. Therefore, the negative effects of illumination variation can be reduced using the proposed DLAN.

Distant Objects. The distance between an object and a sensor is inversely proportional to its apparent size. It may be difficult to detect and segment small objects at certain distances. The tracker might incorrectly segment a single object into multiple masks or classify distant non-target objects as cars or pedestrians, as illustrated in Figure 8. However, the DLAN utilizes rich historical spatio-temporal and appearance features to successfully correct errors in tracking distant objects.

4.3. State-of-the-Art Comparison

This paper compares DLAN with several advanced algorithms in MOTS, including EAMSN [38], BoxMOTS [39], CML-MOTS [40], UNINEXT-L [41], QDTrack [31], MaskFreeVIS [42], MOTRv2 [43], MOTR [44], ByteTrack [45], VMT [46], Unicorn [47], DG-Labeler [48], DIOR [49], PCAN [11], ReMOTS [50], MOTS R-CNN [51], MOTSNet [52], PointTrack-U [53], STEM-Seg [54], MOTSFusion [55], CenterTrack [56], TrackR-CNN [2], MaskTrackRCNN [57], BePix [58], and SortIoU [59] using the validation sets from BDD100K and KITTI MOTS.

Table 2. Comparison of different networks on the KITTI MOTS validation set.

Trackers	MOTSA ↑		MOTSP ↑		IDS ↓
	Car	Ped	Car	Ped	Car	Ped
DLAN (Ours)	90.5	80.1	90.6	82.1	20	18
EAMSN [38]	90.4	69.2	90.5	79.0	24	29
BoxMOTS [39]	68.4	58.3	86.4	72.4	—	—
CML-MOTS [40]	88.2	65.3	88.5	76.1	62	32
DG-Labeler [48]	90.7	—	83.4	—	58	50
DIOR [49]	87.4	80.3	88.1	81.5	649	611
PCAN [11]	89.6	66.4	88.3	76.1	—	—
ReMOTS [50]	84.8	-	-	-	231	-
MOTS R-CNN [51]	90.1	67.3	-	-	35	30
MOTSNet [52]	87.2	69.3	89.6	79.7	—	—
STEM-Seg [54]	83.8	66.1	87.2	77.7	76	19
PointTrack-U [53]	89.9	75.6	90.1	81.4	23	27
MOTSFusion (2D) [55]	89.2	66.6	-	-	85	53
CenterTrack [56]	88.9	65.7	-	-	31	25
TrackR-CNN [2]	87.8	65.1	87.2	75.7	93	78
BePix [58]	89.7	-	86.5	-	88	-

Note: “—“ indicates that the metric is not reported in the original work. The bold numbers highlight the results achieved by our tracker for clearer comparison. Red and blue indicate the best and second-best performance. MOTSA/MOTSP (higher is better ↑) and IDS (lower is better ↓), are used for evaluation.

shows the results of the KITTI MOTS validation set for cars and pedestrians in terms of the MOTSA, MOTSP, and IDS metrics. In this validation set, DLAN achieves outstanding performance across all key metrics. These results demonstrate that DLAN has exceptional capabilities in terms of tracking accuracy and identity consistency on the challenging KITTI MOTS validation dataset.

The KITTI MOTS dataset focuses primarily on two categories: cars and pedestrians. By contrast, the BDD100K MOTS dataset encompasses a broader range of eight categories, offering a more comprehensive evaluation.

As shown in

Table 3. Comparison of different networks on the BDD100K MOTS validation set.

Trackers	mMOTSA ↑	mMOTSP ↑	mIDF1 ↑	IDS ↓
DLAN (Ours)	32.1	67.4	46.9	724
BoxMOTS [39]	12.6	57.5	20.9	1423
QDTrack-mots-fix [31]	23.5	66.3	44.5	973
UNINEXT-L [41]	32.0	60.2	45.4	1634
VMT [46]	28.7	67.3	45.7	825
Unicorn [47]	29.6	67.7	44.2	1737
Unicorn [47] + MaskFreeVIS [42]	23.8	66.7	44.9	2086
PCAN [11]	27.4	66.7	45.1	876
STEM-Seg [54]	12.2	58.2	25.4	8732
MaskTrackRCNN [57]	12.3	59.9	26.2	9116
SortIoU [59]	10.3	59.9	21.8	15951

Note: The bold numbers highlight the results achieved by our tracker for clearer comparison. Red and blue indicate the best and second-best performance. mMOTSA/mMOTSP/mIDF1 (higher is better ↑) and IDS (lower is better ↓), are used for evaluation.

, we apply DLAN to the BDD100K MOTS validation sets to assess and compare metrics, including mMOTSA, mMOTSP, mIDF1, and IDS. The results demonstrate that DLAN excels in several key metrics, including mMOTSA, mIDF1, and IDS, and achieves the best performance across the board. Notably, the improvement in the IDS metric is particularly striking, showing a remarkable 26% reduction compared to QDTrack-mots-fix. This reduction is a clear indicator of DLAN’s exceptional ability to maintain identity consistency across tracks. By significantly lowering the number of IDS, the DLAN not only improves the tracking accuracy but also enhances the reliability and robustness of the tracking system. Additionally, in

Table 4. Comparison of different networks on the BDD100K MOT validation set.

Trackers	mMOTA ↑	mIDF1 ↑	MOTA ↑	IDF1 ↑
DLAN (Ours)	45.7	57.1	69.8	73.9
QDTrack [31]	42.1	54.3	68.2	73.3
MOTRv2 [43]	43.6	56.5	65.6	72.7
MOTR [44]	32.3	44.8	56.2	65.8
ByteTrack [45]	45.5	54.8	69.1	70.4
Unicorn [47]	41.2	54.0	66.6	71.3

Note: The bold numbers highlight the results achieved by our tracker for clearer comparison. Red and blue indicate the best and second-best performance. mMOTSA/mIDF1/MOTA/IDF1 (higher is better ↑) are used for evaluation.

, we present a comparison of the tracking performance of DLAN with state-of-the-art MOT trackers on the BDD100K MOT dataset. The results demonstrate that DLAN achieves leading performance in MOT, further validating its effectiveness.

4.4. Ablation Study and Analysis

This section presents a series of ablation experiments to investigate and validate the impact of attention models and the main hyperparameter settings (prototype number, EM iteration number, and update rates) on the performance of the DLAN algorithm. Unless otherwise specified, all ablation experiments strictly follow the same training configuration as described in Section 4.1.3, and only the ablated component is modified.

Effect of Attention Models and Memory Update Mechanism. We conduct an ablation study on the attention models and the memory update mechanism listed in

Table 5. Ablation of different models on the BDD100K MOTS validation set.

Trackers	mMOTSA ↑	mMOTSP ↑	mIDF1 ↑	IDS ↓
DLAN(Ours)	32.1	67.4	46.9	724
w/o Memory Update	30.3	66.8	45.7	794
w/o Memory Update and FAM	28.7	66.7	45.5	823
w/o Memory Update and IAM	26.5	66.5	45.3	869
w/o IAM and FAM	23.5	66.3	44.5	973

Note: The bold numbers highlight the results achieved by our tracker for clearer comparison. mMOTSA/mMOTSP/mIDF1 (higher is better ↑) and IDS (lower is better ↓), are used for evaluation.

. We find that sequentially removing the memory update mechanism, the FAM, and the IAM leads to a decline in the four main metrics, with the most significant impact observed on mMOTSA and IDS. This confirms that all proposed components are crucial for the tracker’s success. As shown in Figure 9, we present the attention heatmaps of FAM with and without the segmentation-guided memory update mechanism. It is evident that prototypes are highly susceptible to background noise, which causes the attention mechanism to focus on irrelevant information. However, with the guidance of the previous frame’s IAM-refined segmentation, the memory selectively updates the feature corresponding to the foreground region, significantly improving the attention results of FAM.

Qualitative Ablation about FAM. We analyze the role of the FAM in the network through an ablation approach that involves either removing or applying the FAM. In Figure 10, four sets of comparative results are presented. The first set indicates that FAM optimizes the missed detection of densely occluded objects. The second set demonstrates that employing the FAM enhances the detection of distant small objects. The third set reveals that in the absence of FAM, the tracker incorrectly detects and recognizes large plants at night as vehicles. In the final set, without the FAM, the tracker mistakenly identifies barriers to elevated structures as vehicles. The results indicate that the FAM leverages historical information to compensate for missing pixels in the current frame, effectively enhancing the tracker’s detection and recognition capabilities in complex scenarios.

Qualitative Ablation about IAM. Figure 11 shows the qualitative ablation results of instance segmentation in the DLAN. Compared with the model without the IAM, the results demonstrate that incorporating the IAM enhances the mask accuracy. This improvement underscores the critical role of IAM in enhancing segmentation precision. In Figure 12, we compare the original segmentation loss with the optimized segmentation loss after applying IAM in a specific iteration. The results indicate that IAM alone, without backpropagation, can significantly reduce segmentation loss, further proving that IAM can effectively enhance the segmentation accuracy.

Frame Attention Visualization. We visualize the frame-level prototypical attention heatmaps, and the analysis reveals that different prototypes exhibit clearly distinct attention patterns, as shown in Figure 13. In the case of vehicle models, some prototypes mainly concentrate on boundary details—like the top edges and bottom outlines—while others emphasize the internal sections, such as the body of the car and the window areas. The attention map illustrates the comprehensive attention results achieved by effectively integrating all prototypes. Consequently, FAM can provide frame features that are more beneficial for subsequent tracking tasks.

Computation and Memory Efficiency. We analyze computation and memory efficiency across various common input resolutions in

Table 6. Computation comparison of different linear attention mechanisms across different video resolutions. (Batch Size = 2).

Resolution	DLAN			Linformer [13]			RAVLT-S [14]
	FLOPs (B)	Memory (M)	$\mathbf{FPS}\mathbf{\uparrow }$	FLOPs (B)	Memory (M)	$\mathbf{FPS}\mathbf{\uparrow }$	FLOPs (B)	Memory (M)	$\mathbf{FPS}\mathbf{\uparrow }$
640 $\times$ 480	10.3	684	22.5	16.4	1252	17.8	17.3	904	16.1
1280 $\times$ 720	26.5	2032	11.5	44.2	3751	7.2	45.9	2415	6.7
1920 $\times$ 1080	59.6	4597	6.2	97.4	8451	4.8	102.4	5733	4.5

Note: FPS (higher is better ↑) is used for evaluation.

. In addition to FLOPs and memory consumption, we evaluate the FPS on downstream tasks using Faster R-CNN and Mask R-CNN to verify practical efficiency gains. Compared to the representative linear baseline Linformer [13] and the state-of-the-art RAVLT [14], the DLAN, with its recursive structure and condensed prototypes, achieves a significant reduction in FLOPs and memory consumption, as well as improved FPS.

Qualitative Analysis about GMM. To validate the importance of soft clustering, we replaced the GMM clustering method in PLCA with the commonly used k-means hard clustering algorithm. As shown in Figure 14, owing to its rigid “one-size-fits-all” assignment, where each data point must belong to exactly one cluster, the k-means clustering method disrupts spatial continuity. Moreover, the interiors of vehicles consist of smooth regions with low gradients and few textures, where k-means clustering may result in overly strong edge activation and misclassification of internal features as background. In contrast, GMM soft clustering enables a smooth transition between the edges and interior, making it more suitable for mask representation.

Influence of Prototype Number. The prototype number refers to the number of clusters in the clustering process. Generally, increasing this number can improve the clustering accuracy, as it is often difficult to condense the complex information of real-world images into a few simple clusters. However, a very high prototype number can reduce the computational efficiency and lead to overfitting. In

Table 7. Ablation study of prototype number on the BDD100K MOTS validation set.

Prototype Number	mMOTSA ↑	mMOTSP ↑	mIDF1 ↑	IDS ↓
1	30.7	66.8	43.0	788
5	31.2	67.0	44.1	775
10	31.5	67.1	45.6	762
20	31.8	67.2	46.2	751
30	32.0	67.2	46.5	734
35	32.1	67.4	46.9	724
40	31.9	67.3	45.8	733
50	31.9	67.3	45.7	735

Note: The bold numbers indicate the best performance in each column. mMOTSA/mMOTSP/mIDF1 (higher is better ↑) and IDS (lower is better ↓), are used for evaluation.

, we observe that the metrics of the DLAN increase as the prototype number increases from 1 to 50. However, when this number exceeds 35, the metrics begin to decline instead of increasing. Therefore, we set the number of prototypes to 35.

Influence of EM Iteration Number. The EM iteration number represents the number of iterations executed in the expectation-maximization clustering algorithm. While increasing this parameter generally enhances the model fitting accuracy, excessive iterations may lead to overfitting and performance degradation. As presented in

Table 8. Ablation study of EM Iteration Number on the BDD100K MOTS validation set.

EM Iteration Number	mAP ↑	mAP50 ↑	mAP75 ↑	IDS ↓
2	26.2	42.0	25.8	742
4	26.2	43.0	25.8	739
6	26.3	43.3	25.8	731
8	26.5	44.4	25.9	728
10	26.8	44.5	26.0	724
12	26.8	44.4	26.1	727
14	26.7	44.4	25.9	726

Note: The bold numbers indicate the best performance in each column. mAP/mAP50/mAP75 (higher is better ↑) and IDS (lower is better ↓), are used for evaluation.

, our empirical analysis demonstrates that initial improvements in all evaluation metrics are observed as this parameter increases from 2 to 10, after which further iterations yield only marginal gains or may compromise performance. This observation suggests an optimal convergence point at 10 iterations, and we adopt 10 EM iterations as our optimal configuration for the proposed framework.

Impact of Update Rate. DLAN has two types of update rate hyperparameters: the feature update rate ${\lambda }_Y$ and the memory and center update rates $\lambda$. The feature update rate controls the contribution of the prototype-processed feature to downstream tasks. A higher rate increases the semantic influence of the prototypes. However, the prototype outputs are coarse-grained semantic compensations. The excessively large rate can shift the distribution of the backbone features, potentially disrupting the spatial and structural priors already learned by the backbone. As shown in

Table 9. Ablation study of the feature update rate on the BDD100K MOTS validation set.

The Feature Update Rate	mAP ↑	mAP50 ↑	mAP75 ↑	mIDF1 ↑	mMOTSA ↑	mMOTSP ↑
0	25.8	41.5	25.1	45.1	29.3	65.8
0.10	26.2	43.7	25.2	45.6	30.3	66.3
0.20	26.6	44.2	25.3	46.4	32.0	67.2
0.25	26.8	44.5	25.4	46.9	32.1	67.4
0.3	26.0	43.8	25.2	46.4	31.2	67.0
0.5	22.4	36.4	23.6	40.7	27.8	68.4
1	0.4	0.5	0.4	0.8	0.2	38.2

Note: The bold numbers indicate the best performance in each column. mAP/mAP50/mAP75/mIDF1/mMOTSA/mMOTSP (higher is better ↑) are used for evaluation.

, all evaluation metrics are optimal when the feature update rate is set to 0.25. The memory and center update rates play a crucial role in both the EM algorithm and the attention mechanism. While a higher update rate emphasizes the influence of historical spatio-temporal information and segmentation results, excessively high rates can amplify the impact of historical errors, which in turn may degrade tracking performance. As shown in

Table 10. Ablation study of memory and center update rates on the BDD100K MOTS validation set.

The Memory and Center Update Rates	mAP ↑	mAP50 ↑	mAP75 ↑	mIDF1 ↑	mMOTSA ↑	mMOTSP ↑
0	26.0	42.7	25.4	43.3	29.7	66.8
0.10	26.0	43.3	25.5	44.3	32.0	67.0
0.30	26.3	43.0	25.4	45.4	30.7	66.8
0.40	26.6	44.1	25.7	46.4	32.0	66.9
0.50	26.8	44.5	25.7	46.9	32.0	67.2
0.60	26.3	44.0	25.8	45.4	32.1	67.1
0.70	25.8	42.0	25.2	44.9	31.9	67.0
0.80	23.1	36.4	23.4	37.6	28.1	66.7

Note: The bold numbers indicate the best performance in each column. mAP/mAP50/mAP75/mIDF1/mMOTSA/mMOTSP (higher is better ↑) are used for evaluation.

, initial improvements across all evaluation metrics are observed as the rate increases from 0, followed by significant performance degradation beyond an optimal threshold. Notably, the best trade-off is achieved at an update rate of 0.5, suggesting an optimal balance between adaptive feature refinement and information retention.

Cross-Dataset Generalization Evaluation. To evaluate the generalization capability of the DLAN, we trained it on the BDD100K dataset and tested it on the KITTI MOTS dataset. For a fair comparison, we also evaluate QDTrack using its officially released pretrained weights trained on the same BDD100K dataset. As shown in

Table 11. Comparison of different networks in cross-dataset evaluation on the KITTI MOTS validation set.

Trackers	Pre-Training	MOTSA ↑		MOTSP ↑
		Car	Ped	Car	Ped
DLAN	I, KS	90.5	80.1	90.6	82.1
DLAN	I, B	89.6	78.5	89.7	80.3
QDTrack-mots-fix [31]	I, B	86.8	69.6	86.5	76.1
STEM-Seg [54]	I, C, S	83.8	66.1	87.2	77.7
MOTSNet [49]	I, M	83.9	67.8	89.4	79.4
TrackR-CNN [2]	I, C, M	87.8	65.5	87.2	75.7

Note: I: ImageNet. B: BDD100K MOTS. C: COCO. M: MOTS Challenge. KS: KITTI MOTS, S: Cityscapes. The bold numbers highlight the results achieved by our tracker for clearer comparison. MOTSA/MOTSP (higher is better ↑) are used for evaluation.

, DLAN achieves higher MOTSA and MOTSP than QDTrack, indicating that the collaborative effect of FAM and IAM effectively enhances the generalization capability of DLAN. Additionally, compared with other trackers trained on different datasets, DLAN achieves superior performance across all metrics when evaluated on the KITTI MOTS validation set, further demonstrating its strong generalization capability.

5. Limitations

While DLAN demonstrates strong performance in MOTS, several limitations remain that may affect its practical deployment and overall robustness. These limitations can be broadly categorized into real-world efficiency limitations, architectural and functional limitations, and failure cases analysis.

Real-world efficiency limitations. DLAN reduces theoretical FLOPs. However, due to limited resources, it remains uncertain whether it can meet real-time needs on automotive or robotic platforms. Linear complexity alone does not ensure practical efficiency without an optimized implementation.

Architectural and functional limitations. As a kernel function-based linear attention approximation, the RLSA may lose high-rank discriminative spatial information, which is crucial for detecting small objects. Additionally, DLAN relies on a two-stage detection framework (Faster R-CNN with FPN and Mask R-CNN) for initial proposals. Compared with fully end-to-end transformer-based trackers, this architectural dependence may limit global optimization. Moreover, the model is susceptible to the update rates of memory and cluster centers. For example, increasing λ from 0.5 to 0.8 significantly degrades performance, with mMOTSA dropping from 32.0 to 28.1 and mIDF1 decreasing from 46.9 to 37.6.

Failure Case Analysis. Although DLAN uses historical spatio-temporal information to reduce ID switches, tracking errors still happen in cases where the target’s historical pixel information is very incomplete. As shown in Figure 15, when new objects appear or when targets move from distant to closer positions, the historical cues may not be sufficient to maintain the correct associations.

6. Conclusions

In this paper, we presented DLAN, a novel online MOTS framework that processes video frames sequentially without accessing any future information. By effectively utilizing appearance and spatio-temporal information with linear computational complexity, DLAN effectively refines features and segmentation masks. Through memory and cluster center updates, the FAM has compensated for pixel loss caused by occlusion, illumination, and scale changes. IAM has separated foreground and background features to refine segmentation, produce clearer object boundaries, and improve robustness in challenging scenes. Meanwhile, the IAM-refined segmentation guides the FAM’s memory to focus on storing historical features of the foreground regions, effectively alleviating prototype contamination caused by background noise. With better feature refinement and mask accuracy, DLAN has outperformed state-of-the-art methods on the BDD100K MOT, BDD100K MOTS, and KITTI MOTS datasets. DLAN has achieved 32.1 mMOTSA and 90.6 MOTSP (car category). Additionally, on the BDD100K MOTS dataset, experimental results have shown that integrating DLAN into QDTrack has achieved a 26% reduction in the number of IDS and significantly improved the accuracy of the predicted masks. Future research may further enhance DLAN by exploring more expressive soft or hierarchical clustering strategies and extending its framework to broader video understanding tasks. Moreover, integrating DLAN with advanced data association or long-term tracking mechanisms could further improve identity consistency in crowded or highly dynamic scenes.