Contrastive Transformer Network for Track Segment Association with Two-Stage Online Method

Abstract

Interrupted and multi-source track segment association (TSA) are two key challenges in target trajectory research within radar data processing. Traditional methods often rely on simplistic assumptions about target motion and statistical techniques for track association, leading to problems such as unrealistic assumptions, susceptibility to noise, and suboptimal performance limits. This study proposes a unified framework to address the challenges of associating interrupted and multi-source track segments by measuring trajectory similarity. We present TSA-cTFER, a novel network utilizing contrastive learning and TransFormer Encoder to accurately assess trajectory similarity through learned Representations by computing distances between high-dimensional feature vectors. Additionally, we tackle dynamic association scenarios with a two-stage online algorithm designed to manage tracks that appear or disappear at any time. This algorithm categorizes track pairs into easy and hard groups, employing tailored association strategies to achieve precise and robust associations in dynamic environments. Experimental results on real-world datasets demonstrate that our proposed TSA-cTFER network with the two-stage online algorithm outperforms existing methods, achieving 94.59% accuracy in interrupted track segment association tasks and 94.83% in multi-source track segment association tasks.

1. Introduction

In the field of radar data processing, track interruption poses a frequent and significant challenge that hinders continuous target tracking. Factors such as target maneuvering, erroneous associations, and low detection probability contribute to this problem [1]. In addition, there is a growing research focus on the joint detection and tracking of targets using multiple sensor networks. To enable subsequent trajectory fusion processing, it becomes crucial to associate tracks obtained from different sensors, including radars or the automatic identification system (AIS), that correspond to the same target [2].

Traditional methods for interrupted track segment association (ITSA), as described in [3,4], rely on an a priori hypothesis model of target motion. These methods involve forward prediction or backward retrodiction between the end of the old track segment and the start of the new track segment, providing a framework for linking the track segments. If the predicted or retrodicted trajectory satisfies certain discriminative criteria, it is considered associated. In [5], a method was used that combined constant acceleration (CA) and coordinated turn (CT) models. The study explored various model parameters, such as accelerations and turn rates, to accurately model target motion characteristics. Fuzzy similarity measures were then applied to assess correlations between interrupted tracks. Similarly, in [1], researchers addressed interruptions in target tracking using a constant velocity (CV) model for continuous motion and switching to a CT model during interruptions within a two-dimensional (2-D) assignment framework. They optimized interruption associations by varying turn rates and employed a 2-D matching algorithm [6] to achieve a globally optimal solution. When the actual motion of the target does not match the a priori assumptions, it is difficult to guarantee the performance of the association.

Traditional multi-source track segment association (MSTSA) techniques aim to address system biases and random errors inherent in different sensors by employing various criteria such as statistics, structural similarity, fuzzy logic, etc. In [7], a novel track-to-track association algorithm was introduced, which uses the reference topology feature to mitigate sensor biases. Furthermore, the research outlined in [8] took a novel approach by transforming the association problem into a non-rigid point-matching problem and incorporated structural similarity to improve performance. In addition, in [9] an innovative algorithm is proposed that utilizes a dynamic weight function. By integrating target position information into the weight function and dynamically updating the fuzzy weight set, this improved algorithm achieved superior results in dense target scenarios. Inconsistent sensor noise levels can affect the performance of these methods. Moreover, thresholds often have to be determined manually, making it difficult to find a satisfactory global value.

Traditional techniques often suffer from inherent limitations, including unrealistic model assumptions, sensitivity to noise, reliance on manual thresholding, and suboptimal performance limits, etc. [1,10]. Nevertheless, in recent years, data-driven deep-learning methods have gained considerable attention in the domain of track segment association. These methods effectively learn feature representations of trajectories from the data, offering increased flexibility and accuracy.

For the ITSA problem, in [10], a combination of Long Short-Term Memory (LSTM) [11] and convolutional neural network (CNN) was utilized to capture temporal and spatial information from trajectories. This enables the comparison of feature similarity between trajectories to determine whether they belong to segments of the same target. Meanwhile, in [12], a graph neural network was employed to extract local track point embeddings at the node level and track graph embeddings at the graph level, facilitating the association of interrupted track segments. The authors above meticulously designed network architectures tailored to trajectory data and, in static environments, determined whether the trajectories belonged to the same target by comparing their similarity. However, neither considered the dynamic nature of real-world scenarios, where target trajectories can appear or disappear at any time. It is necessary to dynamically calculate similarity and make assessments based on changes in trajectories.

As for the MSTSA problem, the authors of [2] introduced using a spatial-temporal mixing extraction block to extract features from both tracks and scenes. These extracted features were then used to estimate the homography matrix, which helped to provide clear discriminants and improve the credibility of association decisions. In another study [13], the authors employed an autoencoder to efficiently denoise initial track data. They subsequently utilized a fully connected layer to extract high-dimensional features from the tracks, capturing the essential characteristics and patterns inherent in the data. Furthermore, a 3-D convolutional layer was incorporated to capture the features of the scene, providing valuable contextual information about the surrounding environment. Finally, the combination of features extracted from both tracks and scenes was used to assess the likelihood of two tracks originating from a common target. Similar to ITSA, the authors did not account for dynamic changes in the scene. Instead, they [2,13] used the complete temporal trajectory information to make associations between trajectories, rather than using all the information available up to the present moment.

In this paper, we address the problem of associating interrupted and multi-source track segments within a unified framework, which has not been previously discussed together in the literature [2,10,12,13]. Firstly, we transform the track segment association problem into a trajectory similarity measurement problem following [10,12]. By comparing the distances between high-dimensional features of trajectories, we can determine if they belong to the same target. Typically, trajectories from the same target exhibit high similarity. This approach contrasts with traditional methods that rely on motion model-based predictions for associating interrupted trajectories, thus avoiding performance degradation caused by mismatched motion models. Next, we introduce the TSA-cTFER network to extract high-dimensional feature representations of trajectories. For the input trajectory data, we initially convert the continuous and discrete attributes into vector representations using linear mapping and lookup tables. We then leverage the robust feature extraction capabilities of the Transformer Encoder. This is followed by pooling the information across the entire temporal dimension, as illustrated in Figure 1. This process provides comprehensive insights into the characteristics and behaviors of the trajectories. To train the network, we employ contrastive learning. In both the ITSA and MSTSA tasks, we consider track segments from the same target as positive samples and tracks from different targets as negative samples. This objective function allows us to continuously decrease the distance between positive samples while increasing the distance between negative samples, allowing the network to learn discriminative features effectively. Finally, considering the dynamic nature of real-world scenarios, we propose a two-stage online association algorithm that leverages the high-dimensional features obtained from TSA-cTFER. This algorithm efficiently handles trajectories that may appear or disappear at any time. In the first stage, we categorize potential association pairs into easy and hard pairs based on their similarity differences. Then, in the second stage, we implement delayed decisions to enhance the accuracy of challenging association pairs, significantly improving the overall performance of trajectory associations. Effectively addressing the TSA problem not only improves target tracking accuracy but also enables improved situational awareness and decision-making capabilities in various applications, including surveillance, navigation, and transportation.

In summary, the main contributions of this paper are as follows:

• From the perspective of trajectory similarity measurement, we unify the problems of interrupted track segment association and multi-source track segment association under a unified framework.
• To tackle these challenges, we introduce a network called TSA-cTFER, which combines contrastive learning with the Transformer Encoder to produce high-dimensional feature representations of trajectories. This network effectively transforms the task of trajectory similarity calculation into a distance measurement problem in a high-dimensional vector space.
• Furthermore, in dynamic scenarios where targets may appear or disappear unpredictably, we present a two-stage online association algorithm. This algorithm adapts different association strategies based on the difficulty levels of association pairs, enhancing association accuracy and reducing error rates.

The organization of this paper is as follows: Section 2 unifies the problem of ITSA and MSTSA in a unified framework and builds a model for the problem to be solved. Section 3 introduces in detail the network architecture based on the Transformer Encoder, as well as the contrastive loss function, and focuses on the construction method of positive and negative association pairs. In Section 4, we propose a two-stage online association algorithm for the practical application scenario of track segment association. Finally, in Section 5, we verify the effectiveness of the proposed TSA-cTFER network with the two-stage online algorithm through comparative experiments carried out with real-world data.

2. Problem

As for the ITSA problem, when track segmentation occurs, a tracking algorithm may create two tracks. The first track is the old one, which is terminated because there is no new information associated with the tracking target. The second track is a young one, following the same target, which is initialized and confirmed after the break. Given two lists of tracks, a track segment association algorithm must find the correct match between old and new tracks to maintain a consistent track ID.

Suppose that at the current time k, there exists an old track segment list containing tracks that have been terminated and a young track segment list consisting of tracks that have been started recently. They can be expressed as follows [1,3]

$$\begin{array}{cc}\hfill {\mathcal{T}}_k^{\mathcal{O}}& =\{{\mathit{T}}_n^{\mathcal{O}}\left(k\right);n=1,\dots ,N\}\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\mathcal{T}}_k^{\mathcal{Y}}& =\{{\mathit{T}}_m^{\mathcal{Y}}\left(k\right);m=1,\dots ,M\}\hfill \end{array}$$

(2)

where N and M are the total number of old and young track segments, respectively. ${\mathit{T}}_n^{\mathcal{O}}\left(k\right)$ and ${\mathit{T}}_m^{\mathcal{Y}}\left(k\right)$ are the state vector of the target of the old track segment and young track segment following

$$\begin{array}{cc}\hfill {\mathit{T}}_n^{\mathcal{O}}\left(k\right)& =\{{\widehat{\mathit{s}}}_k^n;k=k_{n,s}^{\mathcal{O}},\dots ,k_{n,e}^{\mathcal{O}}\},n=1,\dots ,N\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {\mathit{T}}_m^{\mathcal{Y}}\left(k\right)& =\{{\widehat{\mathit{s}}}_k^m;k=k_{m,s}^{\mathcal{Y}},\dots ,k_{m,e}^{\mathcal{Y}}\},m=1,\dots ,M\hfill \end{array}$$

(4)

In (3), $k_{n,s}^{\mathcal{O}}$ and $k_{n,e}^{\mathcal{O}}$ are the start and end times of the nth old track segment, respectively. Similarly, in (4), $k_{m,s}^{\mathcal{Y}}$ and $k_{m,e}^{\mathcal{Y}}$ are the start and end times of the mth young track segment, respectively. ${\widehat{\mathit{s}}}_k^{·}$ is the state vector of the target at time k containing the moving information. For instance, when considering the information obtained from radar, the target trajectory state ${\widehat{\mathit{s}}}_k^{·}$ can be expressed as $[p{x}_k,p{y}_k,v{x}_k,v{y}_k]$. Here, $p{x}_k$ and $p{y}_k$ denote the target’s position in a 2D Cartesian coordinate system, and $v{x}_k$ and $v{y}_k$ represent the target’s velocity.

As for the MSTSA problem, it is essential to accurately match the target trajectory information reported by different sensors for the same target. Suppose that at the current time k, there are two sources, denoted as $\mathcal{A}$ and $\mathcal{B}$, present in the scenario. These sources may belong to the same category, such as both being radar devices, or they could be of different categories, like radar and AIS devices. It is worth noting that the multi-source track segment association problem can be broken down into pairwise association problems between any two sources. Therefore, here we only consider the association problem between two sources. Taking into account that the target trajectory information reported by source $\mathcal{A}$ and $\mathcal{B}$ are represented as

$$\begin{array}{cc}\hfill {\mathcal{T}}_k^{\mathcal{A}}& =\{{\mathit{T}}_i^{\mathcal{A}}\left(k\right);i=1,\dots ,I\}\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill {\mathcal{T}}_k^{\mathcal{B}}& =\{{\mathit{T}}_j^{\mathcal{B}}\left(k\right);j=1,\dots ,J\}\hfill \end{array}$$

(6)

where $I,J$ is the total number of track segments reported on behalf of source $\mathcal{A}$ and $\mathcal{B}$, respectively. Similarly to ${\mathit{T}}_n^{\mathcal{O}}\left(k\right)$ and ${\mathit{T}}_m^{\mathcal{Y}}\left(k\right)$ in (3) and (4), ${\mathit{T}}_i^{\mathcal{A}}\left(k\right)$ and ${\mathit{T}}_j^{\mathcal{B}}\left(k\right)$ denote the state vector of the target of source $\mathcal{A}$ and $\mathcal{B}$.

It can be seen from (1), (2), (5) and (6) that both the ITSA and MSTSA problems can determine whether two tracks belong to the same target by comparing the similarity of the tracks between two lists. Therefore, this paper considers these two problems together.

For each target’s state vector ${\mathit{T}}_{·}^{·}\left(k\right)$ in (1), (2), (5) and (6), we feed it into a neural network $f_{\theta }$ represented by the parameter $\theta$. This network then maps the state vector to a high-dimensional vector $\mathit{z}\in {\mathbb{R}}^D$ that encompasses almost all the information on the trajectory. In this way, the similarity between the two tracks can be determined by comparing the distances in high-dimensional vector spaces. If two tracks are associated with the same target, their distance is smaller than that of the others. Taking the ITSA problem as an example, the distance between two tracks is calculated as follows

$$\begin{array}{c}{\mathit{z}}_n^{\mathcal{O}}\left(k\right)=f_{\theta }\left({\mathit{T}}_n^{\mathcal{O}}\left(k\right)\right)\\ {\mathit{z}}_m^{\mathcal{Y}}\left(k\right)=f_{\theta }({\mathit{T}}_m^{\mathcal{Y}}\left(k\right))\\ d_{n,m}\left(k\right)=\mathit{sim}({\mathit{z}}_n^{\mathcal{O}}\left(k\right),{\mathit{z}}_m^{\mathcal{Y}}\left(k\right))\end{array}$$

(7)

where $\mathit{sim}(·,·):{\mathbb{R}}^D\times {\mathbb{R}}^D\to \mathbb{R}$ is a distance measure and more details about it is described in Section 3.5.

Once we obtain the distances between two tracks, we can construct the global distance matrix ${\mathit{D}}_k$ at time k.

$${\mathit{D}}_k={\left[d_{n,m}\left(k\right)\right]}_{N\times M}$$

(8)

Therefore, the optimal match can be obtained by maximizing the following expression [3]

$$\underset{a_{n,m}}{max}{\mathit{A}}_k{\mathit{D}}_k=\sum _{n=0}^N\sum _{m=0}^M{a}_{n,m}{d}_{n,m}$$

(9)

where ${\mathit{A}}_k$ is an assignment matrix, whose elements are binary, and

$$a_{n,m}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\mathrm{tracks}{\mathit{T}}_n^{\mathit{O}}\left(k\right)\mathrm{and}{\mathit{T}}_m^{\mathcal{Y}}\left(k\right)\mathrm{are}\mathrm{from}\mathrm{the}\mathrm{same}\mathrm{target}\hfill \\ 0\hfill & \mathrm{else}.\hfill \end{array}\right.$$

(10)

Considering that each track can only be associated with another track or not associated with any track, Equation (9) satisfies the following constraint [3]

$$\begin{array}{c}\hfill \begin{array}{c}{\displaystyle \sum _{n=0}^N}{a}_{n,m}=1,m=1,\dots ,M\hfill \\ {\displaystyle \sum _{m=0}^M}{a}_{n,m}=1,n=1,\dots ,N\hfill \end{array}\end{array}$$

(11)

where $n=0$ and $m=0$ indicate dummy tracks in the old track segment list ${\mathcal{T}}_k^{\mathcal{O}}$ and young track segment list ${\mathcal{T}}_k^{\mathcal{Y}}$, respectively.

Observing that (9) describes a 2-D assignment problem, as the time k varies, Equation (9) evolves into a multi-dimensional (M-D) assignment problem, which is typically complex and time-consuming to resolve. Furthermore, existing methods [2,10,12,13] mainly focus on the static association issue without accounting for temporal changes as described in Section 1. Therefore, this paper first considers solving the 2-D assignment problem at time k, and then adopts a greedy algorithm to solve the temporal variation case, that is, designing a two-stage online matching strategy to achieve track association, thereby avoiding directly solving the M-D assignment. Further details will be provided in Section 4.

3. Proposed Method

In this section, we will begin by presenting the overall framework of TSA-cTFER. Subsequently, a detailed explanation will be provided for each component, including the input embedding module, the Transformer Encoder module, and the output pooling module. In addition, we will discuss the construction of positive and negative pairs and the contrastive loss functions utilized during training.

3.1. Overall Framework

The overall network framework, as depicted in Figure 2, comprises three modules: the input embedding module, the Transformer Encoder module, and the output pooling module. The input embedding module is tasked with processing various input data attributes and converting them into a unified vector representation. Subsequently, the Transformer Encoder module extracts features from the trajectories, which are then passed through the output pooling module to map the trajectories onto high-dimensional vectors. Here, we use two parallel pooling heads to obtain high-dimensional vector representations for the tasks of ITSA and MSTSA, respectively.

3.2. Input Embedding Module

The input embedding module maps different attributes of the targets to vectors of the same dimension, facilitating further processing by the network. Specifically, the attributes of the targets can be divided into continuous attributes and discrete attributes, as shown in Figure 2b.

Continuous attributes such as position, velocity, heading, etc., are first preprocessed through z-score standardization preprocessing and then linearly mapped to a D-dim vector. Suppose that ${\mathit{x}}_c\in {\mathbb{R}}^{x_L\times D_c}$ are the preprocessed continuous attributes, where $x_L$ is the length of the trajectory and $D_c$ is the number of continuous attributes. A linear projector is then used to obtain a D-dim embedding vector, denoted as ${\widehat{\mathit{x}}}_c$, which can be computed as follows

$${\widehat{\mathit{x}}}_c={\mathit{x}}_c{\mathit{W}}_c\in {\mathbb{R}}^{x_L\times D}$$

(12)

where ${\mathit{W}}_c\in {\mathbb{R}}^{D_c\times D}$ are the network parameters that need to be learned.

For discrete attributes such as target type, detection timestamp, source ID, etc., we use one-hot encoding to represent them. In the one-hot encoding, each discrete attribute is represented by a fixed-length binary vector. The length of the binary vector is equal to the number of unique discrete attributes. Each element in the vector corresponds to a specific discrete attribute, and only one element is set to 1 while all the others are set to 0. Thus, discrete attributes can be mapped to a D-dim vector through a lookup table following

$${\widehat{\mathit{x}}}_d={\mathit{x}}_d{\mathit{W}}_d\in {\mathbb{R}}^{x_L\times D}$$

(13)

where ${\mathit{x}}_d\in {[0,1]}^{x_L\times D_d}$ is the one-hot binary vector, $D_d$ is the number of unique discrete attributes, and ${\mathit{W}}_d\in {\mathbb{R}}^{D_d\times D}$ are the network parameters that need to be learned.

After obtaining the continuous and discrete embeddings of a trajectory, we add them at the corresponding time steps and then pad them with a learned padding token ${\mathit{E}}_{pad}\in {\mathbb{R}}^D$ to meet the specified length requirement of the network. Following that, a non-linear transformation is applied along with positional encoding [14] to obtain the final input embedding representation as follows

$$\begin{array}{c}\hfill \begin{array}{c}\widehat{\mathit{x}}=\mathit{NonLinear}\left(\mathit{cat}[{\widehat{\mathit{x}}}_c+{\widehat{\mathit{x}}}_d;{\mathit{E}}_{pad},\dots ,{\mathit{E}}_{pad}]\right)\in {\mathbb{R}}^{N\times D}\hfill \\ {\mathit{z}}_0=\widehat{\mathit{x}}+{\mathit{P}}_{pos}\in {\mathbb{R}}^{N\times D}\hfill \end{array}\end{array}$$

(14)

where $\mathit{cat}[·,·]$ represents the concatenation operation. The non-linear transformation consists of a ReLU activation function [15] and a linear projection. The positional encoding ${\mathit{P}}_{pos}\in {\mathbb{R}}^{N\times D}$ contains information about the point position in the trajectory [14], which is shown as follows

$$\begin{array}{c}\hfill \begin{array}{c}{\mathit{P}}_{pos,2i}=sin(pos/{\mathrm{10,000}}^{2i/D})\\ {\mathit{P}}_{pos,2i+1}=cos(pos/{\mathrm{10,000}}^{2i/D})\end{array}\end{array}$$

(15)

where $pos$ is the point position in the trajectory and i is the dimension.

3.3. Transformer Encoder Module

As shown in Figure 2c, the Transformer encoder structure consists of L layers of identical Transformer Encoder blocks. Each Transformer Encoder block includes a multi-head self-attention (MSA) module and a Feed-Forward Network (FFN) module. Incorporation of residual connections [16] and layer normalization [17] within each block facilitates efficient propagation of information and promotes stable training.

3.3.1. Multi-Head Self-Attention

The multi-head self-attention mechanism in the Transformer Encoder block is a crucial component that allows it to capture dependencies and relationships within an input sequence effectively. Initially, the input embeddings ${\mathit{z}}_l\in {\mathbb{R}}^{N\times D}$ are transformed into multiple sets of query $\mathit{q}$, key $\mathit{k}$, and value $\mathit{v}$ matrices through linear transformations [14,18].

$$[\mathit{q},\mathit{k},\mathit{v}]={\mathit{z}}_l{\mathit{W}}_{qkv},{\mathit{W}}_{qkv}\in {\mathbb{R}}^{D\times 3D_h}$$

(16)

where $D_h$ is the dimension of each attention head. To maintain a constant computational load and keep the number of parameters consistent when adjusting the number of heads h, $D_h$ is usually set to $D/h$.

Next, the self-attention weights $\mathit{A}$ are computed by taking the scale dot product between $\mathit{q}$ and $\mathit{k}$, followed by applying a masked softmax function.

$$\mathit{A}=\mathit{softmax}({\mathit{M}}_A·(\mathit{q}{\mathit{k}}^T/\sqrt{D_h}))\in {\mathbb{R}}^{N\times N}$$

(17)

where ${\mathit{M}}_A\in {[0,1]}^{N\times N}$ is a masked matrix, all padding tokens will be masked and excluded from the self-attention weights $\mathit{A}$ computation.

Finally, the weighted sum of the value matrices $\mathit{v}$ is calculated using the self-attention weights $\mathit{A}$.

$$\mathit{SA}\left({\mathit{z}}_l\right)=\mathit{A}\mathit{v}\in {\mathbb{R}}^{N\times D_h}$$

(18)

Instead of performing a single self-attention operation, the aforementioned process is repeated h times to obtain a multi-head self-attention. Each head in the multi-head self-attention has its own set of weight matrices for query, key, and value calculations. The outputs of all heads are subsequently concatenated into a matrix, which is further projected to yield the final representation.

$$\mathit{MSA}\left({\mathit{z}}_l\right)=\mathit{cat}[{\mathrm{SA}}_1\left({\mathit{z}}_l\right);\dots ;{\mathrm{SA}}_h\left({\mathit{z}}_l\right)]{\mathit{W}}_{msa}\in {\mathbb{R}}^{N\times D}$$

(19)

3.3.2. Feed-Forward Network

The position-wise fully connected feed-forward network employs a non-linear transformation to enhance the expressive capabilities of the network [14,18]. This transformation is applied independently and uniformly to each position. It comprises two linear transformations with a GELU activation function [19] in between.

$$\mathit{FFN}\left(z\right)=\mathit{GELU}(z{\mathit{W}}_{f1}+{\mathit{b}}_{f1}){\mathit{W}}_{f2}+{\mathit{b}}_{f2}$$

(20)

where ${\mathit{W}}_{f1}\in {\mathbb{R}}^{D\times D_{ff}}$, ${\mathit{b}}_{f1}\in {\mathbb{R}}^{D_{ff}}$, ${\mathit{W}}_{f2}\in {\mathbb{R}}^{D_{ff}\times D}$ and ${\mathit{b}}_{f2}\in {\mathbb{R}}^D$ are all learnable parameters.

3.3.3. Block Connecting and Layer Stacking

In each layer of the Transformer Encoder block, a combination of residual connection and layer normalization is utilized to connect the multi-head self-attention block with the feed-forward network block. The output of the previous layer acts as the input of the next layer. By stacking L layers together, the Transformer Encoder structure is constructed [14,18].

$$\begin{array}{ccc}\hfill {\mathit{z}}_l^{\prime }& =\mathit{MSA}\left(\mathit{LN}\left({\mathit{z}}_{l-1}\right)\right)+{\mathit{z}}_{l-1}, \hfill & \hfill l=1,\dots ,L\end{array}$$

(21)

$$\begin{array}{ccc}\hfill {\mathit{z}}_l& =\mathit{FFN}\left(\mathit{LN}\left({\mathit{z}}_l^{\prime }\right)\right)+{\mathit{z}}_l^{\prime },\hspace{1em}\hspace{1em}\hfill & \hfill l=1,\dots ,L\end{array}$$

(22)

where ${\mathit{z}}_0$ is obtained from (14) and a layernorm operation [17] is applied before every block. Layer normalization standardizes each feature dimension to have zero mean and unit variance. This effectively addresses the issues of vanishing and exploding gradients, enhancing the stability and convergence speed of the network.

3.4. Output Pooling Module

The Transformer Encoder module produces an output matrix ${\mathit{z}}_L$ of size $N\times D$. To obtain a fixed-length trajectory representation with a dimension of D, a pooling module is applied to process the context matrix ${\mathit{z}}_L$. As shown in Figure 2a, we obtain trajectory feature representations for the ITSA task and the MSTSA task separately using two parallel projection heads after the same pooling operation. Initially, ${\mathit{z}}_L$ goes through a pooling layer and then outputs trajectory features through a linear mapping.

$$\begin{array}{cc}\hfill {\widehat{\mathit{z}}}_L& =\mathit{pooling}\left({\mathit{z}}_L\right)\in {\mathbb{R}}^D\hfill \\ \hfill {\mathit{z}}_{out}^{\mathit{ITSA}}& ={\widehat{\mathit{z}}}_L{\mathit{W}}_{\mathit{ITSA}}\in {\mathbb{R}}^D\hfill \\ \hfill {\mathit{z}}_{out}^{\mathit{MSTSA}}& ={\widehat{\mathit{z}}}_L{\mathit{W}}_{\mathit{WSTSA}}\in {\mathbb{R}}^D\hfill \end{array}$$

(23)

where ${\mathit{W}}_{\mathit{ITSA}},{\mathit{W}}_{\mathit{WSTSA}}\in {\mathbb{R}}^{D\times D}$ are the learnable projection heads for ITSA and MSTSA task, respectively.

We experiment with three pooling strategies:

•

CLS-strategy

Like class-token described in previous works such as ViT [18] and BERT [20], we introduce a trainable embedding ${\mathit{x}}_{cls}$ at the beginning of the embedded sequence. The state of this embedding at the output of the Transformer Encoder serves as a representation of the entire trajectory.

$$\begin{array}{cc}\hfill {\widehat{\mathit{x}}}_0& =\mathit{NonLinear}\left(\mathit{cat}[{\mathit{x}}_{cls};{\widehat{\mathit{x}}}_c+{\widehat{\mathit{x}}}_d;{\mathit{E}}_{pad},\dots ,{\mathit{E}}_{pad}]\right)\hfill \\ \hfill {\mathit{z}}_0& ={\widehat{\mathit{x}}}_0+{\mathit{P}}_{pos}\hfill \\ \hfill {\widehat{\mathit{z}}}_L& =\mathit{LN}\left({\widehat{\mathit{z}}}_L^0\right)\in {\mathbb{R}}^D\hfill \end{array}$$

(24)

Here, ${\widehat{\mathit{z}}}_L^0$ corresponds to the input class-token ${\mathit{x}}_{cls}$ and is located at the beginning of the output state.

•

MEAN-strategy

MEAN-strategy computes the mean-over-time of the context matrix ${\mathit{z}}_L$. It calculates the average value of each column in ${\mathit{z}}_L$ and returns a D-dim vector ${\widehat{\mathit{z}}}_L$.

$${\widehat{\mathit{z}}}_L=\mathit{mean}({\mathit{M}}_z·{\mathit{z}}_L,\mathit{axis}=0)\in {\mathbb{R}}^D$$

(25)

Here, $\mathit{axis}=0$ indicates that the mean operation is performed on each column of the matrix ${\mathit{z}}_L$ and ${\mathit{M}}_z\in {[0,1]}^{N\times D}$ is a masked matrix similar to ${\mathit{M}}_A$, where the D column vectors are identical, and the first $x_L$ elements in each of the N rows are active.

•

MAX-strategy

Similar to the MEAN-strategy, MAX-strategy computes the max-over-time of the context matrix ${\mathit{z}}_L$ as follows

$${\widehat{\mathit{z}}}_L=\mathit{max}({\mathit{M}}_z·{\mathit{z}}_L,\mathit{axis}=0)\in {\mathbb{R}}^D$$

(26)

Thus, we obtain high-dimensional feature representations for each trajectory.

3.5. Contrastive Loss Function

We use the contrastive loss [21,22] as an objective function. Contrastive learning is a technique that aims to enhance the learning of data representations by creating pairs of positive and negative samples, and continuously minimizing the dissimilarity between similar data instances while maximizing the dissimilarity between dissimilar ones. In the context of ITSA, positive samples are defined as track segments that belong to the same target before and after an interruption, whereas negative samples represent track segments between different targets. For the MSTSA problem, positive samples refer to track segments from different sources that correspond to the same target. In contrast, negative samples indicate track segments between different sources and distinct targets.

As illustrated in Figure 3, let us assume that there are two sources, denoted $\mathcal{A}$ and $\mathcal{B}$, that continuously provide trajectory information for two distinct targets. The track segments of the same target may be interrupted for various reasons; for instance, ${\mathit{T}}_1^{\mathcal{A}}$ and ${\mathit{T}}_2^{\mathcal{A}}$ represent such interrupted track segments. Additionally, track segments of the same target detected by different sources may exhibit positional discrepancies due to source system errors, as seen with ${\mathit{T}}_1^{\mathcal{A}}$ and ${\mathit{T}}_1^{\mathcal{B}}$ representing multi-source track segments. Therefore, it is crucial to determine which track segments belong to the same target. In this context, Track ${\mathit{T}}_1^{\mathcal{A}}$, ${\mathit{T}}_1^{\mathcal{B}}$ and ${\mathit{T}}_2^{\mathcal{A}}$ belong to the track segment of the same target, and Track ${\mathit{T}}_3^{\mathcal{A}}$, ${\mathit{T}}_2^{\mathcal{B}}$ and ${\mathit{T}}_4^{\mathcal{A}}$ belong to the segment of another distinct target. For the ITSA task, consider ${\mathit{T}}_2^{\mathcal{A}}$ as an example, assuming the current old track set is $\{{\mathit{T}}_1^{\mathcal{A}},{\mathit{T}}_3^{\mathcal{A}},{\mathit{T}}_1^{\mathcal{B}}\}$, since ITSA only matches tracks from the same source, so ${\mathit{T}}_1^{\mathcal{B}}$ will not match with ${\mathit{T}}_2^{\mathcal{A}}$. Consequently, $({\mathit{T}}_1^{\mathcal{A}},{\mathit{T}}_2^{\mathcal{A}})$ are positive pairs and $({\mathit{T}}_3^{\mathcal{A}},{\mathit{T}}_2^{\mathcal{A}})$ are negative pairs based on whether the track segments are from the same target. For the MSTSA task, take ${\mathit{T}}_1^{\mathcal{B}}$ as an example. If the current set of existing tracks is $\{{\mathit{T}}_1^{\mathcal{A}},{\mathit{T}}_3^{\mathcal{A}},{\mathit{T}}_2^{\mathcal{B}}\}$, MSTSA only matches tracks generated by different sources, so ${\mathit{T}}_2^{\mathcal{B}}$ will not match with ${\mathit{T}}_1^{\mathcal{B}}$. Then, $({\mathit{T}}_1^{\mathcal{A}},{\mathit{T}}_1^{\mathcal{B}})$ are positive pairs and $({\mathit{T}}_3^{\mathcal{A}},{\mathit{T}}_1^{\mathcal{B}})$ are considered negative pairs based on whether the track segments are from the same target.

Given a collection of paired track samples $\mathcal{D}={\left\{({\mathit{T}}_m,{\mathit{T}}_m^{+})\right\}}_{m=1}^M$, where ${\mathit{T}}_m$ and ${\mathit{T}}_m^{+}$ represent positive track samples from the same target. During training, we randomly select B track pairs from $\mathcal{D}$ to create a mini-batch of $2B$ sample points. Thus, for each track sample ${\mathit{T}}_i$, there is a similar sample ${\mathit{T}}_j$ in the mini-batch, with the remaining $2(B-1)$ samples considered negatives. Following the approach in [21,22], InfoNCE is used as the loss function. The loss function for a given positive sample pair $({\mathit{T}}_i,{\mathit{T}}_j)$ is defined as

$${\ell }_{i,j}=-log{\displaystyle \frac{exp(\mathit{sim}({\mathit{z}}_i,{\mathit{z}}_j)/\tau )}{{\sum }_{k=1}^{2B}\mathbb{1}\left[k\ne i\right]exp(\mathit{sim}({\mathit{z}}_i,{\mathit{z}}_k)/\tau )}}$$

(27)

where B is the mini-batch size, ${\mathit{z}}_i,{\mathit{z}}_j,{\mathit{z}}_k$ are obtained from (7), ${\mathbb{1}}_{[k\ne i]}\in \{0,1\}$ is an indicator function that evaluates to 1 iff $k\ne i$ and $\tau$ denotes a temperature hyperparameter. As described in (7), we adopt cosine similarity as a distance measure here, defined as

$$\mathit{sim}(\mathit{u},\mathit{v})={\mathit{u}}^T\mathit{v}/(\parallel \mathit{u}\parallel \parallel \mathit{v}\parallel )$$

(28)

where $\parallel ·\parallel$ denotes ${\ell }_2$ norm. Therefore, the total loss ${\ell }_{total}$ is defined as

$${\ell }_{total}={\displaystyle \frac{1}{2B}}\sum _{i=1}^{2B}\sum _{j=1}^{2B}{\ell }_{i,j}(i,j)\in \mathrm{positive}\mathrm{pairs}$$

(29)

In the InfoNCE loss function (27), the numerator term represents the similarity of the positive samples, while the denominator term sums the similarities among all samples except for itself. The temperature parameter $\tau$ controls the smoothness or sharpness of the similarity distribution. A higher $\tau$ leads to smaller differences in similarity values between positive and negative samples, while a lower $\tau$ increases these differences. The choice of the temperature parameter $\tau$ is usually based on experimental results as shown in Section 5.6.

4. Two-Stage Online Association Algorithm

The online TSA algorithm divides the entire association process into two parts. The first part is the track state update module, which uses the continuously received target information from the sensors to update the target state. The second part is the proposed two-stage association algorithm. In association scenarios, there are both simple track pairs that can be easily associated and difficult track pairs that pose challenges for the association. To address these varying complexities, we employ different association strategies to handle them accordingly. The comprehensive tracking process is illustrated in Figure 4 and will be expounded on in the subsequent sections.

4.1. Track State Update

The trajectory information of the target is continuously sent by the sensors. The track state update module is responsible for recording the states of the targets, which includes sensor ID, timestamp, position, velocity, and other relevant information. Currently, it utilizes the neural network $f_{\theta }$ for TSA to constantly update the representation of the trajectory. A detailed description of the entire track state update process is provided in Algorithm 1.

Lines 2 to 5 of Algorithm 1 update the state and feature representation of the trajectories using the sensor detection results ${\mathcal{D}}_k$ at the current time step k. The similarity between the two trajectories is then computed using the Equations (7) and (28). The function $\mathit{filtering}(t_i,t_j)$ on line 8 of Algorithm 1 filters out certain pairs of trajectories that do not meet the association criteria. This filtering is performed by taking into account information such as timestamp, position, and velocity. In the case of ITSA, it is necessary to consider that there is a temporal sequential relationship between the trajectories for which similarity must be computed, while trajectories that appear simultaneously do not require similarity computation. In the context of the MSTSA problem, trajectories belonging to the same target are usually close to each other. By comparing the positional differences between the two trajectories at the same time, the distant trajectories can be filtered out to reduce the computational load in subsequent steps.

It is essential to emphasize that the length of the input data for the feature extractor $f_{\theta }$ on line 4 of Algorithm 1 may vary depending on the specific requirements of $f_{\theta }$. For instance, in TSA-cTFER with a fixed input size N, if a trajectory is longer than the specified length, it may need to be truncated by retaining only the most recent data points to meet the length requirement.

Algorithm 1: Track State Update.

4.2. Two-Stage Association Method

As previously mentioned, the two-stage association approach classifies candidate track pairs into easy and hard pairs based on the disparity in their similarity. The entire process of the two-stage association method is described in Algorithm 2.

First, the set of track scores ${\mathcal{T}}_s$ is divided into two distinct sets, denoted as ${\mathcal{T}}_s^I$ and ${\mathcal{T}}_s^J$ according to the specific task at hand, as described in line 1 of Algorithm 2. For the ITSA problem, ${\mathcal{T}}_s$ is categorized into old track sets ${\mathcal{T}}_s^{\mathcal{O}}$ and young track sets ${\mathcal{T}}_s^{\mathcal{Y}}$ based on timestamp information to determine whether the track has ended. For the MSTSA problem, considering the different source IDs and timestamps, tracks from the same source sensor at the current moment are grouped, resulting in ${\mathcal{T}}_s^{\mathcal{A}}$ and ${\mathcal{T}}_s^{\mathcal{B}}$.

Subsequently, a score matrix $scor{e}_m$ is constructed based on the similarities obtained from Algorithm 1. The Hungarian algorithm [6] is then employed to solve the matrix, yielding candidate association pairs. It is important to note that the introduction of dummy tracks in (11) is primarily intended to indicate that certain tracks may not have corresponding associated tracks. In this context, dummy tracks are not explicitly constructed; instead, the matching results of the Hungarian algorithm are re-evaluated, and some track associations with low similarities are discarded. Candidate association pairs with a score value below ${\tau }_{low}$ are directly labeled as invalid. Among the remaining candidate association pairs, those that have another pair with a score difference less than ${\tau }_d$ are categorized as hard pairs; otherwise, if the association score is greater than ${\tau }_{high}$, it is considered an easy pair, while the remaining pairs are considered invalid. The selection procedure for invalid/easy/hard association pairs is detailed in Algorithm 3 and illustrated in Figure 5. We primarily determine the existence of similar association pairs based on score differences, and then judge whether the association pair is considered an easy or hard case.

$$\begin{array}{c}\hfill \begin{array}{c}\mathrm{abs}(scor{e}_m[i,jj]-scor{e}_m[i,j])<{\tau }_d\mathit{for}jj=0,1,\dots ,J-1\\ \mathrm{abs}(scor{e}_m[ii,j]-scor{e}_m[i,j])<{\tau }_d\mathit{for}ii=0,1,\dots ,I-1\end{array}\end{array}$$

(30)

Algorithm 2: Two-Stage Association.

Next, we employ a straightforward M/N logic to determine the successful association of pairs within the set $easy\_pairs$, as described in line 11 of Algorithm 2. This logic operates on the assumption that, out of M consecutive evaluations, if there are N instances of a particular association pair appearing in the set $easy\_pairs$, it is considered a valid association.

For all hard association pairs $hard\_pairs$, the hard score matrix $hard\_scor{e}_m$ is first reconstructed based on the set of track scores ${\mathcal{T}}_s$. Subsequently, Algorithm 3 is used to further classify the pairs into easy association pairs $hard\_e\_pairs$ and hard association pairs $hard\_h\_pairs$ using a difference cost threshold ${\tau }_d^{hard}(<{\tau }_d)$. Then, the M/N logic is applied to easy association pairs $hard\_e\_pairs$ to determine their association relationship. Conversely, the hard association pairs $hard\_h\_pairs$ are retained for additional evaluation in the subsequent time step $k+1$ until the maximum matching time requirement is met. At that point, the association pair with the highest score is directly selected as the output.

In summary, the selection of easy and hard association pairs, outlined in Algorithms 2 and 3, is based on the similarity of characteristics derived from the TSA network $f_{\theta }$. In scenarios involving multiple trajectories with high similarities, we adopt a deferred decision strategy that involves reevaluating associations upon receiving additional target trajectory information. Although this strategy may lead to increased association latency, it contributes to improved accuracy levels.

Algorithm 3: Select Easy/Hard Association Pairs.

5. Experimental Results

5.1. Datasets

To assess the effectiveness of our approach, we utilized the Multi-source Track Association Dataset (MTAD) proposed in [23,24] as a standardized benchmark for evaluation purposes. The MTAD is constructed by performing various processing steps, such as grid partitioning, automatic interruption, and noise injection, on AIS trajectory data. This dataset contains millions of individual tracks, however, for our analysis, we are focusing solely on data from 5000 unique scenes. For each scene, trajectory information is reported from two sensors at different reporting rates: 10 s and 20 s. The entire scenario lasts approximately two hours. Each trajectory in the dataset is comprised of four-dimensional information, namely longitude, latitude, speed, and heading, describing the motion of the targets. As a result, each scene in this dataset contains varying numbers of tracks, ranging from several to hundreds. These tracks exhibit diverse movement patterns, target types, and duration periods. Hence, the MTAD is highly suitable for validating the performance of different track segment association algorithms.

As shown in Figure 6, the MTAD presents four typical association scenarios, which involve ITSA and MSTSA tasks. These scenarios exhibit varying characteristics. Some have a limited number of tracks with straightforward target motion, while others have a larger number of tracks that tend to concentrate on specific areas. Consequently, associating these tracks becomes more challenging.

5.2. Metrics

The two tracks that satisfy the following conditions are referred to as the actual associated track pairs AP [23]:

• The tracks originate from the same target.
• In the case of ITSA, the time interval between interruptions in the two tracks is required to be less than 20 min, and each track must have a duration longer than 2 min.
• For the MSTSA problem, the intersection time of the two-track segments needs to exceed 2 min.

Thus, we define the set composed of all APs in a scene s as ${\mathcal{T}}_{\mathrm{AP}}^s$ and the set composed of the associated pairs produced by the association algorithm as ${\mathcal{O}}_{\mathrm{AP}}^s$.

To measure the association accuracy, we define the association accuracy $P_{CA}^s$ as [23]:

$$P_{\mathrm{CA}}^s={\displaystyle \frac{|{\mathcal{O}}_{\mathrm{AP}}^s\cap {\mathcal{T}}_{\mathrm{AP}}^s|}{|{\mathcal{T}}_{\mathrm{AP}}^s|}}\times 100\%$$

(31)

where $|·|$ indicates the number of association pairs in the set and ∩ is representing the intersection operation between sets.

Similarly, we can also obtain the definitions of association fault rate $P_{\mathrm{FA}}^s$ [23]:

$$P_{\mathrm{FA}}^s={\displaystyle \frac{|{\mathcal{O}}_{\mathrm{AP}}^s-{\mathcal{O}}_{\mathrm{AP}}^s\cap {\mathcal{T}}_{\mathrm{AP}}^s|}{|{\mathcal{O}}_{\mathrm{AP}}^s|}}\times 100\%$$

(32)

and the association latency $T_{\mathrm{AP}}^s$ [23]:

$$T_{\mathrm{AP}}^s={\displaystyle \frac{1}{|{\mathcal{O}}_{\mathrm{AP}}^s\cap {\mathcal{T}}_{\mathrm{AP}}^s|}}\sum _{AP\in {\mathcal{O}}_{\mathrm{AP}}^s\cap {\mathcal{T}}_{\mathrm{AP}}^s}{t}_{\mathrm{AP}}^s$$

(33)

where the latency $t_{\mathrm{AP}}^s$ is defined as the generation time of the association relationship between track pairs minus the association start time of that track pair. For ITSA, the association start time is defined as the time when the first point of the young track appears. For the MSTSA problem, the association start time is defined as the time when the first point of the later track appears.

Therefore, for all scenarios, the average association accuracy ${mP}_{\mathrm{CA}}$ and the average weighted association accuracy ${mwP}_{\mathrm{CA}}$ are, respectively, defined as [23]:

$${mP}_{\mathrm{CA}}={\displaystyle \frac{1}{S}}\sum _{s=0}^{S-1}{P}_{\mathrm{CA}}^s$$

(34)

$${mwP}_{\mathrm{CA}}=\sum _{s=0}^{S-1}{\displaystyle \frac{|{\mathcal{O}}_{\mathrm{AP}}^s|}{{\mathcal{O}}_{\mathrm{AP}}}}{P}_{\mathrm{CA}}^s$$

(35)

Here, S represents the total number of test scenarios and ${\mathcal{O}}_{\mathrm{AP}}={\sum }_{s=0}^{S-1}\left|{\mathcal{O}}_{\mathrm{AP}}^s\right|$. Similarly, we can obtain the definitions of average association fault rate ${mP}_{\mathrm{FA}}$, average weighted association fault rate ${mwP}_{\mathrm{FA}}$, average association latency $m{T}_{\mathrm{AP}}^s$, and average weighted association latency $mw{T}_{\mathrm{AP}}^s$.

5.3. Baselines

5.3.1. Traditional Methods

For the ITSA problem, traditional methods commonly depend on the prediction of the trajectories of targets and subsequently assessing the degree of similarity between two trajectories. If the predefined matching criteria are met, the corresponding pairs are deemed successful matches.

•

Multiple Hypothesis TSA: The Multiple Hypothesis TSA algorithm (MH TSA) [5] starts by constructing multiple potential models of target motion and performing trajectory prediction. Next, based on position and velocity information, a fuzzy correlation function is utilized to depict the relationship between predicted old tracks and new tracks. Finally, the Multiple Hypothesis TSA algorithm constructs a fuzzy similarity matrix between the old and young tracks and then employs the two-dimensional assignment principle to determine the association between targets.

•

Multi-Frame 2-D TSA: The Multi-Frame 2-D TSA algorithm (MF 2-D TSA) [1] assumes that the target motion follows the constant velocity model before and after interruption. Additionally, it incorporates a constant turn with a turning rate of w throughout the breakage period. By exploring various turning rates, if the distance between the predicted state of the old track and the state of the young track meets specific criteria, this correlation is deemed successful.

In the case of the MSTSA problem, it is common to employ distance-based measurements to quantify the similarity between two tracks.

•

Distance-Based TSA: The Euclidean distance (ED) is the most commonly used distance metric for track correlation as used in [23]. However, it assumes that trajectories are the same length. Here, we also utilize Dynamic Time Warping (DTW) [25,26,27] as a criterion to measure the similarity of multi-source tracks. Similar to ED, DTW is also a measure designed for time series and can be applied to trajectory data. However, unlike ED, DTW can align a point of one trajectory with one or more consecutive points of another trajectory. This flexibility makes DTW suitable for cases where track lengths vary or where there are temporal variations in the data.

5.3.2. Deep Learning Based Methods

•

TSADCNN: TSADCNN [10] uses a temporal and spatial information extraction module to extract track information. The temporal module, composed of LSTM, is responsible for capturing temporal information, while the spatial module, consisting of multi-scale CNN, extracts spatial information. Initially, the track information undergoes processing through the LSTM module to capture temporal dependencies. Then, it is fed to the CNN module to extract spatial features. By mapping the track data onto a high-dimensional space, TSADCNN determines whether the trajectories belong to the same target based on the similarity of high-dimensional vectors.

•

TGRA: Similar to TSADCNN, TGRA [12] utilizes a graph network to extract track information. It employs the Graph Isomorphism Network (GIN) [28] and Graph Convolutional Network (GCN) [29] to obtain node-level local representations of track points. Subsequently, the Graph Attention Network (GAT) [30] is employed to acquire representations at the graph level. In this way, This approach enables the extraction of high-dimensional representations that encompass both the spatial and temporal information of the entire trajectory.

5.4. Implementation Details

5.4.1. Data Preparation

The 5000 scenes in MTAD are split into three subsets: a training set, a validation set, and a test set, following an 8:1:1 ratio. During training, trajectory data from 4000 scenes is mixed into one dataset. For each training iteration, a batch of trajectories is randomly chosen from this dataset. Positive and negative samples are created using this batch for the network to learn from, as explained in Section 3.5. In the testing phase, data from each scene are evaluated independently. Simulated data sources provide target trajectory information every 20 s in chronological order. Using the two-stage online association algorithm detailed in Section 4 and involving various networks such as TSADCNN [10], TGRA [12], and our TSA-cTFER to compute trajectory similarity, the final association results are derived for each scene. These outcomes are compared with the ground truth associations to determine the final performance metrics. For deep learning-based TSA methods, the continuous attributes input into the network include longitude and latitude, while the discrete attribute is the timestamp.

5.4.2. Model Settings

In the TSA-cTFER model, we define the maximum input trajectory length as $N=128$, the hidden layer size as $D=64$, the number of model layers as $L=4$, and utilize an attention mechanism with $h=4$ heads. Furthermore, the size of the hidden layer of the FFN is set as $D_{ff}=4D$. For the TSADCNN model, we use the same input embedding module as in the TSA-cTFER model. The features then undergo processing through LSTM and multi-scale CNN layers, followed by the utilization of two parallel fully connected layers to extract features for ITSA and MSTSA tasks, respectively. It is worth mentioning that we do not require the temporal correlation matrix to be a square matrix, hence the symmetric constraint loss discussed in [10] is not applied in this model. In this setup, all feature dimensions are set to 64, a single-layer unidirectional LSTM network is employed, and within the multi-scale convolutional network, each scale’s feature dimension is halved while the channel count is doubled. Similarly, in the TGRA model, we make use of the same input embedding module and incorporate two parallel Track Graph Embedding modules to derive ITSA and MSTSA features, respectively. In this model, the final output feature dimension is 64, while the remaining parameters align with those outlined in [12].

5.4.3. Training Details

During the training phase, we employed random cropping as a data augmentation strategy to improve the model’s generalization ability. Specifically, for two track segments to be associated in the ITSA task, we randomly discarded the tail information of the young track since ITSA primarily focuses on the changes before and after interruptions; for positive track pairs in the MSTSA task, we randomly cropped the trajectory information at the same timestamps for the two tracks.

To compare the feature extraction capabilities of different deep learning models, all of these models were equipped with identical parameters (N/M = 3/6, ${\tau }_{low}=0.5$, ${\tau }_{high}=0.7$, ${\tau }_d=0.1$, ${\tau }_d^{hard}=0.05$) in the context of the two-stage online association algorithm. This consistent configuration allows for a comprehensive evaluation and comparison of the ability of the models to extract meaningful features.

During the training period, we trained all data for 150 epochs with a batch size of 512. We use Adam optimizer [31] with ${\beta }_1=0.9$, ${\beta }_2=0.999$, and $ϵ={10}^{-6}$. After 10 epochs of warmup, the learning rate decreased following a half-cycle cosine [32]. Additionally, a Dropout technique [33] with a dropout probability of $0.1$ was applied as a regularization method during the training of TSA-cTFER. All experiments are carried out on a 64-bit workstation with the PyTorch deep learning framework [34]. The detailed configurations for these experiments are as follows: Ubuntu 22.04.3 LTS, 64 GB RAM, Intel Xeon E5-2690 CPU @ 2.60 GHz, NVIDIA GeForce RTX 3090 GPU.

5.5. Performance Analysis

For deep learning-based TSA methods, we independently trained each network three times and reported the average and standard deviation of the final results. In addition, we separately trained and co-train the ITSA and MSTSA tasks to observe the strength of the network’s ability to represent trajectories.

5.5.1. ITSA Task

The results of the ITSA task using different methods are displayed in

Table 1. Results of Interrupted Track Segment Association.

	#params. (M)	${\mathit{mP}}_{\mathbf{CA}}$ (%)	${\mathit{mP}}_{\mathbf{FA}}$ (%)	${\mathit{mT}}_{\mathbf{AP}}$ (s)	${\mathit{mwP}}_{\mathbf{CA}}$ (%)	${\mathit{mwP}}_{\mathbf{FA}}$ (%)	${\mathit{mwT}}_{\mathbf{AP}}$ (s)
ED-based TSA	-	48.87	55.59	86.92	27.98	75.05	79.44
MH TSA [5]	-	93.79	6.35	49.34	90.87	11.99	43.42
MF 2-D TSA [1]	-	80.54	5.30	156.82	75.73	9.92	132.59
TSADCNN (Sep.) [10]	0.908	79.41 ± 3.03	6.61 ± 0.59	227.27 ± 23.24	76.70 ± 2.66	12.14 ± 0.74	212.43 ± 15.62
TSADCNN [10]	1.301	83.86 ± 0.13	5.52 ± 0.24	201.01 ± 13.63	80.82 ± 0.15	10.85 ± 0.05	189.51 ± 2.17
TGRA (Sep.) [12]	0.534	85.64 ± 0.72	8.63 ± 0.27	152.71 ± 8.68	79.55 ± 0.62	16.16 ± 0.23	153.70 ± 1.26
TGRA [12]	0.534	85.02 ± 0.29	9.02 ± 0.07	160.85 ± 10.76	79.55 ± 0.18	16.17 ± 0.11	150.42 ± 1.00
TSA-cTFER (Sep.)	0.670	94.21 ± 0.05	3.46 ± 0.06	130.91 ± 1.13	90.77 ± 0.09	7.39 ± 0.07	139.19 ± 0.96
TSA-cTFER	0.674	94.59 ± 0.04	3.49 ± 0.08	131.00 ± 9.49	91.29 ± 0.13	7.34 ± 0.15	136.13 ± 2.97

The best results are bolded.

, where “Sep.” indicates that only interrupted track data are utilized for training, and “#params.” represents the number of learnable parameters of the network. Among traditional interrupted TSA methods, Multiple Hypothesis TSA outperforms ED-based TSA and Multi-Frame 2-D TSA by a significant margin. Regarding deep learning-based methods, the results indicate that TSADCNN performs better when trained jointly rather than separately. Conversely, TGRA shows better performance with separate training for the two tasks. The separate and joint training results of TSA-cTFER are comparable, suggesting that TSA-cTFER possesses stronger representation capabilities than TSADCNN and TGRA. Furthermore, the association results of TSA-cTFER significantly outperform TSADCNN and TGRA, regardless of whether they were trained separately or jointly. Compared to the best results achieved by TSADCNN and TGRA, there is an increase of $11\%$ and $9\%$ in ${mP}_{\mathrm{CA}}$, respectively, alongside lower false association rates and faster association times. These findings underscore the superior ability of TSA-cTFER to extract trajectory characteristics.

It should be noted that the results of TSA-cTFER and Multiple Hypothesis TSA are similar, with TSA-cTFER demonstrating slightly superior association accuracy and lower error rates. However, it necessitates a longer average association time. This may be due to the significant correspondence between the motion model assumed by the Multiple Hypothesis TSA approach and the actual motion model in MTAD, contributing to its robust performance.

5.5.2. MSTSA Task

The results of the multi-source track segment association using different models are shown in

Table 2. Results of Multi-source Track Segment Association.

	#params. (M)	${\mathit{mP}}_{\mathbf{CA}}$ (%)	${\mathit{mP}}_{\mathbf{FA}}$ (%)	${\mathit{mT}}_{\mathbf{AP}}$ (s)	${\mathit{mwP}}_{\mathbf{CA}}$ (%)	${\mathit{mwP}}_{\mathbf{FA}}$ (%)	${\mathit{mwT}}_{\mathbf{AP}}$ (s)
ED-based TSA	-	50.99	25.85	464.38	39.30	39.86	247.04
DTW-based TSA	-	54.24	41.77	505.52	35.27	60.56	638.35
TSADCNN (Sep.) [10]	0.908	94.68 ± 0.51	2.73 ± 0.40	235.85 ± 6.74	91.80 ± 0.94	5.13 ± 0.70	269.29 ± 11.55
TSADCNN [10]	1.301	92.85 ± 0.22	4.22 ± 0.22	259.49 ± 2.13	85.36 ± 0.30	8.05 ± 0.27	311.86 ± 3.01
TGRA (Sep.) [12]	0.534	93.81 ± 0.13	3.41 ± 0.11	252.84 ± 1.52	90.03 ± 0.42	6.47 ± 0.38	303.85 ± 3.13
TGRA [12]	0.534	91.61 ± 0.13	5.32 ± 0.11	271.25 ± 2.26	85.60 ± 0.31	10.59 ± 0.27	334.80 ± 2.54
TSA-cTFER (Sep.)	0.670	94.83 ± 0.38	2.61 ± 0.34	234.65 ± 1.10	92.20 ± 0.52	4.79 ± 0.47	269.79 ± 2.01
TSA-cTFER	0.674	94.25 ± 0.09	3.06 ± 0.11	246.95 ± 0.78	91.00 ± 0.12	5.75 ± 0.11	290.20 ± 1.47

. Traditional multi-source association methods based on distance metrics perform poorly in this task. In deep learning-based association methods, unlike the findings in the ITSA task, the performance of TSADCNN and TGRA improves significantly when trained separately for the MSTSA task compared to joint training for both tasks, resulting in an increase of approximately $1.8\%$ and $2.2\%$ in ${mP}_{\mathrm{CA}}$. However, TSA-cTFER shows consistent and better performance with fewer parameters, regardless of whether it is trained separately or jointly, highlighting its superior capability to represent trajectories compared to TSADCNN and TGRA. Furthermore, the performance gap between these three networks is minimal when trained separately, possibly because the MSTSA task is less complex than the ITSA task, as the trajectories of the MSTSA task exhibit greater similarity of spatial movement.

5.5.3. Analysis of a Typical Scenario

As shown in Figure 7a, a track segment association scenario in the MTAD is depicted. The scene depicts trajectory data recorded by two sensors, $\mathcal{A}$ and $\mathcal{B}$, spanning a period of time, denoted as $\{\mathcal{A}\_0,\mathcal{A}\_1,\mathcal{A}\_2,\mathcal{A}\_6\}$ and $\{\mathcal{B}\_0,\mathcal{B}\_3,\mathcal{B}\_5\}$. Each trajectory maintains varying durations, originating from distinct starting points and concluding at different endpoints while displaying similar motion patterns.

As shown in Figure 7b, it illustrates the variation of association scores output by TSA-cTFER over time in the context of the ITSA task. For the track segment $\mathcal{B}\_3$, there are two additional track segments, $\mathcal{B}\_0$ and $\mathcal{B}\_5$, waiting to be associated. Here, $\mathcal{B}\_0$ represents the true track segment, while $\mathcal{B}\_5$ denotes the false one. Furthermore, it can be seen from Figure 7a that the direction of movement of $\mathcal{B}\_5$ is opposite to that of $\mathcal{B}\_3$. By leveraging the robust trajectory representation capability of TSA-cTFER, the graph depicted in Figure 7b demonstrates a consistent decline in the association score of $(\mathcal{B}\_5,\mathcal{B}\_3)$ below the threshold ${\tau }_{low}$, leading to its classification as an invalid pair and subsequent disregard by the association algorithm as described in Algorithm 3. On the contrary, the association score of $(\mathcal{B}\_0,\mathcal{B}\_3)$ exceeds the threshold ${\tau }_{high}$, indicating that it will be considered a correct association by Algorithm 3.

As shown in Figure 7c, it depicts the change in association scores output by TSA-cTFER in the MSTSA task. For track segment $\mathcal{B}\_3$, there are four track segments $\mathcal{A}\_0$, $\mathcal{A}\_1$, $\mathcal{A}\_2$, and $\mathcal{A}\_6$ that could be associated with it, with $\mathcal{A}\_1$ and $\mathcal{A}\_2$ being the true track segments. The substantial spatial overlap among these segments presents challenges for the association. In the initial association stage, $\mathcal{B}\_3$ may be associated with $\mathcal{A}\_0$, $\mathcal{A}\_1$, and $\mathcal{A}\_6$. As more information is collected, the association score of the $(\mathcal{A}\_6,\mathcal{B}\_3)$ output by TSA-cTFER decreases, indicating that $\mathcal{A}\_6$ and $\mathcal{B}\_3$ are becoming less similar. Meanwhile, both $(\mathcal{A}\_0,\mathcal{B}\_3)$ and $(\mathcal{A}\_1,\mathcal{B}\_3)$ show high similarity, but the similarity between $\mathcal{A}\_1$ and $\mathcal{B}\_3$ is higher. Combining this with the M/N logic in Algorithm 2, the association algorithm will consider $(\mathcal{A}\_1,\mathcal{B}\_3)$ as the correct match. In the subsequent association stage, since $\mathcal{A}\_0$ was not associated with $\mathcal{B}\_3$ earlier, it will be excluded, leading to the association between $\mathcal{A}\_2$ and $\mathcal{B}\_3$.

The variation in association scores in Figure 7 not only validates the powerful trajectory representation capability of TSA-cTFER, which combines contrastive learning to quickly and accurately assess trajectory similarity, but also demonstrates the effectiveness of the two-stage online association algorithm. This algorithm can address challenging association pairs by employing delayed matching and using more information to assist in the decision-making process.

5.6. Ablation Study

We performed ablation experiments on different components of TSA-cTFER to evaluate their specific impacts on the MTAD validation dataset. The ITSA and MSTSA tasks were concurrently trained for 100 epochs using the complete training dataset while ensuring that all other parameters remained consistent with those outlined in Section 5.4.3.

Effect of pooling strategy. We conducted tests, as described in Section 3.4, to investigate the influence of different pooling strategies on the results. As shown in Figure 8, regardless of the ITSA or MSTSA task, the MEAN-strategy outperforms others in both the metrics $m{P}_{\mathrm{CA}}$ and $m{P}_{\mathrm{FA}}$. In comparison, the MAX-strategy performs slightly worse than the MEAN-strategy. Surprisingly, the commonly used CLS-strategy in [18,35] shows the worst performance in this experiment. This discrepancy might arise from the fact that the CLS-strategy relies solely on a single temporal feature output, while the MEAN-strategy and MAX-strategy leverage all temporal feature outputs, thus allowing them to capture long-term trajectory features more effectively.

Effect of input trajectory length. Different input trajectory lengths provide different amounts of information, with longer trajectories providing more insight into whether two trajectories originate from the same target. However, longer trajectories also introduce computational overhead and redundancy. Consequently, we experimented to determine the optimal input trajectory length. In Figure 9, it is observed that both ITSA and MSTSA show an improved average accuracy as the input trajectory length increases from 64 to 128, and MSTSA shows an increase of approximately $1.5\%$. However, as the input trajectory length continues to increase from 128 to 256, the average accuracy of ITSA decreases. It is suggested that input trajectory lengths beyond a certain point do not provide additional useful information to improve the learning capabilities of the model.

Effect of hidden state size. The sizes of hidden layers in a model determine its representation abilities. In Figure 10, when the hidden layer size is increased gradually from 32 to 256, there is an observed improvement in average accuracy for both ITSA and MSTSA tasks. The maximum improvement recorded is approximately $2\%$. Therefore, it can be concluded that larger models tend to achieve higher average accuracy. However, it is important to note that larger models also require more GPU memory and training time during the training process.

Effect of the number of layers. The depth of a model also affects its representational capacity. The deeper the model, the stronger its representational capacity. As shown in Figure 11, increasing the number of layers in the model from 2 to 6 resulted in an average accuracy improvement of approximately $1.3\%$.

Effect of training batch size. The TSA-cTFER training utilizes a contrastive learning framework, which leverages data within the same batch to generate positive and negative samples. Generally, larger batch sizes provide more negative samples, enabling the model to achieve the specified accuracy quicker in fewer epochs. However, excessively large batch sizes can pose challenges during model training. As illustrated in Figure 12, both too small and too large batch sizes have an impact on network learning, but overall the influence of batch size on performance remains within $1\%$.

Effect of contrastive loss temperature. The temperature parameter $\tau$ plays a crucial role in controlling similarity scaling during contrastive learning. Different temperature settings affect the smoothness or sharpness of similarity distributions, as described in Section 3.5. By tuning this parameter, we can effectively manage how close or distant the positive and negative samples are in the feature space, which significantly affects the model’s performance. Our experiments show that changing the temperature parameter from 0.05 to 1.0 leads to worse model performance, as shown in Figure 13. Conversely, smaller temperature parameters help the model to learn better features, with the optimal value being 0.07.

6. Discussion

Although the two-stage online association algorithm combined with the TSA-cTFER network has achieved good results, there are still some problems and areas for improvement. In this section, we first analyze the performance of TSA-cTFER in a complex association scenario characterized by multiple adjacent track segments. We then discuss the shortcomings of the TSA-cTFER and suggest possible avenues for future improvement.

6.1. Analysis of a Failure Scenario

In this challenging association scenario, as shown in Figure 14, multiple adjacent track segments lead to a high probability of misjudgment by the TSA-cTFER. Such misjudgments can significantly impact subsequent determinations of association pairs, resulting in the propagation of association errors. Specifically, the trajectories of the targets in the scene, illustrated in Figure 14a, show that the distances between the targets are very close and their motion patterns are similar. This similarity complicates the process of determining which track segments belong to the same target. The scores of various association pairs computed by TSA-cTFER over time, as depicted in Figure 14b, indicate that in this scenario, the scores for false associations surpass those for true ones. Consequently, the model outputs false associations such as $\left(\mathcal{A}\_45,\mathcal{A}\_5\right)$, $\left(\mathcal{A}\_51,\mathcal{A}\_49\right)$, and $\left(\mathcal{A}\_20,\mathcal{A}\_50\right)$. Furthermore, Figure 14c provides a schematic representation of the propagation of association errors. Initially, the model identifies the association pair $\left(\mathcal{A}\_45,\mathcal{A}\_5\right)$ as valid. Due to the one-to-one association constraint, it does not pair $\mathcal{A}\_5$ with its true associated track $\mathcal{A}\_49$. Subsequently, $\mathcal{A}\_49$ is paired with $\mathcal{A}\_51$, further restricting $\mathcal{A}\_51$ from pairing with $\mathcal{A}\_50$. Ultimately, $\mathcal{A}\_50$ can only be paired with $\mathcal{A}\_20$.

This situation leads the model to assign excessively high scores to false association pairs. As illustrated in Figure 14a, there is significant overlap between the track segments $\mathcal{A}\_45$ and $\mathcal{A}\_5$. Moreover, as discussed in Section 5.4.1, we utilized only the latitude and longitude position information during training; other data features, such as heading angle and speed, were excluded due to inaccuracies in the dataset. As a result, from the perspective of model input, the trajectory information exhibits considerable similarity, which can further mislead the model’s associations. Furthermore, we assess similarity based on the distance between feature vectors, processing each trajectory independently through the network. For efficiency reasons, we chose not to implement a framework that concatenates and jointly processes the two trajectories, which restricts the model’s effectiveness in the ITSA task, particularly in scenarios involving numerous similar trajectories.

6.2. Limitations and Future Work

We introduce the TSA-cTFER network to assess trajectory similarity, integrating it with a two-stage online association algorithm to effectively handle interrupted and multi-source track segments. Our experiments show that this approach outperforms existing state-of-the-art deep learning and traditional methods, although some shortcomings remain that require further improvement. To enhance our approach, we consider several key aspects following as:

•

Data Features and Augumentation: As highlighted in the failure scenarios above, recalculating speed and heading angle based on smoothed latitude and longitude and incorporating these features into the input embedding module of TSA-cTFER could improve the model’s ability to assess trajectory similarity. In addition, as noted in Section 5.4.3, this work has only used random cropping for data augmentation. Exploration of other techniques, such as random masking, may further improve the robustness of the model to meet practical application requirements.

•

Training Process: Building upon our data preparation strategy described in Section 5.4.1, we mixed the data from all scenarios and then randomly sampled a batch for training. Although this data sampling method proved effective in the early stages of training, it may not provide sufficient meaningful information for the model to learn effectively in the later stages, especially in failure scenarios where multiple similar trajectories exist. Thus, further fine-tuning of the network based on the specific scenarios may be necessary to optimize performance.

•

End-to-End Association: To achieve online track segment association, we propose a heuristic two-stage online association algorithm that heavily relies on TSA-cTFER’s assessment of trajectory similarity. However, this heuristic algorithm depends on manually defined rules and parameters, limiting its adaptability to different scenarios. Therefore, developing an end-to-end network that integrates trajectory information from the entire scenario and optimizes the association process through deep learning could effectively address these limitations, thereby improving accuracy and significantly reducing the time required for track association.

In summary, addressing these limitations through targeted improvements will enhance the overall performance and applicability of the TSA-cTFER network in real-world scenarios.

7. Conclusions

This paper focuses on addressing the challenges associated with interrupted and multi-source track segment association through a trajectory similarity comparison. The proposed network, TSA-cTFER, exploits the robust representational capabilities of contrastive learning and Transformer Encoder to transform trajectory similarity measurement into distance measurement within high-dimensional vectors. Additionally, in dynamic scenarios where targets continuously appear or disappear, a two-stage online association algorithm is introduced. This algorithm employs different association strategies based on the complexity of the matching trajectories, using M/N logic for easy associations and a delayed strategy for hard pairs. Finally, through the evaluation of real data, the two-stage online association algorithm combined with TSA-cTFER demonstrates excellent results compared to other models, achieving an average accuracy of 94.59% in the ITSA task and an average association accuracy of 94.83% in the MSTSA task, thereby validating the effectiveness of our approach.