Joint Object Detection and Multi-Object Tracking Based on Hypergraph Matching

Abstract

Addressing the challenges in online multi-object tracking algorithms under complex scenarios, where the independence among feature extraction, object detection, and data association modules leads to both error accumulation and the difficulty of maintaining visual consistency for occluded objects, we have proposed an end-to-end multi-object tracking method based on hypergraph matching (JDTHM). Initially, a feature extraction and object detection module is introduced to achieve preliminary localization and description of the objects. Subsequently, a deep feature aggregation module is designed to extract temporal information from historical tracklets, amalgamating features from object detection and feature extraction to enhance the consistency between the current frame features and the tracklet features, thus preventing identity swaps and tracklet breaks caused by object detection loss or distortion. Finally, a data association module based on hypergraph matching is constructed, integrating with object detection and feature extraction into a unified network, transforming the data association problem into a hypergraph matching problem between the tracklet hypergraph and the detection hypergraph, thereby achieving end-to-end model optimization. The experimental results demonstrate that this method has yielded favorable qualitative and quantitative analysis results on three multi-object tracking datasets, thereby validating its effectiveness in enhancing the robustness and accuracy of multi-object tracking tasks.

1. Introduction

Multiple Object Tracking (MOT) is an important task in the field of computer vision, aiming to stably track multiple objects of interest in a video sequence and generate their motion tracklets. It has been widely applied in autonomous driving, action recognition, mobile robotics, and intelligent surveillance. Depending on whether or not the generation of motion tracklets depends on the information of subsequent frames, MOT tasks can be divided into two different modes: offline tracking and online tracking. Offline tracking uses all the detection results in a video segment to jointly infer the motion tracklets of objects in each frame, which can be seen as a global optimization problem, while online tracking generates new tracklets sequentially according to the current detection results and historical tracklets. Due to its ability to process input video sequences in real-time, online MOT is widely used in real-world scenarios.

Early online MOT algorithms focused on complex association optimization methods, which had limited accuracy. With the rapid development of deep learning, online MOT algorithms based on deep learning have gradually become mainstream. At present, deep learning-based MOT methods are primarily divided into Detection-based-Embedding (DBE) [1,2,3,4,5,6,7,8] and Joint Detection and Embedding (JDE) [9,10,11,12,13,14].

DBE separates tracking into two independent tasks: object detection and data association. First, the detector outputs the detection results, which are then sent to the data association module to match with the tracklets of the previous frame, thus forming new tracklet boxes. DBE, with its straightforward approach and excellent tracking accuracy, once became the mainstream paradigm for MOT. The object detection module and data association module are trained separately to achieve the best performance, but they usually have the disadvantage of not being able to propagate errors back through the entire MOT system. In other words, each module is optimized individually, rather than for the entire MOT system. Moreover, when there are a large number of objects in the detection frame, due to the lack of feature sharing between the two modules in the DBE paradigm, a Re-ID model is needed for each bounding box (Bbox) in the detection frame, leading to the disadvantage of not achieving real-time inference speed.

JDE integrates the appearance embedding model used for data association into the detection module, so it can output detection results and corresponding appearance embeddings simultaneously. JDE uses a shared-weight, deep learning network to complete the tasks of object detection and data association, achieving good real-time performance.

Figure 1a depicts the input data for an MOT task, which is a sequence of object tracking images where each frame contains multiple targets represented by rectangular boxes of different colors. Figure 1b is a schematic representation of the target tracklets abstracted from Figure 1a, where the circular dots of different colors correspond to the target rectangular boxes in Figure 1a, triangles represent similar targets, the gray square indicates the occurrence of an intersection between two targets at this time, causing visual occlusion, the red curve represents the correct tracklet of the target, and the blue curve represents the incorrect tracklet of the target. It can be seen from the figure that when there are similar targets nearby or severe occlusions occur, it is easy to accumulate errors or maintain visual consistency. If attention is paid to the higher-order interactions between multiple targets, as indicated by the purple part in the figure, it can effectively mitigate the impact of these situations. However, as shown in Figure 1, due to the difference in features required by the object detection task and the Re-ID task, there is a competitive phenomenon between the detection task and the Re-ID task within the JDE paradigm; for the object detection task, the model aims to increase the similarity of appearance features of objects belonging to the same category, i.e., to narrow the intra-class distance, while for the Re-ID task, the model aims to maximize the differences in appearance features between different objects, i.e., to expand the inter-class distance. In addition, when the tracked objects are occluded by adjacent or cluttered objects, especially in severe occlusion, the reliability of the features detected will decrease, making it difficult to maintain the visual consistency of the occluded objects. Moreover, because most of the association modules in the above two paradigms use the Hungarian matching data association algorithm, the errors generated in the data association module cannot be propagated back to the object detection module and the feature embedding module, preventing the formation of an end-to-end optimization.

In response to the above issues, we propose an MOT method (JDTHM) based on hypergraph matching association, which enhances object detection and feature extraction through the tracklet information formed by the data association module in an end-to-end tracking model to form positive feedback. Specifically, the end-to-end tracking model uses tracklet information to enhance the visual features of object detection, thereby reducing false positives and false negatives. At the same time, the enhanced visual features assist the feature extraction module in generating discriminative features, thereby reducing the impact of occlusion. Based on more robust detection results and aggregated features, the data association module can generate higher quality tracklets. The main contributions are as follows:

• We introduce a feature extraction and detection module to initially achieve object localization and description of detection results.
• We design a deep feature aggregation module that extracts temporal information from historical tracklet information, aggregates features from object detection and feature extraction, enhances the consistency area between the current frame features and tracklet features, and avoids the impact of identity swapping and tracklet breaks caused by object detection loss or distortion.
• We construct a data association module based on the hypergraph neural network, integrating with object detection and feature extraction into a unified network. This integration transforms the data association problem into a hypergraph matching problem between the tracklet hypergraph and the detection hypergraph, thereby achieving end-to-end model optimization.
• The hypergraph framework demonstrates highly competitive performance on three MOT datasets, confirming its effectiveness in enhancing the accuracy of change detection.

The remainder of this paper is organized as follows: Section 2 reviews the related work; Section 3 introduces the relevant theories of hypergraph matching; Section 4 provides a detailed introduction to the proposed hypergraph architecture; Section 5 presents the experimental results and discussion; Section 6 concludes the paper.

2. Related Works

2.1. Detection-Based-Embedding

Online MOT algorithms based on deep learning are divided into two frameworks: detection-based tracking and joint detection tracking, with the difference being the presence or absence of a tracking module integrated with the object detection network. MOT algorithms based on deep object detectors [15,16,17] incorporate high-performance deep learning object detectors in the detection part of the MOT process, such as Region-based Convolutional Neural Networks (R-CNNs) [18,19], Single Shot Multibox Detector (SSD) [20], and You Only Look Once (YOLO) [21,22], to enhance the performance of online MOT algorithms. The typical Simple Online and Real-time Tracking (SORT) algorithm [15] is based on the traditional Hungarian association algorithm and uses the Faster R-CNN object detection network to replace the original Aggregate Channel Feature (ACF), achieving a significant improvement in tracking accuracy and speed. Subsequent research [6] has also shown a high positive correlation between detection accuracy and online MOT performance.

MOT algorithms based on deep object detectors generally take good object detection results as the pre-input and make improvements from the perspectives of appearance feature extraction, object feature prediction, and data association.

Compared to some online trackers [15,23,24] that only use simple motion modeling, the appearance features of the object can more robustly track objects that are occluded for a long time. For example, DeepSORT [25] uses a pre-trained ResNet network [26] on a large-scale dataset to extract the appearance features of the object, and then incorporates the similarity measurement of appearance features into the association cost for association, reducing the problem of object identity switching during the tracking process. To better simulate temporal information, some algorithms, independent of the detector, use the encoded vectors of the object’s geometric features (position, length, width, etc.) in historical frames and the appearance feature vectors and predict the change of the feature vectors through a temporal model for data association. For example, Exponential Moving Average (EMA) [10] aggregates the tracklet features to prevent occlusion and outlier feature points from affecting the match. Dai et al. [27] connect the average appearance features with spatiotemporal features and then use Graph Convolutional Networks (GCNs) [28] to aggregate features to obtain long-term appearance features. In addition, for data association based on matching, a cost matrix is constructed through the similarity of tracklet features and detection features, and then the allocation matrix is solved, with the allocation matrix supervising the learning of motion features and appearance features. For instance, DeepMOT [29] proposes a deep Hungarian network, using a Bidirectional Recurrent Neural Network (BRNN) [30] to convey global information in the cost matrix, and its loss function is designed based on two differentiable evaluation indicators, making the allocation matrix output by the network more conducive to the improvement of the indicators. Shan et al. [31] further integrate the object’s position in the previous frames and appearance features to obtain better global spatiotemporal information, alleviating the problem of object feature variation over time.

Each module of the detection-based tracking algorithm can be optimized separately, and with the improvement of the performance of each module the algorithm can achieve a higher tracking accuracy [32,33,34]. However, the coupling between modules is not high, and the independent training method of multiple modules may cause the model parameters to converge to the optimum of each module, rather than the global optimum of MOT. Moreover, some models improve the tracking effect by stacking multiple modules and using more deep learning techniques, which makes the DBE framework increasingly complex, and the tracking real-time performance does not meet online requirements, making it difficult to deploy in practice.

2.2. Joint Detection and Embedding

In recent years, the Joint Detection and Embedding (JDE) frameworks have reduced the complexity of the DBE frameworks while also improving the accuracy of MOT [35,36,37]. Its strategy is to integrate some functional modules to a certain extent on the basis of DBE, reducing the algorithmic complexity brought by phased processing while increasing the coupling between functional modules, alleviating the problem of local optimization brought by manually connecting each module.

In the detection algorithm, a head network is added to infer the whole picture information through an integrated network, then extract the corresponding pixel points of the object on the feature map as the object’s appearance feature vector and output the object’s detection box at the same time. JDE uses a YOLOv3 framework [38], adding an Re-ID branch parallel to the detection branch and extracting the feature vector at the center point of the positive anchor box on the feature map output by the branch as the object’s appearance feature vector. However, as mentioned in FairMOT [11], the predicted anchor center may not fall within the object area, so the anchor box mechanism is not suitable for training appearance features; they built tracking algorithms on CenterNet [39] by using Deep Layer Aggregation (DLA) [40] in the backbone architecture and extracting feature vectors at the object center point to learn appearance features. One of the main challenges of the joint learning of detection and tracking comes from the conflict between these two tasks. The purpose of the detection task is to identify object categories from the background, such as pedestrians and vehicles, hoping to improve the similarity of appearance features belonging to the same category of objects, that is, to narrow the intra-class gap. While the purpose of the Re-ID embedding is to distinguish different objects, not categories, hoping to maximize the appearance feature differences between different objects, that is, to expand the intra-class gap. Therefore, some methods decouple the feature maps of different tasks. For instance, Yang et al. [41] use a separate Re-ID head to learn appearance features in addition to the original detection branch. CSTrackv2 [12] learns the correlation and difference information between detection and re-identification through a channel mutual attention mechanism, and then use a multi-scale attention module to extract appearance features. SimpleTrack [42] is based on the FairMOT model and uses a feature fusion branch different from detection to extract appearance features. The design of these modules has increased the modeling of feature maps in the head network, enriched the differences between the respective task heads and the original feature maps, and improved the detection performance and recognition performance of the algorithm. Similarly, the appearance features extracted in a single frame image cannot obtain temporal information, so some researchers have used a multi-frame feature extraction method. For example, GSDT [43] is based on the FairMOT algorithm, using a graph convolutional model to model the local correlation between tracklet features and the current feature map, making the extracted appearance features contain historical information and improving the adaptability to object occlusion and deformation. However, since most of the association modules use the Hungarian matching data association algorithm, the errors generated in the data association module cannot be propagated back to the object detection module and the feature embedding module, and an end-to-end optimization cannot be formed.

The feature prediction and detection phase of JDE are carried out simultaneously, through the addition of frame information in the detection network or multi-frame detection. In order to learn the appearance and motion features of the tracklets across multiple frames, model the temporal characteristics, and predict the tracklet direction, some researchers use a multi-frame network or a head aggregation network in the form of joint detection and tracking to learn spatiotemporal feature maps for tracking. For example, CTrack [44] follows the CenterNet anchor-free detection framework, splicing a pair of sequential frames and the previous frame’s heatmap at the input end and predicting the object center position, size, offset, and inter-frame object displacement at the output end, which can achieve two-dimensional and three-dimensional MOT at the same time. TransCenter [45] also embeds Transformer into MOT, processing dense pixel-level, multi-scale detection and tracking queries in two query learning networks based on a variable-shaped Transformer encoder and decoder to obtain detection and predicted displacement. JDE’s data association and object detection are carried out simultaneously, using a single neural network to process the input image and directly infer the tracklet of the object. The object location can be refined through the correlation between the tracklet features and the current feature map. Some researchers have integrated single-object tracking algorithms into multi-object algorithms [46], using the powerful feature extraction and positioning capabilities of single-object algorithms to achieve higher precision end-to-end tracking. SOTMOT [47] is built on a variant of DLA-34, solving the ridge regression coefficients of the previous frame for each object and using the learned coefficients to predict the labels of the next frame. Therefore, discriminative features can be learned in the local area. The disadvantage is that as the number of objects in the scene increases, assigning a single-object tracker to each object will cause efficiency problems. Recently, with the progress of Transformer [48], some researchers have also used the query vector mechanism in the Transformer model for MOT, to learn the tracklet features with the query vector for frame-by-frame positioning and object identification, implicitly completing data association, thereby simplifying the complex and multi-step architecture. Meinhardt et al. [49] were the first to introduce Transformer into MOT. They iterated the DETR (end-to-end object detection with transformers) detector [50] in the time direction. The first frame uses randomly initialized empty queries to detect objects in the image and output the corresponding queries for the objects. In subsequent frames, the object queries from the previous frame and the randomly initialized empty queries are input together into the decoder, where the object queries are responsible for detecting the position of the tracklet in the current frame and passing the identity, implicitly completing data association, while the empty queries are responsible for detecting newly appearing objects.

The joint detection and tracking algorithms integrate different modules with the detection network for joint optimization, expecting the modules of the algorithm to work more collaboratively and simplifying the complex multi-step form, thereby improving the speed of MOT, which has received widespread attention in recent years.

3. Preliminaries of Hypergraph Matching

We introduce the preliminary concept association graph, on which our methods are based. For clarification, we first summarize important notations and definitions in

Table 1. Notations and definitions.

Notation	Definition
$\mathcal{G}$	Graph/Hypergraph
$\mathcal{V}$	The set of vertices of $\mathcal{G}$
$\mathit{\epsilon }$	The set of hyperedges of $\mathcal{G}$
$N$	The number of vertices on $\mathcal{G}$.
$M$	The number of hyperedges on $\mathcal{G}$.
$\mathbf{A}$	The adjacency matrix $\mathbf{A}=\overline{\mathbf{P}}{\overline{Q}}^{\mathrm{T}}$, where $\overline{\mathbf{P}},\overline{Q}$ are the connectivity matrices.
$D_v$	The diagonal matrix of vertex degrees.
$D_e$	The diagonal matrix of hyperedge degrees.

.

Graph Matching (GM) is a fundamental problem, which is NP-complete. It aims to match graphs ${\mathcal{G}}^1=({\mathcal{V}}^1,{\mathit{\epsilon }}^1)$ and ${\mathcal{G}}^2=({\mathcal{V}}^2,{\mathit{\epsilon }}^2)$, involving the establishment of node correspondences between the two graphs based on node-to-node and edge-to-edge affinities. It can be formulated as a Quadratic Assignment Programming (QAP) problem [51], in graph matching tasks, assuming a perfect match corresponds to the highest affinity score, the objective $\mathcal{J}(X)$ needs to be maximized while satisfying certain constraints.

$$\begin{array}{l}\mathcal{J}(X)=\mathrm{vec}{(X)}^{\mathrm{T}}{\rm K}\mathrm{vec}(X),\\ s.t.X\in {[0,1]}^{n_1\times n_2},X{1}_{n_2}=1_{n_1},X^{\mathrm{T}}{1}_{n_1}\le 1_{n_2}\end{array},$$

(1)

where $X$ is a doubly stochastic matrix, where all its rows sum to 1 and all its column sums are ≤1. $1_{n_1},1_{n_2}$ represent column vectors of length $n_1$ and $n_2$, with all elements equal to 1, where $n_1$ and $n_2$ denote the number of vertices in ${\mathcal{G}}^1$ and ${\mathcal{G}}^2$.

Initially, to measure the similarity between nodes in two graphs, an affinity matrix, ${\rm K}\in {ℝ}^{n_1{n}_2\times n_1{n}_2}$, is constructed. The diagonal elements, ${{\rm K}}_{ia,ia}\in s_v({\mathcal{V}}_i^1,{\mathcal{V}}_a^2)$, represent the first-order node similarity between node ${\mathcal{V}}_i^1$ and node ${\mathcal{V}}_a^2$. The off-diagonal elements, ${{\rm K}}_{ia,jb}\in s_e({\mathit{\epsilon }}_{ij}^1,{\mathit{\epsilon }}_{ab}^2)$, indicate the second-order edge similarity between edges ${\mathit{\epsilon }}_{ij}^1$ and ${\mathit{\epsilon }}_{ab}^2$.

Subsequently, as shown in Figure 2, the association graph, ${\mathcal{G}}^{\mathrm{A}}=({\mathcal{V}}^{\mathrm{A}},{\mathit{\epsilon }}^{\mathrm{A}})$, between the two graphs is constructed, transforming the graph matching problem into the node classification problem on the association graph, which simplifies the problem’s structure and allows the application of graph embedding and machine learning techniques. The nodes of the association graph, ${\mathcal{V}}_{ia}^{\mathrm{A}}$, represent potential matching relationships between the nodes ${\mathcal{V}}_i^1$ and ${\mathcal{V}}_a^2$, where the superscript means the index of graphs and the subscript means the index of vertices. Specifically, if ${\mathcal{V}}_i^1$ is a node in ${\mathcal{G}}^1$ and ${\mathcal{V}}_a^2$ is a node in ${\mathcal{G}}^2$, then vertex ${\mathcal{V}}_{ia}^{\mathrm{A}}$ in the association graph represents the match between ${\mathcal{V}}_i^1$ and ${\mathcal{V}}_a^2$. The edges in the association graph, ${\mathit{\epsilon }}_{ia,jb}^{\mathrm{A}}$, represent the consistency between matching pairs. If two matching pairs, $({\mathcal{V}}_i^1,{\mathcal{V}}_a^2)$ and $({\mathcal{V}}_j^1,{\mathcal{V}}_b^2)$, can coexist in an optimal match, then there is an edge between ${\mathcal{V}}_{ia}^{\mathrm{A}}$ and ${\mathcal{V}}_{jb}^{\mathrm{A}}$. The weighted adjacency matrix of association graph $W$ comes from the off-diagonal elements of ${\rm K}$. To better exploit the first-order similarities, diagonal elements, ${{\rm K}}_{ia,ia}$, are further assigned as node attributes, ${\mathcal{V}}^{\mathrm{A}}$.

In order to perform node classification on the association graph, GCNs are first used to embed the nodes of the association graph, capturing the topological structure information between nodes and generating feature representations for each node. We denote $v^{(k)}\in {ℝ}^{n_1{n}_2\times l_k}$ as $l_k$-dimensional vertex embeddings on layer $k$. The initial embeddings at k = 0 are scalar, i.e., $l_k=1$, taken from the diagonal of ${\rm K}$.

By outputting a doubly stochastic matrix through the Sinkhorn network, the node embeddings generated by GCNs are transformed into representations that satisfy matching constraints. By introducing matching constraints in the embedding process, the matching-aware embedding not only utilizes the topological structure information of the nodes but also more directly learns node representations that meet the matching conditions, thereby enhancing the performance of the graph matching task.

The hypergraph is a generalization of the graph. A hypergraph, $\mathcal{G}=(\mathcal{V},\mathit{\epsilon })$, encompasses a collection of vertices and a set of hyperedges, where $\mathcal{V}$ denotes the set of vertices, with the number of vertices in $\mathcal{V}$ represented as $N$. Each vertex possesses certain semantic information. $\mathit{\epsilon }$ represents the set of hyperedges. Unlike edges in a simple graph that can only connect two vertices, each hyperedge can connect any number of vertices, and the number of hyperedges in $\mathit{\epsilon }$ is denoted as $M$. The above QAP models can also be generalized to the higher-order case. One line of work adopted a tensor-marginalization-based model for $p$-order ($p\ge 3$) hypergraph matching, resulting in a higher-order assignment problem [51].

4. Methodology

We follow the online Multiple Object Tracking (MOT) methodology and have designed an end-to-end MOT framework (JDTHM), as shown in Figure 3. This framework mainly includes a feature extraction and object detection module, a feature aggregation module that integrates historical tracklet information, and a data association module.

The feature extraction and object detection module preliminarily achieves the localization and description of detection results for objects. The deep feature aggregation module extracts temporal information from historical tracklet data, amalgamating features from object detection with those from feature extraction. This enhances the consistency between the current frame features and the tracklet features, preventing identity swaps and tracklet breaks caused by the loss or distortion of object detection. The data association module, based on hypergraph matching, receives information from object detection and feature extraction. It uses the tracklet information generated by the data association module to enhance the object-detection and feature extraction modules, thereby reducing false positives and false negatives. The aggregated features further assist the feature extraction module in generating discriminative features, thus minimizing variations caused by occlusions. With more robust detection results and features, the data association module can produce higher-quality tracklets.

4.1. Feature Extraction and Object Detection

To encompass a wider range of low-level and high-level features, and to adjust the receptive field for tracking objects of different shapes and scales, we use Deep Layer Aggregation (DLA) [40] as the backbone to extract feature maps from the input image $X^t$ to obtain the basic feature map $F^t$.

We employ the two-stage detector Faster R-CNN [52] as our detection component, merging the public detection results of the current frame with the tracking results from the previous frames as the candidate proposal ${\mathbb{P}}^t$. The corresponding detection results, ${\mathbb{B}}^t=\{{\mathit{B}}_1^t,{\mathit{B}}_2^t,\dots ,{\mathit{B}}_n^t\}$, are generated by the detector $\mathcal{D}(·)$, where $n$ represents the number of detected objects, and ${\mathit{B}}_i^t$ represents the corresponding candidate bboxes.

4.2. Deep Feature Aggregation

Due to the presence of complex scenarios such as partial occlusions and motion blur in videos, it is difficult to detect occluded objects using only the clues from the current frame. Based on the consistency of object motion, historical information contains rich spatiotemporal contextual information that can assist in detecting occluded objects. By leveraging the abundant temporally complementary information, the description of occluded objects is enhanced, thereby reducing identity switches. Therefore, we first constructed the tracklet embedding module, which maintains a tracklet buffer area to save the image feature information and motion information contained in all historical tracklets. The buffer area is divided into different sub-areas according to the number of historical tracklets. The features in each area are represented as $M_j^{t-1}=\{{\widehat{F}}_j^{t-\eta },{\widehat{F}}_j^{t-\eta +1},\dots ,{\widehat{F}}_j^{t-1}\}$, where ${\widehat{F}}_j^{t-1}$ and $\eta$ are the single-step embedding and cache length of the $j$-th object, respectively. The tracklet feature is updated to $T_j^t=\gamma (M_j^{t-1},{\widehat{F}}_j^t)$, where $\gamma$ denotes the feature aggregation operation.

Subsequently, an attention mechanism based historical tracklet feature enhancement module is adopted. When it is necessary to associate with the current frame object for data correlation, a memory feature fusion network is used to integrate all historical inputs of each tracklet, enhancing the features of the consistent areas between the historical tracklets and the current frame, to generate high-quality tracklets for subsequent modules.

Setting $\xi (x_i,x_j)={\exp }^{\varphi {\left(x_i\right)}^{\mathrm{T}}\phi \left(x_j\right)}$, the attention between the visual features of the current frame and the historical tracklet features is defined as follows:

$$\mathcal{A}(F_{i,\ast }^t,{{\rm T}}^{t-1})={\displaystyle \sum _{k=1}^m\frac{{\alpha }_{\ast ,k}{\displaystyle {\sum }_{\forall j}\xi (F_{i,\ast }^t,{{\rm T}}_{j,k}^{t-1})\theta ({{\rm T}}_{j,k}^{t-1})}}{{\displaystyle {\sum }_{\forall j}\xi (F_{i,\ast }^t,{{\rm T}}_{j,k}^{t-1})}}},$$

(2)

where $F_{i,\ast }^t$ and ${{\rm T}}_{j,k}^{t-1}$ represent random features from $F_{\ast }^t$ and ${{\rm T}}_k^{t-1}$, respectively. ${{\rm T}}_k^{t-1}\in {{\rm T}}^{t-1}$ represents the feature map of the $k$-th tracklet. $\varphi (·)$, $\phi (·)$ and $\theta (·)$ denote Convolution layers. ${\alpha }_{\ast ,k}$ is an indicator function, taking the value 1 if $\mathrm{IoU}({\mathit{P}}_{*}^t,{\mathit{B}}_k^{t-1})>{\alpha }_{\mathrm{IoU}}$ and 0 otherwise. $\mathrm{IoU}$ stands for “Intersection over Union”, which measures the degree of overlap between two regions, often used to compare the similarity between a predicted bbox or segmentation area and the ground truth bbox or segmentation area. $\mathrm{IoU}({\mathit{P}}_{*}^t,{\mathit{B}}_k^{t-1})$ represents the geometric similarity between the candidate proposal ${\mathit{P}}_{*}^t$ and the final bbox ${\mathit{B}}_k^{t-1}$ in the tracklet ${\mathcal{T}}_k^{t-1}$, where ${\alpha }_{\mathrm{IoU}}$ is the geometric threshold. Furthermore, the features enhanced by the historical tracklets are as follows:

$${\tilde{F}}_{\ast }^t=F_{\ast }^t\oplus \mathcal{A}(F_{\ast }^t,T^{t-1}),$$

(3)

where $\oplus$ represents element-wise addition and subtraction operations for matrices. The output is a fusion of features generated from the detection results of the current frame and the enhanced features of historical tracklets.

$${\widehat{F}}_i^t=\mu ({\tilde{F}}^t,{\mathit{B}}_i^t),$$

(4)

where $\mu$ is the convolution layer on top of the RoIAlign layer. The feature corresponding to ${\mathit{B}}^t$ can be obtained through ${\widehat{F}}^t=\{{\widehat{F}}_1^t,{\widehat{F}}_2^t,\dots ,{\widehat{F}}_n^t\}$.

4.3. Data Association

We transform the data association into a hypergraph matching problem between detection results ${\mathbb{B}}^t=\{{\mathit{B}}_1^t,{\mathit{B}}_2^t,\dots ,{\mathit{B}}_n^t\}$ and historical tracklets ${\mathbb{T}}^{t-1}=\{{\mathcal{T}}_1^{t-1},{\mathcal{T}}_2^{t-1},\dots ,{\mathcal{T}}_m^{t-1}\}$, where $n$ and $m$ represent the number of detection results and historical tracklets, respectively. We first construct a detection hypergraph and a tracklet hypergraph following NHGM [51] to describe the relationships between different objects in the detection results and historical tracklets. The hypergraph matching process is shown in Figure 4.

The detection hypergraph is defined as ${\mathcal{G}}_D=({\mathcal{V}}_D,{\mathit{\epsilon }}_D)$, where ${\mathcal{V}}_D=\{({\mathit{B}}_i^t,{\widehat{F}}_i^t)\},i\in [1,n]$ represents the vertex set, with ${\widehat{F}}_i^t$ representing the features of the bbox ${\mathit{B}}_i^t$. ${\mathit{\epsilon }}_D$ represents the set of hyperedges. Similarly, the tracklet hypergraph is defined as ${\mathcal{G}}_T=({\mathcal{V}}_T,{\mathit{\epsilon }}_T)$, where ${\mathcal{V}}_T=\{({\mathit{B}}_j^{t-1},T_j^{t-1})\},j\in [1,m]$ represents the vertex set, with $T_j^{t-1}$ representing the feature of final bbox ${\mathit{B}}_j^{t-1}$ of the $j$-th historical tracklet ${\mathcal{T}}_j^{t-1}$. ${\mathit{\epsilon }}_T$ represents the set of hyperedges.

The connectivity matrices of the detection hypergraph are denoted as ${\overline{\mathbf{P}}}_D$ and ${\overline{Q}}_D$, and those of the tracklet hypergraph are denoted as ${\overline{\mathbf{P}}}_T$ and ${\overline{\mathbf{Q}}}_T$. The adjacency matrices obtained are ${\overline{\mathbf{A}}}_D={\overline{\mathbf{P}}}_D{\overline{\mathbf{Q}}}_D^{\mathrm{T}}$ and ${\overline{\mathbf{A}}}_T={\overline{\mathbf{P}}}_T{\overline{\mathbf{Q}}}_T^{\mathrm{T}}$, respectively. The representation of the hyperedges is constructed by concatenating the vertex features at both ends of the edges.

$${\overline{\mathbf{E}}}_D=[{\overline{\mathbf{P}}}_D^{\mathrm{T}}{\widehat{F}}^t{\overline{\mathbf{Q}}}_D^{\mathrm{T}}{\widehat{F}}^t],$$

(5)

$${\overline{\mathbf{E}}}_T=[{\overline{\mathbf{P}}}_T^{\mathrm{T}}{T}^{t-1}{\overline{\mathbf{Q}}}_T^{\mathrm{T}}{T}^{t-1}],$$

(6)

We enhance the vertex features through cross-graph information transfer between the detection hypergraph and the tracklet hypergraph.

$${\widehat{F}}_i^{t,l+1}={\psi }_D({\widehat{F}}_i^{t,l}+{\displaystyle \sum _{j=1}^m(\cos ({\widehat{F}}_i^{t,l},T_j^{t-1,l})+IoU({\mathit{B}}_i^{t,l},{\mathit{B}}_j^{t-1,l})})),$$

(7)

$$T_j^{t-1,l+1}={\psi }_T(T_j^{t-1,l}+{\displaystyle \sum _{i=1}^n(\cos ({\widehat{F}}_i^{t,l},T_j^{t-1,l})+IoU({\mathit{B}}_i^{t,l},{\mathit{B}}_j^{t-1,l})})),$$

(8)

The enhanced vertex features are denoted as ${\widehat{F}}_i^{t,L}$ and $T_j^{t-1,L}$, meanwhile, the hyperedge features are denoted as ${\overline{\mathbf{E}}}_{D,i1i2}^L$ and ${\overline{\mathbf{E}}}_{T,j1j2}^L$. For hypergraph matching, the above QAP model is generalized to the higher-order case, adopting a tensor-marginalization-based model for $p$-order ($p\ge 3$) hypergraph matching, resulting in a higher-order assignment problem [51]:

$$\begin{array}{l}\mathcal{J}(x)=H{\otimes }_{1}x{\otimes }_{2}x\dots {\otimes }_{p}x\\ s.t.X{1}_{n_2}=1_{n_1},X^{\mathrm{T}}{1}_{n_1}\le 1_{n_2}\end{array},$$

(9)

where $x=\mathrm{vec}(X)$ is the column-vectorized form and $H$ is the $p$-order affinity tensor whose elements record the affinity between two hyperedges, operated by tensor product ${\otimes }_q$, where ${\otimes }_q$ can be regarded as tensor marginalization at dimension $q$.

To measure the similarity between vertices in the two hypergraphs, an affinity tensor, $H$, is constructed. The first-order vertex similarity, the second-order edge similarity, and the third-order angle similarity matrices of the affinity tensor $H$ can be represented as follows:

$$H_{i,j}^{(1)}=\cos ({\widehat{F}}_i^{t,L},T_j^{t-1,L}),$$

(10)

$$H_{i1i2,j1j2}^{(2)}=\cos ({\overline{\mathbf{E}}}_{D,i1i2}^L,{\overline{\mathbf{E}}}_{T,j1j2}^L),$$

(11)

$$H_{{\omega }_1,{\omega }_2,{\omega }_3}^{(3)}=\exp \{-({\displaystyle \sum _{q=1}^3|\cos {\beta }_D^{{\omega }_q}-\cos {\beta }_T^{{\omega }_q}|})/{\sigma }_3\},$$

(12)

where ${\beta }_D^{{\omega }_q}$ and ${\beta }_T^{{\omega }_q}$ represent the angles between $H_D$ and $H_T$, respectively.

Finally, we follow the computational process of hypergraph matching vertex updates as described in NHGM [51], utilizing hypergraph matching with high-order contextual information to match the detection hypergraph with the tracklet hypergraph. Hypergraph convolutional networks are used to embed the vertices of the association graph, capturing the topological structure information between nodes and generating feature representations for each node. A Sinkhorn network outputs a doubly stochastic matrix, transforming the vertex embeddings generated by HGCN into representations that meet the matching constraints. After each HGCN layer, the output of the Sinkhorn network (i.e., the predicted matching matrix) is concatenated with the vertex embeddings of the current layer. By introducing matching constraints in the embedding process, the matching-aware embeddings can not only utilize the topological structure information of the nodes but also more directly learn the node representations that satisfy the matching conditions, thereby improving the performance of the graph matching task. If the corresponding similarity in $H$ is higher than the affinity threshold $\delta$, then two nodes between the detection hypergraph and the tracklet hypergraph are matched. Finally, optimal matching can be obtained. For detections that do not match any tracklet, if the detection confidence score is greater than a certain threshold, we will use it to initialize a new tracklet. Unmatched tracklet representations suggest that they may have left the scene or were not detected.

During the training process, our proposed algorithm can be optimized in an end-to-end manner using the following loss function:

$$\mathcal{L}={\mathcal{L}}_{Det}+{\mathcal{L}}_{Emb}+{\mathcal{L}}_{HGM},$$

(13)

where ${\mathcal{L}}_{Det}$, ${\mathcal{L}}_{Emb}$, and ${\mathcal{L}}_{HGM}$ represent the detection loss, feature loss, and hypergraph matching loss, respectively.

5. Experiments

5.1. Datasets and Evaluation Metrics

We evaluate our method on MOTChallenges [53,54,55], including MOT16, MOT17 and MOT20 benchmarks. The MOT16 and MOT17 datasets both include 14 video sequences, and the MOT20 dataset includes 8 video sequences, with high density and extremely complex scenes. Following the CLEAR MOT metrics [56], HOTA [57], and IDF1 score [58], we apply some basic items for quantitative evaluation, e.g., Multiple Object Tracking Accuracy (MOTA ↑), Higher Order Tracking Accuracy (HOTA ↑), IDF1 (↑), False Positives (FP ↓), False Negatives (FN ↓), and Identity Switches (IDS ↓). MOTA focuses on the performance of the detection branch, HOTA can comprehensively assess the performance of both the detection branch and data association, while IDF1 evaluates the ability to maintain identity, paying more attention to the association performance.

5.2. Benchmark Evaluation

To demonstrate the advantages of our hypergraph-based architecture in the MOT task, we have selected many pioneering and influential methods for comparison. The compared methods are divided into online tracking and offline tracking. We have summarized the MOTChallenge results of different methods in

Table 2. Comparison with different MOT methods on MOT16 benchmark with the provided public detections, showing the quantitative results.

Method	MOTA	HOTA	IDF1	FP	FN	IDS
GMTracker [3]	61.1	51.2	66.6	3891	66,550	503
MPNTrack [6]	58.6	48.9	61.7	4949	70,252	354
DeepMOT [29]	54.8	42.2	53.4	2955	78,765	645
LPC [27]	58.8	51.7	67.6	6167	68,432	435
Lif_T [59]	57.5	49.6	64.1	4249	72,868	335
TMOH [60]	63.2	50.7	63.5	3122	63,376	635
UTM [61]	63.8	53.1	67.1	8328	57,269	428
JDTHM	67.6	56.3	69.1	6137	52,686	377

,

Table 3. Comparison with different methods on MOT17 benchmark with the provided public detections, showing the quantitative results.

Method	MOTA	HOTA	IDF1	FP	FN	IDS
GMTracker [3]	59.0	51.1	65.9	20,395	209,553	1105
MPNTrack [6]	58.8	49.0	61.7	17,413	213,594	1185
FairMOT [11]	73.7	59.3	72.3	27,507	117,477	3303
LPC [27]	59.0	51.7	66.8	23,102	206,947	1122
CTTrack [44]	61.5	48.2	59.6	14,076	200,672	2583
TransCenter [45]	71.9	54.1	62.3	17,378	137,008	4046
Trackformer [49]	74.1	57.3	68.0	34,602	108,777	2829
Lif_T [59]	60.5	51.3	65.6	14,966	206,619	1189
CSTrack [62]	74.9	59.3	72.3	23,847	114,303	3567
PermaTrack [63]	73.1	54.2	67.2	24,577	123,508	3571
JDTHM	73.9	59.6	71.7	25,639	117,756	3733

and

Table 4. Comparison with modern methods on MOT20 benchmark with the provided public detections, showing the quantitative results.

Method	MOTA	HOTA	IDF1	FP	FN	IDS
FairMOT [11]	61.8	54.6	67.3	103,440	88,901	5243
CSTrack [62]	66.6	54.0	68.6	25,404	144,358	3196
GSDT [43]	67.1	53.1	67.5	31,913	135,409	3133
MPNTrack [6]	57.6	46.8	59.1	16,953	201,384	1210
Trackformer [49]	68.6	54.7	65.7	20,348	140,373	1532
CorrTracker [64]	65.2	57.2	69.1	79,429	95,855	5183
RelationTrack [65]	67.2	56.5	70.5	61,134	104,597	4243
GNNMatch [66]	54.5	40.2	49.0	9522	223,611	2038
JDTHM	69.3	56.7	70.9	29,716	127,135	2288

. For the aforementioned methods, if the performance on the corresponding dataset is reported on the official leaderboard, we adopt it directly. If not, we compare it based on the latest data provided in the selected papers.

In most metrics, the proposed method has achieved good performance on most evaluation metrics. Since the MOTA metric emphasizes the detection side more, we believe that our strong HOTA performance indicates that our detector and data association modules perform well. In addition, our impressive IDF1 performance indicates that, due to the spatiotemporal relationship modeling through hypergraph interactions, our method can produce temporally consistent tracklets.

Among the existing methods, the one most related to us is GMTracker [3], which uses a similar graph matching method for identity association. The main differences and novelties lie in our use of knowledge extracted from historical tracklets to enhance the object detection and feature embedding modules. On the MOT16 dataset, our proposed method has improved by 6.5%, 5.1%, and 2.5% in MOTA, HOTA, and IDF1 metrics, respectively. On the MOT17 dataset, compared to GMTracker [3], our proposed method has improved by 14.9%, 8.5%, and 5.8% in MOTA, HOTA, and IDF1 metrics, respectively. Particularly, on the MOT20 dataset, we also compared with the JDE method GSDT [43], and our proposed method has improved by 2.2%, 3.6%, and 3.4% in MOTA, HOTA, and IDF1 metrics, respectively. A positive feedback loop that fuses historical tracklet features is used for enhancement and correction.

At the same time, compared to the second-order information features used in other MOT methods based on graph neural networks, our method can not only extract higher-order information features but also pay more attention to the visual consistency relationship of occluded tracking objects, thereby helping the tracker to associate better in severe occlusion situations. Another point worth mentioning is that our IDS is better than most mainstream methods; the smaller the IDS, the fewer the number of identity switchovers in the tracking tracklet and the higher the reliability of the tracking results.

Figure 5 displays the qualitative analysis results of our hypergraph architecture on the MOT17.

Figure 6 displays the qualitative analysis results of our hypergraph architecture on the MOT20.

5.3. Ablation Study

To verify the impact of the proposed modules on tracking performance, we conducted ablation studies on the modules within the framework, and the results of the impact of different modules on tracking performance are shown in

Table 5. Ablation study of the model and a report of the MOTA/HOTA/IDF1 scores for each configuration on the datasets.

Baseline	Feature Aggregation	Hypergraph	MOTA	HOTA	IDF1
✓	-	-	67.1	56.3	65.9
✓	✓	-	72.3	57.1	66.2
✓	-	✓	71.6	57.9	69.4
✓	✓	✓	73.9	59.6	71.7

. From the perspective of the MOTA metric, the TFE module significantly improved the tracker’s accuracy, with an increase of 5.2% in MOTA, while from the perspective of the IDF1 metric, the hypergraph matching module notably enhanced the association precision, with an increase of 3.5% in IDF1. This proves that the graph matching algorithm overlooked the similarity of third-order angles, a similarity that can model group activities, thereby generating more reliable tracklets and fully demonstrating the potential of the hypergraph architecture in MOT tasks. However, due to the computational cost of high-order structures, which explodes to $O\left(\right(N_D{N}_T{)}^p)$, where $p$ is the order, hypergraph neural networks with structures higher than the third order were not explored in this paper. Moreover, since our proposed method is an end-to-end framework, the performance of each module was greatly enhanced when the two modules were combined, with an increase of 6.8% in MOTA, 3.3% in HOTA, and 5.8% in IDF1.

Furthermore, to verify the impact of high-order structural information in hypergraph matching on tracking performance, we conducted ablation experiments on different orders of high-order structural information in hypergraph matching, with the results shown in

Table 6. Effect of the order of high-order structural information in hypergraph matching on tracking performance and a report of the MOTA/HOTA/IDF1/IDS scores for each configuration on the datasets.

Order	MOTA	HOTA	IDF1
1	64.4	53.6	65.9
2	66.9	54.6	68.6
3	69.3	56.7	70.9

. In the paper, we only discuss the case where $p\le 3$. As the order of high-order information in the hypergraph matching progress increases, both MOTA and IDF1 also show an increasing trend. This trend demonstrates that when learning spatiotemporal consistency, high-order structural information relationship features are more effective.

6. Conclusions

We designed an MOT framework based on hypergraph association. Through the feature extraction and detection module, we achieved initial localization and detection of objects. By integrating the features from object detection with those from feature extraction, we enhanced the consistency between the current frame features and the tracklet features, avoiding issues such as identity swapping and tracklet fragmentation due to the loss or distortion of object detection. We modelled the data association process using a hypergraph neural network, effectively capturing the relational information between objects. Hypergraph matching was used to associate both occluded and non-occluded objects, maintaining the visual consistency of the tracked objects. The experimental results showed that this method achieves good qualitative and quantitative analysis on three MOT datasets, thereby proving its effectiveness in improving the robustness and accuracy of MOT tasks.