Multi-Agent-Supported Tracking Based on Hidden Markov Model with Weighted Entropy

Abstract

Multi-object tracking (MOT) has witnessed significant advancements in recent years, yet it remains challenged by complex uncertainties arising from pedestrian movement patterns. To address this, we present a unified framework that explicitly models pedestrian dynamics through a dual-phase paradigm, combining a Hidden Markov Model (HMM) for motion modeling and weighted entropy for adaptive multi-cue fusion. Furthermore, a multi-agent architecture is employed for track management, enabling parallelized state estimation and seamless integration of the HMM-based Kalman filter with multi-cue fusion. Quantitative evaluations show that our method achieves 82.1 in IDF1, 81.5 in MOTA, 65.9 in HOTA, and 1,255 IDs on the MOT17 benchmark, and achieves 81.2 in IDF1, 78.4 in MOTA, 65.7 in HOTA, and 608 IDs on the MOT20 benchmark, and the application of the multi-agent mechanism significantly improves the scores on FPS as a result of efficient computation. The experimental results demonstrate that the proposed method achieves state-of-the-art performance, particularly in highly crowded scenes.

1. Introduction

With the rapid development of smart cities, the requirement for video surveillance systems has significantly increased. Visual object tracking, a fundamental task in computer vision, is divided into single-object tracking (SOT) and multi-object tracking (MOT). In recent years, MOT, aiming to locate multiple specific objects and maintain their unique identities in sequential frames, has received increasing attention due to its capability to advance video understanding, intelligent surveillance, human–computer interaction, and medical diagnostics. The tracking-by-detection (TBD) framework has recently become the dominant paradigm in MOT [1,2,3]. It transforms the MOT task into a frame-by-frame data association problem, typically implemented in a centralized system.

Extensive research has been dedicated to addressing complex environmental conditions and target occlusion, with the aim of mitigating association failures and target loss. However, pedestrian trajectories—the primary focus of MOT—often deviate from purely deterministic models. This is due to their spatiotemporal evolution, which is jointly governed by objective physical constraints and subjective cognitive decision-making processes. To characterize pedestrian locomotion patterns, we propose a dual-phase paradigm comprising a decision phase and an action phase. The decision phase encompasses non-observable processes including destination planning, obstacle avoidance, and speed adjustment. In contrast, the action phase only contains the observable process of physical movement. This dual-phase structure presents challenges for traditional tracking methods, as it introduces significant uncertainty due to variable cognitive intentions and external environmental perturbations.

The Hidden Markov Model (HMM), a kind of Probabilistic Graphical Model, provides a robust framework for modeling sequential probabilistic problems and performs efficient inference. Within this framework, latent states representing pedestrian intentions are influenced solely by the preceding state transition. This is particularly well-suited to the context of pedestrian tracking due to its ability to represent doubly stochastic processes: an unobservable decision phase governed by internal cognitive states and an observable action phase manifesting as measurement [4].

Further, target features suffer from severe uncertainties as well, influenced by changes in orientation, pose, or appearance. This variability aligns with the entropy framework, which quantifies information disorder. Leveraging this insight, we employ weighted entropy [5] to dynamically characterize feature uncertainty, enabling adaptive fusion strategies that balance noisy detections.

However, the introduction of these two mechanism causes unavoidable computation loads in the traditional approach of track management, in addition to the weak adaptability of the algorithm and limited scalability of the system. Thus, we introduce a multi-agent mechanism to manage the tracking of objects. With the hypothesis that pedestrians do not exhibit abrupt large displacements between consecutive frames, we formulate each object as an agent that maintains a separate HMM-based Kalman filter and independently decide the creation, association, prediction, termination, and weight of features for tracks, which is a Markov decision process. Each agent is trained based on rewards obtained from the environment. Detections are associated based on each filter’s prior prediction about the tracked object’s position. And the weight of features is determined by the evaluation of entropy given by the agents.

Therefore, the main contributions of this paper are as follows:

• Decomposing pedestrian locomotion into a decision phase and an action phase according to human cognitive patterns: This addresses the analysis of dynamic moving patterns instead of objective conditions. And the dual-phase structure naturally accommodates probabilistic modeling frameworks.
• Reconstructing Kalman filters within a unified HMM framework: This architecture explicitly models state transitions and observation processes corresponding to the dual-phase model, enabling probabilistic inference and better prediction of pedestrian trajectories.
• The proposal of a weighted entropy mechanism for adaptive multi-cue fusion, dynamically balancing appearance and motion cues based on real-time uncertainty quantification: By leveraging entropy as a measure of feature reliability, our mechanism adjusts fusion weights to prioritize stable features during tracking, enhancing robustness to occlusions and appearance changes.
• The introduction of a multi-agent mechanism to manage the tracks separately. By assigning each track to an independent agent, the HMM-based Kalman filter and multi-cue fusion with weighted entropy can be effectively conducted and updated according to the obtained rewards.

2. Related Work

In this section, we review recent developments in tracking-by-detection (TBD) frameworks and applications of probabilistic graphical models (PGMs) in multi-object tracking (MOT).

2.1. Tracking-by-Detection

In the TBD paradigm, a class of widely adopted MOT frameworks [1,2,3] employs a two-stage pipeline: objects of interest are first detected, and then cross-frame associations are established using complementary cues such as appearance and motion features. SORT [2] exemplifies this paradigm by leveraging the Kalman Filter for motion state prediction and the Hungarian algorithm for association. To address occlusion challenges, DeepSORT [3] enhances the association strategy by incorporating appearance features extracted from deep convolutional neural networks into the cost matrix. Building on this foundation, Du et al. [1] introduced the StrongSORT framework, which refines detection, embedding, and association modules through lightweight, plug-and-play components. Wang et al. [6] proposed a General Recurrent Tracking Unit that captures long-term temporal dependencies to score track proposals, mitigating detection. Zhang et al. [7] advanced monocular tracking via a detection multiplexing strategy, introducing a multiplex labeling graph model with independent nodes.

Despite their advancements, most of these methods predominantly rely on bipartite graph matching for data association, which often results in the inadequate modeling of intricate situations.

2.2. Multi-Agent Mechanism in MOT

Shen et al. [8] propose a distributed MOT algorithm based on multi-agent reinforcement learning, which formulates MOT as a Markov decision process to improve the adaptability of the algorithm. Focusing on efficient online approaches that can run in real time, Pol Rosello and Mykel J. Kochenderfer [9] present a novel, multi-agent reinforcement learning formulation of multi-object tracking that treats creating, propagating, and terminating object tracks as actions in a sequential decision-making problem.

2.3. Probabilistic Models in MOT

Segal and Reid [10] introduce a latent data association framework that models state nodes across continuous time slices as interconnected through probabilistic dependencies, reframing data association and model selection as a unified inference problem. To decrease computational complexity, Chen [11] employs Markov random fields with a Local Binary Pattern algorithm, replacing cluster graphs with a more tractable model with the assumption of prior knowledge of target count, detection probability, and clutter density. Wang, de La Corce, and Paragios [12] tackle the challenges of image noise, cluttered backgrounds, and camera motion by integrating a pairwise Markov random field into a joint 2.5D layered model, which enables simultaneous image segmentation and tracking.

However, these methods often overlook the distinct stochastic motion patterns of pedestrians and the inherent instability of tracking cues, treating them the same as other dynamic objects in the scene. To address this, we emphasize the dual-phase pattern of targets by formulating tracking within an HMM framework. And we introduce the weighted entropy mechanism to quantify the uncertainty of feature characterization capacity during multi-cue fusion, enabling the adaptive recalibration of weight to mitigate bias in noisy or cluttered conditions.

3. Multi-Agent-Supported Tracking Based on Hidden Markov Model with Weighted Entropy

3.1. Overview

Unlike traditional TBD methods that perform global optimization and track updates centrally, we address the MOT problem using the multi-agent mechanism, assigning independent agents to each track when a new target enters the scene. The agents’ role is to manage a separate Kalman filter and take actions to modify the track under its responsibility. As shown in Figure 1, in each frame the agent obtains detections from the detector and makes its decision according to the Kalman filter and weighted entropy mechanism. This framework can be conducted in both a decentralized way and a multi-thread way. The agents compete with each other to obtain the detections under the communication protocol proposed by Noukhovitch et al. [13] and adjust their strategies through trust-region policy optimization (TRPO) based on rewards determined by the joint tracking performance relative to ground truth object tracks. The reward is denoted by

$$\begin{array}{c}\hfill r_i=\omega dis{t}^m({\tau }_i,d_j)+(1-\omega )dis{t}^a({\tau }_i,d_j)\end{array}$$

(1)

$$dis{t}^m({\tau }_i,d_j)={\displaystyle \frac{{\tau }_i\cap d_j}{{\tau }_i\cup d_j}}$$

(2)

$$dis{t}^a({\tau }_i,d_j)=1-{\displaystyle \frac{{\tau }_i·d_j}{\parallel {\tau }_i\parallel \parallel d_j\parallel }}$$

(3)

where $dis{t}^m({\tau }_i,d_j)$ represents the motion similarity between track ${\tau }_i$ and the selected detection $d_j$, while the appearance similarity is denoted as $dis{t}^a({\tau }_i,d_j)$, and $\omega$ is a weight parameter.

3.2. Details of Process of Tracking

Based on the principle of StrongSORT [1], tracks are initialized with detections and their extracted features and assigned to an agent, as shown in Figure 1. In the t-th frame of the video sequence $V=\{f_1,f_2,\dots ,f_L\}$, the object detector first generates detections $D_t=\{d_1,d_2,\dots ,d_N\}$, where N represents the total number of detections. Each detection contains a location, width, and height parameter, denoted by $d_j=[x,y,w,h]$. These detections are distributed to agents as candidates, and the agents are allowed to either accept the detections or wait until the next frame. For agents choosing to wait, a reward of 1 is assigned if the corresponding ground-truth tracks also opt for waiting. Otherwise, they receive −1 as a penalty. In the case of agents accepting detections, a detection may potentially be assigned to more than one agent. For instance, there are four tracks managed by agents $\{{\tau }_2,{\tau }_4,{\tau }_5,{\tau }_6\}$ that attempt to claim detection $d_2$ at the same time, and it is obvious that $d_2$ cannot be selected repeatedly. To address this, ${\tau }_2$ is designated as the valid assignment, and $\{{\tau }_4,{\tau }_5,{\tau }_6\}$ are allocated to the collision set ${\mathcal{T}}_{collision}$ temporarily. Upon completion of all detection processing, the agents in the collision set locally compute and share the rewards of each candidate, and the total rewards are optimized to produce the final distribution of detections:

$$\begin{array}{c}\hfill r_{total}=\sum _i{r}_i\end{array}$$

(4)

$$\begin{array}{c}\hfill r_i=\omega dis{t}^m({\tau }_i,d_j)+(1-\omega )dis{t}^a({\tau }_i,d_j),{\tau }_i\in {\mathcal{T}}_{collision},d_j\in D_{candidate}\end{array}$$

(5)

Then the agents associate these detections with the managed track according to the cost matrix, constructed through multi-cue fusion, via weighted entropy. The cost matrix represents the similarity between existing tracks and new detections, and models a bipartite graph for the matching problem. Before fusion, the agent performs normalization on appearance and motion features. The entropy of the fused feature is then computed to quantify its uncertainty. Through entropy minimization, the optimal coefficient $\alpha$ is derived to balance feature weights dynamically. Details of the formulations are provided in Section 3.4.

Following association matching, the agent utilizes tracks with detections as measurements for the Kalman filter update, which involves calculating the Kalman gain, updating the state estimate, and refining the error covariance. Subsequently, with the hypothesis that pedestrians do not exhibit abrupt large displacements between consecutive frames, the agent deploys the updated HMM-based Kalman filter (detailed in Section 3.3) to predict object locations in the subsequent frame, thus providing priors for the next iteration of association matching. For missed tracks and missed detections from the initial association, the agent employs IoU matching to generate additional valid track-detection pairs. Remaining unassociated detections are initialized as new tracks, while unmatched tracks are marked for deletion unless the corresponding targets reappear in subsequent frames.

3.3. Kalman Filter Reconstruction Through HMM

With detections acquired by the object detector, establishing continuous trajectories through the matching of new detections with existing tracks constitutes a fundamental challenge. A critical issue arises from the fact that the trajectories generated at time $t-1$ may not accurately represent the exact positions of moving objects at time t, with the complexity of pedestrian motion further exacerbating this difficulty. To address this, the Kalman filter is employed for prediction. The Kalman filter is a method for predicting the future state of a system based on previous ones as a different approach to statistical prediction and filtering [14].

In this section, we present a reformulation of the Kalman filter within the framework of HMMs to better model the dual-phase movement patterns of pedestrians, as previously discussed. As the Kalman filter is rooted in principled probabilistic theory, it can indeed be systematically described within the paradigm of HMMs [15]. This integration aims to enhance the predictive capability for scenarios characterized by abrupt changes in pedestrian dynamics, thereby improving the robustness of trajectory association in complex environments.

The state-space model is described as follows:

• The sequence representing the states of objects is denoted as ${\left({\mathbf{x}}_t\right)}_{t=1,...,n}$, with these states being non-observable and solely influenced by past states through the last observation. Mathematically, ${\mathbf{x}}_t$ is expressed as a Markov process with $\mathbb{P}({\mathbf{x}}_t\mid {\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_{t-1})=\mathbb{P}({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1})$;
• For each state, there exists a corresponding observable variable ${\mathbf{z}}_t$, which is a manifestation of the underlying latent state ${\mathbf{x}}_t$.

To ensure computational tractability of the proposed framework, several constraints are integrated.

• The space and the state exhibit linear dependence (this is shared by both the Kalman filter and HMMs):

  $${\mathbf{z}}_t={\mathbf{H}}_t{\mathbf{x}}_t+{\mathbf{v}}_t$$

  (6)

  $${\mathbf{v}}_t\sim \mathcal{N}\left(0,{\mathbf{R}}_t\right)$$

  (7)

  where ${\mathbf{v}}_t$ represents the zero-mean Gaussian measurement noise, and $\mathbf{H}$ denotes the observation matrix that projects the state space onto the measurement space.
• The states at time $t-1$ and t exhibit linear dependence (this is shared by both the Kalman filter and HMMs):

  $${\mathbf{x}}_t={\mathbf{F}}_t{\mathbf{x}}_{t-1}+{\mathbf{B}}_t{\mathbf{u}}_t+{\mathbf{w}}_t$$

  (8)

  $${\mathbf{w}}_t\sim \mathcal{N}\left(0,{\mathbf{Q}}_t\right)$$

  (9)

  where ${\mathbf{w}}_t$ represents the zero-mean Gaussian process noise, and $\mathbf{F}$ is the state transition matrix. The term ${\mathbf{B}}_t{\mathbf{u}}_t$ accounts for additional control inputs.
• The initial state ${\mathbf{x}}_0$ and noise vectors at each step are all mutually independent (this is shared by both the Kalman filter and HMMs).
• The state variable ${\mathbf{x}}_t$ adheres to a multi-dimensional normal distribution. This is specifically common to the Kalman filter. Meanwhile, for HMMs, the state variable is presumed to follow a multinomial distribution.

Under these constraints, the original state-space model specializes to a more restricted formulation, the Linear-Gaussian State-Space Model (LG-SSM).

Leveraging Equation (8), the unconditional mean of ${\mathbf{x}}_t$ is derived through a recursive formulation as follows:

$${\displaystyle \prod _{k=2}^t{\mathrm{F}}_k{\mathrm{x}}_1+\sum _{k=2}^t\prod _{l=k+1}^t{\mathrm{F}}_l{\mathrm{B}}_k}$$

(10)

And the unconditional co-variance is computed as follows:

$${\mathbf{P}}_t=\mathbb{E}\left[{\mathbf{x}}_t{\mathbf{x}}_t^{\mathrm{T}}\right]$$

(11)

With the constrains on independence, exclusion of control inputs, and Equation (8), we have

$${\mathbf{P}}_t={\mathbf{F}}_t{\mathbf{P}}_{t-1}{\mathbf{F}}_t^{\mathrm{T}}+{\mathbf{Q}}_t$$

(12)

It establishes an equation for the unconditional covariance and is formally recognized as the Lyapunov equation.

The next task is to estimate the state ${\mathbf{x}}_t$ utilizing a subset of output sequences ${\mathbf{z}}_0,\dots ,{\mathbf{z}}_t$. In standard Kalman filter notation, the governing equations for this estimation process are formulated as follows:

$${\widehat{\mathbf{x}}}_{t|t}\triangleq \mathbb{E}[{\mathbf{x}}_t\mid {\mathbf{z}}_0,\dots ,{\mathbf{z}}_t]$$

(13)

$$\begin{array}{ccc}\hfill {\mathbf{P}}_{t|t}& \triangleq & \mathbb{E}[({\mathbf{x}}_t-{\widehat{\mathbf{x}}}_{t|t}){({\mathbf{x}}_t-{\widehat{\mathbf{x}}}_{t|t})}^{\mathrm{T}}\mid {\mathbf{z}}_0,\dots ,{\mathbf{z}}_t]\hfill \end{array}$$

(14)

The time propagation process is structured into two sequential stages to preserve the classical formulations of Kalman filters:

• Time update:

  $$\mathbb{P}[{\mathbf{x}}_t\mid {\mathbf{z}}_0,\dots ,{\mathbf{z}}_t]\to \mathbb{P}[{\mathbf{x}}_{t+1}\mid {\mathbf{z}}_0,\dots ,{\mathbf{z}}_t]$$

  (15)
• Measurement update:

  $$\begin{array}{c}\hfill \mathbb{P}[{\mathbf{x}}_{t+1}\mid {\mathbf{z}}_0,\dots ,{\mathbf{z}}_t]\to \mathbb{P}[{\mathbf{x}}_{t+1}\mid {\mathbf{z}}_0,\dots ,{\mathbf{z}}_{t+1}]\end{array}$$

  (16)

Conditioned on the output sequence ${\mathbf{z}}_0,\dots ,{\mathbf{z}}_t$, the variables ${\mathbf{x}}_{t+1}$ and ${\mathbf{z}}_{t+1}$ follow a joint Gaussian distribution with the mean and covariance matrix respectively denoted by

$$\left.\left[\begin{array}{c}{\widehat{\mathbf{x}}}_{t+1|t}\\ {\mathbf{H}}_{t+1}{\widehat{\mathbf{x}}}_{t+1|t}\end{array}\right.\right]$$

(17)

$$\left[\begin{array}{cc}{\mathbf{P}}_{t+1|t}& {\mathbf{P}}_{t+1|t}{\mathbf{H}}_{t+1}^{\mathrm{T}}\\ {\mathbf{H}}_{t+1}{\mathbf{P}}_{t+1|t}& {\mathbf{H}}_{t+1}{\mathbf{P}}_{t+1|t}{\mathbf{H}}_{t+1}^{\mathrm{T}}+{\mathbf{R}}_{t+1}\end{array}\right]$$

(18)

Further, with ${\mathbf{z}}_0,\dots ,{\mathbf{z}}_t+1$, the Kalman filter can be formally represented through the following recursive equations:

$${\widehat{\mathbf{x}}}_{t|t-1}={\mathbf{F}}_t{\widehat{\mathbf{x}}}_{t-1|t-1}$$

(19)

$${\mathbf{P}}_{t|t-1}={\mathbf{F}}_t{\mathbf{P}}_{t-1|t-1}{\mathbf{F}}_t^{\mathrm{T}}+{\mathbf{Q}}_t$$

(20)

$$\begin{array}{cc}\hfill {\widehat{\mathbf{x}}}_{t+1|t+1}=& {\widehat{\mathbf{x}}}_{t+1|t}+{\mathbf{P}}_{t+1|t}{\mathbf{H}}_{t+1}^{\mathrm{T}}{\left({\mathbf{H}}_{t+1}{\mathbf{P}}_{t+1|t}{\mathbf{H}}_{t+1}^{\mathrm{T}}+{\mathbf{R}}_{t+1}\right)}^{-1}({\mathbf{z}}_{t+1}-{\mathbf{H}}_{t+1}{\widehat{\mathbf{x}}}_{t+1|t})\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill {\mathbf{P}}_{t+1|t+1}=& {\mathbf{P}}_{t+1|t}-{\mathbf{P}}_{t+1|t}{\mathbf{H}}_{t+1}^{\mathrm{T}}{\left({\mathbf{H}}_{t+1}{\mathbf{P}}_{t+1|t}{\mathbf{H}}_{t+1}^{\mathrm{T}}+{\mathbf{R}}_{t+1}\right)}^{-1}{\mathbf{H}}_{t+1}{\mathbf{P}}_{t+1|t}\hfill \end{array}$$

(22)

To infer the final posterior distribution $\mathbb{P}({\mathbf{x}}_t\mid {\mathbf{z}}_1,\dots ,{\mathbf{z}}_T)$, the following formulation is employed:

$${\widehat{\mathbf{x}}}_{t|T}={\mathbf{P}}_{t|T}({\mathbf{P}}_{t|t}^{-1}{\widehat{\mathbf{x}}}_{t|t}+{\mathbf{P}}_{t|t+1}^{-1}{\widehat{\mathbf{x}}}_{t|t+1})$$

(23)

$${\mathbf{P}}_{t|T}={\left({\mathbf{P}}_{t|t}^{-1}+{\mathbf{P}}_{t|t+1}^{-1}-{\Sigma }_t^{-1}\right)}^{-1}$$

(24)

Thus, we formalize the Kalman filter as a special case of the HMM framework.

3.4. Multi-Cue Fusion by Weighted Entropy

The utilization of multiple features enables the acquisition of complementary information for object representation. While various cues may be employed, appearance and motion information remain the most intuitive and critical for cross-frame data association. However, these features are highly susceptible to changes in target characteristics, which diminish their discriminative capacity. To address this limitation, we employ these two widely used features governed by weighted entropy, a metric designed to quantify the uncertainty in their characterization capabilities, to construct a balanced cost matrix for association matching. This framework evaluates the similarity between existing tracks and new detections by integrating feature-specific uncertainties, thereby enhancing the robustness of the data association process against target variations.

We begin by formulating the likelihood of state $Y_t$ of the object conditioned on the observation $O_t$ given by the detections in the t-th frame as follows:

$$p\left(O_t\right|Y_t)=exp\left(-c{∥(F_t-{\overline{F}}_t)-U{U}^T(F_t-{\overline{F}}_t)∥}_2^2\right)$$

(25)

where c denotes a normalization constant, U is the projection matrix of the PCA subspace [16], $F_t$ represents the object feature, and ${\overline{F}}_t$ represents the empirical mean of historical object features across previous frames. The combined assessment of the i-th object in the t-th frame is defined as follows:

$$Y_i^{fused}=\alpha {\tilde{Y}}_i^{app}+(1-\alpha ){\tilde{Y}}_i^{mot}$$

(26)

where ${\tilde{Y}}_i^{app}$ and ${\tilde{Y}}_i^{mot}$ denote the normalized evaluations of the state, and $0\le \alpha \le 1$ is the coefficient applied to ${\tilde{Y}}_i^{app}$.

To reduce the perturbations of the discriminative capability of features, we formally introduce weighted entropy [5] as a metric to quantify the uncertainty of the features:

$$H\left(\alpha \right)=-\sum _i{Y}_i^{fused}lg{Y}_i^{fused}$$

(27)

Smaller $H\left(\alpha \right)$ indicates greater discriminative capacity of the state evaluation metric. Consequently, we formulate the fusion coefficient optimization as a minimization problem of $H\left(\alpha \right)$. And the optimized $\alpha$ enables effective multi-cue fusion and subsequent construction of the assignment cost matrix.

4. Experiment

4.1. Setting

4.1.1. Datasets

Experimental evaluations were conducted on the MOT17 [17] and MOT20 [18] benchmark datasets under the “private detection” protocol. MOT17, a well-established MOT benchmark, comprises 7 training sequences with 5316 frames and 7 testing sequences with 5919 frames, and is widely used. MOT20, specifically designed for densely crowded environments, includes 4 training sequences with 8931 frames and 4 testing sequences with 4479 frames, providing a rigorous challenge for tracking in highly occluded and cluttered scenes.

4.1.2. Metrics

Performance evaluations were conducted using existing metrics, including MOTA, IDs, IDF1, HOTA, and FPS [19,20,21]. MOTA primarily assesses detection performance by integrating FP, FN, and IDs, providing a holistic measure of tracking reliability. IDF1 focuses on the consistency of identity matching between consecutive frames. HOTA fuses DetA and AssA into a single metric, distinguishing itself by evaluating performance across a spectrum of detection similarity thresholds rather than a fixed criterion. This design inherently incorporates localization precision [1]. Notably, direct comparisons of FPS for computational efficiency are often problematic due to inherent variability in hardware configurations and software optimizations across experimental setups. Additionally, for trackers following the TBD paradigm, the computational overhead of object detection is typically excluded.

4.1.3. Implementation Details

All experiments were conducted on a server equipped with a single NVIDIA V100 GPU. And the experimental results were validated through five independent trials. For object detection, we adopted YOLOX-X [22] with a COCO-pretrained model [23] as the initialized weights for its optimized balance between detection speed and accuracy. The training schedule was similar to the configuration of ByteTrack [24], with 80 epochs. A threshold of 0.6 was set for the confidence of detection to filter low-confidence detections while preserving recall performance. The parameters optimized by CMA-ES (covariance matrix adaptation evolution strategy, an evolution strategy optimization method that can accommodate non-convex optimization problems) of the HMM-based Kalman filter are listed below:

$$\mathbf{H}=\left[\begin{array}{c}1\\ 0\end{array}\right],\mathbf{Q}=\left[\begin{array}{ccc}24.8^2& 24.8& 0\\ 24.8& 0& 11.8^2\end{array}\right],\mathbf{R}=\left[\begin{array}{c}46.2\end{array}\right],{\mathbf{P}}_{t=0}=\left[\begin{array}{cc}77.5& 0\\ 0& 67\end{array}\right]$$

(28)

4.2. Ablation Study

We conducted an ablation study experiment on the MOT17 training dataset, as most previous works do, using the StrongSORT framework [1] as the baseline. First, the original Kalman filter was replaced with our HMM-based Kalman filter (denoted as $\mathcal{R}$). As reported in

Table 1. Ablation study on the MOT17 training set.

Sequence	Method	IDF1(↑)	MOTA(↑)	HOTA(↑)	IDs(↓)	FPS(↑)
MOT17-02	Baseline	60.5	79.4	55.0	124	6.2
	+$\mathcal{R}$	65.9	79.4	58.8	96	5.2
	+$\mathcal{R}$+$\mathcal{W}$	68.9	79.4	59.8	85	3.0
	+$\mathcal{R}$+$\mathcal{W}$+$\mathcal{M}$	68.9	79.4	59.8	85	12.6
MOT17-04	Baseline	96.6	98.3	90.0	140	4.7
	+$\mathcal{R}$	97.4	98.4	92.9	99	3.3
	+$\mathcal{R}$+$\mathcal{W}$	97.7	98.4	93.2	90	1.1
	+$\mathcal{R}$+$\mathcal{W}$+$\mathcal{M}$	97.7	98.4	93.2	90	9.3
MOT17-05	Baseline	71.8	80.2	61.8	126	30.4
	+$\mathcal{R}$	73.6	80.2	63.3	105	24.8
	+$\mathcal{R}$ + $\mathcal{W}$	74.8	80.2	63.8	91	17.4
	+$\mathcal{R}$ + $\mathcal{W}$+$\mathcal{M}$	74.8	80.2	63.8	91	54.6
MOT17-09	Baseline	70.3	87.1	65.0	50	20.8
	+$\mathcal{R}$	72.3	87.1	66.1	41	15.5
	+$\mathcal{R}$ + $\mathcal{W}$	73.3	87.1	66.8	25	10.0
	+$\mathcal{R}$ + $\mathcal{W}$+$\mathcal{M}$	73.3	87.1	66.8	25	40.1
MOT17-10	Baseline	77.9	82.7	63.2	112	12.9
	+$\mathcal{R}$	80.5	82.8	65.2	82	10.8
	+$\mathcal{R}$+$\mathcal{W}$	82.5	82.8	65.6	71	6.9
	+$\mathcal{R}$+$\mathcal{W}$+$\mathcal{M}$	82.5	82.8	65.6	71	25.0
MOT17-11	Baseline	87.5	92.6	78.2	127	22.4
	+$\mathcal{R}$	89.8	93.0	80.0	97	19.5
	+$\mathcal{R}$+$\mathcal{W}$	89.9	93.0	80.2	83	14.7
	+$\mathcal{R}$+$\mathcal{W}$+$\mathcal{M}$	89.9	93.0	80.2	83	40.3
MOT17-13	Baseline	81.3	90.9	65.7	117	32.7
	+$\mathcal{R}$	84.0	91.0	69.6	93	27.6
	+$\mathcal{R}$+$\mathcal{W}$	84.3	91.0	71.0	81	22.8
	+$\mathcal{R}$+$\mathcal{W}$+$\mathcal{M}$	84.3	91.0	71.0	81	62.6

Numbers in bold denote the best scores in the corresponding metric.

, this modification reduced the average ID count by 30, demonstrating that the HMM effectively enhances target location estimation and mitigates target missing errors. At the same time, the IDF1 and HOTA metrics exhibited significant improvements due to more reliable association. Subsequently, the default weight parameter (0.98) for multi-cue fusion was replaced with a weighted entropy mechanism (denoted as $\mathcal{W}$), which adaptively balances appearance and motion feature contributions, as shown in Figure 2. The results show that $\mathcal{W}$ achieved an average IDF1 increase of 1.1 over $\mathcal{R}$, accompanied by a further reduction in IDs. This improvement is attributed to the weighted entropy’s capability to dynamically calibrate the trade-off between appearance and motion features, enhancing HOTA at the same time. Notably, since both HMM and the weighted entropy mechanism were not applied in the detecting stage, the proposed method had a negligible impact on MOTA as expected. Further, we replaced the original framework of track management with the multi-agent approach (denoted by $\mathcal{M}$), which assigns each track to an agent for management. The significant increase in FPS demonstrates that the proposed approach can offset the additional computation loads caused by the application of the HMM-based Kalman filter and weighted entropy mechanism effectively.

4.3. Comparison with State-of-the-Art Methods

We compared the performance of our method with that of state-of-the-art methods on the test sets of MOT17 and MOT20 respectively. The results are shown in

Table 2. Performance comparison with state-of-the-art MOT methods on MOT17 test set. Numbers in bold denote the best scores in the corresponding metric.

Method	IDF1(↑)	MOTA(↑)	HOTA(↑)	IDs(↓)	FPS(↑)
STDFormer [25]	73.1	78.4	60.9	5091	24.0
RelationTrack [26]	74.7	73.8	61.0	1374	8.5
MOTRv2 [27]	75.0	78.6	62.0	2619	7.5
TransMOT [28]	75.1	76.7	61.7	2346	9.6
ByteTrack [24]	77.3	80.3	63.1	2196	29.6
P3AFormer [29]	78.1	81.2	-	1893	-
BPMTrack [30]	78.1	81.3	63.6	2010	16.1
StrongSORT [1]	78.5	78.3	63.5	1446	7.5
Hybrid-SORT [31]	78.4	79.3	63.6	-	-
UCMCTrack [32]	79.0	79.0	64.3	-	-
BoT-SORT [33]	79.5	80.6	64.6	1257	6.6
C-BIoU [34]	79.7	81.1	64.1	-	-
MotionTrack [35]	80.1	81.1	65.1	1140	15.7
SparseTrack [36]	80.1	81.0	65.1	1170	19.9
BoT-SORT-ReID [33]	80.2	80.5	65.0	1212	4.5
Ours	82.1	81.5	65.9	1255	16.7

Numbers in bold denote the best scores in the corresponding metric.

and

Table 3. Performance comparison with state-of-the-art MOT methods on MOT20 test set. Numbers in bold denote the best scores in the corresponding metric.

Method	IDF1(↑)	MOTA(↑)	HOTA(↑)	IDs(↓)	FPS(↑)
MOTR [37]	68.6	73.4	57.8	2439	-
RelationTrack [26]	70.5	67.2	56.5	4243	2.7
STDFormer [25]	72.1	76.2	60.2	3166	-
ByteTrack [24]	75.2	77.8	61.3	1223	17.5
StrongSORT [1]	75.9	72.2	61.5	1066	1.5
Hybrid-SORT [31]	76.2	76.4	62.5	-	-
BoT-SORT [33]	76.3	77.7	62.6	1212	6.6
P3AFormer [29]	76.4	78.1	-	1332	-
MotionTrack [35]	76.5	78.0	62.8	1165	15.0
SparseTrack [36]	77.3	78.2	63.4	1116	12.5
UCMCTrack [32]	77.4	75.5	62.8	-	-
BoT-SORT-ReID [33]	77.5	77.8	63.3	1257	2.4
Ours	81.2	78.4	65.7	608	3.9

Numbers in bold denote the best scores in the corresponding metric.

.

MOT17: As presented in Table 2, the proposed method achieves state-of-the-art performance on the MOT17 benchmark, ranking first in the IDF1, MOTA, and HOTA metrics with scores of 82.1, 81.5, and 65.9, respectively. Meanwhile, the method ranks fourth in the IDs metric with a count of 105 more than the first. Our method outperforms the second method by a significant margin (+1.9 in IDF1, +0.8 in HOTA), demonstrating superior cross-frame matching precision. This performance gain is attributed to the adaptive weighting scheme that balances appearance and motion features in constructing the cost matrix, producing a more accurate association.

MOT20: As previously noted, MOT20 serves as a benchmark for highly crowded scenarios, where severe occlusion inherently increases the risk of detection failures and association errors. Unlike many state-of-the-art trackers that require network retraining or fine-tuning on MOT20 or auxiliary datasets to address these challenges, the proposed method achieves superior performance without such adaptation. As shown in Table 3, our method ranks first on MOT20 across key metrics: IDF1 (81.2), MOTA (78.4), HOTA (65.7), and IDs (608). Its performance underscores the robustness of its design, particularly in managing dense occlusion scenarios. The method’s excellence in crowded environments is attributed to its HMM-based Kalman filter, which analyzes the two-phase moving pattern of pedestrians to mitigate occlusion impacts, as illustrated in Figure 3. In particular, the method reduces IDs by 469 compared to the second method, achieving a substantial advantage in IDF1 at the same time.

5. Discussion

Through experimental validation, our method has demonstrated promising results. The reduction in IDs achieved by replacing the conventional Kalman filter with the HMM-based variant underscores the importance of motion context. Traditional models often fail to capture the irregular movements inherent in pedestrian trajectories, particularly during occlusions or abrupt direction changes. By modeling the two-phase motion pattern, the HMM-based Kalman filter provides a probabilistic framework that better accommodates practical situations. This improvement directly translates into improved HOTA scores (65.9 on MOT17 and 65.7 on MOT20), reflecting superior trajectory continuity. The weighted entropy mechanism addresses a fundamental limitation of fixed weighting schemes in multi-cue tracking frameworks. While appearance features dominate in sparse scenes, motion cues become more reliable during occlusions or rapid movements. The proposed dynamic calibration strategy achieves a 1.1 improvement in IDF1 and further reduction in IDs in the ablation study by balancing feature contributions in a self-adaptive way. And the multi-agent mechanism provides efficient computations as the underlying support, leading to satisfying performance on the FPS metric.

However, our work has limitations: The two-phase motion pattern and the model that accompanies it are specialized in tracking pedestrians, resulting in incompatibility for tracking vehicles or other objects. Thus, the suitable datasets for our method are limited. Further, pedestrians from different regions may have vastly different behavioral habits, posing a great challenge to the study of pedestrian movement patterns in this work. And the real-time computation of entropy brings heavy loads to devices, decreasing the FPS metric accordingly.

In addition, we compare the main contributions of previous works with ours in

Table 4. Comparison of main contributions between previous works and ours.

Method	Contributions and Limitations
RTU++ [6]	Constructs a general Extendable-Multiple-Node Tracking framework.
	Proposes RTU++ to score track proposals.
	Establishes a Node-based General Tracking Data Generation method.
	Lacks analysis of moving patterns for pedestrians.
DMARL-Tracker [8]	Proposes a distributed MOT algorithm based on MARL.
	Designs a collaborative blockchain-based MOT framework.
	Fixed weight parameters lack robustness.
SparseTrack [36]	A method to obtain relative depth from 2D images.
	An effective depth cascade matching approach.
	A new IoU-only tracker with depth cascade matching.
	Neglects the probabilistic problems of uncertainties in MOT.
MLT [7]	A detection multiplexing method for mutual occlusion.
	An MLG model using the detection multiplexing method.
	A near-online optimization method of solving MLG.
	Lacks analysis of moving patterns for pedestrians.
StrongSORT [1]	Equips DeepSORT with advanced modules and tricks.
	Proposes two lightweight algorithms, AFLink and GSI.
	Heavy computation loads result in low FPS.
Ours	Decomposes pedestrian locomotion into two phases.
	Reconstructs Kalman filters in a unified HMM framework.
	Proposes a weighted entropy mechanism for adaptive fusion.
	Introduces multi-agents to manage tracks separately.
	Restricted in scenarios of tracking pedestrians.

.

6. Conclusions

In this work, our goal is to tackle the uncertainty problem of two-phase motion patterns when tracking pedestrians. We propose a novel multi-agent framework for MOT, leveraging multi-agent decision-making to enhance the efficiency and scalability of track management in complex scenes. In addition, the framework integrates two key methodological innovations: HMM-based reconstruction of the Kalman filter and a weighted entropy mechanism for the adaptive fusion of appearance and motion features.

Quantitative evaluations on the MOT17 benchmark demonstrate significant advancements over prior methods: Our method achieves 82.1 in IDF1 (+1.9 vs. second-best), 81.5 in MOTA, and 65.9 in HOTA (+0.8 vs. second-best), establishing new state-of-the-art results in all three primary metrics. For the challenging MOT20 benchmark featuring extreme crowd densities, our method achieves 81.2 in IDF1, 78.4 in MOTA, 65.7 in HOTA, and 608 IDs—all ranking first without dataset-specific fine-tuning. And the improvement in the FPS metric demonstrates that the multi-agent approach can enhance the efficiency of computation and track management.

Future work could further explore the decentralization of MOT. On the one hand, centralized computing architectures face inherent limitations in terms of security and robustness, primarily due to their susceptibility to single-point failures—a critical vulnerability that renders them unsuitable for deployment in industrial environments. On the other hand, it brings inevitable risks of data leakage for data owners and the administrator. Meanwhile, emerging blockchain technology has become a promising paradigm to enable trustworthy collaborative frameworks to be established, leveraging its salient attributes of decentralization, transparency, traceability, and immutability. The framework of blockchain-based MOT can consist of edge devices, compute units, and blockchain networks. Edge devices contain surveillance cameras and the communication protocol. Compute units include object detectors, ReID modules, and trackers. This architecture facilitates the seamless integration of multi-agent coordination mechanisms.