Context-Aware Identity Prediction for Anti-UAV Multi-Object Tracking in Remote Sensing Videos

Highlights

What are the main findings?

• A context-aware identity prediction framework is proposed for anti-UAV multi-object tracking in remote sensing videos. By using historical trajectory priors, the proposed framework effectively addresses weak UAV appearance, abrupt motion, occlusion, and intermittent visibility in complex low-altitude surveillance scenarios.
• Two dedicated modules are designed: the adaptive identity disambiguation (AID) module for reliability-aware appearance-motion fusion, and the motion-evolution temporal memory (METM) module for motion-consistent trajectory recovery. Together with the identity decision layer (IDL), these modules improve identity association and trajectory continuity, achieving higher HOTA and IDF1 scores and fewer identity switches and track fragments on the 4th Anti-UAV Benchmark Track-3 and AU-MOT.

What are the implications of the main findings?

• The proposed framework shows that historical trajectory priors are critical for robust anti-UAV multi-object tracking. By using past motion and identity context, the tracker can go beyond frame-wise appearance matching and better handle weak UAV appearance, abrupt maneuvers, occlusion, and intermittent visibility in complex low-altitude surveillance scenarios.
• The AID and METM modules demonstrate that short-term reliability-aware association and long-term motion-aware memory are complementary for maintaining UAV identities. This provides a practical direction for anti-UAV surveillance systems, where multiple small UAVs must be continuously tracked under background clutter, target disappearance, and reappearance.

Abstract

Anti-UAV multi-object tracking in remote sensing videos is challenging because UAV targets are small, weakly textured, and often affected by cluttered backgrounds, abrupt motion, occlusion, and intermittent visibility. To address these challenges, we formulate anti-UAV multi-object tracking as a context-aware identity prediction task, in which target identities and locations are inferred from historical trajectory priors instead of current-frame observations alone. Under this formulation, we propose a dual-track parallel tracking framework. The adaptive identity disambiguation (AID) module combines motion cues with appearance features according to their estimated reliability, improving short-term association when visual evidence is weak. In parallel, the motion-evolution temporal memory (METM) module models trajectory dynamics using motion anomaly detection and time-decayed memory, enabling spatiotemporal recovery after occlusion, temporary disappearance, or abrupt motion. The outputs of the two branches are integrated by a unified identity decision layer to produce stable tracking results. Experiments are conducted on the public 4th Anti-UAV Benchmark Track-3 and our newly constructed Anti-UAV Multi-Object Tracking dataset, AU-MOT. On the 4th Anti-UAV Benchmark Track-3, our method achieves 63.6% HOTA and 64.1% IDF1, outperforming the strongest competing method by 3.5% and 3.9%, respectively, while reducing identity switches and track fragments by 20.8% and 23.8%. On AU-MOT, it achieves 67.2% HOTA and 67.8% IDF1, with 20.2% fewer identity switches and 22.3% fewer track fragments. These results demonstrate its effectiveness under long-range observation, weak target appearance, cluttered backgrounds, abrupt motion, and intermittent target visibility.

1. Introduction

The rapid development of unmanned aerial vehicles (UAVs) has improved operational efficiency in sports event live streaming, urban management [1], emergency response [2], environmental monitoring [3], and many other civilian applications [4]. At the same time, the mobility and flexibility that make UAVs useful also create new risks for low-altitude airspace safety [5,6]. Unauthorized or malicious UAVs may pose a threat to sports training areas, sports event live streaming venues, and regulated airspace operations [7]. UAV monitoring and tracking have therefore become important components of low-altitude surveillance systems [8]. In this setting, robust multi-target tracking (MTT) in remote sensing videos is essential for continuous trajectory estimation, behavior analysis, and downstream decision-making.

Despite recent progress in MTT, anti-UAV multi-object tracking in remote sensing videos remains difficult because UAV targets differ markedly from objects in conventional tracking scenarios [5,9]. They are often observed at long range and occupy only a few pixels, which leads to weak texture, low contrast, and limited appearance information [10]. Against complex backgrounds such as sky, clouds, buildings, or vegetation, appearance-based identity association can become unreliable. UAV motion further increases the difficulty. Unlike pedestrians or vehicles, which usually follow relatively smooth motion patterns, UAVs may accelerate rapidly, turn abruptly, hover, or switch motion states within a short period [11,12]. Such motion often breaks the assumptions behind Kalman filtering and simple linear motion prediction [13]. Background clutter, scale variation, occlusion, and atmospheric disturbance can also cause missed detections, fragmented trajectories, and identity switches. These factors are tightly coupled: weak appearance reduces the discriminative power of visual features, irregular motion weakens trajectory prediction, and intermittent visibility further amplifies association errors. As a result, conventional tracking frameworks often struggle to maintain stable identities and continuous trajectories in complex anti-UAV remote sensing scenarios.

Most multi-target tracking methods adopt the tracking-by-detection paradigm, where detection and association are treated as separate stages. A detector first localizes targets in each frame [14,15], and an association module then links detections over time to form trajectories [16]. This association step is usually based on handcrafted costs that combine appearance similarity, spatial overlap, and motion affinity, followed by a matching algorithm such as the Hungarian algorithm. Although this modular pipeline works well in many general tracking scenarios, its weaknesses become clear in anti-UAV remote sensing videos. The detector is mainly optimized for frame-level localization and does not explicitly learn long-term identity consistency, while the association module is constrained by fixed detection features and predefined matching rules. When UAVs provide weak appearance cues, make abrupt maneuvers, or disappear for short periods, these pipelines have limited ability to recover identity continuity from historical trajectory context [13]. Recent end-to-end MTT methods reduce the reliance on handcrafted association by using query-based transformer architectures, where track queries are propagated across frames to maintain target identities [17]. This design gives a more unified formulation of detection and tracking, but it is still not fully suited to anti-UAV scenarios. Detection and tracking are often optimized within the same decoding process, which can force a compromise between accurate localization and stable identity modeling. Track queries also remain closely tied to visual appearance. When UAVs occupy only a few pixels and provide weak or ambiguous visual evidence, appearance-dominated queries become unreliable for identity discrimination. As a result, both traditional tracking-by-detection methods and recent unified tracking frameworks still struggle to maintain robust identity consistency in complex anti-UAV scenarios. These limitations call for a tracking framework that can make better use of historical trajectory context, reason adaptively about motion evolution, and optimize detection and tracking within a unified model.

Based on this observation, we formulate anti-UAV multi-object tracking as a context-aware identity prediction problem. Rather than determining target identities only from current-frame observations or handcrafted association costs, the proposed framework infers target identities and locations from historical trajectory priors. This formulation allows identity association to be learned from temporal context and links identity prediction with trajectory consistency. The tracker can therefore better adapt to unseen trajectories and preserve identities when visual evidence is weak or unreliable. By jointly modeling localization and trajectory coherence, the proposed formulation provides a more flexible solution for anti-UAV multi-object tracking under weak appearance cues and irregular motion patterns.

Meanwhile, although the high maneuverability of UAVs makes tracking difficult, it also provides useful motion information for identity discrimination. In practical anti-UAV scenarios, UAVs may accelerate rapidly, turn abruptly, hover, or frequently switch between motion states. These dynamic patterns can help distinguish targets, especially when appearance features are degraded by long-range observation, low resolution, or cluttered backgrounds. Effective use of kinematic context is therefore important for maintaining trajectory continuity in anti-UAV multi-object tracking.

To this end, we propose a dual-track parallel tracking framework. The adaptive identity disambiguation (AID) module combines motion cues and appearance features through reliability-aware weighting, improving short-term association when visual features are ambiguous or degraded. In parallel, the motion-evolution temporal memory (METM) module models long-term trajectory dynamics using motion anomaly detection and a time-decayed memory mechanism, which supports trajectory recovery under occlusion, temporary disappearance, and abrupt maneuvers. The outputs of the two branches are further integrated by a unified identity decision layer, which balances short-term association and long-term trajectory recovery to produce stable tracking results.

The main contributions of this work are summarized as follows:

• A context-aware identity prediction framework for anti-UAV multi-object tracking is proposed to infer target identities and locations from historical trajectory priors, reducing the dependence on handcrafted association rules and improving the coordination between detection and identity association.
• The adaptive identity disambiguation (AID) module is designed to achieve dynamic fusion of motion and appearance cues via scene-adaptive reliability modeling, improving identity association when visual features are weak or ambiguous.
• The motion-evolution temporal memory (METM) module is designed to model long-term trajectory dynamics with kinematic anomaly detection and time-decayed memory, enabling motion-consistent trajectory recovery under occlusion, temporary disappearance, and abrupt maneuvers.

The remainder of this paper is organized as follows. Section 2 reviews related work in multi-object tracking and anti-UAV systems. Section 3 details the proposed context-aware identity prediction framework and the dual-track parallel mechanism. Section 4 presents experimental results and ablation studies. Section 5 discusses the limitations of the proposed method. Finally, Section 6 concludes this paper and outlines future research directions.

2. Related Work

2.1. Multi-Object Tracking

Object detection and tracking are important fundamental tasks in remote sensing image processing [18,19]. Multi-object tracking (MOT) has long been studied under the tracking-by-detection (TBD) paradigm, where tracking is decomposed into frame-wise object detection and inter-frame data association. SORT [16], for example, associates detections across frames using Kalman filtering and the Hungarian algorithm. This pipeline is simple and efficient, but it assumes relatively smooth motion and continuous target visibility [20,21]. Later methods, such as DeepSORT [22] and ByteTrack [23], improve association by introducing appearance-based re-identification features or retaining low-confidence detections. Despite these advances, detection and association are still optimized separately. The detector is mainly trained for frame-level localization, whereas the association module depends on predefined costs based on appearance similarity, spatial overlap, and motion affinity. This separation limits end-to-end optimization and becomes unreliable when target appearance is degraded or motion no longer follows simple assumptions.

The rise of deep networks provides new ideas for typical object detection and tracking [24,25]. Transformer-based trackers have recently reduced the reliance on handcrafted association by casting MOT as an end-to-end trainable task. TrackFormer [26] and MOTR [27] extend DETR-style object queries to the temporal domain by propagating track queries across frames to preserve target identities. These methods offer a more integrated formulation of detection and tracking. However, unified query decoding can still lead to competition between localization and identity modeling, especially when the two objectives depend on different cues. Recent methods have attempted to address this issue from different perspectives. MeMOTR [28] introduces long-term memory to improve temporal association, while CO-MOT [29] improves end-to-end tracking through coopetition label assignment and shadow query sets. Even so, many of these methods still depend heavily on visual appearance, which is often unreliable in anti-UAV scenarios where targets are small, low-resolution, and weakly textured [30]. Recent methods such as MOTIP [31] reformulate tracking as in-context identity prediction and show promising generalization ability, but fixed temporal windows and appearance-dominated identity embeddings may remain insufficient for long-range UAV tracking with abrupt motion and intermittent visibility.

In this work, we formulate anti-UAV multi-object tracking (anti-UAV MOT) as a context-aware identity prediction task. Instead of relying on separate detection-association stages or appearance-driven query propagation, the proposed framework infers target identities and locations from historical trajectory priors. This formulation supports unified optimization of detection and tracking, while allowing motion-aware association and memory-based trajectory recovery to be incorporated for challenging anti-UAV tracking scenarios.

2.2. Anti-UAV Tracking in Remote Sensing Videos

Anti-UAV tracking has become increasingly important as low-altitude airspace surveillance and security monitoring receive more attention [10]. Compared with objects in conventional tracking scenarios, UAVs in remote sensing videos are usually observed at long range and appear as small, low-contrast targets. Their visual features are often weak and can be easily affected by background clutter, atmospheric disturbance, scale changes, sensor noise, and motion blur [5,32]. UAV motion also makes tracking difficult, since targets may accelerate rapidly, turn abruptly, hover, or switch motion states within a short period. These behaviors are hard to predict with simple linear motion models.

Most existing anti-UAV tracking studies focus on single-object tracking or joint detection-and-tracking tasks [10]. In these settings, the tracker usually localizes a specified UAV and determines whether it is visible or absent in each frame. This formulation has advanced long-term UAV tracking under small target scale, out-of-view motion, background clutter, and temporary disappearance. However, it does not fully address identity association in multi-UAV scenarios. When multiple UAVs appear at the same time, the tracker must localize each target while keeping its identity consistent across frames. This introduces additional difficulties, including inter-target confusion, identity switches, fragmented trajectories, and re-identification after reappearance.

Recent anti-UAV benchmarks have started to move from single-UAV tracking toward multi-UAV tracking [33]. In these scenarios, several UAVs may appear in the same frame and move against complex backgrounds such as clouds, buildings, vegetation, and terrain. Appearance similarity alone is often insufficient, because small UAVs provide limited discriminative texture. Short-term motion prediction is also unreliable when targets maneuver abruptly or become intermittently invisible. Anti-UAV multi-object tracking therefore requires a framework that can jointly use weak appearance cues, irregular motion patterns, and historical trajectory context.

Motivated by these challenges, we develop a dual-track parallel mechanism for anti-UAV MOT. The adaptive identity disambiguation module performs short-term association through reliability-aware fusion of appearance and motion cues, while the motion-evolution temporal memory module supports long-term trajectory recovery through motion-consistent memory retrieval. By combining short-term identity discrimination with long-term motion-aware recovery, the proposed design helps narrow the gap between existing single-UAV tracking methods and the needs of practical multi-UAV tracking in complex remote sensing videos.

2.3. Occlusion Handling and Trajectory Recovery

Maintaining identity consistency under occlusion and intermittent visibility loss is a central challenge in multi-object tracking. Traditional trackers often propagate target states with motion models such as Kalman filtering during short occlusions [13,16]. This strategy works well when target motion is smooth and the occlusion lasts only a few frames. In anti-UAV tracking, however, UAVs may accelerate abruptly, make sharp turns, hover, or switch motion states within a short period [5,10]. Such motion can quickly accumulate prediction errors and cause the estimated trajectory to drift away from the true target location. When the target reappears, the tracker may then fail to recover its original identity.

Appearance-based re-identification is widely used to recover occluded targets in general MOT methods [22,34]. By comparing visual embeddings before and after occlusion, these methods can re-associate targets whose appearance remains sufficiently distinctive [35]. This assumption is often weak in long-range anti-UAV surveillance. UAV targets may occupy only a few pixels and provide limited texture information, making their appearance features sensitive to resolution degradation, motion blur, background interference, and illumination changes [5,10]. As a result, appearance-based recovery may cause identity switches or incorrectly reactivate disappeared trajectories.

Memory-based trackers have recently been used to store historical target representations and retrieve them when targets reappear. Methods such as MeMOT [36] and MeMOTR [28] use memory banks or memory attention to capture longer-term dependencies. GTR [37] also models object features across a temporal window to produce global trajectories. These designs improve trajectory continuity to some extent, but many existing memory strategies still treat historical features as static records and do not explicitly account for the reliability decay of dormant trajectories. Motion feasibility is also often underused during trajectory recovery, which can lead to incorrect matching when targets maneuver abruptly or when background clutter produces confusing detections.

To address these limitations, we introduce the motion-evolution temporal memory (METM) module for long-term trajectory maintenance and recovery. Rather than storing dormant trajectories as static records, METM updates them through a time-decayed memory mechanism and retrieves disappeared targets under spatiotemporal consistency and kinematic feasibility constraints. It also uses motion anomaly detection to distinguish normal trajectory evolution, temporary disappearance, and abrupt maneuvering. This design improves identity recovery after occlusion or visibility loss, which is particularly important for anti-UAV tracking in complex remote sensing backgrounds.

3. Materials and Methods

3.1. Notation Summary

For clarity, Table 1 summarizes the main symbols used in the proposed framework.

Table 1. Summary of main notation used in the proposed framework.

Symbol	Description
t	Current frame index
T	Length of the historical temporal window
N	Number of detections in the current frame
M	Number of trajectories considered for association
$M_{\mathrm{a}}$	Number of active trajectories
$M_{\mathrm{d}}$	Number of dormant trajectories in the memory bank
S	Number of regular identity tokens in the identity dictionary
$\mathcal{I}$	Learnable identity dictionary
${\mathcal{T}}_{\mathrm{a}}$	Set of active trajectories
${\mathcal{T}}_{\mathrm{d}}$	Set of dormant trajectories
${\mathcal{M}}_{\mathrm{d}}$	Dormant trajectory memory bank
${\mathit{d}}_t^n$	Detection embedding of the n-th detection at frame t
${\mathit{e}}_{\tau }^m$	Identity-conditioned embedding of trajectory m at frame $\tau$
${\mathbf{S}}_{\mathrm{AID}}$	Short-term association score matrix from AID
${\mathbf{S}}_{\mathrm{METM}}$	Long-term recovery score matrix from METM
${\theta }_{\mathrm{match}}$	Matching threshold for active trajectories
${\theta }_{\mathrm{wake}}$	Reactivation threshold for dormant trajectories
$T_{max}$	Maximum dormant duration

3.2. Overall Framework

The proposed anti-UAV tracking framework casts the conventional detection-and-association pipeline as a context-aware identity prediction process. As shown in Figure 1, the framework predicts multi-target states, including target locations and identities, from historical trajectory priors. Detection and tracking are therefore optimized within a unified end-to-end framework rather than treated as separate stages.

Figure 1. Overview of the proposed framework. Given an input video sequence, the DETR detector extracts object-level features and bounding boxes for each frame. The Context-Aware Identity Encoder constructs identity-conditioned tracklet representations by combining visual features with learnable ID tokens from the identity dictionary. These representations are then processed by two parallel branches: the adaptive identity disambiguation (AID) module for reliability-aware short-term association and the motion-evolution temporal memory (METM) module for long-term trajectory recovery. The outputs of the two branches are integrated by the identity decision layer (IDL) to generate final tracking results with consistent identity labels.

The architecture uses a dual-track parallel design to model short-term identity association and long-term trajectory continuity at the same time. Given an input video sequence, the backbone first extracts object-level visual features, which are then passed to two core modules: the adaptive identity disambiguation (AID) module and the motion-evolution temporal memory (METM) module. AID focuses on short-term association under ambiguous observations. It combines kinematic cues with appearance embeddings and adjusts their contributions according to the estimated reliability of each cue. This design reduces the dependence on degraded appearance features when UAV targets are small, blurred, or observed at long range.

In parallel, METM maintains long-term temporal consistency by modeling trajectory evolution over time. It detects abnormal motion patterns caused by abrupt maneuvers, occlusion, or temporary disappearance and stores historical trajectories in a memory bank with time-decayed confidence. When a target becomes temporarily invisible, METM retrieves candidate trajectories from the memory bank based on spatiotemporal consistency and kinematic feasibility. This mechanism helps recover disappeared targets and reduces trajectory fragmentation after they reappear.

The outputs of AID and METM are integrated by a unified identity decision layer. This layer resolves conflicts between short-term association and memory-based recovery by considering matching confidence, temporal consistency, and motion feasibility. With this design, the proposed framework moves beyond passive frame-to-frame matching and predicts identities with support from historical trajectory context. It can therefore maintain more stable identities and more continuous trajectories in challenging anti-UAV scenarios, where appearance cues are weak, motion is abrupt, and targets may disappear intermittently.

3.3. Context-Aware Identity Encoder

In the proposed framework, identity prediction is treated as a context-dependent inference problem rather than a fixed semantic classification task. A target identity is inferred from its trajectory history, including previous appearance, motion state, and temporal continuity. To support this formulation, we introduce a Context-Aware Identity Encoder that maps raw trajectory observations to discriminative identity-conditioned embeddings. These embeddings define a shared representation space for identity association, trajectory prediction, and memory retrieval.

Formally, we construct an identity dictionary $\mathcal{I}$ with $S+1$ learnable C-dimensional embeddings:

$$\mathcal{I}=\{{\mathbf{i}}^1,{\mathbf{i}}^2,\dots ,{\mathbf{i}}^S,{\mathbf{i}}^{\mathrm{new}}\}$$

(1)

where each identity token ${\mathbf{i}}^s\in {\mathbb{R}}^C$ is a learnable embedding. The first S tokens are used as temporary identity representations for existing trajectories, while ${\mathbf{i}}^{\mathrm{new}}$ is a special token for newly appearing or unassigned objects. Unlike sparse one-hot labels, these learnable tokens provide compact continuous identity conditioning and allow the model to reason about identity association in a shared embedding space.

Let ${\mathcal{T}}_{t-1}=\{T_{t-1}^1,\dots ,T_{t-1}^M\}$ denote the set of active trajectories at frame $t-1$, where M is the number of trajectories. For each trajectory $T^m$, we assign a label index $k_m\in \{1,\dots ,S\}$, which is randomly permuted at each training iteration to prevent the model from memorizing fixed identity mappings.

For each trajectory, we collect its historical visual features over the past T frames:

$${\mathcal{F}}_{t-T:t-1}^m=\{f_{t-T}^m,\dots ,f_{t-1}^m\},f_t^m\in {\mathbb{R}}^C$$

(2)

where $f_t^m$ denotes the appearance embedding extracted from the detector.

We then construct identity-conditioned tracklet representations by concatenating visual features with identity tokens:

$${\mathit{e}}_{\tau }^m=W_e\left[f_{\tau }^m;{\mathbf{i}}^{k_m}\right]+b_e,\tau =t-T,\dots ,t-1$$

(3)

where $[·;·]$ denotes feature concatenation, and $W_e\in {\mathbb{R}}^{C\times 2C}$ projects the concatenated representation back to the hidden dimension C. Thus, ${\mathit{e}}_{\tau }^m\in {\mathbb{R}}^C$ encodes both appearance information and identity context.

For the current frame t, let ${\left\{f_t^n\right\}}_{n=1}^N$ denote the appearance features of N detected objects. Since their identities are unknown before association, each detection is combined with the special new-object token ${\mathbf{i}}^{\mathrm{new}}$:

$${\mathit{d}}_t^n=W_d\left[f_t^n;{\mathbf{i}}^{\mathrm{new}}\right]+b_d,n=1,\dots ,N$$

(4)

where $W_d\in {\mathbb{R}}^{C\times 2C}$ is a projection matrix, and ${\mathit{d}}_t^n\in {\mathbb{R}}^C$ represents the candidate detection embedding before identity assignment.

The encoded trajectory features $\left\{{\mathit{e}}_{\tau }^m\right\}$, and current detection embeddings $\left\{{\mathit{d}}_t^n\right\}$ are then fed into the adaptive identity disambiguation (AID) module and the motion-evolution temporal memory (METM) module. By operating in a shared identity-conditioned embedding space, the two modules can jointly reason about short-term identity association and long-term trajectory evolution.

This design allows identities to be dynamically instantiated from trajectory context rather than tied to fixed semantic categories. As a result, the model can generalize to unseen targets during inference while maintaining consistent identity representations for active and reappearing trajectories.

3.4. Adaptive Identity Disambiguation Module

In anti-UAV tracking, appearance cues are often unreliable because targets are observed from long distances, blurred by motion, affected by atmospheric disturbance, or represented by only a few pixels. In these cases, visual features provide limited information for identity association. Motion cues become important when UAVs move quickly or change direction abruptly, but they can also be uncertain when targets hover, move irregularly, or accelerate suddenly. The reliability of appearance and motion therefore changes over time, which makes fixed-weight fusion unsuitable for robust identity association.

We address this issue with the adaptive identity disambiguation (AID) module, shown in Figure 2. AID uses two branches to model appearance similarity and motion consistency while estimating their reliability at the current time step. The appearance branch aggregates temporal visual features to compare identities across frames, and the motion branch measures kinematic consistency from the predicted displacement. Their outputs are fused according to the estimated reliability scores. This allows the tracker to rely more on motion when appearance features are degraded, and to return to appearance-based discrimination when motion prediction becomes uncertain.

Figure 2. Framework of the proposed adaptive identity disambiguation (AID) module.

Given an active trajectory $T^m$ within the temporal window $[t-T,t-1]$, we use its identity-conditioned embeddings ${\left\{{\mathit{e}}_{\tau }^m\right\}}_{\tau =t-T}^{t-1}$ as input to the appearance pathway. These embeddings are projected into a query-key-value space:

$$q_{\tau }^m=W_Q^e{\mathit{e}}_{\tau }^m+b_Q^e,k_{\tau }^m=W_K^e{\mathit{e}}_{\tau }^m+b_K^e,v_{\tau }^m=W_V^e{\mathit{e}}_{\tau }^m+b_V^e$$

(5)

where $W_Q^e,W_K^e,W_V^e\in {\mathbb{R}}^{C\times C}$ are learnable projection matrices.

To preserve temporal order, we add temporal positional encoding ${\rho }_{\tau }\in {\mathbb{R}}^C$ to the query and key:

$${\widehat{q}}_{\tau }^m=q_{\tau }^m+{\rho }_{\tau },{\widehat{k}}_{\tau }^m=k_{\tau }^m+{\rho }_{\tau }$$

(6)

The temporal attention weights are computed as follows:

$${\alpha }_{\tau ,{\tau }^{\prime }}^m={\displaystyle \frac{exp\left({\left({\widehat{q}}_{\tau }^m\right)}^{\top }{\widehat{k}}_{{\tau }^{\prime }}^m/\sqrt{C}\right)}{{\sum }_{s=t-T}^{t-1}exp\left({\left({\widehat{q}}_{\tau }^m\right)}^{\top }{\widehat{k}}_s^m/\sqrt{C}\right)}}$$

(7)

The temporally aggregated representation is then obtained by the following equation:

$$h_{\tau }^m={\displaystyle \sum _{{\tau }^{\prime }=t-T}^{t-1}}{\alpha }_{\tau ,{\tau }^{\prime }}^m{v}_{{\tau }^{\prime }}^m$$

(8)

We use ${\overline{h}}^m=h_{t-1}^m$ as the trajectory-level appearance representation. For each current detection embedding ${\mathit{d}}_t^n$, its query vector is computed as follows:

$$q_t^n=W_Q^d{\mathit{d}}_t^n+b_Q^d$$

(9)

The appearance affinity between detection n and trajectory m is then defined as follows:

$$s_{app}^{n,m}={\displaystyle \frac{exp\left({\left(q_t^n\right)}^{\top }{\overline{h}}^m/\sqrt{C}\right)}{{\sum }_{m^{\prime }=1}^{M}exp\left({\left(q_t^n\right)}^{\top }{\overline{h}}^{m^{\prime }}/\sqrt{C}\right)}}$$

(10)

Simultaneously, the motion pathway evaluates kinematic consistency via first-order temporal modeling. For each active trajectory, let $p_{t-1}^m$ denote the center position of trajectory m at frame $t-1$. The instantaneous velocity is estimated as follows:

$${\mathbf{v}}_{t-1}^m={\displaystyle \frac{p_{t-1}^m-p_{t-2}^m}{\Delta t}}$$

(11)

The predicted position at frame t is as follows:

$${\widehat{p}}_t^m=p_{t-1}^m+{\mathbf{v}}_{t-1}^m\Delta t$$

(12)

For a detection with center position $p_t^n$, the scale-normalized spatial deviation is as follows:

$$d_{mot}^{n,m}={\displaystyle \frac{\parallel p_t^n-{\widehat{p}}_t^m{\parallel }_2}{\sqrt{w_{t-1}^m{h}_{t-1}^m}+ϵ}}$$

(13)

where $w_{t-1}^m$ and $h_{t-1}^m$ denote the width and height of the trajectory bounding box, and $ϵ$ is a small constant for numerical stability.

The motion affinity is normalized over all active trajectories:

$$s_{mot}^{n,m}={\displaystyle \frac{exp\left(-\frac{{\left(d_{mot}^{n,m}\right)}^2}{2{\sigma }^2}\right)}{{\sum }_{m^{\prime }=1}^{M}exp\left(-\frac{{\left(d_{mot}^{n,m^{\prime }}\right)}^2}{2{\sigma }^2}\right)}}$$

(14)

To dynamically balance appearance and motion cues, we construct a global context descriptor:

$${\mathbf{c}}_t=\left[{\displaystyle \frac{1}{N}}{\displaystyle \sum _{n=1}^N}{\mathit{d}}_t^n;{\overline{v}}_t;{\overline{s}}_t;{\sigma }_{\mathrm{conf}}\right]$$

(15)

where ${\overline{v}}_t$ is the mean velocity magnitude, ${\overline{s}}_t$ is the average object scale, and ${\sigma }_{\mathrm{conf}}$ denotes the variance of detection confidence scores. A lightweight MLP is used as the reliability gate:

$$[w_{app},w_{mot}]=\mathrm{Softmax}\left(\mathrm{MLP}\left({\mathbf{c}}_t\right)\right)$$

(16)

The final AID similarity matrix is computed by reliability-aware fusion:

$${\mathbf{S}}_{AID}=w_{app}{\mathbf{S}}_{app}+w_{mot}{\mathbf{S}}_{mot}$$

(17)

This gated output is subsequently forwarded to the identity decision layer for joint arbitration with the METM module.

Overall, the AID module provides a reliability-aware short-term association mechanism for anti-UAV tracking. By adaptively balancing appearance similarity and motion consistency, it improves identity association when visual features are weak, ambiguous, or degraded by long-range observation and cluttered backgrounds.

3.5. Motion-Evolution Temporal Memory Module

Although the AID module improves short-term identity association within the temporal window $[t-T,t-1]$, it may be insufficient when a target remains invisible for a longer period. In anti-UAV scenarios, visibility gaps can be caused by atmospheric disturbance, background clutter, terrain occlusion, sensor limitations, or abrupt maneuvers. Once a UAV disappears for multiple frames, its appearance representation becomes less reliable, and its predicted position may gradually drift, making identity recovery difficult after reappearance. Existing trackers often rely on appearance-based re-identification or simple memory updates to handle such cases. However, for small and low-resolution UAV targets, appearance features are unstable, and memory retrieval without motion feasibility constraints can easily reactivate incorrect trajectories.

To address this problem, we introduce the motion-evolution temporal memory (METM) module for long-term trajectory maintenance and identity recovery. As shown in Figure 3, METM complements the short-term association capability of AID by maintaining a dormant memory bank ${\mathcal{M}}_{\mathrm{d}}$ for temporarily unmatched trajectories. Each dormant trajectory is stored with its recent appearance representation, kinematic state, and a time-decayed confidence score. During retrieval, candidate detections are evaluated under spatiotemporal consistency and kinematic feasibility constraints, allowing METM to recover disappeared targets after occlusion, temporary visibility loss, or abrupt maneuvers while reducing incorrect identity reactivation.

Figure 3. Framework of the proposed motion-evolution temporal memory (METM) module.

When a trajectory $T^m$ becomes unmatched at frame $t_0$, its last observed state is recorded at frame $t_0-1$, and the trajectory is transferred to the dormant memory bank. Let $p_t^m\in {\mathbb{R}}^2$ denote the center position of trajectory m at frame t, and let $\Delta t$ denote the time interval between adjacent frames. The velocity and acceleration at the last observed frame are estimated as follows:

$${\mathbf{v}}_{t_0-1}^m={\displaystyle \frac{p_{t_0-1}^m-p_{t_0-2}^m}{\Delta t}}$$

(18)

$${\mathbf{a}}_{t_0-1}^m={\displaystyle \frac{{\mathbf{v}}_{t_0-1}^m-{\mathbf{v}}_{t_0-2}^m}{\Delta t}}$$

(19)

Assuming a constant-acceleration motion model during the visibility gap, the elapsed time from the last observation to the current frame t is defined as follows:

$$\Delta t_{\mathrm{gap}}=(t-t_0+1)\Delta t$$

(20)

The predicted position of dormant trajectory m at frame t is then computed as follows:

$${\widehat{p}}_t^m=p_{t_0-1}^m+{\mathbf{v}}_{t_0-1}^m\Delta t_{\mathrm{gap}}+{\displaystyle \frac{1}{2}}{\mathbf{a}}_{t_0-1}^m\Delta t_{\mathrm{gap}}^2$$

(21)

To account for the growth of prediction uncertainty over time, we define a motion uncertainty scale:

$${\sigma }_{\mathrm{pred}}\left(t\right)={\sigma }_0+{\sigma }_v\Delta t_{\mathrm{gap}}+{\sigma }_a\Delta t_{\mathrm{gap}}^2$$

(22)

where ${\sigma }_0$, ${\sigma }_v$, and ${\sigma }_a$ denote the initial uncertainty, velocity-induced uncertainty, and acceleration-induced uncertainty, respectively.

Given a candidate detection with center position $p_t^n$, the Motion Anomaly Score (MAS) between detection n and dormant trajectory m is defined as follows:

$${\mathrm{MAS}}^{n,m}={\displaystyle \frac{{∥p_t^n-{\widehat{p}}_t^m∥}_2}{{\sigma }_{\mathrm{pred}}\left(t\right)\sqrt{w_{t_0-1}^m{h}_{t_0-1}^m}+ϵ}}$$

(23)

where $w_{t_0-1}^m$ and $h_{t_0-1}^m$ are the width and height of the last observed bounding box of trajectory m, and $ϵ$ is a small constant for numerical stability. A smaller MAS indicates that the candidate detection is more consistent with the predicted motion evolution of the dormant trajectory.

Candidates with ${\mathrm{MAS}}^{n,m}<{\theta }_{\mathrm{normal}}$ are treated as reliable recovery candidates, while those with ${\mathrm{MAS}}^{n,m}>{\theta }_{\mathrm{anomaly}}$ are regarded as kinematically implausible and suppressed. Candidates in the intermediate range are retained with lower confidence and further evaluated by appearance similarity and memory confidence. This strategy avoids prematurely discarding highly maneuverable UAV targets while filtering out physically unreasonable reactivations.

For each temporarily unmatched trajectory, METM stores a memory entry in the dormant memory bank ${\mathcal{M}}_{\mathrm{d}}$. Each entry contains the trajectory-level appearance representation ${\overline{h}}^m$, the last observed kinematic state ${\{p,\mathbf{v},\mathbf{a}\}}_{t_0-1}^m$, the last bounding-box scale $(w_{t_0-1}^m,h_{t_0-1}^m)$, and the missing-frame index $t_0$.

Since the reliability of an inactive trajectory decreases as the visibility gap becomes longer, a time-decayed memory confidence is assigned to each dormant trajectory:

$$w_{\mathrm{mem}}^m\left(t\right)=exp\left[-{\lambda }_t(t-t_0+1)\right]$$

(24)

where ${\lambda }_t$ controls the temporal decay rate. This decay mechanism reduces the influence of old trajectories and prevents stale memory entries from dominating later identity recovery.

During inference, METM retrieves candidate matches from the dormant memory bank ${\mathcal{M}}_{\mathrm{d}}$. To enforce motion feasibility while keeping the recovery process differentiable during training, we introduce a learnable spatial tolerance factor $\gamma$. To ensure that the tolerance remains positive, $\gamma$ is parameterized as follows:

$$\gamma ={\gamma }_{min}+\mathrm{Softplus}\left(\eta \right)$$

(25)

where $\eta$ is a learnable parameter optimized jointly with the tracking network, and ${\gamma }_{min}>0$ is a small lower bound for numerical stability. The Softplus function ensures that the learned tolerance remains positive.

For each dormant trajectory, the learnable motion-consistent search radius is defined as follows:

$$r_t^m=\gamma {\sigma }_{\mathrm{pred}}\left(t\right)\sqrt{w_{t_0-1}^m{h}_{t_0-1}^m}$$

(26)

where ${\sigma }_{\mathrm{pred}}\left(t\right)$ models the growth of motion prediction uncertainty, and $w_{t_0-1}^m$ and $h_{t_0-1}^m$ denote the width and height of the last observed bounding box of trajectory m.

We define a differentiable feasibility gate for each candidate detection:

$$g_{\mathrm{fea}}^{n,m}=\sigma \left({\displaystyle \frac{r_t^m-{∥p_t^n-{\widehat{p}}_t^m∥}_2}{{\tau }_g}}\right)$$

(27)

where $\sigma (·)$ denotes the sigmoid function, and ${\tau }_g$ is a temperature parameter controlling the sharpness of the soft boundary. When a detection lies close to the predicted position, $g_{\mathrm{fea}}^{n,m}$ approaches 1; when it is far outside the feasible motion range, the gate approaches 0. In this way, candidates that violate motion feasibility are softly suppressed while gradients can still update the learnable tolerance factor $\gamma$.

For a candidate detection, the final METM recovery score is computed by combining memory confidence, motion consistency, learnable feasibility, and appearance similarity:

$$s_{\mathrm{METM}}^{n,m}=w_{\mathrm{mem}}^m\left(t\right)·g_{\mathrm{fea}}^{n,m}·exp\left(-{\displaystyle \frac{{\left({\mathrm{MAS}}^{n,m}\right)}^2}{2}}\right)·{\displaystyle \frac{1+cos({\overline{h}}^m,{\mathit{d}}_t^n)}{2}}$$

(28)

where $cos(·,·)$ denotes cosine similarity, and ${\mathit{d}}_t^n$ is the encoded detection representation produced by the Context-Aware Identity Encoder. The cosine similarity is linearly mapped to $[0,1]$ to keep the recovery score non-negative. The feasibility gate allows METM to suppress physically implausible candidates while preserving differentiability for learning the motion tolerance.

The resulting similarity matrix ${\mathbf{S}}_{\mathrm{METM}}\in {\mathbb{R}}^{N\times M_{\mathrm{d}}}$ represents the recovery confidence between current detections and dormant trajectories, where $M_{\mathrm{d}}$ is the number of trajectories stored in the dormant memory bank. This matrix is then forwarded to the identity decision layer for joint arbitration with the short-term association results from the AID module.

Overall, the METM module provides a motion-aware long-term recovery mechanism for temporarily unmatched UAV trajectories. By integrating time-decayed memory confidence, learnable kinematic feasibility, and appearance consistency, it improves identity recovery under occlusion, intermittent visibility loss, and abrupt maneuvers.

3.6. Identity Decision Layer

The identity decision layer (IDL) integrates the short-term association scores from AID and the long-term recovery scores from METM. AID focuses on active trajectories that are recently observed, while METM retrieves candidate matches from dormant trajectories stored in the memory bank. Since these two sources cover different temporal ranges, their predictions may conflict when observations are ambiguous. IDL resolves such conflicts by constructing a unified matching space and applying a priority-aware arbitration strategy. Recent active trajectories are preferred when their matching confidence is sufficiently high, while dormant trajectories are reactivated only when they satisfy spatiotemporal consistency and kinematic feasibility constraints.

Let ${\mathcal{T}}_{\mathrm{a}}=\{T_{\mathrm{a}}^1,\dots ,T_{\mathrm{a}}^{M_{\mathrm{a}}}\}$ denote the set of active trajectories, and let ${\mathcal{T}}_{\mathrm{d}}=\{T_{\mathrm{d}}^1,\dots ,T_{\mathrm{d}}^{M_{\mathrm{d}}}\}$ denote the set of dormant trajectories stored in the METM memory bank. Given N detections in the current frame, AID produces a short-term association matrix ${\mathbf{S}}_{\mathrm{AID}}\in {\mathbb{R}}^{N\times M_{\mathrm{a}}}$, while METM produces a long-term recovery matrix ${\mathbf{S}}_{\mathrm{METM}}\in {\mathbb{R}}^{N\times M_{\mathrm{d}}}$. The unified similarity matrix is constructed by concatenating the two matrices along the trajectory dimension:

$${\mathbf{S}}_{\mathrm{combined}}=\left[{\mathbf{S}}_{\mathrm{AID}}|{\mathbf{S}}_{\mathrm{METM}}\right]\in {\mathbb{R}}^{N\times (M_{\mathrm{a}}+M_{\mathrm{d}})}$$

(29)

This unified space allows detections to be matched against both recently active trajectories and dormant recovery candidates.

Although METM can recover disappeared targets, recent active trajectories are generally more reliable when their association scores are comparable. Therefore, we introduce a priority-aware arbitration score:

$$s_{\mathrm{arb}}^{n,j}=\left\{\begin{array}{cc}{s}_{\mathrm{AID}}^{n,j}+{\lambda }_{\mathrm{pri}},\hfill & 1\le j\le M_{\mathrm{a}}\hfill \\ s_{\mathrm{METM}}^{n,j-M_{\mathrm{a}}},\hfill & M_{\mathrm{a}}<j\le M_{\mathrm{a}}+M_{\mathrm{d}}\hfill \end{array}\right.$$

(30)

where ${\lambda }_{\mathrm{pri}}>0$ is a bias term that favors high-confidence recent associations. This strategy prevents dormant trajectories from being incorrectly reactivated when an active trajectory already provides a reliable match, while still allowing long-term recovery when the METM score is sufficiently strong.

Based on the arbitration matrix ${\mathbf{S}}_{\mathrm{arb}}$, we perform bipartite matching between detections and trajectory candidates:

$${\mathcal{M}}^{*}=Hungarian(-{\mathbf{S}}_{\mathrm{arb}})$$

(31)

where ${\mathcal{M}}^{*}$ denotes the optimal one-to-one assignment set. For each matched pair $(n,j)\in {\mathcal{M}}^{*}$, the identity assignment is determined as follows:

$$\mathrm{Assignment}\left(n\right)=\left\{\begin{array}{cc}\mathrm{ID}\left(T_{\mathrm{a}}^j\right),\hfill & j\le M_{\mathrm{a}}\mathrm{and}{s}_{\mathrm{arb}}^{n,j}>{\theta }_{\mathrm{match}}\hfill \\ \mathrm{ID}\left(T_{\mathrm{d}}^{j-M_{\mathrm{a}}}\right),\hfill & j>M_{\mathrm{a}}\mathrm{and}{s}_{\mathrm{arb}}^{n,j}>{\theta }_{\mathrm{wake}}\hfill \\ \mathrm{New}\mathrm{ID},\hfill & s_{\mathrm{arb}}^{n,j}\le {\theta }_{\mathrm{match}}\mathrm{and}{c}_t^n>{\theta }_{\mathrm{init}}\hfill \\ \mathrm{Rejected},\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(32)

where ${\theta }_{\mathrm{match}}$ is the active-trajectory matching threshold, ${\theta }_{\mathrm{wake}}$ is the dormant-trajectory reactivation threshold, $c_t^n$ is the detection confidence score, and ${\theta }_{\mathrm{init}}$ is the confidence threshold for initializing new trajectories.

After identity assignment, trajectory states are updated according to the matching results. Matched active trajectories are updated with the corresponding detections. Matched dormant trajectories are reactivated and moved back to the active set. Unmatched active trajectories are transferred to the dormant memory bank and initialized with their latest appearance representation, kinematic state, and missing-frame index. Dormant trajectories that remain unmatched for more than $T_{\max }$ frames are removed from memory to avoid stale identity recovery.

Overall, the IDL provides a unified arbitration mechanism that balances recent short-term associations and long-term memory-based recovery. By combining priority-aware matching, trajectory reactivation, and new identity initialization, it maintains identity continuity while reducing incorrect recovery of disappeared trajectories.

4. Experiments

4.1. Datasets and Metrics

The 4th Anti-UAV Benchmark Track-3 is a multi-UAV tracking benchmark released for the 4th Anti-UAV Challenge. It contains 300 thermal infrared video sequences, including 200 sequences for training and 100 sequences for testing. The videos are provided at a spatial resolution of $640\times 512$ pixels. Track-3 focuses on continuous multi-UAV tracking in complex low-altitude surveillance scenes. The number of UAV targets in each frame ranges from 0 to more than 40, which covers both sparse and crowded multi-UAV scenarios. In addition, the dataset includes dynamic backgrounds, complex target motion, tiny-scale UAVs, and diverse scene types such as clouds, buildings, mountains, and sea regions.

These properties make Track-3 particularly suitable for evaluating multi-UAV identity preservation and long-term trajectory continuity. First, multiple visually similar UAVs may appear simultaneously, overlap with each other, or move across cluttered backgrounds, which directly tests whether a tracker can maintain consistent identities under inter-target ambiguity. Second, frequent target disappearance, reappearance, newly appearing UAVs, and abrupt motion require the tracker to recover trajectories after temporal observation gaps rather than only perform frame-to-frame association. Third, the thermal infrared modality and small target scale weaken appearance cues, making identity association more dependent on temporal context and motion consistency. Therefore, this benchmark provides a rigorous testbed for evaluating identity preservation, long-term recovery, and trajectory continuity in anti-UAV multi-object tracking.

We also construct AU-MOT to evaluate robustness under practical surveillance conditions; detailed information is shown in Table 2. AU-MOT contains 300 RGB visible-light video sequences, including 200 sequences for training and 100 sequences for testing. All videos are captured in practical low-altitude surveillance scenarios and provided at a spatial resolution of $1920\times 1080$ pixels. Each sequence contains approximately 1100 frames and includes multiple UAV targets. The average number of UAV targets per frame is 2.8. During annotation, all visible UAV targets are labeled frame by frame with bounding boxes and temporally consistent identity labels. The original temporal order of video frames is preserved during annotation and evaluation, and no temporal resampling or interpolation is applied. The dataset covers diverse surveillance scenes, including buildings, roads, bridges, sky backgrounds, vegetation, and other cluttered environments. These scenes reflect common challenges in real anti-UAV monitoring, such as small target scale, weak appearance cues, background interference, abrupt motion, occlusion, and short-term visibility loss. Compared with general multi-object tracking datasets, AU-MOT focuses on small aerial targets observed at long range and against complex surveillance backgrounds, making it suitable for evaluating identity association and trajectory recovery in anti-UAV multi-object tracking. Due to data usage restrictions, AU-MOT is not publicly available at this stage. All compared methods are trained and evaluated under the same training and testing protocol to ensure fair comparison.

Table 2. Detailed information of the AU-MOT dataset.

Item	AU-MOT
Modality	RGB visible-light video
Number of sequences	300
Training split	200 sequences
Testing split	100 sequences
Spatial resolution	$1920\times 1080$ pixels
Average frames per sequence	∼1100
Average UAVs per frame	2.8
Annotation type	Frame-level bounding boxes and consistent IDs
Temporal sampling	Original frame order, no resampling
Major scene types	Buildings, roads, bridges, sky, vegetation, cluttered backgrounds

We use Higher Order Tracking Accuracy (HOTA) as the primary metric because it jointly reflects detection accuracy and association quality. Following the HOTA protocol, we also report Detection Accuracy (DetA) and Association Accuracy (AssA) to analyze localization and identity association separately [38]. Since identity stability and trajectory continuity are central to anti-UAV tracking, we further report ID F1 score (IDF1), identity switches (IDs), and track fragments (Frag). IDF1 measures the consistency between predicted and ground-truth identities, while IDs counts how often the tracker changes the identity of the same target. Frag measures trajectory fragmentation and reflects the ability of a tracker to maintain continuous trajectories through occlusion, disappearance, and reappearance. We also report MOTA for comparison with existing tracking methods [39]. Evaluation on the 4th Anti-UAV Benchmark Track-3 follows the official protocol, and experiments on AU-MOT use the same evaluation setting for consistency.

All experiments are conducted on a server with 8 NVIDIA RTX 4090 GPUs, an Intel Xeon CPU, and 128 GB RAM. Unless otherwise specified, all methods are trained and evaluated under the same hardware and software environment. Training on the 4th Anti-UAV Benchmark Track-3 takes about 11 h for 60 epochs under the default configuration. During inference, the proposed method runs at about 17.8 FPS on a single NVIDIA RTX 4090 GPU, providing a practical balance between tracking accuracy and computational efficiency for near-real-time anti-UAV surveillance.

4.2. Implementation Details

We adopt Deformable DETR [40] with a ResNet-50 [41] backbone as the base detector, initialized with COCO [42] pre-trained weights. To ensure architectural compatibility with the DETR framework [43], the hidden dimension is consistently set to $C=256$ throughout the system. The ID dictionary $\mathcal{I}$ comprises $S+1=61$ learnable embeddings, including $S=60$ regular identity tokens and one dedicated newborn token, each of dimension $C=256$. In the Context-Aware Identity Encoder, the concatenated visual and identity features are projected back to the hidden dimension C so that they remain compatible with the downstream modules. The AID module uses temporal attention to aggregate appearance features and a two-layer MLP with a hidden dimension of 128 to estimate adaptive reliability.

Our model is trained end-to-end using the AdamW optimizer. The initial learning rate is set to $2\times {10}^{-4}$ for the DETR backbone and ${10}^{-4}$ for the newly introduced tracking modules, with a decay factor of 0.1 applied at epochs 40 and 50. We train for a total of 60 epochs with a batch size of 4 video clips. Each training sample consists of $T+1=30$ consecutive frames randomly sampled from the training sequences, where $T=29$ frames provide the historical context for in-context identity prediction. To balance training efficiency and memory constraints, gradients are backpropagated through the DETR backbone for only 4 frames per clip, while the remaining frames are processed in inference mode without gradient computation. For data preprocessing, input images are resized to a minimum short side of 800 pixels and a maximum long side of 1440 pixels.

For hyperparameter tuning and model selection, we further split the training sequences of each dataset into training and validation subsets, with 10% of the training sequences used for validation. The validation subset is used only for hyperparameter selection and convergence monitoring, while the official test split remains untouched until the final evaluation. The final checkpoint is selected according to the best validation HOTA, with IDF1 and AssA used as auxiliary indicators of identity association quality. After model selection, the selected checkpoint is evaluated on the test split using the same evaluation protocol for all compared methods.

The overall training objective consists of detection and tracking losses:

$$\mathcal{L}={\mathcal{L}}_{\det }+{\mathcal{L}}_{\mathrm{track}}$$

(33)

The detection loss follows the standard DETR formulation:

$${\mathcal{L}}_{\det }={\lambda }_{\mathrm{cls}}{\mathcal{L}}_{\mathrm{cls}}+{\lambda }_{\mathrm{L}1}{\mathcal{L}}_{\mathrm{L}1}+{\lambda }_{\mathrm{giou}}{\mathcal{L}}_{\mathrm{giou}}$$

(34)

where ${\mathcal{L}}_{\mathrm{cls}}$ is the focal classification loss, and ${\mathcal{L}}_{\mathrm{L}1}$ and ${\mathcal{L}}_{\mathrm{giou}}$ are the bounding-box regression losses.

The tracking loss is defined as

$${\mathcal{L}}_{\mathrm{track}}={\lambda }_{\mathrm{AID}}{\mathcal{L}}_{\mathrm{AID}}+{\lambda }_{\mathrm{METM}}{\mathcal{L}}_{\mathrm{METM}}$$

(35)

where ${\mathcal{L}}_{\mathrm{AID}}$ supervises short-term identity association for active trajectories, while ${\mathcal{L}}_{\mathrm{METM}}$ supervises memory-based trajectory recovery for dormant trajectories. The learnable tolerance parameter $\eta$ is updated through ${\mathcal{L}}_{\mathrm{METM}}$, since it directly affects the differentiable feasibility gate in the METM recovery score.

The weighting coefficients are set to ${\lambda }_{\mathrm{cls}}=2.0$, ${\lambda }_{\mathrm{L}1}=5.0$, ${\lambda }_{\mathrm{giou}}=2.0$, ${\lambda }_{\mathrm{AID}}=1.0$, and ${\lambda }_{\mathrm{METM}}=1.0$. The larger weights for localization losses reflect the need for accurate spatial alignment in small UAV tracking, while the tracking losses encourage stable identity association and trajectory recovery.

During inference, detections with confidence scores below ${\theta }_{\det }=0.3$ are discarded. For identity assignment, we set ${\theta }_{\mathrm{match}}=0.5$ for active trajectory matching and ${\theta }_{\mathrm{wake}}=0.3$ for dormant trajectory reactivation. A new trajectory is initialized when a detection cannot be associated with any active or dormant trajectory and its detection confidence satisfies $c_t^n>{\theta }_{\mathrm{init}}$, where ${\theta }_{\mathrm{init}}=0.6$. The maximum dormancy duration in METM is set to $T_{max}=120$ frames. Dormant trajectories that remain unmatched for more than $T_{max}$ frames are removed from the memory bank to avoid stale identity recovery. The temporal memory decay rate is controlled by ${\lambda }_t$, and the feasibility gate is controlled by the temperature parameter ${\tau }_g$. The spatial recovery range is determined by the learnable tolerance factor $\gamma$, which is parameterized by $\gamma ={\gamma }_{min}+\mathrm{Softplus}\left(\eta \right)$, learned during training, and fixed during inference. For clarity, the main model configuration and training settings are summarized in Table 3, and the detailed values of these tracking-related hyperparameters are summarized in Table 4. The proposed method runs at about 17.8 FPS on a single NVIDIA RTX 4090 GPU, providing a practical balance between tracking accuracy and computational efficiency for near-real-time anti-UAV surveillance.

Table 3. Summary of model configuration and training settings.

Category	Item	Setting
Detector	Base detector	Deformable DETR with ResNet-50 backbone
	Initialization	COCO pre-trained weights
Model	Hidden dimension	$C=256$
	Identity dictionary	$S=60$
	AID reliability estimator	Two-layer MLP with hidden dimension 128
Training	Optimizer	AdamW
	Learning rate	$2\times {10}^{-4}$ for the DETR and ${10}^{-4}$ for tracking modules
	Learning-rate schedule	Decay factor of 0.1 at epochs 40 and 50
	Training epochs	60
	Batch size	4
	Temporal window	$T+1=30$ frames, with $T=29$ historical frames
	Gradient backpropagation	DETR backpropagated for 4 frames per clip
	Validation split	10% of the training sequences
	Checkpoint selection	Best validation HOTA with IDF1/AssA as auxiliaries

Table 4. Summary of key tracking-related hyperparameters.

Parameter	Value	Description
${\theta }_{\det }$	0.3	Detection confidence threshold
${\theta }_{\mathrm{match}}$	0.5	Matching threshold for active trajectories
${\theta }_{\mathrm{wake}}$	0.3	Reactivation threshold for dormant trajectories
${\theta }_{\mathrm{init}}$	0.6	Confidence threshold for initializing a new trajectory
$T_{max}$	120 frames	Maximum duration for retaining unmatched dormant trajectories
${\lambda }_t$	0.02	Temporal decay rate for dormant memory confidence
${\tau }_g$	10 pixels	Temperature parameter of the differentiable feasibility gate
${\gamma }_{min}$	0.1	Lower bound of the learnable spatial tolerance factor
$\gamma$	Learned	Spatial tolerance factor learned during training/fixed at inference
${\theta }_{\mathrm{normal}}$	1.0	MAS threshold for reliable recovery candidates
${\theta }_{\mathrm{anomaly}}$	3.0	MAS threshold for suppressing implausible candidates

4.3. State-of-the-Art Comparison

Before comparing with existing trackers, we first clarify the adopted protocol to ensure a fair comparison. For tracking-by-detection methods, including SORT, DeepSORT, ByteTrack, OC-SORT, and DiffMOT, we use the same detector outputs as input. Specifically, the detector is Deformable DETR with a ResNet-50 backbone, initialized with COCO pre-trained weights and trained on the same training split for each dataset. Therefore, the comparison mainly reflects differences in identity association and trajectory maintenance rather than detection quality. For end-to-end trackers, such as TrackFormer and MOTR, we follow their official implementations while using the same training and test data. During evaluation, all methods use the same input resolution, test split, evaluation scripts, and evaluation metrics.

We compare the proposed framework with representative trackers on the 4th Anti-UAV Benchmark Track-3 test set, including generic multi-object trackers and UAV-oriented methods. Table 5 reports the quantitative comparison with existing tracking methods. As shown in Table 5, the baselines include tracking-by-detection methods, such as SORT [16], DeepSORT [22], ByteTrack [23], and OC-SORT [13], as well as end-to-end trackers, such as TrackFormer [26] and MOTR [27]. The proposed method achieves the best overall performance across all evaluation metrics. Specifically, it obtains 63.6% HOTA, outperforming the strongest competing method, DiffMOT [44], by 3.5%. It also achieves 78.1% DetA, 50.3% AssA, 84.9% MOTA, and 64.1% IDF1, demonstrating better localization accuracy and stronger identity preservation in challenging anti-UAV scenarios. The improvement is particularly clear in identity-related and continuity-related metrics. Compared with the strongest baseline DiffMOT, our method improves HOTA, IDF1, and AssA by 3.5, 3.9, and 1.5 percentage points, corresponding to relative gains of 5.8%, 6.5%, and 3.1%, respectively. It also reduces identity switches from 648 to 513 and track fragments from 1084 to 826, corresponding to relative reductions of 20.8% and 23.8%. These results indicate that the proposed AID module improves identity association under weak visual evidence, while the METM module helps recover temporarily disappeared targets and maintain trajectory continuity. Overall, the proposed framework is more robust to long-range observation, weak target appearance, abrupt motion, occlusion, and temporary disappearance.

Table 5. Performance comparison with state-of-the-art methods on the 4th Anti-UAV Benchmark Track-3 test set.

Methods	HOTA ↑	DetA ↑	AssA ↑	MOTA ↑	IDF1 ↑	IDs ↓	Frag ↓
SORT [16]	45.1	62.4	32.9	71.7	38.3	1738	2738
DeepSORT [22]	48.2	63.9	36.9	74.3	45.4	1598	2611
TrackFormer [26]	51.9	67.6	37.4	75.5	48.8	1417	2406
ByteTrack [23]	52.4	68.7	39.6	78.7	51.2	1308	2293
MOTR [27]	54.5	71.3	39.3	77.6	52.3	1173	2002
OC-SORT [13]	55.7	74.7	43.6	80.5	56.2	1047	1884
SiamDT [5]	58.8	76.4	44.9	81.1	59.1	872	1579
DiffMOT [44]	60.1	77.3	48.8	82.4	60.2	648	1084
Ours	63.6	78.1	50.3	84.9	64.1	513	826

The best results are shown in bold. ↑ indicates that higher values are better. ↓ indicates that lower values are better.

We further evaluate the proposed framework on the AU-MOT test set to examine its effectiveness under practical anti-UAV surveillance conditions. Table 6 reports the quantitative comparison with existing tracking methods. As shown in Table 6, our method achieves the best performance across all evaluation metrics, with 67.2% HOTA, 82.9% DetA, 54.7% AssA, 88.9% MOTA, and 67.8% IDF1. Compared with the strongest baseline DiffMOT, our method improves HOTA, IDF1, and AssA by 5.8, 5.2, and 4.6 percentage points, corresponding to relative gains of 9.4%, 8.3%, and 9.2%, respectively. It also reduces identity switches from 604 to 482 and track fragments from 1019 to 792, corresponding to relative reductions of 20.2% and 22.3%. These improvements indicate that the proposed framework can maintain more stable identities and more continuous trajectories, rather than only improving frame-level localization. Since AU-MOT contains diverse surveillance scenes with frequent scale changes, weak target appearance, and background interference, the consistent gains on this dataset demonstrate the robustness of the proposed method in practical anti-UAV monitoring scenarios.

Table 6. Performance comparison with state-of-the-art methods on the AU-MOT test set.

Methods	HOTA ↑	DetA ↑	AssA ↑	MOTA ↑	IDF1 ↑	IDs ↓	Frag ↓
SORT [16]	44.2	61.6	32.1	71.7	37.1	1796	2707
DeepSORT [22]	47.9	63.2	35.9	73.4	44.6	1694	2597
TrackFormer [26]	50.1	71.4	35.7	73.9	47.4	1517	2508
ByteTrack [23]	53.7	69.4	40.2	79.6	52.5	1427	2397
MOTR [27]	53.9	73.6	38.8	76.2	51.8	1398	2304
OC-SORT [13]	54.8	71.3	44.5	81.6	56.3	1219	2184
SiamDT [5]	59.7	77.4	45.9	82.5	59.8	784	1398
DiffMOT [44]	61.4	75.6	50.1	83.9	62.6	604	1019
Ours	67.2	82.9	54.7	88.9	67.8	482	792

The best results are shown in bold. ↑ indicates that higher values are better. ↓ indicates that lower values are better.

The proposed framework improves performance consistently on both the public 4th Anti-UAV Benchmark Track-3 and our AU-MOT dataset. These gains come mainly from the complementary roles of AID and METM. AID adjusts the relative contribution of appearance and motion cues according to their estimated reliability, reducing association errors when visual features are weakened by long-range observation, motion blur, or cluttered backgrounds. METM improves long-term trajectory continuity by maintaining a time-decayed trajectory memory and recovering temporarily disappeared targets under motion consistency constraints. Working together, the two modules make the tracker more robust to weak appearance, abrupt motion, and intermittent visibility, leading to more stable identity assignment and more continuous trajectories in challenging anti-UAV scenarios.

4.4. Ablation Studies

4.4.1. Effect of Dual-Track Parallel Mechanism

To evaluate the effectiveness of the two primary modules (AID and METM) within the overall tracking system, we conduct ablation experiments on both the 4th Anti-UAV Benchmark Track-3 and AU-MOT. Table 7 presents results on the 4th Anti-UAV Benchmark Track-3, while Table 8 reports corresponding ablations on AU-MOT.

Table 7. Ablation on the dual-track parallel mechanism on the 4th Anti-UAV Benchmark Track-3.

AID		METM	HOTA ↑	DetA ↑	AssA ↑	IDF1 ↑	IDs ↓	Frag ↓
Fixed	Adaptive
-	-	-	48.3	65.1	35.6	49.4	1637	2693
✓	-	-	52.4	66.4	40.1	52.9	1329	2387
-	✓	-	58.7	72.4	46.2	59.3	916	1783
-	-	✓	59.4	72.9	47.1	60.2	763	1329
✓	-	✓	61.1	76.8	48.5	62.4	617	948
-	✓	✓	63.6	78.1	50.3	64.1	513	826

Fixed indicates equal weight fusion. Adaptive denotes the proposed reliability estimation. Bold values indicate the best performance across all metrics. ↑ indicates that higher values are better. ↓ indicates that lower values are better.

Table 8. Ablation on the dual-track parallel mechanism on AU-MOT.

AID		METM	HOTA ↑	DetA ↑	AssA ↑	IDF1 ↑	IDs ↓	Frag ↓
Fixed	Adaptive
-	-	-	50.3	65.7	38.2	51.6	1497	2498
✓	-	-	57.6	73.4	44.5	59.2	1164	1975
-	✓	-	60.7	75.5	46.8	61.8	726	1217
-	-	✓	61.9	76.2	47.6	63.3	689	1174
✓	-	✓	62.8	77.9	49.6	64.1	559	928
-	✓	✓	67.2	82.9	54.7	67.8	482	792

Fixed indicates equal weight fusion. Adaptive denotes the proposed reliability estimation. Bold values indicate the best performance across all metrics. ↑ indicates that higher values are better. ↓ indicates that lower values are better.

Experimental results on the 4th Anti-UAV Benchmark Track-3 are reported in Table 7. The baseline model without AID and METM obtains 48.3% HOTA, 35.6% AssA, and 49.4% IDF1, with 1637 identity switches and 2693 track fragments. These results indicate that direct detection with simple association is insufficient for robust anti-UAV tracking, especially under weak appearance cues and irregular UAV motion. Introducing AID with fixed-weight fusion improves HOTA from 48.3% to 52.4% and AssA from 35.6% to 40.1%, while reducing IDs from 1637 to 1329 and Frag from 2693 to 2387. This shows that combining appearance and motion cues is beneficial for short-term identity association. When the proposed adaptive reliability estimation is used, the performance further increases to 58.7% HOTA, 46.2% AssA, and 59.3% IDF1. Meanwhile, IDs and Frag are reduced to 916 and 1783, respectively. These improvements demonstrate that dynamic cue weighting is important in anti-UAV tracking, where the reliability of appearance and motion cues changes with target scale, motion state, and background complexity. The METM module also provides substantial gains when used independently. Compared with the baseline, adding METM improves HOTA from 48.3% to 59.4%, AssA from 35.6% to 47.1%, and IDF1 from 49.4% to 60.2%. More importantly, Frag is reduced from 2693 to 1329, indicating that motion-evolution temporal memory is effective for maintaining trajectory continuity after occlusion, abrupt maneuvers, and temporary disappearance. Combining METM with fixed-weight AID further improves the performance to 61.1% HOTA, 48.5% AssA, and 62.4% IDF1, while reducing IDs and Frag to 617 and 948. This confirms the complementarity between short-term multi-cue association and long-term trajectory recovery.

The complete dual-track architecture, which integrates adaptive AID and METM, achieves the best performance, with 63.6% HOTA, 78.1% DetA, 50.3% AssA, and 64.1% IDF1. It also obtains the lowest IDs and Frag, with 513 identity switches and 826 track fragments. These results verify that reliability-aware short-term association and motion-aware temporal memory jointly improve identity stability and trajectory continuity in challenging anti-UAV scenarios.

Experimental results on AU-MOT are reported in Table 8. The baseline model without AID and METM obtains 50.3% HOTA, 38.2% AssA, and 51.6% IDF1, with 1497 identity switches and 2498 track fragments. These results indicate that direct detection with simple association is insufficient for stable identity preservation and trajectory continuity in complex anti-UAV surveillance scenes. Introducing AID with fixed-weight fusion improves HOTA from 50.3% to 57.6%, AssA from 38.2% to 44.5%, and IDF1 from 51.6% to 59.2%. Meanwhile, IDs and Frag are reduced from 1497 to 1164 and from 2498 to 1975, respectively. This shows that combining appearance and motion cues is beneficial for short-term identity association. When adaptive reliability estimation is further introduced, the performance increases to 60.7% HOTA, 46.8% AssA, and 61.8% IDF1. IDs and Frag are also reduced to 726 and 1217, respectively, demonstrating that adaptive cue weighting can better handle weak visual features, scale variation, and background interference in AU-MOT. The METM module also brings clear improvements when used independently. Compared with the baseline, adding METM improves HOTA from 50.3% to 61.9%, AssA from 38.2% to 47.6%, and IDF1 from 51.6% to 63.3%. It also reduces IDs from 1497 to 689 and Frag from 2498 to 1174, showing that motion-evolution temporal memory is effective for recovering temporarily disappeared targets and maintaining trajectory continuity. When METM is combined with fixed-weight AID, the performance further increases to 62.8% HOTA, 49.6% AssA, and 64.1% IDF1, with IDs and Frag reduced to 559 and 928. This confirms the complementarity between short-term multi-cue association and long-term memory-based recovery.

The complete dual-track architecture, which integrates adaptive AID and METM, achieves the best overall performance, with 67.2% HOTA, 82.9% DetA, 54.7% AssA, and 67.8% IDF1. It also obtains the lowest IDs and Frag, with 482 identity switches and 792 track fragments. Compared with the baseline, the full model reduces IDs by 67.8% and Frag by 68.3%, demonstrating a clear improvement in identity stability and trajectory continuity. These results verify that adaptive short-term association and motion-aware long-term recovery are both necessary for robust anti-UAV multi-object tracking on AU-MOT.

4.4.2. Components of Adaptive Identity Disambiguation Module

Table 9 presents the ablation results of different components in the AID module. When only the appearance pathway is used, the model achieves 52.6% HOTA, 41.8% AssA, and 52.9% IDF1, with 1324 identity switches and 2385 track fragments. This limited performance indicates that appearance cues alone are insufficient for reliable identity association in anti-UAV scenarios, where UAV targets are often small, low-resolution, and weakly textured. In contrast, using only the motion pathway improves HOTA to 59.2%, AssA to 46.2%, and IDF1 to 57.3%, while reducing IDs and Frag to 739 and 1362, respectively. These results show that motion cues provide more stable information than appearance features under long-range observation and complex backgrounds.

Table 9. Ablation on AID module components on the 4th Anti-UAV Benchmark Track-3 with METM enabled.

Appearance Pathway	Motion Pathway	ARE	HOTA ↑	DetA ↑	AssA ↑	IDF1 ↑	IDs ↓	Frag ↓
✓	-	-	52.6	67.8	41.8	52.9	1324	2385
-	✓	-	59.2	74.6	46.2	57.3	739	1362
✓	✓	-	60.4	75.8	47.8	61.5	598	974
✓	✓	✓	63.6	78.1	50.3	64.1	513	826

Bold values indicate the best performance across all metrics. ↑ indicates that higher values are better. ↓ indicates that lower values are better.

When both appearance and motion pathways are used without adaptive reliability estimation, the performance further increases to 60.4% HOTA, 47.8% AssA, and 61.5% IDF1. IDs and Frag are also reduced to 598 and 974, respectively. This confirms the complementarity between the two pathways: motion cues help maintain association when visual features are degraded, while appearance cues still provide useful discrimination when motion patterns are ambiguous. After introducing the proposed adaptive reliability estimation (ARE), the model achieves the best performance across all metrics, with 63.6% HOTA, 78.1% DetA, 50.3% AssA, and 64.1% IDF1. Compared with the fixed fusion setting, ARE improves HOTA by 3.2%, DetA by 2.3%, AssA by 2.5%, and IDF1 by 2.6%. It also reduces IDs from 598 to 513 and Frag from 974 to 826, corresponding to relative reductions of 14.2% and 15.2%, respectively. These results indicate that ARE enables the tracker to select more reliable cues for each association decision, leading to fewer identity changes and more stable trajectories across frames.

4.4.3. Components of Motion-Evolution Temporal Memory Module

To further verify the effectiveness of the motion-evolution temporal memory (METM) module, we conduct ablation studies on its key designs for long-term trajectory recovery, including the Motion Anomaly Score (MAS), temporal-decayed memory, and learnable spatiotemporal feasibility. All experiments are conducted on the 4th Anti-UAV Benchmark Track-3 with the AID module enabled, so that the analysis focuses on the contribution of METM.

We compare the proposed MAS with three alternatives: no anomaly filtering, IoU-based gating, and fixed-distance gating. In the no-filtering setting, all unmatched trajectories are retained for possible recovery. IoU-based gating keeps a candidate only when the IoU between its bounding box and the predicted box exceeds a threshold ${\tau }_{\mathrm{IoU}}$. Distance-based gating accepts a candidate if its center distance to the predicted position satisfies $\parallel p_t^n-{\widehat{p}}_t^m{\parallel }_2<{\tau }_d$.

As shown in Table 10, directly retaining all unmatched trajectories without filtering achieves only 54.3% HOTA, 40.9% AssA, and 54.8% IDF1, with 1278 identity switches and 2184 track fragments. This result indicates that unfiltered dormant trajectories introduce many unreliable recovery candidates, which can lead to incorrect identity reactivation and fragmented trajectories. IoU-based gating improves the performance to 56.9% HOTA, 44.7% AssA, and 58.2% IDF1, while reducing IDs and Frag to 983 and 1698, respectively. However, its performance remains limited. Fixed-distance gating further improves HOTA to 58.9%, AssA to 47.1%, and IDF1 to 62.2%, suggesting that center-distance constraints are more suitable than IoU-based overlap for UAV trajectory recovery. Nevertheless, this strategy still relies on a fixed spatial threshold and cannot adapt to the growing uncertainty of motion prediction.

Table 10. Ablation on METM module components on the 4th Anti-UAV Benchmark Track-3 with AID enabled. We compare the proposed MAS with baseline gating strategies to evaluate its effectiveness in filtering physically implausible candidates.

Strategy	HOTA ↑	DetA ↑	AssA ↑	IDF1 ↑	IDs ↓	Frag ↓
No Filtering	54.3	69.2	40.9	54.8	1278	2184
IoU-based	56.9	72.3	44.7	58.2	983	1698
Fixed-distance	58.9	74.6	47.1	62.2	893	1524
MAS (Ours)	63.6	78.1	50.3	64.1	513	826

Bold values indicate the best performance across all metrics. ↑ indicates that higher values are better. ↓ indicates that lower values are better.

By contrast, the proposed MAS achieves the best performance across all metrics, with 63.6% HOTA, 78.1% DetA, 50.3% AssA, and 64.1% IDF1. It also reduces IDs to 513 and Frag to 826. Compared with fixed-distance gating, MAS improves HOTA by 4.7%, AssA by 3.2%, and IDF1 by 1.9%, while reducing IDs and Frag by 42.6% and 45.8%, respectively. These results show that scale-normalized motion anomaly estimation provides a more reliable criterion for memory retrieval than fixed spatial filtering. By accounting for both motion uncertainty and target scale, MAS can suppress physically implausible candidates while preserving feasible recovery paths, which is particularly important for anti-UAV tracking with temporary disappearance and irregular target motion.

We evaluate the proposed exponential temporal decay by comparing it with three alternatives. In the fixed-weight setting, all dormant memory entries are assigned a constant confidence weight $w=1.0$, regardless of the missing duration. In the hard cut-off setting, a memory entry is retained only within a fixed window of $T=30$ frames, i.e., $w=1$ if $\Delta t_{\mathrm{miss}}\le T$, and $w=0$ otherwise. In the linear decay setting, the memory confidence decreases linearly as $w_{\mathrm{lin}}\left(t\right)=max(0,1-\Delta t_{\mathrm{miss}}/T)$, where $\Delta t_{\mathrm{miss}}=t-t_0+1$ denotes the number of missing frames. These comparisons examine whether smooth temporal decay provides a more reliable estimate for dormant trajectories.

As shown in Table 11, the fixed-weight strategy obtains 53.6% HOTA, 41.2% AssA, and 50.8% IDF1, with 1229 identity switches and 2207 track fragments. Since all dormant trajectories are assigned the same confidence regardless of their missing duration, outdated memory entries may still participate in identity recovery, leading to unreliable associations and fragmented trajectories. The hard cut-off strategy improves the performance to 55.7% HOTA, 42.9% AssA, and 54.3% IDF1, while reducing IDs and Frag to 1184 and 1974, respectively. However, its improvement remains limited because a fixed temporal window may prematurely discard useful long-term trajectory information when UAV targets reappear after extended disappearance. Linear decay further improves the performance to 60.6% HOTA, 46.8% AssA, and 59.9% IDF1. It also reduces IDs and Frag to 693 and 1087, respectively, showing that gradually decreasing memory confidence is more effective than fixed weighting or hard removal. Nevertheless, linear decay uses a uniform attenuation rule and may not sufficiently model the reliability change of dormant trajectories under different motion states and disappearance durations.

Table 11. Comparison of different temporal decay strategies for the dormant memory bank ${\mathcal{M}}_{\mathrm{d}}$.

Strategy	HOTA ↑	DetA ↑	AssA ↑	IDF1 ↑	IDs ↓	Frag ↓
Fixed-weight	53.6	67.4	41.2	50.8	1229	2207
Hard Cut-off	55.7	69.6	42.9	54.3	1184	1974
Linear Decay	60.6	74.9	46.8	59.9	693	1087
Temporal Decay (Ours)	63.6	78.1	50.3	64.1	513	826

Bold values indicate the best performance across all metrics. ↑ indicates that higher values are better. ↓ indicates that lower values are better.

The proposed temporal decay strategy achieves the best performance across all metrics, reaching 63.6% HOTA, 78.1% DetA, 50.3% AssA, and 64.1% IDF1. Compared with linear decay, it improves HOTA by 3.0%, DetA by 3.2%, AssA by 3.5%, and IDF1 by 4.2%. These results indicate that the proposed temporal decay mechanism provides a better balance between preserving useful long-term memory and suppressing stale trajectory information, leading to more reliable identity recovery after temporary disappearance.

We further investigate the effect of spatiotemporal feasibility constraints on trajectory recovery. We compare three strategies: Global Search, Fixed Radius, and Learnable $\gamma$. In Global Search, no spatial feasibility constraint is applied and all dormant trajectories are considered. In Fixed Radius, candidates are searched within a fixed radius around the predicted position. In Learnable $\gamma$, $\gamma$ is optimized through the differentiable feasibility gate. This comparison evaluates whether adaptive motion-aware search is more effective than fixed spatial constraints for recovering small UAV targets after disappearance.

As shown in Table 12, the Global Search strategy obtains only 54.3% HOTA, 40.9% AssA, and 55.3% IDF1, with 1248 identity switches and 2175 track fragments. Since all dormant trajectories are considered without spatial constraints, many physically implausible candidates are introduced into memory retrieval, which weakens identity recovery and increases trajectory fragmentation. The Fixed Radius strategy improves the performance to 57.9% HOTA, 45.2% AssA, and 59.0% IDF1 by suppressing distant candidates. It also reduces IDs and Frag to 938 and 1739, respectively. However, a fixed search radius cannot adapt to different target scales, motion uncertainties, or disappearance durations, which limits its effectiveness in highly dynamic anti-UAV scenarios.

Table 12. Effect of spatiotemporal feasible region constraints on tracking performance.

Constraint	HOTA ↑	DetA ↑	AssA ↑	IDF1 ↑	IDs ↓	Frag ↓
Global Search	54.3	68.7	40.9	55.3	1248	2175
Fixed Radius	57.9	73.1	45.2	59.0	938	1739
Learnable $\gamma$	63.6	78.1	50.3	64.1	513	826

Bold values indicate the best performance across all metrics. ↑ indicates that higher values are better. ↓ indicates that lower values are better.

The proposed learnable $\gamma$ achieves the best performance across all metrics, with 63.6% HOTA, 78.1% DetA, 50.3% AssA, and 64.1% IDF1. Compared with the Fixed Radius strategy, it improves HOTA by 5.7%, DetA by 5.0%, AssA by 5.1%, and IDF1 by 5.1%. It also reduces IDs from 938 to 513 and Frag from 1739 to 826, corresponding to relative reductions of 45.3% and 52.5%, respectively. These results show that the learnable spatiotemporal feasibility constraint provides a more adaptive recovery range than fixed spatial filtering. By adjusting the candidate search region according to motion uncertainty and trajectory scale, it improves memory retrieval reliability while preserving feasible recovery paths for UAV targets after temporary disappearance or abrupt motion.

4.5. Qualitative Analysis

We provide qualitative comparisons with DiffMOT on the 4th Anti-UAV Benchmark Track-3 and AU-MOT to further examine the tracking behavior of the proposed method. DiffMOT is selected as the comparison method because it is the strongest competing method in our quantitative experiments. The results are shown in Figure 4 and Figure 5, respectively. The qualitative results show that the proposed method maintains more stable identities and produces more continuous trajectories in challenging anti-UAV scenarios.

Figure 4. Qualitative comparison on the 4th Anti-UAV Benchmark Track-3. The results are arranged by columns: (a,d) ground truth; (b,e) our method; and (c,f) DiffMOT. The frame index is shown in the upper-left corner of each image. The selected examples illustrate occlusion, temporary disappearance, and target reappearance in thermal infrared videos. The proposed method correctly re-identifies the reappearing UAV and preserves its identity, whereas DiffMOT assigns a new identity after the observation gap.

Figure 5. Qualitative comparison on AU-MOT. The results are arranged by columns: (a,d) ground truth; (b,e) our method; and (c,f) DiffMOT. The frame index is shown in the upper-left corner of each image. The selected examples illustrate background clutter, weak target appearance, and scale variation in visible-light surveillance scenes. The proposed method maintains stable target identities, whereas DiffMOT tends to lose UAV targets or confuse background structures with real targets.

Figure 4 shows qualitative results on the 4th Anti-UAV Benchmark Track-3. The selected examples mainly illustrate occlusion, target disappearance, and target reappearance in thermal infrared videos. When the UAV is temporarily occluded and reappears in later frames, DiffMOT fails to recover the original identity after reappearance and instead assigns a new identity to the target, resulting in an identity switch. This indicates that DiffMOT has limited ability to recover target identities when the observation is interrupted by occlusion. In contrast, the proposed method correctly re-identifies the UAV after occlusion and preserves the same identity across the sequence. This result demonstrates the effectiveness of the motion-evolution temporal memory (METM) module, which stores dormant trajectories and retrieves reappearing targets under motion consistency constraints.

Figure 5 shows qualitative results on AU-MOT. These examples emphasize background clutter, weak target appearance, and scale variation in practical visible-light surveillance scenes. In the road and bridge scenes, small UAV targets are easily confused with background structures, causing DiffMOT to lose targets or produce false associations. In the sky or vegetation scenes, the target scale changes across frames and the visual appearance remains weak, making identity association more difficult. The proposed method maintains more stable identities under these conditions because AID adaptively balances appearance and motion cues according to their reliability, while METM preserves trajectory continuity when short-term observations become unreliable.

5. Discussion

Although the proposed framework improves identity association and trajectory continuity in challenging anti-UAV multi-object tracking scenarios, several limitations remain. First, when UAV targets appear in extremely dense groups, their small scale and highly similar appearance may still cause inter-target confusion, especially when multiple targets move closely with overlapping trajectories. Second, METM can recover temporarily disappeared targets by using motion-evolution memory, but when a target is invisible for a duration far beyond the memory horizon, the stored trajectory state becomes less reliable and incorrect reactivation may occur after reappearance. Third, severe sensor noise, thermal drift, motion blur, or very low target contrast may degrade the detector and weaken both appearance and motion cues, which can further affect downstream identity association. Future work will investigate stronger long-term re-identification, group-aware motion modeling, and multi-sensor fusion to improve robustness under these extreme conditions.

6. Conclusions

This paper presented a context-aware identity prediction framework for anti-UAV multi-object tracking in remote sensing videos. Instead of treating detection and association as two separate stages, the proposed framework infers target locations and identities from historical trajectory priors, reducing the dependence on handcrafted matching rules and enabling more unified tracking optimization. The framework adopts a dual-track parallel mechanism consisting of the adaptive identity disambiguation (AID) module and the motion-evolution temporal memory (METM) module. AID adaptively balances appearance and motion cues according to their reliability, improving short-term identity association when visual evidence is weak or ambiguous. METM maintains a time-decayed trajectory memory and performs motion-consistent recovery for temporarily disappeared targets, helping preserve trajectory continuity after occlusion, abrupt motion, or intermittent visibility loss. The identity decision layer further integrates short-term association and long-term recovery results to produce stable identity assignments. Experiments on the 4th Anti-UAV Benchmark Track-3 and the newly constructed AU-MOT dataset demonstrate the effectiveness of the proposed method. Compared with representative tracking-by-detection, end-to-end, and anti-UAV tracking methods, our method achieves better overall performance and clear improvements in association-related metrics. Ablation studies further verify the complementary roles of AID and METM, while qualitative results show that the proposed framework can better handle target reappearance, complex backgrounds, and weak UAV appearance.

Future work will focus on modeling interactions among dense UAV targets and incorporating multi-spectral sensing to improve robustness under nighttime conditions, adverse weather, and more complex low-altitude surveillance scenarios.