High-Order Temporal Context-Aware Aerial Tracking with Heterogeneous Visual Experts

Abstract

Visual tracking from the unmanned aerial vehicle (UAV) perspective has been at the core of many low-altitude remote sensing applications. Most of the aerial trackers follow “tracking-by-detection” paradigms or their temporal-context-embedded variants, where the only visual appearance cue is encompassed for representation learning and estimating the spatial likelihood of the target. However, the variation of the target appearance among consecutive frames is inherently unpredictable, which degrades the robustness of the temporal context-aware representation. To address this concern, we advocate extra visual motion exhibiting predictable temporal continuity for complete temporal context-aware representation and introduce a dual-stream tracker involving explicit heterogeneous visual tracking experts. Our technical contributions involve three-folds: (1) high-order temporal context-aware representation integrates motion and appearance cues over a temporal context queue, (2) bidirectional cross-domain refinement enhances feature representation through cross-attention based mutual guidance, and (3) consistent decision-making allows for anti-drifting localization via dynamic gating and failure-aware recovery. Extensive experiments on four UAV benchmarks (UAV123, UAV123@10fps, UAV20L, and DTB70) illustrate that our method outperforms existing aerial trackers in terms of success rate and precision, particularly in occlusion and fast motion scenarios. Such superior tracking stability highlights its potential for real-world UAV applications.

1. Introduction

Aerial tracking is devoted to estimating the states of target (e.g., position and scale) over videos captured by an onboard camera in an unmanned aerial vehicle (UAV) platform. Benefitting from the unique bird’s-eye view and wide field, aerial tracking has found wide utility in a plethora of earth observation applications [1,2], such as environmental monitoring [3], disaster management [4,5], and precision agriculture. Because of the inevitable mobility of the UAV, there is an inherent need for aerial trackers that are robust to the appearance variations of the target caused by viewpoint changes, frequent occlusions, fast motion, and illumination shifts, to name but a few.

One drastic way to overcome the appearance variation is the straightforward exploration of “tracking-by-detection” paradigm, where discriminative appearance representations are learned to differentiate target candidates from complex terrain clutters frame by frame. Early efforts of discriminative correlation filter (DCF)-based trackers [6,7,8,9] are treated as major milestones, whose progress is attributed to approximating geometric translation of appearance by circularly shifting the target template and then simplifying the Moore–Penrose inverse-based learning stage with element-wise operations in Fourier domain. Such shallow appearance classifiers with fast online parameter adaption admits deployment on resource-constrained UAV platforms. Further, motivated by the modern data-driven representation learning, the Siamese-based tracker and its variants [10,11] establish over-parametric metrics regarding target appearance, where visual similarity between the initial exemplar and larger search image is densely evaluated with cross-correlation. Although the deep tracker tends to be criticized for time-consuming feedforward computation, its “uniform” convolution operation can be executed in parallel with the help of onboard hardware accelerators and thus balance the tradeoff between offline appearance discriminability and online inference efficiency.

An alternative but even more rational tracking strategy is to work with temporal contexts of the visual appearance. Instead of “instance detection” for each frame independently, the temporal context-aware trackers could benefit from consecutive frame information and then progressively update appearance parameters to handle the target appearance variations or adjacent similar distractors over time. For instance, DCF-based methods [12,13] integrate temporal regularization, enforcing appearance consistency over time and adaptively updating templates to accommodate tracking drifts. Memory-augmented trackers [14,15] instantiate external memories to store historical target states, mitigating appearance shifts by recalling past appearances and long-term patterns. Also, the empirical success of transformer-based detectors enlightens the recent top-performing trackers [16,17] turn to leverage the best of both worlds: deep-learned appearance features and the attention-based long-range dependency of spatial–temporal contexts.

While effective, the aforementioned aerial trackers still suffer from an inherent unpredictability of the target appearance among consecutive frames. Different from the geometric motion cue that follows physical rules and exhibits temporal continuity, the target appearance, as a semantic description from the aerial perspective, is more sensitive to the aerial challenging attributes: (1) severe viewpoint variations from UAV maneuvers cause drastic appearance changes, (2) illumination conditions fluctuate significantly in outdoor environments, (3) partial or complete occlusions temporarily eliminate robust appearance cue, and (4) appearance-similar objects in cluttered background create ambiguity, as shown in Figure 1. Despite the motion cues (e.g., optical flow field) being incorporated into several temporal context-aware trackers [18,19,20], most of the explorations are restricted to auxiliary inference for the appearance association. Such under-utilization of temporal contexts results from two technical issues: naive temporal invariance assumption and inherent heterogeneity between the appearance and motion cues.

To address these concerns, we present high-order temporal context-aware representation for jointly exploiting heterogeneous visual cues in Siamese-based aerial tracking framework. Beyond appearance consistency constraints between adjacent frames, we explicitly predict and recalibrate the similarity map in the latent representation space with the appearance and motion cues of targets over a temporal context queue. Specific contributions of this paper are summarized as follows:

• High-Order Temporal Context-Aware Representation: We propose a dual-stream framework that enriches motion and appearance features with temporal contexts. Motion features, extracted from optical flow and motion-history images (MHIs), capture dynamic trajectories, while appearance features, refined through temporal-adaptive convolutions, adapt to appearance variation. Both enhanced cues improve robustness against drifting or disruption commonly encountered in aerial tracking.
• Heterogeneous Feature Representation Refinement: We leverage a bidirectional cross-domain interaction that equally refines the motion and appearance features. Motion guides appearance to ensure temporal consistency, while appearance calibrates motion to suppress noise, creating a cohesive representation resilient to occlusions and complex backgrounds.
• Consistent Decision Fusion with Smooth Constraint: We introduce a decision module that dynamically judges the refined feature pairs with velocity-based gating and enforces consistent inference with smooth spatial–temporal constraints so as to avoid unreasonable decision switch between the heterogeneous visual experts.

Extensive evaluations on UAV benchmarks demonstrate that our method achieves 0.616 success rate on UAV123, outperforming pure appearance-based trackers like SiamRPN [21] and SiamAPN++ [11] by 0.035 and 0.037, respectively, in general scenarios. In challenging occlusion scenarios, our approach maintains a 0.563 success rate, surpassing TCTrack [16] by 0.056. The motion stream contributes significantly to the performance gain, particularly in fast-motion cases where we achieve 0.725 success rate compared to 0.637 without motion modeling, validating the necessity of explicit motion context integration.

The remainder of this paper is organized as follows. Section 2 introduces the related works. Section 3 details our proposed framework and methodology. The performance comparison with other SOTA trackers is reported in Section 4, including experimental setup, quantitative results across four UAV benchmarks, and qualitative analysis and ablation studies. Discussion is presented in Section 5. Finally, conclusions and future work are presented in Section 6.

2. Related Works

2.1. Visual Tracking in Aerial Remote Sensing

Early explorations of aerial tracking typically employed a correlation filter (CF) [6,8] or its variants due to their remarkable computational efficiency (i.e., moderate FLOPs and memory consumption). Technically, the CF trackers approximate conventional dense sampling strategy by circularly shifting the target templates and thus learn the appearance filter in the Fourier domain with the property of the circulant matrix. However, due to the lack of robust feature representation of target appearance, the learned correlation filter, as a shallow binary decision hyperplane, often struggles to discriminate the target from complex backgrounds.

Recently, the advancement of deep representation learning has revolutionized visual tracking with the Siamese-based paradigm. SiamFC [10] pioneered this road by formulating tracking as appearance similarity learning between a template and search region. Subsequent works like SiamRPN++ [21] and SiamAPN++ [11] incorporated region proposal networks and attention mechanisms to enhance localization accuracy. While these approaches demonstrated effectiveness in aerial footage, their obsessive concentration on the appearance representation would inevitably be sensitive to the distractor (i.e., similar object) adjacent to the true one.

2.2. Temporal Context in Visual Tracking

Beyond the straightforward “tracking-by-detection” paradigm, temporal contexts have attracted lots of attention and have been integrated into existing visual tracking frameworks. For instance, STRCF [12] and ARCF [22] introduced spatial temporal regularization in correlation filters to enforce consistency among consecutive frames. For the purpose of efficient convex optimization, they typically focus on penalizing abrupt appearance changes (i.e., smooth constraints) rather than capturing explicit temporal dynamics.

Recent deep-learning-based trackers have attempted to conduct sequence modeling through various strategies. Template update mechanisms, such as UpdateNet [13], employ a dedicated network to predict template updates based on historical tracking results. This approach utilizes cumulative learning, where new templates are computed as weighted combinations of previous ones. However, due to changes in viewpoint or occlusion, visual cues become unreliable, resulting in tracking drift. Memory-augmented tracking methods, such as MemTrack [14] and STMTrack [15], introduce external memory mechanisms. MemTrack maintains memory banks to store historical target representations, enabling the retrieval of past appearances during challenging scenarios. STMTrack extends this concept with space–time memory networks that jointly model spatial and temporal contexts across multiple time scales. They can maintain the target representation during temporary visual interruptions and use memory help to recover from occlusion in UAV scenes. Recent transformer-based trackers, such as TCTrack [16], leverage multi-level temporal convolutions and attention mechanisms to capture long-range dependencies. While these methods demonstrate effectiveness in temporal context modeling, they primarily focus on appearance-based temporal consistency rather than explicit motion–appearance fusion.

The aforementioned trackers motivate us to utilize extra visual cues with high-order constraints against intractable appearance variations.

2.3. Motion Cue in Visual Tracking

Compared with the variable appearance, motion cue can be treated as a more predictable and reliable temporal context for visual tracking. Regretfully, unlike the multi-object tracking task where explicit motion modeling is routine for target association, the single-object trackers rarely fully leverage the motion cue. Conventionally, early efforts employ Kalman or particle filters to represent motion cue, yet are inadequate for complex aerial dynamics. As for single-object tracking, FlowTrack [18] incorporates optical flow as an additional input stream along with RGB data (i.e., raw appearance). More recently, approaches such as TrackLSTM [19] and ATOM [23] have advanced temporal modeling by alternatively employing recurrent networks or Intersection-over-Union (IoU) prediction along the temporal dimension. TrackLSTM demonstrated how LSTMs can implicitly learn complex temporal dependencies from feature sequences, offering improved robustness to short-term occlusions. ATOM, by predicting IoU over time, refined bounding box estimation with a temporal awareness. These methods predominantly perform an implicit motion analysis rather than explicitly exploiting motion as an independent predictive signal.

However, most of the methods treat the motion as an individual cue or auxiliary one, limiting robustness to occlusions or fast motion. The rapid viewpoint changes from UAV maneuvers demand trackers capable of handling drastic appearance variations, where motion-based cues can provide trajectory consistency when appearance cues become unreliable. In scenarios characterized by fast target movement or low frame rates, appearance information between consecutive frames can be scarce or heavily blurred. Motion cues can effectively predict the target’s trajectory and infer its likely position, compensating for sparse appearance data and mitigating the impact of motion blur and rapid ego-motion of the UAV. Such concern inspires us to equally treat motion and appearance as complementary temporal contexts, enabling robust tracking when either one changes.

2.4. Heterogeneous Feature Fusion in Visual Tracking

Heterogeneous representation of the appearance and motion challenges dealing with both cues into one unified framework. Straightforward concatenation [24] or weighted summation [25] often degrade motion contributions in appearance-dominant scenarios. Attention-based fusion, as in SiamAttn [26] and SiamGAT [27], emphasizes spatial contexts but neglects the interaction between motion and appearance. Transformer-based approaches like TransT [28] and STARK [17] only enhance the feature integration module to improve tracking robustness. Additionally, MixFormer [29] through multi-scale feature fusion and spatio-temporal relationship modeling can effectively capture the dynamic changes of the target in long time series. TATrack [30] introduces an attention-based multimodal feature fusion strategy, adaptively adjusting the influence of different modalities. However, they still focus on appearance without dedicated motion representations.

In contrast to previous approaches, we treat appearance and motion as equally important cues and enable them to interact with a bidirectional cross-domain interaction module, where the query, key, and value tuple could be inherently heterogeneous. When appearance becomes unreliable during occlusions, our motion stream can guide target localization. Conversely, when motion becomes ambiguous due to background distractors, appearance features provide discriminative filtering. As such, the bidirectional reasoning mechanism, combined with our adaptive temporal enhancement strategy, enables more robust tracking in challenging aerial scenarios where both appearance and motion may undergo variations.

3. Proposed Method

In this section, we instantiate a Siamese-based aerial tracking framework illustrated in Figure 2. It consists of three modules: high-order temporal-aware representation, heterogeneous feature refinement, and consistent decision fusion with smooth constraint. Each of the modules will be detailed in the following subsections.

3.1. High-Order Temporal-Aware Representation

This module encodes both motion and appearance cues across consecutive frames to lay the foundation for the feature representation of targets. Specifically, by analyzing a short sequence of preceding frames with dual-branch architectures, it could capture high-order temporal contexts on both dynamic motion patterns and appearance variations. Such high-order temporal-aware representation enhances the tracker’s robustness to occlusions, motion blur, and viewpoint variations from the aerial view.

3.1.1. Motion Representation

To capture the target’s motion trajectory, we first represent the motion pattern with the optical flow on the motion branch. Given two consecutive frames $I_{t-1}$ and $I_t$, the Farneback algorithm [31] could induce dense optical flow field at each pixel:

$$\begin{array}{c}\hfill \overrightarrow{v}(x,y)=\mathrm{Farneback}(I_{t-1},I_t)\end{array}$$

(1)

where $\overrightarrow{v}(x,y)$ represents the pixel-wise optical flow vector, capturing the displacement at location $(x,y)$. This fine-resolution motion cue allows the tracker to distinguish the movement of target from background clutter.

Then, to integrate longer-term motion contexts, we construct a motion-history image (MHI) by accumulating recent motion responses:

$$\begin{array}{c}\hfill {\mathrm{MHI}}_t(x,y)=\left\{\begin{array}{cc}\tau \hfill & \mathrm{if}|\overrightarrow{v}(x,y)|>\theta \hfill \\ max(0,{\mathrm{MHI}}_{t-1}(x,y)-\delta )\hfill & \mathrm{otherwise}\hfill \end{array}\right.\end{array}$$

(2)

where $\tau$ represents the current timestamp, $\theta$ is a threshold for identifying noticeable motion, and $\delta$ is the temporal decay factor. This formulation amplifies and attends to recent motion responses while suppressing the historical ones.

The obtained optical flow and MHI are then fed into a conventional convolutional neural network (CNN) to extract high-order temporal-aware motion features $f_{\mathrm{mot}}\in {\mathbb{R}}^{C\times H^{\prime }\times W^{\prime }}$, where C is the number of channels, and $H^{\prime }\times W^{\prime }$ denotes spatial dimensions.

3.1.2. Appearance Representation

In parallel, the appearance branch encodes the target’s spatial layout. To be specific, we define an initial feature map $f_{\mathrm{app}}^0\in {\mathbb{R}}^{C\times H^{\prime }\times W^{\prime }}$ that is extracted using a base convolutional layer to represent target appearance (such as the local texture and structural patterns, etc.) within current frame $I_t$.

To accommodate temporal appearance variations, we utilize a temporal-adaptive convolution (TadaConv) [32] for the processing of the following frame. It modulates the convolution kernel dynamically based on inter-frame differences:

$$\begin{array}{c}\hfill K_t=K_{\mathrm{base}}+{\mathrm{Conv}}_{\mathrm{adapt}}(I_t-I_{t-1})\end{array}$$

(3)

where $K_{\mathrm{base}}$ is a fixed kernel, and ${\mathrm{Conv}}_{\mathrm{adapt}}$ produces a residual correction conditioned on latest appearance change. Thus, the feature map of future frames, $f_{\mathrm{app}}\in {\mathbb{R}}^{C\times H^{\prime }\times W^{\prime }}=K_t\star f_{\mathrm{app}}^0$, remains sensitive to the distribution among consecutive frames such as shadow shifts or pose deformation, enabling reliable appearance-based feature discrimination over time.

3.1.3. Temporal Context-Aware Enhancement

This stage refines the current frame features (i.e., $f_{\mathrm{app}}^t$ and $f_{\mathrm{mot}}^t$, both in ${\mathbb{R}}^{C\times H^{\prime }\times W^{\prime }}$) by incorporating historical cues from the previous L frames. All the cues are collected as $F_{\mathrm{app}}^{\mathrm{history}}=\{f_{\mathrm{app}}^{t-1}, f_{\mathrm{app}}^{t-2}, \dots , f_{\mathrm{app}}^{t-L}\}$ and $F_{\mathrm{mot}}^{\mathrm{history}}=\{f_{\mathrm{mot}}^{t-1}, f_{\mathrm{mot}}^{t-2}, \dots , f_{\mathrm{mot}}^{t-L}\}$, forming the temporal contexts that capture the appearance changes and motion consistency.

To refine the temporal contexts effectively, we employ a multi-head attention mechanism that enables the model to dynamically reweight relevant patterns from the past. Specifically, the enhanced appearance and motion features are computed as follows:

$$\begin{array}{c}\hfill f_{\mathrm{app}}^{*}=\mathrm{MultiHeadAttention}\left(\begin{array}{c}{f}_{\mathrm{app}}^t,F_{\mathrm{app}}^{\mathrm{history}},F_{\mathrm{app}}^{\mathrm{history}}\hfill \end{array}\right)\end{array}$$

(4)

$$\begin{array}{c}\hfill f_{\mathrm{mot}}^{*}=\mathrm{MultiHeadAttention}\left(\begin{array}{c}{f}_{\mathrm{mot}}^t,F_{\mathrm{mot}}^{\mathrm{history}},F_{\mathrm{mot}}^{\mathrm{history}}\hfill \end{array}\right)\end{array}$$

(5)

Additionally, the attention mechanism is defined as follows:

$$\begin{array}{c}\hfill \mathrm{MultiHeadAttention}(Q,K,V)=\mathrm{Concat}({\mathrm{head}}_1, \dots ,{\mathrm{head}}_h)W^O,\end{array}$$

(6)

$$\begin{array}{c}\hfill {\mathrm{head}}_i=\mathrm{Softmax}\left({\displaystyle \frac{\left(Q{W}_i^Q\right){\left(K{W}_i^K\right)}^{\top }}{\sqrt{d_k}}}\right)\left(V{W}_i^V\right)\end{array}$$

(7)

where $d_k=C/h$, and h denotes the number of heads. Each attention head specializes in distinct appearance and motion clues, thereby enriching the current features with long-term contextual semantics.

3.2. Heterogeneous Feature Refinement

Given the high-order temporal context-aware representation of motion and appearance, this module involves the interaction and refinement between the heterogeneous features $f_{\mathrm{app}}^{*}$ and $f_{\mathrm{mot}}^{*}$. Unlike conventional fusion representation that yields single modality output, we are devoted to obtaining two refined representations $f_{\mathrm{mot}}^{\mathrm{refined}}$ and $f_{\mathrm{app}}^{\mathrm{refined}}$ with bidirectional feature interaction, where the motion and appearance features guide each other.

3.2.1. Feature Projection Embedding

Due to the inconsistent feature distributions, it would be intractable to align the heterogeneous appearance and motion features. For the purpose of effective cross-domain interaction, we introduce two learnable linear projection matrices that embed the heterogeneous features into a shared Hilbert space, where the embeddings admit computational equivalence. The embedding process is formulated as

$$\begin{array}{c}\hfill f_{\mathrm{app}}^{\mathrm{proj}}=W_{\mathrm{app}}{f}_{\mathrm{app}}^{*}+b_{\mathrm{app}}\end{array}$$

(8)

$$\begin{array}{c}\hfill f_{\mathrm{mot}}^{\mathrm{proj}}=W_{\mathrm{mot}}{f}_{\mathrm{mot}}^{*}+b_{\mathrm{mot}}\end{array}$$

(9)

where $W_{\mathrm{app}},W_{\mathrm{mot}}\in {\mathbb{R}}^{C^{\prime }\times C}$ denote learnable projection matrices, $b_{\mathrm{app}},b_{\mathrm{mot}}\in {\mathbb{R}}^{C^{\prime }}$ represents bias vectors, and $C^{\prime }$ specifies the dimension of the unified embedding space. The projections $f_{\mathrm{app}}^{\mathrm{proj}},f_{\mathrm{mot}}^{\mathrm{proj}}\in {\mathbb{R}}^{C^{\prime }\times H^{\prime }\times W^{\prime }}$ are subsequently reshaped into sequences ${\mathbb{R}}^{\left(H^{\prime }{W}^{\prime }\right)\times C^{\prime }}$ for the following bidirectional cross-domain interaction refinement.

3.2.2. Bidirectional Cross-Domain Interaction

We employ a bidirectional cross-domain interaction that allows motion features $f_{\mathrm{mot}}^{\mathrm{proj}}$ and appearance features $f_{\mathrm{app}}^{\mathrm{proj}}$ to guide each other within a unified embedding space. Specific attention mapping, as illustrated in Figure 3, consists of two parallel branches: appearance-guided motion refinement and motion-guided appearance refinement.

Appearance-Guided Motion Refinement. In this branch, we treat the appearance projections $f_{\mathrm{app}}^{\mathrm{proj}}$ as the query (i.e., Q) and the motion projections $f_{\mathrm{mot}}^{\mathrm{proj}}$ as key and value (i.e., K and V), enabling appearance to guide the refinement of motion:

$$f_{\mathrm{mot}}^{\mathrm{refined}}=\mathrm{Attention}(Q=f_{\mathrm{app}}^{\mathrm{proj}},K=f_{\mathrm{mot}}^{\mathrm{proj}},V=f_{\mathrm{mot}}^{\mathrm{proj}})$$

(10)

where the attention function is defined as

$$\begin{array}{c}\hfill \mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{C^{\prime }}}}\right)V\end{array}$$

(11)

Equation (10) compares each appearance feature vector with all motion features, identifying regions where semantic contents and motion patterns are most correlated. Unlike weighted sum or concatenation, the cross-attention mechanism introduces a learnable alignment between the feature spaces. Appearance features, being structurally stable and semantically expressive, provide spatial priors that rectify the attention toward motion patterns consistent with the target’s visual consistency.

Motion-Guided Appearance Refinement. Conversely, we designate $f_{\mathrm{mot}}^{\mathrm{proj}}$ as the query (i.e., Q) and $f_{\mathrm{app}}^{\mathrm{proj}}$ as key and value (i.e., K and V), allowing motion features to guide appearance ones:

$$\begin{array}{c}\hfill f_{\mathrm{app}}^{\mathrm{refined}}=\mathrm{Attention}(Q=f_{\mathrm{mot}}^{\mathrm{proj}},K=f_{\mathrm{app}}^{\mathrm{proj}},V=f_{\mathrm{app}}^{\mathrm{proj}})\end{array}$$

(12)

Since motion captures the continuity of object trajectories even under changing viewpoints or lighting conditions, it provides robust guidance for target association. Through mapping motion queries to appearance regions, the model could suppress visually misleading features (e.g., clutter or distractors) and attend to the trajectory-consistent targets.

3.3. Consistent Decision Fusion with Smooth Constraint

This decision-making module dynamically integrates the refined feature representation of motion and appearance under smooth optimization constraint so as to avoid unreasonable decision switches between the two heterogeneous visual experts. It follows a three-stage pipeline: dynamic feature gating for context-aware feature selection, adaptive decision fusion for temporal consistency, and failure-aware recovery against disruptions (e.g., long-term occlusions and interruptions), as shown in Figure 4.

3.3.1. Dynamic Feature Gating

To dynamically judge the appearance and motion cues in UAV tracking, we introduce a gating network that adaptively modulates the contributions of appearance features $f_{\mathrm{app}}^{\mathrm{refined}}$ and motion features $f_{\mathrm{mot}}^{\mathrm{refined}}$. The gating is conditioned on two factors: normalized target velocity ${\mathcal{V}}_t={\displaystyle \frac{|p_t-p_{t-1}|}{\Delta t}}$ and the appearance confidence ${\mathcal{C}}_t=max\left(S_{\mathrm{init}}\right)$. Then, the induced gating weights, ${\alpha }_t$ and ${\beta }_t$, are expressed as

$$\begin{array}{cc}\hfill g_t^{\mathrm{mot}}& =\sigma \left(W_m[{\mathcal{V}}_t, \nabla {\mathcal{V}}_t, {\mathcal{C}}_{t-1}]+b_m\right)\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill g_t^{\mathrm{app}}& =\sigma \left(W_a[{\mathcal{C}}_t, \nabla {\mathcal{C}}_t, {\mathcal{V}}_{t-1}]+b_a\right)\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\alpha }_t& ={\displaystyle \frac{exp\left(g_t^{\mathrm{app}}\right)}{exp\left(g_t^{\mathrm{app}}\right)+exp\left(g_t^{\mathrm{mot}}\right)}}\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill {\beta }_t& =1-{\alpha }_t\hfill \end{array}$$

(16)

where $\sigma (·)$ is the sigmoid activation, learnable linear weights are $W_m$, $W_a$, $b_m$, $b_a$, and ∇ denotes the temporal gradient. The proposed dynamic feature gating mechanism offers three core advantages:

• Robust context-driven feature selection. The motion gating response is more sensitive to short-term velocity changes, enabling swift adaptation to rapid movements, while the appearance gating favors stable scenes, ensuring visual consistency when motion is less informative.
• Temporal smoothness. Incorporating temporal gradients suppresses erratic cue shifts when the video contents change slowly (e.g., cruise or hover mode).
• Historical complementarity optimization. Leveraging historical confidence and velocity weighting, enabling the model to offset short-term degradation in either stream by relying on the complementary one.

3.3.2. Smooth Decision Fusion

Sudden fluctuation in the gating weights often results in erratic predictions of target states. Here, we introduce an adaptive decision fusion strategy that constrains the positions within physically plausible trajectories.

More specifically, the fused representation at time t integrates both current modality-specific features and historical fused ones:

$$f_{\mathrm{fused}}^t={\alpha }_t{f}_{\mathrm{mot}}^{\mathrm{refined}}+{\beta }_t{f}_{\mathrm{app}}^{\mathrm{refined}}+\sum _{k=1}^K{\gamma }_k{f}_{\mathrm{fused}}^{t-k}$$

(17)

where ${\gamma }_k=\lambda e^{-k/\tau }$ (with $\lambda =0.5$, $\tau =3$) controls the temporal decay rate, assigning greater weight to latest frames while gradually attenuating the influence of historical ones, and the fused features $f_{\mathrm{fused}}^t$ are utilized to generate the target response map $S_t\left(p\right)$, which encodes spatial confidence scores for candidate locations.

To stabilize the predicted position, we formulate tracking as a regularized optimization problem. At each timestep, the predicted position $p_t$ is obtained by solving the following:

$$p_t=arg\underset{p}{max}\left(\sum _{i=0}^{N-1}{w}_i{S}_{t-i}\left(p\right)-\eta {\parallel p-p_{t-1}\parallel }_2^2\right)$$

(18)

where $S_{t-i}\left(p\right)$ denotes the response map at time $t-i$, and the weights $w_i=2^{-i}/\sum 2^{-k}$ favor recent frames. Additionally, the spatial regularization term $\eta \parallel p-p_{t-1}{\parallel }_2^2$ suppresses abrupt positional variation.

3.3.3. Failure-Aware Recovery

In the case of drastic appearance variations, we introduce two motion-guided strategies to admit continuing tracking:

• Confidence-Aware Fallback: If the appearance cue is unreliable (e.g., ${\mathcal{C}}_t<0.4$), we preserve the gating weight from the previous frame (${\alpha }_t\leftarrow {\alpha }_{t-1}$) and estimate the target’s position using a short-term average of recent velocities, $p_t=p_{t-1}+{\displaystyle \frac{1}{3}}{\sum }_{i=1}^3{\mathcal{V}}_{t-i}$. This strategy assumes that the target continues along a consistent path over a short period, effectively bridging visual disruptions.
• Velocity-Guided Prediction: If the appearance confidence drops below 0.2 (e.g., severe occlusion), we update the position based on extrapolated motion using $p_t=p_{t-1}+{\widehat{\mathcal{V}}}_t$. Here, ${\widehat{\mathcal{V}}}_t$ is a weighted average of the past three velocities, computed as $\mathrm{EMA}({\mathcal{V}}_{t-3:t-1},\alpha =0.9)$. This method extends the motion trend to estimate the target’s location when the appearance cue is unavailable.

4. Experimental Results

In this section, we report the details of our experiment. To fully prove the effectiveness of our aerial tracker, we evaluate our method on four well-established aerial tracking benchmarks, i.e., UAV123 [33], UAV20L [33], UAV123@10fps [33], and DTB70 [34].

4.1. Dataset

UAV123 [33] consists of 123 sequences collected under diverse real-world conditions, including urban roads, industrial sites, highways, sports fields and rural areas, as shown in Figure 5a. It captures diverse challenges including lighting fluctuations, abrupt target motion, partial occlusion, object shape deformation, complex background, and size transformation. It features a wide variety of targets such as pedestrians, vehicles, cyclists, and groups and is representative of common UAV applications like surveillance, traffic monitoring, and infrastructure inspection.

UAV123@10fps [33] contains 123 aerial video sequences captured by UAV-mounted cameras at 10 frames per second (FPS), simulating real-world motion and processing constraints. It covers a wide range of scenarios including urban, rural, and aquatic environments and features various target types such as vehicles and pedestrians, as shown in Figure 5b. With over 113K frames annotated with bounding boxes and attribute tags, UAV123@10fps provides a challenging benchmark for evaluating tracker robustness under reduced frame rates and energy-efficient operational requirements.

UAV20L [33] consists of 20 long-duration UAV-captured sequences, each averaging nearly 3000 frames, with the longest exceeding 5000 frames. It covers a wide range of scenarios including expansive forests, rugged mountains, coastal regions, agricultural lands, and urban outskirts, as shown in Figure 5c. As the sequence length increases, challenges such as object deformation, scale variation, illumination changes, and viewpoint shifts become more pronounced. It focuses on long-term tracking challenges such as persistent occlusion, target reappearance, and background clutter, simulating real-world missions that require sustained attention over extended durations.

DTB70 [34] is a drone-tracking benchmark with 70 sequences capturing vehicles, pedestrians, and animals in diverse scenes, as shown in Figure 5d. It includes 41,000 annotated frames and poses challenges like fast motion, occlusion, and camera viewpoint changes, making it suitable for assessing tracker robustness under UAV-specific constraints such as motion blur and real-time processing. It emphasizes environmental and object diversity. It includes scenarios involving marine vessels, animals, and vehicles, captured across waterfronts, farmlands, urban intersections, and more dynamic outdoor scenes.

4.2. Evaluation Metrics

The aerial tracker is assessed by a one-pass evaluation across all the previously mentioned datasets, with precision and success rate as the primary evaluation criteria. The precision plot presents the percentage of bounding boxes whose center location error (CLE) is less than a given threshold. CLE is used to measure the Euclidean distance between the estimated object location and the ground-truth bounding box center. Additionally, the success rate is employed to measure the IoU of the estimated area and the ground-truth bounding box. The success plot indicates the ratio of the number of frames whose IoU is greater than the given threshold to the total number of frames. The area under the curve (AUC) is applied to rank the success rate of trackers.

4.3. Implementation Details

Our framework is implemented based on a temporally enhanced AlexNet backbone [35]. It integrates dynamic temporal convolutions (kernel size = (3, 1, 1)) for adaptive feature calibration and a motion branch with cascaded convolutions (11 × 11→5 × 5→3 × 3) to generate 384-channel motion descriptors. The training process employs a three-stage curriculum spanning 100 epochs on VID [36], LaSOT [37], and GOT-10K [38] datasets. These datasets are chosen for their complementary characteristics that align with UAV tracking requirements. VID provides diverse object categories with temporal annotations, essential for learning robust appearance representations across different target types commonly encountered in aerial footage (vehicles, persons, animals, and objects). LaSOT offers long-term tracking scenarios (average 2500 frames per sequence) that are crucial for learning motion pattern consistency and temporal context modeling required for extended UAV missions. GOT-10K ensures cross-category generalization through its one-shot evaluation protocol, preventing overfitting to specific object classes and enabling robust performance on novel targets encountered in real-world UAV applications. This learning strategy progressively builds from basic appearance discrimination (VID) through temporal consistency modeling (LaSOT) to general robustness (GOT-10K), aligning with the hierarchical nature of our heterogeneous feature fusion architecture.

We initialize learning rates at 0.005 with log-space decay to 0.00005, using SGD optimizer (momentum = 0.9, weight_decay = 0.0001) and batch size 32 distributed across two NVIDIA GeForce RTX2080Ti GPUs. In the first 10 epochs, we freeze the first three convolutional blocks, and training is focused on the motion head with a higher learning rate ($5\times {10}^{-3}$). Subsequently, the core temporal adaptation layers are then unfrozen with layer-specific learning rate ratios, introducing temporal dropout ($p=0.1$) and flow perturbation ($\mathcal{N}(0,0.1)$) to enhance temporal robustness.

4.4. Quantitative Evaluation

To evaluate the effectiveness of the proposed aerial tracker, we conducted extensive experiments on four UAV benchmarks: UAV123, UAV20L, UAV123@10fps, and DTB70. A total of 38 state-of-the-art (SOTA) trackers were included for fair comparison, covering spatio-temporal context methods (e.g., TCTrack [16] and DeepSTRCF [12]), Siamese-based methods (e.g., SiamAPN++ [11] and SiamDW [39]), DCF-based approaches (e.g., SRDCF [9] and KCF [8]), feature fusion trackers (e.g., ECO_gpu [6] and CCOT [40]), and motion-aware models (e.g., LCT [41] and ARCF [22]). The results are obtained by re-implementing the official codes with their corresponding hyperparameters.

As shown in Figure 6, our method consistently outperforms all compared trackers, achieving precision scores of 0.812, 0.737, 0.787, and 0.828 and success rates of 0.616, 0.552, 0.601, and 0.629 on the UAV123, UAV20L, UAV123@10fps, and DTB70 benchmarks, respectively. To provide comprehensive quantitative comparison,

Table 1. Comparison of precision and success rates on DTB70 benchmark. Results presented in bold indicate superior performance.

Trackers	Succ.	Prec.	Trackers	Succ.	Prec.
Ours	0.629	0.828	UDT [42]	0.422	0.605
TCTrack [16]	0.621	0.815	ARCF_H [22]	0.416	0.618
SiamAPN++ [11]	0.594	0.792	CF2 [43]	0.415	0.645
SiamRPN [44]	0.586	0.794	MCCT_H [45]	0.405	0.606
SiamAPN [46]	0.585	0.786	BACF [7]	0.402	0.601
CCOT [40]	0.517	0.771	CoKCF [47]	0.378	0.592
DeepS0TRCF [12]	0.505	0.736	SRDCF [9]	0.363	0.707
ECO_gpu [6]	0.502	0.724	fDSST [48]	0.357	0.536
SiamDW [39]	0.492	0.727	Staple_CA [49]	0.351	0.535
MCCT [45]	0.484	0.707	SRDCFdecon [50]	0.351	0.506
AutoTrack [51]	0.478	0.698	SAMF_CA [52]	0.346	0.609
DASiamRPN [53]	0.474	0.684	SAMF [52]	0.340	0.524
ARCF [22]	0.472	0.671	CN [54]	0.315	0.514
UDTplus [42]	0.462	0.665	KCC [55]	0.291	0.442
IBCCF [56]	0.460	0.671	LCT [41]	0.282	0.464
TADT [57]	0.459	0.691	KCF [8]	0.280	0.470
ECO_HC [6]	0.453	0.645	DCF [58]	0.280	0.469
CSRDCF [58]	0.438	0.648	DSST [48]	0.276	0.521
STRCF [12]	0.437	0.651	Staple [49]	0.265	0.368
MCPF [59]	0.432	0.666

presents detailed precision and success rates for all compared methods on the DTB70 benchmark. These results outperform all competing methods, demonstrating the effectiveness of our high-order temporal context-aware representation, which equally encodes motion patterns and appearance features within a unified architecture. This representation enables robust target perception under diverse aerial scenarios, effectively mitigating challenges such as occlusion, motion blur, temporal discontinuity, and background interference and significantly enhancing tracking stability and accuracy.

To further evaluate the robustness of our tracker under challenging conditions, we conduct additional tests on four representative scenarios selected from the UAV20L dataset, namely, full occlusion, partial occlusion, background clutter, and out-of-view. As shown in Figure 7, our method consistently achieves the best performance in both success rate and precision across all cases. Specifically, in full occlusion scenarios, our method achieves a 0.353 success rate compared to SiamAPN++’s 0.344, demonstrating the effectiveness of motion-guided recovery when appearance cues are completely unavailable. For partial occlusion, we maintain a 0.543 success rate with 0.735 precision, outperforming appearance-only methods like SiamAPN++’s 0.506 success rate. The fusion of high-order motion patterns with spatial features allows the tracker to infer plausible target positions even in the absence of reliable visual observations. Additionally, in more complex conditions where both appearance and motion cues are compromised, such as background clutter and out-of-view cases, our approach still exhibits superior performance compared to other trackers. In background clutter scenarios, our approach still achieves a 0.334 success rate, surpassing TCTrack by 0.053. For out-of-view cases, our failure-aware recovery enables 0.593 success rate. This is owing to the fact that our appearance–motion fusion mechanism enables the model to maintain robustness to tracking despite severe distractors or sudden disappearance of the target.

4.5. Qualitative Evaluation

As illustrated in Figure 8, we conduct a qualitative comparison between our aerial tracker and other leading methods on the UAV123 dataset.

To illustrate the effectiveness of our method under challenging tracking conditions, such as complex background, frequent short-term occlusions, and target coverage by other objects, we, respectively, select (a) Bike3, (b) Car9, and (c) Group2 as representative sequences. One can see that our method demonstrates robust tracking performance across these scenarios.

In the Bike3 sequence, the bicycle moves through a cluttered scene with pedestrians, poles, and other similar objects. These distractions could mislead appearance-based aerial trackers. Our method leverages MHIs to capture the target motion patterns, allowing us to distinguish the moving target from static distractions. As shown in Figure 8a, our tracker accurately tracks the target, while others drift toward the background.

In the Car9 sequence, the car is frequently occluded by billboards and obstacles. Our velocity-adaptive gating increases the influence of motion features during occlusion, using recent velocity to estimate position. Additionally, the spatio-temporal refinement ensures smooth tracking. As shown in Figure 8b, our tracker maintains accurate localization, avoiding the drift seen in appearance-only methods.

In the Group2 sequence, the person is partially hidden by branches, leaving limited appearance cues. Our method addresses this by refining motion and appearance features through bidirectional attention. As shown in Figure 8c, our tracker accurately recovers the bounding box, avoiding drift and scale errors common in appearance-based methods.

4.6. Ablation and Hyperparameter Sensitivity Study

To validate the effectiveness of each component in our proposed tracking framework, we conduct the ablation studies and hyperparameter sensitivity study on the UAV123, UAV20L, and DTB70 benchmarks. Specific analysis consists of four aspects: (1) component-wise validation of the overall results, (2) sensitivity analysis of MHI parameters, (3) exploration of cross-attention head numbers, and (4) evaluation of different smooth constraint parameters.

4.6.1. Component Ablation of Overall Framework

We analyze the contributions of (1) motion feature and MHI, (2) bidirectional cross-attention mechanisms, and (3) smooth constraint. Starting with a baseline model (appearance-only Siamese tracker), we progressively incorporate our proposed modules to quantify their individual and combined effects.

Table 2. Ablation studies on major components of our framework on UAV123 and DTB70 datasets. Results presented in bold indicate superior performance.

Components			UAV123		DTB70
Motion Feature Extraction	Heterogeneous Feature Refinement	Consistent Decision Fusion	Succ.	Prec.	Succ.	Prec.
×	×	×	0.575	0.765	0.586	0.794
✓	×	×	0.591	0.783	0.601	0.809
✓	✓	×	0.607	0.801	0.614	0.816
✓	✓	✓	0.616	0.812	0.629	0.828

presents the results on UAV123 and DTB70 datasets. Adding our motion feature stream with MHI-based temporal encoding improves success rates by 0.016 and 0.015 on UAV123 and DTB70, respectively, highlighting the importance of explicit motion modeling. The bidirectional cross-attention mechanism further enhances performance by facilitating effective information exchange between appearance and motion streams. Finally, the spatio-temporal refinement module contributes an additional gain of 0.009 and 0.015 in success rates, demonstrating the importance of trajectory consistency constraints.

4.6.2. Sensitivity to MHI Parameters

Then, we investigate different configurations of our motion feature extraction module, specifically analyzing the impact of MHI parameters and temporal window size.

Table 3. Performance comparison with different MHI parameters under challenging scenarios on UAV123 dataset. Results presented in bold indicate superior performance.

Configuration	Fast Motion		Occlusion		Overall
	Succ.	Prec.	Succ.	Prec.	Succ.	Prec.
MHI ($\tau$ = 0, L = 3)	0.459	0.665	0.370	0.595	0.594	0.785
MHI ($\tau$ = 1, L = 3)	0.537	0.752	0.438	0.687	0.602	0.796
MHI ($\tau$ = 3, L = 3)	0.573	0.786	0.527	0.735	0.616	0.812
MHI ($\tau$ = 5, L = 3)	0.551	0.761	0.489	0.702	0.606	0.801
MHI ($\tau$ = 3, L = 5)	0.565	0.773	0.503	0.718	0.609	0.805

shows the effects of different decay parameters ($\tau$) and temporal windows (L) on tracking performance under challenging scenarios.

The results demonstrate that motion-history encoding significantly improves tracking performance in challenging scenarios, particularly for fast motion (+0.114) and occlusion (+0.157) when using optimal parameters ($\tau$ = 3, L = 3). A decay factor of $\tau$ = 3 achieves the best balance between short-term responsiveness and long-term motion pattern preservation. Increasing the temporal window to L = 5 slightly decreases performance, likely due to the introduction of irrelevant historical motion when the direction of target movement changes.

4.6.3. Sensitivity to Cross-Attention Head Number

Additionally, we evaluate different bidirectional cross-domain interaction configurations in our bidirectional fusion module to understand how appearance and motion features should optimally interact.

Table 4. Different configurations of cross-attention head numbers under challenging scenarios on UAV123 dataset. Results presented in bold indicate superior performance.

Head Number	Background Clutter		Viewpoint Change		Overall
	Succ.	Prec.	Succ.	Prec.	Succ.	Prec.
4 heads	0.327	0.551	0.605	0.758	0.601	0.807
8 heads	0.398	0.581	0.632	0.821	0.616	0.812
12 heads	0.362	0.563	0.617	0.793	0.603	0.804

shows that $h=8$ achieves best performance in complex background and viewpoint changes.

4.6.4. Sensitivity to Smooth Constraint Parameters

To analyze the impact of different parameters in our smooth constraint module during long-term tracking, we specifically conducted experiments on UAV20L, focusing on the exponential decay coefficient ($\lambda$) and regularization strength ($\eta$) for trajectory smoothness constraints. The results in

Table 5. Experiment results of different smooth constraint parameters with challenging scenes on UAV20L dataset. Results presented in bold indicate superior performance.

Parameters	Occlusion		Motion Blur		Overall
	Succ.	Prec.	Succ.	Prec.	Succ.	Prec.
$\lambda$ = 0, $\eta$ = 0	0.446	0.633	0.500	0.710	0.519	0.703
$\lambda$ = 0.3, $\eta$ = 0.7	0.492	0.687	0.537	0.761	0.535	0.728
$\lambda$ = 0.5, $\eta$ = 0.7	0.537	0.737	0.557	0.821	0.552	0.737
$\lambda$ = 0.7, $\eta$ = 0.7	0.514	0.713	0.525	0.783	0.539	0.721
$\lambda$ = 0.5, $\eta$ = 0.5	0.509	0.706	0.542	0.804	0.547	0.731
$\lambda$ = 0.5, $\eta$ = 0.9	0.493	0.682	0.507	0.771	0.523	0.718

demonstrate how these parameters affect tracking performance in long-term tracking scenarios.

The results confirm that the constraints significantly enhance tracking robustness, particularly in challenging scenarios like occlusion recovery (+0.091) and motion blur (+0.057). The optimal configuration uses $\lambda$ = 0.5 for temporal decay and $\eta$ = 0.7 for spatial regularization, balancing historical trajectory information with current frame observations. Excessive regularization ($\eta$ = 0.9) reduces adaptability to sudden motion changes, while insufficient constraints ($\eta$ = 0.5) provide inadequate stability during occlusions.

5. Discussion

In this paper, we propose a high-order temporal context-aware tracker that incorporates a failure-aware recovery mechanism to alleviate common tracking failures. However, certain scenarios still pose considerable challenges to the failure-aware recovery mechanism. One such case arises when multiple similar-looking objects move at comparable speeds (e.g., vehicles in traffic). As seen in Figure 9a, both motion and appearance cues become ambiguous, making it difficult to maintain identity consistency. Since our velocity-based prediction relies on the assumption of consistent motion patterns, it may fail to distinguish between targets when such cues are unreliable or conflicting. Another limitation occurs under prolonged or complete occlusion. Without updated visual feedback, motion-based predictions rely solely on outdated cues. Over time, small errors accumulate, leading to temporal drift and unreliable recovery, especially when the target reappears at a position far from the predicted trajectory. This limitation is illustrated in Figure 9b, where the predicted position gradually drifts and fails to reconnect with the actual target once it re-emerges. These cases underscore the limitations of relying solely on local motion history. To address such challenges, future work will explore more robust long-term recovery strategies, such as incorporating global re-detection or re-identification modules.

In addition, the high-order temporal context-aware tracker, by integrating heterogeneous visual cues (appearance and motion) and temporal context-aware representation, provides a robust foundation for modeling dynamic targets with varying appearances in challenging aerial environments. Our design has enabled us to achieve highly effective single-target tracking (SOT), consistently maintaining target localization and minimizing drift amidst complex factors like viewpoint changes, occlusions, and rapid motion. However, the current framework is specifically tailored for SOT and lacks the fundamental components required for multiple object tracking (MOT). Extending our method to MOT directly presents significant challenges as it not only necessitates sophisticated spatial reasoning across multiple objects but also demands robust mechanisms for maintaining inter-frame identity consistency and effectively handling complex inter-target occlusions and interactions. Future work will, therefore, focus on adapting our core modules within a comprehensive MOT framework, potentially incorporating tracking-by-detection paradigms or graph-based association models to address these crucial multiple object complexities.

Lastly, we acknowledge that our tracker also introduces a non-negligible computational burden. While our aerial tracker achieves promising performance in challenging scenarios, it results in a computational cost of 24.6GFLOPs, exceeding the leading trackers such as SiamDW (12.9GFLOPs) [39] and SiamGAT (17.28GFLOPs) [27]. This complexity primarily stems from the additional optical flow computation and the convolutional operations involved in heterogeneous feature fusion. Nevertheless, all the computational cost of the aforementioned trackers remains within the same order of magnitude. Additionally, the computational cost of our method remains accommodated by the edge computing modules. Taking NVIDIA Jetson AGX Orin as an example, its FP32 computing throughput reaches 5.3 TFLOPs, which is two orders of magnitude higher than our model’s FLOPs. In other words, the proposed method consumes only 0.5 percent of available computation resources, leaving a substantial margin for a reliable implementation. In summary, we think this computational burden is worth it for real-time applications. Future work will focus on further reducing the computational burden for deployment on more resource-constrained aerial platforms without sacrificing accuracy.

6. Conclusions

In this work, we proposed a high-order temporal context-aware tracker to address the instability of visual tracking in UAV platform, particularly under challenging conditions such as cluttered background, severe occlusions, and drastic appearance variations. Specifically, we equally encoded appearance and motion cues and leveraged motion-history images to learn high-order temporal representations. Additionally, a bidirectional cross-attention mechanism is introduced to refine these heterogeneous features, yielding cohesive and complementary representation. Then, a velocity-based gating strategy, coupled with consistent inference with smooth spatial–temporal, was designed to guide decision fusion and ensure stable location. Finally, we conduct extensive experiments on four UAV benchmarks to validate the effectiveness and robustness of the proposed method.