A Target Tracking Method Based on Frequency and Spatial Information Perception in UAV Vision

Highlights

What are the main findings?

• We proposed a Transformer-based UAV target tracking framework (FSTrack) that jointly exploits frequency-domain and spatio-temporal information to improve target representation and localization under challenging conditions such as illumination variation, occlusion, and background interference.
• A frequency-spatial enhancement module and a spatio-temporal feature fusion module are introduced to capture complementary target characteristics across frames, resulting in improved tracking robustness and accuracy within a fully end-to-end architecture.

What are the implications of the main findings?

• The findings demonstrate that integrating frequency-domain cues with spatial and temporal representations provides a more discriminative and robust tracking strategy for UAV vision in complex real-world environments.
• The proposed end-to-end design reduces reliance on post-processing and simplifies the conventional tracking pipeline, making FSTrack more suitable for practical real-time UAV tracking applications.

Abstract

Target tracking for Unmanned Aerial Vehicles (UAVs) can be significantly impacted by environmental factors such as lighting variations, background clutter, and target occlusion. To address these challenges, we developed a target tracking method that integrates both frequency-domain and spatial perception capabilities in UAV vision (FSTrack). Specifically: (1) we utilized the Swin Transformer as the core network to extract features from both the template and search images; (2) we introduced a Transformer-based module to enhance both frequency and spatial information, improving tracking accuracy under varying illumination conditions; (3) we designed a spatio-temporal feature fusion module with multiple multi-head self-attention mechanisms to precisely model the tracking state, thus increasing reliability in cluttered and occluded environments; and (4) we created a hybrid loss function to boost accuracy in both classification and regression tasks. Our experimental results on the UAV123, DTB70, and UAVDT datasets show that our approach not only surpasses current state-of-the-art methods in success rates and precision but also operates more swiftly.

1. Introduction

Recently, Unmanned Aerial Vehicles (UAVs) are widely utilized in both military and civilian domains due to their compact size, ease of control, and remarkable maneuverability, making them well-suited for tasks such as surveillance, mapping, and delivery. Advances in computer vision have facilitated the widespread use of UAV visual tracking technologies across diverse fields, including security reconnaissance [1,2], disaster detection [3], and search and rescue missions [4]. As a vital aspect of UAV operations, target tracking algorithms have attracted considerable attention from the computer vision research community. Unlike ground-based platforms, aerial video from UAVs often faces obstacles such as varying viewing angles, low resolution, significant occlusion, motion blur, and scale changes, all of which can significantly challenge the target tracking process.

Target tracking methods can be generally divided into two primary groups: correlation filtering and deep learning approaches. Correlation filtering techniques expedite the tracking process by transferring the convolution operation from the spatial to the frequency domain, as demonstrated in various studies [5,6,7,8,9,10,11,12,13,14,15,16,17]. Bolme and colleagues [5] initiated the use of correlation filters in tracking by employing the MOSSE strategy, which is known for its high-speed tracking capability of over 600 frames per second. Henriques et al. [6] furthered this method’s efficiency by integrating multiple feature channels into the kernel correlation filter, thus augmenting its performance without sacrificing speed. Lin et al. [9] presented the BiCF, a filter that tackles tracking issues related to target and background changes by using a bidirectional incongruity error mechanism. Xu et al. [12] created an ACS-DCF algorithm that bolsters the tracking robustness by prioritizing informative channels and downplaying the impact of less useful ones. Huang et al. [14] introduced an innovative technique for learning correlation filters that are discriminative for both visual and thermal infrared spectrums, also utilizing adaptive spatial regularization to reduce boundary artifacts and increase tracking velocity. However, even with their swift operation, these methods may still struggle with maintaining precision under fluctuating lighting and in the presence of complex backgrounds.

Recently, deep learning-based trackers have become prominent in UAV target tracking due to their ability to efficiently learn and compare feature representations [18,19,20,21,22,23]. The fully convolutional Siamese network (SiamFC) algorithm, introduced by Bertinetto et al. [18], maps both target templates and search regions into a common feature space, allowing the network to learn their similarities for precise tracking. SiamFC has demonstrated strong performance across various UAV tracking benchmarks. Subsequent improvements, such as SiamFC++ [24], SiamBAN [25], SiamCAR [26], and the Siamese graph attention network [20], have further enhanced the robustness of these trackers. Recently, Transformers [21] have shown potential in capturing global spatio-temporal correlations in sequential data, making them suitable for target tracking applications. TCTrack [22] utilizes Transformers to effectively encode spatio-temporal information, using contextual data to accurately localize targets. Similarly, Yan et al. [27] applied Transformers to combine spatial and temporal information, creating discriminative spatio-temporal features for precise target localization. However, these advanced algorithms often involve complex mechanisms for managing spatio-temporal information, which can impact tracking speed. Additionally, the spatio-temporal data produced by these methods can become problematic when targets are blurred due to changes in ambient lighting, affecting the tracker’s robustness.

To enhance target tracking accuracy, especially under varying lighting conditions and complex backgrounds, this paper introduces FSTrack, a target tracking method that integrates both frequency-domain and spatial perception capabilities in UAV vision. Figure 1 illustrates a summary of the performance comparison between FSTrack and other related methods. The key contributions of this work are as follows.

Figure 1. Suc-Pre-FPS comparisons with state-of-the-arts methods on UAVDT [28]. The horizontal axis is FPS, the vertical axis is Suc, and the radius of circle is Pre. Our FSTrack achieves 66.2% Suc, 81.4% Pre on UAVDT with 56.9 FPS running speed.

• To address challenges in target representation during tracking, such as occlusion, illumination changes, and background interference, we developed a Transformer-based tracking method. This approach enhances tracking performance by integrating both frequency information and spatio-temporal data of the target.
• We introduced a frequency and spatial information enhancement module that captures and combines the target’s frequency and spatial features. This module improves the accuracy of target feature representation and localization, particularly in conditions with variable lighting.
• We developed a spatio-temporal feature fusion module that utilizes multiple multi-head self-attention mechanisms to integrate the target’s spatio-temporal state information. This design effectively models the tracking state, improving accuracy in challenging scenarios with background interference and occlusion.
• The proposed method operates in a fully end-to-end network, eliminating the need for post-processing steps and significantly simplifying the traditional tracking pipeline.

The subsequent sections of this paper are organized as follows: The Section 2 conducts an exhaustive examination of preceding scholarly work pertinent to our study. Moving forward, the Section 3 elaborates on the intricacies of the novel tracking approach introduced in this paper. Subsequently, the Section 4 showcases the outcomes of our experiments, complemented by a thorough analytical discourse. Concluding the paper, Section 5 encapsulates the principal discoveries and offers final thoughts on the research presented.

2. Related Work

2.1. Single Target Tracking

The core goal of visual tracking is the precise identification of a target as it moves through a sequence of frames, starting with its initial delineation within a bounding box [29]. Tracking methods that are founded on deep learning can be broadly divided into those that are based on Convolutional Neural Networks (CNNs) and those that employ Transformer architectures. For instance, CNN-based tracking systems like SiamFC [18] and SiamRPN [19], capitalize on CNNs to examine the correlation between the features of a target’s template and those of the broader search region, thus streamlining the process of pinpointing the target. On the other hand, techniques such as DiMP [30] and PrDiMP [31] are designed to boost the exactness and sturdiness of tracking by acquiring a kernel that filters out the target in a manner that is discerning. Nonetheless, CNNs can find it challenging to sustain connections over longer timeframes and broader areas within the spatial and temporal realms, potentially impacting the consistency of tracking accuracy throughout longer image sequences.

The latest strides in the field of computer vision, especially with the advent of the Transformer model [32], have markedly enhanced tracking techniques that are based on the Transformer. These approaches, which include both feature-discriminative and Siamese-style trackers, frequently outperform their CNN counterparts by capturing extensive interdependencies and recognizing targets with higher precision. Pioneering efforts like TransT [33] integrated CNN-based feature extraction with unique attention mechanisms, adeptly modeling the associations between the features of target templates and those of the search regions. In a similar vein, TrDiMP [34] employed Transformers to craft a comprehensive, multi-frame target filtering representation, thus refining tracking proficiency. Wang and colleagues [35] proposed a network that fuses features across different scales, embracing both box attention (concentrating on the features within the bounding box) and instance attention (zeroing in on particular instances), all within the Transformer structure. This strategy more adeptly uses extensive contextual data to counteract the challenges of getting stuck in local optima.

At present, tracking systems that are exclusively founded on the Transformer framework are increasingly becoming popular. Techniques like OSTrack [36], Mixformer [37], and SimTrack [38] leverage conventional visual Transformer models to bolster the learning of features and the mapping of relationships. These strategies have demonstrated notable progress compared to previous methods that were based on the Transformer architecture.

2.2. Transformer in Computer Vision

The Transformer model, pioneered by Vaswani and colleagues [21] in the domain of machine translation, was crafted to address translation challenges. It achieves this by identifying interdependencies among all input elements and synthesizing a comprehensive view of their representations. A central component of the Transformer is its attention mechanism, which facilitates the model’s capability to discern connections between inputs and amalgamate insights across the entire sequence. The self-attention process merges spatial elements into a cohesive representation.

$$\text{Self-Atten}(\mathit{Q},\mathbf{K},\mathit{V})=\mathrm{Softmax}\left({\displaystyle \frac{{\mathit{QK}}^T}{\sqrt{d_k}}}\right)\mathit{V}$$

(1)

where $\mathit{Q}$, $\mathit{K}$, and $\mathit{V}$ denote the query, key, and value, respectively, and $d_k$ indicates the dimension of K. Equation (1) outlines the essential computation for determining the attention score of an individual image token. The single-head self-attention layer is designed to concentrate its analysis on one specific element, whereas the multi-head self-attention (MHSA) mechanism expands on this by enhancing the model’s capacity to simultaneously detect and process multiple relational dynamics present within the dataset. The operational dynamics of MHSA are as detailed below:

$$\mathrm{MHSA}({\mathit{Q}}^{\prime },{\mathit{K}}^{\prime },{\mathit{V}}^{\prime })=\mathrm{Concat}({\mathrm{head}}_1,\dots ,{\mathrm{head}}_{\mathrm{h}}){\mathit{W}}^O$$

(2)

$${\mathrm{head}}_{\mathrm{h}}=\text{Self-Atten}({\mathit{QW}}_h^Q,{\mathit{KW}}_h^K,{\mathit{VW}}_h^V)$$

(3)

where ${\mathit{Q}}^{\prime }$, ${\mathit{K}}^{\prime }$, and ${\mathit{V}}^{\prime }$ represent the concatenated query, key, and value vectors from all attention heads, respectively. ${\mathit{W}}^O$ denotes the output projection matrix, while $\mathit{W}{h}^Q$, $\mathit{W}{h}^K$, and ${\mathit{W}}_h^V$ are the projection matrices for the query, key, and value vectors for each head, respectively.

Over the past year, the Transformer has shown significant potential in visual processing, owing to its enhanced parallelism and strong performance. In visual applications, the Transformer processes input images by dividing them into fixed-size patches, which are then converted into one-dimensional tokens. These tokens, along with a special class token, are combined and processed through the Transformer encoder. The final, refined class tokens provide a comprehensive representation of the image for subsequent classification tasks.

The Transformer’s demonstrated effectiveness in representation learning has led to its widespread use in various visual tasks, including image classification, object detection, semantic segmentation, and action recognition. Its capability to capture dependencies between input tokens enables the Transformer to extract detailed semantic information from images, thereby enhancing performance across these different applications.

3. Method

To enhance UAV tracking accuracy under varying lighting conditions and complex backgrounds, we developed a target tracking network that explicitly models frequency-aware and spatial-temporal representations for UAV vision, as shown in Figure 2. The network consists of three main components: feature extraction, feature interaction, and classification & regression.

Figure 2. Overall architecture of FSTrack. It contains three components: feature extraction, feature fusion, and classification and regression.

During the feature extraction stage, we deploy a pair of Swin Transformers with shared weights to extract foundational spatial representations from both the template and search images. To bolster tracking precision under fluctuating illumination, we have developed a Transformer-driven module for frequency and spatial information enhancement (FSIE). This module adeptly captures and synthesizes both frequency and spatial attributes. Advancing to the feature fusion stage, the spatio-temporal feature fusion (STFF) module leverages an array of multi-head self-attention mechanisms to amalgamate the target’s dynamic spatio-temporal data, precisely capturing its state and elevating tracking efficacy amidst clutter and occlusions. Concluding with the classification and regression phase, we implement a composite loss function, crafted to refine tracking accuracy and expedite the model’s training convergence.

3.1. FSIE

To enhance tracking robustness under illumination variation, we propose a Transformer-driven frequency-spatial information enhancement (FSIE) module, which explicitly injects frequency-domain representations into the attention computation together with spatial features. Unlike conventional spatial-only Transformer trackers, FSIE enables frequency features to participate directly in feature interaction within the attention mechanism rather than serving as an auxiliary enhancement branch. From the perspective of frequency-domain analysis, illumination information is primarily concentrated in the low-frequency components, whereas discriminative structural details and texture characteristics of the target are mainly distributed in the middle- and high-frequency regions. Consequently, frequency-domain representations exhibit greater robustness to illumination fluctuations and appearance perturbations than purely spatial-domain features. By explicitly modeling spectral responses, the proposed FSIE module can effectively suppress illumination-induced disturbances while preserving informative target structures.

Moreover, frequency-domain information provides complementary cues to spatial representations for both dynamic and static target tracking scenarios. For dynamic targets, spectral representations can capture motion-related variations and temporal-frequency characteristics, thereby improving discrimination between targets and background clutter. For relatively static targets, frequency-domain enhancement contributes to maintaining structural consistency and reducing the influence of high-frequency noise and local appearance variations. Therefore, the joint exploitation of spatial-frequency representations significantly improves localization accuracy and tracking robustness in complex UAV tracking environments. This module aims to sharpen the depiction of the target across both the reference and the search frames.

Unlike existing Transformer-based trackers, such as SwinTrack and other attention-driven architectures that mainly model feature interactions in the spatial domain, the proposed FSIE explicitly establishes cross-domain interactions between frequency-domain and spatial-domain representations. Rather than treating frequency information as an auxiliary branch or performing independent spectral enhancement, FSIE jointly embeds frequency responses and spatial structural cues into a unified attention framework. Specifically, the frequency-domain features extracted by FFT are integrated with graph-based spatial representations before multi-head self-attention is performed, enabling the attention mechanism to simultaneously exploit spectral discriminative information and spatial contextual relationships. This design allows the tracker to capture complementary target characteristics that are not explicitly modeled in conventional Transformer tracking frameworks. The layout of our frequency and spatial information enhancement (FSIE) module is illustrated in the accompanying diagram, as referenced in Figure 3.

Figure 3. The structure of FSIE module.

The key distinction between FSIE and existing feature enhancement modules lies in the coupling strategy between frequency and spatial information. Existing approaches generally perform feature refinement within a single representation space, whereas FSIE introduces a frequency-spatial interaction mechanism that first extracts complementary spectral and graph-structured spatial features and subsequently performs joint attention learning. Through this design, frequency-domain cues participate directly in feature aggregation and attention calculation rather than being fused only at the output stage. Consequently, the learned representations become more discriminative under illumination variation, background clutter, and appearance deformation commonly encountered in UAV tracking scenarios.

Initially, the backbone network captures the raw features of the target, denoted as $\mathit{F}$, directly from the visual data. Let the backbone output feature map be $\mathit{F}\in {\mathbb{R}}^{H\times W\times C}$, where H, W, and C denote the spatial height, width, and channel dimension, respectively. Prior to attention computation, the feature map is reshaped into a token sequence of size $N\times C$, where $N=H\times W$. Subsequently, it utilizes the Fourier transform to derive the target’s spectral characteristics and employs a graph convolutional network to gather spatial attribute details. By deploying multi-head self-attention mechanisms, the system is able to distill contextual insights from the combined spectrum and spatial attributes, which in turn refines the feature portrayal and culminates in the generation of the refined characteristic response, represented as $\mathit{F}\mathbf{1}$.

$$\mathit{F}\mathbf{1}=\mathrm{MHSA}({\left(\mathit{SF}\right)}^Q,{(\mathit{SF}+\mathit{FF})}^K,{(\mathit{SF}+\mathit{FF})}^V)$$

(4)

where $\mathit{SF}$ denotes spatial information features, with $\mathit{SF}=GCN\left(\mathit{F}\right)$, and $\mathit{FF}$ represents frequency-domain features. A two-dimensional FFT is applied to $\mathit{F}$ and only the magnitude spectrum is retained to obtain a real-valued representation, ensuring compatibility with subsequent real-valued attention computation. Both $\mathit{SF}$ and $\mathit{FF}$ have dimensions $N\times C$. The query, key, and value tensors are generated through learnable linear projections as $Q=W_Q\mathit{SF}$, $K=W_K(\mathit{SF}+\mathit{FF})$, $V=W_V(\mathit{SF}+\mathit{FF})$, where $W_Q$, $W_K$, and $W_V$ denote trainable projection matrices. Here, $GCN$ refers to the graph convolutional network, and $FFT$ stands for the fast Fourier transform.

This formulation enables spatial features to preserve localization consistency in the query space, while frequency-enhanced representations enrich key-value interactions for improved discrimination.

The asymmetric design is motivated by the observation that spatial features are more suitable for defining query representations due to their structural stability, while frequency features are more effective as contextual modulation signals in key-value construction. This separation improves training stability and enhances cross-domain feature interaction.

In scenarios of enduring occlusions or image degradation, the reference image from preceding frames might contain extraneous spatio-temporal details. To mitigate this challenge, we’ve integrated a filtering mechanism that employs global average pooling in tandem with a feed-forward neural network. Incorporating the multi-head self-attention (MHSA) in conjunction with our filtering system, the frequency and spatial information enhancement (FSIE) module refines the distinctness of feature information and diminishes interfering factors. This integration strengthens the tracking process’s resilience and reliability over extended durations. The mathematical formulation of the FSIE module is delineated in Equations (5)–(8).

$${\mathit{F}}^{\prime }=Norm\left(\mathit{F}\mathbf{4}+FFN\left(\mathit{F}\mathbf{4}\right)\right)$$

(5)

$$\mathit{F}\mathbf{4}=Norm(\mathit{F}\mathbf{1}+\mathit{SF})\oplus \mathit{F}\mathbf{3}$$

(6)

$$\mathit{F}\mathbf{3}=FFN\left(GAP\left(\mathit{FF}\right)\right)\otimes \mathit{F}\mathbf{2}\otimes Norm(\mathit{F}\mathbf{1}+\mathit{SF})$$

(7)

$$\mathit{F}\mathbf{2}=Conv\left(Cat\left(Norm(\mathit{F}\mathbf{1}+\mathit{SF}),\mathit{FF}\right)\right)$$

(8)

where $GAP$, $Norm$, $FFN$, $Cat$, and $Conv$ denote global average pooling, normalization, feed-forward network, concatenation, and convolution operations, respectively. Specifically, $Norm(·)$ represents Layer Normalization (LN), which is applied after residual connections to stabilize feature distributions and facilitate optimization.

The design of $F_3$ is motivated by the need to jointly model global frequency statistics and spatially guided feature modulation. Specifically, $FFN\left(GAP\right(\mathit{FF}\left)\right)$ captures global frequency-aware channel-wise descriptors, while $Norm(\mathit{F}\mathbf{1}+\mathit{SF})$ preserves spatial consistency. The intermediate term $\mathit{F}\mathbf{2}$ further bridges spatial-frequency interactions, enabling multi-level feature refinement.

3.2. STFF

The response maps, denoted as $\mathit{S}$ and ${\mathit{S}}^{*}$, contain the spatial and temporal attributes obtained from both the template and the search images, respectively. Specifically, $\mathit{S}$ and ${\mathit{S}}^{*}$ are generated from the template branch and search branch after the feature matching stage. Both response maps are represented as feature tensors of size $H_s\times W_s\times C_s$ and are reshaped into token sequences of size $N_s\times C_s$ ($N_s=H_s{W}_s$) before being fed into the Transformer encoder. To achieve enduring and robust target tracking, we have crafted a spatio-temporal feature fusion mechanism grounded in the Transformer framework. This mechanism adeptly amalgamates the two response maps, thereby enhancing the precision of target positioning. Different from conventional Transformer tracking frameworks that mainly focus on feature interactions within a single frame or rely on template-search matching alone, the proposed STFF explicitly models spatio-temporal dependencies between historical template responses and current search responses. Unlike temporal-context trackers such as TCTrack, TrDiMP, and STARK, which typically incorporate temporal information through spatial feature aggregation or template updating strategies, the proposed STFF performs spatio-temporal modeling directly on response maps using a hierarchical encoder-decoder architecture. This enables explicit modeling of temporal consistency at the interaction level rather than implicit embedding within spatial features. Instead of performing direct feature concatenation or standard attention-based fusion, STFF introduces an encoder-decoder architecture that progressively extracts temporal consistency information and spatial localization cues. This design enables the tracker to maintain target continuity across frames while preserving discriminative spatial structures, thereby improving robustness against occlusion, appearance variation, and fast motion. The comprehensive layout of the spatio-temporal feature fusion (STFF) module is depicted in the accompanying diagram, as indicated in Figure 4.

Figure 4. The structure of STFF module.

The Spatio-Temporal Feature Fusion (STFF) module is constituted by two integral elements: an encoder and a decoder. As illustrated in Figure 4, both response maps are concurrently introduced into the multi-head self-attention (MHSA) framework to assimilate stronger spatio-temporal foundational insights. For the encoder, query, key, and value tensors are obtained through independent linear projections of the response maps. Specifically, the template response map provides the primary spatial-temporal query representation, while both template and search responses contribute complementary information for key-value construction. Following standard Transformer practice, these representations are generated through learnable linear projections before being processed by the MHSA module. Since the attention mechanism follows the conventional Transformer formulation, no additional attention operators are introduced in STFF. The resulting attention operation enables information exchange between template-derived and search-derived responses, thereby capturing temporal consistency and spatial correspondence across consecutive frames. This methodology allows the encoder to distill spatial and temporal foundational information from the maps via the MHSA process coupled with a feed-forward neural network. Subsequently, this information is employed by the decoder to precisely pinpoint the target’s position. This response-level temporal modeling is particularly beneficial for UAV tracking scenarios, where motion blur, low-resolution imaging, and abrupt target motion often degrade the reliability of spatial features. By relying on response-map consistency rather than only appearance-based spatial features, STFF reduces sensitivity to visual degradation and improves temporal stability under challenging conditions.

As shown in Figure 4, the decoder employs a dual multi-head self-attention approach to augment the response map, capitalizing on the spatio-temporal foundational insights furnished by the encoder. The first attention layer captures intra-response dependencies within the template response map, while the second attention layer performs interaction between the encoder outputs and the search response map. It should be noted that spatial and temporal information are not represented as separate channels or dedicated network branches. Spatial information corresponds to the localization patterns encoded in the response maps, whereas temporal information is implicitly carried by the updated template response map, which accumulates target representations from previous frames. Therefore, the decoder performs spatio-temporal modeling through attention-based interactions among response-map tokens rather than through explicit spatial and temporal feature streams. Such a hierarchical attention design progressively aggregates spatial and temporal information for robust target localization.

Initially, the decoder isolates spatial and temporal details from the response map associated with the original template through its first self-attention process. Subsequently, the second self-attention process merges these foundational insights with the encoder’s outputs, refining and amalgamating the spatial and temporal data to yield the conclusive response map. This dual mechanism facilitates the generation of intricate spatio-temporal context mappings. Furthermore, unlike standard Transformer decoders that primarily utilize encoder outputs for semantic enhancement, the proposed dual-attention decoding strategy separately models intra-response dependencies and inter-response spatio-temporal correlations. This hierarchical fusion process enables more effective propagation of temporal context while avoiding the loss of fine-grained spatial details, leading to more accurate target localization in challenging UAV tracking environments. By amalgamating the response maps from both the baseline and the search templates, the model enriches its semantic comprehension, thus bolstering the tracking algorithm’s resilience to disruptions in intricate settings.

Overall, the key distinction between STFF and existing Transformer-based temporal trackers lies in that STFF performs hierarchical spatio-temporal modeling at the response level, whereas existing methods mainly rely on spatial feature-level temporal aggregation or template-search correlation. Temporal information is represented through the interaction between the historical template response map and the current search response map. The template is continuously updated during tracking and therefore implicitly preserves target information accumulated from previous frames. Consequently, STFF models temporal consistency through response-level interactions without explicitly storing or processing a long sequence of preceding frames. This design allows more stable temporal propagation and more accurate localization in dynamic UAV environments.

3.3. Loss Function

The loss function is pivotal in shaping the precision of a model’s forecasts regarding the target’s whereabouts and the dimensions of its bounding box. A well-crafted loss function aids in the model’s ability to more accurately anticipate the target’s whereabouts, contours, and trajectory, thus boosting the precision of the tracking process. The loss function for target tracking is commonly bifurcated into two distinct elements: the classification component and the regression component. The classification element is dedicated to differentiating the target from its surroundings, whereas the regression element is designed to fine-tune the parameters of the target’s bounding box.

In the target tracking network presented in this paper, the classification loss function ${\mathcal{L}}_{\mathrm{cls}}$ is computed using Focal Loss [39], as specified in Equation (9).

$${\mathcal{L}}_{\mathrm{cls}}={(1-P_t)}^{\gamma }·log\left(P_t\right)$$

(9)

where $P_t$ indicates the probability that the network classifies an instance as a positive category, and $\gamma$ represents the focusing parameter.

Our approach leverages a hybrid regression loss function that integrates the L1 loss [40] with the CIoU loss [41]. The L1 loss is employed as a bounding-box regression loss to measure the absolute difference between the predicted bounding-box parameters and the corresponding ground-truth values. By providing stable gradients during optimization, the L1 loss complements the CIoU loss and contributes to more accurate target localization. It is formulated as presented in Equation (10). The CIoU loss, on the other hand, assesses the overlap of the predicted bounding box with the ground truth, factoring in the centroid distance and the aspect ratio consistency. By incorporating these additional geometric insights, CIoU furnishes richer gradient details during the optimization phase, accelerating the convergence rate and enhancing the efficiency of the training process. The methodology for calculating CIoU loss is delineated in Equations (11)–(13).

$${\mathcal{L}}_{\mathrm{L}1}={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}\left|B_{gt}^{\left(i\right)}-B_{pr}^{\left(i\right)}\right|$$

(10)

$${\mathcal{L}}_{\mathrm{CIoU}}=\mathrm{IoU}-{\displaystyle \frac{{\rho }^2}{c^2}}-\alpha v$$

(11)

$$v={\displaystyle \frac{4}{{\pi }^2}}{\left(arctan{\displaystyle \frac{w_{gt}}{h_{gt}}}-arctan{\displaystyle \frac{w_{pr}}{h_{pr}}}\right)}^2$$

(12)

$$\alpha ={\displaystyle \frac{v}{1-\mathrm{IoU}+v}}$$

(13)

where n denotes the number of samples, $B_{gt}$ and $B_{pr}$ denote the ground-truth and predicted bounding-box parameters, respectively, P represents the probability of the true class, $w_{gt}$ and $h_{gt}$ are the width and height of the ground truth box, $w_{pr}$ and $h_{pr}$ are the width and height of the predicted box, IoU stands for Intersection over Union, $\rho$ indicates the distance between the center points of the bounding boxes, and c is the diagonal of the outer bounding box.

In summary, the loss function for the proposed target tracking network is defined by Equation (14).

$${\mathcal{L}}_{\mathrm{ours}}={\mathcal{L}}_{\mathrm{cls}}+{\lambda }_{\mathrm{L}1}{\mathcal{L}}_{L1}+{\lambda }_{\mathrm{CIoU}}{\mathcal{L}}_{\mathrm{CIoU}}$$

(14)

where ${\lambda }_{\mathrm{L}1}$ and ${\lambda }_{\mathrm{CIoU}}$ are the balancing coefficients. Experimental results in this paper validate that ${\lambda }_{\mathrm{L}1}=2$ and ${\lambda }_{\mathrm{CIoU}}=3$. The overall loss function consists of a classification loss and two regression losses. Specifically, ${\mathcal{L}}_{\mathrm{cls}}$ is used to distinguish the target from the background, ${\mathcal{L}}_{\mathrm{L}1}$ constrains the coordinate regression error between the predicted and ground-truth bounding boxes, and ${\mathcal{L}}_{\mathrm{CIoU}}$ further optimizes the geometric consistency by considering overlap, center distance, and aspect ratio. The balancing coefficients ${\lambda }_{\mathrm{L}1}$ and ${\lambda }_{\mathrm{CIoU}}$ are introduced to regulate the relative contributions of the two regression terms during training.

4. Experiments

4.1. Datasets and Metrics

4.1.1. Datasets

The experiments are conducted on UAV123 [42], DTB70 [43], and UAVDT [28] datasets, which contain diverse aerial tracking scenarios under complex environments. In addition to illumination variation, occlusion, background clutter, and scale changes, these datasets also exhibit diverse target motion characteristics, including slow motion, fast motion, abrupt motion, and irregular trajectory variations. The motion speed and trajectory complexity vary significantly across different sequences, providing comprehensive evaluations for tracker robustness under dynamic UAV tracking conditions. Therefore, these datasets are well suited for validating the effectiveness of the proposed method in handling complex target motion patterns.

Specifically, UAV123 comprises 123 aerial video sequences, showcasing a multitude of scenarios and subjects, including but not limited to pedestrians, automobiles, and bicycles. Tailored for assessing visual tracking efficacy in UAV environments, DTB70 offers 70 high-fidelity video sequences captured aerially, encompassing diverse settings and subjects. UAVDT spans a spectrum of settings, such as urban landscapes, roadways, and academic grounds, under varying conditions like time of day, climatic conditions, and flight altitudes. This dataset presents a diverse array of targets ranging from pedestrians to various vehicles and poses extensive challenges for detection and tracking tasks. We convert video sequences into single-object tracking format by using the first-frame ground truth bounding box as initialization and tracking a single target per sequence.

4.1.2. Evaluation Metrics

In our study, we apply the one-pass evaluation (OPE) technique [44] to measure the efficacy of the target tracking approach, encompassing metrics such as the success rate (Suc) and precision (Pre). The algorithm’s computational demand is gauged by examining the frame rate and parameter count. The success rate metric quantifies the percentage of frames where the intersection between the estimated bounding box and the actual bounding box exceeds a predefined threshold, compared to the overall frame count. This concept is articulated in Equation (15).

$$\mathrm{Suc}={\displaystyle \frac{Count(R_{\mathrm{IoU}}>\alpha )}{Count\left(R_{\mathrm{IoU},all}\right)}}\times 100\%$$

(15)

where $\alpha$ represents the threshold value, typically set to 0.5. The overlap rate $R_{IoU}$ is the intersection ratio between the predicted box and the ground truth box, calculated using Equation (16):

$$R_{\mathrm{IoU}}={\displaystyle \frac{\left|B\cap B^{gt}\right|}{\left|B\cup B^{gt}\right|}}$$

(16)

where B and $B^{gt}$ denote the prediction box and the ground truth box, respectively.

Precision is defined as the ratio of the number of frames where the distance $\delta$ between the center of the predicted box and the center of the ground truth box is below a specified threshold, relative to the total number of frames. It is calculated using Equation (17):

$$\mathrm{Pre}={\displaystyle \frac{Count(\delta <\beta )}{Count\left({\delta }_{all}\right)}}\times 100\%$$

(17)

where $\beta$ denote threshold value, which is usually set to 20. The target centre position error $\delta$ is given by:

$$\delta =\sqrt{{(x_p-x_r)}^2+{(y_p-y_r)}^2}$$

(18)

where $(x_r,y_r)$ and $(x_p,y_p)$ represent the coordinates of the center of the predicted box and the coordinates of the center of the ground truth box, respectively.

4.2. Implementation Details

FSTrack has been developed using the PyTorch framework, specifically version 2.11, and underwent testing on a hardware setup featuring an Intel Core i9-13900K CPU with a 3.0 GHz clock speed and an NVIDIA RTX 4090 GPU. For the training phase, we employed datasets including the video object detection (VID), the object detection (DET), and the comprehensive COCO dataset [45], collectively offering over 760,000 video sequences and more than 10 million images with annotations. The model’s input consists of frame pairs depicting the same object from the annotated sequences, which assists the model in discerning object motion. The model operates on template images of dimensions 128 × 128 pixels that are moved across search images of 256 × 256 pixels with an interval of 8 pixels. The training was conducted with a batch size of 24, employing the AdamW optimizer [46]. The learning rate was initialized to ${10}^{-4}$, complemented by a weight decay factor of ${10}^{-5}$, and the training spanned a total of 300 epochs.

We conducted a comparative analysis of our proposed FSTrack against 14 state-of-the-art tracking algorithms, encompassing SiamFC [18], PrDiMP [31], TransT [33], TrDiMP [34], OSTrack [36], MixFormer [37], SiamRPN++ [47], SiamR-CNN [48], STARK [27], SeqTrack [49], E.T.Track [50], SMAT [51], HiFT [52], and SwinTrack [53]. Each experiment was executed on an identical experimental platform, with assessments focusing on metrics of precision, success, and computational efficiency. We use officially released pretrained models for all compared methods. For methods without available weights, we follow their original training protocols. All methods are evaluated using the same input resolution for fair comparison.

4.3. Ablation Experiments

The ablation studies conducted on the UAV123 dataset with the RTX 4090 GPU have yielded significant insights, as delineated in Table 1. Initially, the integration of the FSIE module into the foundational SwinTrack algorithm resulted in a noteworthy 1.4% increase in the success rate and a substantial 2.3% boost in precision. Further enhancement was observed when both the FSIE and STFF modules were incorporated into the baseline algorithm. This dual integration led to a more pronounced improvement in the success rate, which rose by 3.5%, and an even more significant surge in precision, increasing by 5.5%. Lastly, the comprehensive integration of the FSIE module, the STFF module, along with the novel loss function ${\mathcal{L}}_{\mathrm{ours}}$, into the SwinTrack algorithm, yielded the most substantial results. The success rate saw an impressive 3.8% improvement, and the precision achieved the highest increment of 6.9%. These results underscore the individual and collective contributions of the proposed modules and loss function to the tracking performance.

Table 1. Ablation experiments on UAV123. The best performance is displayed in bold. ↑ means that the larger the value, the better the result. All improvements are recalculated based on consistent baseline results for fair comparison. ✔ indicates that the corresponding component is included in the ablation study.

Baseline	FSIE	STFF	${\mathcal{L}}_{\mathbf{ours}}$	UAV123
				Suc (%) ↑	Pre (%) ↑
✔				54.6	73.4
✔	✔			57.8	75.7
✔	✔	✔		62.1	78.9
✔	✔	✔	✔	65.4	80.1

4.4. Comparisons with State-of-the-Arts Methods

4.4.1. Experimental Results on UAV123 Dataset

In this study, the FSTrack algorithm has undergone rigorous evaluation on the UAV123 tracking benchmark, where it was juxtaposed with 14 alternative tracking methodologies. The comparative analysis is visually represented in Figure 5, Table 2, and Figure 6, which depict the success and precision rates of FSTrack in relation to the other 14 methodologies. Our findings indicate that FSTrack has attained a success rate of 65.4% and a precision rate of 80.1%, signifying respective enhancements of 1.1% and 2.2% over the SwinTrack algorithm. Although SwinTrack is positioned as the runner-up in terms of success rate, HiFT secures the second spot for precision rate. Collectively, these outcomes underscore the formidable competitiveness of the FSTrack algorithm.

Figure 5. Comparison of success plot and precision plot of tracking results.

Table 2. Quantitative results on the UAV datasets. The best performance is displayed in bold. ↑ means that the larger the value, the better the result.

Methods	Backbone	UAV123		DTB70		UAVDT
		Suc (%) ↑	Pre (%) ↑	Suc (%) ↑	Pre (%) ↑	Suc (%) ↑	Pre (%) ↑
SiamFC [18]	AlexNet	46.7	68.9	48.3	70.1	47.5	69.9
PrDiMP [31]	ResNet	57.5	76.3	59.1	79.5	56.3	76.2
TransT [33]	ResNet	58.8	78.6	59.6	80.4	57.8	75.3
TrDiMP [34]	ResNet	54.9	76.3	61.5	77.5	59.8	76.7
OSTrack [36]	ViT	64.1	75.2	64.3	75.8	63.9	74.9
MixFormer [37]	MAM	62.3	76.9	62.5	77.1	63.1	77.6
SiamRPN++ [47]	ResNet	61.0	75.2	61.6	75.8	62.5	76.9
SiamR-CNN [48]	ResNet	59.7	72.4	60.3	72.6	60.7	73.1
STARK [27]	ResNet	59.6	73.8	60.1	74.1	59.8	73.7
SeqTrack [49]	ViT	63.8	74.9	64.2	75.5	64.3	75.7
E.T.Track [50]	LT-Mobile	56.9	74.1	57.1	74.8	56.7	73.9
SMAT [51]	ViT	57.8	74.7	59.2	74.9	58.5	74.7
HiFT [52]	AlexNet	58.7	78.7	59.4	79.3	59.1	78.9
SwinTrack [53]	ViT	64.3	77.9	65.2	79.9	65.1	79.3
FSTrack	ViT	65.4	80.1	67.9	80.9	66.2	81.4

Figure 6. Radar graphs of Success (Suc) and Precision (Pre) metrics with state-of-the-art methods on UAV datasets.

4.4.2. Experimental Results on DTB70 Dataset

Figure 5, Table 2, and the visual representation in Figure 6 collectively present the comparative tracking performance of 14 algorithms on the DTB70 benchmark. Our proposed algorithm exhibits robust performance across the 70 challenging scenarios characterized by vigorous UAV movements, securing a success rate of 67.9% and a precision rate of 80.9%. When juxtaposed with the baseline SwinTrack, which possesses a notably larger parameter set, our algorithm demonstrates a marked improvement, with a 2.7% increase in success rate and a 1% enhancement in precision rate. These comparative outcomes serve to accentuate the efficacy of our proposed algorithm in managing the pronounced motion dynamics inherent to UAV operations.

4.4.3. Experimental Results on UAVDT Dataset

To substantiate the efficacy and practicality of our proposed algorithm for UAV target tracking, we executed comprehensive experiments utilizing the UAVDT dataset. This dataset encompasses a wealth of detailed annotations, such as the target’s location, size, and speed, which lays a solid groundwork for the assessment of tracking algorithms. Our algorithm was pitted against a spectrum of sophisticated UAV tracking methodologies. As depicted in Figure 5, Table 2, and Figure 6, our algorithm secured the top tracking success and precision rates within the UAVDT dataset. This accomplishment underscores its exceptional performance across a multitude of scenarios. Moreover, our algorithm exhibits significant resilience across varying conditions, including diverse environments, weather patterns, and lighting changes, adeptly tracking targets with negligible susceptibility to external perturbations.

4.5. Qualitative Evaluation

To comprehensively evaluate the effectiveness and practicality of the proposed FSTrack algorithm in UAV-based object tracking scenarios, we conducted extensive experimental analyses on multiple benchmark datasets. The qualitative and quantitative results are visualized in Figure 7, Figure 8 and Figure 9, clearly demonstrating the performance differences among various state-of-the-art tracking algorithms. As observed, the SiamFC tracker consistently underperforms across multiple sequences, exhibiting frequent tracking failures and high sensitivity to illumination variations. While the baseline SwinTrack algorithm demonstrates moderately stable tracking behavior, it suffers from suboptimal localization accuracy when compared to more advanced approaches such as TransT and TrDiMP. Although TrDiMP achieves competitive results under normal conditions, its performance deteriorates in the presence of background distractors that closely resemble the target, often leading to tracking drift or misidentification.

Figure 7. Comparison with state-of-the-arts on UAV123.

Figure 8. Comparison with state-of-the-arts on DTB70.

Figure 9. Comparison with state-of-the-arts on UAVDT.

In contrast, the proposed FSTrack framework exhibits superior robustness and accuracy across a wide range of challenging scenarios. Notably, it maintains precise target localization even under significant illumination changes, in cluttered backgrounds, and in the presence of similar-looking objects. This robustness is attributed to the algorithm’s enhanced feature discrimination capability and adaptive spatiotemporal modeling, which together ensure reliable tracking performance. These results indicate that FSTrack achieves a favorable balance between performance and computational efficiency, demonstrating strong robustness under the evaluated benchmarks.

4.6. Computational Complexity Experiments

The computational efficiency of a tracking algorithm is a critical consideration. Two key indicators—the number of trainable parameters (Params) and frames per second (FPS)—serve as fundamental metrics to evaluate both model complexity and inference speed. Specifically, the Params metric reflects the total number of learnable weights within the model, indicative of its architectural scale, while FPS measures the rate at which input frames are processed, directly correlating with computational efficiency under a unified evaluation platform.

Table 3 presents a comparative assessment of the computational characteristics across several representative tracking algorithms. All FPS results are measured on the same high-performance GPU (RTX 4090) under identical input settings to ensure fair comparison of algorithmic efficiency. For instance, SiamFC employs 2.3 million parameters and achieves an inference speed of 58.9 FPS, demonstrating a well-balanced trade-off between model size and runtime efficiency. In comparison, PrDiMP features the smallest parameter footprint at only 0.76 million; however, its FPS is comparatively lower than that of SiamFC, highlighting differences in architectural design and computational overhead. Notably, the proposed FSTrack framework integrates a significantly larger model with 10.89 million parameters, yet it sustains 56.4 FPS under the same evaluation setting. This performance demonstrates that FSTrack maintains competitive computational efficiency while incorporating a more expressive representation learning architecture.

Table 3. Results of complexity experiments on three datasets. The best performance is displayed in bold (Params and FPS). ↑ means that the larger the value, the better the result. ↓ means that the small the value, the better the result.

	Params (M) ↓	FPS ↑
		UAV123	DTB70	UAVDT
SiamFC [18]	2.3	58.9	60.1	59.4
PrDiMP [31]	0.76	29.1	28.4	31.5
TransT [33]	4.85	31.3	32.9	34.6
TrDiMP [34]	2.37	41.6	41.9	42.4
OSTrack [36]	86	10.9	9.7	11.3
MixFormer [37]	5.44	24.9	24.3	25.6
SiamRPN++ [47]	16.36	25.1	25.8	27.2
SiamR-CNN [48]	11.16	34.9	33.5	35.7
STARK [27]	107.78	16.7	17.8	16.9
SeqTrack [49]	16.13	42.3	44.1	44.0
E.T.Track [50]	23.76	51.2	51.6	50.9
SMAT [51]	15.83	52.3	53.8	53.5
HiFT [52]	33.19	47.8	47.9	47.8
SwinTrack [53]	21.46	45.3	46.1	45.9
FSTrack	10.89	56.4	57.1	56.9

These results collectively suggest that FSTrack achieves a favorable balance between computational complexity and tracking performance. The reported efficiency indicates its potential for real-time inference under GPU-based evaluation settings. We note that actual deployment on UAV platforms depends on additional constraints such as embedded hardware specifications, power consumption, and memory limitations, which are beyond the scope of this work.

5. Conclusions

This paper introduces an innovative strategy for boosting the precision of target tracking in UAV imagery through the integration of frequency-domain insights, spatial characteristics, and spatio-temporal data. Our proposed technique, dubbed FSTrack, encompasses several pivotal elements: (1) feature extraction: It capitalizes on a pair of Swin Transformers with synchronized weights to distill essential features from both the template and search images, ensuring a robust feature representation. (2) frequency and spatial Information enhancement (FSIE): This module amalgamates frequency and spatial domain data to bolster tracking accuracy, particularly under challenging illumination variations. (3) spatio-temporal feature fusion (STFF): It deploys a module equipped with multi-head self-attention (MHSA) mechanisms to effectively integrate spatio-temporal state information, thereby enhancing tracking performance in complex scenarios such as background clutter and occlusions. (4) hybrid loss function: A novel loss function is designed by combining classification and regression components to improve tracking accuracy and accelerate training convergence. Empirical evidence from our experiments substantiates that FSTrack significantly outperforms existing state-of-the-art methods in both precision and success rate, while maintaining real-time performance.

In practical UAV systems, the predicted target states generated by FSTrack can further serve as high-level perception inputs for downstream guidance, navigation, and control (GNC) modules.

Looking ahead, our research will focus on integrating multi-modal sensor data, including infrared and event cameras, to obtain more comprehensive target information and further improve tracking robustness and reliability.