Visual Object Tracking Based on the Motion Prediction and Block Search in UAV Videos

Abstract

With the development of computer vision and Unmanned Aerial Vehicles (UAVs) technology, visual object tracking has become an indispensable core technology for UAVs, and it has been widely used in both civil and military fields. Visual object tracking from the UAV perspective experiences interference from various complex conditions such as background clutter, occlusion, and being out of view, which can easily lead to tracking drift. Once tracking drift occurs, it will lead to almost complete failure of the subsequent tracking. Currently, few trackers have been designed to solve the tracking drift problem. Thus, this paper proposes a tracking algorithm based on motion prediction and block search to address the tracking drift problem caused by various complex conditions. Specifically, when the tracker experiences tracking drift, we first use a Kalman filter to predict the motion state of the target, and then use a block search module to relocate the target. In addition, to improve the tracker’s ability to adapt to changes in the target’s appearance and the environment, we propose a dynamic template updating network (DTUN) that allows the tracker to make appropriate template decisions based on various tracking conditions. We also introduce three tracking evaluation metrics: namely, average peak correlation energy, size change ratio, and tracking score. They serve as prior information for tracking status identification in the DTUN and the block prediction module. Extensive experiments and comparisons with many competitive algorithms on five aerial benchmarks, UAV20L, UAV123, UAVDT, DTB70, and VisDrone2018-SOT, demonstrate that our method achieves significant performance improvements. Especially in UAV20L long-term tracking, our method outperforms the baseline in terms of success rate and accuracy by 19.1% and 20.8%, respectively. This demonstrates the superior performance of our method in the task of long-term tracking from the UAV perspective, and we achieve a real-time speed of 43 FPS.

1. Introduction

In recent years, UAV technology and industry have developed rapidly. Object tracking technology from the UAV perspective has been widely used and has become the core technology of UAVs [1,2,3]. Visual target tracking technology is not only widely used in civil fields such as security, logistics, and rescue but also plays an important role in military fields such as intelligence collection, guidance, and remote sensing [4]. However, object tracking technology from the UAV perspective faces several challenges and problems [5]. (1) Tracking robustness under complex scenarios: there are many challenging statuses in object tracking tasks from an aerial perspective, such as fast motion, object deformation, background clutter, occlusion, and being out of view. (2) Resource constraints: UAVs have limited computational resources and energy, and must achieve efficient object tracking and meet the real-time requirements with limited resources. Therefore, researchers still need to explore and solve the problems faced by object tracking technology from the UAV perspective.

Traditional object-tracking algorithms usually use filtering methods (e.g., Kalman filter [6]) to estimate the motion state of the object, and predict the position of the object by establishing state equations and observation equations. However, these algorithms are prone to the problem of tracking drift when the object undergoes sudden changes in motion or when the sensor wobbles. In recent years, with the development of deep learning (DL) technology, researchers have begun to apply DL techniques to visual object tracking and have made significant progress. Due to their excellent performance, Siamese network trackers [7,8] have been favored by many researchers. In visual object tracking, Siamese networks are deep neural networks for learning target features and computing object similarities. It consists of two similar convolutional neural networks (CNNs) that process the template image and the search image respectively and output their feature representations. Some derived methods such as STMTrack [9], SiamGAT [10], and SiamRN [11] are based on Siamese networks. These methods usually improve the training method of the algorithm and the similarity calculation process but do not address the tracking drift problem caused by disturbances such as object occlusion and being out of view. As a result, these methods are not advantageous for long-term tracking tasks in complex scenes.

With the improvement of computing resources and the success of the Transformer architecture in the field of natural language processing, target tracking algorithms based on Transformer architecture have become a hot research topic, such as DropTrack [12], SwinTrack [13], and AiATrack [14]. In visual object tracking, tracking algorithms based on the Transformer architecture mainly consist of an encoder and a decoder. The encoder and decoder consist of multiple identical encoder layers, each containing a self-attention mechanism and a feedforward neural network. The self-attention mechanism can model associations between different locations in the input stream. Feedforward neural networks introduce nonlinear transformations into the feature representation. SwinTrack [13] uses the Transformer architecture for both representation learning and feature fusion. The Transformer architecture allows for better feature interaction for tracking than the pure CNN framework. AiATrack [14] proposes an attention-in-attention (AiA) module, which enhances appropriate correlations by seeking consensus among all correlation vectors. The AiA module can be applied to both self-attention blocks and cross-attention blocks to facilitate feature aggregation. It is due to the attentional mechanism of the Transformer architecture and the feed-forward neural network that the tracker performs the tracking task with better robustness and generalization. However, for the same reason, trackers based on the Transformer architecture are more computationally intensive. Therefore, certain limitations exist in performing tracking tasks on UAV platforms.

In order to solve the tracking drift problem caused by complex conditions such as occlusion and being out of view, this paper proposes a visual object tracking algorithm based on motion prediction and block search and improves it for the tracking task from the UAV perspective. We introduce three metrics for evaluating tracking results: average peak correlation energy (APCE), size change ratio (SCR), and tracking score (TS). These metrics aim to jointly identify the tracking status and provide prior information for the proposed dynamic template updating network (DTUN). The proposed DTUN employs the optimal template strategy based on different tracking statuses to improve the tracker’s robustness. We utilize the Kalman filtering for motion prediction of the object when the object is temporarily occluded or subject to other disturbances that cause tracking failure. In cases of near-linear motion, the Kalman filter can be applied simply and efficiently. For cases of the nonlinear motion of the object, the Kalman filter predicts the approximate motion direction of the object, which is crucial for the block-search module. The proposed block-prediction module mainly solves the long-term tracking drift problem. Considering cases of tiny objects from the UAV perspective, we process the enlarged search region with block search, significantly improving search efficiency for tiny objects. Overall, our method performs well in solving the tracking drift problem in long-term tracking caused by complex statuses such as object occlusion and being out of view. Figure 1 shows a visualization of our method compared with the baseline algorithm. When tracking drift occurs due to object occlusion, it will continue if no action is taken (as shown by the blue bounding box). In contrast, our method employs a target prediction and block search framework, which can effectively relocate the tracker to the object (as shown by the red bounding box).

Extensive experiments on five aerial datasets, UAV20L [15], UAV123 [15], UAVDT [16], DTB70 [17], and VisDrone2018-SOT [18], demonstrate the superior performance of our method. In particular, on the large-scale long-term tracking dataset UAV20L, our method achieves a significant improvement of 19.1% and 20.8% in terms of success and accuracy, respectively, compared to the baseline method. In addition, our method achieves a tracking speed of 43 FPS, which far exceeds the real-time requirement. This demonstrates the high-efficiency and high-accuracy performance of our method in performing the object tracking task from the UAV perspective. Figure 2 shows the results of our method compared with other methods in terms of success rate and tracking speed.

In summary, the main contributions of this paper can be summarized in the following three aspects:

(1)

We introduce three evaluation metrics: the APCE, SCR, and TS, which are used to evaluate the tracking results for each frame. These evaluation metrics are used to jointly identify the tracking status and provide feedback information to the DTUN in the tracking of subsequent frames. The proposed DTUN adjusts the template strategy according to different tracking statuses, enabling the tracker to adapt to changes in the object and the tracking scenario.

(2)

We propose a motion prediction and block search module. When tracking drift occurs due to complex statuses (e.g., occlusion and being out of view), we first predict the motion state of the object using a Kalman filter, and then utilize the block search to re-locate the object. Its performance is excellent for solving the tracking drift problem from the UAV perspective.

(3)

Our proposed algorithm achieves significant performance improvement on five aerial datasets, UAV20L [15], UAV123 [15], UAVDT [16], DTB70 [17], and VisDrone2018-SOT [18]. In particular, on the long-term tracking dataset UAV20L, our method achieves 19.1% and 20.8% increase in success and precision, respectively, compared to the baseline method, and achieves 43 FPS real-time speed.

This paper is organized as follows: Section 2 describes other work related to our approach. Section 3 describes the framework and specific details of our proposed approach. Section 4 shows the results of experiments on five aerial datasets and compares them with other methods. Finally, our work is summarized in Section 5.

2. Related Work

Visual object tracking is an important task in computer vision that has developed rapidly in recent years. It aims to accurately track objects in video sequences in real time. In addition, the application of visual object tracking in the field of UAVs is receiving increasing attention and plays an important role. In this chapter, the research on visual object tracking algorithms and their development on UAVs is discussed.

2.1. Object Tracking Algorithm

With the development of computer vision and DL, researchers began to apply DL to visual object tracking. The introduction of the Siamese network has made a significant contribution to the progress of visual object tracking. It learns the representation of the template and search images and calculates the similarity between them. The template and search images are fed into two networks with shared weights. Siamese network-based trackers have achieved high accuracy and real-time performance in object-tracking tasks. Recently, the Transformer model achieved great success in natural language processing and computer vision with the improvement of computational resources. Transformer-based tracking algorithms achieve more accurate tracking by introducing an attention mechanism to model the relationship between the object and its surrounding context. These methods take a sequence of object features as input and use a multilayer Transformer encoder to learn the object’s representation.

In recent years, Siamese network-based tracking algorithms have been extensively researched. Li et al. proposed SiamRPN [19] by introducing the regional proposal network (RPN) [20] into the Siamese network. RPN consists of two branches, object classification and bounding box regression. The introduction of RPN eliminates the traditional multiscale testing and online fine-tuning process, and greatly improves the accuracy and speed of object tracking. Guo et al. argued that anchor-based regression networks require tricky hyperparameters to be set manually, so they proposed an anchor-free tracking algorithm (SiamCAR) [21]. Compared to the anchor-based approach, the anchor-free regression network has fewer hyperparameters. It directly calculates the distance from the center of the object to the four edges during the bounding box regression process. Fu et al. argued that the fixed initial template information has been fully mined and the existing online learning template update process is time-consuming. Therefore, they proposed a tracking framework based on space-time memory networks (STMTrack) [9] that utilizes the historical templates from the tracking process to better adapt to changes in the object’ appearance. RTS [22] proposed a segmentation-centered tracking framework that can better distinguish between object and background information. It can generate accurate segmentation masks in the tracking results, but there is a reduction in the tracking speed. Although the above trackers improve tracking accuracy by improving the model training method and bounding box regression, they do not effectively solve the tracking drift problem due to complex situations. In addition, Zheng et al. [23] argued that the imbalance of the training data makes the learned features lack significant discriminative properties. Therefore, in the offline training phase, they made the model more focused on semantic interference by controlling the sampling strategy. In the inference phase, an interference-aware module and a global search strategy are used to improve the tracker’s resistance to interference. However, this global search strategy is not good for tracking tiny objects from the UAV perspective, especially when there are similar objects around, background clutter, or low resolution.

After Siamese networks, Transformer large-model-based trackers also achieved excellent results. Yan et al. [24] presented a tracking architecture with an encoder–decoder transformer (STARK). The encoder models the global spatio-temporal feature information of the target and the search region, and the decoder predicts the spatial location of the target. The encoder–decoder transformer captures remote dependencies in both spatial and temporal dimensions and does not require subsequent hyperparameter processing. Cao et al. proposed an efficient Hierarchical Feature Transformer (HiFT) [25], which inputs hierarchical similarity maps into the feature transformer for the interactive fusion of spatial and semantic cues. It not only improves the global contextual information but also efficiently learns the dependencies between multilevel features. Ye et al. [26] argued that the features extracted by existing two-stage tracking frameworks lack target perceptibility and have limited target–background discriminability. Therefore, they proposed a one-stage tracking framework (OSTrack), which bridges templates and search images with bidirectional information flows to unify feature learning and relation modeling. To further improve the inference efficiency, they proposed an in-network candidate early elimination module to gradually discard candidates belonging to the background. The above Transformer model-based tracker achieves significant performance improvement in visual object tracking, but it requires high computational resources, which are not advantageous for applications on UAV platforms.

2.2. Tracking Algorithms in the UAV

Visual object-tracking technology is widely used as the core technology of UAV applications. The fast and automatic tracking of moving targets can be achieved by equipping UAVs with visual sensors and object-tracking algorithms. However, tiny objects detected by aerial sensors are easily occluded and susceptible to environmental influences. In addition, there are limitations in the computational power of UAV devices. Therefore, researchers have focused on developing, researching, and improving object-tracking algorithms specifically for aerial tracking to address these challenges.

Cao et al. [25] argued that using only the last layer of image features will reduce the accuracy of aerial tracking, and simply using multiple layers of features will increase the online inference time. Therefore, they proposed the hierarchical feature Transformer framework. This framework enables the interactive fusion of spatial and semantic information to achieve efficient aerial tracking. TCTrack [27] proposed a temporal context information tracking framework that can fully utilize the temporal context of aerial tracking. Specifically, they proposed online temporal adaptive convolution to enhance temporal information of spatial features and an adaptive temporal converter that uses temporal knowledge for encoding-decoding. The proposed online temporal adaptive convolution can be used to enhance the temporal information of spatial features. Since visual trackers may perform the tracking task at night, low-light conditions are unfavorable for tracking. Therefore, Li et al. [28] proposed a novel discriminative correlation filter-based tracker (ADTrack) with illumination adaptive and anti-dark capability. This tracker first extracts the illumination information from the image, then performs enhancement preprocessing, and finally generates an object-aware mask to realize object tracking. For airborne tracking, if the object is within a complex environment, tracking performance is severely compromised. With the application of depth cameras on UAVs, adding depth information can more effectively deal with complex scenes such as background information interference. Therefore, Yang et al. [29] proposed a multimodal fusion and feature-matching algorithm and constructed a large-scale RGBD (RGB-Depth map) tracking dataset. In addition, RGB-T (RGB-thermal) [30] tracking is also an effective means in the field of visual object tracking to solve the difficulty of UAV tracking in complex scenes. However, adding additional sensor devices also significantly increases the cost and flight burden of UAVs.

3. Proposed Method

This section provides a comprehensive overview of the proposed method. Section 3.1 outlines the overall structure, Section 3.2 delves into the dynamic template updating network, Section 3.3 discusses the specifics of the search–evaluation network, and Section 3.4 presents details related to the block-prediction module.

3.1. Overall Framework

Figure 3 illustrates the structure of the proposed MPBTrack algorithm, comprising three parts: the dynamic template updating network, the search–evaluation network, and the block-prediction module. The dynamic template updating network adjusts the number of templates according to the evaluation results regarding the tracking condition. The tracking results for each frame are filtered, and high-quality templates are stored in the template memory. The template extraction module extracts a corresponding number of diverse high-quality template features from the template memory and then concatenates these template features with the initial templates. The number of templates used for the current frame tracking is determined by the received APCE feedback results. The search–evaluation network calculates the similarity between the template image and the search image and evaluates the tracking results. Following the similarity calculation, the network obtains results for the object’s classification, center-ness, and regression from three branches. The tracking status evaluation network yields three metrics: APCE, TS, and SCR. The block-prediction module first recognizes the tracking status based on the three joint evaluation metrics. When the tracker detects tracking drift caused by occlusion or other interference, it will predict the target’s motion trajectory using a Kalman filter. If the object’s position is not accurately predicted within 20 frames, a block search in an expanded area is utilized to relocate the object.

3.2. Dynamic Template Updating Network

Since fixed templates alone cannot adapt to changes in the object’ appearance and tracking scenarios (e.g., illumination variation and background clutter), they can easily lead to tracking failures. Therefore, we propose a dynamic template updating network to adapt to the changes in the object’ appearance and tracking scenarios during the tracking process. For normal tracking conditions, the tracker can achieve accurate tracking by using only a few templates. For complex tracking conditions, we will employ more historical templates to adapt them to improve the robustness of object tracking. The method dynamically switches the number of templates according to the tracking status, which improves the tracking robustness as well as the computational efficiency.

Figure 4 shows the structure of the DTUN. The network includes a dynamic template in addition to the traditional initial template. The dynamic templates are obtained from high-quality regions filtered by the tracking results. The dynamic features extracted by the feature extraction network are stored in the feature memory. In poor tracking status (e.g., background clutter, motion blur, and object occlusion), the target region of the tracking result is affected when the template quality is low. If low-quality templates are stored in the template memory, the subsequent tracking accuracy will be greatly affected. The tracking score is the most intuitive response to the quality of the tracking region. In order to improve the quality of templates in the template memory, we filter the tracking results to ensure that high-quality templates are stored in the feature memory while low-quality templates are discarded.

The TS metric is employed to assess the quality of the tracked region for each frame. If the TS of the current region exceeds a predefined threshold (0.6), it will be incorporated into the feature memory as a new template, following the processes of cropping and feature extraction. Specifically, the new template will perform the following operations to yield the optimal template features: (1) obtain the cropping size of the current image according to Equation (23), and then crop with the target as the center point; (2) resize the cropped region to $289\times 289$ to obtain a template containing the foreground region and the background region; (3) generate a foreground–background mask map with the same size as the foreground and the background in the template; and (4) the cropped template and the mask map are jointly fed into the feature extraction network to obtain the template features (the purpose is to improve the tracker’s discrimination between the target and the background), and finally the template features are stored in the feature memory.

The APCE score measures the quality of the response map of the tracking results (described in more detail in Section 3.3.2). The tracker receives feedback on the APCE results at each frame and transforms the APCE values into the corresponding number of templates. The equation for this transformation process is as follows:

$$T=N_{max}-\frac{N_{max}-1}{1+exp\left(-a\left(APCE-b\right)\right)}$$

(1)

where T denotes the number of templates, and $N_{max}$ denotes the maximum value of the template range. We set the template range to 1–10, and a and b denote the slope and horizontal offset of the function, respectively. If the tracking quality of the previous frame is high, a small number of templates will be used in the next frame to achieve better tracking results. When the tracker receives a lower tracking quality in the previous frame, more historical templates will be utilized to adapt to the current complex tracking situation.

The template extraction mechanism is an important part of the dynamic template-updating process. A large number of historical templates are stored in the template memory, and effective utilization of these templates is crucial to the robustness and accuracy of tracking. By using Equation (1), we can calculate the number of templates that need to be used in the next frame, and then use the template extraction mechanism to select high-quality and diverse templates from the historical ones, suitable for tracking in the next frame. The template extraction can be denoted as:

$${\tau }_i=\lfloor \frac{t}{N}\rfloor \times i,i=0,1,2,\dots ,N$$

(2)

$$T_j=ma{x}_{rag}\left[{\tau }_j,{\tau }_{j+1}\right],j=0,1,2,\dots ,N-1$$

(3)

$$T_{con}=concat(T_0,\dots ,T_{N-1})$$

(4)

where t is the total number of templates in the library, and N is the number of tracked templates. $ta{u}_j$ denotes the j-th segmentation point, and $ma{x}_{arg}\left[{\phi }_1,{\phi }_2\right]$ denotes the maximum value in the interval $\left[{\phi }_1,{\phi }_2\right]$. $concat(,)$ denotes the concatenation operation. Assuming that N templates are needed for the next frame, the specific extraction steps are as follows: (1) The initial template is necessary, as it contains primary information about the tracking target. Note that we exclude the last frame template, as it may introduce additional interfering information. (2) The templates in the template memory are sequentially divided into $N-1$ segments, and then the template with the highest tracking score is selected from each segment to serve as the optimal tracking template. This extraction mechanism can enhance the diversity of templates and extract high-quality templates, thus significantly improving the robustness of the tracker.

3.3. Search–Evaluation Network

3.3.1. Search Subnetwork

The search network utilizes STMTrack [9] as the baseline and GoogleNet [31] as the feature extraction network. As shown in Figure 5, the template image z and search image x undergo feature extraction to obtain their respective features $\phi \left(z\right)$ and $\phi \left(x\right)$. In the tracking process, the extracted historical template features are concatenated, and then the concatenated template features and the search image features are subjected to a cross-correlation operation to obtain the response map $R^{*}$. The process can be represented as:

$$R^{*}=concat\left({\phi }_1\left(z\right),\dots ,{\phi }_i\left(z\right)\right)★\phi \left(x\right)$$

(5)

where $\phi$ denotes the feature extraction operation, ★ denotes the cross correlation operation, and $concat(,)$ denotes the concatenation operation. Following the response map, the classification convolutional neural network and regression convolutional neural network are used to obtain the classification feature maps $R^{cls}$ and regression feature maps $R^{reg}$, respectively. The purpose of the classification branch is to classify the target to be tracked from the background. The classification branch includes a center-ness branch that boosts confidence for positions closer to the center of the image. Multiplying the classification response map $s_{cls}$ with the center-ness response map $s_{ctr}$ suppresses the classification confidence for locations farther from the center of the target, resulting in the final tracking response map. The purpose of the regression branch is to determine the distance from the center of the target to the left, top, right, and bottom edges of the target bounding box in the search image.

During the training phase of the network model, the classification branch is trained using the focal loss function, which can be expressed as:

$$L_{cls}=-{\alpha }_t{\left(1-p_t\right)}^{\gamma }log\left(p_t\right)$$

(6)

where ${\left(1-p_t\right)}^{\gamma }$ is a modulating factor, $\gamma$ is a tunable focusing parameter, and $p_t$ is the predicted probability of a positive or negative sample. The regression branch utilizes intersection over union (IoU) as a loss function, which can be expressed as:

$$L_{reg}=1-\frac{Intersection\left(B,B^{*}\right)}{Union\left(B,B^{*}\right)}$$

(7)

where B is the predicted bounding box and $B^{*}$ is its corresponding ground-truth bounding box. The center-ness branch uses a binary cross-entropy loss (BCE), which can be written as:

$$L_{cen}=c_{(i,j)}log{p}_{(i,j)}+(1-c_{(i,j)})log(1-p_{(i,j)})$$

(8)

where $p_{(i,j)}$ represents the center-ness score at point $(i,j)$. $c_{(i,j)}$ denotes the label value at position $(i,j)$. The final loss function is:

$$Loss=\frac{1}{N}\sum _{x,y}{L}_{cls}+\frac{\lambda }{N}\sum _{x,y}{L}_{reg}+\frac{\lambda }{N}\sum _{x,y}{L}_{cen}$$

(9)

where N denotes the number of point, $\lambda$ is the weight value.

3.3.2. Evaluation Network

In the evaluation network, we introduce three evaluation metrics: APCE, SCR, and TS. These three metrics will be jointly used for status identification in the block-prediction module (detailed in Section 3.4). In addition, APCE will also be fed back to the template extraction mechanism as prior information about the tracking quality. The tracking score will also be used to filter high-quality templates and place them in the template memory.

Average Peak Correlation Energy: Inspired by the LMCF [32] method, we introduce the APCE metric to evaluate the quality of the target region in the tracking response map. When the target is not disturbed by poor conditions, the APCE value is high, and the 3D heatmap of the tracking response map shows a stiff single peak shape. If the target is affected by a cluttered background or occlusion, the APCE value will be lower, and the 3D heatmap of the tracking response map will show a multipeak shape. The more the target is affected, the lower the APCE value. The APCE is calculated as follows:

$$APCE=\frac{{\left|F_{max}-F_{min}\right|}^2}{mean\left({\displaystyle \sum _{w,h}}{\left(F_{w,h}-F_{min}\right)}^2\right)}$$

(10)

where $F_{max}$ and $F_{min}$ represent the maximum value and minimum value of the response map of the tracking result, respectively. $F_{w,h}$ is the response values in the response map at position $(w,h)$.

Figure 6 shows the APCE values and response maps of the tracking results for different statuses. The APCE values are high under normal conditions, and the response map of the tracking result exhibits a stiff single peak form. However, when the target is affected by a cluttered background, the APCE value decreases, and the response map of the tracking result begins to show a trend of multiple peaks. When the target is occluded, the value of APCE decreases significantly, and the response map shows a low multipeak form. Our proposed DTUN is able to adapt to tracking situations where the impact on the target is small. However, when the target suffers from more serious impacts (e.g., occlusion), it will lead to tracking drift and further to complete failure in subsequent tracking if it cannot be effectively addressed. With the APCE metric, we can better measure the tracking results and determine the extent to which the target is affected by the external environment. This provides a more effective a priori guide for us to identify the tracking status in the block-prediction module.

Size Change Ratio: During the target-tracking process, the size of the target typically changes continuously, without any significant changes between consecutive frames. If the size of the target changes significantly in consecutive frames, external interference has likely caused the tracker to drift. Smaller influences are usually insufficient to cause tracking drift, and only severe influences (e.g., target occlusion) can cause tracking drift. To identify tracking drift in the block-prediction module, we introduce the size change ratio as an indicator. The size change ratio is expressed as:

$$SCR=\frac{\frac{2}{m}{\sum }_{i=\lfloor \frac{m}{2}\rfloor }^m{F}_{w\times h}^i}{F_{w\times h}^c}$$

(11)

where m denotes the number of templates in the template memory. $F_{w\times h}^i$ denotes the size of the i-th target template in the template memory, and $F_{w\times h}^c$ denotes the template size of the current frame.

Specifically, in order to obtain a more accurate SCR value, target sizes of poor quality are discarded. The average of the latter $m/2$ target sizes in the template memory is used to calculate the historical target size. Then, the ratio of the historical target size relative to the current target size is calculated. If this ratio falls within the threshold, the target size is considered to be within the normal range of variation. Otherwise, a sudden change in size is considered to have occurred. Such sudden size changes are often caused by tracking drift due to severe impacts (e.g., occlusion) on the target. Therefore, we use the SCR as a tracking status identification metric to assess whether the tracker is experiencing tracking drift. Considering that the rapid movement of the sensor may also cause the target size to change rapidly in a short period, which may lead to the misjudgment of the tracker, so we add the tracking score and the APCE score to support the judgment. Tracking drift is only considered to have occurred when all three conditions meet the threshold, which is then transmitted to the condition recognizer. This auxiliary judgment prevents the misjudgment of the tracker when the tracking condition is good and effectively improves the recognition accuracy of tracking drift.

Figure 7 displays the SCR variation curve during target tracking. The blue curve and bounding box represent the baseline algorithm, while the red curve and bounding box represent our method. Tracking drift occurs, and the size of the target bounding box gradually increases after the target is occluded starting from frame 1837. The blue curve and target bounding box indicate that the baseline algorithm has entered a tracking drift state. At frame 2148, when the SCR exceeds the threshold, it prompts our method to utilize the block prediction to relocate the target. As a result, the target is successfully tracked at frame 2343. This demonstrates the significance of the SCR metric in determining tracking drift and the effectiveness of our method in resolving such situations.

Tracking score: The tracking score is a direct measure of the quality of tracking results. In Siamese networks, the tracking score is calculated by multiplying the classification confidence and center-ness score. The classification confidence reflects the similarity between the template and the search region, while the center-ness reflects the distance between the target and the center point in the search image. The classification confidence $s_{cls}$ can be multiplied by the center-ness $s_{ctr}$ to suppress the score for positively classified targets that are far from the target center. The tracking score $s_{trc}$ can be expressed as: $s_{trc}=s_{cls}\times s_{ctr}$. To suppress large variations in the target scale, the scale penalty function is used to penalize large-scale variations in the target, and the process can be written as:

$$s^{*}=s_{trc}\times p^n=s_{trc}\times e^{k\times max(\frac{r}{r^{\prime }},\frac{r^{\prime }}{r})\times max(\frac{s}{s^{\prime }},\frac{s^{\prime }}{s})}$$

(12)

where k is a hyperparameter. r represents the proposal’s ratio of height and width, and $r^{\prime }$ represents that of the last frame. s and $s^{\prime }$ represent the overall scale of the proposal and last frame, respectively. s is calculated by:

$$(w+p)\times (h+p)=s^2$$

(13)

where w and h denote the width and height of the target, respectively, and p is the padding value. Additionally, to suppress scores away from the center, the response map is post-processed using a cosine window function. The final tracking score $TS$ can be expressed as:

$$TS=max\left(s^{*}\times (1-d)+H\times d\right)$$

(14)

where d is a hyperparameter, and H denotes the cosine matrix.

Finally, a response map with a tracking score of $5\times 5$ is obtained. The maximum confidence score in the response map indicates a higher probability that the target is present. If the target is affected by external factors, the maximum confidence score will be lower. The tracker records tracking results with confidence scores below the threshold and sends this information to the condition recognizer. The condition recognizer receives three evaluation scores from the evaluation network and performs further processing in the block-prediction module.

Algorithm 1 shows the details of the DTUN.

Algorithm 1: Dynamic template-updating network.

3.4. Block-Prediction Module

The block-prediction module comprises three parts: the condition recognizer, the Kalman filter, and block search. Its purpose is to address the tracking drift problem that arises from target occlusion and from targets moving out of the field of view. These problems are difficult to solve with traditional trackers. If the target is completely occluded and then reappears, tracking drift may occur, preventing the target from appearing in the tracker’s search area and resulting in tracking failure. Similarly, if the target moves out of view and then reappears, the tracker may not be able to locate the target in the search area. These challenging situations can cause tracking drift and subsequent tracking failures. Therefore, it is of great significance to solve the tracking drift problem in long-term tracking.

In our analysis of the search–evaluation network, we have determined the significance of the APCE, SCR, and TS in identifying tracking drift. The status recognizer identifies the tracking status for each frame during the tracking process. The Kalman filter is used to predict the target trajectory in the subsequent frame if the target is occluded. The block search is used to re-locate the target if the predictor fails to predict the target within 20 frames. The block search network has two modes: block search and expanded block search. If the block search fails to locate the target, the expanded block search will be used. Additionally, we check if the target moves out of the field of view during each frame’s tracking process. If the target moves out of the field of view, the expanded block search will be used to relocate it.

3.4.1. Motion Prediction

In the field of target tracking, most trackers do not include a target prediction process. This can result in tracking drift when the target is disturbed by external factors, such as occlusion. This is particularly problematic for long-term tracking, as tracking drift can lead to complete failure of subsequent tracking. Therefore, it is crucial to introduce a target prediction process to address tracking drift caused by target occlusion. In airborne tracking, due to the characteristics of distant sensors and small targets, the motion process before the target is occluded and can be regarded as approximately linear.

The Kalman filter [6] is used in the target prediction stage. The state and observation equations of the Kalman filter can be expressed as:

$$x_k=A{x}_{k-1}+w_{k-1}$$

(15)

$$z_k=H{x}_k+v_k$$

(16)

where k denotes the moment of the kth frame, $x_k$ is the state vector, and $z_k$ is the observation vector. H is the observation matrix, and $H=I$, where I is the unit matrix. A is the state transition matrix. $w_{k-1}$ and $v_k$ are the process error and observation error, respectively, and are assumed to be subject to Gaussian distributions with covariance matrices Q and R. We set $Q=I\times 0.1$ and $R=I$. During the process of object tracking, the updating process takes up most of the time, while the prediction process takes up very little time. Therefore, it can be assumed that the size of the target will not change significantly in a short period. The state space of the target is set as follows:

$$X={[x,y,w,h,v_x,v_y]}^T$$

(17)

where x, y denote the center coordinates of the target, and w and h denote the width and height of the target, respectively. $v_x$ and $v_y$ denote the change ratio of the center coordinates of the target, respectively.

The Kalman filtering computational steps consist of the prediction process and the update process. The prediction process focuses on predicting the state and error covariance variables. It can be expressed as:

$${\widehat{x}}_k^{-}=A{\widehat{x}}_{k-1}^{-}$$

(18)

$${\widehat{p}}_k^{-}=A{\widehat{p}}_{k-1}{A}^T+Q$$

(19)

where $P_{k-1}$ is the error covariance of the prediction for the $k-1$st frame. The state update phase includes the optimal estimation of the system state and the update of the error covariance matrix. It can be expressed as:

$${\widehat{x}}_k={\widehat{x}}_k^{-}+K_k(z_k-H{\widehat{x}}_k^{-})$$

(20)

$$p_k=(I-K_{k}H)p_k^{-}$$

(21)

$$K_k=p_k^{-}{H}^T{(H{p}_k^{-}{H}^T+R)}^{-1}$$

(22)

where $K_k$ is the Kalman filter gain of the kth frame. $z_k$ is the actual observation of the kth frame.

3.4.2. Block Search

The Kalman filter can effectively solve most target occlusion problems with near-linear motion during target prediction. However, the motion trajectory of the target after occlusion may sometimes exhibit significant nonlinearity. To deal with this situation, we first predict the approximate motion direction of the target using the Kalman filter, and then perform a block search to further locate the target. In target tracking from the UAV perspective, the target is relatively small and moves slowly. Therefore, if the Kalman filter fails to locate the target within 20 frames, we will use block search to further search for it.

Figure 8 illustrates the process of block search. Figure ➀ shows the process of target prediction using Kalman filtering after the target is occluded. When the target is occluded, its motion trajectory exhibits a large nonlinearity, which differs from the predicted trajectory, resulting in prediction failure. Therefore, the block search is necessary for further target searching. In Figure ➁, the red dot indicates the predicted position of the Kalman filter, and the green rectangular box indicates the original search area. The target is not within the search area due to the small size of the search area. Therefore, in the block search module, the search area will be expanded first. The process can be expressed as follows:

$$\begin{array}{cc}\hfill & s_w=\frac{\sqrt{t_w^{i-1}}}{b_w}\times 289\times n\hfill \\ \hfill & s_h=\frac{\sqrt{t_h^{i-1}}}{b_h}\times 289\times n\hfill \end{array}$$

(23)

where $s_w$ and $s_h$ represent the width and height of the search area, respectively. n is the magnification factor. In block search, the value of n is 3 when tracking drift occurs because the object moves out of view, and 2 when tracking drift occurs in other situations. $b_w$ and $b_h$ are the width and height of the initially sampled image, obtained as follows:

$$b_w=\frac{t_w^1}{t_s},b_h=\frac{t_h^1}{t_s}$$

(24)

$$t_s=\frac{\sqrt{t_w^1\times t_h^1\times p^2}}{289}$$

(25)

where $t_w^1$ and $t_h^1$ are the width and height of the initial target, respectively. p is the search region factor, typically set to 4. Due to the enlarged search region being larger than the target, accurately localizing the target can be difficult if searching for it directly within the region. Figure ➂ shows the response map results of searching in this way. As the response map shows, this direct search makes target localization difficult. Therefore, as shown in Figure ➃, we segment the enlarged search area into $3\times 3$ blocks and search for the target in each block. Block search effectively overcomes the challenge of accurately localizing small targets in a larger search area. Figure ➄ shows the response map result of the block search, which accurately localizes the target in the block image containing it. The target region has the highest response value score, while the other blocks have significantly lower scores.

Figure ➅ illustrates the calculation of the target center position. The center coordinates $(x,y)$ of the target in the original search image are calculated as follows:

$$\begin{array}{cc}\hfill & x=(c_w-\lfloor \frac{\sqrt{C}}{2}\rfloor )\times S+x^{\prime }-x_1+x_0\hfill \\ \hfill & y=(c_h-\lfloor \frac{\sqrt{C}}{2}\rfloor )\times S+y^{\prime }-y_1+y_0\hfill \end{array}$$

(26)

where C represents the number of block subimages. $(c_w,c_h)$ represents the coordinate position of the block subimage where the target is located, with the horizontal and vertical axes being represented by w and h respectively. S denotes the width and height of the square block subimage. $(x_0,y_0)$ represents the coordinates of the center position of the search area $x_0=\frac{s_w}{2}$, $y_0=\frac{s_h}{2}$. $(x_1,y_1)$ represents the coordinates of the center position of the block subimage where the maximum score of the target is located, $x_1=y_1=\frac{S}{2}$. $(x^{\prime },y^{\prime })$ represents the coordinates of the position with the maximum score in the block search response map. The width w and height h of the target bounding box are calculated using the following equations:

$$\begin{array}{cc}\hfill & w=(1-r)w_{i-1}+r\times w^p\hfill \\ \hfill & h=(1-r)h_{i-1}+r\times h^p\hfill \end{array}$$

(27)

where $w^p$ and $h^p$ denote the width and height of the predicted target, respectively. $w_{i-1}$ and $h_{i-1}$ denote the width and height of the target in the previous frame, respectively. r is derived by:

$$r=p^n\left(argmax\left\{s_{trc}\right\}\right)\times max\left\{s_{trc}\right\}\times q$$

(28)

where q is a hyperparameter, $s_{trc}$ is the tracking score, and $p^n$ represents the scale penalty function.

Tracking drift can also be caused by targets moving out of view, which makes it difficult for the tracker to localize the target due to the uncertainty in the center of the search area. To address this issue, the enlarged block search method is utilized. To bring the search area closer to the target’s reappearance, we calculate the average of the target’s historical trajectory coordinates and use it as the center of the new search area. The search area is expanded to three times the original size, and a $5\times 5$ block search is used to search the target.

When the target moves out of the field of view, the tracker is unable to track the target correctly. During this time, the tracker will perform expanded block search every 20 frames. (It should be noted that the Kalman filter is not effective in acquiring the target when it moves out of view or reappears. In this case, only block search will be employed, and the Kalman filter will not be used). It is not until the target reappears within the field of view (e.g., the target changes its direction of motion or the UAV turns its camera towards the target) that the expanded block search can correctly localize the target. Given the uncertainty surrounding the moment of target reappearance, the expanded block search is conducted every 20 frames, which avoids repeated search calculations and improves the tracking efficiency.

Figure 9 shows the overlap rate graphs of the proposed method compared to the baseline algorithm in the UAV20L long-term tracking dataset. The overlap ratio is defined as the intersection-over-union ratio between the predicted target bounding box and the ground truth. A low overlap ratio in the graph indicates tracking drift, which may be caused by occlusion or the disappearance of the target. Tracking drift leads to an almost complete failure of subsequent tracking, as shown by the blue curve in the figure. Our method (red curve) effectively re-tracks the target after tracking drift, demonstrating its effectiveness in solving the tracking drift problem.

Algorithm 2 shows the complete details of the MPBTrack.

Algorithm 2: The proposed MPBTrack algorithm.

4. Experiments

This section presents the experimental validation of the MPBTrack algorithm. Section 4.1 describes the details of model training and experimental evaluation. In Section 4.2, we perform ablation experiments for analysis. Section 4.3, Section 4.4, Section 4.5, Section 4.6 and Section 4.7 present quantitative experimental results, and Section 4.8 presents a qualitative experimental analysis.

4.1. Experimental Details

The experiments were conducted on an Intel(R) Xeon(R) Silver 4110 2.10 GHz CPU and NVIDIA GeForce RTX2080 GPU platform. We use GoogLeNet [31] as the backbone. The algorithmic model is trained on the TrackingNet [33], LaSOT [34], and GOT-10k [35] training sets, as well as the ILSVRC VID [36], ILSVRC DET [36], and COCO [37] datasets. The model was trained with 20 epochs using the SGD optimizer. The learning rate increased from $1\times {10}^{-2}$ to $8\times {10}^{-2}$ with a warmup technology in the first epoch and then decreased from $8\times {10}^{-2}$ to $1\times {10}^{-6}$ with a cosine annealing learning rate schedule. The momentum and weight decay rate were set to 0.9 and $1\times {10}^{-4}$, respectively. During the inference phase, the block subimages were uniformly cropped to $289\times 289$ to be fed into the feature extraction network. The block search was performed every 20 frames after tracking drift, while the expanded block search was performed every 15 frames.

To demonstrate the effectiveness of our approach in tracking an object on UAV platforms, quantitative and qualitative experiments were conducted on five aerial object tracking datasets: UAV20L [15], UAV123 [15], UAVDT [16], DTB70 [17], and VisDrone2018-SOT [18]. The evaluation of the experimental results included overall performance assessment and evaluation of the challenge attributes. The metrics used to evaluate the experiment were success and precision. Success is expressed as the area under curve (AUC) plotted as the overlap at different thresholds. The overlap is the ratio of the intersection over union (IoU) between the predicted bounding box region A and the ground truth region B. It can be expressed as $Overlap=\frac{|A\cap B|}{|A\cup B|}$, where $|.|$ denotes the area of the region. Due to the potential for inconsistent evaluation results with different overlap thresholds, AUC was used to rank the success scores of the trackers. Precision measures the distance between the center position of the predicted bounding box and the center position of the ground truth. Traditionally, this distance threshold is set to 20 pixels.

4.2. Ablation Experiments

To validate the effectiveness of the proposed modules, we performed ablation experiments on the UAV123 and UAV20L datasets. Our tracking framework comprises two parts: the dynamic template-updating network (DTUN) and the block-prediction module (BPM). We added DTUN, BPM, and DTUN+BPM to the baseline algorithm to validate the effectiveness of each module separately.

Table 1. Analysis of ablation experiments using DTUN and BPM on the UAV123 and UAV20L datasets.

Module	UAV20L		UAV123
	Success	Precision	Success	Precision
Baseline	0.589	0.742	0.647	0.825
Baseline + DTUN	0.617	0.784	0.651	0.834
Baseline + BPM	0.683	0.875	0.651	0.841
Baseline + DTUN + BPM	0.706	0.905	0.656	0.842

presents the experimental results for the UAV20L and UAV123 datasets. Our method demonstrates a 16.5% and 18.9% improvement in success and precision, respectively, compared to the baseline on the UAV20L dataset. Specifically, DTUN shows a 4.4% and 5.3% improvement in success and precision, respectively, while BPM shows a 14.3% and 16.0% improvement in success and precision, respectively. The ablation experiments demonstrate that both the proposed DTUN and BPM achieve excellent performance improvement. Our method provides a significant improvement in solving the tracking drift problem in long-term airborne object tracking.

The target in the UAV view is small and moves slowly over a short period. To avoid reduced tracking speed, the block search should not be used for every frame when tracking drift occurs. Instead, appropriate time intervals should be used to balance the success rate and speed.

Table 2. Ablation experiments with different search intervals.

Interval Frames	UAV20L			VisDrone2018-SOT
	Success	Precision	FPS	Success	Precision	FPS
10	0.697	0.893	32.4	0.637	0.822	27.9
15	0.700	0.898	32.6	0.637	0.823	28.3
20	0.706	0.905	43.5	0.665	0.864	43.4
25	0.689	0.811	32.8	0.633	0.818	26.7

shows the changes in tracking performance and speed using different time intervals. Tracking speed may decrease without improving performance if a smaller time interval is used. Conversely, using a larger time interval can decrease tracking performance. The decrease in tracking speed may be due to unresolved tracking drift. The experimental analysis determined that the frame interval suitable for tracking in UAVs is 20.

Section 3.3.2 discusses the importance of the SCR metric in determining tracking drift conditions in UAV video object tracking. Choosing a suitable SCR threshold can improve the accuracy of recognizing tracking conditions.

Table 3. Experimental results for various SCR thresholds on the UAV20L dataset.

SCR	2.0	2.4	2.8	3.2	3.6
Success	0.696	0.704	0.706	0.695	0.688
Precision	0.892	0.901	0.905	0.890	0.882

demonstrates the impact of different SCR values on tracking performance. The table shows that choosing a threshold that is too large or too small results in misjudging the tracking drift condition and further leads to the degradation of tracking performance.

4.3. Experiments on UAV20L Benchmark

UAV20L [15] is a long-term object tracking dataset, which contains 20 long-term sequences with an average of 2933 frames per sequence. The dataset comprises approximately 58k frames, with the longest sequence consisting of 5527 frames. It presents a significant challenge for the tracker’s long-term tracking capabilities. The challenging attributes of UAV20L include aspect ratio change (ARC), background clutter (BC), camera motion (CM), fast motion (FM), full occlusion (FO), illumination variation (IV), low resolution (LS), out of view (OV), partial occlusion (PO), scale variation (SV), similar object (SO), and viewpoint change (VC). Precision and success serve as evaluation metrics for this dataset.

Figure 10 shows a comparison of our method with other competitive methods, such as TaMOs [38], ARTrack [39], RTS [22], SLT-TransT [40], OSTrack [26], and TransT [41] based on Transformer architecture, and STMTrack [9], SiamBAN [42], SiamCAR [21], and SiamGAT [10] based on Siamese network derivation. Our method outperforms the baseline STMTrack by improving the success by 19.1% and improving the precision by 20.8%. In terms of success, our method surpasses the second-ranked OSTrack by 2.3% and achieves faster tracking speeds. The results demonstrate the superior tracking robustness and accuracy of our method for long-term tracking on UAVs, and the tracking speed far exceeds the real-time requirements.

Figure 11 and Figure 12 show the success and precision plots for attribute evaluation, respectively. The proposed method achieves excellent performance in both attribute evaluations. In terms of success, the scores are ARC (0.683), BC (0.613), CM (0.698), FO (0.599), IV (0.663), OV (0.703), PO (0.689), SV (0.706), and VC (0.742); in terms of precision, the scores are ARC (0.881), BC (0.870), CM (0.900), FO (0.857), IV (0.856), OV (0.898), PO (0.894), SV (0.900), and VC (0.906). The attribute evaluations demonstrate the remarkable performance of the proposed DTUN and BPM in dealing with various complex situations, proving the effectiveness of our approach.

4.4. Experiments on UAV123 Benchmark

UAV123 [15] consists of 123 low-altitude video sequences captured by UAVs, with a total of more than 110K frames. This dataset poses significant challenges for object tracking due to its inclusion of numerous video sequences with complex conditions.

Table 4. Experimental results comparing our method with other methods on the UAV123 dataset. Trackers are ranked based on their success scores.

Tracker	Ta-MOs [38]	HiFT [25]	TC-Track [27]	PAC-Net [45]	Siam-CAR [21]	Light-Track [44]	CNN-InMO [46]	Siam-BAN [42]	Siam-RN [11]	Auto-Match [47]	SiamPW-RBO [43]	Siam-GAT [10]	STM-Track [9]	MPB-Track
Succ.	0.571	0.589	0.604	0.620	0.623	0.626	0.629	0.631	0.643	0.644	0.645	0.646	0.647	0.656
Prec.	0.791	0.787	0.800	0.827	0.813	0.809	0.818	0.833	-	-	-	0.843	0.825	0.842

outlines the results of the comparison experiments between our method and other competitors, such as TaMOs [38], HiFT [25], STMTrack [9], SiamPW-RBO [43], and LightTrack [44]. Compared with the baseline method, our method delivers competitive performance, improving success and precision by 1.4% and 2.1%, respectively.

4.5. Experiments on UAVDT Benchmark

UAVDT [16] is a dataset for object tracking and detection captured by UAVs, which contains 50 video sequences with moving vehicles as its targets of interest. This dataset encompasses various challenges such as long-term tracking (LT), large occlusion (LO), object blur (OB), small object (SO), background clutter (BC), camera rotation (CR), object motion (OM), camera motion (CM), illumination variation (IV), and scale variation (SV).

Figure 13 illustrates the experimental results of our method in comparison with other methods (TaMOs [38], ARTrack [39], DropTrack [12], ROMTrack [48], TransT [41], LightTrack [44], SiamCAR [21], and STMTrack [9]) on the UAVDT dataset. Our approach demonstrates superior performance compared to other Transformer-based large model structures and Siamese network trackers. Compared to the baseline STMTrack, our method shows an improvement of 2.0% and 1.2% in terms of success and precision, respectively. Figure 14 shows the success plots for attribute evaluation. Our method secures excellent success scores across various challenging attributes, including BC (0.608), CM (0.651), IV (0.695), LO (0.615), LT (0.767), OB (0.677), OM (0.687), SV (0.678), and SO (0.675). The evaluation results highlight the excellent performance of our method in adapting to various complex conditions on UAVs.

4.6. Experiments on DTB70 Benchmark

DTB70 [17] dataset is a highly diverse dataset that consists of 70 videos captured by UAVs. Figure 15 shows the success and precision results of the proposed method on the DTB70 dataset. Our method achieves an excellent score of 0.670 in terms of success and outperforms the baseline method by 2.3% and 2.3% in terms of success and precision. Compared to Transformer architecture-based methods such as TaMOs [38], ROMTrack [48], ARTrack [39], TransT [41], and OSTrack [26], our method achieves better tracking results without requiring large model parameters and computational resources. Similarly, compared to Siamese network-based methods such as STMTrack [9], SiamAPN [49], SiamGAT [10], and SiamCAR [21], our method achieves significant performance improvement in terms of success and precision. In summary, our Siamese network-based tracker achieves a higher success score than the Transformer-based tracker, and it demonstrates significant competitiveness in tracking from UAVs.

4.7. Experiments on VisDrone2018-SOT Benchmark

VisDrone2018-SOT [18] contains 35 video sequences totaling approximately 29k frames. Figure 16 shows the success rate plot and accuracy plot of some competitive trackers (e.g., TaMOs-R50 [38], ARTrack [39], ToMP-101 [50], CNNInMo [46], and STMTrack [9]) on the VisDrone2018-SOT dataset. Our method outperforms the baseline method by 5.6% and 4.9% in terms of success and precision, achieving significant performance improvements. The success vs. speed and precision vs. speed results of the trackers are shown in Figure 17. Compared to the Transformer architecture-based and Siamese network-based approaches, our method not only achieves leading results in terms of success, but also achieves a real-time speed of 43 FPS, which is a great advantage for object tracking from UAVs.

4.8. Qualitative Analysis

In order to visually compare the tracking performance of the proposed method, a qualitative comparison with the ground truth as well as other methods (e.g., ARTrack [39], TCTrack [27], TransT [41], SiamCAR [21], STMTrack [9]) is performed. The study analyzes seven sequences from UAV20L and VisDrone2018-SOT datasets. Figure 18 shows the sequences in the following order from top to bottom: car1, group2, person14, uav180, uav1, group1, and uav93.

(1)

car 1: This sequence presents two challenges: occlusion of the object and the object moving out of view. Some trackers experience tracking drift after the object is occluded, and more trackers experience tracking drift after the target moves out of the field of view. Our method, however, successfully tracks the object again after both challenges.

(2)

group2, person14, uav180: These three sequences present the challenge of object occlusion. The visualization results demonstrate that when the object is occluded, only our tracker successfully tracks it, while the other trackers experience tracking drift or track the wrong object for a prolonged period in the subsequent frames. This highlights the significant advantages of our tracker in long-term tracking and handling challenging situations.

(3)

uav1: The uav1 sequence involves the challenges of camera motion, background clutter, and fast object motion. The simultaneous interference of these three challenges in the tracking of this uav1 sequence leads to tracking drift in multiple trackers. However, our tracker relies on dynamic template updating and block search to remain relatively resistant to interference from complex conditions.

(4)

group1 and uav93: These two sequences present challenges with similar targets and object occlusion. When mutual occlusion between objects occurs, other trackers appear to track the wrong object. Our tracker can still accurately track the correct object in this challenging scenario.

5. Conclusions

This paper proposes a visual target-tracking algorithm based on object prediction and block search, aiming to solve the problem of tracking drift in UAV view. Specifically, when the tracker experiences tracking drift due to object occlusion or moving out of view, our approach predicts the motion state of the object using a Kalman filter. Then, the proposed block search module efficiently searches for the tracking of the drifting target. In addition, to enhance the adaptability of the tracker in changing scenarios, we propose a dynamic template update network. This network employs the optimal template strategy based on various tracking conditions to improve the tracker’s robustness. Finally, we introduce three evaluation metrics: APCE, SCR, and TS. These metrics are used to identify tracking drift status and provide prior information for object tracking in subsequent frames. Extensive experiments and comparisons with many competitive algorithms on five aerial benchmarks, namely, UAV123, UAV20L, UAVDT, DTB70, and VisDrone2018-SOT, have demonstrated the effectiveness of our approach in resisting tracking drift in a complex UAV viewpoint environment, and it achieved a real-time speed of 43 FPS.