TPTTrack: A Lightweight Transformer-Based Tracker with Target Prompt Token and Hierarchical Attention Decoupling

Abstract

Although Transformer-based trackers have achieved impressive tracking accuracy owing to their strong capability for global context modeling, they still suffer from substantial model complexity and high computational latency. To address these limitations, this paper proposes a lightweight Transformer-based single object tracking method, termed TPTTrack. Specifically, a target-state-guided prompt token is introduced and concatenated with the template and search region features. Constructed from compact target-state information, this token guides cross-region feature interaction toward target-relevant information, thereby enhancing tracking robustness. Furthermore, a hierarchical attention decoupling mechanism is developed to improve shallow feature extraction efficiency and reduce redundant self-attention in deeper layers. In addition, a lightweight autoregressive prediction module is employed for dynamic target-state modeling and efficient state estimation. The results of experiments such as 65.0% AO on GOT-10k and 68.5% precision on LaSOT demonstrate an effective balance between accuracy and efficiency that provides a good trade-off between performance and computational cost.

1. Introduction

Target tracking technology aims to continuously locate specific targets from video sequences and is the core of real-time applications such as autonomous driving, drone navigation, mobile augmented reality and intelligent monitoring [1,2]. With the increasing complexity of deep learning models, although mainstream trackers have excellent accuracy on benchmark datasets, their huge computational load and memory consumption seriously hinder the real-time deployment of edge devices with limited resources, such as mobile, embedded platform and Internet of Things devices [3]. Lightweight design and real-time performance remain critical bottlenecks in the practical deployment of visual object tracking technologies. The main challenges are threefold: (1) Complex deep architectures such as Transformer [4] and large-scale CNN [5] incur high inference latency, making it difficult to meet the high frame rate requirements of video streams. (2) The large number of parameters and high computational complexity hinder deployment on mobile platforms equipped with CPUs or low-power GPUs. (3) Real-time model updating strategies in online tracking frameworks, such as template refinement, often incur additional latency, thereby introducing new performance bottlenecks. Therefore, it is of both urgent practical need and significant research value to design lightweight and real-time tracking models that effectively balance accuracy, speed and model size.

In the field of single object tracking, the existing methods can be divided into two categories: correlation-filter-based trackers and deep learning-based trackers.

Correlation-filter-based tracking methods received considerable attention in early studies due to their high computational efficiency and favorable real-time performance. Their core idea is: in the target initialization frame, by modeling the features of the target area and training the filter, the features of the same area are mapped to the frequency domain in the subsequent frames, and the response map is obtained by the relevant operation, and the area with the highest score is selected as the target position. Typical methods such as MOSSE [6], KCF [7] and CSK [8] accelerate the correlation operation by fast Fourier transform, which significantly improves the running speed.

With the development of deep learning technology, target tracking algorithms based on deep neural networks have gradually become a major research focus. These methods generally provide stronger feature representation capability and higher tracking accuracy, thereby further advancing the development of the field. Deep learning-based tracking methods can be broadly divided into two categories: one is the tracking algorithm based on a Siamese network, which uses a two-branch structure to extract the features of the target template and the search area respectively, and generates the response map through the similarity measure. The area with the largest response value is regarded as the position of the target in the current frame. Representative works include SiamFC [9], SiamRPN [10], SiamMask [11] and Siam-DW [12], which effectively combine end-to-end training with fast inference mechanisms, achieving a good balance between accuracy and efficiency. Another emerging category consists of Transformer-based tracking methods [13], which have gained increasing attention in recent years. By introducing the self-attention mechanism, the Transformer architecture overcomes the limitations of conventional convolutional neural networks (CNNs) in modeling long-range dependencies and capturing global contextual information. It enables the modeling of interdependencies across spatial regions simultaneously, providing a more global representation of the target. This significantly enhances tracking robustness in challenging scenarios such as cluttered backgrounds and rapid target motion. Methods such as TransT [14], OSTrack [15], STARK [16], SwinTrack [17] and SeqTrack [18] have demonstrated remarkable improvements in both accuracy and generalization, achieving state-of-the-art performance on several benchmark datasets including GOT-10k [19] and LaSOT [20].

In recent years, with the continuous development of single object tracking research, the design philosophy of related methods has gradually evolved from merely improving feature representation capability to several more representative core research directions. Temporal modeling [21] emphasizes the use of historical cross-frame state information to characterize the dynamic evolution of the target in continuous video sequences, thereby improving long-term tracking stability. Motion-aware tracking [22] focuses on incorporating target displacement, velocity, and motion pattern information to enhance localization capability under fast motion and drastic pose variations. Target-aware modeling [23] explicitly introduces target prior information to improve the network’s discriminative ability toward target-relevant regions, thereby reducing the influence of background interference and confusion caused by similar distractors. Query-guided tracking [24] employs query vectors or prompt representations to guide feature interaction and target localization, thus enhancing the model’s ability to focus on critical information. In addition, multimodal tracking [25] further improves robustness in complex environments by integrating visible, infrared, or other sensing modalities. These research directions indicate that the core of current single object tracking research lies not only in improving representation capability itself, but also in more effectively modeling target priors, temporal states, and cross-region interaction relationships. Under this perspective, how to jointly consider target prior guidance, temporal state propagation, and efficient feature interaction under lightweight constraints has become an important issue that deserves further investigation in current single object tracking research.

While the application of Transformer architectures has significantly improved object tracking performance, most Transformer-based trackers suffer from large model sizes and high computational demands [26]. As a result, they often achieve high tracking accuracy at the cost of runtime efficiency, which becomes a critical limitation for practical real-time applications on edge devices.

To address the above issues, this paper proposes a lightweight tracking model, termed TPTTrack, based on a single-stream Transformer. Rather than being a simple combination of multiple modules, the proposed model deeply couples explicit target prior injection, hierarchical attention allocation, and autoregressive state evolution within a unified architecture. In this way, it forms a lightweight framework for single object tracking and achieves simultaneous improvements in runtime efficiency and tracking robustness. The main contributions of this paper can be summarized as follows:

• Target-State-Guided Prompt Token: A target-state-guided prompt token is introduced to explicitly encode target prior information and inject it into the feature interaction process between the template and search regions. Different from approaches that introduce an additional high-dimensional appearance branch, the proposed prompt token is constructed from compact target-state cues. This design enables the network to focus more effectively on target-relevant regions during cross-region feature aggregation, while incurring only negligible computational overhead.
• Hierarchical Decoupled Attention Mechanism: A hierarchical decoupled attention mechanism is designed in this work. In the shallow layers, mixed attention is retained to strengthen feature extraction and representation learning. In the deep layers, redundant intra-region self-attention is removed, while key cross-region interactions are preserved for target-oriented information propagation. This design reduces unnecessary computation and effectively balances tracking accuracy and inference efficiency.
• Lightweight Autoregressive State Generation Module: A lightweight autoregressive state generation module is developed to model the temporal dynamics of target states. By recursively propagating target-state information across consecutive frames, the proposed module enables continuous temporal modeling of target motion and appearance variations. Under challenging conditions such as fast motion, occlusion, and severe scale changes, this mechanism effectively improves the accuracy and stability of tracking.

2. Related Work

In recent years, from the perspective of method design, single object tracking research has gradually developed several technical routes that are more closely aligned with core model innovations, mainly including target-aware methods, efficiency-oriented methods, and temporally aware methods. Target-aware methods emphasize the explicit introduction of target priors, query representations, or prompt information to guide the feature interaction process toward target-relevant regions, thereby improving discriminative capability under challenging conditions such as complex backgrounds and similar distractors. Efficiency-oriented methods are mainly designed for resource-constrained scenarios, and focus on reducing model size and computational complexity through lightweight backbone design, attention pruning, structural re-parameterization, or detection-head simplification, so as to balance inference speed and tracking accuracy. Temporally aware methods emphasize the modeling of cross-frame target motion states and historical information, and enhance target localization stability in continuous video scenarios through state propagation, temporal feature fusion, or dynamic update mechanisms. Compared with the traditional taxonomy based on methodological paradigms, the above classification more directly reflects the core design ideas of current single object tracking methods, and is also more closely related to the target prior guidance, lightweight interaction modeling, and state propagation mechanism emphasized in this work.

Among these research directions, efficiency-oriented methods have attracted extensive attention in recent years because they are more closely related to the practical deployment requirements of resource-constrained scenarios. The core objective of such methods is to maintain high tracking accuracy and real-time performance while reducing model size and computational complexity as much as possible. Driven by this objective, a variety of lightweight trackers have been proposed. LightTrack [27] introduced a neural architecture search framework specifically designed for single object tracking. By constructing a lightweight search space composed of depthwise separable convolutions and inverted residual modules, it jointly optimizes both the backbone and the prediction head. On this basis, LightTrack combines a one-shot weight-sharing strategy with an evolutionary search algorithm to dynamically identify the optimal architecture under a fixed computational budget, thereby achieving a favorable balance between tracking accuracy and inference efficiency. Such a search-based design effectively improves the lightweight nature of the model and enhances its potential for applications on resource-constrained devices. However, the method still relies primarily on conventional convolution operators, making it difficult to fully capture long-range dependencies, and its robustness remains limited in challenging scenarios such as complex backgrounds and distractor interference.

FEAR [28] proposed a lightweight tracking method that introduces a single-parameter dual-template update mechanism together with a pixel-level feature fusion module, with the aim of improving adaptability to target appearance variation under limited computational overhead. This design enhances the flexibility of template updating to a certain extent and improves local feature fusion. Nevertheless, its overall feature extraction capability remains limited, leading to unstable tracking performance when the target undergoes occlusion, fast motion, or significant deformation. Moreover, without an explicit global attention mechanism, the method cannot effectively model cross-region associations and global dynamic context, thereby limiting its performance in complex scenarios.

MixFormerV2 [29] is a lightweight tracking solution built upon a pure Transformer architecture. By replacing the conventional detection head with a learnable prediction token, it enables end-to-end target-state prediction and thus exhibits a highly integrated design. Compared with conventional convolution-based trackers, MixFormerV2 can better exploit the global modeling capability of Transformers and therefore shows advantages in target representation learning. However, the adoption of a unified attention strategy in shallow layers preserves a considerable amount of redundant interactions, which introduces unnecessary computational overhead. In addition, the method still suffers from a generalization bottleneck in key-region modeling. Furthermore, its strong performance is achieved partly at the cost of a relatively large parameter scale, which limits its applicability in ultra-low-power scenarios.

HiT [30] fuses features from different levels through a bridging module and employs dual-graph positional encoding to enhance spatial relationship modeling, thereby significantly improving tracking accuracy. By effectively exploiting complementary information across multiple feature levels, the method demonstrates strong representation capability in challenging scenes. Nevertheless, the backbone of HiT remains relatively complex and substantial overlap exists in the inter-layer computational graph, making structural decoupling, redundancy pruning and efficient optimization difficult. As a result, from the perspective of lightweight design, its computational overhead remains relatively high, leaving limited room for further efficiency improvement.

MVT [31] adopts MobileViT [32] as the backbone and generates target localization features by integrating template and search region information, exhibiting favorable lightweight characteristics. Benefiting from the compromise offered by MobileViT between local convolutional modeling and global representation learning, the method achieves a certain balance between parameter efficiency and tracking accuracy. However, the overall performance of MVT depends heavily on the representational capability of MobileViT itself, while the interactions among modules are not sufficiently fine-grained. Consequently, it is difficult to fully exploit the deep associations between the template and the search region, which in turn limits further improvements in tracking robustness.

Overall, existing lightweight tracking methods have made notable progress in reducing model complexity and improving runtime efficiency. However, they still generally suffer from insufficient long-range dependency modeling, redundant attention computation and limited robustness in challenging scenarios. Therefore, how to further enhance global modeling capability and key-region interaction efficiency while preserving lightweight characteristics remains an important issue in current research on lightweight single object tracking.

3. Method

This section will introduce how our model TPTTrack method implements the tracking process and related training information.

3.1. Model Architecture

The overall architecture of TPTTrack is shown in Figure 1. The template image $I_Z$ and the search image $I_X$ are partitioned into non-overlapping patches of fixed size $P\times P$ to achieve a balance between tracking accuracy and computational cost. After patch partitioning, each image patch is flattened and projected by a linear embedding into a unified feature space, yielding the visual tokens of the template branch and the search branch, respectively, which are then used as the inputs to the subsequent Transformer encoder. In addition, a target prompt token is introduced and concatenated with the visual tokens of the template and search branches along the token dimension to form a unified input sequence, which is then fed into the Transformer encoder. It should be emphasized that the target prompt token is not an additional high-dimensional appearance branch, but rather a low-dimensional prior-guided token constructed from target-state information. Its purpose is to explicitly inject prior state cues, such as the target position and scale, into the feature interaction process.

Figure 1. Overall architecture of the proposed TPTTrack framework. (a) Overall architecture of TPTTrack. (b) Shallow attention module. (c) Deep attention module.

Let the target bounding box state at frame $t$ be defined as

$$b_t=(x_t,y_t,w_t,h_t)$$

(1)

where $(x_t,y_t)$ denotes the top-left coordinate of the target box, and $w_t$ and $h_t$ denote its width and height, respectively. To unify the state representation, it is first transformed into a corner-based form:

$${\tilde{b}}_t=(x_t,y_t,x_t+w_t,y_t+h_t)$$

(2)

Subsequently, the coordinates are constrained within a predefined range to obtain a clipped state representation:

$${\bar{b}}_t=\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}({\tilde{b}}_t,b_{\mathrm{m}\mathrm{i}\mathrm{n}},b_{\mathrm{m}\mathrm{a}\mathrm{x}})$$

(3)

where $\mathrm{c}\mathrm{l}\mathrm{i}\mathrm{p}(\cdot )$ denotes the truncation operation, and $b_{\mathrm{m}\mathrm{i}\mathrm{n}}=-\frac{R}{2}+\frac{1}{2},b_{\mathrm{m}\mathrm{a}\mathrm{x}}=\frac{R}{2}+\frac{1}{2}$ represent the lower and upper bounds allowed by the discrete encoding scheme, respectively. This step ensures that the continuous bounding box coordinates can be mapped to valid indices after discrete quantization, while also reducing the influence of abnormal coordinates on the stability of prompt-token encoding.

On this basis, the continuous coordinates are discretized into four state indices:

$$q_t=(q_t^1,q_t^2,q_t^3,q_t^4)$$

(4)

where the $i$-th state index is defined as

$$q_t^i=\lfloor ({\bar{b}}_t^i+\mathsf{\Delta })(N-1)\rfloor ,i\in \{1,2,3,4\}$$

(5)

in which $N$ denotes the number of discrete bins for each coordinate dimension, $\mathsf{\Delta }=\frac{R}{2}-\frac{1}{2}$ is the offset introduced to shift the coordinates into a non-negative index space, and $\lfloor \cdot \rfloor$ denotes the floor operation. The target prompt token is then constructed through embedding mapping and positional encoding:

$$W_t=E\left(q_t\right)+P$$

(6)

where $E(\cdot )$ denotes the embedding mapping of the state indices, and $P$ denotes the positional encoding. Therefore, the target prompt token is essentially a prompt-sequence representation obtained from the target bounding box state through continuous-coordinate representation, discrete quantization, and embedding encoding. Compared with directly introducing an additional high-dimensional appearance branch, this design explicitly incorporates target-state priors into Transformer-based feature modeling with only limited additional parameters and computational cost.

At the initialization stage, the initial target state is constructed from the target bounding box provided in the first frame, and the target prompt token $W_0$ is generated through the above mapping process. During tracking at frame $t$, the template feature $Z_t$, the search region feature $X_t$, and the target prompt token $W_t$ are jointly fed into the Transformer backbone to obtain the fused feature of the current frame:

$$F{T}_t=f(Z_t,X_t,W_t)$$

(7)

Subsequently, state-related regression tokens are extracted from the fused feature $F{T}_t$, and the discrete state prediction ${\widehat{q}}_t$ of the current frame is obtained through a linear mapping. The continuous coordinates are then recovered by inverse mapping:

$${\widehat{b}}_t^i=\frac{{\widehat{q}}_t^i}{N-1}-\mathsf{\Delta }$$

(8)

Through this process, the predicted box in corner form $({\widehat{x}}_1,{\widehat{y}}_1,{\widehat{x}}_2,{\widehat{y}}_2)$ is first obtained, which is then converted into the center-based bounding box representation ${\widehat{b}}_t=({\widehat{c}}_x,{\widehat{c}}_y,\widehat{w},\widehat{h})$. The predicted bounding box ${\widehat{b}}_t$ of the current frame is further used to update the center and scale of the search region in the next frame, thereby constructing a new search region $X_{t+1}$. Accordingly, the frame-to-frame recursive relationship in this work can be expressed as

$${\widehat{b}}_t\to X_{t+1}\to {\widehat{b}}_{t+1}$$

(9)

In this way, the target-state prior continuously participates in target-state estimation over time, enabling the model to maintain stable target-state modeling capability across consecutive video frames. Mechanistically, the target prompt token provides explicit target-state constraints for the feature interaction of the current frame, while the subsequent prediction result influences the construction of the next-frame search region through frame-wise recursion. Together, these two components form a state propagation mechanism for continuous tracking.

Through this design, the proposed model not only enhances target discrimination under challenging conditions such as complex backgrounds, similar distractors, and appearance variations, but also improves localization stability in dynamic scenarios such as fast motion and occlusion recovery. Meanwhile, since the prompt token is represented only by compact state information and has a fixed and short sequence length, it does not significantly increase the overall computational burden of the model.

3.2. Backbone Network

This study adopts the lightweight pre-trained Transformer model ViT-Tiny [33] as the backbone network. Compared with ViT-Base [34], although its feature extraction dimension is only one-quarter of that of ViT-Base, its performance surpasses that of most carefully designed lightweight ViT variants owing to the fast pre-training distillation module, while also achieving faster inference speed. Within this framework, ViT-Tiny takes the template image $Z_{\mathrm{i}\mathrm{n}}\in {\mathrm{R}}^{W_z\times H_z\times 3}$ and a search image $X_{\mathrm{i}\mathrm{n}}\in {\mathrm{R}}^{W_x\times H_x\times 3}$ as inputs, partitions Z and X into $Z_{\mathrm{t}\mathrm{o}\mathrm{k}\mathrm{e}\mathrm{n}}$ and $X_{\mathrm{t}\mathrm{o}\mathrm{k}\mathrm{e}\mathrm{n}}$, respectively, and concatenates them with the target prompt token $W_{\mathrm{t}\mathrm{o}\mathrm{k}\mathrm{e}\mathrm{n}}$ before feeding them into the encoder for attention computation.

3.3. Layered Attention

In the previous tracking model, Mixformer [35] proposed a mixed attention module to simultaneously achieve feature extraction and feature fusion. This module adopts an asymmetric mixed attention mechanism by removing the query connection from the template to the search region, thereby effectively reducing the computational cost and preventing the target template from being disturbed by the dynamic search region.

In the attention module, $Q_z$, $K_z$ and $V_z$ are generated from the template-frame token sequence $Z_{\mathrm{t}\mathrm{o}\mathrm{k}\mathrm{e}\mathrm{n}}$. $Q_x$, $K_x$ and $V_x$ are generated from the search-frame token sequence $X_{\mathrm{t}\mathrm{o}\mathrm{k}\mathrm{e}\mathrm{n}}$. $Q_w$, $K_w$ and $V_w$ are generated from the target prompt token sequence $W_{\mathrm{t}\mathrm{o}\mathrm{k}\mathrm{e}\mathrm{n}}$. The final query, key and value matrices are then concatenated as $Q=[Q_z;Q_x;Q_w]$, $K=[K_z;K_x;K_w]$ and $V=[V_z;V_x;V_w]$. Accordingly, the attention computation in most single-stream trackers can be formulated as

$$A\left(Q,K,V\right)=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{Q{K}^T}{\sqrt{d}}\right)\cdot V$$

(10)

To investigate the hybrid attention mechanism, the above formula can be rewritten as follows.

$$\begin{array}{c}A\left(\left[Q_z;Q_x;Q_w\right],\left[K_z;K_x;K_w\right],\left[V_z;V_x;V_w\right]\right)\\ =\mathrm{softmax}\left(\frac{\left[Q_z;Q_x;Q_w\right]{\left[K_z;K_x;K_w\right]}^T}{\sqrt{d}}\right)\cdot \left[V_z;V_x;V_w\right]\\ =\mathrm{softmax}\left(\frac{1}{\sqrt{d}}\mathrm{*}\left[\begin{array}{ccc}{Q}_z{K}_z^T& Q_z{K}_x^T& Q_z{K}_w^T\\ Q_x{K}_z^T& Q_x{K}_x^T& Q_x{K}_w^T\\ Q_w{K}_z^T& Q_w{K}_x^T& Q_w{K}_w^T\end{array}\right]\right)\cdot \left[\begin{array}{c}{V}_z\\ V_x\\ V_w\end{array}\right]\end{array}$$

(11)

Here, $Q_z{K}_z^T$, $Q_x{K}_x^T$, and $Q_w{K}_w^T$ denote the self-attention mechanisms corresponding to the template region, the search region and the target prompt token region, respectively, whose primary role is to extract features within their respective regions. In contrast, $Q_i{K}_j^T(i,j\in \left\{z,x,w\right\}i\ne j)$ represents the cross-attention mechanisms among the template, search and target prompt token regions, which are intended to promote feature interaction and correlation modeling across different regions. Meanwhile, the computational complexity of the attention mechanism in Equation (12) directly affects the tracking speed. Assuming that the length of the template token sequence Z is m, the length of the search region X is n, and the target prompt token region contains only low-dimensional state information corresponding to the target bounding box coordinates, with a fixed length of l is 4, the time complexity of each attention term can be expressed as follows:

$$\begin{array}{cc}{Q}_z{K}_z^T& :O\left(m^2\right)\\ Q_x{K}_x^T& :O\left(n^2\right)\\ Q_w{K}_w^T& :O\left(16\right)\\ Q_z{K}_x^T\mathrm{and}{Q}_x{K}_z^T& :O\left(mn\right)\\ Q_z{K}_w^T\mathrm{and}{Q}_w{K}_z^T& :O\left(4m\right)\\ Q_x{K}_w^T\mathrm{and}{Q}_w{K}_x^T& :O\left(4n\right)\end{array}$$

(12)

Since the dimensional relationship among the three token groups typically satisfies $n\gg m\gg l=4$, appropriately reducing the computation associated with the $Q_x{K}_x^T$ term in the attention operation can effectively lower the overall time complexity.

As illustrated in Figure 2, the first $k$ layers perform mixed attention computation, whereas the subsequent $12-k$ layers only preserve cross-region feature interaction and the self-attention of the target prompt token region, while omitting the computations of $Q_z{K}_z^T$ and $Q_x{K}_x^T$. The attention formulation for the shallow part is given as follows:

$$A_{\mathrm{S}\mathrm{h}\mathrm{a}\mathrm{l}\mathrm{l}\mathrm{o}\mathrm{w}}=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{1}{\sqrt{d}}\mathrm{*}\left[\begin{array}{ccc}{Q}_z{K}_z^T& 0& Q_z{K}_w^T\\ Q_x{K}_z^T& Q_x{K}_x^T& Q_x{K}_w^T\\ Q_w{K}_z^T& Q_w{K}_x^T& Q_w{K}_w^T\end{array}\right]\right)\cdot \left[\begin{array}{c}{V}_z\\ V_x\\ V_w\end{array}\right]$$

(13)

Figure 2. Specific computational details of the shallow attention module and the deep attention module.

The attention formulation for the deep part is given as follows:

$$A_{\mathrm{D}\mathrm{e}\mathrm{e}\mathrm{p}}=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{1}{\sqrt{d}}\mathrm{*}\left[\begin{array}{ccc}0& Q_z{K}_x^T& Q_z{K}_w^T\\ Q_x{K}_z^T& 0& Q_x{K}_w^T\\ Q_w{K}_z^T& Q_w{K}_x^T& Q_w{K}_w^T\end{array}\right]\right)\cdot \left[\begin{array}{c}{V}_z\\ V_x\\ V_w\end{array}\right]$$

(14)

Studies on Vision Transformers have demonstrated that different network layers assume distinct modeling responsibilities. In the shallow layers, the network primarily focuses on capturing local textures and low-level semantic patterns, where the intra-region spatial self-correlation within the template and search regions plays a crucial role in constructing robust feature representations. By contrast, in the deeper layers, the network is more inclined to model high-level semantic interactions as well as the discriminative relationship between the target and the background. At this stage, redundant intra-region self-attention not only imposes additional computational overhead but may also introduce unnecessary background interference. To this end, a hierarchical decoupling strategy is proposed to enable a smooth transition from region-wise feature extraction to global feature interaction, which is also consistent with the hierarchical processing paradigm of biological vision, evolving from local perception to global cognition. Specifically, mixed attention computation similar to that adopted in MixFormer is retained in the shallow layers, whereas only the feature interaction mechanism is preserved in the deeper layers. Moreover, since the target prompt token has a relatively short sequence length and plays a crucial role in the subsequent loss computation during training, all attention terms involving $Q_w$ and $K_w$ are retained.

In the experiments, we set k = 6, where the size of the template image Z is 128 × 128 and that of the search image X is 256 × 256. Thus, m = 128 and n = 256. Based on these settings, the attention computation is reduced by 38.3% compared with Equation (12).

3.4. Lightweight Autoregressive Prediction Module

To enable dynamic modeling of target states while avoiding the additional computational overhead introduced by conventional detection heads, a lightweight autoregressive prediction module is introduced after the Transformer encoder to continuously estimate the bounding box state of the target over time.

Specifically, the target-related features output by the Transformer encoder are treated as latent state representations of a temporal sequence. On this basis, the model progressively predicts the target bounding box parameters of the current frame in an autoregressive manner. In this way, the tracking task is formulated as a conditional sequence generation problem, in which the target position at the next time step is predicted conditioned on the historical target states and the current visual features. Compared with static prediction schemes that directly regress the entire bounding box in a single step, autoregressive modeling explicitly captures the temporal continuity of target states, thereby improving tracking stability in challenging scenarios such as fast motion, occlusion and scale variation.

Compared with conventional detection heads, the proposed module neither requires additional feature pyramid construction nor relies on dense proposal generation or complex post-processing. Instead, it directly predicts the target bounding box from the compact state representation encoded by the Transformer, thereby resulting in lower parameter overhead and computational complexity. Meanwhile, since the prediction results continuously participate in subsequent state estimation through frame-to-frame recursion, the proposed module is able to maintain continuous modeling of target motion variations over time, thus yielding more stable tracking performance in challenging scenarios such as fast motion, partial occlusion, and scale variation.

The target-related features produced by the encoder are further used to predict the IoU between the estimated bounding box and the ground-truth bounding box, which serves as an auxiliary metric for evaluating the quality of the learned feature representation. The probabilistic formulation of the autoregressive sequence is given as

$$p\left(Y^t\right|Y^{t-1},\left(W^t,Z^t,X^t\right))$$

(15)

In this formulation, $Z^t$ and $X^t$ denote the template image and the search image at time step $t$, respectively, while $Y^{t-1}$ represents the target state at the previous time step. The prompt token $W^t$ is re-encoded from $Y^{t-1}$ and incorporated into the feature interaction at the current time step as an explicit prior condition. The target state predicted by the autoregressive module is not only used for bounding box estimation in the current frame, but is also re-encoded into a new target prompt token for the next time step and fed back into the model. In this way, the target prompt token and the autoregressive prediction module form a stable closed-loop mechanism: the former injects explicit target-state prior information into current-frame feature interaction, while the latter continuously estimates and updates this prior across frames. This design enables tighter coupling between target prior guidance and temporal state evolution, which distinguishes the proposed framework from trackers relying only on static token interaction or one-step box regression.

By integrating the autoregressive prediction mechanism with the target prompt token and the hierarchical attention structure, the proposed model is able not only to obtain discriminative feature representations within a single frame, but also to perform continuous modeling of the target motion trajectory along the temporal dimension, thereby providing effective support for robust tracking in complex dynamic scenarios.

3.5. Training Loss

For the autoregressive bounding box generation model, the overall loss function is constructed by jointly combining the $L_1$ loss and the GIoU loss, which is defined as

$$L={\lambda }_1\times L_{L1}+{\lambda }_2\times L_{GIoU}$$

(16)

where ${\lambda }_1$ and ${\lambda }_2$ are weighting coefficients used to balance the contributions of the $L_1$ loss and the GIoU loss in the overall objective function.

The $L_1$ loss is used to measure the discrepancy between the predicted bounding box and the ground-truth bounding box in the parameter space, and is defined as

$$L_{L1}=\parallel B_{pred}-B_{gt}{\parallel }_1$$

(17)

where $B_{pred}=(x,y,w,h)$ denotes the predicted bounding box, $B_{gt}=(x^{\ast },y^{\ast },w^{\ast },h^{\ast })$ denotes the ground-truth bounding box, and $\parallel \cdot {\parallel }_1$ denotes the $L_1$ norm. This loss directly constrains the numerical deviation between the predicted box and the ground-truth box in terms of center position, width, and height, thereby helping improve the stability of bounding box regression and the accuracy of local parameter estimation.

To further enhance the overall spatial matching ability between the predicted box and the ground-truth box, the GIoU loss is introduced, which is defined as

$$L_{\mathrm{G}\mathrm{I}\mathrm{o}\mathrm{U}}=1-\mathrm{G}\mathrm{I}\mathrm{o}\mathrm{U}(B_{pred},B_{gt})$$

(18)

where GIoU is defined as

$$\mathrm{G}\mathrm{I}\mathrm{o}\mathrm{U}=\mathrm{I}\mathrm{o}\mathrm{U}-\frac{\mid C-(B_{pred}\cup B_{gt})\mid }{\mid C\mid }$$

(19)

Here, IoU denotes the intersection-over-union between the predicted bounding box and the ground-truth bounding box, $C$ denotes the smallest enclosing rectangle covering both the predicted box and the ground-truth box, and $\mid \cdot \mid$ denotes the area of the corresponding region. Compared with regression schemes that optimize only coordinate errors, the GIoU loss constrains the overlap quality between the predicted box and the ground-truth box from the perspective of overall spatial geometry, and can still provide effective optimization signals even when the two boxes have very small overlap or no overlap at all.

Therefore, the $L_1$ loss and the GIoU loss jointly optimize the model from two complementary aspects, namely bounding box parameter regression accuracy and overall spatial overlap quality. The former helps improve the numerical accuracy of position and scale estimation, while the latter enhances the overall geometric alignment between the predicted box and the ground-truth box. Their combination enables the model to account for both local parameter regression and global spatial alignment, thereby improving the accuracy and stability of bounding box prediction during target tracking.

4. Experiments

4.1. Training

The proposed tracker is implemented based on Python 3.6 and PyTorch 1.7.0. Different training data partitioning strategies are adopted for different benchmark datasets. Specifically, the results reported on GOT-10k are obtained by training the model using only the training split of GOT-10k, whereas the results reported on LaSOT are obtained under a joint training setting using the training sets of both GOT-10k and LaSOT. During training, standard data augmentation strategies are employed, including horizontal flipping and brightness perturbation. Following the settings of mainstream trackers, the template and search images are resized to 128 × 128 and 256 × 256, respectively. The model is trained on an NVIDIA GeForce RTX 3060 GPU for 300 epochs with a batch size of 32. The initial learning rate is set to 4 × 10⁻⁴, and a weight decay of 10⁻⁴ is applied after 240 epochs. The AdamW optimizer [36] is adopted to ensure efficient training and stable convergence of the model.

4.2. Evaluation on the GOT-10k and LaSOT Benchmarks

GOT-10k is one of the most widely used benchmarks in single object tracking. Built upon the WordNet hierarchy, it emphasizes generalization evaluation under open-world conditions. The dataset covers 563 target categories, including animals, vehicles, persons and various everyday objects, exhibiting substantial diversity in both scene content and category distribution. GOT-10k contains approximately 1.5 million frames with manually annotated bounding boxes on a frame-by-frame basis, thereby ensuring the stability and reproducibility of evaluation. It adopts a strict training/validation/test split and follows a one-shot protocol, in which the object categories in the training and test sets do not overlap. This setting makes it particularly suitable for evaluating a tracker’s transferability and generalization capability to unseen categories.

LaSOT is a large-scale benchmark designed for long-term single object tracking, with the aim of evaluating robustness and temporal consistency over long sequences. The dataset contains 1400 video sequences with more than 3.5 million frames in total, and each frame is provided with a high-quality bounding box annotation. The average sequence length exceeds 2500 frames, which is significantly longer than that of most short-term benchmarks. LaSOT covers 70 target categories, including both rigid and non-rigid objects, and involves various challenging factors such as occlusion, fast motion, illumination variation, background clutter and viewpoint change. It is therefore well suited for evaluating the ability of a tracker to handle appearance variation, occlusion recovery and long-term stability.

To ensure fair comparison among different methods, the standard evaluation protocols of both datasets are adopted in this study. Let $B_t$ denote the predicted bounding box in the $t$-th frame, $G_t$ the ground-truth bounding box, and $N$ the total number of frames. The intersection-over-union (IoU) is defined as

$${\mathrm{I}\mathrm{o}\mathrm{U}}_t=\frac{\mid B_t\cap G_t\mid }{\mid B_t\cup G_t\mid }$$

(20)

The success rate is used to measure the proportion of frames whose overlap between the predicted box and the ground-truth box exceeds a predefined threshold, which is formulated as

$$\mathrm{S}\mathrm{R}\left(\alpha \right)=\frac{1}{N}\sum _{t=1}^{N}q({\mathrm{I}\mathrm{o}\mathrm{U}}_t>\alpha )$$

(21)

where $\alpha \in [0,1]$ denotes the threshold and $q(\cdot )$ is the indicator function.

Under the standard evaluation protocol of GOT-10k, ${\mathrm{S}\mathrm{R}}_{0.5}$ denotes the proportion of frames with $\mathrm{I}\mathrm{o}\mathrm{U}>0.5$, reflecting the success rate of usable localization, whereas ${\mathrm{S}\mathrm{R}}_{0.75}$ denotes the proportion of frames with $\mathrm{I}\mathrm{o}\mathrm{U}>0.75$, placing greater emphasis on high-precision localization capability.

Under the standard evaluation protocol of LaSOT, AUC is typically defined as the area under the success-rate curve $\mathrm{S}\mathrm{R}\left(\alpha \right)$ over $\alpha \in [0,1]$. AUC jointly reflects the overall tracking performance under different precision requirements and provides a more stable measure of comprehensive tracking quality over long sequences. It is expressed as

$$\mathrm{A}\mathrm{U}\mathrm{C}={\int }_0^1\mathrm{S}\mathrm{R}\left(\alpha \right)\mathrm{d}\alpha$$

(22)

The average overlap (AO) measures the overall overlap quality and is defined as the average IoU over all frames. Compared with success rates under a single threshold, AO provides a finer-grained evaluation and better reflects the average overlap between the predicted and ground-truth bounding boxes. It is therefore commonly adopted as a core metric on GOT-10k. Its formulation is given by

$$\mathrm{A}\mathrm{O}=\frac{1}{N}\sum _{t=1}^N{\mathrm{I}\mathrm{o}\mathrm{U}}_t$$

(23)

Let the center of the predicted box be $c_t=(x_t,y_t)$ and the center of the ground-truth box be $g_t=(x_t^{\ast },y_t^{\ast })$. The center location error (CLE) is then defined as

$${\mathrm{C}\mathrm{L}\mathrm{E}}_t={\parallel c_t-g_t\parallel }_2$$

(24)

Under the standard evaluation protocol of LaSOT, precision (P) measures the proportion of frames whose center location error is smaller than a pixel threshold $\tau$ (commonly $\tau =20$ pixels), thereby directly reflecting the center localization accuracy. It is defined as

$$\mathrm{P}=\frac{1}{N}\sum _{t=1}^{N}q({\mathrm{C}\mathrm{L}\mathrm{E}}_t<\tau )$$

(25)

To alleviate the influence of different image resolutions and target scales on the center error, LaSOT further introduces the normalized center location error, defined as

$${\mathrm{N}\mathrm{C}\mathrm{L}\mathrm{E}}_t=\sqrt{{\left(\frac{x_t-x_t^{\ast }}{w_t}\right)}^2+{\left(\frac{y_t-y_t^{\ast }}{h_t}\right)}^2}$$

(26)

where $w_t$ and $h_t$ denote the width and height of the ground-truth bounding box, respectively. The normalized precision, denoted as ${\mathrm{P}}_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}$, focuses more on relative error, making it possible to compare errors of targets at different scales under a unified criterion. It is therefore more suitable for comprehensive evaluation over long sequences and multi-scale targets. Its formulation is

$${\mathrm{P}}_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}=\frac{1}{N}\sum _{t=1}^{N}q({\mathrm{N}\mathrm{C}\mathrm{L}\mathrm{E}}_t<{\tau }_n)$$

(27)

The inference speed is used to measure the throughput capability of a tracker under a given hardware platform and input configuration. Let $T$ (in seconds) denote the total time required to process $N$ frames continuously with a batch size of 1 under a fixed test environment. The inference speed is defined as

$$\mathrm{F}\mathrm{P}\mathrm{S}=\frac{N}{T}$$

(28)

where $T$ usually accounts only for the forward inference time of the model. Therefore, FPS primarily reflects inference throughput rather than being strictly equivalent to the end-to-end system latency.

In this study, the model size is represented by the total number of trainable parameters. Let the model contain $L$ trainable parameter tensors ${\left\{W_i\right\}}_{i=1}^L$, where $\mid W_i\mid$ denotes the number of elements in the $i$-th tensor. The parameter count reflects the lower bound of the model storage and loading cost, and is also correlated with memory consumption during inference, although it does not directly correspond to the peak resource usage in actual deployment. For convenience of comparison, the parameter count is usually reported in millions, which is defined as

$$\mathrm{P}\mathrm{a}\mathrm{r}\mathrm{a}\mathrm{m}\mathrm{s}\left(\mathrm{M}\right)=\frac{\sum _{i=1}^L\mid W_i\mid }{{10}^6}$$

(29)

Under the unified evaluation protocol, this study adopts AO, SR, AUC, P and ${\mathrm{P}}_{\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}}$ to evaluate tracking performance from two aspects, namely the overlap quality between predicted and ground-truth boxes and the target center localization error. Meanwhile, FPS, the number of parameters and the computational complexity in terms of FLOPs are jointly used to quantify model performance and resource cost, thereby providing a basis for performance comparison and lightweight analysis among different methods.

4.2.1. Quantitative Analysis

The proposed TPTTrack is evaluated against existing tracking methods on the GOT-10k and LaSOT benchmarks in terms of tracking performance, model complexity and runtime speed. The evaluation results are summarized in Table 1. Among the compared methods, LightTrack, FEAR-XS, HCAT, MVT and HiT-B are lightweight trackers, whereas TransT, DiMP-50, Ocean and SiamRPN++ are non-lightweight trackers. The speed results reported in Table 1 were all obtained under a unified hardware platform. The specific experimental environment consists of a 13th Gen Intel Core i7-13700 CPU and an NVIDIA GeForce RTX 3060 GPU.

Table 1. Comparison of TPTTrack with state-of-the-art trackers on the GOT-10k and LaSOT benchmarks.

	Trackers	GOT-1OK			LaSOT			Speed/fps		Parameters/M	FLOPS/G
		AO	${\mathbf{S}\mathbf{R}}_{0.5}$	${\mathbf{S}\mathbf{R}}_{0.75}$	AUC	PNorm	P	GPU	CPU
Lightweight	LightTrack	58.2	66.8	44.2	52.3	62.3	55.6	81.58	10.32	2.0	0.53
	FEAR-XS	57.3	68.1	45.5	53.5	64.1	54.5	104.26	12.84	1.4	1.59
	HCAT	64.1	72.5	55.7	59.3	68.7	61.0	86.25	13.85	4.3	2.19
	MVT	62.4	73.0	54.6	55.3	64.1	58.3	90.8	11.4	5.5	0.77
	HiT-B	64.0	72.1	56.1	60.2	72.1	67.1	120.67	18.73	6.6	1.42
	TPTTrack (ours)	65.0	73.7	56.4	61.4	70.9	68.5	111.6	20.76	5.5	1.79
Non-lightweight	TransT	65.8	74.8	58.5	62.9	71.8	67.0	40	4	26.1	32.9
	DIMP-50	61.1	69.7	49.2	58.1	65.6	60.4	35.5	7.5	23.0	21.5
	Ocean	61.1	70.1	47.3	57.3	66.2	61.9	30.5	6.2	44.3	26.0
	SiamRPN++	61.8	71.9	51.8	58.6	65.8	63.7	21.4	3.8	54.0	48.9

Note: Bold numbers indicate the best results among the lightweight tracking methods.

As shown in Table 1, on the GOT-10k benchmark, TPTTrack achieves an AO of 65.0%, outperforming real-time lightweight trackers such as LightTrack, FEAR-XS, HCAT, MVT and HiT-B, and trailing the non-lightweight tracker TransT by only 0.8%. On the LaSOT benchmark, TPTTrack attains an AUC of 61.4%, which is higher than those of LightTrack, FEAR-XS, HCAT, MVT and HiT-B. Its precision reaches 68.5%, also ranking among the best-performing real-time lightweight trackers. In addition, TPTTrack runs at 111.6 FPS on GPU and 20.76 FPS on CPU. Although its GPU speed is slightly lower than that of HiT-B, it is still substantially faster than non-lightweight trackers such as TransT, DiMP-50, Ocean and SiamRPN++.

From the perspective of lightweight design, TPTTrack contains only 5.5M parameters, which is significantly fewer than those of non-lightweight trackers such as TransT, DiMP-50, Ocean and SiamRPN++. Its computational complexity is 1.79G FLOPs, which is also markedly lower than that of the above non-lightweight methods, demonstrating the clear lightweight characteristics of the proposed approach. Compared with existing lightweight trackers, although the parameter count and FLOPs of TPTTrack are not the smallest, they remain at a relatively low level overall. More importantly, TPTTrack achieves superior or near-best performance in accuracy metrics among current lightweight tracking methods.

As illustrated in Figure 3, there exists a clear accuracy–efficiency trade-off between AO and GPU speed among different tracking methods on the GOT-10k benchmark. In general, non-real-time methods tend to achieve higher accuracy at the cost of slower inference speed, whereas most real-time methods, although faster in inference, exhibit varying degrees of performance degradation in terms of AO. By contrast, TPTTrack is located in the upper-right region of the comparison plot, achieving an AO of 65.0% while running at 111.6 FPS on GPU, which is only slightly lower than that of the fastest model. This indicates that TPTTrack delivers the best overall performance among real-time methods. Compared with TransT, TPTTrack shows only a marginal decrease of 0.8% in AO, while its inference speed is improved to 2.79 times that of TransT.

Figure 3. Comparison of average overlap (AO) and GPU speed of different trackers on the GOT-10k dataset.

As shown in Figure 4, TPTTrack maintains overall favorable success curves in typical challenging scenarios, including similar-object interference, low-contrast conditions, occlusion-and-reappearance, scale variation, fast motion, and out-of-view, which indicates that the proposed method has strong overlap matching capability and tracking stability under complex conditions. Specifically, in the scenarios of similar-object interference, low contrast, and out-of-view, the overall curve of TPTTrack remains among the best, suggesting that the target-state-guided prompt token can effectively enhance the model’s focus on target-relevant regions. In the scenarios of occlusion-and-reappearance and fast motion, TPTTrack still achieves high success rates under medium and high overlap thresholds, showing that the lightweight autoregressive state generation module helps improve target-state propagation and recovery ability. In the scale-variation scenario, TPTTrack still outperforms most compared methods overall, which reflects that the hierarchical decoupled attention mechanism can effectively preserve critical feature interactions while reducing redundant computation.

Figure 4. Tracking success rate curves of different algorithms in different scenarios.

As shown in Figure 5, the precision curves of TPTTrack also exhibit good stability and overall superiority under the above typical challenging scenarios, indicating that the proposed method has strong advantages in target center localization accuracy. In the scenarios of similar-object interference, low contrast, and out-of-view, TPTTrack still maintains high precision over a relatively large error-threshold range, which suggests that the target prompt token can provide clearer target prior constraints for center localization. In the scenarios of occlusion-and-reappearance and fast motion, the curve of TPTTrack remains overall higher than those of most lightweight methods, showing that the autoregressive state propagation mechanism can enhance the model’s ability to localize targets under continuous motion changes and short-term target loss. In the scale-variation scenario, TPTTrack still maintains high precision over most threshold ranges, which further verifies the positive effect of the hierarchical decoupled attention mechanism on localization stability. Overall, the results in Figure 5 demonstrate that the proposed method achieves strong center localization capability and overall robustness across multiple challenging scenarios.

Figure 5. Tracking precision rate curves of different algorithms in different scenarios.

4.2.2. Qualitative Analysis

As illustrated in Figure 6, to provide a more intuitive understanding of the role of the target prompt token during the tracking process, three representative tracking sequences are selected for visualization of the feature response heatmaps produced by the model. In the heatmaps, color intensity is used to indicate the response strength of the model to different regions, where red or highlighted regions represent stronger attention and blue or low-intensity regions indicate weaker responses. It can be observed that, after introducing the target prompt token, the model responses become more concentrated on the target region, while the responses to background and irrelevant distractor regions are significantly suppressed. This demonstrates that the target prompt token provides the model with more explicit target prior information, thereby guiding the feature interaction process to focus more effectively on target-relevant regions and further improving target localization accuracy and tracking stability.

Figure 6. Heat map, (a) original image, (b) without target prompt token, (c) with target prompt token.

To evaluate the tracking performance of the proposed method under different challenging conditions, comparative experiments were conducted against five representative lightweight tracking algorithms across four typical challenge scenarios. The complete qualitative comparison results are shown in Figure 7.

Figure 7. Qualitative comparison of six lightweight trackers (LightTrack, FEAR-XS, HCAT, MVT, HiT-B and the proposed TPTTrack) across four scenarios.

In the similar-object scenarios, the presence of distractors with appearances similar to that of the target often leads to mismatching and tracking drift. As shown in the second example of Figure 7a, when a similar object appears, the competing methods exhibit varying degrees of mismatching or inaccurate scale estimation. In contrast, the proposed TPTTrack introduces the target prompt token to provide prior target information and further strengthen the response to target-relevant regions in subsequent network processing, thereby enabling more accurate target tracking under similar-object interference.

In the low-contrast scenarios, where the target exhibits only slight differences in brightness and texture from the background, localization tends to become unstable. As illustrated in Figure 7b, when the distinction between the target and the background is weak, the compared methods show different degrees of localization deviation, and the tracking box of FEAR-XS in particular suffers from irreversible drift. By contrast, the proposed TPTTrack, benefiting from the enhanced modeling of target-relevant regions through the hierarchical attention mechanism, is still able to stably cover the target and maintain relatively high tracking accuracy.

In the occlusion-and-reappearance scenarios, the target is partially or fully occluded and then reappears, which often causes tracking interruption or difficulty in re-localization. As shown in Figure 7c, when the target is occluded, methods such as HCAT, MVT and HiT-B all exhibit inaccurate tracking boxes, which further leads to noticeable drift after the target reappears. In contrast, the proposed TPTTrack continuously models the target-state information through the autoregressive prediction module, thereby maintaining more reliable tracking during occlusion and demonstrating stronger robustness.

In the scale-variation scenarios, the target undergoes significant changes in size and shape during tracking, which imposes higher demands on the scale adaptation capability of the tracker. As illustrated in Figure 7d, when the target experiences scale and shape changes caused by rotation, the competing trackers suffer from varying degrees of scale mismatch. By contrast, the proposed TPTTrack, by continuously modeling target-state information through the autoregressive prediction module and imposing persistent constraints on target representation via the target prompt token, is able to adapt the predicted bounding box more effectively to target size changes, thus exhibiting stronger adaptability.

In summary, TPTTrack demonstrates better localization accuracy and tracking stability across different challenging scenarios, including similar-object interference, low contrast, occlusion recovery and scale variation. These results indicate that, by integrating the target prompt token, the hierarchical attention decoupling mechanism and the lightweight autoregressive prediction module, the proposed method can effectively enhance target discrimination, suppress background interference and improve target-state modeling in complex scenes, thereby achieving superior accuracy and robustness under diverse challenging conditions.

4.3. Ablation Study

To validate the effectiveness of the hierarchical attention strategy in TPTTrack, the model was evaluated on the GOT-10k benchmark to investigate the impact of different attention mechanism components. Equations (30)–(35) present the formulations of the corresponding attention variants. $A_1$ and $A_2$ are mainly designed to verify whether retaining partial intra-region self-attention in the shallow layers, together with the interaction of the target prompt token, helps enhance low-level feature extraction. $A_3$ and $A_4$ are used to analyze the role of preserving the key cross-region interactions among the template region, the search region, and the target prompt token in target-state modeling after redundant intra-region self-attention is reduced in the deep layers. $A_5$ and $A_6$, as control configurations without the target prompt token, are further introduced to evaluate the specific contribution of the target prompt token to target prior modeling and cross-region feature interaction.

$$A_1=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{1}{\sqrt{d}}\ast \left[\begin{array}{ccc}{Q}_z{K}_z^T& 0& Q_z{K}_w^T\\ Q_x{K}_z^T& Q_x{K}_x^T& Q_x{K}_w^T\\ Q_w{K}_z^T& Q_w{K}_x^T& Q_w{K}_w^T\end{array}\right]\right)\cdot \left[\begin{array}{c}{V}_z\\ V_x\\ V_w\end{array}\right]$$

(30)

$$A_2=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{1}{\sqrt{d}}\ast \left[\begin{array}{ccc}{Q}_z{K}_z^T& 0& Q_z{K}_w^T\\ 0& Q_x{K}_x^T& Q_x{K}_w^T\\ Q_w{K}_z^T& Q_w{K}_x^T& Q_w{K}_w^T\end{array}\right]\right)\cdot \left[\begin{array}{c}{V}_z\\ V_x\\ V_w\end{array}\right]$$

(31)

$$A_3=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{1}{\sqrt{d}}\ast \left[\begin{array}{ccc}0& Q_z{K}_x^T& Q_z{K}_w^T\\ Q_x{K}_z^T& 0& Q_x{K}_w^T\\ Q_w{K}_z^T& Q_w{K}_x^T& Q_w{K}_w^T\end{array}\right]\right)\cdot \left[\begin{array}{c}{V}_z\\ V_x\\ V_w\end{array}\right]$$

(32)

$$A_4=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{1}{\sqrt{d}}\ast \left[\begin{array}{ccc}0& 0& Q_z{K}_w^T\\ Q_x{K}_z^T& 0& Q_x{K}_w^T\\ Q_w{K}_z^T& Q_w{K}_x^T& Q_w{K}_w^T\end{array}\right]\right)\cdot \left[\begin{array}{c}{V}_z\\ V_x\\ V_w\end{array}\right]$$

(33)

$$A_5=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{1}{\sqrt{d}}\ast \left[\begin{array}{cc}{Q}_z{K}_z^T& 0\\ Q_x{K}_z^T& Q_x{K}_x^T\end{array}\right]\right)\left[\begin{array}{c}{V}_z\\ V_x\end{array}\right]$$

(34)

$$A_6=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}\left(\frac{1}{\sqrt{d}}\ast \left[\begin{array}{cc}{Q}_z{K}_z^T& 0\\ 0& Q_x{K}_x^T\end{array}\right]\right)\left[\begin{array}{c}{V}_z\\ V_x\end{array}\right]$$

(35)

To further verify the superiority of the proposed hierarchical attention decoupling strategy, comparative experiments under different configurations were designed in this section. First, the full attention mechanism $A_{\mathrm{full}}$ was introduced as the performance baseline, in which all network layers employ the standard attention computation defined in Equation (11). Subsequently, the proposed decoupling mechanism was applied to different stages of the network: in the shallow stage (the first $k$ layers), $A_1$ or $A_2$, both containing the main diagonal terms, was adopted for feature extraction; in the deep stage (the remaining $12-k$ layers), $A_3$ or $A_4$, both excluding the main diagonal terms, was employed for feature interaction. This results in four attention combinations, namely $A_{13}$, $A_{14}$, $A_{23}$ and $A_{24}$. Table 2 summarizes the experimental results of these combinations, together with the full-attention baseline, on the GOT-10k benchmark.

Table 2. Performance comparison of different hierarchical attention combinations on the GOT-10k test set.

#	AO	${\mathbf{S}\mathbf{R}}_{0.5}$	${\mathbf{S}\mathbf{R}}_{0.75}$	Speed/FPS
1 ($A_{13})$	65.0	73.7	56.4	111.6
2 ($A_{14})$	63.8	72.2	55.2	113.6
3 ($A_{23})$	64.5	73.1	55.8	112.9
4 ($A_{24}$)	62.6	70.4	54.0	115.8
5 ($A_{full}$)	65.3	74.0	56.8	69.1

Note: Bold numbers indicate the best results among different attention combination schemes.

In this ablation study, the ratio between the shallow and deep stages was set to 1:1, i.e., $k=6$. Table 2 presents a comparative analysis of different attention mechanism combinations on the GOT-10k test set. It can be observed that different attention configurations exhibit a clear trade-off between tracking accuracy and inference speed. Among them, the configuration using the proposed hierarchical decoupled attention mechanism (#1) achieves the best results on the core accuracy metrics, including AO, ${\mathrm{S}\mathrm{R}}_{0.5}$ and ${\mathrm{S}\mathrm{R}}_{0.75}$, while maintaining a relatively high inference speed, demonstrating favorable overall performance.

According to the results in Table 2, compared with configurations #2, #3 and #4, which preserve only part of the attention terms in either the shallow or the deep stage, configuration #1 consistently achieves higher accuracy. This indicates that retaining the necessary intra-region self-attention in the shallow stage is beneficial for enhancing feature extraction capability, whereas emphasizing cross-region feature interaction in the deep stage contributes to more discriminative modeling between the target and the background. In contrast, when attention computation is excessively reduced in the deep stage (as in configuration #4), although the inference speed is slightly improved, the target localization accuracy degrades noticeably, suggesting that deep semantic interaction remains crucial for stable tracking.

It should be noted that configuration #5, which adopts the full attention mechanism, achieves slightly higher accuracy than the hierarchical decoupling scheme; however, its inference speed drops significantly to only 69.1 FPS, making it difficult to satisfy the requirement of real-time tracking. This indicates that, although the full-attention strategy can enhance feature modeling capability, it introduces a substantial amount of redundant computation, thereby reducing the overall efficiency of the model.

Overall, the hierarchical decoupled attention mechanism effectively preserves critical feature interaction capability while significantly reducing unnecessary computational overhead, thereby achieving a better balance between tracking accuracy and inference efficiency. These results validate the rationality and effectiveness of the proposed attention decoupling strategy in the lightweight Transformer-based tracking framework.

To further investigate the influence of the ratio between shallow and deep layers on performance, this study also compares the effects of different shallow-to-deep layer ratios on the GOT-10k benchmark. Specifically, the experiments are conducted with $k=3,6,$ and $9$, corresponding to $12-k=9,6,$ and $3$, respectively, in order to examine three different shallow-to-deep layer configurations. In this ablation study, the shallow stage adopts $A_1$, while the deep stage adopts $A_3$. As shown in Table 3, the model achieves the best performance on the GOT-10k benchmark when the shallow-to-deep layer ratio is set to 6:6.

Table 3. Performance comparison under different shallow-to-deep layer ratios on the GOT-10k test set.

Shallow-to-Deep Layer Ratio	AO	${\mathbf{S}\mathbf{R}}_{0.5}$	${\mathbf{S}\mathbf{R}}_{0.75}$
3:9	63.8	72.2	55.4
6:6	65.0	73.7	56.4
9:3	64.1	72.1	55.0

Note: Bold numbers indicate the best results.

To further verify the role of the lightweight autoregressive prediction module in the overall tracking performance, an ablation study on different target-state prediction strategies was conducted on the GOT-10k benchmark. In this experiment, the Transformer backbone, the target prompt token, and the hierarchical decoupled attention mechanism were kept exactly the same, while only the bounding box prediction strategy was replaced, so as to ensure the comparability of the experimental results.

Specifically, under the premise that the Transformer backbone, the target prompt token, and the hierarchical decoupled attention mechanism remained unchanged, different bounding box prediction strategies were comparatively analyzed. First, the proposed autoregressive sequence generation module was replaced with a static prediction head, in which a corner head was adopted to predict the target location. When the image features to be predicted are fed into the corner head, they are first weighted according to the attention map, such that local features can be enhanced or suppressed based on global information. In this component, the search region features are first extracted by the encoder and then refined through the attention map. These enhanced features are reshaped into $(H,W)$ and subsequently fed into a fully convolutional network. This fully convolutional network consists of multiple sequential convolutional layers, batch normalization layers and rectified linear unit activation functions, and is responsible for generating probability maps corresponding to the top-left and bottom-right corners of the target bounding box, respectively. Finally, the predicted coordinates are obtained by calculating the expected values of the corner probability distributions.

To further distinguish the performance gain brought by the autoregressive prediction architecture itself from that introduced by temporal state modeling, a weak autoregressive variant was also constructed. This variant preserves the structural form of the autoregressive prediction head, but does not use the prediction result from the previous time step as a conditional input during inference; instead, it estimates the bounding box solely based on the Transformer features of the current frame. As shown in Table 4, the tracker equipped with the lightweight autoregressive prediction module achieves the best performance among the three configurations.

Table 4. Performance comparison of different target-state prediction strategies on the GOT-10k test set.

Configuration	AO	${\mathbf{S}\mathbf{R}}_{0.5}$	${\mathbf{S}\mathbf{R}}_{0.75}$
Corner Point Prediction Head	63.6	71.9	54.8
Weak Autoregression	64.3	72.6	55.6
Autoregression	65.0	73.7	56.4

Note: Bold numbers indicate the best results.

To further distinguish the performance gain contributed by the explicit target prior provided by the target prompt token from that brought by the underlying feature matching mechanism, an additional variant without the target prompt token was constructed. This variant retains the original backbone network, the hierarchical decoupled attention mechanism, and the lightweight autoregressive prediction module, while removing only the target prompt token. As a result, the compact target-state information is no longer encoded as an independent prompt token to participate in the Transformer-based feature interaction, and target localization relies solely on the visual features extracted from the template and search branches, with the $A_{56}$ attention combination adopted. As shown in Table 5, the tracker equipped with the target prompt token achieves superior performance compared with the configuration without the target prompt token.

Table 5. Performance comparison with and without the target prompt token on the GOT-10k test set.

Target Prompt Token	AO	${\mathbf{S}\mathbf{R}}_{0.5}$	${\mathbf{S}\mathbf{R}}_{0.75}$
Not used	64.2	72.8	55.4
Used	65.0	73.7	56.4

Note: Bold numbers indicate the best results.

5. Conclusions

To address the challenges of single object trackers based on Transformer architectures, including relatively large model size, redundant attention computation, and consequently high inference latency, which make deployment in real-time and resource-constrained scenarios difficult, this paper proposes a lightweight Transformer-based single object tracking method, termed TPTTrack. By introducing a target prompt token, the proposed method explicitly incorporates target-state priors into the feature interaction process, thereby enhancing the representation of target-relevant regions and suppressing background interference. In addition, a hierarchical attention decoupling mechanism is designed to allocate attention computation more appropriately across different network layers, thereby reducing redundant computational overhead. Meanwhile, a lightweight autoregressive prediction module is integrated to continuously model the target state in a frame-by-frame manner, improving the adaptability of the model in complex dynamic scenarios. Experimental results show that TPTTrack achieves an AO of 65.0% on GOT-10k and a precision of 68.5% on LaSOT, while running at 111.6 FPS on GPU and 20.76 FPS on CPU. The model contains only 5.5M parameters with a computational complexity of 1.79G FLOPs. These results demonstrate that the proposed method achieves competitive tracking accuracy and favorable real-time performance while maintaining low model complexity and computational cost, thereby striking a reasonable balance between lightweight design and tracking performance. Under a unified Transformer architecture, the proposed method organically integrates explicit target prior modeling, hierarchical attention allocation, and autoregressive state propagation, thereby forming a complete framework for lightweight object tracking and providing a new design perspective for related research.