MPVT: An Efficient Multi-Modal Prompt Vision Tracker for Visual Target Tracking

Abstract

Visual target tracking is a fundamental task in computer vision. Combining multi-modal information with tracking leverages complementary information, which improves the precision and robustness of trackers. Traditional multi-modal tracking methods typically employ a full fine-tuning scheme, i.e., fine-tuning pre-trained single-modal models to multi-modal tasks. However, this approach suffers from low transfer learning efficiency, catastrophic forgetting, and high cross-task deployment costs. To address these issues, we propose an efficient model named multi-modal prompt vision tracker (MPVT) based on an efficient prompt-tuning paradigm. Three key components are involved in the model: a decoupled input enhancement module, a dynamic adaptive prompt fusion module, and a fully connected head network module. The decoupled input enhancement module enhances input representations via positional and type embedding. The dynamic adaptive prompt fusion module achieves efficient prompt tuning and multi-modal interaction using scaled convolution and low-rank cross-modal attention mechanisms. The fully connected head network module addresses the shortcomings of traditional convolutional head networks such as inductive biases. Experimental results from RGB-T, RGB-D, and RGB-E scenarios show that MPVT outperforms state-of-the-art methods. Moreover, MPVT can save 43.8% GPU memory usage and reduce training time by 62.9% compared with a full-parameter fine-tuning model.

1. Introduction

Visual target tracking is widely used in various scenarios, such as video surveillance and autonomous driving. Target tracking based on RGB images [1,2,3] has high performance due to its large-scale training data. However, such images often make tracking fail under poor lighting conditions, such as darkness, cluttered backgrounds, and high-speed motion. With the development of multi-modal technology, auxiliary modalities can enhance the success rate, precision, and robustness of tracking by utilizing the complementarity between modalities. Some classical scenarios include RGB-T tracking [4,5,6], RGB-D tracking [7,8], and RGB-E tracking [9,10,11].

For RGB-T tracking, some researchers have proposed fusion methods based on feature decoupling. Li et al. [12] constructed a multi-branch network to handle RGB and thermal infrared images separately. ADRNet [13] further utilizes adaptive universal branches for decoupling. MANet [14] jointly learns shared, modal-specific, and instance-aware features through universal, modal, and instance adapters. MANet++ [15] incorporates a hierarchical difference loss to maximize the distribution difference between modal-specific and shared features. DMCNet [16] adopts a dual-door control mutual condition module to suppress noise through cross-modal feature guidance. With the deepening development of attention mechanisms, some researchers have proposed dynamic fusion methods based on feature selection. For example, FANet [17] involves a hierarchical feature aggregation module to adaptively fuse multi-layer features. CMPP [18] includes a cross-modal propagation framework that achieves complementarity through spatio-temporal domain feature diffusion. ViPT [19] uses infrared modalities as visual prompts for visible modalities and utilizes the attention mechanism to achieve cross-modal interaction. In addition, some methods incorporate the unique characteristics of Transformer models to design multi-modal interaction modules for RGB-T target tracking. For example, MTNet [20] utilizes channel aggregation and distribution modules to eliminate redundancy. SiamCAF [21] uses complementary coupling modules to enhance feature similarity through cross-connections. Very recently, Wang et al. [22] proposed a novel lightweight adapter to boost fusion performance for RGB-T target tracking.

Traditional RGB-D target tracking methods mainly adopt manually designed features and shallow fusion strategies. For example, DS-KCF [23] achieves bimodal feature-level fusion through HOG feature extraction, while STC [24] improves tracking robustness through decision-level fusion. With the development of deep learning, researchers have begun to explore deep feature fusion. For example, Jiang et al. [25] proposed a dual-branch CNN architecture, which extracts RGB and deep features separately and enhances discriminability through weighted fusion. OTR [26] introduces deep dynamic spatial constraints to optimize DCF (discriminative correlation filter) learning, significantly improving tracking stability in occluded scenarios. In recent years, researchers have focused more on cross-modal interaction modeling to fully utilize the complementary information of multiple modalities. DeT [7] incorporates a symmetrical double-branch structure and achieves early fusion through feature concatenation. Recently, Gao et al. [27] proposed a new RGB-D approach to capture the interactive information between modalities.

The RGB-E tracking method [28] often uses a cross-modal attention mechanism for feature fusion. AFNet [10] utilizes the high temporal resolution of event data to design a cross-frame rate alignment module, achieving a tracking frequency of 1000 Hz. HRCEUTrack [11] introduces a Mask Autoencoder (MAE) to randomly mask RGB and event tokens, enhances cross-modal representation consistency through reconstruction tasks, and combines orthogonal high-rank regularization to suppress network fluctuations.

In recent years, the visual prompt learning paradigm has been introduced in target tracking, which adapts specific downstream tasks through prompt modules to fully utilize pre-trained knowledge and reduce dependence on downstream data. Yang et al. [29] proposed ProTrack, which applies prompt learning methods to multi-modal tracking. This model utilizes visual prompts to convert RGB-T and RGB-D data into a unified RGB space. With the diversification of tracking tasks, some researchers are focusing on a universal visual prompt framework. Hong et al. [30] proposed a OneTracker framework, which freezes the basic tracking model and only fine-tunes the cross-modal prompt module, achieving the unified processing of multiple tracking tasks, enabling the model to quickly adapt to diverse tracking scenarios. Due to target tracking being a temporal task, some methods rely on prompt learning to fuse spatio-temporal information. Shi et al. [31] proposed an EVPTrack method, which effectively conveys spatio-temporal information to the current frame by generating an explicit visual prompt, avoiding frequent template update operations. This method enhances the model’s adaptability to scale changes through a multi-scale prompt mechanism.

Although various methods have achieved great success in target tracking, they mainly adopt full-parameter fine-tuning strategies, which have some limitations: (1) Low transfer efficiency: The introduction of multi-modality further increases the complexity of a model, resulting in a significant increase in the resources required for fine-tuning. (2) Catastrophic forgetting of pre-trained knowledge: Unbalanced data sizes can easily lead to the over-fitting of downstream noise in a model. (3) High cost of cross-task deployment: Each downstream task model needs to be stored separately and cannot share common capabilities.

This paper focuses on the issues of multi-modal tracking, combining thermal infrared (T), depth (D), event (E), or other data from different sensors with RGB images, and makes full use of the complementary characteristics between modalities to enhance tracking performance. In this paper, a multi-modal efficient fine-tuning model based on prompt learning, MPVT, is proposed to address the problems of low transfer efficiency, catastrophic forgetting, and weak multi-modal correlation modeling in the traditional full-parameter fine-tuning transfer paradigm.

The remainder of this paper is organized as follows: In Section 2, the multi-modal prompt vision tracker (MPVT) is proposed based on an efficient prompt-tuning paradigm. Decoupled input enhancement, dynamic adaptive prompt fusion, and fully connected head network modules are included in the model. Furthermore, a joint loss function is introduced so as to train the model efficiently. The experimental settings, datasets, details of the experimental results, and analysis are presented in Section 3. Finally, conclusions are summarized in Section 4.

2. Materials and Methods

The framework of MPVT is shown in Figure 1. MPVT aims to fine-tune the pre-trained RGB target tracking model for multi-modal tracking tasks, including RGB-T, RGB-D, and RGB-E, through efficient transfer learning.

Based on the ViT encoder architecture, three components are designed to achieve efficient multi-modal transfer learning. (1) The decoupled input embedding enhancement module first enhances the multi-modal resolution-decoupled position embedding of inputs and then enhances type embedding for both modalities. (2) The dynamic adaptive prompt fusion module is the core of prompt learning, which uses N multi-modal prompt learning modules for transfer learning, while significantly reducing the number of learning parameters by freezing the backbone encoder layer. (3) The fully connected head network module implements two branches of classification and regression, avoiding the inductive bias of traditional convolutional head networks and demonstrating stronger global feature interaction capabilities.

2.1. Decoupled Input Embedding Enhancement Module

For the input template images ${\mathbf{Z}}_{RGB},{\mathbf{Z}}_{AUX}\in {\mathbb{R}}^{H_z\times W_z\times 3}$ and search regions ${\mathbf{X}}_{RGB},$${\mathbf{X}}_{AUX}\in {\mathbb{R}}^{H_x\times W_x\times 3}$, two image embedding layers are used to map them to token sequences. Since the backbone has been pre-trained, only the auxiliary modal embedding layer needs to be fine-tuned, while the RGB embedding layer is frozen. The embedding process can be written as

$$\left.\begin{array}{c}{\tilde{\mathbf{X}}}_{RGB}^0={\mathcal{E}}_{RGB}\left({\mathbf{X}}_{RGB}\right)\hfill \\ {\tilde{\mathbf{Z}}}_{RGB}^0={\mathcal{E}}_{RGB}\left({\mathbf{Z}}_{RGB}\right)\hfill \\ {\tilde{\mathbf{X}}}_{AUX}^0={\mathcal{E}}_{AUX}\left({\mathbf{X}}_{AUX}\right)\hfill \\ {\tilde{\mathbf{Z}}}_{AUX}^0={\mathcal{E}}_{AUX}\left({\mathbf{Z}}_{AUX}\right)\hfill \end{array}\right\},$$

(1)

where ${\tilde{\mathbf{X}}}_{RGB}^0,{\tilde{\mathbf{X}}}_{AUX}^0\in {\mathbb{R}}^{L_x\times d}$, ${\tilde{\mathbf{Z}}}_{RGB}^0,{\tilde{\mathbf{Z}}}_{AUX}^0\in {\mathbb{R}}^{L_z\times d}$, $L_x={\displaystyle \frac{H_x}{s}}\times {\displaystyle \frac{W_x}{s}}$, and $L_z={\displaystyle \frac{H_z}{s}}\times {\displaystyle \frac{W_z}{s}}$, with the patch size being $(s,s)$.

After obtaining the embedding token sequences for each image, this module enhances the input token using two procedures: resolution-decoupled positional embedding enhancement and type embedding enhancement.

Traditional multi-modal tracking models independently embed the positions of the template and searching regions by utilizing one-dimensional absolute embedding. If the resolution of the template region is significantly lower than that of the searching region, this will disrupt the input distribution of the pre-trained ViT model, resulting in the failure of fine-tuning. To solve this problem, unified position embedding through a resolution-decoupled position embedding method is presented. Specifically, let the one-dimensional absolute embedding be $\mathbf{R}$; the corresponding two-dimensional form is ${\mathbf{R}}_{2D}$, as shown in Equation (2). For a searching region of the size $h_x\times w_x$, if $h_x=hand{w}_x=w$, ${\mathbf{R}}_{2D}$ can be directly used as the position embedding of the search region, i.e., ${\mathbf{R}}_x={\mathbf{R}}_{2D}$.

$${\mathbf{R}}_{2D}=\left(\begin{array}{ccc}{r}_{11}& \dots & r_{1w}\\ ⋮& \ddots & ⋮\\ r_{h1}& \dots & r_{hw}\end{array}\right),$$

(2)

A sub-matrix of ${\mathbf{R}}_x$ is used as the position embedding of a template region of the size $h_z\times w_z$ due to its resolution being smaller than that of the searching region. Specifically, starting from the top-left corner of ${\mathbf{R}}_x$, we select a region whose size equals the size of the template region, which can be written as

$${\mathbf{R}}_z=\left(\begin{array}{ccc}{r}_{11}& \dots & r_{1w_z}\\ ⋮& \ddots & ⋮\\ r_{h_{z}1}& \dots & r_{h_z{w}_z}\end{array}\right).$$

(3)

Since the position encoding of the template region ${\mathbf{R}}_z$ and that of the searching region ${\mathbf{R}}_x$ are derived from the same position encoding ${\mathbf{R}}_{2D}$, which is consistent with the pre-trained ViT model, the uniformity of position encoding can be ensured.

Additionally, two learnable embeddings, ${\mathbf{E}}_z^{type}\in {\mathbb{R}}^d$ and ${\mathbf{E}}_x^{type}\in {\mathbb{R}}^d$, are designed to distinguish between the template and searching region embedding and between the targets in the template region and background embedding. ${\mathbf{E}}_z^{type}$ is further divided into the target region embedding ${\mathbf{E}}_{z_o}^{type}\in {\mathbb{R}}^d$ and the background region embedding ${\mathbf{E}}_{z_b}^{type}\in {\mathbb{R}}^d$. Through resolution-decoupled position and type embedding enhancement, the template region embedding ${\mathbf{E}}_z$ and the searching region embedding ${\mathbf{E}}_x$ can be represented as

$${\mathbf{E}}_x^{(i,j)}={\mathbf{R}}_x^{(i,j)}+{\mathbf{E}}_x^{type},$$

(4)

and

$${\mathbf{E}}_z^{(i,j)}=\left\{\begin{array}{cc}{\mathbf{R}}_z^{(i,j)}+{\mathbf{E}}_{z_o}^{type},\hfill & z(i,j)\mathrm{is}\mathrm{the}\mathrm{target}\mathrm{region}\\ {\mathbf{R}}_z^{(i,j)}+{\mathbf{E}}_{z_b}^{type},\hfill & otherwise\hfill \end{array}\right.,$$

(5)

For this reason, the enhanced features can be obtained as

$$\left.\begin{array}{c}{\mathbf{X}}_{RGB}^0={\tilde{\mathbf{X}}}_{RGB}^0+{\mathbf{E}}_x\hfill \\ {\mathbf{Z}}_{RGB}^0={\tilde{\mathbf{z}}}_{RGB}^0+{\mathbf{E}}_z\hfill \\ {\mathbf{X}}_{AUX}^0={\tilde{\mathbf{X}}}_{AUX}^0+{\mathbf{E}}_x\hfill \\ {\mathbf{Z}}_{AUX}^0={\tilde{\mathbf{z}}}_{AUX}^0+{\mathbf{E}}_z\hfill \end{array}\right\}.$$

(6)

2.2. Dynamic Adaptive Prompt Fusion Module

As shown in Figure 1, we assume that there are N encoder layers, ${\mathcal{M}}_1,\dots ,{\mathcal{M}}_N$, in the ViT backbone and the corresponding N multi-modal prompt learning modules ${\mathcal{P}}_1,\dots ,{\mathcal{P}}_N$. The prompt learning process is

$$\left.\begin{array}{c}{\mathbf{C}}^0=Concat({\mathbf{X}}_{RGB}^0,{\mathbf{Z}}_{RGB}^0)\hfill \\ {\mathbf{C}}_{AUX}^0=Concat({\mathbf{X}}_{AUX}^0,{\mathbf{Z}}_{AUX}^0)\hfill \\ {\mathbf{C}}_{AUX}^i={\mathcal{P}}_i({\mathbf{C}}^{i-1},{\mathbf{C}}_{AUX}^{i-1})\hfill \\ {\mathbf{C}}^i={\mathcal{M}}_i({\mathbf{C}}^{i-1}+{\mathbf{C}}_{AUX}^i)\hfill \end{array}\right\},$$

(7)

where Concat represents the concatenation operation of the tokens ${\mathbf{C}}^i,{\mathbf{C}}_{AUX}^i\in {\mathbb{R}}^{L\times d}$, and $L=L_x+L_z$. ${\mathcal{P}}_i$ is the i-th dynamic adaptive prompt fusion module, which mainly includes three core components: (1) dynamic channel scaling convolution for modality-specific weighting, (2) low-rank cross-modal attention for multi-modal interaction, and (3) grouped linear mapping for efficient output mapping, as shown in Figure 2.

(1)

Dynamic Channel Scaling Convolution

For the modal feature $\mathbf{C}$, the statistical value $\mathbf{s}$ of the feature is calculated by

$$\mathbf{s}={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathbf{C}[i,:]\in {\mathbb{R}}^d,$$

(8)

and the dynamic scaling factor $\mathit{\alpha }$ is calculated by

$$\mathit{\alpha }=Softmax({\mathbf{W}}_2·GELU\left({\mathbf{W}}_{\mathbf{1}}\mathbf{s}\right)),$$

(9)

where ${\mathbf{W}}_{\mathbf{1}}\in {\mathbb{R}}^{r\times d}$ and ${\mathbf{W}}_{\mathbf{2}}\in {\mathbb{R}}^{K\times r}$ are learnable parameters, and K is the number of dynamic convolution kernels.

Two types of features are separately learned using depth-wise separable convolution through

$$\left.\begin{array}{c}\mathbf{X}=Reshape\left(\mathbf{C}\right)\in {\mathbb{R}}^{d\times H\times W}\hfill \\ \mathbf{M}=DWConv\left(\mathbf{X}\right)\odot \sum _{k=1}^K\mathit{\alpha }\left[k\right]·{\mathbf{C}}_k\hfill \end{array}\right\},$$

(10)

where $DWConv:{\mathbb{R}}^{d\times H\times W}\to {\mathbb{R}}^{(d/\beta )\times H\times W}$ is a depth-wise separable convolution, ${\mathbf{C}}_k\in {\mathbb{R}}^{(d/\beta )\times 1\times 1}$ is a dynamic channel scaling vector, and $\beta$ is a hyper-parameter of the dimensionality reduction ratio.

The intermediate results $\mathbf{M}$ and ${\mathbf{M}}_{AUX}$ can be obtained by implementing the above dynamic channel scaling convolution on two features, ${\mathbf{C}}^{i-1}$ and ${\mathbf{C}}_{AUX}^{i-1}$.

(2)

Low-Rank Cross-Modal Attention

A low-rank attention mechanism for cross-modal learning is proposed to overcome the large number of parameters and the high complexity of traditional attention mechanisms. For $\mathbf{M}$ and ${\mathbf{M}}_{AUX}$, low rank projection can be obtained by

$$\left.\begin{array}{c}\mathbf{Q}=\mathbf{M}{\mathbf{W}}_Q\in {\mathbb{R}}^{N\times d^{\prime }}\hfill \\ \mathbf{K}=[\mathbf{M},{\mathbf{M}}_{AUX}]{\mathbf{W}}_K\in {\mathbb{R}}^{2N\times d^{\prime }}\hfill \\ \mathbf{V}=[\mathbf{M},{\mathbf{M}}_{AUX}]{\mathbf{W}}_V\in {\mathbb{R}}^{2N\times d^{\prime }}\hfill \end{array}\right\},$$

(11)

where ${\mathbf{W}}_Q,{\mathbf{W}}_K,{\mathbf{W}}_V\in {\mathbb{R}}^{(d/\beta )\times d^{\prime }}$ are learnable parameters. Based on Equation (11), a multi-modal interaction feature, $\mathbf{O}$, can be achieved by fusing the attention mechanism and residual features dynamically, as follows:

$$\left.\begin{array}{c}Attn=Softmax\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{d}}}\right)\in {\mathbb{R}}^{N\times 2N}\hfill \\ \mathbf{O}=\gamma ·\left(Attn\mathbf{V}\right)+(1-\gamma )·{\mathbf{M}}_{AUX}\hfill \end{array}\right\},$$

(12)

where the fusion coefficient $\gamma$ is a learnable parameter.

(3)

Grouped Linear Mapping

Efficient grouped linear mapping is used to calculate prompt output features that in Equation (7), ${\mathbf{C}}_{AUX}^i\in {\mathbb{R}}^{L\times d}$, as follows:

$${\mathbf{C}}_{AUX}^i=\mathrm{GroupLinear}\left(\mathbf{O}\right)={\mathrm{Concat}}_{j=1}^g\left(\mathbf{O}[:,j·d/g:(j+1)·d/g]{\mathbf{W}}_j\right),$$

(13)

where ${\mathbf{W}}_j\in {\mathbb{R}}^{(d^{\prime }/g)\times (d/g)}$ is the group learnable parameter and g is the number of groups. For the sake of computational efficiency, g is taken as 8 in this study.

2.3. Fully Connected Head Network Module

Traditional models such as OSTrack [32] use convolutional head networks for classification and regression, which have strong inductive biases, such as locality assumptions and translation invariance. The local receptive field of convolutional kernels limits the global feature interaction ability. For this reason, a fully connected head network is utilized in this study, which entirely uses multi-layer perceptrons for classification and regression.

In this module, a multi-task joint loss function, ${\mathcal{L}}_{total}={\mathcal{L}}_{cls}+{\mathcal{L}}_{box}$, is used to train the model. The classification loss ${\mathcal{L}}_{cls}$ adopts binary cross-entropy loss (BCE) and can be defined by

$${\mathcal{L}}_{cls}={\displaystyle \frac{1}{N_{pos}}}\sum _{i,j}[-{\overline{y}}_{i,j}log{\widehat{y}}_{i,j}-(1-{\overline{y}}_{i,j})log(1-{\widehat{y}}_{i,j})],$$

(14)

where ${\overline{y}}_{i,j}$ is the IoU value of the predicted box $B_{i,j}^{pred}$ and the true label $B^{gt}$ for each foreground point, $(i,j)$. ${\widehat{y}}_{i,j}=C[i,j]\in [0,1]$ is the predicted score of the classification branch, and $N_{pos}$ is the total number of foreground points.

The regression loss ${\mathcal{L}}_{box}$ adopts generalized IoU (GIoU), which can provide effective gradients even if the predicted and label boxes do not overlap. It can be defined as

$${\mathcal{L}}_{box}={\displaystyle \frac{1}{N_{pos}}}\sum _{i,j}{\mathrm{II}}_{y_{i,j}=1}\left(1-\mathrm{GIoU}(B_{i,j}^{pred},B^{gt})\right),$$

(15)

where II is the indicator function, which only calculates the loss of foreground points.

3. Experimental Results and Discussion

3.1. Datasets

The performance of MPVT was evaluated and compared using three multi-modal datasets, LasHeR [33], DepthTrack [7], and VisEvent [9], corresponding to three multi-modal scenarios: RGB-T, RGB-D, and RGB-E, respectively. LasHeR is the largest and most diverse publicly available dataset in RGB-T target tracking, consisting of 1224 visible light and thermal infrared video pairs with over 730K frames. DepthTrack includes 200 video sequences, with an average sequence length of 1473 frames, providing sufficiently long and continuously changing training data. In addition, it covers 40 different scenario types and 90 target categories. VisEvent contains 820 video pairs, which were collected in complex real-world scenarios such as those with low light, high-speed motion, and cluttered backgrounds. It is widely used for the performance evaluation of RGB-E tracking methods.

3.2. Evaluation Indices

Four evaluation indices are used to evaluate the performance of the methods, that is, the success rate ($Sr$), the tracking recall rate ($TRe$), precision ($Pr$), and the F1-score.

The success rate is the ratio of successfully tracked frames to the total number of frames at a specific IoU threshold, $\beta$. If the IoU value of a frame is greater than or equal to $\beta$, the tracking of that frame is successful. $Sr$ can be defined as

$$Sr={\displaystyle \frac{1}{N}}\sum _{i=1}^N\mathrm{I}(Io{U}_i\ge \beta ),$$

(16)

$$Io{U}_i={\displaystyle \frac{|p_i\cap g_i|}{|p_i\cup g_i|}},$$

(17)

where N is the total number of frames in a video, $Io{U}_i$ is the IoU value of the i-th frame, and $\mathrm{I}(·)$ is the indicator function whose value is 1 if the condition is satisfied and 0 otherwise. $p_i$ is the predicted target position, box and $g_i$ is the label position box.

Precision measures the degree of overlap between predicted and real bounding boxes. It can be defined as

$$Pr={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}Io{U}_i,$$

(18)

The tracking recall rate ($TRe$) is also used as an evaluation index to evaluate the tracking method. Unlike traditional recall, $TRe$ measures the tracker’s ability to correctly detect targets in visible frames. It can be defined as

$$TRe={\displaystyle \frac{{\sum }_{t=1}^N\mathrm{I}({\widehat{v}}_t>\beta ,v_t=1)}{{\sum }_{t=1}^N\mathrm{I}(v_t=1)}},$$

(19)

where $v_t\in \{0,1\}$ is an annotation variable that determines whether the target is visible in frame t, and ${\widehat{v}}_t\in [0,1]$ is the confidence score for frame t.

Generally, a large $Pr$ will lead to a low $TRe$ and vice versa. For a better comparison, a unified index, i.e., the F1-score, combining $Pr$ and $TRe$ is introduced to evaluate the performance of a method. It can be defined as

$$F1=2\times {\displaystyle \frac{Pr·TRe}{Pr+TRe}}.$$

(20)

3.3. Model Parameter Settings

The model was trained for 60 epochs with a batch size of 16; each batch contained $6\times {10}^5$ pairs of data. AdamW was used as the optimization algorithm, with the weight decay being ${10}^{-4}$. The learning rate was updated by the CosineLR scheduling algorithm [34], with the maximal and minimal learning rates being ${10}^{-4}$ and ${10}^{-6}$, respectively. The parameters were set as frozen or learnable for different modules, as shown in Figure 1. The learnable prompt parameters were initialized using the Xavier algorithm [35], and other training parameters are shown in

Table 1. The model parameter settings.

Parameter Name	Value
Layers (N)	40
Hidden dimensions (d)	1536
Size of searching image ($H_x=W_x$)	378
Size of template image ($H_z=W_z$)	196
Feature size of searching region	27
Feature size of template region	14
Epochs	60
Batch size	16
Learning rate	Maximal rate was ${10}^{-4}$; minimal rate was ${10}^{-6}$

.

3.4. Main Results

Various methods were compared with MPVT using three datasets, including TBSI [4], APFNet [5], FANet [17], CAT [12], MDNet [36], PrDIMP50 [37], LTMU [38], OSTrack [32], TransT [39], DAL [8], TATrack [40], TABBTrack [41], ProTrack [29], ViPT [19], and Un-Track [42].

For the RGB-T scenario, various types of RGB-T tracking models were compared with MPVT. Early methods such as FANet [5] mainly improved the complementarity of multi-modal information through feature fusion. Feature decoupling-based methods such as CAT [12] and APFNet [5] introduced difficult attribute branches and progressive fusion mechanisms, respectively. TransT [39] applied Transformer to visual tracking, significantly enhancing the global information interaction between templates and searching regions. OSTrack [32] and TBSI [4] further optimized the Transformer architecture, achieving significant improvements in precision and generalization performance. ProTrack [29] utilized a multi-modal prompt tracking framework, and ViPT [19] further fine-tuned the pre-trained model through visual prompts. We conducted extensive comparisons based on the various methods mentioned above, and the experimental results are shown in the third and fourth columns of

Table 2. The results on LasHeR, DepthTrack, and VisEvent.

Method	${\mathit{R}}_{\mathbf{TP}}$ *	LasHeR		DepthTrack			VisEvent
		$\mathbf{Sr}$ (↑)	$\mathbf{Pr}$ (↑)	$\mathbf{F}$1 (↑)	$\mathbf{TRe}$ (↑)	$\mathbf{Pr}$ (↑)	$\mathbf{Sr}$ (↑)	$\mathbf{Pr}$ (↑)
TBSI [4]	100%	0.506	0.638	-	-	-	-	-
APFNet [5]	100%	0.362	0.500	-	-	-	-	-
FANet [17]	100%	0.309	0.441	-	-	-	-	-
CAT [12]	100%	0.314	0.450	-	-	-	-	-
MDNet [36]	100%	-	-	-	-	-	0.426	0.661
PrDIMP50 [37]	100%	-	-	-	-	-	0.453	0.644
LTMU [38]	100%	-	-	0.460	0.417	0.512	0.459	0.655
OSTrack [32]	100%	0.412	0.515	0.529	0.522	0.536	0.534	0.695
TransT [39]	100%	0.394	0.524	-	-	-	0.474	0.650
DAL [8]	100%	-	-	0.429	0.369	0.512	-	-
TATrack [40]	100%	0.559	0.702	-	-	-	-	-
TABBTrack [41]	100%	-	-	0.618	0.615	0.622	-	-
ProTrack [29]	<1%	0.420	0.538	0.578	0.573	0.583	0.471	0.632
ViPT [19]	0.4%	0.525	0.651	0.594	0.596	0.592	0.592	0.758
Un-Track [42]	6.74%	0.536	0.667	-	-	-	0.589	0.755
MPVT	0.9%	0.578	0.724	0.628	0.626	0.631	0.624	0.761

* $R_{TP}$—ratio of trainable parameters.

. MPVT achieved the best performance among all models, with a success rate of 0.578 and a precision of 0.724. Compared with the second best model, TATrack [40], it increased the success rate and precision by 1.9% and 2.2%, respectively, and compared with ViPT [19], it increased the success rate and precision by 5.3% and 7.3%, respectively.

For the RGB-D scenario, to conduct a comprehensive comparison, various models were introduced as baselines. Besides the methods used in the RGB-T scenario, some other methods were also compared with MPVT. DAL [8] constructed a long-term tracking framework for deep perception through an end-to-end deep feature fusion network. TABBTrack [41] adopted a three-stream architecture and combined temporal features to fuse depth images. The results are shown in columns 5–7 in Table 2. MPVT achieved the highest performance among all the compared models, with an F1-score of 0.628, a tracking recall rate of 0.626, and a precision of 0.631. Compared with ViPT [19], it increased the F1-score, $TRe$, and $Pr$ by 3.4%, 3%, and 3.0%, respectively, and compared with the second-best model, TABBTrack [41], it increased the F1-score, $TRe$, and $Pr$ by 1%, 1.1%, and 0.9%, respectively.

Among all the compared models in the RGB-E scenario, MPVT achieved the highest tracking success rate (0.624) and precision (0.761), with improvements of over 3.2% and 0.3%, respectively, compared with those of the second-best model, ViPT [19], in the RGB-E scenario, as shown in the eighth and ninth columns of Table 2.

Regarding degraded scenarios, Figure 3 and Figure 4 present precision plots for 15 degraded scenarios involving the LasHeR and VisEvent datasets, respectively (DepthTrack is excluded due to unavailable degradation labels). MPVT consistently outperformed baselines across all scenarios. The performance gain was notable under abrupt illumination variation (AIV), hyaline occlusion (HO), and background clutter (BC) in LasHeR and also under low illumination (LI), AIV, and HO in the VisEvent dataset. For instance, MPVT outperformed ViPT by 0.306 under AIV. These results demonstrate the model’s robust performance in degraded conditions.

Regarding efficiency, MPVT has 0.9% (2.8 M) trainable parameters while achieving competitive tracking performance. It shows a higher transfer efficiency compared with traditional fully fine-tuning models, whose trainable parameter ratios are 100%, and learnable parameters are generally greater than 100 M. ViPT is a prompt-based model and has 0.4% (0.84 M) trainable parameters, and although MPVT has more trainable parameters than this, it achieved better tracking performance across all three datasets. Experimental results across the RGB-T, RGB-D, and RGB-E scenarios demonstrate that MPVT preserves the feature extraction capabilities of pre-trained models. Through decoupled input enhancement and dynamic prompt fusion, it avoids feature degradation while effectively integrating complementary multi-modal information. Compared with full-parameter fine-tuning, MPVT maintains higher transfer efficiency.

3.5. Ablation Study

To analyze the effectiveness of the proposed components, ablation experiments were conducted on the three datasets. The experimental results are shown in

Table 3. Ablation results on LasHeR, DepthTrack, and VisEvent.

Method	${{\mathit{R}}_{\mathbf{TP}}}^{†}$	LasHer		DepthTrack			VisEvent
		$\mathbf{Sr}$ (↑)	$\mathbf{Pr}$ (↑)	$\mathbf{F}$1 (↑)	$\mathbf{TRe}$ (↑)	$\mathbf{Pr}$ (↑)	$\mathbf{Sr}$ (↑)	$\mathbf{Pr}$ (↑)
MPVT	0.9%	0.578	0.724	0.628	0.626	0.631	0.624	0.761
MPVT-FC	0.8%	0.562	0.711	0.613	0.612	0.614	0.609	0.758
MPVT-PF	100%	0.523	0.654	0.570	0.569	0.571	0.594	0.754
MPVT-IE	100%	0.517	0.648	0.556	0.554	0.557	0.587	0.752

^†$R_{TP}$—ratio of trainable parameters.

. MPVT, due to the use of efficient training paradigms such as prompt learning, achieved the best performance in RGB-T, RGB-D, and RGB-E, although only 0.9% of the parameters were used for training. First, if the fully connected head network module was removed and the traditional convolutional head network was used instead (MPVT-FC), the performance decreased by 0.01 to 0.02. This indicates that fully connected head networks can effectively avoid the inductive bias of traditional convolutional head networks. Second, if the dynamic adaptive prompt fusion module was removed based on MPVT-FC (MPVT-PF), the model just utilized a full-parameter fine-tuning strategy. In this case, each index decreased by 0.04 to 0.05 on all three datasets. This indicates that the fusion module is crucial for efficient parameter learning and can efficiently transfer from a single-modal pre-trained model on large-scale RGB data to multi-modal scenarios such as RGB-T, RGB-D, and RGB-E. Finally, if the input enhancement module was further removed (MPVT-IE), the performance decreased by about 0.01, proving that the proposed position and type embedding enhancement can contribute to accurate target tracking.

3.6. Model Analysis

3.6.1. Sensitivity Analysis of Prompt Depth

The dynamic adaptive prompt fusion (DAPF) module can be inserted before any ViT encoder layer. The number of DAPF modules used determines the prompt depth: shallow integration applies DAPF only in the first layer, whereas deep integration applies it in every layer. To further assess the impact of prompt fusion depth on tracking performance, we conducted experiments using a scaled-down MPVT model (with 12 encoder layers and 768 hidden dimensions). Performance variations were observed through primary metrics across the three datasets.

This experiment evaluated tracking performance at prompt depths $N_p\in \{1,3,5,9,12\}$. As shown in Figure 5, increasing the prompt depth consistently enhanced performance across all three datasets, demonstrating the effectiveness and scalability of our dynamic adaptive prompt fusion (DAPF) module.

3.6.2. Feature Visualization

A visualization of the tracking results of MPVT and a baseline model in three multi-modal scenarios is shown in Figure 6. From Figure 6d,e, it can be seen that the feature map of MPVT is more concentrated, clearer, and less noisy, while that of the baseline model is more scattered. This indicates that MPVT has a more stable discrimination of targets. From the tracking results in Figure 6b,c, it can also be observed that MPVT (red box) has higher tracking precision compared with the baseline model (green box).

3.6.3. Model Efficiency

The efficiency of MPVT was analyzed from three aspects: training time, GPU memory usage, and inference speed, as shown in

Table 4. Results of efficiency analysis.

Type	Training Time	GPU Memory Usage	Inference Speed
	(Hours)↓	(GB)↓	(fps)↑
Full fine-tuning	62.2	38.1	18
(removing prompt fusion module)
Prompt learning	23.1	21.4	17
(enabling prompt fusion module)

. The training time and GPU memory usage by prompt learning decreased significantly compared with those of full-parameter fine-tuning models, saving 43.8% of GPU memory and 62.9% of training time. In terms of inference speed, due to the addition of model parameters and computational complexity, the inference speed slightly decreased by about 1 fps. The dynamic adaptive prompted fusion (DAPF) module maintained consistent computational complexity for all auxiliary modalities. This minimal inference performance difference enables significant advantages in practice. Regarding edge deployment, 43.8% memory reduction allows operation on resource-constrained devices, and efficiency gains enable the simultaneous processing of more streams.

3.6.4. Differences Between MPVT and Vision–Language Prompting

MPVT shares core principles with CLIP-style prompting [43] but differs in three aspects: (1) Token construction: CLIP uses discrete text tokens, while MPVT employs continuous visual prompts requiring no vocabulary constraints. (2) Fusion granularity: CLIP fuses global semantic features, whereas our low-rank cross-attention preserves spatial details critical for localization. (3) Knowledge transfer: Both leverage frozen backbones, but MPVT’s dynamic scaling adapts to sensor-specific noise patterns unseen in CLIP. These differences highlight how prompt engineering must be rethought for dense prediction tasks.

4. Conclusions

Focusing on the issues of low transfer efficiency, catastrophic forgetting of pre-trained knowledge, and weak multi-modal correlation modeling in traditional full-parameter fine-tuning, an efficient multi-modal prompt vision tracker is proposed to achieve efficient prompt learning on the basis of an RGB pre-trained model. The proposed model can minimize the cost of fine-tuning downstream tasks while achieving higher multi-modal tracking performance. The model mainly includes three key components: a decoupled input enhancement module, a dynamic adaptive prompt fusion module, and a fully connected head network module. The decoupled input enhancement module enhances input from position and type embedding. The dynamic adaptive prompt fusion module achieves efficient prompt learning and multi-modal interaction through scaling convolution and low-rank cross-modal attention learning. The fully connected head network module overcomes the shortcomings of traditional convolutional head networks, such as inductive biases.

The performance of MPVT was evaluated under three multi-modal scenarios: RGB-T, RGB-D, and RGB-E. The experimental results show that the model achieved the best performance on all three multi-modal datasets, achieving tracking precisions of 72.4%, 63.1%, and 76.1%, respectively, and outperforming the second-best model by 2.2%, 0.9%, and 0.3%, respectively. Further ablation studies showed that all three proposed components contribute to more accurate target tracking. Furthermore, the model achieved efficient multi-modal transfer with less than 1% of learnable parameters. The efficiency analysis showed that MPVT is much more effective in training time and GPU memory usage than full-parameter fine-tuning models, saving up to 43.8% of GPU memory and 62.9% of training time.

Recently, many researchers have fused vision and text in various large models [44]. Extending MPVT to vision–language tracking requires solving the fundamental challenge of spatial–semantic alignment, i.e., bridging textual descriptions with visual coordinates while maintaining real-time efficiency. Key research directions include the following: (1) developing geometric-aware attention mechanisms to ground linguistic concepts (e.g., “left”, “occluded”) to pixel-level locations; (2) designing memory-augmented architectures that preserve object identity across evolving text descriptions; and (3) designing efficient fusion schemes to avoid computational explosion when combining visual tokens with linguistic tokens. In future work, we will introduce lingual information into MPVT to further extend multi-modal scenarios.