VETA-CLIP: Lightweight Video Adaptation with Efficient Spatio-Temporal Attention and Variation Loss

Abstract

Full fine-tuning of large-scale vision-language models for video action recognition incurs prohibitive computational cost and often degrades pre-trained spatial representations. To address this, we propose VETA-CLIP, a Video Efficient Temporal Adaptation framework that enhances temporal modeling while preserving cross-modal alignment. By incorporating lightweight adapters into a frozen backbone, VETA-CLIP introduces only 3.55M trainable parameters (a 98% reduction compared to full fine-tuning). Our approach features two key innovations: (1) an Efficient Spatio-Temporal Attention (ESTA) mechanism with a parameter-free boundary replication temporal shift (BRTS) module, which explicitly decouples spatial and temporal attention heads to capture inter-frame dynamics while minimizing disruption to the pre-trained spatial representations; and (2) a novel Variation Loss that maximizes both local inter-frame differences and global temporal variance, encouraging the model to focus on action-related changes rather than static backgrounds. Extensive experiments on HMDB-51, UCF-101, and Something-Something v2 demonstrate that VETA-CLIP achieves competitive performance across zero-shot, base-to-novel, and few-shot protocols, while and remains competitive on the Kinetics-400 dataset. Notably, our eight-frame variant requires only 4.7 GB of peak GPU memory and 2.47 ms of inference per video, demonstrating exceptional computational efficiency alongside consistent accuracy gains.

1. Introduction

Action recognition is one of the core research directions in the field of video understanding, with broad applications in video surveillance [1] and motion analysis [2], and across a wide range of interdisciplinary domains [3,4]. With the remarkable success of large-scale vision-language pre-trained models (e.g., CLIP [5] and ALIGN [6]) in image recognition tasks, a key research focus has shifted toward efficiently adapting these models to the more temporally complex domain of video understanding.

Existing video action recognition methods based on vision-language models still face three critical challenges: (1) High Computational Cost: Approaches like ActionCLIP [7] rely on full model fine-tuning and introduce complex external modules. With a large number of trainable parameters, such methods incur significant computational overhead during training and inference, restricting their deployment in resource-constrained environments. (2) Decreased cross-modal generalization: Extensive fine-tuning of the backbone network inevitably disrupts the pre-trained model’s cross-modal alignment capability, thus affecting zero-shot and cross-dataset transfer performance. (3) Inadequate temporal modeling: Prevailing methods like ViFi-CLIP [8] rely on implicit temporal perception, which is insufficient for tasks with high temporal sensitivity. Conversely, introducing explicit temporal modules [9,10] risks disturbing the carefully pre-trained spatial representations. This dilemma highlights the need for a lightweight mechanism capable of capturing inter-frame motion while minimizing the compromise to spatial alignment.

This conflict stems from representation misalignment. CLIP’s visual encoder is pre-trained to align perfectly with its text encoder. Aggressive temporal modeling disrupts this learned spatial feature manifold, weakening cross-modal generalization—particularly in base-to-novel transfer tasks. To resolve this, our decoupled design enhances cross-frame interaction while minimizing perturbation to original spatial representations.

To address these intertwined challenges, we propose a Video Efficient Temporal Adaptation framework, termed VETA-CLIP. Rather than fully fine-tuning the heavy backbone, VETA-CLIP inserts lightweight Adapters into the frozen CLIP visual encoder, significantly mitigating computational costs while helping retain the pre-trained cross-modal generalization ability. To resolve the conflict between temporal modeling and spatial representation, we design an Efficient Spatio-Temporal Attention (ESTA) mechanism. ESTA explicitly decouples the Transformer attention heads into spatial and temporal groups, operating in tandem with a zero-parameter boundary replication temporal shift (BRTS) module to capture cross-frame dynamics while minimizing entanglement with the pre-trained spatial representations. Furthermore, since standard video-text contrastive objectives primarily focus on global alignment, they often overlook fine-grained temporal dynamics. To counteract this, we introduce a novel Variation Loss that maximizes local inter-frame differences and global temporal variance, compelling the model to focus on salient action cues rather than static backgrounds. Extensive experiments demonstrate that VETA-CLIP achieves competitive performance against state-of-the-art methods on standard benchmarks, including HMDB-51 [11], UCF-101 [12], and Something-something-v2 (SSv2) [13].

The main contributions of our work are as follows:

• Efficient Framework: We propose the VETA-CLIP framework, which introduces only 3.55M trainable parameters (achieving a 98% reduction compared to full fine-tuning). Extensive comparisons and ablation studies validate the effectiveness of this lightweight design.
• Temporal Modeling: We design a decoupled efficient spatio-temporal attention (ESTA) mechanism alongside a zero-parameter boundary replication temporal shift (BRTS) module. This design is intended to facilitate targeted cross-frame interaction while preserving the pre-trained spatial representations.
• Variation Loss: We propose a Variation Loss to capture fine-grained temporal dynamics. By maximizing both local inter-frame differences and global temporal variance, it encourages the encoder to focus on action-related changes rather than static backgrounds, improving discrimination in temporally sensitive tasks.

2. Related Work

2.1. Video Action Recognition

Video action recognition aims to understand and classify human actions from video sequences, with its core challenge lying in the effective modeling of both spatial information and motion-related temporal cues. Vision Transformers (ViTs) [14] have emerged as powerful alternatives to 3D CNNs [15], capturing long-range dependencies more effectively than earlier models such as SlowFast [16], I3D [17], and I-VL [18]. Recently, the rise of multimodal pre-training has propelled vision-language models into video understanding. Pioneering approaches like ActionCLIP [7] and X-CLIP [19] employ the pre-trained CLIP [5] as a backbone, integrating specific temporal modules for video-level recognition. Rasheed et al. [8] further proposed a “Bridge-and-Prompt” framework that learns visual-language prompts while freezing the pre-trained backbone. This parameter-efficient adaptation strategy motivates our work, as it reduces training overhead and helps preserve the pre-trained vision–language alignment.

2.2. Vision-Language Models

Vision-Language (VL) models are pre-trained by leveraging contrastive objectives on massive collections of image-text pairs, which endows them with strong transferability to downstream tasks such as zero-shot [20] and few-shot recognition [21], detection [22,23], and segmentation [24,25]. Nevertheless, being optimized entirely on static images, these foundational models inherently lack the temporal modeling mechanisms critical for video understanding. To address this issue, researchers have proposed various strategies in recent years to adapt vision-language models for video tasks. ActionCLIP [7] introduces a temporal transformer for video–text multimodal learning to enhance temporal reasoning; however, it fully fine-tunes the CLIP backbone, leading to a large trainable parameter budget (168.5M) and high computational cost. In contrast, ViFi-CLIP [8] pursues a more lightweight transfer by encoding videos as unified sequences and learning video-level representations with minimal modification to the pre-trained backbone.

Recent studies have further advanced video-VLM adaptation through parameter-efficient tuning. ViLT-CLIP [26], VLPA-CLIP [27], ST-adapter [28], and M-adapter [29] provide stronger baselines for evaluating the trade-off between adaptation capacity and efficiency.

3. Methodology

In this section, we first introduce VETA-CLIP (Section 3.1) then present the Efficient Spatio-Temporal Attention (Section 3.2), the fine-tuning strategy with adapters (Section 3.3), and finally the proposed Variation Loss (Section 3.4).

3.1. VETA-CLIP

VETA-CLIP is a lightweight framework that enhances the temporal modeling capabilities of CLIP for video action recognition while preserving the stability of its cross-modal alignment. As illustrated in Figure 1, VETA-CLIP adopts a dual-stream architecture comprising a visual encoder and a text encoder. To preserve pre-trained spatial representations and cross-modal alignment, we freeze both backbones and introduce lightweight trainable adapters into the visual branch. In our implementation, the visual encoder adopts the CLIP ViT-B/16 backbone [5] and incorporates the efficient spatio-temporal attention (ESTA) module to enhance cross-frame interaction and temporal modeling. Meanwhile, the text encoder utilizes a standard frozen Transformer to extract text representations. Finally, the entire framework is optimized using the standard contrastive loss for cross-modal alignment and the proposed Variation Loss to enhance temporal sensitivity.

Figure 1. Overview of the parameter-efficient VETA-CLIP framework. The visual encoder incorporates the proposed Efficient Spatio-Temporal Attention (ESTA) module with boundary replication temporal shift (BRTS) and a lightweight adapter, while the text encoder remains frozen.

To prepare the input for the text encoder, we adopt the LLM-generated action descriptions released by Ahmad et al. [30]. Specifically, each action category is converted into a natural-language description using the template “describe [category] as an action performed by humans”. Each description is first tokenized into a sequence of text tokens, which subsequently serves as the input to the text encoder for feature extraction.

To prepare the input for the visual encoder, we sample a video $V=\{f_1,f_2,\dots ,f_T\}\in {\mathbb{R}}^{T\times H\times W\times C}$ consisting of T frames, each with a spatial resolution $(H\times W)$ with C channels. Each frame is divided into $N={\displaystyle \frac{H\times W}{P^2}}$ non-overlapping image patches of size $(P\times P)$. For the t-th frame, its patches ${\left\{f_t^{\left(i\right)}\right\}}_{i=1}^N\in {\mathbb{R}}^{P\times P\times C}$ are linearly projected into D-dimensional patch tokens, denoted as $x_t^{patch}\in {\mathbb{R}}^{N\times D}$. A class token $x_{cls}\in {\mathbb{R}}^D$ is prepended to $x_t^{patch}$, resulting in $[x_{cls};x_t^{patch}]\in {\mathbb{R}}^{(N+1)\times D}$. To inject spatial positional information, a spatial positional embedding $E_{pos}\in {\mathbb{R}}^{(N+1)\times D}$ is added, yielding the final input token sequence to the visual encoder for the t-th frame: $x_t^{image}=[x_{cls};x_t^{patch}]+E_{pos}$. The full video representation is formed by stacking these frame-level tokens: $X=[x_1^{image},x_2^{image},\dots ,x_T^{image}]\in {\mathbb{R}}^{T\times (N+1)\times D}$.

3.2. Efficient Spatio-Temporal Attention

In traditional Multi-Head Self-Attention [31] (MHA), all attention heads are used for spatial modeling, which leads to a redundant distribution of heads. For video inputs, the modeling requirements for spatial texture and temporal features are distinct. To address this, we propose the Efficient Spatio-Temporal Attention (ESTA) mechanism. ESTA explicitly divides the attention heads into two groups: spatial heads and temporal heads. Furthermore, we design a boundary replication temporal shift (BRTS) module specifically applied before the attention projection of the temporal heads. This enables cross-frame interaction with minimal computational overhead while helping preserve the pre-trained spatial representations.

The ESTA module structure is illustrated in Figure 2a, where the h attention heads are divided into two groups: a set of $(h-k)$ heads for spatial modeling and a set of k heads for temporal modeling. The temporal–head ratio is chosen to balance temporal modeling capacity with the preservation of pre-trained spatial representations. Based on the sensitivity analysis in Section 4.4.2, we set k to $1/6$ and $1/4$ of the total heads for 8-frame and 16-frame inputs, respectively, as these settings perform best in our experiments. For an 8-frame input, we allocate $1/6$ of the total heads for temporal modeling ($k={\displaystyle \frac{h}{6}}$), while for a 16-frame input, the number of temporal heads increases slightly ($k={\displaystyle \frac{h}{4}}$) to better capture temporal features.

Figure 2. Illustration of the proposed Efficient Spatio-Temporal Attention (ESTA) module. (a) Attention heads are explicitly split into spatial and temporal groups. The boundary replication temporal shift (BRTS) is applied before the projection of temporal heads to enable efficient cross-frame interaction. (b) BRTS shifts channel features along the temporal dimension and fuses them with the original input via a residual connection.

To enhance cross-frame interaction, we apply the BRTS module for frame feature migration on each video input $X\in {\mathbb{R}}^{T\times (N+1)\times D}$. For the input to the BRTS module, we first strip the [CLS] token from the token sequence and reshape the remaining spatial patches into a 4D spatio-temporal tensor, denoted as $X_{\mathrm{input}}=\{{\tilde{x}}_1,{\tilde{x}}_2,\dots ,{\tilde{x}}_T\}\in {\mathbb{R}}^{T\times \tilde{H}\times \tilde{W}\times D}$, where D is the embedding dimension, and $\tilde{H}$ and $\tilde{W}$ represent the downsampled spatial grid dimensions (i.e., $\tilde{H}\times \tilde{W}=N$). As shown in Figure 2b, the BRTS module splits the channel dimension D into three groups based on a hyperparameter $n_{\mathrm{div}}$ (default is 8).

• Left-Shift Group: The feature of the current frame at m (where $m={\displaystyle \frac{D}{n_{\mathrm{div}}}}$) dimensions is replaced by the corresponding feature from the previous frame. The first frame uses self-replication as the source.
• Right-Shift Group: For another m dimensions, the features of the current frame are replaced by the corresponding features from the next frame. The last frame uses self-replication for the right shift.
• Retained Group: The remaining $D-2m$ dimensions retain their original features.

In the traditional Temporal Shift Module [32] (TSM), which was primarily designed for CNN architectures, zero-padding is typically used for boundary frames. However, applying zero-padding to a pre-trained Vision Transformer introduces artificial zero-values that act as disruptive noise to the well-aligned feature distribution. To address this, our BRTS module employs a self-replication strategy for boundary frames, preserving information integrity without introducing noise. More importantly, unlike CNN-based TSM, BRTS is tailored for ViT: we strip the [CLS] token before shifting and re-concatenate it afterward, ensuring that the global semantic representation, which is critical for CLIP’s vision-language alignment, remains largely isolated from temporal perturbations. Subsequently, the temporally shifted tensor $X_{\mathrm{shifted}}$ is fused with the original input $X_{\mathrm{input}}$ via a residual connection. Finally, the updated spatial patches are flattened back into a sequence and re-concatenated with the previously stripped [CLS] token. This additive operation and token reintegration ensure that the final output $X_{\mathrm{output}}\in {\mathbb{R}}^{T\times (N+1)\times D}$ encapsulates both the pre-trained spatial representations and the newly integrated cross-frame dynamics.

To better preserve the pre-trained semantic alignment, the spatial heads derive their queries, keys, and values directly from the unshifted input X. In contrast, the temporal heads project from the BRTS output $X_{\mathrm{output}}$ to perform targeted cross-frame modeling. The linear projection matrices for the i-th attention head are denoted as $W_{\mathrm{attn}}^{Q,i},W_{\mathrm{attn}}^{K,i},W_{\mathrm{attn}}^{V,i}\in {\mathbb{R}}^{D\times d}$, where D is the embedding dimension and $d={\displaystyle \frac{D}{h}}$ represents the dimension of each individual head. The attention calculation for the spatial and temporal heads is formulated as follows:

$$Q_s^i=X{W}_{\mathrm{attn}}^{Q,i},K_s^i=X{W}_{\mathrm{attn}}^{K,i},V_s^i=X{W}_{\mathrm{attn}}^{V,i}$$

(1)

$$Q_t^i,K_t^i,V_t^i=X_{\mathrm{output}}{W}_{\mathrm{attn}}^{Q,i},X_{\mathrm{output}}{W}_{\mathrm{attn}}^{K,i},X_{\mathrm{output}}{W}_{\mathrm{attn}}^{V,i}$$

(2)

$${\mathrm{S}\text{-}\mathrm{head}}_i=\mathrm{Attention}(Q_s^i,K_s^i,V_s^i),\mathrm{where}i\in \{k+1,\dots ,h\}$$

(3)

$${\mathrm{T}\text{-}\mathrm{head}}_i=\mathrm{Attention}(Q_t^i,K_t^i,V_t^i),\mathrm{where}i\in \{1,\dots ,k\}$$

(4)

Finally, the outputs of all temporal and spatial heads are concatenated and linearly transformed by an output projection matrix $W_{\mathrm{attn}}^O\in {\mathbb{R}}^{D\times D}$ to produce the ESTA module’s output:

$$\mathrm{ESTA}\left(X\right)=\mathrm{Concat}({\mathrm{T}\text{-}\mathrm{head}}_1,\dots ,{\mathrm{T}\text{-}\mathrm{head}}_k,{\mathrm{S}\text{-}\mathrm{head}}_{k+1},\dots ,{\mathrm{S}\text{-}\mathrm{head}}_h)W_{\mathrm{attn}}^O$$

(5)

Unlike previous methods such as TDN [9], which compute temporal differences via intrusive structural modifications suitable for train-from-scratch CNNs, or approaches relying on uniform temporal attention [33], our method explicitly decouples spatial and temporal modeling. Applying intrusive modifications to frozen vision-language models would severely distort their pre-trained spatial representations. By decoupling the attention heads, ESTA helps preserve the pre-trained zero-shot capability in the spatial pathway, while the dedicated temporal heads focus on cross-frame dynamics extracted via BRTS. This method simplifies the model architecture, improves spatio-temporal modeling performance, and reduces the adaptation difficulty for transferring vision-language models from image to video tasks.

3.3. Fine-Tuning VETA-CLIP

To reduce computational costs during the fine-tuning of VETA-CLIP, we draw inspiration from efficient fine-tuning techniques in Natural Language Processing (NLP) [34,35,36] and adopt the lightweight bottleneck adapter technique [37]. As shown in Figure 3, the main function of the adapter is to perform dimensional transformation and non-linear activation on the input visual features. The first fully connected layer (Linear 1) projects the input features into a lower-dimensional hidden space, while the second fully connected layer (Linear 2) restores the original dimension. We insert the adapter after the ESTA module and before the MLP layer. Since post-ESTA representations already encode a rich spatio-temporal context, modulating features at this point efficiently injects task-specific knowledge without directly modifying the attention computation itself. This placement allows lightweight adaptation of the contextualized token representations before the MLP block. The adapter uses a skip connection for training stability, and its linear projections are initialized with a uniform Kaiming distribution. This overall design introduces only 3.55M trainable parameters.

Figure 3. The adapter module, featuring a down-projection (Linear 1), a GELU activation, and an up-projection (Linear 2) for visual feature adaptation.

3.4. Variation Loss

In the traditional CLIP [5] framework, a bidirectional contrastive loss (${\mathcal{L}}_c$) is used to align visual and textual embeddings, consisting of an image-to-text and a text-to-image term:

$${\mathcal{L}}_{\mathrm{visual}}=-{\displaystyle \frac{1}{B}}\sum _{i=1}^{B}log{\displaystyle \frac{exp(S_{i,i}/\tau )}{{\sum }_{j=1}^{B}exp(S_{i,j}/\tau )}},{\mathcal{L}}_{\mathrm{text}}=-{\displaystyle \frac{1}{B}}\sum _{i=1}^{B}log{\displaystyle \frac{exp(S_{i,i}/\tau )}{{\sum }_{j=1}^{B}exp(S_{j,i}/\tau )}}$$

(6)

$${\mathcal{L}}_c={\displaystyle \frac{1}{2}}({\mathcal{L}}_{\mathrm{visual}}+{\mathcal{L}}_{\mathrm{text}})$$

(7)

where $S_{i,j}$ denotes the cosine similarity between the i-th visual representation ($v_i$) and the j-th text representation ($y_j$). Here, B is the batch size, and $\tau$ is a temperature parameter. This loss encourages maximization of similarity between matching pairs while minimizing the similarity of non-matching pairs.

Although ${\mathcal{L}}_c$ excels at multimodal alignment, it operates primarily on an aggregated video-level representation, which can obscure fine-grained temporal dynamics. To alleviate temporal over-smoothing and model dynamic inter-frame variations with minimal disruption to pre-trained alignment, we introduce a Variation Loss (${\mathcal{L}}_v$) that maximizes both local temporal differences and global sequence diversity.

We define the input video as a set of T frames captured via a uniform temporal sampling strategy. After passing through the visual encoder, we obtain frame-level embeddings: $\{e_1,e_2,\dots ,e_T\}\in {\mathbb{R}}^{T\times D}$. Note that these frame-level embeddings $e_t$ are obtained from the adapted visual encoder. Because these representations are inherently constrained by the contrastive learning space, they provide a stable foundation for the Variation Loss to capture inter-frame dynamics without the risk of feature magnitude instability. As illustrated in Figure 4, ${\mathcal{L}}_v$ captures inter-frame variation from two complementary perspectives: (i) the local temporal difference ($D_{\mathrm{local}}$) and (ii) global temporal variance ($\mathrm{Var}$).

Figure 4. A schematic of the Variation Loss (${\mathcal{L}}_v$) computation.

• Local Temporal Difference ($D_{\mathrm{local}}$): We propose a local temporal difference measure $D_{\mathrm{local}}$ to quantify temporal variations between consecutive video frames. Specifically, for the t-th frame, we calculate the difference for each feature dimension as a measure of change between the two adjacent frames, defined as

  $$D_{\mathrm{local}}^{\left(t\right)}={\displaystyle \frac{1}{D}}\sum _{d=1}^D{\left(e_{t+1}^{\left(d\right)}-e_t^{\left(d\right)}\right)}^2$$

  (8)

Averaging the differences across all adjacent frames yields the total local temporal difference $D_{\mathrm{local}}$:

$$D_{\mathrm{local}}={\displaystyle \frac{1}{T-1}}\sum _{t=1}^{T-1}{D}_{\mathrm{local}}^{\left(t\right)}={\displaystyle \frac{1}{D(T-1)}}\sum _{t=1}^{T-1}\sum _{d=1}^D{\left(e_{t+1}^{\left(d\right)}-e_t^{\left(d\right)}\right)}^2$$

(9)

where $e_t^{\left(d\right)}$ is the feature at the d-th dimension of the t-th frame, D is the feature dimension per frame, and T is the number of input frames. Computing differences per dimension helps the model independently analyze variations in each feature dimension, thereby enhancing its sensitivity to dimensions with significant changes.

• Global Temporal Variance ($\mathrm{Var}$): To ensure diversity across the entire video sequence, we calculate the feature variance along the temporal dimension. The variance for the d-th dimension is defined as ${\mathrm{Var}}_d$:

  $${\mathrm{Var}}_d={\displaystyle \frac{1}{T}}\sum _{t=1}^T{\left(e_t^{\left(d\right)}-e_{\mathrm{mean}}^{\left(d\right)}\right)}^2,\mathrm{where}{e}_{\mathrm{mean}}^{\left(d\right)}={\displaystyle \frac{1}{T}}\sum _{t=1}^T{e}_t^{\left(d\right)}$$

  (10)

The overall diversity variance $\mathrm{Var}$ is then computed by averaging across all D dimensions:

$$\mathrm{Var}={\displaystyle \frac{1}{D}}\sum _{d=1}^D{\mathrm{Var}}_d$$

(11)

To better capture action dynamics, the model is encouraged to increase both local temporal differences ($D_{\mathrm{local}}$) and global temporal variance ($\mathrm{Var}$). We formulate the Variation Loss ${\mathcal{L}}_v$ in a reciprocal form:

$${\mathcal{L}}_v={\displaystyle \frac{1}{\delta +D_{\mathrm{local}}+\mathrm{Var}}}$$

(12)

Here, $\delta =1$ is introduced for numerical stability. We use a reciprocal formulation instead of an unbounded additive negative objective (e.g., $-(D_{\mathrm{local}}+\mathrm{Var})$) because it yields smoother and more stable optimization behavior. As $D_{\mathrm{local}}+\mathrm{Var}$ increases, ${\mathcal{L}}_v$ gradually approaches zero, and the gradient is scaled by a factor of $1/{(\delta +D_{\mathrm{local}}+\mathrm{Var})}^2$, introducing a natural diminishing-return effect. This property helps prevent the auxiliary objective from dominating the fine-tuning process and helps preserve the primacy of the contrastive loss, thereby reducing the risk of overly perturbing the pre-trained cross-modal semantic alignment.

The final total loss function ${\mathcal{L}}_{\mathrm{total}}$ integrates the traditional contrastive loss ${\mathcal{L}}_c$ and the proposed Variation Loss ${\mathcal{L}}_v$:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_c(v,y)+\lambda {\mathcal{L}}_v$$

(13)

In all experiments, we set $\lambda =1$, which is selected based on the ablation results in Section 4.4.2. The proposed Variation Loss is fundamentally different from existing temporal constraints and regularization methods. Unlike temporal order constraints that impose explicit directional supervision, such as forward/backward sequence prediction, our objective measures the undirected magnitude of temporal variation. This design is more suitable for a frozen CLIP backbone, as it enhances temporal sensitivity without explicitly enforcing sequence ordering on the pre-trained vision–language representation space. Moreover, unlike conventional feature variance regularization, which is mainly introduced to prevent representation collapse from a global statistical perspective, the proposed loss serves as a task-oriented temporal enhancement objective. By jointly combining a local temporal difference term ($D_{\mathrm{local}}$) and a global temporal variance term ($\mathrm{Var}$), it captures both short-range inter-frame dynamics and sequence-level temporal diversity. In addition, the reciprocal formulation introduces a natural self-limiting effect: as $D_{\mathrm{local}}$ and $\mathrm{Var}$ increase, the gradient magnitude of the auxiliary objective decreases accordingly. This property helps the proposed loss enhance motion-sensitive variation while reducing the risk of overwhelming the primary contrastive objective (${\mathcal{L}}_c$), thereby helping preserve the robust semantic structure of the frozen backbone.

4. Experimental Results and Discussion

We first introduce the datasets (Section 4.1) and implementation settings (Section 4.2). Next, we compare VETA-CLIP with existing methods (Section 4.3) and conduct ablation studies to validate our designs (Section 4.4). Finally, we provide a comprehensive efficiency analysis to highlight the practical advantages of our framework (Section 4.5).

4.1. Datasets

We evaluate our approach on four widely used action recognition benchmark datasets: two smaller-scale datasets, HMDB-51 [11] and UCF-101 [12], as well as two large-scale datasets, Something-something-v2 (SSv2) [13] and Kinetics-400 [38]. Across all datasets, we adopt Top-1 classification accuracy (%) as the primary evaluation metric.

HMDB-51 contains 6766 videos across 51 categories, sourced from movies and online videos. The dataset features significant camera motion and a low resolution, making it a robust testbed for evaluating model generalization under challenging conditions.

UCF-101 comprises 13,320 realistic action videos from YouTube, covering 101 categories. It exhibits large variations in camera motion, object appearance, viewpoint, and background clutter, primarily testing a model’s robustness to complex spatial environments.

SSv2 is a large-scale dataset comprising 220,847 video clips spanning 174 classes, posing a substantial challenge for temporal modeling due to its emphasis on fine-grained human–object interactions.

Kinetics-400 [38] is another large-scale dataset containing 306,245 video clips across 400 action categories, with each class comprising approximately 400 to 1150 clips, each around 10 s in duration. Given its large scale and comprehensive category coverage, it serves as a fundamental benchmark for large-scale video representation learning.

4.2. Implementation Details

In the experiments, we use a ViT-B/16-based CLIP model as the visual encoder. To balance performance with computational throughput, we sparsely sample a sequence of 8 frames from each video instance unless specified, resizing spatial dimensions to $224\times 224$. For data preprocessing, we adopt a simple and controlled setting without additional RandAugment or temporal random shifting, and rely only on the standard CLIP image normalization pipeline.

Unless otherwise specified, all main experiments are conducted on a single NVIDIA GeForce RTX 5090 (32 GB) GPU with a batch size of 50. For optimization, we employ the AdamW optimizer. To ensure optimization stability and strict reproducibility, we utilize a cosine annealing learning rate schedule coupled with a linear warmup phase for the first 5 epochs, gradually scaling the learning rate to the peak value of $1\times {10}^{-5}$. The model is trained for over 50 epochs. Furthermore, regularization is enforced via a weight decay factor of 0.2 and a dropout rate of 0.1 after the fully connected layers to mitigate overfitting. To ensure reliability, all results are averaged over three independent runs. Standard deviations are reported in Table 1 to indicate result stability, and the corresponding 95% confidence intervals are provided in Appendix A (Table A3). For readability, they are omitted from subsequent dense tables.

Table 1. Zero-shot action recognition performance on HMDB-51 and UCF-101.

Method	Backbone	HMDB-51 Top-1 (%)	UCF-101 Top-1 (%)
Uni-modal zero-shot action recognition models
ASR (ECML′17) [39]	BiDiLEL	$21.8\pm 0.9$	$24.4\pm 1.0$
ZSECOC (CVPR′17) [40]	Fisher Vectors	$22.6\pm 1.2$	$15.1\pm 1.7$
UR (CVPR′18) [41]	ResNet-200	$24.4\pm 1.6$	$17.5\pm 1.6$
E2E (CVPR′20) [42]	R(2+1)D-18	32.7	48.0
ER-ZSAR (ICCV′21) [43]	TSM	$35.3\pm 4.6$	$51.8\pm 2.9$
Adapting pre-trained image VL models
Vanilla CLIP (ICML′21) [5]	ViT-16	$40.8\pm 0.3$	$63.2\pm 0.2$
ActionCLIP (TNNLS′23) [7]	ViT-16	$40.8\pm 5.4$	$58.3\pm 3.4$
XCLIP (ECCV′22) [19]	ViT-16	$44.6\pm 5.2$	$72.0\pm 2.3$
A5 (ECCV′22) [18]	ViT-16	$44.3\pm 2.2$	$69.3\pm 4.2$
ViFi-CLIP (CVPR′23) [8]	ViT-16	$51.3\pm 0.6$	$76.8\pm 0.7$
ViLT-CLIP (AAAI′24) [26]	ViT-16	$45.3\pm 0.9$	$69.3\pm 1.1$
VLPA-CLIP (PR′25) [27]	ViT-16	$49.2\pm 0.4$	$76.1\pm 0.4$
VETA-CLIP	ViT-16	$53.5\pm 0.5$	$80.3\pm 0.7$

4.3. Evaluation and Comparisons

This section evaluates the effectiveness of the proposed method through comprehensive testing on the HMDB-51, UCF-101, SSv2, and Kinetics-400 datasets. The experimental evaluation consists of four settings: (1) zero-shot, (2) base-to-novel, (3) fully supervised, and (4) few-shot.

4.3.1. Zero-Shot Evaluation

For zero-shot evaluation, we compare VETA-CLIP with both specialized unimodal zero-shot methods and existing vision-language models adapted to video tasks. Following standard protocols, all models are trained on Kinetics-400 and tested on HMDB-51 and UCF-101, whose categories are disjoint from those of Kinetics-400. We report the top-1 accuracy as the evaluation metric. As shown in Table 2, existing video-adapted models often append heavy trainable layers to capture motion dynamics, which severely bottlenecks their inference throughput. In contrast, VETA-CLIP circumvents this issue by inserting lightweight adapters into a frozen backbone and employing the ESTA module for efficient temporal modeling. By introducing merely 3.55M trainable parameters, our framework maintains competitive computational complexity (GFLOPs) while achieving a favorable trade-off between parameter efficiency and inference throughput.

Table 2. A comparison of performance parameters.

Method	GFLOPs ↓	TP ↑	Total Params (M) ↓	Tunable Params (M) ↓
ActionCLIP (TNNLS′23) [7]	563	67.7	168.5	168.5
XCLIP (ECCV′22) [19]	287	58.5	131.5	131.5
ViFi-CLIP (CVPR′23) [8]	281	71.1	124.7	124.7
ViLT-CLIP (AAAI′24) [26]	287	72.1	-	-
VLPA-CLIP (PR′25) [27]	159	109.7	148.8	24.1
VETA-CLIP	101	330.5	152.6	3.55

As shown in Table 1, the baseline CLIP model already exhibits stronger generalization than traditional unimodal methods. Building on this, our VETA-CLIP outperforms existing vision language models adapted for video, achieving gains of +2.2% and +3.5% on HMDB-51 and UCF-101, respectively. We further evaluate the efficiency of VETA-CLIP in Table 2, comparing computational cost (GFLOPs), inference throughput (TP), and parameter counts (total and trainable, in millions) against leading competitors.

4.3.2. Base-to-Novel Evaluation

Base-to-Novel Generalization: To evaluate transferability to unseen categories, we follow the base-to-novel protocol established by ViFi-CLIP [8]. Specifically, action categories are divided into disjoint Base and Novel sets, with training restricted exclusively to the Base classes. Table 3 reports the top-1 accuracy (%) on both sets, along with their harmonic mean (HM), across HMDB-51, UCF-101, SSv2, and Kinetics-400.

Table 3. Base-to-novel generalization performance on HMDB-51, UCF-101, SSv2, and Kinetics-400.

Method	HMDB-51 Top-1 (%)			UCF-101 Top-1 (%)			SSv2 Top-1 (%)			Kinetics-400 Top-1 (%)
	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM
Vanilla CLIP (ICML′21) [5]	53.3	46.8	49.8	78.5	63.6	70.3	4.9	5.3	5.1	62.3	53.4	57.5
ActionCLIP (TNNLS′23) [7]	69.1	37.3	48.5	90.1	58.1	70.7	13.3	10.1	11.5	61.0	46.2	52.6
XCLIP (ECCV′22) [19]	69.4	45.5	55.0	89.9	58.9	71.2	8.5	6.6	7.4	74.1	56.4	64.0
A5 (ECCV′22) [18]	46.2	16.0	23.8	90.5	40.4	55.8	8.3	5.3	6.4	69.7	37.6	48.8
ViFi-CLIP (CVPR′23) [8]	73.8	53.3	61.9	92.9	67.7	78.3	16.2	12.1	13.9	76.4	61.1	67.9
ViLT-CLIP (AAAI′24) [26]	76.7	57.5	65.7	95.2	70.5	81.0	17.3	12.8	14.7	77.4	63.0	69.5
VETA-CLIP	78.1	59.3	67.4	95.4	78.2	85.9	17.6	14.5	15.9	75.9	61.0	67.6

Table 3 demonstrates that VETA-CLIP exhibits strong base-to-novel generalization capabilities across most evaluated benchmarks. On UCF-101, we improve the Novel class accuracy from 70.5% (ViLT-CLIP [26]) to 78.2%, demonstrating that our lightweight adapter captures temporal dynamics and generalizes well to unseen novel classes without severe overfitting to base classes. Similar improvements are observed on both HMDB-51 and SSv2, further confirming the robustness of our design.

However, on the large-scale Kinetics-400 dataset, although VETA-CLIP remains competitive with earlier methods such as ViFi-CLIP, it underperforms the more recent ViLT-CLIP [26]. This gap reflects an inherent trade-off between extreme parameter efficiency and model capacity. Specifically, Kinetics-400 covers a broader and more diverse semantic space, where stronger task-specific adaptation and richer expressive power become increasingly important. Under such a setting, our ultra-lightweight design, with only 3.55M trainable parameters, may encounter a mild capacity bottleneck in modeling highly diverse action categories.

In contrast, methods with richer multimodal adaptation strategies, such as ViLT-CLIP [26], may offer greater expressive flexibility for large-scale category variation. As reported in Table 2, ViLT-CLIP is also associated with substantially higher computational cost and lower throughput than VETA-CLIP (287 vs. 101 GFLOPs; 72.1 vs. 330.5 TP). Therefore, while VETA-CLIP is particularly advantageous for efficient transfer and deployment under constrained computational budgets, its deliberately limited parameter budget may impose a natural upper bound on peak performance for very large and semantically diverse benchmarks.

To further visualize this transferability, Figure 5 provides the confusion matrices on the novel categories. The pronounced diagonal dominance and sparse off-diagonal misclassifications corroborate our quantitative results, confirming that VETA-CLIP maintains high discriminative power without severe confusion among unseen classes.

Figure 5. Confusion matrices of the proposed VETA-CLIP on the novel categories under the base-to-novel generalization protocol. (a) Results on HMDB-51 (25 novel classes); (b) results on UCF-101 (50 novel classes).

4.3.3. Fully-Supervised Evaluation

We evaluate VETA-CLIP on the Kinetics-400 dataset under the fully supervised setting. Following standard protocols, we report top-1 and top-5 accuracy to compare with current state-of-the-art methods.

As shown in Table 4, VETA-CLIP achieves a competitive performance on Kinetics-400 with exceptional parameter efficiency. Notably, our 8-frame variant achieves 83.2% Top-1 accuracy, outperforming several 16-frame adapters, such as ST-adapter [28] and M-adapter [29]. When using 16 frames, VETA-CLIP reaches 83.7%, matching the accuracy of VLPA-CLIP [27] while utilizing nearly seven times fewer trainable parameters (3.55M vs. 24.1M). Its performance naturally trails behind massive models like ViCLIP [44], which achieves 88.7% Top-1 accuracy with 300M parameters, reflecting the inherent capacity trade-off of our ultra-lightweight design. The superior accuracy of ViCLIP [44] is primarily driven by extensive pre-training on the large-scale InternVid dataset and a complex architecture that combines masked modeling with spatio-temporal attention.

Table 4. Performance comparison under the fully-supervised setting on the Kinetics-400 dataset. ✓ indicates full fine-tuning of the pre-trained backbone, while × denotes parameter-efficient fine-tuning (the backbone is frozen).

Method	Finetuning	Tunable Param	Frames	Views	Top-1 (%)	Top-5 (%)
Uni-modal architectures
Uniformer-B (ICLR′22) [45]	✓	50.3M	32	$4\times 3$	83.0	95.4
TimeSformer-L (ICML′21) [46]	✓	121M	96	$1\times 3$	80.7	94.7
Mformer-HR (NeurIPS′21) [47]	✓	382M	16	$10\times 3$	81.1	95.2
Swin-L (CVPR′22) [48]	✓	197M	32	$4\times 3$	83.1	95.9
Tuning pre-trained image VL models
ActionCLIP (TNNLS′23) [7]	✓	168.5M	32	$10\times 3$	83.8	96.2
XCLIP (ECCV′22) [19]	✓	131.5M	16	$4\times 3$	84.7	96.8
ViFi-CLIP (CVPR′23) [8]	✓	124.7M	16	$4\times 3$	83.9	96.3
ViCLIP (ICLR′24) [44]	✓	300M	16	$4\times 3$	88.7	98.2
ST-adapter (NeurIPS′22) [28]	×	7M	16	$1\times 3$	82.5	96.0
M-adapter (CVIU′24) [29]	×	4M	16	$4\times 3$	83.1	96.1
ViLT-CLIP (AAAI′24) [26]	×	-	16	$4\times 3$	77.6	94.5
VLPA-CLIP (PR′25) [27]	×	24.1M	16	$4\times 3$	83.7	96.7
VETA-CLIP	×	3.55M	8	$4\times 3$	83.2	96.0
VETA-CLIP	×	3.55M	16	$4\times 3$	83.7	96.4

4.3.4. Few-Shot Evaluation

Few-shot Evaluation: To evaluate adaptation in data-scarce scenarios, we construct K-shot training subsets by randomly sampling 2, 4, 8, or 16 videos per category. The fine-tuned model is then evaluated on the standard validation split of the corresponding dataset.

Table 5 summarizes the few-shot learning performance of VETA-CLIP in comparison with existing CLIP-based video adaptation methods. As expected, performance exhibits a steady upward trend as the shot number K increases. While the multimodal prompt-based ViLT-CLIP [26] shows strong results in extremely data-scarce regimes (e.g., $K=2$ on HMDB-51), VETA-CLIP consistently secures the top position as the number of training samples increases, which underscores its robustness and strong adaptability in few-shot settings. Specifically, at $K=16$, VETA-CLIP outperforms the prior state-of-the-art by absolute margins of 1.8%, 1.1%, and 0.7% on HMDB-51, UCF-101, and SSv2, respectively. Nevertheless, its advantage becomes less pronounced in extremely data-scarce regimes. For example, on HMDB-51 with $K=2$, the multimodal prompt-based ViLT-CLIP [26] slightly outperforms VETA-CLIP. This observation suggests that, under ultra-low-shot conditions, methods exploiting textual priors through prompt learning may retain an advantage, whereas our lightweight temporal adaptation strategy, which mainly enhances visual feature dynamics, may require slightly richer supervision to fully realize its benefits.

Table 5. Few-shot learning performance comparison on HMDB-51, UCF-101, and SSv2.

Method	HMDB-51 Top-1 (%)				UCF-101 Top-1 (%)				SSv2 Top-1 (%)
	K = 2	K = 4	K = 8	K = 16	K = 2	K = 4	K = 8	K = 16	K = 2	K = 4	K = 8	K = 16
Vanilla CLIP (ICML′21) [5]	41.9	41.9	41.9	41.9	63.6	63.6	63.6	63.6	2.7	2.7	2.7	2.7
ActionCLIP (TNNLS′23) [7]	47.5	57.9	57.3	59.1	70.6	71.5	73.0	91.4	4.8	5.8	8.4	11.1
XCLIP (ECCV′22) [19]	53.0	57.3	62.8	64.0	48.5	75.6	83.7	91.4	3.9	4.5	6.8	10.0
A5 (ECCV′22) [18]	39.7	50.7	56.0	62.4	71.4	79.9	85.7	89.9	4.4	5.1	6.1	9.7
ViFi-CLIP (CVPR′23) [8]	57.2	62.7	64.5	66.8	80.7	85.1	90.0	92.7	6.2	7.4	8.5	12.4
ViLT-CLIP (AAAI′24) [26]	60.6	61.9	66.9	69.6	85.3	90.0	91.3	93.8	7.8	9.4	10.3	13.2
VLPA-CLIP (PR′25) [27]	54.7	59.4	63.7	66.6	78.2	85.0	89.5	92.7	5.8	6.4	8.6	11.4
VETA-CLIP	58.7	66.0	69.2	71.4	85.3	90.1	92.8	94.9	7.8	9.6	10.5	13.9

4.4. Ablation Study

We perform ablation studies on HMDB-51, UCF-101, and SSv2 to assess two aspects: (1) the contribution of each proposed component (Section 4.4.1) and (2) the impact of key hyperparameters (Section 4.4.2).

4.4.1. Module Ablation

To validate the contributions of Efficient Spatio-Temporal Attention (ESTA) and Variation Loss, we construct a baseline by freezing the CLIP [5] backbone and inserting lightweight adapters only into the video encoder, without ESTA or Variation Loss. As shown in Table 6 and Table 7, adding these components consistently improves performance over the baseline across all benchmarks.

Table 6. Ablation analysis of VETA-CLIP across zero-shot and base-to-novel generalization protocols.

Method	HMDB-51	UCF-101	HMDB-51 Top-1 (%)			UCF-101 Top-1 (%)			SSv2 Top-1 (%)
	Zero-Shot Top-1 (%)		Base	Novel	HM	Base	Novel	HM	Base	Novel	HM
Baseline	49.6	76.1	72.3	49.6	58.8	92.9	73.1	81.8	11.9	8.7	10.1
+ Variation Loss	50.4	77.0	73.7	54.7	62.8	93.4	73.8	82.4	13.5	10.9	12.1
+ ESTA	51.9	78.7	75.2	55.9	64.3	94.7	75.8	84.2	16.2	12.6	14.2
+ ESTA + Variation Loss	53.5	80.3	78.1	59.3	67.4	95.4	78.2	85.9	17.6	14.5	15.9

Table 7. Ablation study results under the few-shot learning experimental setting of VETA-CLIP.

Method	HMDB-51 Top-1 (%)				UCF-101 Top-1 (%)				SSv2 Top-1 (%)
	K = 2	K = 4	K = 8	K = 16	K = 2	K = 4	K = 8	K = 16	K = 2	K = 4	K = 8	K = 16
Baseline	54.3	58.5	62.4	64.2	81.7	86.1	87.0	90.6	6.0	7.1	8.5	11.4
+ Variation Loss	55.2	60.8	64.9	66.5	82.6	87.0	88.9	91.2	6.5	7.7	9.3	12.6
+ ESTA	56.9	63.0	67.8	69.0	83.9	88.2	90.5	92.4	7.2	8.8	9.7	13.1
+ ESTA + Variation Loss	58.7	66.0	69.2	71.4	85.3	90.1	92.8	94.9	7.8	9.6	10.5	13.9

As detailed in Table 6 and Table 7, ESTA and the Variation Loss work synergistically to enhance the performance of VETA-CLIP. While ESTA provides the primary performance gain by explicitly modeling informative temporal dynamics, the Variation Loss further refines the learned representations by enhancing sensitivity to fine-grained motion variations. The complementarity of these two components is particularly evident on the challenging SSv2 benchmark and the unseen Novel splits. Notably, under the base-to-novel protocol, the joint integration of both components boosts the harmonic mean from 58.8% to 67.4% on HMDB-51 and from 10.1% to 15.9% on SSv2. Such results demonstrate that the proposed design not only improves recognition accuracy on seen action classes but also significantly strengthens generalization to unseen categories. Overall, the ablation results confirm that the integration of ESTA and the Variation Loss provides a highly effective strategy for spatio-temporal representation learning.

As shown in Figure 6, the baseline model exhibits an overly broad and diffuse attention distribution, which fails to precisely localize regions of motion change. In contrast, VETA-CLIP pays more concentrated attention to motion-relevant regions, particularly around dynamic objects and their associated actions. This qualitative evidence further substantiates the synergistic capability of the proposed ESTA module and Variation Loss in capturing critical spatiotemporal cues, directly contributing to the improved action recognition performance.

Figure 6. Attention Heatmap. Red and blue regions correspond to high and low attention weights, respectively.

4.4.2. Hyperparameter Ablation

Having established the synergistic effectiveness of the ESTA module and the Variation Loss, we further investigate the internal structural design of the visual encoder path. Specifically, we ablate three critical architectural hyperparameters: (1) the allocation of temporal heads in ESTA and the channel partition ratio ($n_{\mathrm{div}}$) in BRTS; (2) the bottleneck dimension (reduction ratio) of the lightweight adapters; and (3) the weight of the Variation Loss ($\lambda$). This allows us to examine how these hyperparameters jointly influence the balance between model efficiency, adaptability, and overall performance.

Table 8 examines the effect of temporal head (k) allocation under different input frame lengths. For the eight-frame input, the optimal temporal capacity is found at $k={\displaystyle \frac{h}{6}}$ (where total heads $h=12$ for ViT-B/16). Increasing this ratio degrades performance, suggesting that allocating too many temporal heads may weaken the preservation of essential spatial representations. For the 16-frame input, the optimal allocation shifts to $k={\displaystyle \frac{h}{4}}$. This trend suggests that, under longer input sequences, a relatively larger temporal modeling capacity can be beneficial. Overall, the results indicate that the optimal temporal-head allocation varies with the input frame length. Additional analysis of the input frame count is provided in Appendix A (Table A1). While increasing the number of input frames generally improves model accuracy, it also leads to higher training time, greater GPU memory consumption, and longer inference latency. A detailed analysis of these computational trade-offs is provided in Section 4.5.

Table 8. Ablation on ESTA configurations. The impact of temporal head (k) allocation and frame count on our ESTA module is also shown.

Frames	Temporal Heads (k)	HMDB-51 Top-1 (%)			UCF-101 Top-1 (%)			SSv2 Top-1 (%)
		Base	Novel	HM	Base	Novel	HM	Base	Novel	HM
8	1	77.0	58.4	66.4	94.2	77.2	84.9	16.2	13.3	14.6
	2	78.1	59.3	67.4	95.4	78.2	85.9	17.6	14.5	15.9
	3	77.4	58.7	66.8	94.7	77.6	85.3	16.9	14.0	15.3
	4	77.6	58.7	66.8	95.0	77.8	85.5	17.3	14.2	15.6
16	1	78.9	60.0	68.2	96.4	79.0	86.8	18.7	15.4	16.9
	2	79.2	60.1	68.3	96.6	79.2	87.0	19.0	15.6	17.1
	3	79.5	60.6	68.8	97.1	79.8	87.6	19.4	15.9	17.5
	4	79.2	60.0	68.3	96.7	79.1	87.0	18.9	15.6	17.1

As shown in Table 9, the best performance is consistently achieved at $n_{\mathrm{div}}=8$ on both HMDB-51 and UCF-101. This suggests that an excessively small $n_{\mathrm{div}}$ may introduce overly strong temporal perturbation, while an overly large $n_{\mathrm{div}}$ weakens cross-frame interaction. Therefore, $n_{\mathrm{div}}=8$ provides the most favorable balance between temporal modeling and preservation of pre-trained spatial representations.

Table 9. Ablation analysis of the BRTS channel partition ratio $n_{\mathrm{div}}$ under the base-to-novel setting.

${\mathit{n}}_{\mathbf{div}}$	HMDB-51 Top-1 (%)			UCF-101 Top-1 (%)
	Base	Novel	HM	Base	Novel	HM
4	77.4	58.2	66.4	95.0	77.5	85.4
8	78.1	59.3	67.4	95.4	78.2	85.9
16	77.6	58.5	66.7	95.1	77.7	85.5

As illustrated in Table 10, the ratio represents the proportion of the bottleneck dimension relative to the input feature dimension (i.e., Bottleneck Width = Ratio × Input Dimension). When the bottleneck width is too small, the adapter’s representational capacity is insufficient to fully transform and adapt the input features, limiting both base-class fitting and novel-class generalization. Increasing the width to 192 (ratio = 0.25) allows the adapter to effectively learn richer feature transformations, yielding the best harmonic mean. Further enlarging the bottleneck preserves strong performance on base classes but leads to a decline on novel classes, suggesting over-specialization. This ablation confirms that an intermediate bottleneck achieves an optimal balance between representational capacity and generalization performance.

Table 10. Ablation study on bottleneck width under the base-to-novel generalization setting.

Ratio	Bottleneck Width	HMDB-51 Top-1 (%)			UCF-101 Top-1 (%)
		Base	Novel	HM	Base	Novel	HM
0.0625	48	73.9	53.6	62.1	93.9	76.5	84.3
0.125	96	77.9	58.6	66.9	95.2	77.2	85.3
0.25	192	78.1	59.3	67.4	95.4	78.2	85.9
0.5	384	78.2	56.9	65.9	95.4	76.5	84.9
1.0	768	78.1	55.1	64.6	95.2	74.6	83.7

With the optimal physical architecture established, the final performance of VETA-CLIP depends on the balance among the optimization objectives. To this end, we conducted an ablation study on the weight of the Variation Loss ($\lambda$) under the few-shot ($K=8$) setting. We chose the eight-shot setting as it offers a representative trade-off between data scarcity and training stability, enabling a clear assessment of the model’s sensitivity to the Variation Loss weight ($\lambda$).

As shown in Figure 7, the Top-1 accuracy on both UCF-101 and HMDB-51 initially increases with the Variation Loss weight ($\lambda$), reaching a peak at $\lambda =1.0$. Beyond this value, further increasing $\lambda$ results in a decline in performance, indicating that overly large weights cause the model to overemphasize the Variation Loss at the expense of classification accuracy. These results suggest that $\lambda =1.0$ provides an optimal balance between enhancing temporal variation sensitivity and maintaining stable class discrimination. Additional component-wise ablation of the proposed Variation Loss under the $K=8$ few-shot setting is provided in Appendix A (Table A2).

Figure 7. Ablation study on the Variation Loss weight ($\lambda$). The figure illustrates the impact of different $\lambda$ values on the Top-1 accuracy for both HMDB-51 and UCF-101 datasets under the 8-shot setting.

4.5. Efficiency Analysis

To assess the practical deployment viability of VETA-CLIP and validate its lightweight design, we evaluate its computational efficiency on UCF-101 under the Base-to-Novel setting. For a fair comparison, all baseline methods in Table 11 were reproduced and evaluated on the same hardware environment (a single NVIDIA GeForce RTX 5090 GPU) using their official implementations.

Table 11. Computational efficiency comparison of VETA-CLIP and other CLIP-based video adaptation methods under base-to-novel evaluation on UCF-101. Metrics include the batch size, sampled frames, peak GPU memory (GB), inference time per video (ms), and training time per epoch (min).

Method	Batch Size	Frames	Peak Memory (GB)	Inference Time (Ms/Video)	Training Time (Min/Epoch)
ActionCLIP (TNNLS′23) [7]	6	16	20.9	24.8	22.1
XCLIP (ECCV′22) [19]	6	16	15.7	28.7	25.6
ViFi-CLIP (CVPR′23) [8]	6	16	14.9	23.6	21.1
VETA-CLIP	6	16	7.9	5.09	2.3
VETA-CLIP	6	8	4.7	2.47	1.3

As shown in Table 11, VETA-CLIP substantially reduces computational cost compared with ActionCLIP, XCLIP, and ViFi-CLIP. With 16-frame inputs, it requires only 7.9 GB of GPU memory and 5.09 ms per video for inference, while the 8-frame variant further lowers memory usage to 4.7 GB and inference time to 2.47 ms per video. The training time per epoch is similarly reduced. These results highlight the lightweight design of VETA-CLIP and its practical advantages for real-time or resource-constrained deployment. As our default configuration, we adopt eight-frame inputs to balance efficiency and accuracy. Although the current efficiency analysis is conducted on a unified GPU platform for fair comparison, dedicated evaluation on edge-device hardware remains an important direction for future work.

5. Conclusions

In this paper, we introduced VETA-CLIP, a highly parameter-efficient adaptation framework for video action recognition. By freezing the CLIP backbone and inserting lightweight adapters, VETA-CLIP reduces the computational cost of video fine-tuning to only 3.55M trainable parameters. The core of our framework is the synergy between two novel components: the Efficient Spatio-Temporal Attention (ESTA) module, which, via decoupled attention heads and the BRTS mechanism, enables targeted cross-frame interaction while preserving pre-trained spatial representations, and the Variation Loss, which emphasizes inter-frame changes and sequence-level diversity to capture fine-grained motion patterns. Extensive experiments on HMDB-51, UCF-101, SSv2, and Kinetics-400 demonstrate that VETA-CLIP achieves strong or competitive performance under zero-shot, base-to-novel, fully-supervised, and few-shot settings. Ultimately, VETA-CLIP offers a practical parameter-efficient solution for adapting static vision language pre-training to video understanding. Furthermore, comprehensive efficiency analyses demonstrate its practical applicability, significantly reducing peak memory usage and inference latency compared to prior methods. While our ultra-lightweight design encounters a mild capacity bottleneck on massive, highly diverse datasets like Kinetics-400, and exhibits certain adaptation limitations under extreme ultra-low-shot scenarios (e.g., 2-shot), it ultimately demonstrates a favorable efficiency–accuracy trade-off across standard evaluation protocols. Future work will explore slightly stronger adaptation modules to further improve expressive capacity on large-scale and semantically diverse benchmarks.