Hybrid Knowledge Distillation for Edge-Efficient Video Action Recognition: Improving Lightweight 3D CNNs via Joint Distillation

Abstract

One of the remaining challenges in deploying 3D CNN models in resource-constrained environments is the high computational demand. In this paper, we design three lightweight architectures that have distinct spatiotemporal topologies, namely, Lite-R21D, Lite-MC3, and Lite-LF, to reduce computational cost. However, these compact models have restricted representational capacity, which consequently limits their ability to capture complex spatiotemporal features. To overcome this, we employ Knowledge Distillation (KD) and further investigate hybrid combinations of response-based, spatiotemporal attention, and intermediate feature alignment paradigms. By analyzing knowledge transfer across these diverse architectures, our experiments on UCF101 and HMDB51 demonstrate that combining these distillation configurations consistently outperforms single KD methods, resulting in a substantial increase in accuracy across all Student models. Our optimal hybrid setup achieves 92.07% accuracy on UCF101 and 65.56% on HMDB51, compared to the Teacher’s 94.74% and 69.48%, reducing the accuracy gap to only 2.67% and 3.92%. These gains are achieved alongside significant efficiency improvements. The proposed models operate with up to 87% fewer parameters and an 89% reduction in Floating-Point Operations (FLOPs), achieving 6.7× faster inference. Our findings highlight that hybrid distillation is an effective approach for transferring and utilizing complex spatiotemporal knowledge in lightweight models.

1. Introduction

Computer vision has progressed rapidly in the last decade, mainly due to two factors: the evolution of high-performance computing hardware and the advancement of deep learning algorithms [1]. Video analysis is one of the sub-fields most influenced by this evolution. Unlike static image classification, which relies entirely on the spatial features of height and width, video analysis requires the modeling of complex spatiotemporal patterns which add time as a third dimension [2,3]. Human Action Recognition (HAR) is an important field in video understanding which has recently gained attention from researchers due to its use in different smart-city applications. For instance, its implementations include sports analysis [4,5], public safety applications [6,7], and healthcare monitoring [8], in addition to other industrial utilities, such as advanced digital twin systems [9,10], human–robot interaction [11,12], and context-aware digital assistance [13].

After the notable success of 2D Convolutional Neural Networks (CNNs) in image classification, researchers started to explore their use for video-based tasks. However, these 2D models are incapable of processing complex spatiotemporal data. Since then, the focus has shifted toward a different direction, with new designs needing to handle both motion dynamics and spatial data. This development was supported by the availability of video datasets, such as Kinetics-400 [14], UCF101 [15], and HMDB51 [16]. Many approaches were introduced to model the spatiotemporal features in videos, including two-stream architectures [17], Recurrent Neural Networks (RNNs) [18], 3D CNNs [19], and, more recently, video Transformers [20]. Among these techniques, 3D CNNs are the most utilized architectures in HAR due to their simple design, which allows the extraction of spatiotemporal features within a single neural block [21] rather than relying on multiple streams or combined architectures. Nevertheless, despite their high-performance capabilities, they remain computationally expensive. As a result, their deployment in real-time applications remains challenging on constrained devices [22,23].

One approach to reduce the complexity associated with large 3D CNNs is to compress them by reducing the depth and width of their neural blocks or by applying network pruning. Although such compression techniques can lead to a reduction in model size and, consequently, the inference time, these operations often degrade the original accuracy of the full-capacity architecture [24,25,26]. To cope with this issue, Knowledge Distillation (KD) has emerged as a training paradigm that helps recover lost performance [27,28,29]. The concept was first introduced for 2D CNN-based image classification [30], where a compact “Student” model mimics a larger “Teacher” framework. Moreover, different KD techniques were developed based on how the knowledge is transferred. For instance, response-based distillation aligns the final output probabilities (dark knowledge) between the Teacher and the Student [31]. Feature-based methods, on the other hand, such as FitNet [32], work by matching the intermediate feature representations of the Teacher, while attention transfer (AT) [33] guides the Student to focus on the same regions as the Teacher. Transferring these KD methods to the video domain introduces challenges that are related to the increased complexity of spatiotemporal information and the noise in temporal data during training [34].

Due to these temporal complexities, applying KD methods in vision-based HAR has remained limited. In early attempts to distill knowledge in video classifiers, response-based KD, i.e., Hinton KD, was primarily used, where knowledge was transferred from an RGB Teacher or another modality, such as optical flow or depth, to a more compact Student [35,36]. Since then, most HAR research has focused either on multi-stream setups or on evaluating single KD paradigms independently. As a result, the literature still lacks a clear understanding of the behavior of different KD mechanisms when they interact within lightweight 3D CNN Students. For example, although some distillation techniques, such as FitNet and AT, have shown effectiveness in image classification, these distillers remain largely underexplored for the RGB video domain. The main reasons behind this are the challenges in handling temporal noise and the joint modeling of where and when the Teacher provides guidance when applying these distillation methods to the video domain [37]. As a result, the behavior of these distillation mechanisms in capturing different aspects of spatiotemporal knowledge remains not well understood.

Moreover, the potential combination of these KD paradigms has not yet been systematically studied in video classification. For example, feature-level transfer can help the Student model recover rich intermediate spatiotemporal representations, while the alignment of logit distributions adds regularization to the final recognition. Additionally, attention transfer focuses on guiding the Student toward the most informative spatiotemporal regions of the Teacher. For this reason, understanding the impact of combining these KD methods to enhance each other or create redundancy remains an open question in vision-based HAR. Addressing this gap is important in cases where KD paradigms are used to enhance the performance of lightweight models deployed on edge devices. In particular, analyzing how different KD types interact during knowledge transfer allows a better understanding of their complementary or redundant effects in hybrid configurations.

In this paper, we address this open question by focusing on hybrid KD for lightweight 3D CNNs in RGB-based HAR. Using three compact Students built on residual 3D CNN blocks resembling those from R (2+1) D and MCx [38], with different topologies for spatiotemporal modeling, we evaluate not only the individual KD methods but also their four possible hybrid combinations. Our goal is to determine the extent to which hybrid distillation techniques can enhance the recovery of complex spatiotemporal knowledge from a high-capacity Teacher and thereby improve the practical deployment of efficient HAR models in edge and smart-environment scenarios.

Our key contributions are summarized as follows:

• Systematic study of hybrid KD in vision-based HAR: We provide a comprehensive investigation of hybrid combinations of KD methods in video classification by combining three distinct paradigms based on logits, attention, and intermediate feature transfer (Hinton, AT, and FitNet). Unlike prior works, which evaluated single KD paradigms in isolation or focused on cross-modal distillation, we systematically analyze standalone, dual, and triple hybrid KD setups on three lightweight 3D CNN Students using two benchmark datasets, UCF101 and HMDB51. To the best of our knowledge, this is the first systematic study to analyze hybrid KD combinations within lightweight 3D CNNs for RGB-based video action recognition.
• Efficient deployment of lightweight models: We demonstrate that our lightweight designs achieve significant reductions in computational cost. These models operate with up to 87% fewer parameters and 89% fewer FLOPs, resulting in up to 6.7 times faster inference compared to the full-capacity Teacher. Additionally, the proposed hybrid KD training allows the compact models to successfully achieve up to a 17.6% improvement in accuracy over scratch-trained baselines. Furthermore, our optimal configuration reaches 92.07% accuracy, narrowing the performance gap to just 2.67% behind the heavy Teacher (94.74%) on UCF101. These results directly address the challenge of deploying high-accuracy models on resource-constrained devices.
• Analysis of action-level knowledge transfer: We select highly challenging spatiotemporal action classes and analyze the impact of different KD methods and their hybrid configurations on these specific actions. By applying this evaluation to our lightweight Student variants, we examine how structural differences influence the absorption of distilled knowledge and therefore affect the learning process. This class-wise analysis reveals that optimal hybrid configurations are architecture-dependent (e.g., AT-FitNet for Lite-R21D and Lite-LF, Hinton-FitNet for Lite-MC3), providing practical guidelines for model selection in real-world deployment.

2. Literature Review

This section provides an overview of HAR advancement detailing the evolution of deep learning architectures designed for spatiotemporal modeling. Moreover, it presents an analysis of KD employment within the HAR domain to identify the specific gaps in the literature that motivate our research.

2.1. Spatiotemporal Modeling in Human Action Recognition

Capturing spatiotemporal features is a key factor in video classification tasks. HAR models have evolved over the years to provide solutions that effectively model this type of information. For example, early research considered the construction of two-stream setups in which the backbones are based on image 2D CNN models. In this dual design, the spatial appearance from RGB frames and temporal motion from optical flow are separately processed in parallel pathways before being fused [17,39]. This configuration was later enhanced by using, for instance, sparse temporal sampling, such as Temporal Segment Networks (TSN) [40]. Other researchers explored employing additional streams to build a three-stream 2D CNN framework, which benefits from incorporating a static image obtained by applying rank pooling to sequential frames [41], while another study adopted saliency maps as a third pathway to help suppress background noise [42] and thus improve the recognition capacity.

Alternatively, the HAR community has considered RNNs for handling the sequence of motion dynamics. To better handle the vanishing gradient issue, gated variants of RNNs such as Long Short-Term Memory (LSTM) networks were employed [43]. This architecture, which combines a standard 2D CNN for spatial content and LSTM for temporal data, was adopted in other works that implemented different types of attention, such as integrated spatial [44], temporal [45], or hybrid spatiotemporal attention mechanisms [46]. Moreover, other researchers utilized the 2D CNN-LSTM design to establish two streams operating on the RGB and optical flow paths separately, as in [21]. While all these architectures showed improvements in performance, they are associated with heavy computational demands. Consequently, studies such as [47] benefited from the Gated Recurrent Unit (GRU) as a lighter solution that still achieves similar results to LSTM adoption.

To overcome the complexities of two-stream and RNN-based models, where spatial and temporal data are handled in different networks, a third paradigm emerged, in which the spatiotemporal features are extracted in a single block. The idea was to develop 3D filters that handle the temporal data together with spatial features. Early designs such as C3D [19,48] and the deeper Res3D [49] frameworks demonstrated the effectiveness of these 3D filters, while other concepts like the I3D model [50], which relies on inflating an existing 2D image classifier into a 3D architecture, proved effective in improving performance. Furthermore, some researchers focused on increasing the efficiency of the 3D models, for instance, the R (2+1) D and MCx architectures [38], which showed how factorizing 3D filters into sequential and separate 2D spatial and 1D temporal operations, or mixing them with spatial-only kernels, can improve both optimization and accuracy. Building upon this, the S3D-G framework [51] achieved state-of-the-art efficiency by restricting these factorized CNN blocks to deeper layers and integrating a channel-gating mechanism. Moreover, the authors of the SlowFast network [52] introduced a dual-pathway design to process high and low frame rates separately. Meanwhile, the X3D framework [53] achieves high efficiency by adopting a lightweight 2D network and then gradually expanding its spatial and temporal dimensions, without the need for large 3D models. Finally, to make these architectures suitable for deployment on resource-constrained devices, several designs were proposed, such as MoViNets [54], which implement causal stream buffers to reduce memory consumption, while the CoVIAR model [55] operates directly on compressed video streams. Additionally, the Temporal Shift Module (TSM) [56] achieved 3D-level modeling on 2D backbones with minimal added computational overhead.

Apart from the previous paradigms, Vision Transformers were adopted in HAR to model long-range dependencies using self-attention mechanisms [20,57]. However, because these frameworks require a massive number of computational operations, researchers developed more efficient variants. To address this, models such as TimeSformer [58] employ a mechanism that divides attention to handle both the spatial relationships and the temporal data in two separate steps. Moreover, the Multiscale Vision Transformer (MViT) [59] achieves state-of-the-art performance by introducing an architecture that mimics a CNN pyramid, utilizing multi-head pooling to capture fine-grained details in actions. The Shifted Chunk Transformer (SCT) [60] is another example of how efficiency is employed in Transformer-based designs. It operates by limiting the attention mechanism to local shifting windows, and, to avoid redundancy in computations, it groups similar features together.

The HAR community has also investigated how combining different architectures can help achieve a balance between accuracy and efficiency. The UniFormer [61] model, for instance, unifies local CNNs in the early layers with global self-attention in the deeper ones within the architecture. Differently, the CVTN [62] decouples these tasks by using a CNN block for extracting spatial data followed by a Transformer that models the temporal dependencies. The two other main examples of efficiency-driven models are ViT-ReT [63] and MEACI-Net [64]: ViT-ReT utilizes recurrent attention mechanisms, and MEACI-Net incorporates cross-modal interactions. In addition, to capture complex human behaviors, recent studies have adopted multi-stage fusion mechanisms that combine different neural network architectures at various depths of the model. For example, authors in [65] integrated ResNet backbones, LSTM, and global Transformers into a single framework. Meanwhile, the work in [66] utilized self-attention blocks within a residual bottleneck framework that contains multiscale filters to reduce and control the visual noise. To explore a different multiscale approach, the authors in [67] employed an Inception-v3 backbone [68] to extract spatial features which are then refined by a feed-forward attention mechanism and processed through a GRU block for final action classification. Additionally, another concept for decoupling spatiotemporal modeling was studied in [69] which implements 3D Histogram Transformer Blocks (HTB) within CNN frameworks to convert spatiotemporal operations into highly efficient attention weights.

Furthermore, recent advancements in the HAR field have increasingly shifted toward minimizing the computational cost of vision-based HAR models for real-time and edge-oriented deployment. For example, MNv3-MFAE [70] introduces a lightweight framework based on the MobileNetV3 architecture, in which multi-feature action extraction is utilized to reduce model complexity while maintaining strong performance. Similarly, CA3D [71] targets efficient video classification on edge devices by combining 3D CNNs with attention mechanisms and quantization-aware training. Other researchers have relied on Transformer architectures, such as the ViT-HAR [72] framework, which addresses action recognition in cluttered and occluded environments using contextual patch re-weighting and attention-guided occlusion masking. These studies reflect the growing interest in efficient HAR architectures for real-time deployment.

Despite the existence of lightweight video classification architectures, large-scale models continue to dominate in terms of overall accuracy. However, their high computational cost remains the most significant obstacle to real-world edge deployment. Rather than exclusively building new lightweight frameworks from scratch, an alternative paradigm has emerged. This concept relies on advanced training techniques to transfer the rich feature representations of computationally heavy models into more compact architectures. This is achieved through distillation of knowledge. When successfully tuned, KD enables a compact model to approximate the performance of a heavier framework while remaining highly efficient. In this paper, we utilize the R (2+1) D model as the reference and source of the distillation model.

2.2. Knowledge Distillation Techniques in Human Action Recognition

While the main purpose of KD in image classification is to transfer learned spatial representations between models, the transition into the video domain is significantly more complicated. First, in video classification, the features are a complex combination of spatial and temporal data. Second, HAR utilizes different modalities, such as RGB, skeleton, depth, and hybrid combinations of these. This implies that the data targeted for transfer vary across setups. Early video KD methods focused on cross-modal frameworks, where the primary goal was to transfer temporal dynamics from an optical flow Teacher to an RGB-only Student at inference time [73]. A notable example of this is the D3D framework [74], which distills motion knowledge into spatial networks to avoid heavy optical flow computations. Alternatively, researchers explored multi-Teacher frameworks to fuse weighted soft targets from diverse expert models [75,76]. An example of this is the compressed CoViAR framework, where a single Student model receives knowledge from multiple distinct Teachers in different spatiotemporal streams.

As 3D CNN architectures continued to advance, researchers shifted toward distilling these unified spatiotemporal models. Response-based strategies were the most dominant in this context, for example, when applied to compressed R (2+1) D and C3D models to enhance their performance [34], or when used in cases of limited data, as investigated in [36]. However, other studies explored different mechanisms by employing generative feature-based KD that utilizes conditional variational autoencoders, aiming to preserve the important temporal information when applied to lighter versions of the I3D model [37]. In addition, other works have introduced different approaches that eliminate the external Teacher, such as bidirectional feature distillation, which separates features based on their dominance [77], and self-distillation paradigms, where the network learns from its own previous training states [78].

Although KD has been increasingly adopted in the HAR field, most of the current literature is generally dominated by feature-based or self-distillation frameworks. Consequently, there remains a notable gap in adapting different successful paradigms from the image classification domain. Specifically, neither FitNet nor AT KD techniques have been comprehensively benchmarked against the standard response-based Hinton approach within a strictly controlled Teacher–Student framework on a lightweight 3D CNN design. Building on this, our work offers a detailed evaluation of these distinct distillation mechanisms to identify the most effective training strategy when applied separately or as joint hybrid combinations to three lightweight 3D CNN frameworks. These models share conceptual similarities in architecture with the R (2+1) D and MCx frameworks but vary in how they are customized internally. Furthermore, we extend this assessment to study the behavior of each KD setup and how it influences learning at the action class level.

3. Methodology

Our main goal in this work is to investigate the effectiveness of certain KD types, namely, FitNet, AT, and Hinton KD, when applied separately or in joint combinations for training lightweight 3D CNN models that differ in their layers’ spatiotemporal topologies. The following subsections detail the Teacher model, the Student architectures, and the distillation strategies covered in this study.

3.1. The Teacher Model

We adopted the 18-layer R (2+1) D architecture as the full-capacity expert Teacher. This framework factorizes the standard 3D CNN kernel (d × h × w) into sequential 2D spatial (1 × h × w) and 1D temporal (d × 1 × 1) operations, which leads to better optimization and computational efficiency.

We utilized this Teacher model in two distinct configurations. First, as a baseline trained from scratch on UCF101, and, second, as an expert source for distillation that gains enhanced spatiotemporal modeling strength from being pre-trained on Kinetics-400 and then fine-tuned on UCF101. Moreover, by comparing these two versions, we were able to identify the specific action classes where the pre-trained expert outperforms the scratch baseline. These challenging classes were then used as an analytical tool to evaluate the extent to which our KD strategies can transfer such expert-level knowledge to the lightweight Students that have never been trained on large-scale data.

3.2. The Lightweight Student Models

Our proposed lightweight Students were developed based on residual 3D CNN blocks similar to those used in R (2+1) D and MCx frameworks [39]. We applied a uniform 50% channel reduction across all network blocks by scaling the capacity from 512 in the original framework down to 256 channels on the Student variants. The 50% channel-width setting was selected as the main configuration because it provides a balanced trade-off between reduced computational cost and sufficient representational capacity for evaluating the proposed single and hybrid KD strategies. To further analyze the effect of stronger compression, additional third-scale Student variants and a fourth-scale Lite-R21D variant were evaluated in a width-scaling ablation study (Section 5.7). In contrast to the Teacher’s factorized stem, we replaced the first layer of each Student with a standard 3D CNN kernel (3 × 7 × 7) across all three models. As Figure 1 shows, the Teacher and Students differ in their internal configurations; however, they follow a shared structural hierarchy consisting of an initial stem followed by four main residual stages (Layer 1–Layer 4) before the final classifier unit.

Although all three Student variants have the same stem layer configuration, they vary in their subsequent residual stages to exhibit different spatiotemporal behaviors:

• Lite-R21D: This model copies the Teacher’s decoupled structure, by applying factorized blocks (2D spatial followed by 1D temporal) across all four residual stages (Layers 1–4 following the stem).
• Lite-MC3: This hybrid approach adopts standard joint 3D CNNs (3 × 3 × 3) in the first two stages (Layers 1 and 2), and spatial-only 2D (1 × 3 × 3) in the deeper third and fourth stages.
• Lite-LF: This variant relies on spatial 2D CNNs (1 × 3 × 3) for the first three stages (Layers 1–3) and then switches to factorize the final spatiotemporal stage.

A clear comparison of these distinct structural configurations is shown in Figure 1, which visualizes the complete architecture of the full-capacity Teacher alongside the three proposed lightweight Student variants.

Although the proposed lightweight Student architectures substantially reduce parameters and computational cost, channel reduction and simplified spatiotemporal topology can still reduce their representational capacity compared with the full-capacity Teacher. Therefore, in this work, KD was introduced as a complementary training strategy to recover part of the accuracy gap caused by compression while preserving the lightweight inference structure of the Students. For this purpose, we established a distillation framework based on response-based, attention-based, and feature-based KD mechanisms, and further evaluated their dual and unified hybrid combinations to analyze how different supervision signals can affect the performance of each lightweight architecture.

3.3. The Knowledge Distillation Paradigms

As mentioned previously, we defined three distinct KD mechanisms and their combinations used for transferring expertise from the expert Teacher to our proposed lightweight Students. Our implemented KD paradigms are response-based (operating on final classifier probabilities), feature-based (transferring intermediate spatiotemporal representations), and attention-based (guiding the Students to follow the Teacher’s spatiotemporal focus regions). Furthermore, we optimized their hybrid combinations simultaneously, which differentiates our work from the single or multi-staged KD pipelines commonly found in the literature. The formulations of the single and hybrid paradigms are defined in the following paragraphs.

3.3.1. Response-Based Distillation

The first distillation method adopted in our work is response-based Hinton KD. This approach uses the Teacher’s final probabilities, known as “soft targets”, to guide the Student during the distillation process. Rather than relying completely on hard labels, this method allows the Student to learn the relative similarities between different actions directly from the Teacher’s predictions. Furthermore, because this distillation operates at the level of final class probabilities, its core mechanism is independent of the input modality. In other words, Hinton KD does not require any further structural adjustments when applied to the 3D video domain.

The total loss function (${\mathcal{L}}_{Hinton}$) is defined as the linear combination of the standard cross-entropy loss with the distillation loss, as presented in Equation (1):

$${\mathcal{L}}_{Hinton}=(1-\alpha ){\mathcal{L}}_{CE}+\alpha T^2{\mathcal{L}}_{KL}$$

(1)

where

• ${\mathcal{L}}_{CE}$: the standard cross-entropy loss calculated using the hard labels;
• ${\mathcal{L}}_{KL}$: the Kullback–Leibler divergence between the softened output probabilities of the Student and Teacher;
• T: the temperature parameter that is applied to the softmax function, which controls the smoothness of the probability distributions;
• $\alpha$: a constant coefficient that balances the ground-truth supervision and distillation.

3.3.2. Attention-Based Distillation

The original AT KD algorithm was designed for 2D image classification, which utilizes 2D adaptive pooling to align the spatial dimensions of height and width between the Teacher and Student. However, applying this directly to the 3D video domain creates a structural mismatch due to the added temporal dimension. We handled this by replacing the standard 2D alignment from the image classification setup with 3D adaptive average pooling. This technique allows the synchronization of the spatiotemporal data of the Student’s feature maps with those of the Teacher. This ensures full 3D alignment before computing the attention maps and calculating the Mean Squared Error (MSE). The total loss in this configuration is defined as in Equation (2):

$${\mathcal{L}}_{AT}={\mathcal{L}}_{CE}+{\displaystyle \frac{\beta }{2}}\sum _{j\in I}{∥{\displaystyle \frac{G\left(A_S^j\right)}{\parallel G\left(A_S^j\right){\parallel }_2}}-{\displaystyle \frac{G\left(A_T^j\right)}{\parallel G\left(A_T^j\right){\parallel }_2}}∥}_2^2$$

(2)

where

• ${\mathcal{L}}_{CE}$: the cross-entropy loss of the ground-truth labels;
• $\beta$: the constant coefficient for the attention weight;
• $j\in I$: the layers selected to be paired for distillation;
• G: the function for mapping spatiotemporal attention (power parameter $p=2$);
• The summation term: the MSE which is computed between the normalized spatiotemporal attention maps of the Teacher and Student.

3.3.3. Feature-Based Distillation

The final single distillation strategy we employed is FitNet, which utilizes intermediate feature regression to align the hidden activations of the Student with those of the Teacher. By explicitly transferring the representational geometry from the Teacher’s intermediate layers, this method provokes the Student to develop a feature hierarchy that mimics the Teacher’s internal knowledge structure. This supervision of the hidden layers acts as a regularizing mechanism that helps the lightweight Student extract more meaningful features.

Similar to the adaptation required for AT, the standard FitNet approach was also developed for 2D image data. This implies that applying it to 3D video architectures requires a transition from spatial to spatiotemporal alignment. Because we scaled the channel widths by half in our three Student models, which naturally leads to a channel mismatch between the Teacher and the Student, we introduced a learnable regressor r(.). While the original image-based implementations utilize a 2D CNN, we employed, in contrast, a (1 × 1 × 1) 3D CNN, to project the Student’s channel dimensions to match the Teacher’s capacity. This 3D kernel is essential for handling the temporal dimension of the feature maps during the projection process of selected layers. Once the channels are aligned, the video FitNet loss is calculated as in Equation (3):

$${\mathcal{L}}_{FitNet}={\mathcal{L}}_{CE}+{\lambda }_{Hint}{∥u_T\left(x\right)-I\left(r\left(u_S\left(x\right)\right)\right)∥}_2^2$$

(3)

where

• ${\mathcal{L}}_{CE}$: the standard cross-entropy loss based on the ground-truth labels;
• $u_T$ and $u_S\left(x\right)$: feature maps of the Teacher and Student, respectively;
• $r(·)$: a learnable 3D CNN regressor for Student channel projection that operates to match the Teacher;
• $I(·)$: a dynamic trilinear interpolation function that synchronizes the spatiotemporal dimensions between the models;
• ${\lambda }_{Hint}$: the weighting coefficient that balances the feature alignment loss.

3.3.4. Simultaneous Distillation

In the previous sections, we defined our three distinct distillation mechanisms. To comprehensively analyze their combined impact rather than evaluating them separately, we introduced an integration of these KD paradigms into joint loss formulations. Our configuration differs from the common practices in the existing literature, which mostly employ multi-stage training pipelines where different KD methods operate sequentially. Instead, we combined these losses simultaneously. By jointly optimizing these multiple objectives during a continuous training phase, we formulated four distinct hybrid combinations to evaluate their effects.

• Response and Attention Hybrid

This configuration guides the Student by simultaneously combining the Teacher’s final response logic via Hinton KD with spatiotemporal focus achieved by AT KD. Equation (4) demonstrates this joint loss formulation:

$${\mathcal{L}}_{H+AT}={\mathcal{L}}_{Hinton}+({\mathcal{L}}_{AT}-{\mathcal{L}}_{CE})$$

(4)

2.

Response and Feature Hybrid

In this setup, the Student is distilled by pairing Hinton and FitNet KDs at the same time.

$${\mathcal{L}}_{H+FitNet}={\mathcal{L}}_{Hinton}+({\mathcal{L}}_{FitNet}-{\mathcal{L}}_{CE})$$

(5)

3.

Attention and Feature Hybrid

This is a purely internal configuration in which the Student is forced to learn strictly from the Teacher’s internal feature geometry via FitNet KD and the spatiotemporal focus that the AT KD offers. This loss formulation is presented in Equation (6):

$${\mathcal{L}}_{AT+FitNet}=({\mathcal{L}}_{AT}-{\mathcal{L}}_{CE})+({\mathcal{L}}_{FitNet}-{\mathcal{L}}_{CE})$$

(6)

4.

Unified Hybrid Distillation

All three single KD types are set to operate together simultaneously, as presented in Equation (7):

$${\mathcal{L}}_{H+AT+FitNet}={\mathcal{L}}_{Hinton}+({\mathcal{L}}_{AT}-{\mathcal{L}}_{CE})+({\mathcal{L}}_{FitNet}-{\mathcal{L}}_{CE})$$

(7)

4. Experimental Setup

In this section, we provide details about the benchmark datasets used in our experiments, the data preprocessing, and the hyperparameters adopted for these methods.

4.1. Datasets and Evaluation Protocols

In this study, we relied on the UCF101 dataset as the main benchmark for training the Student and Teacher models from scratch, as well as for evaluating both the single and hybrid strategies. HMDB51, on the other hand, was used to assess the cross-dataset transferability of the distilled designs. For both datasets, we adopted the official split 1.

The UCF101 dataset contains 13,320 short video clips covering a total of 101 human action classes. We selected this dataset as the primary training benchmark because it provides a balance between actions diversity and computational feasibility. Training three Student architectures under seven distillation configurations (3 single + 4 hybrid KDs) on a large-scale dataset such as Kinetics-400 would create a substantial computational burden. At the same time, UCF101 offers a suitable setting for analyzing how effectively the proposed KD strategies transfer the richer spatiotemporal knowledge acquired by the Teacher during its pre-training on Kinetics-400. This setting also enabled us to identify the five action classes that benefit most from the Teacher’s prior large-scale pre-training, and to use them as a diagnostic metric for analyzing how effectively the distilled Students inherit this spatiotemporal knowledge.

HMDB51, in contrast, is a more challenging dataset containing 6766 videos across 51 action classes. We used this dataset exclusively in the transfer stage to evaluate whether the Students can successfully apply the distilled representations to a different and more demanding set of video actions.

4.2. Data Preprocessing and Augmentation

In this study, we implemented a unified spatiotemporal preprocessing pipeline for all training experiments to enhance model generalization. For spatial augmentation, the frames were first resized to 256 pixels on the shorter edge, a process which was followed by a random crop of 224 × 224. Additional spatial transformations included color jittering and random horizontal flipping with a probability of 50%. Temporally, we sampled a single clip of 16 consecutive frames starting from a random index within each video.

During evaluation, the spatial preprocessing consisted of a 224 × 224 center crop obtained from the 256-pixel resized frames. For the temporal transformation, 10 distinct 16-frame clips were uniformly sampled across the decoded video. The final class prediction was then derived by calculating the mean of softmax probabilities across these 10 temporal views, which provides a stable estimate of the overall measured accuracy.

4.3. Training and Optimization

All our models, along with the training and evaluation functions, were implemented in the PyTorch v2.11 framework. In our training configuration, we used the Stochastic Gradient Descent (SGD) with a momentum of 0.9 and a weight decay of $\mathbf{5}\times {\mathbf{10}}^{-\mathbf{4}}$. Moreover, we set the batch sizes to 16 for training and 4 for evaluation. We also incorporated the Cosine Annealing learning rate scheduler, which decays the learning rate down to a minimum value of $\mathbf{1}\times {\mathbf{10}}^{-\mathbf{6}}$. The numbers of epochs used in our experiments can be described as follows:

• Training from Scratch and KD on UCF101: We set the learning rate initially to 0.01, and trained for 200 epochs to ensure complete convergence.
• Fine-Tuning on HMDB51: We defined a learning rate of 0.001 and a total number of 80 epochs when using this dataset for fine-tuning the pre-trained distilled models or when running secondary distillation.

4.4. Distillation Hyperparameters

In all the experimental setups, we maintained consistency by keeping the KD hyperparameters fixed across all Student models for both the single and hybrid distillation training settings. We selected these values based on the ablation study detailed in Section 5.7. Moreover, the parameter values used in the hybrid distillation configurations were directly inherited from the optimal values established for the related single KD methods. For the feature-based methods (AT and FitNet), we paired the Teacher and Student at the final residual block (Layer 4) to ensure the transfer of high semantic features.

Table 1. KD types, paired layers/logits, and corresponding hyperparameters applied in the distillation experiments.

KD Type	Paired Layers/Logits	Hyperparameters
Hinton	Logits	$T=8,\alpha =0.9$
AT	Layer 4	$\beta =2000$
FitNet	Layer 4	${\lambda }_{Hint}=5$

details these optimal hyperparameters for the individual KD mechanisms.

5. Result and Discussion

5.1. Performance Analysis on UCF101

We started our experiments by establishing the performance baselines for our three lightweight models and the full-capacity R (2+1) D architecture. The R (2+1) D framework was utilized in two distinct training setups: First, we trained it from scratch on UCF101 to serve as a reference for the lightweight models trained on the same dataset. Second, we took the same R (2+1) D architecture, which was pre-trained on Kinetics-400, and fine-tuned it on UCF101. Because this fine-tuned version has better performance, we considered it the expert Teacher required for the KD training.

Table 2. Baseline performance of R (2+1) D fine-tuned Teacher-FT, scratch (Sch), and Student models on UCF101. Number of parameters in millions is presented as Params (M) and computational operations in giga FLOPs (GFLOPS).

Model	Top-1 Acc. (%)	Params (M)	GFLOPs
Teacher-FT	94.74	31.51	162.58
Teacher-Sch	82.95	31.51	162.58
Lite-R21D	78.72	8.33	43.81
Lite-MC3	70.31	4.2	40.58
Lite-LF	74.41	6.97	17.9

summarizes the performance, model parameters, and FLOPs across all models. This is also visualized in Figure 2, which plots Top-1 accuracy against FLOPs, with the bubble size representing the parameter count.

As shown in Table 2, the pre-trained expert Teacher achieves a top accuracy of 94.74%, which is a performance difference of 11.79% compared to the UCF101 scratch-trained reference, which records only 82.95%. This performance gap between the two versions highlights how the expert Teacher learns more complex spatiotemporal features when pre-trained on a much larger dataset, whereas the UCF101 by itself has limited scale and diversity.

When evaluating the lightweight models, the results showed that the Lite-R21D model is the most accurate variant among the three proposed models, with an accuracy of 78.72%, which is only 4.23% behind the scratch-trained Teacher. As this model is the closest in architecture to the Teacher, it utilizes fully spatiotemporal layers the most among all our lightweight models, despite operating with 3.71 times fewer FLOPs and 3.78 times fewer parameters than the full-capacity Teacher.

Table 2 shows that Lite-MC3 is the most memory-efficient design, requiring only 4.20 million parameters, which translates to 7.5 times fewer parameters than the Teacher. Furthermore, since fewer FLOPs directly correlate to faster inference times, the Lite-LF variant is the winning model in terms of computational speed, as it has only 17.90 GFLOPs (approximately 9.1 times fewer than the Teacher).

Moreover, when comparing these two ultra-lightweight architectures, we concluded that the placement of 3D spatiotemporal factorization in the deeper layer (as in Lite-LF, which uses a unified 3D stem and a factorized Layer 4) yields a noticeably higher accuracy than the early architectural dependency on unified spatiotemporal layers (as seen in Lite-MC3, which unifies the stem and first two layers). Ultimately, these results demonstrate the importance of these proposed architectures. Despite their lightweight nature, they preserve strong representational capabilities, making them a very good choice to deploy in resource-constrained environments. However, when considering them for deployment, there is an obvious need to boost their accuracy. This motivated our evaluation of KD strategies, which will be defined and analyzed in Section 5.2.

5.2. Comparative Analysis of Distillation Mechanisms

After we established the baseline architectural configurations (Section 5.1), we turned our attention to evaluating the efficacy of the proposed distillation pipelines. For this, we aimed to quantify the impact of this knowledge transfer by relying on the Top-1 accuracy and introducing a normalized metric termed the Recovery Ratio (R). This calculates the exact proportion of the performance gap, specifically the gap between the scratch-trained Student and the fine-tuned expert Teacher, that is successfully bridged by the distilled model. It is defined as in Equation (8):

$$R={\displaystyle \frac{Acc\left(S_{KD}\right)-Acc\left(S_{scratch}\right)}{Acc\left(T_{FT}\right)-Acc\left(S_{scratch}\right)}}$$

(8)

where $Acc\left(S_{KD}\right)$ denotes the accuracy of the Student that is trained by KD, $Acc\left(S_{scratch}\right)$ represents the baseline accuracy of the scratch-trained Student, and $Acc\left(T_{FT}\right)$ is the accuracy of fine-tuned Teacher.

Table 3. Top-1 accuracy (%) of the proposed lightweight 3D CNN models trained from scratch (Sch) and under various single and hybrid KD configurations on UCF101.

Model	Sch	Hinton	AT	FitNet	Hinton-FitNet	Hinton-AT	AT-FitNet	Unified Hybrid
Lite-R21D	78.72	85.36	86.81	90.83	91.25	89.88	92.07	91.49
R	—	0.41	0.5	0.76	0.78	0.7	0.83	0.8
Lite-MC3	70.31	80.97	79.78	83.21	87.91	84.69	85.65	86.49
R	—	0.44	0.39	0.53	0.72	0.59	0.63	0.66
Lite-LF	74.41	82.84	83.19	87.92	89.45	87.18	89.96	89.48
R	—	0.41	0.43	0.66	0.74	0.63	0.76	0.74

comprehensively details the resulting accuracies and Recovery Ratios across all tested configurations.

To facilitate visual comparison of the performance improvements, Figure 3 illustrates the accuracy gains achieved by each KD strategy across all three distinct Student models. Furthermore, to highlight the compatibility between specific Student architectures and the applied KD paradigms, we visualize the recovery values (R) in Figure 4, which provides a performance matrix heatmap. This figure demonstrates that, when evaluating the single-method distillation paradigms (Hinton KD, AT, and FitNet), the R values clearly demonstrate that intermediate feature distillation (AT and FitNet) outperforms the classic logit-based distillation (Hinton). Although the Unified Hybrid (Hinton-AT-FitNet) successfully recovers a large portion of the Teacher’s knowledge, it fails to outperform the best dual-hybrid methods.

As can be observed in Table 3 and also from Figure 4’s heatmap, the FitNet KD serves as the most powerful single method across all architectures. For instance, the FitNet training improves the Lite-R21D model from a scratch baseline of 78.72% to a remarkable 90.83%, which indicates a successful recovery of 76% (R = 0.76) of the performance gap with the Teacher. Similarly, the same single KD boosts the Lite-LF variant to 87.92% (R = 0.66) and the Lite-MC3 to 83.21% (R = 0.53). In contrast, the Hinton KD method yields the lowest gains, recovering only 41% to 44% of the gap across the three models.

From Table 3 and Figure 4, we conclude that, for complex spatiotemporal tasks, aligning the Teacher’s intermediate features is more effective than matching its final outputs in our designs. The advantage of feature-level transfer is further amplified when integrating the hybrid KD strategies. In these settings, the Students are pushed closer to the Teacher’s performance. For the Lite-R21D and Lite-LF architectures, the AT-FitNet hybrid combination emerges as the best configuration to bring the highest accuracy. For example, by simultaneously transferring spatial attention maps in AT KD and the intermediate feature representations that FitNet offers, the Lite-R21D reaches a peak accuracy of 92.07% (a recovery value of 83%, R = 0.83), while Lite-LF achieves 89.96%. Differently, for the Lite-MC3 model, the Hinton-FitNet combination proves to be the most effective, scoring an accuracy of 87.91% and closing 72% of the performance gap with the Teacher. Moreover, although the Unified Hybrid (Hinton-AT-FitNet) consistently outperforms both the scratch baselines and all single KD approaches across all models, it fails to outperform the best dual-hybrid configurations, such as AT-FitNet for the Lite-R21D and Lite-LF models and Hinton-FitNet for Lite-MC3. This suggests some level of saturation, where forcing the Student to simultaneously optimize across logits, attention, and feature maps leads to optimization conflicts. Ultimately, these results confirm that intelligently pairing structural distillation mechanisms allows highly efficient, lightweight models to achieve the expert-level recognition power of the Teacher.

Although all models were trained under the same experimental protocol, including the UCF101 split 1, 200 training epochs, and 10-clip probability averaging during inference, we further conducted McNemar’s test [79] to verify that the observed improvement of hybrid KD over single KD was not due to random variation. The test was applied to the final video-level predictions of the Lite-R21D architecture, which was our strongest Student model. Specifically, we compared the best single KD method (FitNet) against the best hybrid configuration (AT-FitNet) on the same UCF101 test set. The test identified 56 cases where FitNet was correct while AT-FitNet was incorrect, compared with 103 cases where AT-FitNet was correct while FitNet was incorrect. This resulted in a statistically significant improvement (p = 0.000264 < 0.05), indicating that the performance gain achieved by the hybrid KD strategy is reliable and not due to randomness.

Despite the strong overall performance of hybrid KD, certain limitations emerged. First, combining all three paradigms does not always produce the best result. For Lite-R21D, the Unified Hybrid achieves 91.49%, which is lower than AT-FitNet at 92.07%. This suggests that adding more distillation objectives does not necessarily lead to better optimization for a compressed Student model. Second, the most effective hybrid strategy varies across Student architectures. For Lite-MC3, Hinton-FitNet achieves 87.91% and outperforms AT-FitNet at 85.65%, whereas, for Lite-R21D and Lite-LF, AT-FitNet achieves the highest accuracy, reaching 92.07% and 89.96%, respectively. These results indicate that hybrid KD effectiveness is architecture-dependent, and that the best combination of response, attention, and feature-level supervision depends on the Student’s internal spatiotemporal design.

5.3. Class-Wise Performance and Feature Analysis

In this section, we move our attention beyond the overall Top-1 accuracy achieved by each KD method across the lightweight models. For this, we conducted a detailed class-level analysis to better understand how structural distillation transfers semantic knowledge. As mentioned previously, we observed that the performance gap between the scratch-trained and the fine-tuned Teacher (pre-trained on Kinetics-400) highlights the specific actions that depend most heavily on pre-training on a large dataset. We extracted, based on this gap, the top five hardest classes, which we defined as the actions that provide the maximum gain in accuracy after Kinetics-400 pre-training compared to what they learned from scratch on UCF101. Essentially, these specific actions involve complex spatiotemporal movements that are difficult for these architectural designs to learn when trained on a limited dataset like UCF101. Thus, we wanted to evaluate how our various single and hybrid KD paradigms affect the performance on these five actions across our three Student architectures. With such an experiment, we quantified the extent to which such a KD approach can compensate for the lack of large-scale pre-training.

Table 4. Class-wise Top-1 accuracy (%) of single and hybrid KD methods on the five most challenging UCF101 action classes. Teacher-FT denotes the fine-tuned Teacher, Teacher-Sch denotes the scratch-trained Teacher, and Unified Hybrid denotes the simultaneous combination of Hinton, AT, and FitNet. All values are reported with two decimal places.

Class Name	Teacher-FT	Teacher-Sch	Scratch	Hinton	AT	FitNet	Hinton-FitNet	Hinton-AT	AT-FitNet	Unified Hybrid
Lite-R21D
Nunchucks	91.43	22.86	28.57	51.43	42.86	71.43	74.29	65.71	77.14	77.14
BrushingTeeth	72.22	22.22	36.11	47.22	44.44	66.67	55.56	38.89	72.22	50.00
HandstandWalking	67.65	26.47	20.59	58.82	52.94	70.59	67.65	70.59	67.65	70.59
Basketball	97.14	57.14	40.00	71.43	74.29	71.43	85.71	88.57	88.57	85.71
FrontCrawl	78.38	40.54	62.16	59.46	59.46	78.38	64.86	72.97	75.68	72.97
Lite-MC3
Nunchucks	91.43	22.86	22.86	34.29	20.00	40.00	54.29	60.00	45.71	60.00
BrushingTeeth	72.22	22.22	19.44	27.78	50.00	61.11	58.33	38.89	58.33	72.22
HandstandWalking	67.65	26.47	11.76	35.29	32.35	38.24	38.24	41.18	41.18	47.06
Basketball	97.14	57.14	34.29	57.14	62.86	54.29	51.43	62.86	60.00	71.43
FrontCrawl	78.38	40.54	45.95	43.24	35.14	72.97	56.76	70.27	62.16	64.86
Lite-LF
Nunchucks	91.43	22.86	25.71	40.00	34.29	62.86	65.71	62.86	71.43	65.71
BrushingTeeth	72.22	22.22	30.56	41.67	55.56	66.67	55.56	58.33	66.67	63.89
HandstandWalking	67.65	26.47	23.53	35.29	41.18	55.88	67.65	52.94	64.71	67.65
Basketball	97.14	57.14	34.29	57.14	60.00	60.00	60.00	65.71	57.14	62.86
FrontCrawl	78.38	40.54	40.54	54.05	37.84	75.68	72.97	59.46	81.08	64.86

details the class-wise accuracy for these five challenging actions, comparing both configurations of the Teacher against its baseline scratch version, and the distilled Student variants.

To help explain the spatiotemporal complexity in these five categories, Figure 5 visualizes samples of extracted frames. When looking at these actions, one can notice that they commonly share rapid and fine-grained interactions with small objects that can blur within the background (e.g., Nunchucks, Basketball). Another complexity is due to severe occlusion and small movements of the object (e.g., BrushingTeeth). Furthermore, classes like HandstandWalking and FrontCrawl introduce heavy body pose variation, which makes tracking the individual for recognition a hard task in such environments. These characteristics make such classes even more difficult for lightweight models trained from scratch on the limited-scale UCF101 dataset.

These challenges motivate the use of hybrid KD strategies, which aim to simultaneously transfer complementary spatiotemporal cues from the Teacher, enabling lightweight Students to better capture both spatial details and temporal dynamics. To visualize how these KD types can effectively compensate for the lack of pre-training on the large Kinetics-400 dataset, Figure 6 provides radar charts that map the class-wise accuracy gains, as provided by the results in Table 4.

The scratch baseline accuracy across these five classes is plotted as a dashed gray area in all subplots. This serves as a reference that demonstrates the impact of single KD types (left panels) versus the hybrid combinations (right panels). The scratch-trained baseline models have notably low recognition performance on these five classes, sometimes around an accuracy of 20%. For instance, the scratch-trained Lite-R21D achieves only 28.57% on Nunchucks and 20.59% on HandstandWalking, whereas the expert fine-tuned Teacher establishes robust upper boundaries of 91.43% and 67.65%, respectively, on those two classes.

Figure 6 demonstrates the performance gain of these actions under different KD strategies. This figure shows that, when evaluating single KD types, the standalone FitNet consistently provides the most comprehensive coverage by yielding the most substantial accuracy gains across these five challenging actions. Furthermore, the hybrid paradigms generally enhance class-wise accuracy more than single KD types, except for in some cases where the combined KD techniques introduce minor optimization conflicts on specific actions, i.e., the single FitNet KD achieves 72.97% on BrushingTeeth for the Lite-R21D model, compared to 67.57% with the hybrid AT-FitNet. Despite these rare exceptions, the advantage of these hybrid configurations is still clear.

For example, on the Lite-R21D architecture, the AT-FitNet combination acts as the optimal configuration, boosting Nunchucks accuracy from a scratch value of 28.57% to 77.14%, which is approximately 5.7% higher than the standalone FitNet. Even more notably, on HandstandWalking, AT-FitNet completely recovers the target performance, jumping from 20.59% to exactly match the Teacher’s 67.65%.

Similarly, for the highly compressed Lite-MC3 model, the Hinton-FitNet configuration further enhances the model’s capabilities to successfully restore the needed capability for these complex actions, such as Nunchucks, from a scratch accuracy value of 22.86% to 54.29%. Finally, these class-wise results confirm that such hybrid distillation not only improves the overall average accuracy of the model, but also targets and recovers the hardest spatiotemporal classes that otherwise require pre-training on larger datasets.

We further employed the t-SNE to demonstrate how the best single and hybrid KD mechanisms align the internal feature representations of the best Student Lite-R21D with those of the Teacher. Figure 7 visualizes the 2D feature embeddings for the hardest five actions. The top panel establishes the optimal target distribution, displaying the highly discriminative, well-separated class clusters formed by the fine-tuned Teacher. In obvious contrast, the feature space of the scratch-trained Lite-R21D (bottom left) is heavily entangled, with severe overlap across all categories, reflecting its inability to learn distinct spatiotemporal boundaries from limited data. When applying single-method distillation via FitNet (bottom middle), the Student successfully begins to isolate these representations, grouping complex actions like FrontCrawl and BrushingTeeth into recognizable clusters. However, the structural advantage of the hybrid paradigm can clearly be noticed in the AT-FitNet configuration (bottom right). This hybrid approach shows much tighter and more separated clusters of these classes. Crucially, the topological arrangement of the AT-FitNet KD feature space closely mirrors the Teacher’s target distribution. This suggests the clear advantage of pairing these techniques. By leveraging FitNet to align local features and AT to direct spatiotemporal focus, the hybrid KD yields a highly organized and structured embedding.

Class-wise results revealed that combining more distillations does not always guarantee better recognition. For Lite-R21D on BrushingTeeth, FitNet alone achieves 66.67%, while the Unified Combined setting drops to 50.00%. Similarly, for Lite-LF on FrontCrawl, AT-FitNet reaches 81.08%, compared with 64.86% for Unified Combined. These cases show that the best-performing global KD configuration may not always be optimal for every action class. Finally, the effectiveness of hybrid versus single distillation remains highly dependent on both the target architecture and the specific action category. For instance, on FrontCrawl, Lite-MC3 achieves 72.97% using the single FitNet KD, which outperforms both AT-FitNet at 62.16% and the Unified Combined configuration at 64.86%. Thus, while hybrid KD generally improves semantic alignment, it is not universally superior at the class level, and single structural distillation can sometimes provide a more stable optimization path for specific compressed architectures and action categories.

5.4. Cross-Domain Transfer to HMDB51

Having trained our models using different KD approaches, we further extended our experiments to the HMDB51 dataset in order to evaluate their generalization capabilities on a dataset with higher spatiotemporal complexity and domain variability compared to UCF101. Here, we designed a two-stage transfer learning pipeline, as detailed in

Table 5. Cross-domain transfer performance on HMDB51 using UCF101-initialized models.

Model	Scratch	FitNet	Hinton	AT	Hinton-AT	AT-FitNet	Hinton-FitNet	Unified Hybrid
Teacher	69.48	—	—	—	—	—	—	—
Lite-R21D	49.15	55.56	50.00	56.60	57.32	60.52	58.50	58.30
Lite-MC3	39.67	45.62	44.84	47.52	47.25	48.69	52.22	48.37
Lite-LF	44.90	49.41	48.95	52.68	54.05	54.18	56.80	53.20

. Our main goal was to determine which UCF101 training paradigm provides the strongest recognition performance when transferred to a secondary dataset. To this end, we took the Student models previously trained on UCF101 (scratch, single KD, and hybrid KD) and directly fine-tuned them on HMDB51, as presented in Table 5.

The results indicate that, although the AT KD method offers the strongest initialization among all single approaches, showing that it can reach to 56.60% for the case of Lite-R21D model, compared to its scratch baseline of 49.15%, hybrid KD strategies, on the other hand, still significantly outperform these single KD method baselines. For instance, fine-tuning the Lite-R21D model trained using the combined AT and FitNet KD types on UCF101 achieves an accuracy of 60.52% on HMDB51. Moreover, the FitNet-Hinton hybrid configuration emerges as the most effective initialization for both Lite-MC3 and Lite-LF variants, achieving accuracies of 52.22% and 56.80%, respectively.

The second stage of our experiments on HMDB51 aimed to further maximize the performance capacity of these lightweight models. In this stage, we introduced a second round of active distillation, as shown in

Table 6. Second-stage KD results on HMDB51 based on best selected UCF101 KD initializations.

Model	AT	FitNet	Hinton	Hinton-AT	AT-FitNet	Hinton-FitNet	Unified Hybrid
Teacher	—	—	—	—	—	—	—
Lite-R21D	62.48	61.57	63.20	64.90	65.29	65.56	64.84
Lite-MC3	52.75	46.99	53.33	55.36	55.10	55.56	55.88
Lite-LF	58.63	54.38	58.04	58.43	59.87	60.33	60.26

. Instead of standard fine-tuning, we utilized the corresponding UCF101-trained weights of the best-performing configurations from the first stage as initialization for a second round of full KD training on the HMDB51 dataset. By performing such sequential training, we systematically evaluated all single and hybrid KD combinations.

The results in Table 6 further confirm that hybrid distillation strategies consistently yield the highest performance across all architectures. For the Lite-R21D model, initializing from AT+FitNet and applying Hinton-FitNet in the second stage increases the accuracy to 65.56%, representing an improvement of approximately 5% over its fine-tuned standard baseline and 16.41% over its scratch baseline. For the Lite-MC3 model, combining all three KD mechanisms (Hinton-AT-FitNet) achieves the highest accuracy of 55.88%, while the Lite-LF model reaches its peak performance of 60.33% using the Hinton-FitNet configuration.

From this two-stage setup on HMDB51, we observed that hybrid KD training on UCF101 provides strong generalization when transferred to a new dataset. Furthermore, the second-stage distillation demonstrated that sequential KD on a secondary dataset can still help refine more complex spatiotemporal patterns.

Figure 8 plots the summary of this important evolution by demonstrating the highest accuracies achieved across all KD methods. This progression from scratch baselines to the optimal Stage 1 and then Stage 2 peaks highlights the robust generalization of these KD techniques and the importance of sequential distillation in driving the models to their maximum performances.

5.5. Deployment Feasibility and Real-Time Inference Latency

To assess the deployment of our models in real-time applications, we evaluated the inference latency on two computational mobile devices: the first equipped with an Intel i7-12700H CPU and NVIDIA RTX 3050 Ti GPU, and the second a more powerful device that has an Intel i9-13980HX CPU and NVIDIA RTX 4080 GPU. In addition to a fixed video decoding and augmentation time (2567 ms on Device 1 and 978 ms on Device 2),

Table 7. Inference time (ms) of the full-capacity Teacher R (2+1) D versus the three proposed lightweight models across two devices.

	Device 1 (ms)		Device 2 (ms)
Model	CPU-1	GPU-1	CPU-2	GPU-2
Teacher-R (2+1) D	1482	113	544.4	40.2
Lite-R21D	609.26	46.64	200.9	13.7
Lite-MC3	331.2	19.77	113.8	10.1
Lite-LF	290.4	16.6	101.5	6

reports the CPU and GPU inference times averaged over ten runs for each setup. The results show that the Teacher model remains computationally expensive across both setups (e.g., 1482 ms on the Device 1 CPU), while all Student models achieve a notable reduction in latency.

Testing on Device 1 showed that all three Student models demonstrate substantial GPU speedups compared to the Teacher’s 113 ms. The Lite-R21D reduces latency to 46.64 ms, which translates to being approximately 2.4 times faster than the full-capacity R (2+1) D, while the Lite-MC3 drops to 19.77 ms (around 5.7 times speedup). However, the ultimate winner in terms of inference speed is the Lite-LF model, which operates at only 16.60 ms per clip, which is approximately 6.8 times faster than the Teacher.

The speed of this model is directly related to its architectural configuration, which processes lightweight 2D spatial features first (Layers 1, 2, and 3), and only applies factorized spatiotemporal operations at the network’s final stages. Furthermore, these gains in inference time persist even under CPU execution, confirming the models’ robustness in constrained hardware environments. We also observed that, on both devices, the efficiency ranking of the models remained the same, indicating that these improvements are driven by the architectural designs rather than a dependency on the hardware, as visualized in Figure 9.

By achieving inference times shorter than 20 ms, we ensured that our lightweight models are ready to be deployed in real-time surveillance systems. This level of computational efficiency enables instant anomaly detection, traffic monitoring, and automated public safety responses without relying on high-latency cloud computing infrastructure.

5.6. Comparative Analysis with State-of-the-Art Architectures

There were several challenges to consider when comparing the performance of our proposed models with state-of-the-art architectures. First, there is a relative scarcity of studies that evaluate the adopted KD mechanisms and their hybrid interactions in video action recognition. Second, state-of-the-art studies in this domain utilize different Teacher–Student setups with distinct architectural backbones. In addition, these distillation designs generally incorporate multi-modal frameworks, which makes direct comparison a challenging task. Furthermore, the variety of data preprocessing, including augmentation strategies used across the field, adds another layer of complexity to fair benchmarking.

Despite these challenges, we selected several representative models to establish a comparative baseline, as detailed in

Table 8. Comparison of our proposed models with state-of-the-art architectures. The table shows the parameter count in millions (M), computational complexity in giga FLOPs (GFLOPs), and the spatial resolution of frames (Resolution), in addition to the Top-1 accuracy on the UCF101 and HMDB51 datasets and the corresponding references.

Model	Params (M)	GFLOPs	Resolution	UCF101 %	HMDB51 %	Reference
TSM	24.3	65	224	95.9	73.5	[56]
MEACI-Net	—	89	224	96.1	74.0	[64]
C3D	—	38.5	224	82.3	51.6	[55]
CoViAR	—	4.2	224	90.4	59.1	[55]
Res3D	—	19.3	112	85.8	54.9	[49]
I3D	27.43	57.08	224	95.06	73.46	[50]
R2plus1D-18	33.3	162.58	224	94.74	69.48	[38]
Top-I3D	—	47.4	—	66.6	54.5	[37]
MNv3-MFAE	1.45	7.31	224	96.74	—	[70]
CA3D	7.00	6.30	112	94.80	63.20	[71]
ViT-HAR	23.40	4.70	224	93.90	—	[72]
Lite-R21D	8.33	43.81	224	92.07	65.56	Proposed Model
Lite-LF	6.97	17.9	224	89.96	60.33	Proposed Model
Lite-MC3	4.2	40.58	224	87.91	55.88	Proposed Model

. The results demonstrate that our proposed lightweight models achieve a highly competitive accuracy and efficiency trade-off. For instance, the Lite-R21D model reaches an accuracy of 92.07% on UCF101, which approaches the performance of computationally expensive backbones like the I3D, which achieved 95.06%, and R (2+1) D, which achieved 94.74%. Notably, our Lite-R21D architecture operates with only 8.33 M parameters and 43.81 GFLOPs, which is a massive reduction compared to the full-capacity R (2+1) D Teacher’s 33.3 M parameters and 162.58 GFLOPs.

Our other efficient models also demonstrate strong performance. For instance, the Lite-LF operates at only 17.9 GFLOPs while still achieving 89.96% on UCF101 and 60.33% on HMDB51, a performance that outperforms heavier 3D CNNs like C3D (82.3%) and Res3D (85.8%). Similarly, the proposed Lite-MC3 offers the smallest memory footprint, running at just 4.2 M parameters while maintaining competitive accuracy.

Furthermore, when comparing our frameworks against recent edge-efficient architectures, the proposed lightweight models remain highly competitive. For example, while models such as MNv3-MFAE and CA3D provide high efficiency through their low parameter and FLOP counts, proposed models such as the Lite-LF architecture (17.9 GFLOPs) and Lite-MC3 (4.2 M parameters) achieve a comparable balance of efficiency and Top-1 accuracy through compact architectural design combined with hybrid KD-based accuracy recovery.

When comparing our approach to other distilled frameworks, all three lightweight models require fewer parameters and computational FLOPs than the generative Top-I3D backbone, which operates at 47.4 GFLOPs to reach 66.6% on UCF101 and 54.5% on HMDB51, while still achieving significantly higher accuracies.

5.7. Ablation Study

To determine the best hyperparameters for our distillation framework, we conducted a comprehensive ablation study by setting experiments on the Lite-R21D baseline, as detailed in

Table 9. Ablation study of hyperparameter tuning for individual knowledge distillation methods on the UCF101 dataset using the Lite-R21D baseline (120 epochs). The optimal configuration for each method is highlighted in bold.

KD Type	Target Type	$\mathit{T}/{\mathit{\lambda }}_{\mathit{Hint}}/\mathit{P}$	Weight	Top 1 Accuracy %
Hinton	Logits	$\mathit{T}$ Values	$\mathit{\alpha }$
	Logits	4	0.4	83.95
	Logits	8	0.4	83.37
	Logits	4	0.9	82.92
	Logits	8	0.9	85.06
	Logits	12	0.9	84.48
FitNet	Layers	—	${\mathit{\lambda }}_{\mathit{Hint}}$ Values
	2, 3	—	5.0	74.09
	3	—	5	74.17
	4	—	5	88.63
	4	—	8	87.68
AT	Layers	$\mathit{P}$	$\mathit{\beta }$
	3, 4	2	1000	82.26
	4	3	1000	83.14
	4	2	2000	85.14
	4	2	2400	84.67
	4	3	3000	83.48

. All the experiments were carried out by selecting the Lite-R21D model and a total number of 120 epochs. Each KD method is governed by specific parameters. Hinton KD relies on temperature (T) and the loss weight ($\alpha$) to soften the target logits. FitNet, on the other hand, is controlled by the selected hint layers and the feature weight (${\lambda }_{\mathrm{Hint}}$). Finally, AT KD depends on several parameters to function, including the paired layers between the Teacher and Student, the spatiotemporal power mapping value (P), and the attention weight ($\beta$).

Table 9 reports these experiments and highlights in bold the best hyperparameters adopted for each method. For instance, the optimal Hinton configuration utilizes $T=8$ and($\alpha =0.9$, while AT achieved its highest accuracy by pairing Layer 4 with the Teacher and having $P=2$ and $\beta =2000$. Similarly, FitNet reached its best accuracy by having ${\lambda }_{\mathrm{Hint}}=5$. These optimal hyperparameter configurations were subsequently applied to all hybrid distillation models to ensure fair and consistent evaluation across the remaining experiments.

To justify the choice of the 50% channel-width capacity used in our experiments, we conducted an additional width-scaling ablation study.

Table 10. Scratch-trained performance and computational cost on UCF101 for different architectural width configurations. Channel capacity indicates the scaled channel width relative to the full-capacity R (2+1) D Teacher; Params [M] reports the number of parameters in millions; and FLOPs [G] reports computational cost in giga Floating-Point Operations.

Model	Channel Capacity	Parms (M)	GFLOPs	Top-1 Accuracy %
Lite-R21D	50%	8.33	43.81	78.72
Lite-R21D	33%	3.68	22.73	77.61
Lite-R21D	25%	2.10	11.72	76.50
Lite-MC3	50%	4.2	40.58	70.31
Lite-MC3	33%	1.89	21.23	69.87
Lite-LF	50%	6.97	17.9	74.41
Lite-LF	33%	3.05	9.59	72.73

presents the scratch-trained performance, parameter counts, and computational costs of the Student architectures at half-scale (1/2), third-scale (1/3), and, for Lite-R21D, fourth-scale (1/4) capacity on UCF101.

The results demonstrate that stronger channel-width compression substantially reduces parameters and FLOPs while causing moderate decreases in scratch accuracy across the Student models. For example, Lite-LF decreases from 74.41% at 50% channel capacity to 72.43% at 33% capacity. Interestingly, Lite-R21D exhibits strong structural robustness under aggressive compression: its 25% capacity variant achieves 76.50% accuracy, still outperforming the 50% variants of both Lite-MC3 and Lite-LF. This suggests that the Teacher-aligned factorized topology of Lite-R21D preserves stronger representational capacity under channel-width reduction. However, to ensure that all topologies retain sufficient foundational capacity for absorbing complex spatiotemporal alignments during single and hybrid KD training, the 50% channel-width setting was retained as the main unified baseline for the complete distillation experiments.

6. Limitations and Challenges

Although our proposed models demonstrate strong accuracy–efficiency trade-offs, several limitations remain. First, the experimental evaluation was conducted on UCF101 and HMDB51, which are widely used benchmarks for video action recognition but are relatively limited in scale and spatiotemporal diversity compared with larger and more complex datasets such as Kinetics-400 and Something-Something V2. These larger datasets were not evaluated in this study due to the extensive computational cost of our experimental pipeline, which required training three distinct lightweight architectures across multiple single and hybrid KD configurations.

Second, our efficiency analysis focused on model-level indicators, including parameter count, FLOPs, and measured CPU/GPU inference latency. While these metrics provide useful evidence of computational efficiency, comprehensive hardware-level measures such as peak memory usage, power consumption, thermal behavior, and memory bandwidth, in addition to deployment on dedicated edge platforms such as NVIDIA Jetson and Raspberry Pi, were not examined in this work.

Finally, the full hybrid KD evaluation was conducted under the 50% channel-width setting to keep the experimental space computationally feasible. Although the width-scaling ablation showed that third- and fourth-scale models remain promising, extending all single and hybrid KD configurations to additional compression ratios would require substantially more training and is left for future work.

7. Conclusions

In this paper, we focused on the computational constraints associated with deploying 3D CNN-based video action recognition systems on resource-limited devices, which is a critical requirement for real-time applications in resource-constrained environments. To address this, we designed a suite of lightweight spatiotemporal architectures, Lite-R21D, Lite-MC3, and Lite-LF, which are built on the residual 3D CNN architectural principles but differ in the spatiotemporal modeling strategies across their network layers.

To compensate for the performance drops that generally result from model compression, we developed a comprehensive Knowledge Distillation (KD) framework in which we adopted the R (2+1) D model as the Teacher to supervise the learning of our three lightweight Students. Furthermore, we integrated three distinct KD paradigms and their four hybrid combinations. In contrast to the majority of state-of-the-art works, which mostly rely on response-based distillation (Hinton KD), our study explored the effectiveness of combining these KD approaches within our lightweight 3D CNN architectures for robust spatiotemporal knowledge extraction.

In this work, we conducted extensive experimental evaluations across 30 training configurations on UCF101, followed by a second domain validation on HMDB51. The results demonstrate that hybrid distillation strategies consistently outperform both single-method KD and scratch baselines. Specifically, by jointly combining two different KD types, the three proposed frameworks significantly narrow the performance gap between lightweight Students and the full-capacity Teacher. Our findings show that the optimal hybrid KD configuration pushed the performance of Lite-R21D model from its scratch values of 78.72% on UCF101 and 49.15% on HMDB51 to 92.07% and 65.56%, respectively. This impressive performance reduced the accuracy gap with the Teacher (94.74% on UCF101 and 69.48% on HMDB51) to only 2.67% and 3.92%.

Having approximated the Teacher’s performance, our proposed models showed significant efficiency improvements. The Lite-R21D model operated with only 8.33 M parameters and 43.81 GFLOPs, which represents approximately 75% fewer parameters and a 73% lower computational cost than the Teacher, while still maintaining close performance. This profile ensures its practical readiness for deployment on low-resource computational devices, as it runs with 2.4 times faster inference compared to the Teacher. Moreover, our other compact models, Lite-MC3 and Lite-LF, demonstrated that vision-based action recognition can be achieved with minimal resource requirements, with extreme reductions of approximately 87% in parameter count and 89% in FLOPs compared to the Teacher.

For future work, we aim to extend this framework to emerging video architectures, such as Transformer-based models, in addition to investigating multi-Teacher distillation strategies to further enhance the transfer of complex spatiotemporal knowledge.