Video Human Action Recognition Based on Motion-Tempo Learning and Feedback Attention

Abstract

In video human action-recognition tasks, motion tempo describes the dynamic patterns and temporal scales of human motion. Different categories of actions are typically composed of sub-actions with varying motion tempos. Effectively capturing sub-actions with different motion tempos and distinguishing category-specific sub-actions are crucial for improving action-recognition performance. Convolutional Neural Network (CNN)-based methods attempted to address this challenge, by embedding feedforward attention modules to enhance the action’s dynamic representation learning. However, feedforward attention modules rely only on local information from low-level features, lacking contextual information to generate attention weights. Therefore, we propose a Sub-action Motion information Enhancement Network (SMEN) based on motion-tempo learning and feedback attention, which consists of the Multi-Granularity Adaptive Fusion Module (MgAFM) and Feedback Attention-Guided Module (FAGM). MgAFM enhances the model’s ability to capture crucial sub-action intrinsic information by extracting and adaptively fusing motion dynamic features at different granularities. FAGM leverages high-level features that contain contextual information in a feedback manner to guide low-level features in generating attention weights, enhancing the model’s ability to extract more discriminative spatio-temporal and channel-wise features. Experiments are conducted on three datasets, and the proposed SMEN achieves top-1 accuracies of 52.4%, 63.3% on the Something-Something V1 and V2 datasets, and 76.9% on the Kinetics-400 dataset. Ablation studies, evaluations, and visualizations demonstrate that the proposed SMEN is effective for sub-action motion tempo and representation learning, and outperforms compared methods for video action recognition.

1. Introduction

Video human action recognition and analysis is one key research topic pertaining to applications including healthcare monitoring, human–computer interaction, gait analysis, and security surveillance [1,2,3,4]. For example, in healthcare settings, recognizing human activities allows for understanding the motion patterns and habits of elderly individuals. Thus, timely emergency responses, through video action recognition, will contribute to the creation of safer and healthier living environments for communities. However, for sports and daily activities, action motion’s visual tempo and temporal representation learning, similar action categories with long-range temporal dynamic structure, sub-action sharing within similar action categories, and non-periodic interactions make video action recognition more challenging.

Early researchers strived to explore hand-crafted feature-extraction methods for action-recognition tasks, which perform well in simple actions but struggle to handle complex ones, especially for subtle actions characterized by rapid changes over short periods, such as slight hand movements or brief interactions with objects. Researchers have proposed various 2D CNNs [5,6,7] and 3D CNNs [8,9,10] approaches to learn motion representation for video action recognition, with end-to-end manner.

However, motion-tempo modeling at a distinct temporal scale which tends to describe the motion dynamics and the long-term or short-term motion tempos of actions, thus benefits for video action recognition, is often overlooked [11,12]. For instance, due to the distinctive age, gender, and objects interacted with, the activity habits and motion patterns of persons are different when performing the same action categories. Speed variations result in large intra-class variations. Failing to model the motion tempo and the representation learning of action’s dynamic information deteriorates the action-recognition performance, especially for recognizing similar actions with less detailed dynamic or appearance differences, such as “Long jumping” and “Running”, “Hugging” and “Approaching”, “Pushing” and “Kicking”, etc.

As shown in Figure 1, ubiquitous sub-action sharing phenomenons between similar action categories make video action recognition more challenging. For example, in the Kinetics-400 dataset [13], the action category “Triple jump” is often confused with similar categories like “High jump” and “Long jump”, since they share the common sub-actions “Approach” and “Landing”, which contribute less to distinguishing them. It noted that the unique category-specific sub-action “Flight” is crucial and contributes more to the action category “High jump”.

Actually, from Figure 1, it is observed that different action categories have varying motion tempos; “Take-off” is a fast-tempo sub-action, while the action-specific sub-actions, such as “Hop”, “Skip”, “Flight”, and “Action in the air” have slower tempos. During the training of CNNs, more computational resources are allocated to extract the information of same video frames, but overlook the category-specific motion dynamic tempos, thus struggling to distinguish similar actions. In contrast, the unique sub-action information specific to each category is crucial for recognizing them. For instance, the sub-action “Action in the air” is crucial for recognition of the action category “Long jump”. Therefore, effectively capturing and distinguishing the crucial sub-actions of action categories is vital for video action classification.

To effectively enhance the ability of CNNs to capture sub-action information, two-stream methods [14] and temporal segment networks (TSNs) [15] utilize pre-extracted optical flow to extract fast motion-tempo information of sub-actions. However, time-consuming optical flow computation requires substantial computational resources. To this, Motion Squeeze [16] is proposed by employing lightweight motion-feature learning to replace the resource-intensive optical flow calculation, directly from adjacent video frames. Additionally, researchers have applied attention mechanisms to capture key sub-action information for video action recognition. For instance, attention modules [17,18] that assess the importance of different parts of the input features are utilized along the temporal, channel, or spatial dimensions by calculating a set of weights. The weights are then applied to the input features to extract critical information within the corresponding dimensions. The self-attention-based modules are proposed [19,20,21,22] for processing input features, by calculating the similarity between each position and all other positions of feature maps. Applying these weights to the input features allows us to capture global information more effectively.

However, on one hand, the models used to capture the aforementioned sub-action information can effectively extract the sub-action’s fast-tempo information and struggle to capture slow tempos. On the other hand, these attention modules operate in a feedforward manner, where information flows from the input layer to the output layer in a single direction, relying solely on low-level features before the attention module generates attention weights. The feedforward manner has the advantages of computational simplicity and ease of training but limits the attention mechanism to the representational capacity and low-level semantic information of a single feature. Actually, in the visual system [23], high-level semantic information and task-specific contextual information can be obtained from high-level features through feedback connections, in which high-level information is transmitted back to lower layers. Feedback connections have been proven effective in the visual cortex of primates by neuroscientists [24] and are considered to be a key component of visual attention [25,26]. Similarly, human visual perception transmits high-level semantic information through feedback connections and attention, selectively enhancing and suppressing neuronal activation [27].

Therefore, inspired and motivated by the aforementioned truths, we propose a Multi-Granularity Adaptive Fusion Module (MgAFM), to capture sub-action temporal information at various motion tempos and fuse it with the original RGB features for later representation learning. MgAFM captures crucial sub-action intrinsic information by adaptive motion dynamic features fusion at different granularities.

On the other hand, we propose a Feedback Attention-Guided Module (FAGM), which integrates the action semantic and action category-specific contextual information in a feedback manner. FAGM aims to guide the low-level feature learning by the high-level contextual information, focusing on sub-action spatio-temporal and channel-wise information, enhancing the model to focus on information crucial for action classification. In summary, we propose the Sub-action Motion information Enhancement Network (SMEN), mainly consisting of MgAFM and FAGM, designed to improve action classification performance by effectively capturing and discriminating sub-actions with diverse motion tempos. The main contributions of this paper can be summarized as follows:

• To capture sub-action motion tempos that characterize the dynamic motion information at different temporal granularities, we propose a Multi-Granularity Adaptive Fusion Module (MgAFM). This module captures sub-action temporal information at various motion tempos and adaptively fuses it with the original RGB features for subsequent representation learning.
• The Feedback Attention-Guided Module (FAGM) is proposed, which utilizes high-level features containing action semantics and action category-specific contextual information to guide low-level features in a feedback manner, focusing on important spatio-temporal and channel-wise information for distinguishing category-specific sub-actions.
• Extensive experiments on Something-Something V1, Something-Something V2, and Kinetics-400 datasets achieve top-1 accuracies of 52.4%, 63.3%, and 76.9% respectively, demonstrating the effectiveness of the proposed model.

2. Related Work on Video Action Recognition Methods

According to neural network backbones, sub-action motion-tempo learning, and attention mechanisms for video action recognition, existing methods can be classified into CNN-based, sub-action motion learning-based, and attention mechanism-based methods.

2.1. Methods Based on CNNs

The existing CNN-based methods tailored for human action recognition mainly have two categories, 2D CNN and 3D CNN approaches. As a typical 2D CNN, TSN [15] leverages a sparse temporal sampling strategy to model long-term temporal dependencies. Meanwhile, Temporal Shift Module (TSM) [7] employs a zero-parameter mechanism that shifts specific channels along the temporal dimension, enhancing the flow of information between consecutive frames.

In contrast, 3D CNNs inherently possess the capability to directly model temporal information with 3D convolutional kernels. Convolutional 3D (C3D) [8] adopts a straightforward yet effective spatio-temporal feature-learning method, effectively capturing dynamic video content for video action recognition tasks. However, resource-intensive 3D CNNs often entail a substantial number of model parameters, leading to computational burden and training inefficiency. To this, Inflated 3D Convolutional Networks (I3D) [9] extends 2D convolutional kernels into 3D by expanding along the temporal axis, thus enabling the efficient processing of spatio-temporal features within video data. Pseudo-3D Networks (P3D) [28] decompose 3D convolutions into a 2D spatial and 1D temporal convolutions. I3D [9] and P3D [28] successfully mitigate the computational burden while maintaining robust spatio-temporal feature extraction.

Nevertheless, compared to 3D CNNs, 2D CNNs offer computational efficiency and simpler training pipelines, highlighting a trade-off between temporal modeling accuracy and resource demands.

2.2. Methods for Capturing Motion Tempos of Sub-Action

Accurately capturing sub-action information at varying motion tempos is essential for improving a model’s ability to capture the temporal dynamics of video actions. Sub-actions often exhibit considerable differences in their motion tempos. To address this challenge, SlowFast [29] employs a dual-pathway architecture that captures motion information of slow–fast tempos. MotionSqueeze [16] utilizes adjacent RGB features to generate frame-level motion representations, which aids in capturing fast-tempo sub-action information, whereas struggles to adequately capture slow-tempo sub-actions. Similarly, AGPN [30] uses RGB features sampled at two different frame intervals to capture both fast-tempo and slow-tempo sub-action motion information.

However, these methods are limited to fully capture the sub-action information that is most pertinent to action classification. To capture important sub-actions with varying motion tempos, we propose a Multi-Granularity Adaptive Fusion Module (MgAFM) to extract a crucial sub-action’s intrinsic information with fast and slow motion tempos, and fuses them adaptively at different granularities.

2.3. Video Action Recognition Method Based on Attention Mechanism

Attention mechanism-based video action recognition methods have demonstrated significant improvements to capture key sub-action information. For instance, TEA [17] adopts an attention mechanism to propose the Motion Excitation (ME) module, which operates along the channel dimension, enhancing the weights of channels that focus on dynamic motion patterns. The famous non-local network [20] utilizes a feedforward self-attention mechanism to design a non-local module for capturing long-range dependencies.

However, these methods lack the exploitation of high-level features that contain semantic and task-specific contextual information. In contrast, we proposed FAGM to focus on the sub-action motion-tempo information that is essential for video action classification.

3. The Proposed Critical Action Information Extraction Network

In this section, we will describe the proposed SMEN. As depicted in Figure 2, SMEN uses ResNet50 as its backbone and is composed of two main components: the Multi-Granularity Adaptive Fusion Module (MgAFM) and the Feedback Attention-Guided Module (FAGM). Initially, the T visual frames of the action videos are processed via convolution operations and encoded into feature maps. To capture motion features of sub-actions with varying tempos, the designed MgAFM is introduced following the layer Res3. The temporal dynamics are explored across multiple temporal spans, incorporating various temporal granularities, and then are fused with RGB appearance information adaptively. To identify the most important contextually relevant feature maps, a feedback attention module (FAGM) is designed for video representation learning along temporal, channel, and spatial dimensions.

3.1. Multi-Granularity Adaptive Fusion Module (MgAFM)

The MgAFM aims to capture sub-action motion dynamics with fast and slow tempos of varying motion amplitudes and speeds, from RGB features. MgAFM includes the Motion-Extraction Module (MEM) and the Adaptive Fusion Module (AFM), which extract and adaptively fuse sub-action motion features with varying tempos.

3.1.1. Motion-Extraction Module (MEM)

The MEM is responsible for extracting motion features from sub-actions at different tempos. As illustrated in Figure 3, we employ motion transformations with $(n-1)$ frame intervals to represent sub-actions motion features at different tempos. For the video data, T frames are sampled as input to the model and subsequently converted into feature representations through convolution calculation. The features corresponding to different frame intervals are represented as ${\mathbf{F}}^t\in {\mathbb{R}}^{C\times H\times W}$ and ${\mathbf{F}}^{t+n}\in {\mathbb{R}}^{C\times H\times W}$, where C, H, and W represent the channel dimension, height, and width, respectively. The C dimensional features at spatial position x are denoted by ${\mathbf{F}}_x$. The correlation score for position x with a displacement p is defined as:

$$s(x,p,t)={\mathbf{F}}_x^t·{\mathbf{F}}_{x+p}^{t+n}.$$

(1)

where · denotes the dot product. To enhance computational efficiency, x is computed only within a neighborhood of size $P=2R+1$, limiting the displacement to $p\in {[-R,R]}^2$. For the $t_{th}$ frame, the resulting correlation tensor ${\mathbf{S}}^{\left(t\right)}$ has dimensions $H\times W\times P^2$. In our implementation, considering the spatial resolution of $28\times 28$, we set $P=15$.

To establish the motion relationship between video frames at different intervals, the correlation tensor ${\mathbf{S}}^{\left(t\right)}$ is utilized to generate a displacement field that describes the motion trajectory. To mitigate the influence of noisy outliers in the correlation tensor on the displacement estimation, we adopt the kernel-soft-argmax method [32]. The kernel-soft-argmax for displacement computation is defined as:

$$d(x,t)=\sum _p{\displaystyle \frac{exp\left(g\right(x,p,t\left)s\right(x,p,t)/\tau )}{{\sum }_{p^{\prime }}exp(g(x,p^{\prime },t)s(x,p^{\prime },t)/\tau )}}p.$$

(2)

where

$$g(x,p,t)={\displaystyle \frac{1}{\sqrt{2\pi }\sigma }}exp({\displaystyle \frac{p-{\mathrm{argmax}}_{p}s(x,p,t)}{{\sigma }^2}}).$$

(3)

the function $g(x,p,t)$ represents a Gaussian kernel, with the standard deviation $\sigma$ empirically set to 5. The temperature factor $\tau$ is introduced to adjust the softmax distribution, and based on empirical observations in our experiments, $\tau$ is set to 0.01. Additionally, a confidence map of the correlation tensor is applied as supplementary motion information. This confidence map is generated by pooling the highest correlation at each position x:

$$s^{*}(x,t)=\underset{p}{max}s(x,p,t).$$

(4)

To represent sub-action motion features at different tempos, we define a displacement tensor ${\mathbf{D}}_i\in {\mathbb{R}}^{H\times W\times 3}$, which is formed by concatenating the single-channel confidence map and the two-channel displacement map.

3.1.2. Adaptive Fusion Module (AFM)

The AFM adaptively fuses motion features from sub-actions at different tempos to capture the varying dynamics. As shown in Figure 4, the displacement tensor resulting from MEM processing is denoted as ${\mathbf{D}}_1,{\mathbf{D}}_2,\dots ,{\mathbf{D}}_n$. Initially, the displacement tensors from the n branches, each corresponding to different motion tempos sub-actions, are connected and passed through average pooling to obtain a global representation. This global representation is then fed into MLP and sigmoid function to compute the adaptive weight vector:

$${\mathbf{D}}_c=[{\mathbf{D}}_1;{\mathbf{D}}_2;···;{\mathbf{D}}_n]\in R^{3n\times H\times W}$$

(5)

$${\mathbf{W}}_c=\sigma \left(\mathrm{MLP}(\mathrm{AvgPool}\left({\mathbf{D}}_c\right)\right))\in R^{3n\times 1\times 1}$$

(6)

where $\sigma$ denotes the sigmoid function and ${\mathbf{D}}_c$ represents the features concatenated along the channel dimension. Secondly, the concatenated ${\mathbf{D}}_c$ is element-wise multiplied with the weight vector ${\mathbf{W}}_c$ to yield the feature vector ${\mathbf{D}}_f$:

$${\mathbf{D}}_f={\mathbf{D}}_c\odot {\mathbf{W}}_c\in R^{3n\times H\times W}$$

(7)

where ⊙ denotes the element-wise multiplication. Finally, we use three $1\times 3\times 3$ separable convolutions to convert the displacement tensor ${\mathbf{D}}_f$ into features with the same dimensions as the original RGB features.

3.2. Feedback Attention-Guided Module (FAGM)

To further enhance the capability of CNNs to capture sub-action information, we propose a Feedback Attention-Guided Module (FAGM). The FAGM consists of attention modules applied to the temporal, channel, and spatial dimensions. Firstly, the low-level and high-level features are separately fed into the Temporal Attention Module (TAM), to take more attention to frames that are informative with salient motion regions. Next, the features processed by TAM are then combined and passed into the Channel Attention-Guided Module (CAGM) and the Spatial Attention-Guided Module (SAGM). CAGM and SAGM modules leverage contextual information to guide the low-level feature learning, to focus on the channels and spatial regions most relevant for promoting video action classification.

3.2.1. Temporal Attention Module (TAM)

As shown in Figure 2, within FAGM, the features before and after the convolution $1\times 1$ operation are divided into low-level and high-level features, denoted as ${\mathbf{F}}_l\in {\mathbb{R}}^{N\times C_l\times H_l\times W_l}$ and ${\mathbf{F}}_h\in {\mathbb{R}}^{N\times C_h\times H_h\times W_h}$, respectively. For the low-level features ${\mathbf{F}}_l$, as illustrated in Figure 5, we first perform a feature reshape to expand ${\mathbf{F}}_l$ to ${{\mathbf{F}}_l}^{\prime }\in {\mathbb{R}}^{N_1\times T\times C_l\times H_l\times W_l}$ ($N_1=N/T$). Secondly, $W_l\in {\mathbb{R}}^{N_1\times T\times 1\times 1\times 1}$ is obtained through the operation of average pooling ${\mathbf{F}}_l^{\prime }$ along the temporal dimension and processing it with a sigmoid function. Finally, after performing element-wise multiplication on $W_l$ and ${\mathbf{F}}_l^{\prime }$, the result is added to ${\mathbf{F}}_l^{\prime }$ and reshaped into the feature ${\mathbf{F}}_l^T\in {\mathbb{R}}^{N\times C_l\times H_l\times W_l}$, which contains key frame information. The high-level features ${\mathbf{F}}_h$ are processed in the same way to obtain the feature ${\mathbf{F}}_h^T$. The specific formulas are as follows:

$${\mathbf{F}}_l^{\prime }=\mathrm{reshape}\left({\mathbf{F}}_l\right)$$

(8)

$$W_l=\sigma \left(\mathrm{AvgPool}\left({\mathbf{F}}_l^{\prime }\right)\right)$$

(9)

$${\mathbf{F}}_l^T=\mathrm{reshape}\left(W_l\odot {\mathbf{F}}_l^{\prime }+{\mathbf{F}}_l^{\prime }\right)$$

(10)

After processing with TAM, the model can focus on video frames’ rich motion information, reducing the impact of frames with less noticeable motion changes on video action recognition.

3.2.2. Channel Attention-Guided Module (CAGM)

As shown in Figure 6, the features ${\mathbf{F}}_l^T$ and ${\mathbf{F}}_h^T$ are first average-pooled along the channel dimension to generate feature vectors $V_l\in {\mathbb{R}}^{N\times C_l\times 1\times 1}$ and $V_h\in {\mathbb{R}}^{N\times C_h\times 1\times 1}$. Next, the number of channels is reduced into $C_l/\alpha$ through separate Multilayer Perceptron (MLP). The resulted vectors are then concatenated along the channel dimension and processed through another MLP to generate the visual attention-guided feature $V_g\in {\mathbb{R}}^{N\times C_l\times 1\times 1}$. Finally, $V_g$ is fed into a sigmoid function to produce the channel attention weight vector $W_c\in {\mathbb{R}}^{N\times C_l\times 1\times 1}$. This weight vector $W_c$ is used for element-wise multiplication with the low-level feature ${\mathbf{F}}_l$ to obtain the feature ${\mathbf{F}}_l^{T,C_l}\in {\mathbb{R}}^{N\times C_l\times H_l\times W_l}$. The specific formulas are as follows:

$$V_l=\mathrm{AvgPool}\left({\mathbf{F}}_l^T\right)$$

(11)

$$V_h=\mathrm{AvgPool}\left({\mathbf{F}}_h^T\right)$$

(12)

$$V_g={\mathrm{MLP}}_3[{\mathrm{MLP}}_1\left(V_l\right);{\mathrm{MLP}}_2\left(V_h\right)]$$

(13)

$$W_c=\sigma \left(V_g\right)$$

(14)

$${\mathbf{F}}_l^{T,C_l}=W_c\odot {\mathbf{F}}_l$$

(15)

After processed by the CAGM, different channels of the low-level features are assigned different weights by the guidance of high-level contextual information leverage.

3.2.3. Spatial Attention-Guided Module (SAGM)

As demonstrated in Figure 7, ${\mathbf{F}}_l^{T,C_l}$ undergoes pooling separately to capture global and local salient information. Then, the results are concatenated to obtain the feature ${\mathbf{F}}_s\in {\mathbb{R}}^{N\times 2\times H_l\times W_l}$. Afterward, ${\mathbf{F}}_s$ is processed through a convolutional layer for dimension reduction, followed by a sigmoid function. Finally, the result is then element-wise multiplied with ${\mathbf{F}}_l^{T,C_l}$ to obtain ${\mathbf{F}}_l^{T,C_l,S}\in {\mathbb{R}}^{N\times C_l\times H_l\times W_l}$. The specific formulas are as follows:

$${\mathbf{F}}_s=[\mathrm{AvgPool}\left({\mathbf{F}}_l^{T,C_l}\right);\mathrm{MaxPool}\left({\mathbf{F}}_l^{T,C_l}\right)]$$

(16)

$${\mathbf{F}}_l^{T,C_l,S}=\sigma \left[Con{v}_{7\times 7}\left({\mathbf{F}}_s\right)\right]\odot {\mathbf{F}}_l^{T,C_l}$$

(17)

where ${\mathrm{Conv}}_{7\times 7}$ represents a convolution operation with a kernel size of $7\times 7$. By iterating through the above steps multiple times, the model can fully exploit the contextual information within high-level features.

To enhance the ability of CNNs to activate key sub-actions and motion tempos learning, we propose SMEN. SMEN employs ResNet50 as its backbone and consists of two important components: MgAFM and FAGM. MgAFM is designed to capture motion features of sub-actions with varying tempos and motion dynamics, and adaptively fuse sub-action motion features with multi-granularity. FAGM leverages contextual information and salient motion frames from high-level features, to guide low-level feature learning, thus focusing on channel and spatial information which facilitates video action classification.

4. Experiments

Experiments are performed on three large action datasets: Something-Something V1, Something-Something V2 [33], and Kinetics-400 [13]. Furthermore, ablation studies are carried out to evaluate the impact of different motion granularities, and feedback attention on the performance of our proposed methods. We also compare our approach with the state of the art and perform an activation map visualization to explore the working mechanism of the proposed method.

4.1. Datasets and Experiment Setup

4.1.1. Datasets

To verify the effectiveness of the proposed method, experiments, and evaluations were conducted on three large publicly available human video action-recognition datasets.

Something-Something V1 is a richly annotated video action dataset that contains basic interactions between humans and various objects. It includes a wide range of subtle actions characterized by rapid changes over short periods, such as slight hand movements or brief interactions with objects. The dataset comprises 174 action categories and 108,499 videos, where 86,017 are used for training, 11,522 are used for validation, and 10,960 for testing.

Something-Something V2 is an expanded action datasets, based on Something-Something V1, including 174 action categories and total 220,847 videos, with 168,913 training videos, 24,777 validating videos, and 27,157 testing videos. Figure 8 illustrates sample frames from three action categories from Something-Something V2.

Kinetics-400 is a dataset collected from the YouTube video containing 400 action classes, with each class featuring at least 400 clips of around 10 s. Kinetics-400 presents a significant challenge due to background clutter, complex motion tempos, and long-range or short-range temporal dependencies. Figure 9 shows sampled frames from three action categories within Kinetics-400.

4.1.2. Experiment Setup

In the training stage, we adopt TSM as the baseline network, utilizing ResNet50 pre-trained on ImageNet [34] as the backbone. The same data augmentation strategy used in TSM is adopted. Each video is sampled with 8 or 16 frames as input. Training is performed on 8 NVIDIA GeForce RTX 3090 GPUs using distributed learning, with a per-GPU batch size of 16. The server includes an Intel Xeon CPU, 128 GB of RAM, and SSD storage for fast data loading. The experimental software includes Python 3.9, PyTorch 1.10, and CUDA 11.1, running on Ubuntu 20.04. Optimization is conducted via stochastic gradient descent (SGD) with a momentum of 0.9 and a dropout rate of 0.5. For the Something-Something V1 and V2 datasets, the initial learning rate is set to 0.02 with a weight decay of 5 × 10⁻⁴. Training spans 60 epochs, with the learning rate reduced by a factor of 10 at epochs 30, 40, and 45. For Kinetics-400, the initial learning rate is set to 0.01 with a weight decay of 5 × 10⁻⁴. Training extends to 100 epochs, with the learning rate decreasing by a factor of 10 at epochs 50, 70, and 90.

In the evaluation stage, the experimental software and hardware conditions remain consistent with the training stage. For the Something-Something V1 and V2 datasets, we sample a single clip with T frames from each video as input. Frames are uniformly sampled with a center crop of size 224 × 224 (center crop × 1 clip). For the Kinetics-400 dataset, 10 clips are sampled with T frames from each video. Each frame is uniformly sampled using three crops of 256 × 256 (three crops × 10 clips).

4.2. Ablation Study

Ablation study is a commonly used analysis method in the field of artificial intelligence. Its goal is to analyze the contribution of different parts of a model to its overall performance by gradually removing or adding certain components or functionalities of the model. This section presents ablation experiments and analysis on the Something-Something V1 dataset to assess the effectiveness of each module. This paper primarily uses Top-1 (%) and Top-5 (%) as model performance evaluation metrics, which are widely used in human action-recognition research [7,11,35]. Among them, the human action-recognition model classifies input samples and ranks multiple predicted results in descending order based on probability. Top-1 (%) refers to the percentage of test samples where the predicted category ranked first matches the actual category. Top-5 (%) represents the percentage of test samples where the actual category appears within the top five predicted rankings.

4.2.1. The Ablation of FAGM and MgAFM

As illustrated in

Table 1. Performance comparison with different modules.

FAGM	MgAFM	Top-1 (%)	$\Delta \mathbf{Top}-1(\%)$
		45.6	Baseline (TSM)
✓		49.6	+4.0
	✓	51.3	+5.7
✓	✓	52.4	+6.8

, to more directly demonstrate the necessity of each module, we use $\Delta \mathrm{Top}‐1$ (%) to represent the impact of adding our proposed modules to the baseline model on action-recognition accuracy. Incorporating the FAGM into the baseline yields a 4.0% improvement in Top-1 accuracy. Adding the MgAFM to TSM baseline [7] results in a 5.7% increase in Top-1 accuracy, as MgAFM effectively captures sub-action motion information across varying tempos, thereby improving the model’s capacity to recognize essential sub-action details. When both FAGM and MgAFM are integrated into the baseline, Top-1 accuracy rises by 6.8%, indicating that the two modules complement each other and work synergistically to boost video action-recognition performance.

4.2.2. The Impact of Motion Tempos at Different Temporal Granularities

The MgAFM improves the model’s ability to model motion information by integrating sub-action motion features with different tempos into the original input features. To verify which granularities of motion features contribute to improving model performance, we conducted ablation experiments by incrementally adding motion features with different frame intervals to the model. As shown in

Table 2. The impact of motion features at different temporal granularities on the accuracy of video action recognition.

Branch	Temporal Granularities	Top-1 (%)
1	0	51.5
2	0, 1	51.7
3	0, 1, 2	52.3
4	0, 1, 2, 3	52.4
5	0, 1, 2, 3, 4	52.3
6	0, 1, 2, 3, 4, 5	52.1
7	0, 1, 2, 3, 4, 5, 6	51.9

, when only the fast-tempo motion features extracted from adjacent frames are added to the original input, the Top-1 accuracy is merely 51.5%. As the types of motion features increase, the action-recognition accuracy gradually improves. When motion features with four granularities (frame intervals 0, 1, 2, and 3) are added to the original input, the Top-1 accuracy reaches a maximum of 52.4%.

When the motion features with frame intervals of 4, 5, and 6 are further added to the original RGB features, the model’s performance begins to decline. We analyzed that the correlation of larger temporal granularity cannot provide useful motion change information for subsequent representation learning. Using such motion features as input can negatively impact the model’s recognition performance.

4.2.3. The Impact of Feedback Iteration Times

As illustrated in

Table 3. The impact of feedback iteration times for video action-recognition accuracy.

Iteration Times	Top-1 (%)
0	51.3
1	51.8
2	52.1
3	52.4
4	52.2
5	51.6
6	49.9
7	49.0

, setting the times of FAGM iterations to 1 improves Top-1 accuracy by 0.5% compared to the baseline. A peak Top-1 accuracy of 52.4% is achieved with 3rd iterations. However, increasing the iterations further results in accuracy drops. We speculate that as the time of iterations grows, the computational steps increase, potentially causing a decrease in gradient values during training (similar to the “vanishing gradient” issue), which could limit the effectiveness of attentional guidance and reduce model performance.

4.3. Comparison with the State of the Art

Utilizing ImageNet for pre-training has been shown to speed up model convergence on new datasets [36], significantly lowering computational costs compared to training from scratch. Most video action-recognition models with -50 backbones pre-trained on ImageNet. To achieve a fair comparison with existing methods, we also use ImageNet for pre-training.

4.3.1. Something-Something V1

Table 4. Performance comparison on Something-Something V1, where “-” denotes a result that is not given in the original references.

Method	Backbone	Pre-Trained	Frame × Clip × Scrops	Params	Top-1 (%)	Top-5 (%)
3DCNN:
ECOEnLite [37]	BNIncep+3D ResNet18	Kinetics-400	92 × 1 × 1	150 M	46.4	-
I3D from [9]	ResNet50	ImageNet+ Kinetics-400	32 × 2 × 1	28.0 M	41.6	72.2
NL-I3D from [9]	ResNet50	ImageNet+ Kinetics-400	32 × 2 × 1	35.3 M	44.4	76.0
NL-I3D + GCN [9]	ResNet50	ImageNet+ Kinetics-400	32 × 2 × 1	62.2 M	46.1	76.8
3D DenseNet-121 [10]	3D DenseNet-121	ImageNet	16 × 1 × 1	21.4 M	50.2	78.9
2DCNN:
TSN [15]	BNInception	ImageNet	8 × 1 × 1	10.7 M	19.5	-
TSM [7]	ResNet50	ImageNet	8 × 1 × 1	24.3 M	45.6	74.2
TSM [7]	ResNet50	ImageNet	16 × 1 × 1	24.3 M	47.2	77.1
TANet [39]	ResNet50	ImageNet	16 × 1 × 1	25.6 M	47.6	77.7
TCM [11]	ResNet50	ImageNet	8 × 1 × 1	24.5 M	52.2	80.4
MSNet [16]	ResNet50	ImageNet	8 × 1 × 1	24.6 M	50.9	80.3
AGPN [30]	ResNet50	ImageNet	8 × 1 × 1	27.6 M	51.6	80.9
TEA [17]	ResNet50	ImageNet	16 × 1 × 1	24.5 M	51.9	80.3
MDAF [35]	ResNet50	ImageNet	8 × 1 × 1	24.5 M	49.1	78.0
MT-Net [38]	ResNet50	ImageNet	16 × 1 × 1	-	53.4	81.3
SMEN (ours)	ResNet50	ImageNet	8 × 1 × 1	26.1 M	52.4	80.6
SMEN (ours)	ResNet50	ImageNet	16 × 1 × 1	26.1 M	54.4	82.7

shows our results on the Something-Something V1 datasets. The table is divided into three parts, where the top section lists methods based on 3D CNN [9,10,37] the middle section lists methods based on 2D CNN [7,11,15,16,17,30,35,38,39] and the proposed SMEN-based method is presented in the bottom section. The “Params” in the table represents the number of learnable parameters in each model, mainly including the weights of the convolution kernels and the bias terms. The larger the number of parameters, the more memory the model requires.

Compared to 3D CNN-based methods [9,37], our model has fewer parameters and achieves higher accuracy in video action-recognition tasks. Compared to 2D CNN-based methods [7,11,15,16,17,30,39], our SMEN achieves top-1 accuracies of 52.4% and 54.4% when using 8 or 16 video frames as input, respectively, showing improvements of 6.8% (52.4% vs. 45.6%) and 7.2% (54.4% vs. 47.2%) over the TSM baseline. Furthermore, even in comparison with recent 2D CNN-based methods [35,38], our model continues to show competitive performance improvement for video action-recognition tasks.

Especially, on one hand, compared to feedforward attention-based methods [17,39], our model effectively exploits contextual information from high-level features through the proposed FAGM attention mechanism. The contextual semantic information is leveraged to guide the low-level feature learning significant sub-action information, leading to higher recognition accuracy. On the other hand, compared to methods that capture sub-action information [11,16,30], our model can adaptively fuse sub-actions with different motion tempos, focusing more on key sub-action information.

4.3.2. Something-Something V2

To evaluate the robustness of the proposed method, we test it on the Something-Something V2 datasets.

Table 5. Performance comparison on Something-Something V2, where “-” denotes a result that is not given in the original references.

Method	Backbone	Pre-Trained	Frame × Clips × Crops	Params	Top-1 (%)	Top-5 (%)
3DCNN:
3D DenseNet-121 [10]	3D DenseNet121	ImageNet	16 × 1 × 1	21.4 M	62.9	88.0
CIDC [41]	ResNet50	ImageNet	32 × 1 × 1	87 M	56.3	83.7
RubiksNet [42]	ResNet50	ImageNet	8 × 1 × 1	-	58.8	85.6
2DCNN:
TSN [15]	BNInception	ImageNet	8 × 1 × 1	10.7 M	33.4	-
TSM [7]	ResNet50	ImageNet	8 × 1 × 1	24.3 M	58.8	85.4
TSM [7]	ResNet50	ImageNet	16 × 1 × 1	24.3 M	63.4	88.5
STM [12]	ResNet50	ImageNet	16 × 1 × 1	35.3 M	64.2	89.8
MSNet [16]	ResNet50	ImageNet	8 × 1 × 1	24.6 M	63.0	88.4
AGPN [30]	ResNet50	ImageNet	8 × 1 × 1	27.6 M	63.1	88.6
ACTION-Net [18]	ResNet50	ImageNet	16 × 1 × 1	28.1 M	64.0	89.3
GSFNet [43]	ResNet50	ImageNet	8 × 1 × 1	24.0 M	62.1	-
MDAF [35]	ResNet50	ImageNet	8 × 1 × 1	24.5 M	62.3	87.8
TEINet [40]	ResNet50	ImageNet	16 × 1 × 1	30.4 M	61.3	-
SMEN (ours)	ResNet50	ImageNet	8 × 1 × 1	26.1 M	63.3	88.8
SMEN (ours)	ResNet50	ImageNet	16 × 1 × 1	26.1 M	65.0	89.5

compares our results with 2D CNN and 3D CNN-based methods. As shown in the Table 5, when using 8 or 16 video frames as input, our SMEN reports top-1 accuracies of 63.3% and 65.0% respectively, achieving improvements of 4.5% (63.3% vs. 58.8%) and 1.6% (65.0% vs. 63.4%) over the TSM baseline. Compared to feedforward attention-based methods [12,18,40] and recent 2D CNN-based methods [30,35], the proposed SMEN outperforms them in video action-recognition tasks.

4.3.3. Kinetics-400

Kinetics-400 is a larger dataset with diverse appearances and backgrounds, compared to Something-Something V1 and V2. To assess the proposed method’s effectiveness in modeling appearance features and interactions, we take performance comparisons on the Kinetics-400 dataset, as shown in

Table 6. Performance comparison with the state of the art on the Kinetics-400 dataset. The best results are highlighted. “N/A” and “-” denote a result that is not given in the original references.

Method	Backbone	Pre-Trained	Frame × Clip × Crops	Params	Top-1 (%)	Top-5 (%)
3DCNN:
ECO [37]	BNIncep+3D-ResNet-18	Scratch	92 × 1 × 1	150 M	70.0	89.4
I3D [9]	Inception	ImageNet	64 × N/A × N/A	28.0 M	72.1	90.3
SlowFast [29]	ResNet50	Scratch	(4 + 32) × 1 × 3	-	75.6	92.1
2DCNN:
TSN [15]	BNInception	ImageNet	25 × 1 × 10	10.7 M	72.5	90.2
TSM [7]	ResNet50	ImageNet	8 × 10 × 3	24.3 M	74.1	91.2
TCM [11]	ResNet50	ImageNet	8 × 10 × 3	24.5M	76.1	92.3
STM [12]	ResNet50	ImageNet	16 × 10 × 3	24.0 M	73.7	91.6
MSNet [16]	ResNet50	ImageNet	8 × 10 × 3	24.5 M	75.0	-
GSFNet [43]	ResNet50	ImageNet	8 × 10 × 3	24.0M	74.8	-
MDAF [35]	ResNet50	ImageNet	8 × 10 × 3	24.5 M	76.2	92.0
TEINet [40]	ResNet50	ImageNet	8 × 10 × 3	30.4 M	74.9	91.9
MVFNet [44]	ResNet50	ImageNet	8 × 10 × 3	24.3 M	76.0	92.4
GTDNet [45]	ResNet50	ImageNet	8 × 10 × 3	33.9 M	75.5	92.5
GTDNet [45]	ResNet50	ImageNet	16 × 10 × 3	33.9 M	76.8	92.3
SMEN (ours)	ResNet50	ImageNet	8 × 10 × 3	26.1 M	76.9	92.8
SMEN (ours)	ResNet50	ImageNet	16 × 10 × 3	26.1 M	77.9	93.3

.

Compared to the 3D CNN-based method SlowFast [29], our SMEN model also obtains remarkable results in the Top-5 accuracy (93.3% vs. 92.1%) and its Top-1 accuracy is higher (76.9% vs. 75.6%) since our model can acquire more granular motion features. Compared to TSM, our method achieves a Top-1 accuracy of 76.9%, an improvement of 2.8%. For other 2D CNN-based methods, SMEN provides further performance gains with only a slight increase in model parameters, demonstrating its efficiency and effectiveness in appearance-based action datasets. Furthermore, our SMEN model obtains competitive performances on the Kinetics-400 datasets. Specifically, its Top-1 and Top-5 accuracy outperforms the state-of-the-art 2D CNN-based method MDAF [35] by 0.7% (76.9% vs. 76.2%) and 0.8% (92.8% vs. 92.0%), respectively. All these results highlight that our SMEN is more effective and accurate than the feedforward network in modeling the sub-action motion tempos and representation learning for video action-recognition classification.

4.4. Visualization

To further understand the working mechanism of the proposed SMEN, Grad-CAM [46] is applied to visualize activation maps and the most discriminative feature locations on the Kinetics-400 dataset. As presented in Figure 10, the feature visualization results of TSM [7] and our SMEN are illustrated for the actions “Triple jump”, “Long jump”, and “High jump”. In this activation map, the redder the area, the stronger the model’s attention to that region. The input frames are sampled at equal intervals and cropped to 256 × 256.

In Figure 10, for the first action category “Triple jump”, it can be observed that the TSM method focuses more on the “take-off line” rather than the motion tempos of human motion. In contrast, our method keeps its attention on human motion dynamics and tempos. Our method effectively focuses attention on the temporal information of the sub-actions “Hop”, “Skip”, and “Jump”, which are more critical for classifying human actions. On the other hand, for the second action category “Long jump”, it can be observed that the TSM method pays more attention to the “Take-off line” from frames T0 to T7, rather than to the motion tempos in human motion. In contrast to TSM, our method effectively captures part of the information related to the key sub-actions “Action in the air” and the fast-paced action “Take-off”. For the third action category “High jump”, from frames T3 to T7, it can be observed that TSM places more attention on the “High jump bar” rather than the various motion tempos of action. Our method focuses in a balanced manner on both action motion tempos and the action-specific contextual information “High jump bar”, which can be observed from the brighter blue areas compared to the background in the class activation map.

Besides, we employed Grad-CAM [46] to visualize class activation maps on the Something-Something V2 dataset. As presented in Figure 11, the feature visualization results of TSM [7] and our SMEN are illustrated on the Something-Something V2 dataset for the action categories “Covering something with something” and “Holding something over something”. In these visualizations, we sampled eight frames from each video as input. Since the Something-Something V2 dataset primarily consists of short videos with durations of about 3 seconds, the tempo of sub-action motions within the videos changes rapidly.

The visualization results of Figure 11 show that, for the first action category, compared to TSM, the proposed SMEN captures the motion tempos and the temporal changes of sub-action “Picked up” more promptly. Furthermore, for the subsequent sub-action “Covering” the cup with a towel, our model captures the entire “Covering” tempo more accurately. For the second action category, SMEN outperforms TSM by more effectively capturing the sub-actions “Placing one object on another” and “Removing the arm” while focusing more attention on these dynamic sub-actions. Our method avoids the allocation of resources to static information that has less impact on video action classification.

Ablation studies and results confirm the effectiveness of modules in the proposed SMEN method. The performance comparisons demonstrate the effectiveness of the proposed SMEN, due to the model’s capacity to capture and represent key sub-action information with different motion tempos more effectively. Additionally, the visualization analysis offers a more intuitive insight into the working mechanism of SMEN.

5. Conclusions

In this paper, to explore the potential of the human action-recognition model for capturing and representing vital sub-action information with different motion tempos, we propose a Sub-action Motion information Enhancement Network (SMEN) based on motion tempo learning and feedback attention, comprising two key components: the Multi-Granularity Adaptive Fusion Module (MgAFM) and the Feedback Attention-Guided Module (FAGM). The MgAFM captures the intrinsic information of sub-actions with motion tempos at different granularities and adaptively fuses it with the original RGB features. The FAGM leverages global contextual information to guide low-level feature learning with a feedback mechanism, focusing on vital spatio-temporal and channel-wise information. Extensive experiments on three public datasets, ablation studies, and visualization analyses demonstrate that the proposed method outperforms compared methods, especially for videos containing sub-actions with different motion tempos.

The proposed SMEN focuses less on the contextual background information of human actions and the object categories involved in interactive actions, which is useful for complex human activities. The future plan is to optimize motion-extraction methods to better capture the contextual background of actions or explore the generalization of SMEN using heatmaps [47], which are converted from human skeleton data for skeleton-based human action recognition.