Human Action Recognition Based on 3D Convolution and Multi-Attention Transformer

Abstract

To address the limitations of traditional two-stream networks, such as inadequate spatiotemporal information fusion, limited feature diversity, and insufficient accuracy, we propose an improved two-stream network for human action recognition based on multi-scale attention Transformer and 3D convolutional (C3D) fusion. In the temporal stream, the traditional 2D convolutional is replaced with a C3D network to effectively capture temporal dynamics and spatial features. In the spatial stream, a multi-scale convolutional Transformer encoder is introduced to extract features. Leveraging the multi-scale attention mechanism, the model captures and enhances features at various scales, which are then adaptively fused using a weighted strategy to improve feature representation. Furthermore, through extensive experiments on feature fusion methods, the optimal fusion strategy for the two-stream network is identified. Experimental results on benchmark datasets such as UCF101 and HMDB51 demonstrate that the proposed model achieves superior performance in action recognition tasks.

1. Introduction

Human action recognition involves extracting and analyzing human movements and behavioral features from video sequences to determine the specific categories of actions being performed. This technology is essential for enabling machines to interpret human behavior and serves as a cornerstone in the field of computer vision. It has significant importance and broad application potential across various domains, including intelligent security, virtual reality, human–computer interaction, and intelligent caregiving, where it can deliver substantial economic and social benefits.

Currently, deep learning-based human action recognition methods have attracted a great deal of attention and have achieved numerous research results. Depending on the different network architectures used, deep learning-based human action recognition methods can be categorized into four types: 3D convolutional(3D CNN) [1,2,3], two-stage network [4,5], Transformer model [6,7], and graph convolutional (GCN) [8,9].

Three-dimensional convolution has the advantage of simultaneously modeling spatio-temporal information and has been widely used in action recognition tasks. For example, Duan [1] emphasized that 3D CNNs can better integrate with multiple modalities such as RGB and optical flow information, thus improving the recognition results. Ou [2] proposed a 3D deformable convolutional reasoning network (DCTR), which is used to model and reason about the dependency relationships among multiple entities in the spatiotemporal dimensions to enhance the recognition performance. Although 3D CNN-based methods can model spatio-temporal features simultaneously, there is still room for improvement in handling long-term dependencies.

Two-stream networks extract spatiotemporal information through two separate branches, enabling richer feature representations and more precise extraction of motion information, leading to higher recognition accuracy. For example, Pang [4] improved recognition performance via contrastive learning in the two-stream network. Xie [5] designed a multimodal two-stream fusion network to enhance the saliency of action-related characteristics. Although two-stream networks can extract spatiotemporal features, the interaction and fusion strategies of the spatial and temporal streams, as well as the representation and learning methods of temporal information, still need improvement. Additionally, most methods using 2D CNN cannot effectively capture the temporal information between video frames.

In recent years, Transformers have been introduced into the field of video understanding. The application of the Transformer model provides a new paradigm for action recognition. Its powerful long-range dependency modeling, multimodal fusion ability, and good scalability make it particularly suitable for handling long sequence data and complex temporal contextual relationships in videos. For example, Zhao [6] proposed a spatiotemporal two-stream multi-scale Transformer (STDM Transformer) model for action recognition. Wu [7] proposed a Transformer-based multi-view spatiotemporal feature interaction fusion framework, which achieves effective action recognition through deep fusion of multi-view information. Although the performance of action recognition based on Transformer has greatly improved, Transformer models still face challenges such as large data requirements and insufficient local information capture.

The method based on a GCN can leverage the topological structure of skeletal data to capture the relationships between different joint points, providing a more natural representation of spatiotemporal features. For example, Xie [8] proposed a dynamic semantic-based graph convolutional network (DS-GCN) action recognition method. Liu [9] introduced an action recognition method based on graph prototype learning (ProtoGCN). Although graph convolutional networks (GCNs) perform well in modeling human skeleton sequences and joint relationships, they face challenges in fusing spatiotemporal features and handling long-term temporal data.

In summary, to fully leverage the advantages of various networks, we propose a two-stream network model for human action recognition based on a multi-scale attention Transformer and 3D convolution (MAT-C3D). By constructing a temporal network based on 3D convolution to capture the temporal dependencies between video frames, it can better understand the continuous changes of actions. By constructing a spatial network based on a multi-scale Transformer model, it can capture long-term dependencies and generate receptive fields of different sizes for feature fusion. Moreover, an adaptive weighting mechanism is employed for feature fusion, which enriches the diversity of features and better adapts to long-term, multi-scale action recognition tasks in complex environments.

The main work and contributions are summarized as follows:

1.

A brand-new two-stream network structure containing a C3D temporal network and a multi-scale Transformer spatial network has been constructed, which fully utilizes the correlation between spatial and temporal information to extract spatiotemporal features with interactive relationships, effectively improving the action recognition ability with long-term dependencies.

2.

A multi-scale convolution module is introduced to replace the original FFN module in the Transformer; after inputting context-based relative position encoding, adaptive integrated multi-scale features with stronger representation capabilities are obtained, effectively improving the recognition accuracy of the model in complex environments.

3.

A more efficient feature fusion strategy has been designed, and the spatiotemporal feature fusion method of the two-stream network has been improved by using weighted fusion, which enhances the model’s ability to model complex spatiotemporal interaction relationships and further improves the recognition accuracy in complex environments.

4.

Experimental results on the datasets of UCF101 and HMDB51 demonstrate that our algorithm is a highly accurate human action recognition model.

The rest of this paper is organized as follows: Section 2 reviews related work in action recognition, including 3D convolution, two-stream networks, and Transformer models, and discusses the limitations of existing methods. Section 3 introduces our proposed two-stream network of action recognition based on Transformer and C3D, including a C3D time-stream convolutional network, multi-scale convolution Transformer, positional encoding, and a multi-scale residual model. Section 4 discusses the feature fusion methods and identified the optimal fusion strategy. Section 5 describes the experimental setup, including the software and hardware environments and the used datasets, and presents the results of our experiments, including the main results, ablation studies, and an extended analysis on experimental results. Finally, Section 6 concludes the paper.

2. Related Work

2.1. Action Recognition Based on 3D Convolution

Compared to 2D convolution, 3D convolutional networks are more suitable for learning spatiotemporal features and have been widely used in the field of human action recognition. Tran [10] first used 3D convolutional video (Convolutional 3D, C3D), which streamlines the network structure and can process multiple frames at a time thus improving computational efficiency. Qiu [11] used a parallel serial residual connection method to construct a network based on a small convolutional kernel, which was optimized in terms of the running rate and the number of parameters, but suffered from long training time and easy overfitting in the case of uneven data distribution. Tran [12] decomposed the 3D convolutional filter into independent spatiotemporal components, which significantly improved the accuracy but caused the problems of easily overfitting with a large number of parameters and insufficient processing of global information in the video. Yang [13] proposed an integrated model that combined a 3D convolutional network (C3D), LSTM, and a spatiotemporal attention mechanism, effectively enhancing the recognition accuracy. The above methods directly use 3D networks to learn human action representations from videos. Due to the limitation of video frame length, the ability to learn spatial domain contour information and temporal domain motion information is limited, and a model cannot fully learn and capture the spatiotemporal dependencies during long-term motion processes. To fully leverage the advantages of 3D convolution, we embed 3D convolution into the time-stream branch of a two-stream network to better extract global contextual features.

2.2. Action Recognition Based on Two-Stream Networks

Feichtenhofer [14] proposed a SlowFast network for video behavior recognition using a low frame rate to capture semantics and a high frame rate to capture actions, which has higher accuracy with lower computational complexity, but the performance on small datasets is unstable. Li [15] used residual frames instead of optical flow to capture motion information, avoiding complex optical flow computation with simple structure and better results. However, it requires a lot of adjustments and experiments on model configuration and parameter optimization, which has certain limitations in processing long-term series. Li [16] proposed a Short-term Action Discriminant Attention (SADA) model based on a two-stream structure, which incorporates a fast–slow attention mechanism to emphasize the varying importance of short-term actions and enhance recognition performance. Zhou [17] introduced a highly efficient two-stream network utilizing multi-head attention for action recognition. This network captures key action information through an attention mechanism to differentiate similar actions in videos. However, these methods employ simple feature fusion approaches, which fail to fully integrate the interactive relationships between spatiotemporal features, overlooking the correlation between spatial and temporal information. To address these limitations, we improve the two-stream network by introducing multi-scale attention into the spatial stream and 3D convolutional fusion into the temporal stream, enriching the diversity of feature extraction. Furthermore, the recognition performance is significantly enhanced through an adaptive weighted feature fusion approach.

2.3. Action Recognition Based on Transformer Model

With the tremendous success of the Transformer model in Natural Language Processing, it has been widely applied in the field of action recognition [18,19,20,21,22,23,24,25,26,27,28,29]. For example, in order to model spatiotemporal features, Pareek [18] improved motion detection and enhanced understanding of human motion in adjacent frames by utilizing spatial and temporal attributes. Ma [19] proposed a relative position embedding (RPE-STDT) model based on spatial and temporal decoupling for action recognition. This model replaces absolute position embedding with relative position embedding, effectively captures token sequences, reduces computational costs, and optimizes recognition performance. Qiu [20] proposed the STTFormer network, which utilizes a spatiotemporal tuple self-attention mechanism module to capture the relationships between different joints in consecutive frames. Do [21] proposed the SkateFormer network, which partitions joints and skeletons based on different types of skeletal–temporal relationships, and performs a self-attention mechanism within each partition, alleviating the requirement of a large amount of resources to obtain correlations between joints. Liu [22] proposed the TranSkeleton action recognition method that can effectively capture remote dependencies and subtle time structures. In addition, Sun [23] used k-NN attention instead of the original self-attention, optimizing the training process by ignoring irrelevant or noisy labels in the input sequence, achieving efficient recognition. This improved the attention mechanism to make the original Transformer model more suitable for action recognition tasks. Zhang [24] proposed ActionFormer, a single-stage temporal action location model based on anchor-free. It uses a lightweight decoder to classify actions at every moment and estimate corresponding action boundaries. Zheng [25] employed the attention mechanism and adopted a two-stage training strategy for mutual enhancement of the two motions during training. Yang [26] proposed the Recurrent Vision Transformer (RViT) method for spatiotemporal feature learning to perform video behavior recognition. However, the above methods are all based on a single scale for action recognition, ignoring that the feature scale changes during the recognition process, resulting in uneven attention allocation and limited recognition performance. Therefore, we introduce a multi-scale Transformer model under the two-stream network to model spatial features. Compared with single-scale methods, the attention allocation is more balanced and accurate, and can better identify actions of different granularities.

3. Transformer and C3D-Based Two-Stream Network Modeling

The original two-stream convolutional network utilizes two parallel networks: one branch learns the spatial dimension of static features such as appearance and background, and its input is the RGB image of a single video image sequence frame; the other branch uses a convolutional neural network of the same structure to process multiple frames of the optical streaming image to extract the dynamic features in the time dimension. The final prediction of the video is obtained by fusing the results of the two branches. The same two-dimensional CNN is used for both temporal and spatial dimensions, and the two-dimensional feature maps are outputted after convolution, which leads to some limitations in the temporal information and insufficient model accuracy. In this paper, three-dimensional convolution is introduced in the temporal flow network and the timing problem is optimized using the C3D network model. In the spatial flow network, a multi-scale convolutional Transformer is used to generate feature representations using residual-connected multi-head self-attention to further enhance the feature representation capability of the model. The optimal network architecture is finally obtained by testing different spatiotemporal feature fusions. The overall network architecture of this paper is shown in Figure 1.

3.1. C3D Time-Stream Convolutional Neural Network

Two-dimensional convolutional neural networks work well for feature extraction on single frames but do not capture the temporal information well when faced with video-based tasks, hence three-dimensional convolutional neural networks are proposed. Compared to 2D convolutional neural networks, 3D convolutional neural networks can model temporal information better. The C3D convolutional structure is shown in Figure 2.

Figure 2a shows the application of a two-dimensional convolutional network to a video image sequence frame, and after the convolution operation, the output is a two-dimensional feature map, which loses the time sequence information. Figure 2b shows that the three-dimensional convolutional network is subjected to a convolution operation, and the output is a three-dimensional feature map that retains the time information of the video image sequence frame. Using the C3D network for feature extraction in the temporal dimension enables better extraction of temporal features of continuous optical flow images. The C3D network structure is shown in Figure 2c, which employs five convolutional layers, five pooling layers, two fully connected layers, and a softmax classifier. All layers use 3 × 3 × 3 convolutional kernels; the first pooling kernel is a 1 × 2 × 2 structure and the rest of the pooling kernels are 2 × 2 × 2. The C3D network model has a simple structure and trains the model quickly for better extraction of temporal features.

3.2. Multi-Scale Convolution Transformer

We improve the Transformer model and apply it to feature extraction on the spatial dimension of dual-stream networks. The encoder module for feature extraction consists of two sublayers, multi-head self-attention and adaptive-scale convolutional neural network, both of which have a residual join followed by layer normalization. The overall structure is shown in Figure 3.

The multihead self-attention layer generates a hidden representation from the input using the multi-head self-attention mechanism, which can directly integrate the information in the entire input sequence, overcoming the long-time dependency problem of RNN and LSTM. Assuming the input sequence is X, $X\in R^{(L\times d)}$, using the scaled dot product attention mechanism, the query vector Q, key vector K, and value vector V are generated by linear projection:

$$Q=X{W}^Q;K=X{W}^K;V=X{W}^V$$

(1)

where $W^Q\in R^{(d\times d_k)},W^K\in R^{(d\times d_k)},W^V\in R^{(d\times d_v)}$ are the projection parameters and the weight matrix. Compare the query and the key vectors of dimension $d_k$. The input is a vector of values, where the weight assigned to each value is computed from the dot product of the query and the corresponding key, and the output is a weighted sum of the following values:

$$Attention(Q,K,V)=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

(2)

Using $\sqrt{d_k}$ instead of $d_k$ prevents the inner product from becoming too large while smoothing out the scaling factor. Multi-head self-attention uses multiple self-attention modules to compute in parallel, combining the outputs of all the attention heads into a final result. The input vector X is passed to each of the h different self-attention modules, and the output matrix is stitched together and passed into the linear layer to obtain the final output:

$$MultiHead(Q,K,V)=Concat(〖head{〗}_1,\cdots ,〖head{〗}_h)W^O$$

(3)

$$hea{d}_i=Attention(X{W}_i^Q,X{W}_i^K,X{W}_i^V)$$

(4)

where $W^O\in R^{(〖hd{〗}_v\times d)}$ is the final projection parameter. To facilitate residuals connection, all sublayers use the same dimension as the input layer, i.e., $d_k=d_v=d/h$. In the traditional Transformer, the output of the multi-head self-attention layer is sent to the feed-forward neural network module, which consists of two fully connected layers. In this paper, a multi-scale convolutional module is used to replace the feed-forward neural network module, which captures region feature information at different scales by using convolutional kernels at different scales and an adaptive scale-attention mechanism to adjust between scales. Given a set of kernel size $K=k_1,\cdots ,〖k{〗}_n,P_i$ is the eigenmatrices with ith kernel $k_i$.

$$P_i=ReLU\left(Dropout\left(BN\left(Conv(W^{k_i};X)\right)\right)\right)$$

(5)

where X is the output feature of the multi-head self-attention module; $W^{k_i}\in R(d\times k_i\times d)$ is a filter with a specific receptive field, consisting of d filters, each of size $k_i\times d$. The module is computed by four operations: convolution (Conv), batch normalization (BN), dropout, and ReLU activation unit. The core is the Conv module. Fill zeros on both sides of feature X with $(k_i-1)/2$ so that the generated feature map is $L\times d$ in size. Then, the output of multiple convolution blocks is $P=P_1,\cdots ,P_n$. The output of all feature vectors containing features with problems such as incomplete information, redundant information, etc., are integrated into the feature vector using an FFN adaptive-scale attention mechanism to adaptively integrate features at different scales. The integrated features $P_{ens}$ are defined as follows:

$$P_{ens}=\alpha P=\sum 〖{\alpha }_i{P}_i〗$$

(6)

where ${\alpha }_i$ is the attention score of $P_i$ on the scale $k_i,\alpha =({\alpha }_1,\cdots ,〖\alpha {〗}_i)$ is the vector of attention scores, and $\alpha$ is computed by

$$\alpha =softmax\left(FFN\right(P\left)\right)$$

(7)

where $FFN(·)$ is a two-layer fully connected feed-forward neural network:

$$FFN\left(P_i\right)=ReLU(P_i{W}_1+b_1)W_2+b_2$$

(8)

where $W_1\in d\times d_h$ and $W_2\in d_h\times 1$ are matrices of parameters with hidden dimensions $d_h$, and the parameter matrices of $b_1$ and $b_2$ are the biases. An example of using a convolution kernel with scales 1, 3, and 7 to generate three feature vectors is given in Figure 4, in which the three single-feature vectors are integrated as the final integrated feature through an adaptive-scale attention mechanism.

3.3. Positional Encoding

Since the Transformer operates in parallel with the input data, resulting in the loss of information about the position of the image blocks, some information about the position of the sequence needs to be injected, so positional encoding is added to the bottom layer of the encoder. The original Transformer model used absolute conditional encoding, where only the absolute position of the sequence was input. Relative position coding, in addition to the absolute position of the input sequences, encodes the relative distance between the input sequences and learns pairwise relationships between the sequences. In this paper, context-based relative position coding is used in the attention mechanism to improve the expressiveness of the model [13]. Suppose that the input sequence is $X=(X_1,\cdots ,X_n)$, and the output sequence is $Z=(Z_1,\cdots ,Z_n)$. Each output element $Z_i$ represents the relationship of the current position i to all positions in the sequence, which is the result of linear weighting. The calculation is published as follows:

$$Z_i=\sum _{j=1}^n{\beta }_{ij}\left(X_j{W}^V\right)$$

(9)

where ${\beta }_{ij}$ is the weight. This is obtained by the softmax function

$${\beta }_{ij}=softmax\left(e_{ij}\right)={\displaystyle \frac{exp\left(e_{ij}\right)}{{\sum }_{k=1}^{n}exp\left(e_{ik}\right)}}$$

(10)

where $e_{ij}$ is obtained by scaling the dot product of the two elements:

$$e_{ij}={\displaystyle \frac{\left(X_i{W}^Q\right){\left(X_j{W}^K\right)}^T}{\sqrt{d_k}}}$$

(11)

Context-based relative positional encoding takes into account the interactions with queries, keys, and values; the core idea is to add learnable relative positional encoding to the $e_{ij}$, incorporating learnable relative position encoding in the following:

$$e_{ij}={\displaystyle \frac{\left(X_i{W}^Q\right){\left(X_j{W}^K\right)}^T+b_{ij}}{\sqrt{d_k}}}$$

(12)

$$b_{ij}=\left(X_i{W}^Q\right)r_{ij}^T$$

(13)

where $r_{ij}^T$ is a trainable vector. To calculate the relative position on the 2D image and define the relative weight $r_{ij}$, it is mapped directionally using the crossover method, where the position encoding in the horizontal and vertical directions are computed separately and combined as follows:

$$r_{ij}=p_{I^x(i,j)}^x+p_{I^y(i,j)}^y$$

(14)

$$I^x(i,j)=g(x_i-x_j)$$

(15)

$$I^y(i,j)=g(y_i-y_j)$$

(16)

where $p_{I^x(i,j)}^x$ and $p_{I^y(i,j)}^y$ are learnable vectors that store the weights of relative positions, and $g(·)$ is a many-to-one segmented indexing function that maps relative positions to weights, mapping multiple relative positions to the same position encoding, reducing the number of parameters introduced by the relative positions.

3.4. Multi-Scale Residual Model

The multi-scale residual model consists of two parts: residual connection and layer normalization. In order to ensure better flow of multi-scale information, we improve the residual module of the Transformer to process outputs from multi-head attention and multi-scale convolution modules, respectively. The residual connections are calculated as follows:

$$Z_{mha}=X+MHA\left(X\right)$$

(17)

$$Z_{msa}=X+MSA\left(X\right)$$

(18)

where X is the input of multi-head attention layer or multi-scale convolution module, $MHA\left(X\right)$ is the output of multi-head attention layer, $MSA\left(X\right)$ is the output of multi-scale convolution module, and $Z_{mha}$, $Z_{msa}$ is the output of residual connection. Each small batch of data is normalized by layer normalization, so that the distribution characteristics of each dimension are the same, so as to improve the stability and learning speed of the network. The formula for layer normalization is as follows:

$$LayerNorm\left(Z_{mha}\right)$$

(19)

$$LayerNorm\left(Z_{msa}\right)$$

(20)

The output of residual connection is normalized by Formulas (19) and (20) to obtain the final output. The operational visualization of residual connection and layer normalization is shown in Figure 5.

4. Feature Fusion

The original two-stream network judges the features of temporal and spatial streams separately after obtaining them and obtains the final result by both averaging and SVM. This method does not consider the connection between the information of temporal and spatial flow features, and the recognition accuracy is not high. In this paper, multiple feature fusion methods are used to compare the advantages of different feature fusion strategies. The accuracy of using different fusion methods is compared through multiple sets of experiments and the best feature fusion method is determined after experimental comparison.

4.1. Summing Fusion Approach

Firstly, calculate the sum of pixel values of the corresponding pixels of two feature maps with the same dimension $(i,j)$ and the sum of the pixel values of the two feature maps as the fusion result.

$$y_{i,j,d}^{sum}=z_{i,j,d}^s+z_{i,j,d}^T$$

(21)

Among them, $z_{i,j,d}^s$, $z_{i,j,d}^T$ denote, respectively, the spatial and temporal features with channel number d at pixel point $(i,j)$ and the pixel values of the spatial and temporal features at the pixel point; $y_{i,j,d}^{sum}$ denotes the result of feature fusion by the summation fusion method.

4.2. Average Fusion Approach

Calculate the average pixel value of the corresponding pixel points of two feature maps with the same dimension $(i,j)$ with the average pixel value as the fusion result.

$$y_{i,j,d}^{ave}={\displaystyle \frac{z_{i,j,d}^s+z_{i,j,d}^T}{2}}$$

(22)

where $y_{i,j,d}^{ave}$ denotes the result of feature fusion by averaging fusion.

4.3. Maximum Fusion Approach

The maximum pixel value of the corresponding pixel point of two feature maps with the same dimension is taken as the fusion result. $(i,j)$ of two feature maps with the same dimension as the maximum pixel value as the fusion result.

$$y_{i,j,d}^{max}=max\{z_{i,j,d}^s,z_{i,j,d}^T\}$$

(23)

where $y_{i,j,d}^{max}$ denotes the result of feature fusion by averaging fusion.

4.4. Weighted Fusion Approach

Two feature maps with the same dimensions are weighted at pixels $(i,j)$ at the pixel point for weighted fusion and take the output as the fusion result.

$$y_{i,j,d}^{we}=\sigma z_{i,j,d}^s+(1-\sigma )z_{i,j,d}^T$$

(24)

where $y_{i,j,d}^{we}$ denotes the result of feature fusion by weighted fusion approach, and $\sigma$ is the weighted number.

4.5. Fusion Mode Experiment

Make the size of RGB frames and optical flow maps reset to 224 × 224, feed the RGB frames into the pretrained spatial flow network to extract spatial features, stack the 16 optical flow maps and feed them into the optical flow network to extract temporal features, and output the final result after feature fusion. The results were fused by different fusion methods and the different fusion accuracies on the dataset UCF101 are shown in

Table 1. Accuracy of different fusion methods on UCF101.

Integration Approach	Recognition Accuracy (ACC)/%
Seek peace and integration	87.6
Average fusion	84.3
Maximum fusion	85.5
Weighted integration	88.6

.

It can be concluded that the weighted fusion approach is slightly higher than the rest of the fusion approaches with an accuracy of 88.6%. The following experiments will use the weighted fusion approach for feature fusion.

5. Experimental Results and Analyses

5.1. Datasets

We evaluate our approach on three widely used action recognition datasets: UCF101 [30], HMDB51 [31], and KTH [32]. To reduce the training difficulty of the model and enhance its performance, we employ a pretrained model on the Kinetics-400 [33] dataset.

UCF101: The UCF101 dataset is an expanded version of the UCF50 dataset, comprising over 13,000 videos sourced from YouTube and organized into 101 distinct action classes. This dataset offers a diverse range of actions captured from various angles, featuring complexities such as object appearance, varied viewpoints, camera motion, cluttered backgrounds, and different lighting conditions. The videos within each action class are divided into 25 groups, with each group containing 4 to 7 videos. The action classes can be broadly categorized into five types: human–object interaction, human–human interaction, body motion, sports, and playing musical instruments. The UCF101 dataset is partitioned into a training set with approximately 9500 videos and a test set with around 3700 videos.

HMDB51: The HMDB51 dataset comprises over 6000 videos, primarily sourced from internet movies. These videos are categorized into 51 action classes, with most actions representing everyday activities. Each action class contains at least 101 videos. The action classes can be broadly divided into two main categories: facial actions and body movements. All videos in the HMDB51 dataset are annotated with detailed information, including action class labels, video conditions, and metadata. The annotations specify the position of the body, the visibility of the body parts, and the number of objects involved in the actions. The HMDB51 dataset is divided into a training set with approximately 3500 videos and a test set with around 1500 videos.

KTH: The KTH action dataset, released by the Royal Institute of Technology (KTH) in Sweden in 2004, is a widely used video dataset for action recognition research. The dataset consists of 600 video sequences, each annotated with specific action categories and performer information. It includes six basic action categories, with each action performed by 25 different individuals across four distinct scenarios: outdoor, outdoor with scale variation, outdoor with different clothing, and indoor.

Kinetics-400: The Kinetics-400 dataset is a large-scale and meticulously labeled video dataset, encompassing 400 diverse action classes. It comprises 240,000 training videos, 40,000 testing videos, and 20,000 validation videos. Each action class contains over 600 video instances. The dataset contains a wide range of actions, including human–object interactions such as riding a bike and typing, as well as human–human interactions like shaking hands and salsa dancing.

5.2. Experimental Implementation Details

This experiment was conducted under the Ubuntu 20.04 platform using the MMaction2 repository and deep learning framework PyTorch2.1.1 via Python3.10 under NVIDIA GeForce RTX3090 GPU. The video RGB frames as well as the optical flow maps in the UCF101 and HMDB51 dataset are extracted by preprocessing through MATLAB R2024A, respectively. One RGB image is generated every 4 frames, and one optical flow image in the x and y directions is generated every 1 frame, which is stored in the corresponding directories, respectively. To enhance the model’s generalization capability, we employed data augmentation techniques on the dataset to enrich the diversity of the data. Specifically, we applied random cropping and randomly flipped the images horizontally with a probability of 0.5, which can improve the model performance by 1.3% on the dataset UCF101. Let the initial learning rate be 0.01 and the learning rate decay value be 0.001; the batch size is set to 16; and on the UCF101 and HMDB51 datasets, the epochs is set to 100 and 80, respectively. Training with a pretrained model takes about 4.5 h on UCF101. The training process on the UCF101 dataset is shown in Figure 6.

The results in Figure 6 are the training accuracy rate and loss value curves of the algorithm in this paper, respectively. As can be seen from the figure, with the increase in training times, the accuracy rate keeps increasing, the training loss keeps decreasing, and the recognition result gradually converges after 60 epochs, that is, the recognition result gradually approaches the real result. The final recognition accuracy of the dataset was used as an evaluation criterion for the human behavior recognition model.

5.3. Ablation Experiment

To verify the importance of the proposed two parts, the C3D time-stream module and multi-scale convolutional Transformer module, in the dual-stream network, ablation experiments were conducted separately. The recognition accuracy of the original two-stream network is compared with the network after the addition of the two modules, and the experimental results are shown in

Table 2. Recognition accuracy of different modules.

Modeling	Recognition Accuracy (ACC)/%
Two-stream	88.6
C3D-Two-stream	91.8
Multi-Transformer-Two-stream	90.5
C3D+Multi-Scale Transformer	92.9

.

From the results in Table 2, it can be seen that the recognition accuracy on the dataset UCF101 is improved by 3.2% and 1.9% after adding the C3D module and the multi-scale convolutional Transformer module, respectively. By employing data augmentation and optimizing the model training strategies, the recognition accuracy of the model can be further enhanced. To further verify the effectiveness of the proposed module, we conducted ablation experiments on six categories of the KTH video dataset, and the results are shown in Figure 7. In the figure, TS is the abbreviation of two-stream, C3D-TS is the abbreviation of two-stream based on C3D, and C3D-MS-TS is the abbreviation of two-stream based on multi-scale Transformer module and C3D.

The results indicate that the use of 3D convolution instead of 2D convolution can enhance the extraction of dynamic features, and the fusion of multi-scale attention integration information can enhance the expression of features. Finally, our behavior recognition algorithm is compared with other behavior recognition algorithms on the UCF101 dataset and the comparison results are shown in

Table 3. Recognition accuracy of different methods.

Modeling	UCF101 (ACC)/%	HMDB51 (ACC)/%
Two-stream [34]	87.6	59.4
P3D [4]	88.6	-
C3D [12]	85.4	56.8
VTN [27]	92.7	72.3
IDT [35]	85.9	57.2
AVID-CAM [36]	87.5	-
TDN [37]	91.5	75.3
STS [38]	90.1	62.4
YOWOv3 [39]	95.4	68.8
TDD [40]	90.3	63.2
Ours	92.9	74.6
Ours-DA	94.2	-

Ours-DA: used Data Augmentation.

.

As can be seen from Table 3, the accuracy of the two-stream algorithm with the addition of the C3D module and the multi-scale convolutional Transformer module is obviously enhanced in comparison with the other algorithms, which indicates that the behavioral recognition method designed in this paper achieves a better recognition effect after strengthening the extraction of the dynamic features as well as improving the accuracy of the feature extraction.

6. Conclusions and Future Works

6.1. Conclusions

We proposed an improved two-stream convolutional neural network for a human action recognition algorithm based on the C3D and Transformer model. Firstly, the C3D network is used in the time dimension to replace the 2D convolution, which enhances the extraction of dynamic features. Secondly, in the spatial dimension, the multi-scale Transformer encoder is used for feature extraction, and after inputting the context-based relative position encoding, the features are adaptively integrated into different scales under the action of adaptive scale-attention mechanism and the accuracy of extracted features is improved. Finally, the optimal network architecture is obtained by experimenting with the fusion method of two-stream features. The experimental results demonstrate that the proposed algorithm achieves superior recognition accuracy compared to the original two-stream network, C3D network, and other state-of-the-art methods.

6.2. Future Works

The action recognition algorithm we proposed in this paper demonstrates superior recognition performance. However, the incorporation of 3D convolution and multi-scale Transformer has led to an increase in the model’s parameter count and complexity. In future research, we plan to delve into the application of channel pruning and knowledge distillation techniques to achieve network compression and mitigate the training burden. The challenge of recognizing actions when parts of the human body are occluded is a critical area for further investigation. In real-world applications, partial occlusion of the human body is a frequent occurrence, significantly complicating the task of action recognition. Consequently, developing new algorithms and methods to enhance the accuracy and robustness of action recognition systems under occlusion conditions will be a primary focus of subsequent research endeavors. Additionally, under the framework of a two-stream network architecture, leveraging the fusion of multimodal information for action recognition is another promising avenue for future exploration.