Multi-Grained Temporal Clip Transformer for Skeleton-Based Human Activity Recognition

Abstract

Skeleton-based human activity recognition is a key research topic in the fields of deep learning and computer vision. However, existing approaches are less effective at capturing short-term sub-action information at different granularity levels and long-term motion correlations, which affect recognition accuracy. To overcome these challenges, an innovative multi-grained temporal clip transformer (MTC-Former) is proposed. Firstly, based on the transformer backbone, a multi-grained temporal clip attention (MTCA) module with multi-branch architecture is proposed to capture the characteristics of short-term sub-action features. Secondly, an innovative multi-scale spatial–temporal feature interaction module is proposed to jointly learn sub-action dependencies and facilitate skeletal motion interactions, where long-range motion patterns are embedded to enhance correlation modeling. Experiments were conducted on three datasets, including NTU RGB+D, NTU RGB+D 120, and InHARD, and achieved state-of-the-art Top-1 recognition accuracy, demonstrating the superiority of the proposed MTC-Former.

1. Introduction

Human activity recognition (HAR) and behavior analysis play key roles in empowering embodied intelligent systems with the capability to comprehend and adapt to human living and working environments. They therefore have wide applications, such as video surveillance [1], human–computer interaction [2,3], and medical healthcare [4]. Human activity can be represented through various data modalities [5], such as RGB videos, Depth maps, skeleton joints, infrared images, etc. RGB videos [6] provide rich appearance information such as color and texture, while Depth maps offer geometric cues like distance and shape. Most modern frameworks jointly utilize RGB and Depth maps to improve recognition performance through the early or late fusion of the complementary strengths of RGB and Depth modalities, depending on the application. Different from RGB and Depth modalities, 3D skeletons offer precise positional and dynamic details about human skeleton joints, without being influenced by changes in background, lighting, or viewing angle. Additionally, skeleton data are easily obtained via pose estimation algorithms [7] or Microsoft Kinect [8,9]. Thus, skeleton-based HAR has gradually garnered widespread interest from researchers.

Driven by the rapid advancements in deep learning for skeleton-based activity recognition, contemporary techniques are predominantly categorized into Graph Convolutional Network (GCN)-based [10,11,12,13,14,15] and transformer-based approaches [16,17,18,19,20,21,22]. GCN-based models demonstrate strong capabilities in processing skeleton data and capturing spatiotemporal correlations within skeleton sequences. Liu et al. [11] introduced a multi-scale aggregation strategy to broaden the receptive field and addressed the biased weight issue. They found that when handling rapidly changing sub-actions, the short-term features may lack sufficient granularity. Wu et al. [14] proposed a local–global GCN combined with a dynamic frame weighting module for joint global temporal modeling, relying on a fixed skeleton topology and predefined adjacency relationships, which limited its ability to model long-range periodic actions (e.g., dance sequences). Liu et al. [19] introduced hierarchical temporal partitioning and aggregation to capture long-term dependencies, whereas global temporal relationships were constrained by the fixed-window mechanism of transformers. Despite the remarkable advancements made by the aforementioned methods, these methods solely focus on spatial dependencies among joints within individual frames, while neglecting the complex spatiotemporal correlations across consecutive frames, thereby failing to handle both short-term sub-action features and long-term motion correlations.

To address these problems, Qiu et al. [18] proposed a segment partitioning approach and employed self-attention mechanisms to model short-term sub-action information within the intra-clip. Cui et al. [20] introduced a temporal–local spatial–temporal attention mechanism (TL-STA) designed to model inter-joint correlations across sequential frames. Liu et al. [22] utilized physical topology constraints of the human skeleton to fully explore dependency between joints within a tuple to capture local and global temporal motion information. However, most aforementioned approaches failed to account for the inherent variations in temporal dynamics across different actions, thereby lacking in human motion domain knowledge-embedded feature extraction and interpretability.

In actuality, the domain knowledge of human motion plays a critical role in skeleton-based activity recognition, including the dynamics of human motion, behavioral habit patterns, and spatial–temporal contextual information. As shown in Figure 1, the activities of “clapping” and “long jumping” differ in terms of motion phases, motion patterns, and motion speeds. Consequently, selecting an appropriate clip partition strategy is crucial for extracting short-term sub-action information. To address this, an innovative multi-grained temporal clip transformer (MTC-Former) employing self-attention mechanisms is proposed to model correlations among joints, thereby improving activity recognition accuracy by integrating both fine-grained details and contextual information. Our contributions can be summarized from three perspectives, as follows:

• A multi-grained temporal clip attention (MTCA) module is proposed to capture multi-granularity short-term action clip features. This module consists of several intra-clip self-attention (ICSA) modules and a multi-grained feature fusion (MGFF) module for multi-granularity feature extraction and fusion, respectively.
• Additionally, a multi-scale spatial–temporal feature interaction (MSTI) module is proposed to effectively capture long-term correlations between short action clips and enhance spatial information interaction between human skeleton joints.
• Extensive experiments were conducted on two large-scale datasets and an industrial scenario dataset, demonstrating the effectiveness and strong generalization of our method.

Figure 1. The action of clapping can be regarded as consisting of three sub-actions, “Open hands”, “fold hands”, and “Put down hands”. The action of jumping can be regarded as consisting of two sub-actions, “Preparation Phase”, “Take off”, and “Landing”.

Figure 1. The action of clapping can be regarded as consisting of three sub-actions, “Open hands”, “fold hands”, and “Put down hands”. The action of jumping can be regarded as consisting of two sub-actions, “Preparation Phase”, “Take off”, and “Landing”.

Section 2 first presents the related works of skeleton-based HAR and its theoretical foundations. Subsequently, Section 3 introduces the proposed MTC-Former, including the research motivations and technical details of the proposed ICSA, MTCA, and MSTI modules. Finally, the experiments, comprehensive ablation, and comparative studies conducted on three standard large-scale datasets and one industrial scenario dataset are presented in Section 4. Conclusion and future works are presented in Section 5.

2. Related Works

2.1. Skeleton-Based Activity Recognition

Pioneering studies utilized curved manifold Lie groups to represent skeleton structures, modeling 3D human joint points through geometric algebra in three-dimensional space [23]. Skeleton optical spectra [24] were introduced to map skeleton joints onto color images, enabling convolutional neural networks (CNNs) to capture discriminative information of action. Recurrent Neural Network (RNN) models [25,26,27] were utilized to capture both local and global temporal dynamics. Concurrently, CNN-based approaches [28,29,30] transformed skeleton data into pseudo-data through specific methodologies. However, both CNN-based and RNN-based methods demonstrate limitations in effectively capturing the correlations among joints.

In contrast, GCN-based methods [10,12,31,32] demonstrate superior performance in capturing spatial–temporal dependencies due to their ability to handle irregular graph structures. Yan et al. [10] first applied graph convolution to skeleton data. Shi et al. [32] introduced a dual-stream framework with an adaptive method for dynamic learning, significantly improving the accuracy of human activity classification. Despite these advancements, GCN-based methods struggle to model long-term motion correlations in skeleton sequences. To address this limitation, Chen et al. [12] expanded the spatial–temporal receptive field by stacking the MT-GC module to capture long-term correlations. Similarly, Duan et al. [31] employed a multi-branch architecture to model both long- and short-range correlations. While GCN-based models demonstrate superior capability in capturing spatial–temporal dynamics compared to RNN and CNN methods, they remain limited in modeling short-term sub-actions.

2.2. Transformer for Skeleton-Based Activity Recognition

Transformer [33] and its variants [18,19,34,35,36,37,38] have demonstrated strong effectiveness in the field of activity recognition. Plizzari et al. [36] integrated self-attention and graph convolution to model spatial–temporal dependencies among joints. Shi et al. [37] utilized attention blocks specifically designed to address the unique characteristics of skeleton data, enabling the modeling of spatial–temporal correlations among joints. Gao et al. [38] explored the dependencies between joints and parts using cross self-attention mechanism within the intra-frame. Lin et al. [35] applied self-attention to model multi-scale spatial–temporal features, and leveraged cross-attention to combine cross-features. However, these methods only focus on modeling dependencies between joints or body parts within a single frame, neglecting the correlations between joints in consecutive frames and failing to model the characteristics of sub-actions.

3. Method

As illustrated in Figure 2, the architecture of the proposed multi-grained temporal clip transformer (MTC-Former) is provided. First, a position embedding layer was adopted to encode the skeleton sequences. Then, the MTCA module was designed to capture short-term sub-action information at various granularities. In addition, the MSTI module was designed, aimed at capturing long-term correlations among action clips and facilitating interactions between joints.

3.1. Position Embedding

Compared with traditional RGB data, skeleton data contain the positional information of human skeleton joints, unaffected by background and lighting conditions. The spatial–temporal position information of joints plays a crucial role in activity recognition [39]. Therefore, we encode the skeleton sequence to incorporate spatial–temporal position information. Specifically, the initial skeleton sequence ${\mathbf{X}}_0\in {\mathbb{R}}^{C_0\times T\times V}$, which contains $C_0$ channels, T frames, and V joints, is transformed into a $C_1$-dimensional feature tensor $\mathbf{X}\in {\mathbb{R}}^{C_1\times T\times V}$ through a linear mapping layer implemented by a $1\times 1$ 2D convolution, as follows:

$$\mathbf{X}={\mathrm{Conv}2\mathrm{D}}_{1\times 1}\left({\mathbf{X}}_0\right)\in {\mathbb{R}}^{C_1\times T\times V}$$

(1)

However, the feature tensor obtained through linear mapping fails to distinguish joint identities due to the absence of positional information, leading to a performance decline. To this, a position encoding module is introduced to encode the spatial location of each joint. We encode the relative positional information of joints, as follows:

$$PE(p_j,2i)=sin\left({\displaystyle \frac{p_j}{{1000}^{2i/C_1}}}\right)$$

(2)

$$PE(p_j,2i+1)=cos\left({\displaystyle \frac{p_j}{{1000}^{2i/C_1}}}\right)$$

(3)

where $p_j$ denotes the location of a joint and i corresponds to the index of the encoded feature tensor. Subsequently, the feature vector sequence and the position encoding tensor $\mathbf{PE}$ are combined through element-wise addition, as follows:

$$\check{\mathbf{X}}=\mathbf{X}+\mathbf{PE}\in {\mathbb{R}}^{C_1\times T\times V}$$

(4)

3.2. Intra-Clip Self-Attention

By kinematic analysis of human action categories, it is observed that inter-joint correlations across consecutive frames is important for the HAR. For instance, in “long jump” action, arm movements in frame t exhibit strong dynamical coupling with leg motions of frame $t+i$ at various temporal resolutions, demonstrating the necessity of cross-frame joint feature extraction. Therefore, it is very useful to extract the related features of different joints between adjacent frames and the sub-action clip information of short time intervals at various temporal resolutions.

Based on the above analysis, we propose the intra-clip self-attention (ICSA) module to effectively capture short-term motion information, as shown in Figure 3. Specifically, the encoded skeleton feature $\check{\mathbf{X}}\in {\mathbb{R}}^{C_1\times T\times V}$ is first partitioned into $T_0$ clips. Each clip is then flattened by reshaping the tensor, as follows:

$$\check{\mathbf{X}}=[{\mathbf{X}}_1,{\mathbf{X}}_2,\dots ,{\mathbf{X}}_{T_0}],{\mathbf{X}}_i\in {\mathbb{R}}^{C_1\times n\times V}$$

(5)

$$\check{\mathbf{X}}\in {\mathbb{R}}^{C_1\times T_0\times n\times V}\to {\mathbb{R}}^{C_1\times T_0\times V_0}$$

(6)

where $T_0=T/n$, $V_0=V\times n$, and n indicates the total frames within a clip.

Subsequently, the flattened tensor $\check{\mathbf{X}}$ is projected into ${\mathbf{X}}_p$ through a $1\times 1$ 2D convolution layer. Then, the query ($\mathbf{Q}$), key ($\mathbf{K}$), and value ($\mathbf{V}$) are obtained by splitting ${\mathbf{X}}_p$ along the channel dimension using the Split operation, as follows:

$${\mathbf{X}}_p={\mathrm{Conv}2\mathrm{D}}_{1\times 1}\left(\check{\mathbf{X}}\right)\in {\mathbb{R}}^{3C_1\times T_0\times V_0}$$

(7)

$$\mathbf{Q},\mathbf{K},\mathbf{V}=\mathrm{Split}\left({\mathbf{X}}_p\right)\in {\mathbb{R}}^{C_1\times T_0\times V_0}$$

(8)

To model the intrinsic correlations between joints, we introduce a learnable parameter matrix $\mathbf{A}\in {\mathbb{R}}^{T_0\times V_0\times V_0}$ to encode the topology information of the human body. Additionally, a learnable parameter $\beta$ is employed to dynamically reweigh the correlation matrix. The output ${\mathbf{X}}_{attn}$ is then computed by multiplying the correlation matrix with the corresponding value $\mathbf{V}$, as follows:

$${\mathbf{A}}_{dot}=\mathrm{Tan}\mathrm{h}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^T}{\sqrt{C}}}\right)$$

(9)

$${\mathbf{X}}_{attn}=\mathbf{V}(\beta \times {\mathbf{A}}_{dot}+\mathbf{A})\in {\mathbb{R}}^{C_1\times T_0\times V_0}$$

(10)

Finally, we employ H multi-head self-attention mechanisms to capture richer and more complex dependencies among joints.

$${\mathbf{X}}_{ICSA}=\mathrm{Concat}({\mathbf{X}}_{attn}^1,{\mathbf{X}}_{attn}^2,\dots ,{\mathbf{X}}_{attn}^H)\in {\mathbb{R}}^{C_1\times T_0\times V_0}$$

(11)

In the following, a feed-forward network (FFN) and two residual connections are utilized to improve the model’s representation capacity and training stability.

3.3. Multi-Grained Temporal Clip Attention Module

The researchers of [40] revealed that in the visual cortex, the local receptive field (RF) sizes of neurons in the same area are different, which enables the neurons to collect multi-scale spatial information in the same processing stage. Furthermore, the RF sizes of neurons in the visual cortex are not fixed, but modulated by the stimulus. We employed an adaptive approach to generate attention weights for temporal motion features at different granularities, assigning varying importance weights to short-term motion features across scales before ultimately fusing them.

Based on the above analysis, we propose the multi-grained temporal clip attention (MTCA) module, as shown in Figure 4. Specifically, the input feature $\mathbf{X}\in {\mathbb{R}}^{C_1\times T\times V}$ is segmented into K clips along the channel. These K features are then processed by ICSA modules with varying granularity to capture short-term motion information at different granularities. Finally, the outputs from the ICSA modules are reshaped for further processing, as follows:

$$[{\mathbf{X}}^1,{\mathbf{X}}^2,\dots ,{\mathbf{X}}^K]=\mathbf{Split}\left(\mathbf{X}\right)\in {\mathbb{R}}^{C_1/K\times T\times V}$$

(12)

$${\mathbf{X}}_{ICSA}^i={\mathrm{ICSA}}_{n=i}\left({\mathbf{X}}^i\right)\in {\mathbb{R}}^{C_1\times T_0\times V_0}$$

(13)

$${\mathbf{X}}_{ICSA}^i\in {\mathbb{R}}^{C_1\times T_0\times V_0}\to {\mathbb{R}}^{C_1\times T\times V}$$

(14)

where ${\mathbf{X}}^i$ is obtained by splitting $\mathbf{X}$ along the channel dimension, with $i=\{1,2,3,\dots ,K\}$, and ${\mathbf{X}}_{ICSA}^i$ indicates the output features from the i-th ICSA module.

Next, as shown in Figure 4 for MGFF, we aggregate the output features from the ICSA modules with K different granularities through summation, followed by an average pooling operation to generate a global representation ${\mathbf{X}}_{ICSA}^{global}\in {\mathbb{R}}^{C_1\times 1\times 1}$.

Subsequently, ${\mathrm{f}}_1$ is applied on ${\mathbf{X}}_{ICSA}^{global}$ for channel reduction, and the result is sequentially passed through ${\mathrm{f}}_2$ and a softmax function to adjust the weights of features at different granularities, obtaining weight vector $\mathbf{W}\in {\mathbb{R}}^{K{C}_1\times 1\times 1}$. Finally, $\mathbf{W}$ is divided along the channel dimension using a Split operation to obtain weight vectors with different granularities.

$${\mathbf{X}}_{ICSA}^{global}=\mathrm{GAP}\left(\sum _{i=1}^K{\mathbf{X}}_{ICSA}^i\right)$$

(15)

$$\mathbf{W}=\mathrm{Softmax}\left({\mathrm{f}}_2\left({\mathrm{f}}_1\left({\mathbf{X}}_{ICSA}^{global}\right)\right)\right)\in {\mathbb{R}}^{K{C}_1\times 1\times 1}$$

(16)

$${\mathbf{W}}_1,{\mathbf{W}}_2,\dots ,{\mathbf{W}}_K=\mathrm{Split}\left(\mathbf{W}\right)\in {\mathbb{R}}^{C_1\times 1\times 1}$$

(17)

where ${\mathbf{W}}_i$ represents the weight vector corresponding to the i-th granularity, ${\mathrm{f}}_1$ and ${\mathrm{f}}_2$ denote $1\times 1$ 2D convolutional operations. Then, a dot-product operation is applied between $\mathbf{W}$ and ${\mathbf{X}}_{ICSA}$ to reweight the output features from different ICSA modules. Finally, to strengthen the training stability, a residual connection is incorporated to obtain the output ${\mathbf{X}}_{MTCA}\in {\mathbb{R}}^{C_1\times T\times V}$:

$${\mathbf{X}}_{MTCA}=\sum _{i=1}^K{\mathbf{W}}_i\odot {\mathbf{X}}_{ICSA}^i+\mathbf{X}$$

(18)

3.4. Multi-Scale Spatial–Temporal Feature Interaction (MSTI)

From a human action kinematic perspective, human motion exhibits multi-scale temporal characteristics, encompassing both rapidly changing local details (e.g., finger movements) and slower-evolving global patterns (e.g., trunk motion). Consequently, multi-scale temporal convolution is designed to extract these hierarchical motion features. Furthermore, human movement requires coordinated interactions among multiple joints. For instance, a throwing motion demands precisely timed coordination of the shoulder, elbow, and wrist joints [41]. Thus, modeling individual joints solely through temporal convolution is insufficient; it is essential to simultaneously account for kinematic interactions across different joints.

To this, we introduce a multi-scale spatial–temporal feature interaction (MSTI) module to capture the correlations between sub-actions at different granularity levels and enhance the skeleton motion information interactions among joints, as illustrated in Figure 5. Specifically, the feature ${\mathbf{X}}_{MTCA}\in {\mathbb{R}}^{C_1\times T\times V}$ first undergoes an average pooling operation along the joint dimension, producing a global representation ${\mathbf{X}}_{global}\in {\mathbb{R}}^{C_1\times T\times 1}$. Then, both ${\mathbf{X}}_{MTCA}$ and ${\mathbf{X}}_{global}$ are processed by multi-scale temporal convolution (MSTC) to capture multi-scale temporal information. The MSTC consists of six branches, each of which applies a 2D convolution for channel reduction. The first four branches employ temporal convolutions with different dilation rates of 1, 2, 3, and 4, respectively. The fifth branch incorporates a 3 × 1 max-pooling layer, while the sixth branch retains the original input. Finally, all the outputs from the six branches are concatenated along the channel dimension.

Then, a learnable vector $\mathbf{V}\in {\mathbb{R}}^{1\times V}$ is introduced to facilitate spatial information interaction among joints. These features are then combined to fuse spatial–temporal information among joints, obtaining ${\mathbf{X}}_{out}\in {\mathbb{R}}^{C_1\times T\times V}$, as formulated below:

$${\mathbf{X}}_{out}=\mathrm{MSTC}\left({\mathbf{X}}_{MTCA}\right)+\mathrm{MSTC}\left({\mathbf{X}}_{global}\right)\times \mathbf{V}$$

(19)

4. Experiments

We begin with a brief introduction of the two large public human action recognition datasets, one industrial human action recognition dataset, and the experimental settings. Subsequently, ablation studies and visualizations that were carried out to examine each component’s functionality are presented. Finally, performance comparisons with mainstream methods and analysis are given.

4.1. Datasets

NTU RGB + D [8] is a standard activity recognition dataset, which is widely adopted for HAR. In this dataset, 56,000 samples were gathered and classified into 60 categories. Each sample comprises four data modalities: RGB video, Depth maps, 3D skeleton data, and infrared video. This study specifically utilized 3D skeleton data for experimentation. Following [8], training and testing sets were selected from the dataset using two distinct strategies, where Cross-View (X-View) was determined by camera IDs and Cross-Subject (X-Sub) separated the dataset based on subject IDs.

By introducing 60 action categories with 57,600 new samples, NTU RGB+D 120 [9] has 114,489 samples and 120 activity classes. Consistent with NTU RGB+D, each sample contains all four data modalities, with skeleton data being selected for experimental purposes. Following the same setup of [9], we employed two distinct evaluation protocols. One is the Cross-Subject (X-Sub), where 50% of subjects are used for training. Alternatively, the Cross-Setup (X-Set) protocol divides the setup IDs into two groups: even-numbered IDs are assigned to the training set, while odd-numbered IDs are allocated to the testing set. These partitioning strategies facilitate the evaluation of activity recognition models across the dataset. Some samples are shown in Figure 6.

InHARD [42] is an industrial human action recognition dataset designed for production environments, primarily targeting human behavior recognition and analysis tasks in human–robot collaborative settings. The dataset comprises 14 industrial action categories with 5303 action samples. Each sample includes RGB video and skeleton modalities captured from three distinct viewpoints. InHARD partitions the samples into training and test sets based on both performers and viewing angles. Some samples are shown in Figure 7.

4.2. Experimental Setting

The proposed MTC-Former (MATTOLI SRL, Jesi, Italy) was trained utilizing four RTX 3090 GPUs, with each skeleton sequence uniformly extended to 60 frames via activity replay to ensure consistency. The model was optimized using SGD and trained for 120 epochs with a batch size of 32. The architecture comprises seven multi-grained temporal clip (MTC) blocks, with the output channels sequentially set to 72, 72, 144, 144, 288, 288, and 288.

The Impact of Hyperparameters

As shown in Figure 8, graphical illustrations are incorporated to demonstrate how variations in key hyperparameters affect experimental performance, the ratio of dilation in MSTI, and the number of heads in the multi-head attention mechanism (MHSA).

As shown in Figure 8a, the model achieves 91.0% accuracy (X-Sub) with a dilation rate of 4, effectively capturing both short-term sub-action features and long-term motion relationships. A smaller rate limits modeling to local dynamics, while a larger rate (e.g., 6) reduces accuracy (90.66%) due to diminished per-branch feature capacity from fixed total channels. Figure 8b shows that the model achieves 89.81% accuracy with one attention head, peaking at 91.0% (1.19% improvement) with 4 heads before declining to 88.60% at 8 heads. Fewer heads (e.g., 0–2) limit dependency learning, while more heads (e.g., 6–8) reduce per-head channel dimensions, degrading attention patterns due to constrained total capacity.

4.3. Ablation Studies

On the NTU RGB+D skeleton dataset, our experiments aimed to evaluate the effectiveness of each component in MTC-Former, including the ICSA module with different granularities, the MGFF fusion strategy, and the MSTI module with different numbers of branches.

4.3.1. Ablation Study of ICSA

The intra-clip self-attention (ICSA) module is designed to capture short-term motion features, where n denotes clip length. When n takes different values, the ICSA module can capture short-term motion information at varying granularities. As shown in Figure 9, $n=1$ denotes the number of successive frames within a clip. When $n=1$, the ICSA module models joint correlations within a single frame, whereas $n=3$ enables the capture of inter-joint dependencies across three consecutive frames.

As shown in Figure 9, on the NTU RGB+D dataset, when $n=1$, the recognition performance of the proposed MTC-Former achieves 89.0% (X-Sub) and 93.8% (X-View). As n increases, the model’s recognition performance reaches its optimal level at $n=3$, with accuracy rates of 90.3% (X-Sub) and 95.0% (X-View), achieving improvements of 1.3% (90.3% vs. 89.0%) and 1.2% (95.0% vs. 93.8%), respectively, compared to ${\mathrm{ICSA}}_{n=1}$. However, when $n>3$, the recognition performance of the model gradually declines.

When n = 3, the model achieved the best performance because it captures a sufficient range of temporal dynamics without incorporating irrelevant information. Too small n (e.g., 1 or 2) fails to reflect temporal variations, leading to underutilized motion cues. Too large n (e.g., 4 or more) includes redundant or noisy temporal data, which may distract the attention mechanism.

4.3.2. Ablation Study of MSTI Module

The multi-scale spatial–temporal feature interaction (MSTI) module employs multiple $3\times 1$ convolutional layers with different receptive fields to capture long-term motion correlations. Additionally, it incorporates a learnable vector to improve spatial information interaction among joints. As shown in

Table 1. Ablation study of MSTI module on the NTU RGB+D 60 dataset.

Methods	X-Sub (%)	X-View (%)
MSTC-4	90.2	94.2
MSTC-6	90.5	94.9
MSTI-4 + fixed weight	90.7	95.4
MSTI-4 + dynamic weight	90.8	95.4
MSTI-6 + dynamic weight	91.0	95.8

, the ablation study was employed by varying the number of branches and exploring different weight fusion strategies. On the NTU RGB+D dataset, the MSTC-4, i.e., the multi-scale temporal convolution with 4 branches, achieves 90.2% (X-Sub) and 94.2% (X-View). Compared to MSTC-4, MSTC-6, the multi-scale temporal convolution with 6 branches further improves accuracy by 0.3% (X-Sub) and 0.7% (X-View).

In addition, compared to MSTC, the MSTI module incorporates a learnable weight vector to enhance spatial information interaction among joints. As shown in Table 1, with fixed weights, MSTI-4 outperforms MSTC-4 by 0.5% (90.7% vs. 90.2%) and 1.2% (95.4% vs. 94.2%). Dynamic weighting further widens the gap to 0.6% (90.8% vs. 90.2%) while retaining the 1.2% advantage (95.4% vs. 94.2%). The MSTI module enhances joint-level interaction within frames. When expanded to six branches with dynamic weight, it achieves optimal performance (91.0% X-Sub, 95.8% X-View), surpassing MSTI-4 with dynamic weight by 0.2% and 0.4%, respectively. This highlights the significance of long-term motion modeling, as MSTI effectively captures long-term motion dependencies and facilitates joint motion information interaction.

4.3.3. Ablation Study of Fusion Strategy

The MGFF module integrates output features from ICSA of varying granularities through three fusion approaches, average fusion, linear fusion, and dynamic fusion. Specifically, $\mathrm{MGFF}(n=1,2,3)$ denotes the fusion of three ICSA modules, with granularities of 1, 2, and 3, respectively. Average fusion refers to summing the output features of the three ICSA modules and then taking their mean. Linear fusion involves passing the output features of the three ICSA modules through a linear layer for integration. Dynamic fusion, on the other hand, processes the output features of the three ICSA modules through the MGFF module.

As shown in

Table 2. Ablation study of fusion strategy on NTU RGB+D 60 skeleton dataset with the joint modality.

Method	Fusion Strategy	X-Sub (%)	X-View (%)
$\mathrm{MGFF}(n=1,2,3)$	Average	90.8	95.6
	Linear	90.6	95.3
	Dynamic Fusion	91.0	95.8

, MGFF’s dynamic fusion achieves optimal performance (91.0% X-Sub, 95.8% X-View), surpassing linear fusion by 0.4–0.5% and average fusion by 0.2%. This is because linear fusion and average fusion merely perform simple combinations of the output features from different ICSA modules, failing to fully utilize their potential. In contrast, the MGFF module adopts a data-driven approach to generate weight vectors, which effectively leverages the output features of each ICSA module to adaptively produce weight vectors with different granularities. This demonstrates that the proposed MGFF module can effectively fuse output features at different granularity levels.

4.3.4. Ablation Study of Multimodal Data

Due to the inherent characteristics and complementary nature of different modal data, the integration of multiple modalities has been demonstrated to substantially enhance the model’s overall performance and robustness in various applications. Our proposed framework strategically incorporates four distinct yet interrelated data types: joint positions, bone vectors, joint motion trajectories, and bone motion patterns. This multimodal strategy allows the model to extract both static structural information and dynamic temporal evolution from different perspectives. The joint data provide fundamental spatial information, while the bone data convey relative orientation and distance between connected joints. Furthermore, the motion modalities (both joint and bone) effectively model the temporal dynamics and movement patterns, which are crucial for understanding complex activities and gestures.

As shown in

Table 3. Ablation study of multimodal data on the NTU RGB+D 60 skeleton dataset.

Methods	X-Sub (%)	X-View (%)
MTC-Former (joint)	91.0	95.8
MTC-Former (bone)	91.2	95.2
MTC-Former (joint motion)	88.8	94.4
MTC-Former (bone motion)	88.3	92.9
Fusion	93.2	97.1

, using only the most common joint modality data, MTC-Former achieves a performance of 91.0% (X-Sub) and 95.8% (X-View) accuracy. However, by integrating data from four modalities, it achieves a performance improvement of 2.2% (93.2% vs. 91.0%) and 1.3% (97.1% vs. 95.8%), respectively. This demonstrates the effectiveness of multimodal fusion.

4.3.5. Per Class Accuracy

As illustrated in Figure 10, on the NTU RGB+D dataset, we compared the per category performance between the proposed MTC-Former and that of the baseline ST-GCN, across 60 activity categories. The horizontal axis indicates the 60 distinct activity categories. Compared to ST-GCN, the majority of activity classes, particularly “clapping ”(10), “carrying” (29), “watching TV ” (31), and “eating while standing” (33) show notably substantial performance gains in our model.

4.3.6. Visualizations

To evaluate the effectiveness of MTC-Former, we visualize and analyze the ICSA outputs at different granularity levels. Figure 11 presents two representative activities (“put on glasses” and “reading”) for comparative analysis. The horizontal and vertical axes indicate the joint and frame indices, respectively. In the activation map, the darker the color, the more critical the frame is to the short-term motion.

An analysis of the two distinct behaviors reveals that the feature activation for “put on glasses” is primarily concentrated in the mid-temporal region, while for the activity “reading”, the activation is predominantly localized in the early-temporal region. This pattern aligns with the temporal dynamics of the respective actions, indicating that the proposed method effectively focuses on skeleton frames highly relevant to action-specific categorization, while filtering irrelevant information, consequently enhancing human activity recognition (HAR) performance.

Our analysis reveals that different ICSA modules focus on distinct temporal regions, capturing short-term motion features at varying granularity levels. For “put on glasses”, the output feature activations of ICSA (n = 1) and ICSA (n = 2) are mainly concentrated between frames 25 and 35, corresponding to the middle motion phase. In contrast, ICSA (n = 3) focuses on the range of 31 to 35 frames, capturing the motion’s critical phase. This observation demonstrates that the proposed ICSA modules effectively capture short-term motion information at different granularities.

We employed the confusion matrix to evaluate the confusion levels among different categories of human activities. As illustrated in Figure 12a, most activity categories achieve high classification accuracy. Notably, as shown in Figure 12b, action categories 95–100, which require long-term temporal modeling, exhibit high classification accuracy. This indicates that our model can effectively capture long-term motion features. However, as depicted in Figure 12c, the accuracy is relatively lower for certain behaviors within categories 71 to 76. In activity categories involving significant limb movements, such as arm swings and kicks, our method achieves excellent recognition accuracy. However, activities involving subtle movements, such as “making an OK sign” or “make a victory sign”, exhibit lower recognition accuracy. Additionally, activities involving interaction with small tools, such as “cutting nails or paper”, also exhibit lower recognition rates. Specifically, the 73rd activity, “staple book”, exhibits the lowest recognition accuracy and is frequently misclassified as the 76th activity, “cutting paper”, as skeleton data lack object interaction details. Similarly, the 71st activity, “making an OK sign”, and the 72nd activity, “making a victory gesture”, show lower accuracy and are often misclassified as each other. The critical distinction resides in subtle finger posture variations, which are challenging to capture owing to the constrained number of hand joints.

4.4. Comparison with State-of-the-Art Methods

Our MTC-Former is evaluated against state-of-the-art methods on the NTU RGB+D, NTU RGB+D 120, and InHARD datasets The symbol “-” denotes cases where results were not reported in the original papers. As shown in

Table 4. Comparison results of experiment on the NTU RGB+D and NTU RGB+D 120 datasets.

Categories	Methods	Prams(M)	NTU RGB+D		NTU RGB+D 120
			X-Sub (%)	X-View (%)	X-Sub (%)	X-Set (%)
CNN	RotClips+MTCNN [43]	-	81.1	87.4	62.2	61.8
	Ta-CNN+ [44]	1.06	90.7	95.1	85.7	87.3
GCN	ST-GCN [10]	3.1	81.5	88.3	70.7	73.2
	AS-GCN [47]	-	86.8	94.2	78.3	79.8
	2s-AGCN [32]	3.5	88.5	95.1	79.2	81.5
	Shift-GCN [48]		90.7	96.5	85.9	87.6
	MS-G3D [11]	2.8	91.5	96.2	86.9	88.4
	MSS-GCN [13]	-	92.7	96.9	88.9	90.6
	SPIA-Net [49]	3.2	92.8	96.8	89.2	90.4
	FR-Head [46]	2.0	92.8	96.8	89.5	90.9
	LG-SGNet [14]	1.73	93.1	96.7	89.4	91.0
Transformer	ST-TR [16]	12.1	89.9	96.1	82.7	84.7
	STSA-Net [18]	1.9	92.7	96.7	88.5	90.7
	TranSkeleton [19]	2.2	92.8	97.0	89.4	90.5
	LG-STFormer [22]	10.1	92.8	96.7	88.6	90.4
MLP	SiT-MLP [45]	0.6	92.3	96.8	89.0	90.5
	MTC-Former (ours)	2.5	93.2	97.1	89.7	91.4

, on the NTU RGB+D dataset, our MTC-Former with four-stream fusion achieves 93.1% (X-Sub) and 97.1% (X-View) accuracies, achieving the best performance. On the NTU RGB+D 120 dataset, compared to GCN-based methods [14], our MTC-Former achieves competitive results. Moreover, our MTC-Former outperforms transformer-based methods [19,22], CNN-based methods [43,44], and SIT-MLP [45]. Compared with the lightweight model [14,18,19,44,45,46], our model demonstrates significant performance advantages. In summary, our method achieves high performance while maintaining a low model parameter count.

To further evaluate the performance and generalization ability of the proposed method in different scenarios, we conducted comparisons with other state-of-the-art methods on the industrial human action recognition dataset InHARD. As shown in

Table 5. Comparison results on the InHARD datasets.

Categories	Methods	Modality	Params (M)	Top-1 (%)
CNN	TSN [50]	RGB	25.5	84.4
	TSM [51]	RGB	25.5	88.1
	I3D [52]	RGB	12.3	82.1
	Non Local [53]	RGB	30.25	86.3
Transformer	Timesformer [54]	RGB	121	85.2
	VideoSwin [55]	RGB	28.2	87.6
-	MTC-Former (ours)	Skeleton	2.5	88.3

, the proposed method achieves a Top-1 recognition accuracy of 88.3% on the InHARD datasets, outperforming transformer-based methods TSM and VideoSwin. Furthermore, our approach utilizes skeleton-based modality for human action recognition, which not only delivers superior recognition performance but also maintains lower parameter count. These results demonstrate that our method exhibits excellent generalization capability in industrial production scenarios.

5. Conclusions

To model inter-sub-action relationships and enable skeleton joint-level motion interaction, we proposed a multi-grained temporal clip transformer (MTC-Former) for skeleton-based activity recognition. The proposed MTCA can effectively learn multi-grained short-term sub-action motion information, where domain-specific priors on short-range motion patterns are embedded to enhance correlation modeling. The MSTI module can model long-term motion dependencies, correlations of short action clips, and skeletal motion interactions between human skeleton joints. Experimental results on NTU RGB+D, NTU RGB+D 120, and InHARD datasets demonstrate that the MTC-Former achieves state-of-the-art performance, achieving improved recognition performance compared to fixed transformer-based methods. Future work involves exploring the fusion of integrating RGB modality to compensate for the lack of fine-grained color and texture information for multimodal HAR in complex scene-understanding tasks.