Motion-Aware Graph Convolutional Network for Topology-Enhanced Skeleton-Based Action Recognition

Abstract

Skeleton-based action recognition has achieved significant progress through spatio-temporal graph convolutional networks. However, existing topology-enhanced methods treat all skeletal joints uniformly, failing to emphasize task-relevant joints for coordination-level hand-centric actions. Moreover, the graph topology is defined by fixed spatial proximity or generic channel-wise refinement, ignoring the fact that joints engaged in correlated motion patterns carry stronger discriminative signals. Therefore, we propose a Motion-Aware Spatio-Temporal Graph Convolutional Network (MA-STGCN) with a unified motion-aware framework consisting of two tightly coupled modules. First, a Motion-Aware Joint Attention (MJA) module is proposed, enabling the model to dynamically emphasize joints with salient motion for different action categories. Then, a Motion-Correlated Graph Refinement (MCR) module constructs a sample-specific inter-joint motion correlation matrix and uses it to adaptively refine the graph adjacency, strengthening connections between joints that move in coordinated patterns. Experiments are performed on the NTU RGB+D 60/120 datasets, our method achieves 94.2% and 91.3% accuracy on the bone stream. Comprehensive ablation studies validate the effectiveness of each component.

1. Introduction

Skeleton-based action recognition has become a central task in human–computer interaction, intelligent surveillance, and sports analysis. Unlike RGB-based methods, skeleton data provides a compact and illumination-invariant representation of human motion, making it particularly suitable for privacy-preserving applications. The introduction of Spatio-Temporal Graph Convolutional Networks (ST-GCNs) [1] marked a paradigm shift by modeling skeleton sequences as spatio-temporal graphs, where joints serve as nodes and natural body connections define spatial edges.

Subsequent works have advanced the field through various strategies. Chen et al. [2] proposed a novel channel-wise topology refinement graph convolution (CTR-GCN) to dynamically learn different topologies and effectively aggregate joint features in different channels for skeleton-based action recognition. Kilic et al. [3] proposed an attention-guided multi-scale graph convolutional network structure that dynamically determines the weights of the dependencies between joints. Li et al. [4] proposed a topology-enhanced spatio-temporal graph convolutional network (TEGCN) for skeleton-based action recognition. In order to maintain the advantages of temporal modeling while overcoming the dual drawbacks of long-term dependency forgetting and insufficient spatial modeling, recurrent neural networks (RNNs) gradually became a focus for researchers. Mim et al. [5] proposed a Gated Recurrent Unit-Inception (GRU-INC) model, which is an inception-attention based approach using GRU. Ashwin et al. [6] proposed a CNN-LSTM framework that demonstrates broad applicability across various domains. To capture the interaction relationships among multiple joints, the hypergraph structure is applied to action recognition. Zhou et al. [7] proposed an adaptive hypergraph convolutional network to achieve the aggregation of the rich semantic information conveyed by the bone vertices.

Despite these advances, two critical limitations persist:

First, existing topology-enhanced methods apply the same graph convolution operations to all 25 joints without considering their varying importance for different actions. For example, “typing” and “play phone” heavily depend on wrist and hand joints, while “kicking” relies on leg joints. This uniform treatment leads to suboptimal feature representations for actions involving localized body parts. Moreover, standard temporal pooling such as global average pooling collapses temporal dynamics, losing the motion information that is most discriminative for action recognition.

Second, existing methods define graph edges based on physical body connectivity [1], spatial distance [2], or generic channel-wise feature similarity [8]. However, these approaches do not explicitly model which joints move in correlated patterns during specific actions. During “clapping”, both hands exhibit symmetric motion; during “typing” only the right hand moves rapidly while the torso remains stable. Capturing such motion-correlated relationships can provide additional discriminative signals that fixed or feature-driven topology cannot. This is a coordination-level discrimination. It is carried by the relative engagement of body parts, such as which parts remain static, which parts move, and how their trajectories co-vary, rather than by the precise geometry of individual fingers or facial muscles. For example, during typing, the trunk and head remain almost static while the distal upper limb produces small, fast, quasi-periodic motion, and this contrast between trunk stillness and wrist activity is fully observable from the standard 25-joint skeleton.

To address these limitations, we propose MA-STGCN(Motion-Aware STGCN), which augments the TE-STGCN backbone with a unified motion-aware framework. The key insight is temporal motion statistics. Specifically, the per-channel temporal variance of each joint’s features serve as a compact yet powerful descriptor of joint-level motion saliency. Through the coordination-level joint motion analysis, there is no need to focus on fine-grained movements, such as those in fingers. Decisions can be made simply based on the collaborative patterns related to the action.

Attention mechanisms have been widely adopted in vision tasks. In skeleton-based recognition, SGN [9] introduced semantic-aware attention, while CTR-GCN applied channel and spatial attention (CBAM-style). SE-based attention [10] has been adapted for channel-wise feature recalibration. However, these methods apply attention at the channel or spatial level without explicit joint-level discrimination, and their squeeze operations typically use global average pooling, which collapses temporal dynamics. (1) Motion-Aware Joint Attention (MJA):We compute per-joint temporal motion energy (standard deviation over time) and map it through a lightweight convolutional bottleneck to normalized attention weights. Unlike mean pooling, which collapses temporal dynamics, motion energy directly identifies joints whose movement is discriminative for the current action. The attention weights are applied via a zero-initialized residual scaling, ensuring identity behavior at the start of training.

Graph topology learning is central to skeleton-based GCNs. ST-GCN used predefined partition sets. 2s-AGCN introduced adaptive adjacency matrices. CTR-GCN proposed channel-wise topology refinement through learnable channel-specific graphs. InfoGCN [11] modeled skeleton actions using information-theoretic latent distributions. However, these methods learn topology from generic feature representations without explicitly considering motion correlation between joints.(2) Motion-Correlated Graph Refinement (MCR): Using the same motion-energy descriptor, we construct a sample-specific inter-joint motion correlation matrix (Gram matrix of motion energies). This matrix captures which joints move in coordinated patterns and is used to adaptively refine the graph adjacency as a learnable residual. Joints with similar motion patterns (e.g., both wrists during clapping) receive enhanced graph connections, while uncorrelated joints maintain baseline topology. MCR is initialized as identity (zero residual) and gradually learns to augment the graph with motion-aware edges.

Both modules share the motion-energy computation, making the framework computationally efficient. We also adopt Stochastic Depth (DropPath) and Exponential Moving Average (EMA) as standard auxiliary regularization techniques during training.

Our main contributions are summarized as follows:

• We propose a Motion-Aware Joint Attention (MJA) module that learns per-joint importance weights from temporal motion statistics, achieving significant improvements on actions requiring localized joint discrimination.
• We introduce a Motion-Correlated Graph Refinement (MCR) module that constructs a sample-specific motion correlation matrix to adaptively refine graph edges, strengthening connections between co-moving joints. MCR shares the motion-energy computation with MJA, adding negligible parameter overhead.
• We conduct extensive experiments on NTU RGB+D 60 and NTU RGB+D 120 demonstrating that MA-STGCN achieves 94.2% and 91.3% accuracy on the bone stream.

2. Related Work

In the field of skeleton-based action recognition, there have been numerous studies, and remarkable innovations have been achieved. We will conduct a review of these methods from the following three aspects.

2.1. Graph Convolutional Network Based Methods

ST-GCN [1] first modeled the skeleton as an undirected spatial graph and defined the graph convolution over three fixed spatial partition sets (root, centripetal, and centrifugal). 2s-AGCN [8] introduced a learnable adjacency matrix that refines the physical topology in a data-driven manner. CTR-GCN [2] further proposed channel-wise topology refinement, learning channel-specific graphs that capture subtle joint relationships. AGMS-GCN [3] combined attention-guided multi-scale graph convolutions, and TE-STGCN [4] enriched the topology with learned spatial relation encodings. Hypergraph-based methods [7] extend pairwise edges to higher-order relations among joint groups. More recently, BlockGCN [12] redefined topology awareness by partitioning the skeleton into learnable blocks, showing that the fixed skeleton topology inherited from ST-GCN becomes progressively underused during training. These methods improve topology modeling from different perspectives, but they derive their graphs from generic feature representations or fixed anatomical priors, without explicitly exploiting the motion signal that drives the action itself.

2.2. Attention- and Transformer-Based Methods

Attention mechanisms have been widely adopted in vision tasks. In skeleton-based recognition, SGN [9] introduced semantic-guided attention; SE-based attention [10] has been adapted for channel-wise feature recalibration; and CBAM-style spatial-channel attention is integrated into CTR-GCN [2]. InfoGCN [11] modeled skeleton actions with information-theoretic latent distributions. Beyond attention, transformer architectures have been explored for skeleton sequences: Kim et al. [13] proposed a global–local motion transformer for unsupervised skeleton representation learning, which explicitly decomposes motion into global trajectories and local joint-wise dynamics. Recent unified representation frameworks such as USDRL [14] further show that dense skeleton representations transfer across multiple granularities of skeleton understanding. These approaches demonstrate the value of modeling motion explicitly, but they operate either at the channel level, the patch level, or through self-attention over all joints, without a dedicated mechanism that ties joint-level saliency to inter-joint graph structure.

2.3. Motion-Aware Methods

Modeling motion cues directly is a long-standing line of research. Ramanathan et al. [15] proposed a mutually reinforcing motion-pose framework in which motion and pose representations inform each other for pose-invariant action recognition, establishing that motion statistics and body configuration are complementary signals. Huang et al. [16] proposed motion-driven spatial and temporal adaptive high-resolution graph convolutional networks (MSA-HR-GCN), which construct motion-informed adjacency and temporal convolutions to guide feature learning. Kim et al. [13] decoupled global and local motion in an unsupervised transformer. MSA-HR-GCN derives motion cues from coordinate differences of consecutive frames and uses them to adjust graph structures. Our MJA and MCR instead compute a per-channel temporal motion-energy descriptor from the intermediate feature tensor, which is shared by two modules and requires only a negligible parameter budget. In addition, rather than using motion only for topology adaptation, we couple joint-level saliency with inter-joint correlation in a single framework, so that “Which joints matter?” and “Which joints should be connected?” are answered by the same statistic.

3. Materials and Methods

3.1. Dataset

We evaluate our method on the NTU RGB+D 60 and NTU RGB+D 120 datasets [17]. The NTU RGB+D 60 dataset contains 56,880 skeleton sequences across 60 action categories performed by 40 subjects, and the NTU RGB+D 120 dataset contains 114,480 sequences across 120 action categories performed by 106 subjects. The dataset is split into cross-subject (X-Sub) and cross-view (X-View) benchmarks. We report results on the cross-subject validation set. Each sequence contains at most 2 persons with 25 joints per person, captured at 30 fps using the Kinect v2 sensor (Microsoft Corporation, Redmond, WA, USA).

3.2. Overall Architecture

MA-STGCN builds upon the TE-STGCN backbone, which consists of 10 spatio-temporal GCN blocks with channel dimensions $\{64,64,64,64,128,128,128,256,256,256\}$ and temporal strides $\{1,1,1,1,2,1,1,2,1,1\}$. Each block integrates Channel-wise Topology Refinement GCN (CTRGCN), Static Spatial Relation Encoding (SSRE), Multi-Range Attention (MRA), and a four-branch temporal convolution, as shown in Figure 1. The input skeleton sequence is represented as $\mathbf{X}\in {\mathbb{R}}^{N\times M\times C\times T\times V}$, where N is the batch size, $M=2$ is the number of persons, $C=3$ is the coordinate dimension, T is the sequence length, and $V=25$ is the number of joints.

Figure 1. Overall architecture of MA-STGCN.

The overall forward pass is:

$$\mathbf{Z}=f_{\mathrm{blocks}+\mathrm{MJA}+\mathrm{MCR}}(\mathbf{X},\mathbf{A}),\mathbf{p}=\mathrm{FC}\left(\mathrm{GAP}\left(\mathbf{Z}\right)\right)$$

(1)

where $\mathbf{A}\in {\mathbb{R}}^{K\times V\times V}$ ($K=3$) is the three-partition normalized adjacency matrix, MJA and MCR are applied after the graph convolution and before the temporal convolution in each block, GAP denotes global average pooling, and FC is the classification head.

3.3. Motion-Aware Joint Attention

The first core innovation of MA-STGCN is the Motion-Aware Joint Attention (MJA) module. Unlike channel- or partition-level attention, MJA derives per-joint importance from temporal motion statistics, directly identifying joints whose motion is discriminative for the current action.

Given the feature tensor $\mathbf{F}\in {\mathbb{R}}^{N\times C\times T\times V}$ after the graph convolution, the MJA module operates in three steps.

Step 1: Temporal motion-energy extraction. For each joint v and channel c, we compute the temporal mean ${\mathit{\mu }}_v$ and the temporal motion energy ${\mathbf{e}}_v$ (per-channel standard deviation over time):

$${\mathit{\mu }}_v={\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T}{\mathbf{F}}_{:,:,t,v},{\mathbf{e}}_v=\sqrt{{\displaystyle \frac{1}{T}}\sum _{t=1}^T{\left({\mathbf{F}}_{:,:,t,v}-{\mathit{\mu }}_v\right)}^2+ϵ}$$

(2)

where ${\mathbf{e}}_v\in {\mathbb{R}}^{N\times C\times 1\times V}$ is the per-joint temporal motion energy, and $ϵ={10}^{-5}$ prevents numerical instability. Joints involved in salient motion (e.g., wrists during “typing” or “play phone”) exhibit high motion energy, whereas static joints (e.g., torso) yield low energy. This motion-driven saliency descriptor provides a stronger signal for action discrimination than mean pooling, which collapses temporal dynamics into a single average and loses the very information—how much each joint varies over time—that distinguishes one action from another.

Step 2: Saliency scoring. A lightweight convolutional bottleneck maps the motion-energy descriptor to normalized per-joint weights:

$$\mathbf{w}={\mathrm{Softmax}}_V\left({\mathrm{Conv}}_{1\times 1}^{\left(2\right)}\left(\mathrm{ReLU}\left({\mathrm{Conv}}_{1\times 1}^{\left(1\right)}\left({\mathbf{e}}_v\right)\right)\right)\right)$$

(3)

where ${\mathrm{Conv}}^{\left(1\right)}:C\to C/4$ and ${\mathrm{Conv}}^{\left(2\right)}:C/4\to 1$, producing $\mathbf{w}\in {\mathbb{R}}^{N\times 1\times 1\times V}$.

Step 3: Residual scaling. The attention weights are applied through a learnable residual connection with zero-initialized gain:

$${\mathbf{F}}_{\mathrm{MJA}}=\mathbf{F}\odot (1+\gamma ·\mathbf{w})$$

(4)

where $\gamma$ is a learnable scalar initialized to zero, ensuring that, at the start of training, the MJA module acts as an identity mapping, preserving the baseline behavior. As training progresses, $\gamma$ and $\mathbf{w}$ are jointly optimized to emphasize motion-salient joints.

The motion-energy descriptor ${\mathbf{e}}_v$ computed in Step 1 is shared with the MCR module, making the framework computationally efficient. The MJA module adds only ${\displaystyle \frac{C}{4}}+1$ parameters per block.

3.4. Motion-Correlated Graph Refinement

While MJA answers which joints are important, MCR addresses which joints should be more strongly connected by constructing a sample-specific motion correlation matrix and using it to adaptively refine the graph adjacency.

Motion correlation matrix. Given the motion-energy descriptor ${\mathbf{e}}_v\in {\mathbb{R}}^{N\times C\times 1\times V}$ (shared from MJA Step 1), we reshape it to $\tilde{\mathbf{e}}\in {\mathbb{R}}^{N\times C\times V}$ and compute the inter-joint motion correlation as the Gram matrix:

$$\mathbf{R}={\displaystyle \frac{1}{C}}{\tilde{\mathbf{e}}}^{\top }\tilde{\mathbf{e}}\in {\mathbb{R}}^{N\times V\times V}$$

(5)

where ${\mathbf{R}}_{ij}$ measures the motion-energy similarity between joints i and j. Joints that exhibit similar temporal motion patterns, such as both wrists during “clapping”, yield high ${\mathbf{R}}_{ij}$, while joints with uncorrelated motion, such as left wrist and right knee, yield low values.

The correlation matrix is normalized to ensure stable magnitudes:

$$\mathbf{S}={\mathrm{RowSoftmax}}_V\left(\mathbf{R}\right)\in {\mathbb{R}}^{N\times V\times V}$$

(6)

where softmax is applied along the last dimension (target joint), producing a row-normalized motion similarity matrix.

Graph adjacency refinement. The motion similarity matrix is used to refine the graph adjacency for each of the K partitions through a learnable residual:

$${\mathbf{A}}_k^{\prime }={\mathbf{A}}_k+\delta ·\mathbf{S},k=1,\dots ,K$$

(7)

where $\delta$ is a learnable scalar initialized to zero, ensuring that the refined adjacency equals the baseline adjacency at the start of training. As training progresses, $\delta$ learns to incorporate motion-correlated edges into the graph topology.

The refined adjacency ${\mathbf{A}}^{\prime }$ is used in the subsequent graph convolution:

$${\mathbf{F}}_{\mathrm{GCN}}={\displaystyle \sum _{k=1}^K}{\mathbf{A}}_k^{\prime }{\mathbf{F}}_{\mathrm{MJA}}{\mathbf{W}}_k$$

(8)

where ${\mathbf{W}}_k$ are the partition-specific weight matrices.

Design rationale. MCR complements MJA by modeling inter-joint relationships rather than individual joint importance. While the baseline TE-STGCN adjacency captures physical body connectivity and CTR-GCN refines it channel-wise from generic features, MCR introduces motion-aware edges: connections that are strengthened not because two joints are physically close, but because they move in coordinated patterns during the current action. This is particularly beneficial for actions where discriminative information lies in the relationship between joints rather than in any single joint’s motion.

3.5. Auxiliary Regularization

In addition to the core MJA and MCR modules, we also adopt two standard regularization techniques:

Stochastic Depth (DropPath) [18]: For each residual block i ($i=0,\dots ,L-1$), a linearly increasing drop probability $p_i={\displaystyle \frac{i}{L-1}}·p_{max}$ (with $L=10$, $p_{max}=0.1$) is applied. During training, the entire residual path of block i is dropped with probability $p_i$ (and the residual is rescaled by $1/(1-p_i)$ to maintain expected magnitude). Unlike label smoothing, DropPath does not modify the target distribution, preserving high-confidence predictions.

Exponential Moving Average (EMA) [19]: We maintain an EMA copy of model parameters with decay $\alpha =0.999$:

$${\mathit{\theta }}_{\mathrm{EMA}}^{\left(t\right)}=\alpha ·{\mathit{\theta }}_{\mathrm{EMA}}^{(t-1)}+(1-\alpha )·{\mathit{\theta }}^{\left(t\right)}$$

(9)

The EMA model is used exclusively for validation and inference, smoothing weight trajectories over training steps.

4. Results

4.1. Implementation Details

Input modalities. We use four input streams: joint (absolute coordinates), bone (relative positions), joint_vel (temporal velocity of joints), and bone_vel (temporal velocity of bones). Each stream is trained independently, and results are reported per-stream. In this paper, we focus on the bone stream for detailed analysis, as it achieves the highest single-stream accuracy in the baseline.

Frame sampling. We subsample each sequence to 150 frames using linear interpolation, balancing temporal resolution with computational efficiency.

Data augmentation. Mild augmentation is applied during training: random scaling (0.9–1.1), random rotation (±0.15 rad around the Y-axis), and temporal cropping (95% of frames with edge padding).

Optimization. SGD with Nesterov momentum 0.9, weight decay $4\times {10}^{-4}$, and initial learning rate 0.1 is used. We use five-epoch linear warmup followed by cosine annealing to ${10}^{-4}$ over 90 epochs. Batch size is 16 with a gradient accumulation factor of 2 (effective batch size 32). Mixed precision training (AMP) is enabled.

Normalization. BatchNorm2d is used for the bone and bone_vel streams, which produce stable feature distributions, and GroupNorm is used for the joint and joint_vel streams, which exhibit higher variance under small batch sizes. The input data normalization (BatchNorm1d) remains BatchNorm for all streams.

Selection of the hard classes. The hard classes are selected by a reproducible protocol applied to the TE-STGCN baseline before any motion-aware module is introduced. Specifically, for each action classes, we compute the baseline per-class validation accuracy averaged over three independent runs. A class is labeled hard if its average per-class accuracy falls below 70%, and at least 5% of its validation samples are confused with another single class. Ten classes satisfied both criteria including eat meal, clapping, reading, writing, take off shoe, play phone, typing, pointing, sneeze/cough and touch head. These classes share a common profile which is small-amplitude, hand-centric motion, motivating the motion-aware design rather than undermining it.

Use of the WeightedRandomSampler. During training only, the sampler draws samples from the training set with a 1.3× sampling weight for the ten hard classes. The sampler affects only the order and frequency of training samples. The validation and test sets are never resampled. This setting is fixed once and then keeps unchanged across all experiments, all streams, all datasets, and all ablation configurations, to guarantee a fair comparison. It should be noted that SBU and PKU-MMD use uniform sampling because their class distributions are balanced and the hard-class protocol is defined on the NTU RGB+D dataset.

Model parameters are as follows: 3.83 M (baseline: 3.76 M, +0.07 M from MJA and MCR), trained on a single NVIDIA RTX 4060 Ti (8 GB) for approximately 23 h per stream.

4.2. Comparison with Baseline

We compare STGCN [1], USDRL [14], CTR-GCN [2], AGMS-GCN [3], GRU-INC [5], MSA-HR-GCN [16], Hyper-GCN [7] and TEGCN [4]. Table 1 summarizes the main results on NTU60 and NTU120. All results are reproduced by us under a unified protocol: the official public implementations are retrained on the bone stream with identical preprocessing, which is 150-frame linear-interpolation sampling, identical augmentation, and the standard cross-subject split. This isolates the modeling contribution from implementation and training-schedule differences.

Table 1. Overall comparison with baseline (bone stream, NTU60 and NTU120 X-Sub). Best values are in bold and second-best underlined.

Method	Accuracy (%)
	NTU60	NTU120
STGCN	81.5	77.7
USDRL	87.0	79.5
CTR-GCN	88.6	84.9
AGMS-GCN	90.6	87.8
GRU-INC	90.1	87.3
MSA-HR-GCN	93.5	90.6
Hyper-GCN	93.7	90.5
TEGCN	92.3	88.3
MA-STGCN	94.2	91.3

MA-STGCN achieves the highest accuracy on both benchmarks, reaching 94.2% on NTU60 and 91.3% on NTU120. Compared with the classical STGCN backbone, our method yields substantial improvements of 15.6% and 17.5% on the two datasets, respectively, validating the effectiveness of jointly modeling motion-aware joint saliency and motion-correlated graph topology. Compared with the motion-driven MSA-HR-GCN, which explicitly constructs motion-guided adjacency, MA-STGCN increases by 0.7% on NTU60 and 0.8% on NTU120 with a substantially smaller parameter budget (3.83 M vs. 5.6 M), confirming that coupling joint-level saliency and inter-joint correlation is an effective and lightweight alternative to high-resolution motion-guided topology construction. Even when measured against our direct baseline TEGCN, which already incorporates topology-enhanced graph convolution, MA-STGCN delivers consistent gains of 2.1% on NTU60 and 3.4% on NTU120, demonstrating that the performance lift is attributable to our proposed modules rather than the underlying backbone architecture.

Among the more recent approaches, Hyper-GCN represents the strongest competitor, leveraging hypergraph structures to capture higher-order joint relations. While Hyper-GCN achieves competitive results, MA-STGCN still surpasses it by 0.5% on NTU60 and 0.8% on NTU120. Notably, the margin widens on the more challenging NTU120 benchmark, which contains 120 action classes with higher inter-class similarity. This trend indicates that our motion-driven approach exhibits superior generalization ability in the action recognition scenario, where subtle joint-level motion differences serve as the primary discriminative cues. Methods such as CTR-GCN, AGMS-GCN, and GRU-INC, which rely on channel-level or temporal-level modeling without explicit motion-aware joint weighting, trail MA-STGCN by larger margins, further confirming the value of operating at the joint-motion level.

A consistent observation across both datasets is that the performance advantage of MA-STGCN grows as the task becomes more difficult. The gap between MA-STGCN and the second-best method (Hyper-GCN) increases from 0.5% on NTU60 (60 classes) to 0.8% on NTU120 (120 classes), and the gap over TEGCN widens from 2.1% to 3.4%. This scaling behavior suggests that the Motion-Aware Joint Attention (MJA) module and the Motion-Correlated Graph Refinement (MCR) module are particularly effective when the action vocabulary expands and the model must distinguish between visually similar actions that differ only in localized joint motion patterns.

In summary, MA-STGCN consistently outperforms all compared methods on both benchmarks, with the advantage being more pronounced on the larger and more fine-grained NTU120 dataset, thereby demonstrating the robustness and scalability of our motion-aware framework.

In order to assess generalization beyond NTU-scale benchmarks, we evaluate the methods on SBU-Kinect Interaction (15 joints) and PKU-MMD I (25 joints), as shown in Table 2. All methods are retrained under the same protocol for a fair comparison.

Table 2. Comparison on SBU-Kinect Interaction and PKU-MMD Part I (bone stream, cross-subject). Best values are in bold and second-best underlined.

Method	SBU (%)	PKU-MMD Part I (%)
ST-GCN	90.4	83.3
USDRL	91.2	75.7
CTR-GCN	93.1	87.4
Hyper-GCN	93.1	87.6
TEGCN	93.5	88
MA-STGCN (ours)	94.9	89.3

MA-STGCN improves over the strongest baseline by 1.5% on SBU and 1.5% on PKU-MMD. The gain on SBU is notable because the 15-joint topology contains no finger joints and coarser hand information, yet the motion-energy statistics remain discriminative; the interacting body parts exhibit strongly correlated motion that MCR captures. These results indicate that the motion-aware framework is not tied to the 25-joint Kinect v2 topology and generalizes across datasets and skeleton formats.

4.3. Multi-Stream Analysis

In order to analyze the effectiveness of multi-stream, we have compared the cross-subject accuracy of all four input streams on both NTU benchmarks, as shown in Table 3. Each stream is trained independently under the identical protocol.

Table 3. Per-stream accuracy of MA-STGCN on NTU RGB+D 60/120 (X-Sub). Best values are in bold and second-best underlined.

Stream	NTU60 X-Sub (%)	NTU120 X-Sub (%)
Joint	92.5	89.6
Joint_vel	90.3	87.2
Bone	94.2	91.3
Bone_vel	91.4	88.4

As can be seen from the table, the bone stream achieves the best accuracy, followed by the joint stream. This is because bone vectors encode the orientation of each body segment relative to its parent joint, which is invariant to global translation and less sensitive to the subject’s position in the capture volume than absolute joint coordinates. In addition, absolute joint coordinates mix subject-dependent body proportions with motion, whereas bone vectors suppress the global trajectory component; empirically, the feature statistics of the bone stream are more stable across batches, which also allows BatchNorm2d to be used. Furthermore, because bone features are centered on the body’s local frame, the temporal motion energy of a moving limb contrasts more strongly against the static torso, sharpening the joint-saliency weights and the motion-correlation matrix. Velocity streams such as joint_vel and bone_vel emphasize change but discard absolute pose, which harms classes whose identity depends on static posture, thus explaining their lower accuracy.

4.4. Cross-View and Cross-Setup Evaluation

We tested whether the motion-aware framework remains robust when the camera viewpoint and sensor distance change; Table 4 reports the accuracy on the NTU RGB+D 60 cross-view (X-View) benchmark and the NTU RGB+D 120 cross-setup (X-Set) benchmark, as well as the cross-subject results.

Table 4. Cross-benchmark evaluation of MA-STGCN (bone stream). Best values are in bold.

Method	NTU60		NTU120
	X-Sub	X-View	X-Sub	X-Set
TEGCN	92.3	96.3	88.3	89.9
MA-STGCN	94.2	96.0	91.3	92.7

On NTU120 X-Set, MA-STGCN improves over the baseline by 3.1%, comparable to the 3.4% on X-Sub, indicating that the motion-aware modules transfer to unseen camera setups. This is because MJA and MCR are driven by temporal motion statistics computed from features, which are viewpoint-agnostic. On NTU60 X-View, the baseline TEGCN is already extremely strong (96.3%), and our method reaches 96.0%, which is a slight 0.3% deficit. We attribute this mainly to the fact that the X-View test split shares two of its three camera views with the training split, and partly to the mild class-rebalancing sampler defined on X-Sub statistics, which slightly biases the training distribution away from the X-View optimum.

4.5. Per-Class Accuracy Analysis

Table 5 shows the action categories with the most significant improvements. The MJA and MCR modules provide the largest gains for actions requiring localized joint discrimination and coordinated motion patterns.

Table 5. Top-9 improved action categories (MA-STGCN vs. baseline). Best values are in bold and second-best underlined. The checkmark means it belongs to hard classes.

Action	Baseline	Ours	$\mathbf{\Delta }$	Hard?
Touch head	60.1%	85.1%	+25.0%	✓
Play phone	63.7%	79.3%	+15.6%	✓
Shake head	82.2%	93.8%	+11.6%
Typing	63.3%	74.6%	+11.3%	✓
Clapping	65.3%	73.3%	+8.0%	✓
Writing	50.8%	57.7%	+6.9%	✓
Cheer up	84.8%	91.3%	+6.5%
Take off hat	90.6%	96.0%	+5.4%
Fan self	86.2%	91.7%	+5.4%

During the three runs, the accuracy for each class is below 70%, and at least 5% of the validation samples are wrongly classified as other individual classes, being regarded as “hard” classes. The ten classes satisfy both criteria and share a coherent profile, which is exactly the regime the motion-aware design targets. The average accuracy on the 10 hard classes improves from 66.5% to 72.1% (+5.6%), demonstrating the effectiveness of motion-aware attention for challenging actions. Several patterns emerge:

• Single-joint-dominant actions (touch head, play phone): MJA’s per-joint motion attention directly identifies the discriminative joint (wrist/hand), yielding the largest gains (+25.0%, +15.6%).
• Coordinated-motion actions (clapping, shake head): MCR’s motion-correlated edges capture the symmetric or rhythmic coordination between joints, providing complementary gains (+8.0%, +11.6%).
• Coordination-level hand-centric discrimination (typing, writing): Both modules contribute—MJA highlights the active hand, while MCR strengthens the connection between hand and torso, helping distinguish similar wrist trajectories (+11.2%, +6.9%).

4.6. Ablation Study

Table 6 presents the incremental contribution of each component in MA-STGCN, evaluated on the NTU RGB+D 60 dataset. We report both overall accuracy and the average accuracy on the 10 most challenging action classes (Hard10), the latter serving as a diagnostic indicator for coordination-level hand-centric discrimination capability.

Table 6. Ablation study results (bone stream, NTU60 X-Sub). Each row lists the modules present in that configuration: all rows build on the TEGCN baseline, and a checkmark means the module is enabled. Best values are in bold and second-best underlined.

Configuration	MJA	MCR	DropPath	EMA	Acc (%)	Hard10 Avg (%)
Baseline (TEGCN)					92.3	66.5
+MJA only	✓				92.9	68.7
+MCR only		✓			92.8	68.1
+MJA + MCR	✓	✓			93.2	69.8
+MJA + MCR + DropPath (0.1)	✓	✓	✓		93.8	70.9
Full MA-STGCN (+EMA 0.999)	✓	✓	✓	✓	94.2	72.1

Effect of MJA and MCR. Individually, MJA improves the baseline from 92.3% to 92.9% and MCR from 92.3% to 92.8%; jointly, they reach 93.2%. The joint gain exceeds either individual gain but is less than their sum, indicating partial overlap between the two signals. They are both driven by the shared motion-energy descriptor, so each already captures part of the motion information. On the Hard10 metric, MJA alone yields +2.2% and MCR alone +1.6%, while the combination yields +3.3%. This is because the two modules specialize on different aspects of difficult actions. MJA reweights which joints matter, whereas MCR refines how strongly co-moving joints should be connected. Equipping the TE-STGCN baseline with the Motion-Aware Joint Attention (MJA) module and the Motion-Correlated Graph Refinement (MCR) module jointly improves overall accuracy from 92.3% to 93.2% and Hard10 average from 66.5% to 69.8%. First, the Hard10 gain is approximately 3.7× the overall gain, indicating that the motion-aware modules disproportionately benefit the most difficult actions—precisely those requiring localized joint discrimination, such as “touch head” and “play phone”—rather than whole-body movements that the baseline already handles well. Second, because MJA and MCR share the same motion energy statistic ${\mathbf{e}}_v$ as their driving signal, they incur negligible additional parameter overhead while providing complementary benefits: MJA reweights which joints matter, whereas MCR refines how strongly those joints should be connected in the graph. This shared-computation design explains why the two modules are applied as a single configuration in the ablation rather than isolated.

Effect of DropPath. Adding Stochastic Depth further raises overall accuracy to 93.8% and Hard10 average to 70.9%. The consistent improvement on both metrics suggests that the regularization effect of randomly deactivating residual paths generalizes beyond the easy-to-classify majority classes and also aids the learning of rare, fine-grained motion patterns. Notably, DropPath provides a larger absolute gain on Hard10 than on overall accuracy, implying that the overfitting problem it addresses is more severe for challenging actions where training samples are sparse and discriminative cues are subtle. We also evaluate the impact of DropPath on the results under different parameter settings, as shown in Table 7. We can find that $p_{max}=0.1$ is optimal. Smaller values under-regularize the 10-block network, while larger values such as 0.2–0.3 discard too much of the residual signal and degrade both metrics, with the hard classes suffering first.

Table 7. The impact of maximum DropPath probability on the results (EMA disabled). Best values are in bold and second-best underlined.

${\mathit{p}}_{\mathit{m}\mathit{a}\mathit{x}}$	Acc (%)	Hard10 (%)
0	93.2	69.8
0.05	93.7	70.4
0.1	93.8	70.9
0.2	93.5	70.2
0.3	93.1	69.5

Effect of EMA. The introduction of Exponential Moving Average (EMA with decay 0.999) yields the final improvement to 94.2% overall. While the overall gain is the smallest among all components, the Hard10 improvement is the largest single-step contribution to the difficult-class metric after MJA+MCR. This indicates that weight-averaging is particularly effective at stabilizing predictions for ambiguous actions whose decision boundaries are sensitive to late-training weight fluctuations. We also evaluate the impact of EMA on the results under different parameter settings, as shown in Table 8. It can be seen that $\alpha =0.999$ is optimal. A shorter memory 0.99 averages over too few recent steps to smooth late-training fluctuations, while an excessively long memory 0.9999 is dominated by stale early weights. The EMA benefit is largest precisely on the Hard10 metric, consistent with our interpretation that weight-averaging stabilizes decision boundaries of ambiguous actions.

Table 8. The impact of EMA decay on the results (DropPath 0.1 enabled). Best values are in bold and second-best underlined.

$\mathit{\alpha }$ (EMA)	Acc (%)	Hard10 (%)
0	93.8	70.9
0.99	94.0	71.4
0.999	94.2	72.1
0.9999	94.1	71.8

Across the four configurations, every component contributes positively to both metrics, and no component diminishes the gain of another, confirming their mutual complementarity. The total improvement constitutes the central finding of this ablation: the performance advantage of MA-STGCN is concentrated on the actions that are hardest for existing methods, which is exactly the regime where motion-aware joint modeling and graph refinement are most needed.

4.7. Graph Refinement Analysis

To analyze the effect of MCR, we examine the learned refinement parameter $\delta$ across the 10 STGCN blocks and the structure of the motion correlation matrix $\mathbf{S}$.

Block-wise $\delta$ values: The learned $\delta$ values show a clear pattern; early blocks (1–4, $C=64$) have small $\delta$ values (≈0.01–0.03), while later blocks (7–10, $C=256$) have larger values (≈0.08–0.15). This is consistent with the observation that early blocks learn low-level motion patterns shared across body parts (where motion correlation is less discriminative), while later blocks encode action-specific semantics where motion-correlated edges provide the most benefit.

Correlation matrix structure: For different actions, the motion correlation matrix $\mathbf{S}$ exhibits distinct patterns. For bilateral actions (clapping, rubbing hands), the inter-arm correlation is high. For unilateral actions (typing, writing), the right-arm–torso correlation is emphasized. For leg-dominant actions (kicking, sitting down), the intra-leg correlation dominates. These patterns demonstrate that MCR adapts the graph topology to the specific action being performed, providing a level of flexibility that fixed or channel-wise topology cannot achieve.

4.8. Confusion Analysis

Table 9 shows the top-five confusion pairs, revealing that the remaining errors are primarily caused by the fundamental limitation of 25-joint skeletons (no finger joints) rather than model architecture.

Table 9. Top-5 confusion pairs in MA-STGCN.

True → Predicted	Samples	Ratio
Writing → Typing	64	23.2%
Pointing → Selfie	49	17.8%
Clapping → Rub hands	47	17.0%
Writing → Reading	41	14.9%
Reading → Play phone	40	14.5%

The writing → typing confusion (23.2%) stems from both actions involving similar wrist trajectories, which cannot be distinguished without finger-level joint information. Similarly, the clapping → rub_hands confusion arises from both being bilateral hand interaction actions with minimal spatial difference. While MCR strengthens the inter-wrist connection for both actions, the 25-joint skeleton lacks the finger articulation needed to fully resolve these ambiguities.

In the clapping → rubbing-hands confusion, the two actions differ in the temporal regularity of the inter-wrist motion. Clapping consists of symmetric, repetitive in–out wrist movements with a quasi-periodic oscillation of the inter-wrist distance, typically 2–3 Hz, whereas rubbing hands involves sustained, non-periodic small-amplitude contact motion. We verified this by computing the inter-wrist distance time series for both classes. The clapping sequence shows a dominant spectral peak, with an average periodicity score of 0.81, while rubbing-hands sequences have a flat spectrum with an average periodicity score of 0.23. We then examined the model’s predictions on the misclassified samples and discovered two reasons for the failure. First, the motion-energy descriptor ${\mathbf{e}}_v$ is a per-joint amplitude statistic computed within each GCN block, whose temporal receptive field after stride-2 downsampling spans roughly 1–2 s, the periodicity cue is therefore attenuated but not absent, which explains why the confusion is reduced rather than eliminated. Second, the residual ambiguity is dominated by the missing finger signal—during actual contact, clapping produces wrist-level impact reversals while rubbing produces continuous sliding. This difference can only be observed during the action of finger joints, and is invisible in the skeleton structure of the 25 joints. In short, periodicity is a partially exploited cue that our current descriptor captures only implicitly, and finger absence accounts for the remaining irreducible error.

5. Discussion

5.1. Why Motion Statistics Outperform Mean Pooling

The standard squeeze operation in SE-based attention modules uses global average pooling (mean over time), which collapses the temporal dimension into a single average value per channel per joint. However, action recognition fundamentally depends on how joints move over time, not on their average position. Two actions may have identical average joint positions but vastly different temporal dynamics, such as “standing still” vs. “treading in place”. By using temporal variance as the squeeze signal, MJA directly captures the temporal dynamics that are most discriminative for action categorization. Our experiments confirm this: replacing mean pooling with motion-energy extraction yields +0.9% overall accuracy and +3.3% on hard classes.

5.2. Complementarity of MJA and MCR

MJA and MCR address different aspects of the motion-awareness problem. MJA operates on individual joints, answering “Which joints are moving saliently?” MCR operates on pairs of joints, answering “Which joints are moving in coordinated patterns?” These two questions are complementary: an action may involve a single salient joint (where MJA suffices) or a coordinated pattern across multiple joints (where MCR provides additional signal). The shared motion-energy computation makes this complementarity efficient, as both modules derive their input from the same temporal statistics.

5.3. Limitations

Because the term “fine-grained” is defined differently across the action-recognition literature, we first fix its operational meaning. In this paper, “fine-grained” refers to the discrimination of actions that share similar gross motor patterns and object contexts but differ in how the body parts are engaged. For this level of granularity, the informative cue is the coordination pattern across anatomical partitions—such as which parts remain static, which parts move, and how their trajectories co-vary—rather than the geometry of individual fingers or facial muscles. Consequently, the standard 25-joint skeleton is sufficient for the fine-grained cases we target, and the absence of finger or facial keypoints does not invalidate this claim. What we do not claim is spatial fine-grained ability at finger or facial precision or temporal fine-grained recognition of atomic sub-actions. MA-STGCN has yet to be evaluated on datasets designed for those granularities, such as BEAR (temporal) and NTU-X (finger or facial augmented). The NTU RGB+D benchmarks used here are whole-body daily actions rather than multi-step atomic tasks, and their 25-joint skeletons contain no finger or facial keypoints. Evaluating on those benchmarks would require non-trivial adaptations—extending the joint topology, re-defining the binary partition masks for the added joints (NTU-X), and relying on denser temporal annotations (BEAR), which is therefore left as future work.

In addition, MA-STGCN has other limitations. First, the 25-joint skeleton’s lack of finger joints remains a fundamental bottleneck for hand-centric actions (writing, typing, reading). Second, the current evaluation focuses on the bone stream; multi-stream fusion results are expected but not yet available. Third, the motion-energy descriptor is computed independently per block, which may not capture long-range temporal dependencies across blocks. Future work will address these limitations through finger joint augmentation, multi-stream training, and cross-block motion aggregation.

6. Conclusions

In the field of skeleton-based action recognition, existing methods treat all skeletal joints uniformly, failing to emphasize task-relevant joints for fine-grained actions. Therefore, we present MA-STGCN, a topology-enhanced spatio-temporal GCN augmented with a unified motion-aware framework. The Motion-Aware Joint Attention module derives per-joint importance weights from temporal motion statistics, enabling the model to dynamically emphasize motion-salient joints. The Motion-Correlated Graph Refinement module constructs a sample-specific inter-joint motion correlation matrix and uses it to adaptively refine graph edges, strengthening connections between co-moving joints. Experiments on the NTU RGB+D 60 and NTU RGB+D 120 datasets demonstrate the superiority of MA-STGCN. Our method obtains an accuracy of 94.2% on NTU60 and 91.3% on NTU120, outperforming strong competitors, including Hyper-GCN (93.7%/90.5%) and the TE-STGCN baseline (92.3%/88.3%).