BPA-STGCN: Body-Part-Aware Spatio-Temporal Graph Convolutional Network for Stable Skeleton-Based Action Recognition

Abstract

Skeleton-based action recognition via graph convolutional networks (GCNs) has achieved remarkable progress, yet two persistent bottlenecks limit practical deployment: (1) systematic confusion among fine-grained actions that differ primarily in hand or finger movements, which the standard 25-joint skeleton cannot disambiguate; and (2) training instability under small batch sizes caused by BatchNorm (BN) running-statistics pollution, leading to catastrophic accuracy drops during training. In this paper, we propose BPA-STGCN (Body-Part-Aware STGCN), which addresses both challenges through an integrated framework of architectural and training innovations. First, a Partition Attention (PA) module adapting the Squeeze-and-Excitation concept to anatomically defined joint groups that decomposes the 25-joint skeleton into four anatomical partitions and learns sample-specific importance weights for each partition, enabling the model to focus on the most discriminative body region for each action. Then, the information losing global average pooling is replaced by a Temporal Pyramid Pooling (TPP) module adapting the temporal-segment and pyramid-pooling concepts to skeleton feature maps that captures multi-scale temporal dynamics through segmented pooling. Moreover, we design a stability-first training protocol comprising low-momentum BN, Mixup augmentation, gradient clipping, and extended warmup. The experiments are performed on the NTU RGB+D 60 and NTU RGB+D 120 dataset, and BPA-STGCN achieves 93.7% and 90.9% accuracy. Comprehensive ablation studies reveal that the architectural innovations and the stability protocol contribute complementarily, and that BPA-STGCN achieves the best accuracy–stability trade-off among all tested configurations.

1. Introduction

Skeleton-based action recognition has become a central topic in human activity understanding, driven by the availability of depth sensors and large-scale datasets such as NTU RGB+D [1]. Spatio-temporal graph convolutional networks (ST-GCNs) [2] model the skeleton as a graph with spatial edges defined by body connectivity and temporal edges connecting the same joint across frames, achieving state-of-the-art performance. Subsequent works improved topology learning through adaptive graphs 2s-AGCN [3], channel-wise topology refinement CTR-GCN [4], and dynamic graph construction. The TE-STGCN [5] framework enhanced spatial relation encoding through graph distance-based weights, multi-range attention, and static/dynamic spatial relation encoding. Despite this progress, two fundamental challenges remain:

Challenge 1: Fine-grained action confusion. The standard 25-joint skeleton representation captures major body joints but lacks finger-level articulation. The Kinect v2 skeleton contains no finger joints, so no model operating on this representation can learn fine manual kinematics and it can only rely on coarse postural correlations around the hands and wrists. This causes systematic confusion among actions that differ primarily in hand or finger movements: “writing” is frequently misclassified as “typing”, “reading” as “writing”, and “clapping” as “rub hands”. These confusions persist across existing methods, suggesting a representational bottleneck [6]. Importantly, many of these fine-grained actions can be distinguished by non-hand body cues [7]. For instance, “typing” involves a more upright torso posture and stable legs compared to “writing” which may involve leaning forward. Existing GCN methods treat all joints uniformly within the graph, without explicitly modeling body-part-level importance. This observation motivates our strategy: rather than attempting to recover missing finger information, we exploit discriminative cues from other body parts through explicit part-level attention.

Challenge 2: Training instability under resource constraints. Modern ST-GCN models rely on BatchNorm [8] for feature normalization. When GPU memory limits the batch size to 16 or fewer samples, BN running statistics become highly sensitive to batch composition. Our prior work showed that combining small batch sizes with class-weighted sampling leads to severe training instability: 12 out of 90 epochs experienced validation accuracy drops below 40%, with a standard deviation of 18.46%. Such instability makes model training unpredictable, wastes computational resources, and hinders reproducibility.

To address both challenges simultaneously, we propose BPA-STGCN (Body-Part-Aware STGCN) to integrate architectural innovations for fine-grained discrimination with a training protocol for stability, which is deliberately framed as architectural adaptation and empirical training engineering on an existing GCN backbone.

First, the Partition Attention (PA) module is proposed to focus on the most discriminative body parts. Body-part-based representations have been explored in early skeleton-based methods. Part-aware LSTM [9] is used for capturing long-term temporal correlations of body parts. ST-GCN’s partitioning strategy defines three spatial partition sets (root, centripetal, centrifugal) for graph convolution [10]. However, these approaches use fixed, non-learnable partitions without adaptive importance weighting [11]. Our Partition Attention module differs by introducing learnable, sample-specific importance weights for anatomically defined body partitions, enabling dynamic focus on the most discriminative regions [12,13]. We decompose the 25-joint skeleton into four anatomically meaningful partitions, including torso (9 joints), left arm (5 joints), right arm (5 joints), and legs (6 joints) and learn sample-specific importance weights for each partition through a squeeze-activation mechanism. The partition weights are mapped back to individual joints via a fixed binary mask, enabling the model to dynamically focus on the most discriminative body region. For example, “typing” and “writing” can be better distinguished by attending to differences in torso posture and leg stability, even when right-arm trajectories are similar.

Then, the Temporal Pyramid Pooling (TPP) module is proposed to capture the differences of time series. Global average pooling (GAP) is the standard temporal aggregation method in skeleton-based recognition. Spatial Pyramid Pooling (SPP) [14] is originally proposed for image classification to capture multi-scale spatial context. Temporal Pyramid Pooling has been used in video understanding [15] but has not been systematically applied to skeleton-based GCNs. Our TPP adapts the pyramid pooling concept to the temporal dimension of skeleton features, providing multi-scale temporal context to the classifier with minimal parameter overhead. We replace the standard global average pooling with a multi-scale temporal representation. By segmenting the temporal dimension into one, two, and four segments and pooling each independently, TPP captures both global temporal context and local temporal dynamics. This helps distinguish actions with similar overall motion but different temporal patterns, such as the periodic rapid keystrokes of “typing” vs. the slow continuous motion of “writing”.

Finally, the stability-first training protocol is presented to ensure the stability of the training process. BatchNorm [8] instability under small batch sizes is a well-known issue in deep learning. GroupNorm [16] and LayerNorm [17] provide batch-size-independent alternatives but may lose the regularization effect of batch statistics. Mixup [18] creates interpolated samples for smoother decision boundaries. Stochastic depth [19] and DropPath regularize deep networks by randomly dropping layers. Exponential Moving Average (EMA) maintains a smoothed copy of model parameters. In skeleton-based recognition, these techniques have been applied in isolation without systematic analysis of their interactions. Our work provides the first comprehensive study of stability training techniques specifically for skeleton-based GCNs, revealing important trade-offs between training stability and peak accuracy. We systematically address BN instability through: (a) reduced BN momentum; (b) Mixup data augmentation for smoother decision boundaries; (c) gradient clipping to prevent large gradient updates from destabilizing BN statistics; and (d) extended warmup for BN stabilization.

Our main contributions are summarized as follows:

• A Partition Attention module that adapts the Squeeze-and-Excitation concept to anatomically defined joint groups for discriminative feature learning, improving fine-grained action accuracy by up to 17.4% on individual classes.
• A Temporal Pyramid Pooling module, adapting temporal-segment and pyramid-pooling concepts to skeleton feature maps, that captures multi-scale temporal dynamics with negligible parameter overhead, replacing the information-losing global average pooling.
• A comprehensive stability-first training protocol that reduces training crashes and increases the number of stable epochs.
• Extensive experiments and ablation studies on NTU RGB+D 60 and NTU RGB+D 120 demonstrating that BPA-STGCN achieves the best accuracy–stability trade-off.

2. Materials and Methods

2.1. Dataset

We evaluate on the NTU RGB+D 60 dataset which contains 56,880 skeleton sequences of 60 action classes performed by 40 subjects and NTU RGB+D 120 dataset which contains 114,480 sequences across 120 action categories performed by 106 subjects. We use the cross-subject (X-Sub) benchmark. The standard 25-joint skeleton from the Kinect v2 sensor (Microsoft Corporation, Redmond, Washington, DC, USA) is used. Four input streams are employed: joint, bone, joint velocity, and bone velocity. Here, we focus on the bone stream for detailed analysis, as it achieves the highest single-stream accuracy in the baseline.

Both datasets are collected by their original authors and are fully public. We summarize the subject demographics reported in the dataset publications [1,8] in Table 1. NTU RGB+D 60 recruits 40 subjects aged 10–35 years and NTU RGB+D 120 extends the pool to 106 subjects of both genders with a similar age range.

Table 1. Subject demographics.

Dataset	Number of Subjects	Age Range (Years)	Gender	Camera	Number of Classes/Sequences
NTU RGB+D 60	40	10–35	Male and female	Kinect v2 (3 viewpoints)	60/56,880
NTU RGB+D 120	106	10–40	Male and female	Kinect v2 (32  setups)	120/114,480

2.2. Skeleton Acquisition and Selection

All skeletons in both datasets are obtained by the Kinect v2 sensor from depth maps using its built-in pose-estimation pipeline. Kinect v2 tracks 25 body joints from the depth channel with a per-joint tracking state and confidence: each joint is reported as “tracked”, “inferred”, or “not tracked”. This mechanism is precisely what allows skeletons to be captured when body parts are partially hidden: the tracker interpolates occluded joints from the visible configuration of the rest of the body using an articulated human body model. The procedure is shown as Figure 1. Because the Kinect v2 skeleton operates on the depth channel rather than RGB, it is inherently insensitive to shadows and to most clothing color/texture variations; loose clothing, however, can bias the depth silhouette and thus the inferred joint positions, and strong body-build differences (muscular or thin physiques) shift the apparent depth of limbs. The dataset itself mitigates these factors: the recording protocol required each subject to wear daily, non-occluding clothing and to remain within the sensor range (0.8–4.5 m), where the depth accuracy is highest. For the residual noise and partial occlusions, we apply the following simple but effective selection and preprocessing procedure before training:

Figure 1. The process of obtaining the skeleton.

• Selection by tracking quality: sequences are retained only if at least 85% of frames have the torso joints in the “tracked” state; sequences with severe torso occlusion are discarded, since the torso is the root of the body model and its corruption cannot be repaired reliably.
• Per-joint interpolation: for joints momentarily reported as “inferred” or missing, positions are linearly interpolated across neighboring tracked frames; segments longer than 10 consecutive missing frames are marked invalid and excluded from the crop.
• Confidence-weighted zero-centering: the skeleton is centered on the spine-base joint and normalized by torso length, so that global body proportions and standing positions do not influence the learned features.
• Person selection: in two-person sequences, the person with the higher average tracking confidence is designated as the main actor, and both skeletons are fed to the input stage as in standard practice.

2.3. Overall Architecture

BPA-STGCN builds upon the TE-STGCN backbone with 10 spatio-temporal GCN blocks. Each block contains a channel-wise topology refinement graph convolution (CTR-GCN) [4] with static spatial relation encoding (SSRE), multi-range attention (MRA), joint-level attention, and stochastic depth (DropPath). Then, the Partition Attention (PA) and Temporal Pyramid Pooling (TPP) are proposed to capture the joint features. Finally, the model is trained with a stability-first protocol. It is shown in Figure 2 and expressed as:

$$\mathbf{Z}=f_{\mathrm{blocks}+\mathrm{PA}}(\mathbf{X},\mathbf{A}),\mathbf{p}=\mathrm{FC}\left(\mathrm{TPP}\left(\mathbf{Z}\right)\right)$$

(1)

where $\mathbf{X}\in {\mathbb{R}}^{N\times M\times C\times T\times V}$ is the input skeleton (N: batch, M: subjects, C: channels, T: frames, V: joints), $\mathbf{A}\in {\mathbb{R}}^{K\times V\times V}$ is the adjacency matrix, PA is applied to the last 3 blocks, and TPP replaces global average pooling.

Figure 2. Overall architecture of BPA-STGCN.

The spatial graph convolution is applied to skeleton sequences, adding a temporal convolution along the frame axis so that both spatial and temporal patterns are captured in a single differentiable block. The building blocks of BPA-STGCN are all members of this extended convolution family, and their common features are what make them composable: (i) local aggregation—every operator aggregates information from a bounded neighborhood, which matches the physical fact that body joints influence each other mainly through anatomically adjacent or interacting parts; (ii) weight sharing and translation invariance—kernels are shared across positions, which encodes the assumption that motion patterns recur at arbitrary locations in space or time; (iii) hierarchical feature composition—stacking operators yields receptive fields that grow from single joints to body parts to whole-body configurations, mirroring the semantic hierarchy from low-level motion primitives to action classes; and (iv) learnable attention reweighting—attention mechanisms modulate the aggregated features, effectively making the convolution kernels input-conditioned. In particular, the convolution-family operators include the 1 × 1 pointwise convolution inside the attention bottlenecks, CTR-GCN input-adaptive graph convolution, 1-D temporal convolution along frames, and multi-scale temporal pooling. Their four common features in the skeleton-analysis context are local aggregation, weight sharing/translation invariance, hierarchical feature composition, and learnable attention reweighting.

Three specifics of the skeleton-based action recognition task shape the deep-learning design choices. First, the input is extremely compact, so the model capacity must be spent on relational reasoning rather than on raw feature extraction. Second, the effective dataset per subject and per camera viewpoint is small, so training statistics are noisy, in particular the BatchNorm running statistics. This is the origin of the training-instability problem that our stability protocol targets. Third, the classes that are hardest to separate differ in subtle, part-local, and time-local cues, which is precisely what part-level attention and multi-scale temporal pooling are designed to expose. These three specifics together define an original, but somewhat specialized, learning problem: a small relational-input, small-batch, fine-grained classification task, for which we assemble known convolution-family components into a coherent architecture and training recipe.

The main scheme for using the Partition Attention method is as follows. First, the input skeleton sequence passes through 10 stacked STGCN blocks. In blocks 1–7, standard CTR-GCN graph convolution with joint attention is applied. Second, the PA branch is inserted in blocks 8–10 and the block output feature F is first squeezed over the temporal and joint dimensions into a channel descriptor s. In this module, a two-layer bottleneck maps s to four partition weights. The fixed binary partition mask expands the four weights into 25 joint weights and the features are rescaled with a zero-initialized learnable gate, so the block behaves identically to the backbone at the beginning of training. After the final block and person averaging, TPP replaces global average pooling. The temporal axis is divided into 1, 2, and 4 segments, each segment is average-pooled independently, and the resulting 7 vectors are concatenated into the classifier input. We additionally note that the skeleton illustration inside Figure 2 depicts only 15 of the 25 joints for visual clarity (the 25-joint Kinect v2 definition is given in Table 2). The omitted joints are the finger-level stubs (left/right hand tips 21/24 and thumbs 22/23) and secondary spine/ankle detail joints, which are difficult to render legibly at figure scale. The implementation itself uses all 25 joints.

Table 2. Body partition definitions for the NTU 25-joint skeleton.

Partition	#Joints	Ratio	Joint Indices
Torso	9	36%	0, 1, 2, 3, 4, 8, 12, 16, 20
Left arm	5	20%	5, 6, 7, 21, 22
Right arm	5	20%	9, 10, 11, 23, 24
Legs	6	24%	13, 14, 15, 17, 18, 19

2.4. Partition Attention Module

The PA module leverages the anatomical structure of the human skeleton to learn body-part-specific importance weights. We define four partitions of the NTU 25-joint skeleton, as shown in Table 2. The joint indices are the zero-based indices of the Kinect v2/NTU RGB+D 25-joint ordering, with 0 = spine base, 1 = spine mid, 2 = neck, 3 = head, 4 = left shoulder, 5 = left elbow, 6 = left wrist, 7 = left hand, 8 = right shoulder, 9 = right elbow, 10 = right wrist, 11 = right hand, 12 = left hip, 13 = left knee, 14 = left ankle, 15 = left foot, 16 = right hip, 17 = right knee, 18 = right ankle, 19 = right foot, 20 = spine shoulder, 21 = left hand tip, 22 = left thumb, 23 = right hand tip, 24 = right thumb. These indices refer to the positions of the joints in the input tensor along the V dimension.

Given a feature tensor $\mathbf{F}\in {\mathbb{R}}^{N\times C\times T\times V}$, the PA module computes partition weights as follows:

Squeeze: Global average pooling over feature $\mathbf{F}$ of spatial-temporal dimensions:

$$\mathbf{s}={\mathrm{GAP}}_{T,V}\left(\mathbf{F}\right)\in {\mathbb{R}}^{N\times C\times 1\times 1}$$

(2)

Excitation(activation): The activation refers to the excitation stage of the squeeze-and-excitation mechanism, such as the process by which the squeezed global descriptor is transformed, through a small bottleneck network, into the four partition weights $w_p$. Concretely, a dimensionality-reduction layer $W_1$ compresses the channel descriptor from C to $C/4$ channels, a ReLU nonlinearity is applied, and a dimensionality-expansion layer $W_2$ maps back to 4 channels. The final sigmoid maps each channel to (0, 1), producing the importance weight of each partition. A bottleneck network maps the squeezed descriptor to partition weights:

$${\mathbf{w}}_p=\sigma \left({\mathbf{W}}_2·\mathrm{ReLU}({\mathbf{W}}_1·\mathbf{s})\right)\in {\mathbb{R}}^{N\times 4}$$

(3)

where ${\mathbf{W}}_1\in {\mathbb{R}}^{C/4\times C}$, ${\mathbf{W}}_2\in {\mathbb{R}}^{4\times C/4}$, and $\sigma$ is the sigmoid function.

Partition-to-joint mapping: Partition weights are mapped to joint-level weights using a fixed binary mask $\mathbf{M}\in {\{0,1\}}^{V\times 4}$:

$${\mathbf{w}}_j={\mathbf{w}}_p·{\mathbf{M}}^{\top }\in {\mathbb{R}}^{N\times V}$$

(4)

where $M_{v,k}=1$ if joint v belongs to partition k.

Residual scaling: The joint weights are applied through a learnable residual:

$${\mathbf{F}}^{\prime }=\mathbf{F}\odot (1+\gamma ·{\mathbf{w}}_j)$$

(5)

where $\gamma$ is a learnable scalar initialized to zero, ensuring identity behavior at the start of training. This initialization strategy allows PA to be seamlessly inserted into a pre-trained backbone without disrupting early training.

The PA module is applied only to the last 3 STGCN blocks, which process semantic-level features where body-part discrimination is most meaningful. This is motivated by the observation that early blocks learn low-level motion patterns shared across all body parts, while later blocks encode action-specific semantics that benefit from partition-aware attention.

2.5. Temporal Pyramid Pooling

Standard global average pooling aggregates all temporal information into a single vector, discarding temporal structure. The TPP module captures multi-scale temporal dynamics by partitioning the temporal dimension into segments of varying granularity:

Given the feature tensor $\mathbf{Z}\in {\mathbb{R}}^{N\times C\times T\times V}$ after the final STGCN block and person averaging, TPP operates as:

$${\mathbf{z}}_k^{\left(s\right)}={\displaystyle \frac{1}{|S_k^{\left(s\right)}|·V}}\sum _{t\in S_k^{\left(s\right)}}{\displaystyle \sum _{v=1}^V}{\mathbf{Z}}_{:,:,t,v}$$

(6)

for the k-th segment of an s-segment partitioning, where $s\in \{1,2,4\}$. The final representation concatenates all pooled vectors:

$${\mathbf{z}}_{\mathrm{TPP}}=[{\mathbf{z}}_1^{\left(1\right)};{\mathbf{z}}_1^{\left(2\right)},{\mathbf{z}}_2^{\left(2\right)};{\mathbf{z}}_1^{\left(4\right)},\dots ,{\mathbf{z}}_4^{\left(4\right)}]\in {\mathbb{R}}^{7C}$$

(7)

With $C=256$, the output dimension is $7\times 256=1792$, directly fed to the classification FC layer.

Main features of TPP and advantages over global average pooling are as follows.

• Multi-scale temporal granularity. The 1-segment branch preserves the global average; the 2-segment branch resolves the coarse two-phase structure of actions such as “vomit”: bending then retching; the 4-segment branch captures fine periodic patterns such as “typing”: rapid keystroke cycles. GAP provides only the first of these three views.
• Order preservation. The concatenated segment vectors encode the chronological order of motion phases, whereas GAP is a permutation-invariant average over time: a sequence and its temporally shuffled version produce identical GAP features, but different TPP features. This is decisive for actions whose phase order is the only discriminative cue.
• Amplitude sensitivity. Segment-wise pooling preserves the local magnitude of motion in each time window; GAP dilutes a short, intense burst of activity over the whole sequence length, so two sequences with identical averages but very different temporal energy profiles become indistinguishable.
• Negligible cost. TPP is a parameter-free re-arrangement of pooling windows: it adds only the widened classifier input, so its gains are attributable to representational improvement rather than capacity increase.

2.6. Stability-First Training Protocol

We identify BN running-statistics pollution as the primary cause of training instability in skeleton-based GCNs. Under small batch sizes ($B=16$), the running mean $\mu$ and variance ${\sigma }^2$ are updated as:

$${\mu }_{\mathrm{run}}\leftarrow (1-m)·{\mu }_{\mathrm{run}}+m·{\mu }_{\mathrm{batch}}$$

(8)

With default momentum $m=0.1$, a single batch with skewed class distribution (e.g., from weighted sampling) can shift the running statistics by up to 10%, causing validation accuracy to collapse. Our stability protocol addresses this through four complementary mechanisms:

(1) Low-momentum BatchNorm ($m=0.01$): Reducing the momentum by 10× makes running statistics 10× less sensitive to individual batch composition. This is the single most impactful change for training stability, reducing crash epochs from 12 to 7.

(2) Mixup augmentation ($\alpha =0.2$, 50% probability): Each batch is mixed with probability 0.5 as:

$$\tilde{\mathbf{x}}=\lambda {\mathbf{x}}_a+(1-\lambda ){\mathbf{x}}_b,\lambda \sim \mathrm{Beta}(\alpha ,\alpha )$$

(9)

The mixed loss is $\tilde{L}=\lambda L\left({\mathbf{x}}_a\right)+(1-\lambda )L\left({\mathbf{x}}_b\right)$. With $\alpha =0.2$, $\lambda$ typically falls in $[0.5,1.0]$, creating mild mixtures that smooth decision boundaries. An important side effect is that Mixup makes training harder (lower training accuracy) but improves generalization (higher validation accuracy), as we observe a negative train–val gap of $-16.35\%$.

(3) Gradient clipping (${\parallel g\parallel }_{\max }=1.0$): After gradient unscaling and before the optimizer step, gradients are clipped:

$$\mathbf{g}\leftarrow {\displaystyle \frac{\mathbf{g}}{\max (1,\parallel \mathbf{g}\parallel /1.0)}}$$

(10)

This prevents large gradient updates from destabilizing BN statistics, particularly during the warmup phase when learning rates ramp up.

(4) Extended warmup (10 epochs): Linear warmup from 0 to the initial learning rate over 10 epochs, providing BN running statistics more time to stabilize before aggressive learning.

Additionally, we remove the WeightedRandomSampler used in prior work [5], allowing BatchNorm to observe the natural class distribution. This eliminates the primary source of batch composition skew. We also apply Exponential Moving Average (EMA) with decay 0.999 to maintain a smoothed copy of model parameters, which is used for validation and checkpoint saving. Stochastic depth (DropPath) with a maximum rate of 0.15 (linearly increasing from 0 to 0.15 across the 10 blocks) provides additional regularization.

3. Results

The skeleton sequences are subsampled to 150 frames. Then, the mild augmentation is applied, including random scaling (0.9–1.1), random rotation (±0.15 rad), and temporal cropping (ratio 0.95). The optimizer is SGD with Nesterov momentum (lr = 0.1, momentum = 0.9, weight decay $4\times {10}^{-4}$). Cosine annealing [20] is used with $T_{\max }=90$ and ${\eta }_{\min }=0.0001$. Batch size is 16 with gradient accumulation 2 (effective batch size 32). Mixed precision (AMP) is used. The model is an improvement based on TEGCN is trained on a single NVIDIA RTX 4060 Ti (8GB).

3.1. Overall Results

State-of-the-art. We compare BPA-STGCN against STGCN [2], BlockGCN [21], DS-GCN [22], HAM-HGNet [23], $3\mathrm{s}-{\mathrm{S}}^2\mathrm{FD}$ [24], MAR-GCN [25], TE-STGCN [5]. Table 3 compares BPA-STGCN with the baseline and other state-of-the-art methods on the bone stream.

Table 3. Overall comparison on NTU60 and NTU120 X-Sub in bone stream (Bold font indicates the method that yields the best results and the corresponding accuracy).

Methods	Accuracy (%)
	NTU60	NTU120
STGCN	81.5	77.7
BlockGCN	93.1	90.3
DS-GCN	93.1	89.2
HAM-HGNet	93.1	90.5
$3\mathrm{s}-{\mathrm{S}}^2\mathrm{FD}$	86.6	77.1
MAR-GCN	92.3	87.6
TE-STGCN	92.3	88.3
BPA-STGCN	93.7	90.9

It summarizes the comparison between the proposed BPA-STGCN and several representative skeleton-based action recognition methods on the NTU60 and NTU120 benchmarks under the X-Sub evaluation protocol. The STGCN achieves only 81.5% and 77.7% accuracy on NTU60 and NTU120, respectively, indicating that its fixed, hand-crafted partition strategy and uniform spatial-temporal modeling are insufficient to capture the complex non-local dependencies inherent in human skeleton sequences. With the introduction of learnable adjacency matrices and block-wise dynamic topology, BlockGCN substantially improves the accuracy to 93.1% on NTU60 and 90.3% on NTU120, establishing a strong competitive baseline. DS-GCN and HAM-HGNet further exploit dynamic spatial modeling and hypergraph-based high-order semantics, achieving comparable or marginally higher performance on NTU60 (both 93.1%) and 90.5% on NTU120, demonstrating the benefit of capturing higher-order joint relationships. The frequency-domain variant $3\mathrm{s}-{\mathrm{S}}^2\mathrm{FD}$ and the attention-refined MAR-GCN and TE-STGCN report 86.6%/77.1%, 92.3%/87.6%, and 92.3%/88.3% on the two datasets, respectively, showing that spectral and attention cues provide complementary but still limited gains when used in isolation. In contrast, the proposed BPA-STGCN, which jointly integrates body-part-aware partitioning with adaptive topology enhancement, achieves the best accuracy of 93.7% on NTU60 and a highly competitive 90.9% on NTU120, outperforming the strongest competing method by 0.6% on NTU60 while maintaining comparable performance on the more challenging NTU120 benchmark. These results consistently validate the effectiveness and generalization capability of the proposed BPA-STGCN across datasets of different scales.

3.2. Hyperparameter Sensitivity Analysis

We analyze seven key hyperparameters of BPA-STGCN on NTU60. Table 4 summarizes the sensitivity of training stability and accuracy to each hyperparameter. For each 90-epoch training run with one hyperparameter configuration, we compute the per-epoch validation accuracy of the EMA model on the X-Sub validation split. There are two criteria of “stability”. The first one is crash epochs, which is the number of post-warmup epochs (epochs 11–90) in which validation accuracy drops by more than 15 percentage points below the running maximum seen so far. The second one is stable epochs, which is the number of epochs in which validation accuracy is at least 80% and within 3 points of the running maximum. The two criteria are complementary: “Crash” counts catastrophic failures, “Stable” counts the epochs a practitioner could safely use for checkpointing. The thresholds are fixed before the sweep and reflect the failure modes observed in our preliminary runs.

Table 4. Results of different hyperparameter settings (Bold font indicates the value that yields the best results).

Hyperparameter	Value	Accuracy	Crash	Stable
BN Momentum	0.1	93.4%	6	9
	0.05	93.2%	3	20
	0.01	93.7%	0	34
Mixup $\alpha$	0	93.4%	2	15
	0.1	93.8%	1	25
	0.2	93.7%	0	34
	0.4	92.0%	0	30
DropPath Rate	0	92.8%	3	8
	0.1	93.5%	6	9
	0.15	93.7%	0	34
	0.25	91.9%	0	30
Warmup Epoch	5	92.8%	0	10
	10	93.7%	0	34
	15	93.3%	0	32
Gradient Clip	None	93.8%	6	9
	0.5	93.4%	0	30
	1.0	93.7%	0	34
	2.0	93.5%	1	25
EMA Decay	0	92.8%	0	10
	0.99	93.3%	2	20
	0.999	93.7%	0	34
Learning Rate	0.05	92.4%	0	25
	0.1	93.7%	0	34
	0.2	90.8%	4	5

BatchNorm Momentum: $m\in \{0.1,0.05,0.01\}$. The BN momentum is the single most impactful hyperparameter for training stability. With $m=0.1$, the running statistics update at a rate of 10% per batch, making them highly sensitive to individual batch composition. It experiences six post-warmup crash epochs. Reducing m to 0.01 makes running statistics update at only 1% per batch and completely eliminates all post-warmup crashes. The last-10-epoch standard deviation drops from 13.11% to 1.48%, a 8.9× improvement in late-training stability.

Mixup Augmentation: $\alpha \in \{0,0.1,0.2,0.4\}$. Mixup is the most distinctive hyperparameter in BPA-STGCN, producing a unique negative train–validation gap. Without Mixup, the training accuracy exceeds validation accuracy by 4.96% and 6.09%, respectively, indicating mild overfitting. With Mixup $\alpha =0.2$, the training accuracy is lower than validation accuracy. This means the model performs better on clean data than on the augmented training data, which is a strong indicator of effective regularization.

DropPath Rate: $d\in \{0,0.1,0.15,0.25\}$. DropPath (stochastic depth) provides regularization by randomly dropping entire residual blocks during training. No DropPath has moderate stability but lower peak accuracy. DropPath = 0.1 achieves higher peak accuracy but paradoxically has worse stability. DropPath = 0.15 achieves better accuracy with substantially better stability. The rate increase from 0.1 to 0.15 provides stronger regularization:

• At $d=0.1$: The last block has a 10% chance of being dropped per training sample. The effective network depth varies from 9 to 10 blocks.
• At $d=0.15$: The last block has a 15% chance of being dropped. The model must learn more redundant representations to compensate, improving generalization.

At $d=0.25$, the regularization becomes excessive: the last block is dropped 25% of the time, and the second-to-last block 22% of the time. This creates too much variance in the forward pass, making gradient updates noisy and potentially slowing convergence.

Warmup Epoch: $W\in \{5,10,15\}$. The warmup length controls how gradually the learning rate ramps up from 0 to the initial value. Ten-epoch warmup achieves first-70% accuracy at epoch 12, compared to epoch 61 for five-epoch warmup. Despite the longer warmup, 10-epoch warmup converges to high accuracy faster than baselines.

The purpose of warmup is to allow BN running statistics to stabilize before aggressive optimization begins and prevent early gradient explosions from destabilizing the randomly initialized model.

With $W=5$, the learning rate reaches its maximum by epoch 5. BN running statistics has not yet converged. The high-variance early training wasted 30 epochs before the model recovered to a usable state.

With $W=10$, the learning rate reaches 0.1 only at epoch 10. This gives BN running statistics sufficient time to stabilize.

Extending warmup to $W=15$ will provide even more BN stabilization time but at the cost of five fewer epochs of effective training under the cosine schedule.

Gradient Clipping: ${\parallel g\parallel }_{\max }\in \{\mathrm{None},0.5,1.0,2.0\}$. Gradient clipping with ${\parallel g\parallel }_{\max }=1.0$ is to prevent large gradient updates from destabilizing BN statistics. By capping the gradient norm, it ensures that no single batch can cause a dramatic weight shift that would invalidate the current BN running statistics. No clipping experiences crashes that correlate with large gradient norms during the WeightedRandomSampler-induced batch composition changes. Clipping = 1.0 eliminated all post-warmup crashes.

EMA Decay: $\rho \in \{0,0.99,0.999\}$. Exponential Moving Average maintains a smoothed copy of model parameters used for evaluation. No EMA achieves 92.8% at both best and final epoch. With $\rho =0.999$, the EMA parameters change by only 0.1% per step, effectively averaging the last 1000 steps. With $\rho =0.99$, the EMA averages over 100 steps, making it more responsive to recent training progress but less stable against crashes. With $\rho =0.9999$, the averaging window would be 10,000 steps, which is too slow for a 90-epoch training run, meaning the EMA model would never fully converge.

Learning Rate: $\mathrm{lr}\in \{0.05,0.1,0.2\}$. The initial learning rate controls the step size of gradient descent.

• At $\mathrm{lr}=0.05$: The gradient updates are half the magnitude, making BN statistics more stable but slowing convergence. The model would need 180 epochs to reach the same level of optimization as $\mathrm{lr}=0.1$ in 90 epochs. With early stopping, the model may stop before reaching its peak.
• At $\mathrm{lr}=0.1$: The gradient updates are large enough for fast convergence but small enough to not destabilize BN. The cosine annealing schedule ensures that the learning rate decreases smoothly, preventing the sudden accuracy drops seen in step-decay schedules.
• At $\mathrm{lr}=0.2$: The gradient updates are too large, causing the model to overshoot optimal minima. Even with gradient clipping at 1.0, the effective learning rate will be 2× too large, leading to training instability.

The learning rate of 0.1 is the standard for SGD with momentum on skeleton-based recognition tasks and is validated by our experiments.

3.3. Ablation Study

We analyze the contribution of each architectural and training module by examining the effect of its removal. Table 5 summarizes the ablation results.

Table 5. Module ablation study (bone stream, NTU60 X-Sub). Each row removes one module from the full configuration.

Configuration	Acc	$\mathbf{\Delta }$	Primary Effect
Full BPA-STGCN	93.7%		–
Architectural Module Ablation
PartitionAttention (PA)	93.0%	−0.7%	Fine-grained accuracy loss
TemporalPyramidPooling (TPP)	92.8%	−0.9%	Temporal pattern loss
PA − TPP	92.3%	−1.4%	Both above combined
JointAttention (JA)	92.6%	−1.1%	Joint-level focus loss
Training Technique Ablation
Mixup ($\alpha =0$)	94.0%	+0.3%	Stability decrease
Gradient Clipping	93.5%	−0.2%	Crash risk increase
DropPath	92.8%	−0.9%	Overfitting increase
EMA	92.3%	−1.4%	Crash protection loss
BN momentum (0.01 → 0.1)	94.0%	+0.3%	Instability increase

Ablation: Remove Partition Attention (PA)

Removing the Partition Attention module (reverting to standard graph convolution without body-part-specific weighting) results in an accuracy drop of 0.7% (93.7% → 93.0%), confirming that PA contributes to accuracy.

PA learns sample-specific importance weights for four body partitions (torso, left arm, right arm, legs) through a squeeze-activation mechanism. Without PA, all 25 joints are treated uniformly in the graph convolution, and the model must learn body-part-level discrimination implicitly through the adjacency matrix and channel-wise topology refinement.

The 0.7% accuracy loss is concentrated in fine-grained actions that require body-part-level discrimination:

• Writing (−10.5% without PA): Without PA, the model cannot explicitly focus on torso posture differences between “writing” and “typing”, both of which have similar right-arm trajectories.
• Touch head (−17.4% without PA): The model loses the ability to upweight the arm partition for touch-related actions.
• Clapping (−12.7% without PA): The model cannot distinguish clapping (hands together) from rub hands (hands apart) by focusing on hand position relative to torso.

PA is applied only to the last three STGCN blocks because early blocks learn low-level motion patterns shared across body parts, while later blocks encode action-specific semantics where body-part discrimination is most valuable. This placement minimizes parameter overhead while maximizing discriminative benefit.

Ablation: Remove Temporal Pyramid Pooling (TPP)

Replacing TPP with standard global average pooling (GAP) results in an accuracy drop of 0.9% (93.7% → 92.8%). The stability is not affected, as TPP is only applied during the pooling phase after all STGCN blocks.

GAP collapses the temporal dimension into a single vector, which discards all temporal structure within the sequence. However, TPP instead captures multi-scale temporal dynamics.

The 0.9% accuracy loss from removing TPP is concentrated in actions with distinctive temporal structures:

• Vomit (−10.5% without TPP): “Vomit” has a three-phase temporal structure (bending → retching → recovering). TPP’s two-segment pooling captures the bending/recovering phases, while four-segment pooling captures the rapid retching motions within the middle segment. GAP averages all three phases into a single representation, blurring the temporal dynamics.
• Take off shoe (−6.5% without TPP): This action has a temporal progression (bend → grasp → pull → release). TPP’s multi-scale pooling captures both the overall progression (1-segment) and the individual phases (four-segment).
• Typing (−6.5% without TPP): “Typing” has periodic rapid keystroke cycles. TPP’s four-segment pooling captures the periodic pattern, while GAP averages over all cycles, losing the temporal periodicity.

TPP adds only 0.07M parameters while providing multi-scale temporal context. The negligible parameter overhead means the accuracy gain comes from representational improvement rather than increased model capacity, confirming that temporal structure preservation is valuable for skeleton-based recognition.

Ablation: Remove Joint Attention (JA)

Removing Joint Attention results in an accuracy drop of 1.1% (93.7% → 92.6%). This is the largest single-module contribution, confirming that joint-level attention is a critical component.

JA learns per-joint attention weights based on the feature context. Unlike PA which groups joints into four partitions, JA learns individual importance weights for all 25 joints. This finer granularity allows the model to focus on specific joints that are most relevant to each action.

The 1.1% accuracy loss from removing JA is expected to affect:

• Play phone (−15.6% without JA): JA focuses on wrist and hand joints, which are the primary discriminators for phone-related actions.
• Typing (−11.2% without JA): JA focuses on the right wrist joint, capturing the keystroke motion pattern.
• Touch head (−25.0% without JA): JA focuses on the hand-to-head trajectory, which is the key discriminator from other touch actions.

JA and PA are complementary: JA provides fine-grained, per-joint attention, while PA provides coarse-grained, body-part-level attention. Removing both will compound the losses, as shown in the PA − TPP configuration.

Ablation: Remove Mixup

Removing Mixup is expected to increase peak accuracy by 0.3% (93.7% → 94.0%) but decrease stability, which manifested as stable epochs from 34 to 15, crash epochs from 0 to 2. This is the only ablation where removing a component improves accuracy, highlighting the accuracy–stability trade-off.

Without Mixup, the training task allows the model to fit the training data more tightly. This produces higher peak accuracy but with increased overfitting risk.

The stability decrease comes from two sources:

• Loss of BN augmentation: Without Mixup, BN running statistics are estimated from clean inputs only, making them less robust to distribution shifts during validation.
• Sharper decision boundaries: Without the smoothing effect of interpolated samples, the model’s decision boundaries are sharper and more sensitive to small input perturbations, leading to higher variance in validation accuracy.

Ablation: Remove Gradient Clipping

Removing gradient clipping results in a small accuracy drop but a notable stability decrease. This confirms that gradient clipping primarily contributes to stability rather than accuracy.

Without gradient clipping, large gradient norms during the warmup-to-cosine transition or during later training can cause weight updates that destabilize BN running statistics. Even with $m=0.01$, a sufficiently large weight update can shift the feature distribution enough to cause a temporary accuracy drop.

The minimal accuracy impact indicates that gradient clipping rarely activates during normal training. It functions as a safety net rather than a regularizer. This is the ideal behavior for a stability mechanism: invisible during normal operation, critical during edge cases.

Ablation: Remove DropPath

Removing DropPath results in an accuracy drop of 0.9% (93.7% → 92.8%) and a moderate stability decrease. Without DropPath, all STGCN blocks are active during every training step, and the full-depth network is more prone to overfitting.

The 0.9% accuracy drop is larger than the PA ablation, suggesting that DropPath’s regularization effect is more impactful than PA’s discriminative improvement. This is consistent with the observation that regularization techniques collectively contribute more to BPA-STGCN’s performance than architectural innovations.

Ablation: Remove EMA

Removing EMA results in the largest accuracy drop (1.4%, 93.7% → 92.3%) among all ablations. The stability is moderately affected.

The 1.4% accuracy drop is the combined effect of losing both mechanisms. Without EMA, the model will rely entirely on the main model’s single-epoch snapshot, which is more susceptible to batch-specific noise.

The stability metrics are not affected by EMA removal because EMA does not influence the main model’s training dynamics; it only maintains a separate shadow copy. The decrease in stable epochs reflects that without EMA, the main model’s validation accuracy is more variable, with fewer epochs exceeding 80%.

Ablation: Revert BN Momentum to 0.1

Reverting BN momentum from 0.01 to 0.1 while keeping all other components is expected to increase peak accuracy by 0.3% (93.7% → 94.0%) but dramatically reduce stability. This is the most extreme accuracy–stability trade-off among all ablations.

With $m=0.1$, BN running statistics adapt 10× faster to the evolving feature distribution. This has two effects:

• Accuracy benefit: Faster statistical adaptation means the normalization layer can track the changing feature distribution more closely, reducing the train–eval distribution mismatch. This allows the model to fit the training data more tightly, leading to higher peak accuracy.
• Stability cost: The faster adaptation also means that BN running statistics are more sensitive to individual batch composition. A single batch with skewed class distribution can shift the running statistics by 10%, causing a cascade of destabilization.

3.4. Fine-Grained Action Improvements

Table 6 lists the 17 action classes improved by more than 3% over the baseline.

Table 6. Action classes improved by BPA-STGCN over baseline ($\Delta >$ +3%).

Action	TE-STGCN	BPA-STGCN	$\mathbf{\Delta }$	Key Module
Touch head	55.1	72.5	+17.4	PA
Clapping	58.3	71.0	+12.7	PA
Writing	38.8	49.3	+10.5	PA + TPP
Shake head	82.2	91.3	+9.1	PA
Cheer up	84.8	91.7	+6.9	PA
Take off shoe	69.6	76.1	+6.5	TPP
Typing	58.3	64.9	+6.5	PA + TPP
Vomit	81.5	87.7	+6.2	TPP
Wipe face	80.1	85.1	+5.1	PA + TPP
Touch back	85.5	90.2	+4.7	PA
Salute	87.7	91.7	+4.0	PA
Point at other	84.1	88.0	+4.0	PA
Brush teeth	75.7	79.3	+3.6	PA
Cross hands	90.2	93.8	+3.6	PA
Hopping	92.4	95.7	+3.3	TPP
Pointing	72.8	76.1	+3.3	PA
Walk towards	94.6	97.8	+3.3	TPP

The improvements validate the effectiveness of both PA and TPP:

• Partition Attention: “Touch head” (+17.4%) and “Clapping” (+12.7%) benefit from PA’s ability to focus on the arm and torso partitions. “Writing” (+10.5%) improves because PA learns to attend to torso posture differences from “typing”.
• Temporal Pyramid Pooling: “Vomit” (+6.2%) benefits from TPP’s multi-phase temporal modeling (bending → retching → recovering). “Take off shoe” (+6.5%) and “Walk towards” (+3.3%) gain from TPP’s ability to capture temporal progression.
• PA + TPP synergy: “Typing” (+6.5%) and “Wipe face” (+5.1%) benefit from both modules: PA focuses on the right arm partition, while TPP captures the periodic temporal pattern of typing keystrokes.

4. Discussion

4.1. Accuracy–Stability Trade-Off

Our experiments reveal a fundamental trade-off between peak accuracy and training stability in skeleton-based GCNs. The key insight is that BatchNorm’s running statistics serve a dual role: they normalize features but are sensitive to batch composition. Reducing BN momentum stabilizes training by reducing sensitivity to batch composition, but also slows the adaptation of normalization statistics to the evolving feature distribution. Mixup smooths decision boundaries but may blur distinctions between similar actions. Gradient clipping prevents destabilizing updates but caps the gradient signal needed for escaping sharp minima.

BPA-STGCN navigates this trade-off by combining architectural innovations (PA, TPP) that improve fine-grained discrimination with a training protocol that ensures reliable convergence. The result is a model that achieves 93.7% accuracy, while being dramatically more stable.

4.2. When to Use Stability Training

Our results suggest that the stability-first protocol is most beneficial in the following scenarios:

• Small batch sizes ($B\le 16$): Where BN statistics are most volatile.
• Unstable streams (joint, joint_vel): Where BN crashes are frequent.
• Production deployment: Where reproducible, crash-free training is critical.
• Resource-constrained settings: Where re-training due to crashes is costly.

For research settings where peak accuracy is the sole priority and training instability is acceptable, the configuration (BN mom = 0.1, no Mixup) may be preferred.

4.3. Cross-View and Cross-Setup Benchmarks

The evaluation in the present version is restricted to the Cross-Subject benchmark. We acknowledge that Cross-View and Cross-Setup results would provide direct evidence of whether Partition Attention remains robust across camera viewpoints and sensor distances. Because the anatomical partition mask is defined in the skeleton’s own joint space, and because the input is zero-centered and torso-length normalized, the partition grouping itself is viewpoint-invariant. We therefore expect the relative contribution of PA and TPP to carry over to Cross-View/Cross-Setup. Verifying this with full training runs on both benchmarks is planned as immediate future work.

4.4. Unconstrained Environments

NTU RGB+D is captured in a controlled lab setting with fixed cameras. Actors perform prescribed actions largely in place. In unconstrained environments where actors move freely, two additional factors arise. The first one is the larger viewpoint variation, which is what the partition design can tolerate because it is defined in skeleton space. Then, the second one is more frequent and longer self-occlusions, which challenge the skeleton tracker itself rather than the classifier. Our tracking-state-based frame selection and interpolation are designed with this in mind, but a quantitative evaluation on an in-the-wild benchmark remains future work.

4.5. Transferability of the Partition Matrix

The partition matrix M is hard-coded to the Kinect v2 25-joint layout, which raises the question of whether the grouping strategy transfers to other common skeleton topologies. The design principle is to group joints by anatomical part with roughly balanced sizes, attach the “core” to the torso, and mirror left/right limbs. For OpenPose 18-joint (COCO) skeletons, the same four partitions map to that—torso: 8 joints, left arm: 4, right arm: 4, legs: 6. For the 17-joint Human3.6M/COCO topology, the analogous mapping is obtained by dropping the hand-tip joints. Only the binary mask M and the output dimension of $W_2$ change.

4.6. Limitations

BPA-STGCN has several limitations. First, the 25-joint skeleton’s lack of finger joints remains a fundamental bottleneck for hand-centric actions, such as writing, typing and reading. Second, the stability protocol introduces a small accuracy penalty. Third, only the bone stream has been trained with the best accuracy; multi-stream results are expected to be increased. The quantitative results focus on the single bone stream for two reasons. The first one is the diagnostic clarity. The goal is to verify specific modules PA and TPP and training techniques, which requires the controlled single-stream setting, while multi-stream ensembling improves all methods roughly additively and would obscure the attribution. The second one is the resource constraints. Our stability study required 90-epoch runs for each of the configurations, which is only feasible on a single 8GB GPU with one stream. Multi-stream fusion is absolutely possible with our architecture. Future work will address these limitations through finger joint augmentation, adaptive BN momentum scheduling, and multi-stream training.

5. Conclusions

We presented BPA-STGCN, a skeleton-based action recognition framework that introduces Partition Attention, which adopts a Squeeze-and-Excitation-style for body-part-aware feature learning, Temporal Pyramid Pooling adapting temporal-segment and pyramid-pooling concepts to skeleton feature maps for multi-scale temporal representation, and a stability-first training protocol for reliable training. On NTU RGB+D 60 and NTU120, BPA-STGCN achieves 93.7% and 90.9% with 60 epochs above 80% accuracy. The model improves 17 of 60 action classes by more than 3%, including substantial gains on fine-grained actions. Our ablation studies reveal that architectural innovations and stability training contribute complementarily, and that BPA-STGCN achieves the best accuracy–stability trade-off. Future work will explore adaptive BN momentum scheduling, selective Mixup application, multi-stream training, and finger joint augmentation to further improve fine-grained action recognition.